E-Commerce price index prediction with time series mining and automated machine learning

Abstract

The Consumer Price Index (CPI) serves as a critical economic indicator reflecting the overall prices of goods and services. While surveys of household consumption behaviors have been reliable to date, the survey-based method of compiling the CPI has recently been under strain. Price measurement has become more complex, thereby increasing demands on the data needed to attain accuracy, coverage, and timeliness.¹ This study introduces an integrated data pipeline incorporating a time-sensitive composite similarity model tailored for multivariate time-series correlation analysis to generate an alternative CPI utilizing price data from an e-commerce portal. The pipeline also offers an implementation of the preferred CPI forecasting model derived from an automated machine learning tool. The composite e-commerce index is 90.75% correlated with the official national food and non-alcoholic beverages CPI. Furthermore, ridge regression resulted in an R-squared value of 0.74 and an MAE of 1.269, hence proving that this novel approach can offer a robust complement to traditional methods for deriving a composite index and forecasting the CPI.

Keywords

Consumer Price Index E-Commerce Big Data Time-Series Machine Learning

1. Introduction

Governments have tracked consumer prices for decades by compiling the Consumer Price Index (CPI), the primary statistic used for gauging the aggregate price level in an economy. The CPI heavily influences the policy actions of central banks and governments. These actions include changing policy rates to keep inflation within a target range.

Generating this highly monitored data entails several data-gathering activities, such as phone interviews, in-person visits, or online inquiries to thousands of establishments. For example, the Philippine Statistics Authority (PSA) conducts the Retail Prices Survey of selected goods and services to gather prices of commonly consumed commodities and generate the monthly CPI. However, the need for prompt evidence-based policies and programs to address national economic issues and concerns has emphasized the demand for high-frequency and granular data, setting traditional data collection methods at a disadvantage in terms of timeliness and cost.²

On top of the tedious and costly process of compiling the CPI, the recent pandemic highlighted that the CPI may not be accurate during sudden shifts in spending patterns due to infrequent updates in weighting methodology.³ Traditional methods for compiling economic statistics, such as the CPI, have yet to account for sudden overnight transformations in the economy. These circumstances have presented unique challenges in compiling and interpreting these statistics, given the ongoing changes in economic conditions resulting from the pandemic and geo-political conflicts.

Given the limitations of the official CPI, big data in the form of web-scraped information has been seen to help address the issue of limited and untimely access to primary data inputs needed to compile the CPI, and promptly consider the changing consumption patterns. National Statistics offices of Australia, New Zealand, Malaysia, the Philippines, Vietnam, and Indonesia have explored web-scraped data from selected online marketplaces to generate an alternative CPI using classical forecasting techniques such as ARIMA, Exponential Smoothing, Holt Winter, Linear Trend, Exponential and Level Aggregation.⁴

With the advancements in Machine Learning and the availability of alternative datasets, this research endeavors to delve into novel methodologies to ascertain a complementary index capable of aiding statistical agencies in analyzing and forecasting the CPI. This study adopts Dynamic Time Warping (DTW) technique to discern composite commodity prices index and uncover key products whose price changes coincide with the movement of the official CPI, and Automated Machine Learning (AutoML) to anticipate price fluctuations with a one-month lead time.

2. Methodology

2.1 Dataset

Given the data security issues relevant to scraping e-commerce data, this study uses data requested directly from a leading e-commerce company in the Philippines. The dataset covers raw information from product postings on an e-commerce website from April 2021 to September 2023 (28 months). In addition, analysis is focused on the food and non-alcoholic beverages commodity group, given its weight of 37.75% of the CPI basket.¹ To evaluate the model, the generated e-commerce CPI is compared with monthly official food CPI figures published by the PSA.

The dataset features over 11 million price quotes, averaging around 5,500 quotes per day. It encompasses nearly half a million unique items distributed across over 16,000 product categories sourced from 1,633 online merchants. Within this expansive dataset, 181 food products are identified as relevant for analyzing consumer price trends, particularly within the context of CPI forecasting in the e-commerce domain.

2.2 Pipeline

As illustrated in Figure 1, this study introduces a comprehensive data pipeline tailored for generating and forecasting e-commerce CPI, which involves several key stages to ensure data quality and leverage advanced techniques for more accurate predictions. The pipeline begins with data pre-processing, including ingestion, validation, unit-based conversion, price standardization, outlier detection, and category recognition. This stage ensures that the e-commerce data is cleaned, normalized, and enriched to mitigate noise and inconsistencies.

Subsequently, time series mining technique, particularly time warping, is employed to capture temporal patterns and similarities between web-scraped prices and the official food CPI. Time warping allows for flexible time series alignment, accommodating variations in time intervals and ensuring robust pattern detection.

Finally, machine learning-based forecasting models are applied to the pre-processed and mined data, leveraging the DTW results with other ML techniques for accurate CPI predictions. This integrated approach within the data pipeline ensures a comprehensive and effective methodology for forecasting CPI components, combining traditional statistical methods and modern ML techniques for enhanced accuracy and reliability.

2.3 Pre-processing

In ingesting the data, the files from the e-commerce company are saved in a specific repository, and a script reads and consolidates the files. After which, the datasets are checked for completeness (i.e., no null data point) and redundancy.

2.4 Unit-based methodology

Given that the raw granular dataset only contains the item name, its category, size, selling price, merchant name, date collected and lacks quantity, each transaction was normalized using the size of the item. The size is measured by a metric unit and, hence, can be converted to a single unit system. The metric system units, gram and milliliter, were chosen, and the converted value was considered the daily base price.

2.5 Price standardization

To consider the pricing strategies of different merchants, prices were further standardized per day for each merchant by the formula below.

The average price of product i in a certain month, $P_{n i}$ , is calculated by taking the mean of $P_{d i j}$ , the daily base price of product i per merchant j, normalizing it by $J$ , the total number of merchants, and dividing by $D$ , the number of days that the product is sold by each specific merchant.

2.6 Outlier detection

As there are anomalies in the e-commerce pricing system,⁵ outliers were identified through the interquartile range (IQR). IQR is the range between the first and the third quartiles, namely Q1 and Q3: IQR = Q3 – Q1. The data points that fall below Q1 – 1.5 IQR or above Q3 + 1.5 IQR are considered outliers and, therefore, disregarded in the analysis.

2.7 Cleaning of categories

Product category names were cleaned using the Regular Expression (Regex) and Scikit-learn packages in Python to ensure general commodity groupings are observed. Backward imputations are used to fill in data gaps considering temporal dependency, as this assigns missing values based on the most recent observation in line with the study of Pavlovová.⁶

Given its significant share of the Philippine CPI basket, the food group primarily causes fluctuations in the overall CPI. For this reason and given the scope of the data available, this study focuses on food commodities. As proposed by Cenikj et al.,⁷ this study uses FOODB (https://foodb.ca/) database to filter food commodities from the raw dataset.

2.8 Time series mining

The DTW algorithm is known as an efficient method to measure the similarity between two sequences of time series data (Figure 2). Liang et al.⁹ and Grzejszczak et al.¹⁰ used DTW to effectively screen and adjust investment portfolios for investors. This research employs the same methodology to identify composite e-commerce products exhibiting patterns akin to the CPI movement.

With DTW, the sequences are warped in a non-linear fashion to match each other. The DTW minimizes distortion effects due to time-dependent movement by using an elastic transformation of time series data to recognize the similar phases between different patterns along time. The DTW algorithm is essentially an optimization implementation, as detailed in Figure 3.

DTW is subsequently employed to quantify the similarity between price time series, enabling the clustering of trend-related products. Through this clustering process, the selection of top products is guided by a cost function given by:

D T W (x, y) = \begin{matrix} m i n \\ π \end{matrix} \sqrt{\sum_{(i, j) \in π} d {(x_{i}, y_{j})}^{2}}

Where

π = [π_{0}, \dots, π_{K}]

is a path that satisfies the following properties:

it is a list of index pairs $π_{k} = (i_{k}, j_{k})$ with $0 \leq i_{k} < n$ and $0 \leq j_{k} < m$

$π_{0} = (0, 0)$ and $π = (n - 1, m - 1)$

For all $k > 0, π_{k} = (i_{k}, j_{k})$ is related to $π_{k - 1} = (i_{k - 1}, j_{k - 1})$ as follows:

\begin{aligned} i_{k - 1} \leq i_{k} \leq i_{k - 1} + 1 \end{aligned}

\begin{aligned} j_{k - 1} \leq j_{k} \leq j_{k - 1} + 1 \end{aligned}

To derive a composite index, the mean of the top 10% of commodities, as determined by the least cost or distance from the actual CPI, is computed. Considering the expected lower value of the composite index due to the unit-based methodology, linear regression is then utilized to reindex the composite index, yielding a Reindexed Composite Index (RCI) that aligns with the benchmark. Subsequently, the RCI is assessed against the benchmark, specifically the actual food CPI, and visualized to evaluate if it can capture prevailing market trends and fluctuations.

2.9 Forecasting

After generating the e-commerce index, commodities that significantly affect the index were identified as variables for forecasting. Various forecasting ML methodologies were explored to develop a model that can give a one-month ahead prediction of the CPI component. Deep learning models, such as recurrent neural network and long short-term memory, were initially considered. However, both models are deemed inappropriate, given that these are designed for more extended historical data to avoid overfitting.¹¹ Hence, the study employed regression algorithms, such as linear, decision tree (DT), random forest (RF), ridge, Huber, AdaBoost, k-Neighbors (KNN), and gradient boosting (GB) regression instead.

Each regression algorithm serves a distinct purpose in forecasting the e-commerce food CPI. Linear regression provides a simple, interpretable model by fitting a linear relationship between input features and the target variable. DT regression partitions the feature space into regions, making predictions based on the majority target class within each region. RF regression aggregates predictions from multiple decision trees to reduce overfitting and improve accuracy. Ridge regression introduces regularization to mitigate multicollinearity and overfitting by penalizing significant coefficients. Huber regression is robust to outliers by combining squared and absolute loss functions. AdaBoost regression sequentially builds an ensemble of weak learners, focusing on misclassified instances to improve model accuracy. The KNN makes predictions based on the proximity of data points, leveraging its non-parametric nature to handle complex and non-linear relationships. Meanwhile, GB regression constructs a sequence of models, each targeting the residual errors of the previous model, gradually improving prediction accuracy. Collectively, these algorithms offer a diverse toolkit for forecasting e-commerce CPI, each uniquely catering specific aspects of the dataset's complexity and characteristics.

In implementing these ML methodologies, this study uses an AutoML (Automated Machine Learning) tool named PyCaret. It is an open-source Python library that automates the end-to-end machine learning workflow, from data preparation to model deployment, allowing for efficient model comparison, hyperparameter tuning, and deployment. Using PyCaret's features, the researchers efficiently tested various algorithms for predicting the food CPI using e-commerce commodity data.

2.10 Validation

Each model is trained using cross-validation techniques for robustness, and grid search is used to adjust hyperparameters. Standard regression measures such as Root Mean Squared Error (RMSE), Mean Average Error (MAE), and R-squared are used to evaluate model performance. Cross-validation is used for the final model evaluation.

3. Limitations

The study's foremost challenges include a noisy dataset, the assumptions of similar patterns, and the computational intensity required for big data processing. Despite pre-processing efforts, e-commerce transaction data often contains inconsistencies, complicating DTW analysis and risking inaccurate pattern detection.^12,13 Moreover, given the unpredictable economic landscape, the DTW's assumption of similar patterns may not always hold and lead to inaccurate trend prediction. The required computational intensity for implementing the pipeline can also hinder performance with larger datasets. To address these challenges, rigorous pre-processing from data validation to commodity category recognition is applied. Additionally, parallel processing and optimization strategies are employed to enhance scalability and efficiency.

4. Results

4.1 Time series mining

The results of employing DTW to derive a composite e-commerce price index (CEPI) based on food commodity prices demonstrate promising correlations with the official food CPI, as shown in Figure 4. Over the last 12 months, the composite index has exhibited a positive correlation of 80.64% with the National CPI, suggesting a significant relationship between e-commerce food prices and consumer inflation trends within the timeframe. Moreover, extending the analysis to 28 months reveals a stronger correlation of at least 90.75%, underscoring the robustness and reliability of the index in capturing longer-term inflation dynamics. These findings highlight the efficacy of DTW in constructing a composite e-commerce index that can track fluctuations in food commodity prices, thereby providing valuable insights into consumer price trends and inflationary pressures. Further analysis of the Food CPI trend in the country's capital region (Capital) can also be undertaken noting its relatively low correlation to the CEPI in a 12-month timeframe at 60.87% as shown in Table 1.

Table 1.

Correlations of Composite E-Commerce Price Index (CEPI) with Official CPI.

Timeframe	National	Capital	Outside Capital
12-month	0.8064	0.6087	0.8213
28-month	0.9075	0.9039	0.9061

*All correlations with p-value < 0.05 – statistically significant relationship.

Source: Author's calculations.

The CEPI exhibits significant variability in its deviation from the official food CPIs. These findings, as shown in Table 2, suggest that while the e-commerce Price Index generally tracks official food CPI trends, there are considerable deviations, particularly relative to the food CPI outside the capital region of the Philippines, warranting further investigation into the factors driving these disparities and their implications for inflation analysis and forecasting.

Table 2.

CEPI deviations from the Official Food CPIs.

Food Indices	Maximum	Average
National	5.32	2.09
Capital	5.31	2.09
Outside Capital	5.42	2.10

Source: Author's calculations.

As shown in Table 3 and Figure 5, the CEPI demonstrates a slightly higher mean growth rate of 0.0072 and a higher standard deviation of 0.0192, implying potentially greater volatility in food commodity prices within the e-commerce domain. These results highlight the importance of considering e-commerce data alongside official food CPI metrics for a comprehensive understanding of food price dynamics, particularly given the observed differences in growth rates and volatility between official and e-commerce indices.

Table 3.

Descriptive Statistics of Official Food CPIs and the CEPI Growth Rates.

Measure	National	Capital	Outside Capital	CEPI
Mean	0.0060	0.0058	0.0060	0.0072
Min	−0.0121	−0.0144	−0.0113	−0.0301
25%	−0.0018	0.0004	−0.0016	−0.0027
50%	0.0061	0.0076	0.0048	0.0059
75%	0.0127	0.0116	0.0125	0.0117
Max	0.0272	0.0293	0.0298	0.0578
Standard Deviation	0.0104	0.0109	0.0108	0.0192

Source: Author's calculations.

4.2 Forecasting

The forecasting results revealed that ridge regression emerged as the top-performing model among various ML algorithms, as detailed in Table 4. With an R-squared of 0.74 and an MAE of 1.269, the model emerges to have the best predictive power and accuracy in forecasting the target variable.

Table 4.
ML Modelling Results.

Model MAE RMSE R-squared

Ridge Regression 1.269 1.553 0.736

Decision Trees Regression 1.417 1.689 0.508

Huber Regression 1.427 1.734 0.482

Gradient Boosting 1.511 1.717 0.194

AdaBoost 1.609 1.893 −0.309

Random Forest Regression 1.709 2.013 −0.348

K-Nearest Neighbor 2.108 2.2353 −1.057

Linear Regression 8.223 10.619 −30.752

Model	MAE	RMSE	R-squared
Ridge Regression	1.269	1.553	0.736
Decision Trees Regression	1.417	1.689	0.508
Huber Regression	1.427	1.734	0.482
Gradient Boosting	1.511	1.717	0.194
AdaBoost	1.609	1.893	−0.309
Random Forest Regression	1.709	2.013	−0.348
K-Nearest Neighbor	2.108	2.2353	−1.057
Linear Regression	8.223	10.619	−30.752

Source: Author's calculations.

The resulting forecasting pipeline was constructed using PyCaret. As detailed in Figure 6, the pipeline ensures that the ridge regression model performs optimally by leveraging advanced techniques such as power transformation, imputation, normalization, and model training. This robust pipeline is poised to deliver reliable forecasts along with more data training in the future.

The result suggests that the ridge regression algorithm captures the underlying patterns and relationships within the dataset, yielding reliable predictions with minimal error using an out-of-sample validation, as shown in Figure 7. The superior performance of ridge regression underscores its suitability for the forecasting task at hand and highlights its potential utility in similar predictive modeling scenarios.

The feature importance analysis, as shown in Figure 8, revealed that coffee products hold the highest significance level in forecasting the national food CPI, with a weight of 9.55%, followed by milk (5.45%), chocolate (4.55%), and several other staples. These results underscore the utility of ML methodologies in pinpointing critical commodities that can function as alternatives for gauging price dynamics, thereby enhancing the accuracy and efficiency of price monitoring strategies in the e-commerce context.

5. Conclusion

This study developed a comprehensive pipeline to derive a composite index based on e-commerce food commodity prices using DTW and a forecasting model to predict price movements of food and non-alcoholic beverages using online prices. The pipeline involved pre-processing the e-commerce data, including data cleaning, normalization, and outlier detection. Subsequently, DTW was applied to construct a composite index, aligning the time series of food commodity prices to capture temporal patterns and similarities. This composite index was then correlated with the official food CPI over different categories and timeframes to assess its effectiveness in tracking inflation dynamics. Multiple ML algorithms were also explored to forecast food CPI. Among these models, ridge regression provides the most accurate forecast of the official food CPI. By combining these steps into a coherent pipeline, the study provided a robust framework for deriving composite indices and predicting e-commerce price trends, offering an alternative or complementary online price index and valuable insights into consumer price dynamics within the e-commerce domain.

The derived composite e-commerce index is 90.75% correlated with the national food CPI, with an average deviation of 2.09. This result suggests that the e-commerce index captured the price dynamics inherent to the official food CPI. Notwithstanding, the CEPI also demonstrates a higher mean growth rate of 0.0072 and standard deviation of 0.0192 compared to the official indices, implying greater volatility in food commodity prices within the e-commerce domain. These results underscore the importance of considering e-commerce data alongside official food CPI metrics for a comprehensive understanding of food price dynamics, particularly given the observed differences in growth rates and volatility between official and e-commerce indices.

Furthermore, ridge regression was identified as the best forecasting model with an R-squared value of 0.74 and an MAE of 1.269 relative to higher errors derived from other ML models. The feature importance analysis also revealed that coffee products hold the highest significance level in forecasting the national food CPI, with a weight of 9.55%, followed by milk, chocolate, and several other key items.

In conclusion, integrating DTW with an AutoML pipeline presents a novel yet promising approach, offering a robust alternative or complement to traditional methods for deriving a composite index and forecasting the CPI. By leveraging DTW's ability to capture temporal patterns and AutoML's efficiency in model selection and feature engineering, this combined approach demonstrates the potential for enhancing the accuracy and reliability of CPI forecasts. Moving forward, statistical offices should prioritize further research and experimentation with DTW coupled with AutoML, exploring its applicability across diverse datasets and refining methodologies for optimal performance. Additionally, investing in training and capacity-building initiatives to equip staff with the necessary skills to implement and interpret these advanced techniques will be essential in harnessing the full potential of DTW and AutoML for CPI forecasting and economic analysis.

Footnotes

Acknowledgements

We would like to express our gratitude to our supervisor, Mr. Rossvern S. Reyes for his invaluable insights and assistance in editing the manuscript. We also wish to thank Department of Economic Statistics (DES) Senior Director Redentor Paolo M. Alegre, Jr. for his support to our team during the competition.

ORCID iD

Carmelita Esclanda-Lo

Notes

References

Sichel

Aizcorbe

de Haan

, et al. Modernizing the consumer price index for the 21st century [Internet]. Washington, DC: National Academies Press eBooks, 2022, Available from: https://doi.org/10.17226/26485.

Del Prado

Varona

Robles

, et al. Web scraping of prices of commodities included in the Generation of Consumer Price Index (CPI) for the National Capital Region (NCR). Philippines: UNECE [Internet], 2023. Available from: https://unece.org/documents/2023/05/presentations/web-scraping-prices-commodities-included-generation-consumer-price.

Reinsdorf

. Data disruption: The impact of COVID-19 on inflation measurement [Internet]. Washington, DC: IMF, 2020, Available from: https://www.imf.org/en/Blogs/Articles/2020/11/10/data-disruption-the-impact-of-covid-19-on-inflation-measurement.

Bernal

. United Nations economic and social commission for asia and the pacific. Thailand: Big Data for Economic Statistics, 2021, Issue No 28. 2021. Available from: https://www.unescap.org/sites/default/d8files/knowledge-products/Stats_Brief_Issue28_Big_data_for_economic_statistics_Mar2021.pdf.

Ramakrishnan

Shaabani

, et al. Anomaly detection for an E-commerce pricing system [Internet]. United States: KDD ‘19: Proceedings of the 25th ACM SIGKDD International Conference on knowledge discovery & data mining. 2019. Available from: https://doi.org/10.1145/3292500.3330748.

Pavlovová

. High frequency food prices: evidence from the Czech Republic [Internet]. Prague: Univerzita Karlova, Fakulta sociálních věd, 2017, Available from: https://dspace.cuni.cz/bitstream/handle/20.500.11956/150510/120398379.pdf?sequence=1&isAllowed=y.

Cenikj

Petelin

Seljak BK et al. SciFoodNER: Food Named Entity Recognition for Scientific text. Japan: 2022 IEEE International conference on big data (big data) [Internet]. 2022 Dec 17. Available from: https://doi.org/10.1109/bigdata55660.2022.10020459.

Ratanamahatana

Keogh

. Making time-series classification more accurate using learned constraints [Internet]. USA: Society for Industrial and Applied Mathematics, 2004, Available from: https://doi.org/10.1137/1.9781611972740.2.

Liang

Wang

. A novel time-sensitive composite similarity model for multivariate time-series correlation analysis. Basel, Switzerland: Entropy [Internet] 2021; 23: 731.

10.

Grzejszczak

Probierz

Galuszka

, et al. Dynamic time warping in Financial data – Modification of algorithm in context of stock market similarity analysis. Athens, Greece: European Research Studies [Internet] 2022; XXV: 967–79.

11.

Zohuri

. Deep learning limitations and flaws. New York, United States: Modern Approaches on Material Science [Internet] 2020; 2. Available from: https://doi.org/10.32474/mams.2020.02.000138.

12.

Jank

Shmueli

Dass

, et al. Statistical Challenges in eCommerce: Modeling dynamic and networked data. Maryland, United States: State of the art decision making tools in the information [Internet]. INFORMS, 2008, pp.31–54. Available from: https://doi.org/10.1287/educ.1080.0042.

13.

Tang

. Analysis and demand forecasting based on e-Commerce data [Internet]. Chengdu, China: IEEE, 2023, Available from: https://doi.org/10.1109/icaibd57115.2023.10206072.

14.

Herrmann

Tan

Webb

. Parameterizing the cost function of dynamic time warping with application to time series classification. Berlin, Heidelberg: Data Mining and Knowledge Discovery [Internet] 2023; 37: 2024–45.

15.

Müller

. Dynamic time warping. In: Information retrieval for music and motion [Internet]. Berlin, Heidelberg: Springer Science & Business Media; 2007. Available from: https://link.springer.com/chapter/10.1007/978-3-540-74048-3_4#citeas.