Mixed Reality-Enhanced Digital Twin for Adaptive Photovoltaic Shutter Operation: A Pilot Study

Abstract

Adaptive Photovoltaic Shutter (APS) is an innovative shading device that integrates photovoltaic materials into its slats, enabling it to convert sunlight into electricity while adaptively adjusting frame and slat angles to optimize multiple performance objectives. Due to the complexity of its mechanical system and operation, an advanced digital twin (DT) is essential for effective monitoring and control. Current research integrates mixed reality (MR) with DT to provide a more immersive and intuitive management experience. An Arduino-based IoT system collects real-time data from the APS, uploads it to the cloud, and enables visualization through a MR application. Within the MR interface, users can observe the status of the physical APS and control it by interacting with its digital counterpart. A survey was conducted to evaluate the DT's usability and user engagement. The results suggest that the MR-DT can enhance user engagement and support anomaly detection tasks, although it is associated with longer task initiation time and perceived interaction latency. These findings highlight both the potential and limitations of MR-enhanced DT in APS operation.

Keywords

Mixed reality digital twin data visualization shading device interaction design

Introduction

Adaptive façades are capable of altering their configuration based on predefined strategies to optimise building performance, such as reducing glare and enhancing indoor visual comfort (Böke et al., 2020a; Tabadkani et al., 2019; Tabadkani et al., 2021). A well-known early example is the Arab World Institute in Paris, where photo-sensitive apertures dynamically responded to changes in external daylight. However, due to the complexity and high maintenance demands of its mechanical system, this adaptive façade eventually ceased to function and remains inoperable (FILT3RS, filt3rs.net). Recently, adaptive shading devices have evolved by incorporating photovoltaic (PV) materials into their slats, enabling simultaneous solar energy harvesting and environmental adaptability (Orhon, 2016). However, this integration also brings complexities which make operation and maintenance even more challenging.

Digital Twin (DT) technology has emerged as a potential solution to address such challenges (Angelova et al., 2024). Initially proposed by Grieves in 2002 in the context of Product Lifecycle Management (Grieves, 2005), DT is now widely defined as a digital representation of a physical asset, synchronised through continuous data exchange. ISO 23247 series establishes general principles and attributes for DT. The ISO 23247 series provides a standardised framework outlining DT attributes and principles. DTs can be categorised as monitoring, simulation, or operational twins (Khan et al., 2022). While conceptually related to Cyber-Physical Systems (CPS) (Böke et al., 2020b), DTs typically emphasise higher visual fidelity and data integration (Davari et al., 2022). Although previous studies have applied DTs for fault detection in PV systems (Jain et al., 2020), and CPS for façade monitoring (Böke et al., 2022), no research to date has explored the use of DTs for real-time operation of adaptive façades—especially those incorporating PV components.

Furthermore, the visualisation and interaction aspects of DTs have received limited attention. Conventional DTs are often presented via web-based dashboards (Haag & Anderl, 2018). However, when a digital replica and its real-time sensor data are complex, their information can be challenging and difficult to interpret intuitively. To overcome this problem, XR (Extended Reality) offers a promising solution. XR, an umbrella term encompassing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), represents a set of innovative technologies that revolutionize how users interact with digital systems and the physical world (Zhao et al., 2023). By vividly rendering virtual models and enabling intuitive interaction with digital objects, XR bridges the divide between the physical and digital realms. VR creates a fully immersive, artificial environment, whereas AR overlays virtual elements onto the physical world. MR extends AR by enabling users to interact with virtual objects using their hands and allows these objects to dynamically engage with the real environment (Zhao et al., 2023). These three XR technologies each have distinct features (Schmidt & Yigitbas, 2024; Zhao et al., 2023) and have been applied to various industrial sectors (Cantoral-ceballos, 2022). In the AEC industry, preliminary technical studies have used XR to overlay hidden facility information from BIM or existing models onto the real environment, providing visualization and step-by-step guidance to support inspection and maintenance (Haroon & Yucel, 2026).

Prior research has investigated CPS that have visualized IoT sensor data from built environments in XR (Blanco-novoa et al., 2020; Dias et al., 2025; Natephra & Motamedi, 2019; Stark et al., 2020; Zhao et al., 2022). However, studies reaching the level of sophistication associated with DTs (Carvalho, 2021; Kunnen et al., 2020; Tu et al., 2023) remain limited. VR was considered unsuitable in this scenario because it does not offer direct feedback from the physical world which is a disadvantage for on-site inspection (Zhao et al., 2022). In contrast, MR holds great potential to enhance DT performance by enabling more intuitive data visualization and user interaction with both the physical world and the digital environment.

Therefore, this study employs an Adaptive Photovoltaic Shutter (APS) as the testbed which automatically adjusts its position to optimize solar energy conversion. Then, this study utilizes an MR-based Digital Twin (MR-DT) to monitor and control it, to improve the operational efficiency and clarity of APS data visualisation. A pilot validation experiment indicates that the proposed system offers a more intuitive and efficient interface compared with CPS-based or conventional DT approaches. The study contributes to the growing field of smart building technologies by showcasing the feasibility of MR-DT integration for adaptive façade management (Figure 1).

Figure 1.

The research scope of this study.

Following this introduction, the paper is organized into four main sections. Section 2 presents a literature review, exploring DT and XR integrated DT systems for building management. Section 3 outlines the research methodology, comprising three subsections: the first provides an overview of the mixed-reality digital twin (MR-DT) development; the second details the physical APS and its integrated IoT sensors; and the third describes the procedures for developing the digital twin. Section 4 presents and discusses the results of the validation experiment. Finally, Section 5 summarizes the key conclusions of the study and highlights opportunities for future research.

Literature Review

DT Applications for Facility and Building Management

DT technology has been extensively explored in the context of construction management, safety assessment (Jiang et al., 2024; Zhu et al., 2025), and facility operations (Krayem et al., 2026; Mahmoodian et al., 2022). Its applications span various infrastructure types, including roads (Lu et al., 2025), bridges (Fawad et al., 2024; Mahmoodian et al., 2022), sewer systems (Hartmann et al., 2025), and time-varying water level tanks (Edington et al., 2023).

In PV system management, DTs have been employed to optimize tasks such as maximum power point tracking (MPPT) (Angelova et al., 2024; Shukla et al., 2026). In certain instances, reinforcement learning has been integrated into DT frameworks to improve the control logic of PV systems (Artetxe et al., 2023). Another study proposes a genetic algorithm-based approach to accurately parameterize unknown photovoltaic systems by optimizing seven key parameters from measured power data, enabling high-fidelity digital twins and PV power simulations with limited prior system information (Tool & Simulation, 2020). A more focused research area concerns Building-Integrated Photovoltaics (BIPV), where studies have explored transitioning from Building Information Modeling (BIM) to DT (Wang et al., 2024) and the use of artificial neural networks in DT modelling for predicting BIPV performance (Sanz-saiz, 2022).

Beyond BIPV systems, DTs have been widely applied in building operations and maintenance (Biagini et al., 2024; Coupry et al., 2021; de Barros Lima, 2020; Eneyew et al., 2022; Hu et al., 2026; Jiang et al., 2023; Walczyk & Andrzej, 2024). Several studies conceptualize DTs as the fusion of BIM and Internet of Things (IoT) data streams (Eneyew et al., 2022; Li & Fang, 2025; Walczyk & Andrzej, 2024), with some proposing further extensions toward the Metaverse (Noroozinejad Farsangi et al., 2024), and blockchain (Adu-amankwa & Rahimian, 2025). Notably, targeted efforts have focused on applying DTs to façade management. For example, one study conducted four rounds of experimentation, collecting over 25,000 sensor readings to develop and evaluate a prototype DT for monitoring a dynamic façade element in an office building (Khajavi et al., 2019).

From the perspective of building performance, DT applications are primarily driven by two key objectives: enhancing energy efficiency and improving indoor environmental quality (Arowoiya et al., 2024; Donkers, 2022; Gordo-Gregorio et al., 2025; Saadatifar et al., 2024). Additionally, structural health monitoring has gained significant attention, with DTs enabling real-time diagnostics and anomaly detection in load-bearing systems (Jiang et al., 2023; Sun et al., 2025; Xie et al., 2025).

XR Enhanced DTs

The integration of XR and DT was first proposed in 2019 (Ke et al., 2019) and has since garnered growing interest, particularly in industrial manufacturing. Researchers developed an AR-DT for laser-based manufacturing processes (Stavropoulos et al., 2021), a VR-DT for brake disc production (Catalano et al., 2022), immersive DT models for wind turbine assembly (Yildiz et al., 2020), and an MR-DT for smart home planning and automation (Yigitbas & Karch, 2025). XR-DT frameworks have also been adopted for training future workforces in smart manufacturing contexts (García et al., 2022).

Beyond the manufacturing sector, the integration of XR and DT has been extended to various fields including construction progress monitoring (Alizadehsalehi & Yitmen, 2021), real-time control of multi-tank systems (Jeršov, 2020), bridge asset management (Fawad et al., 2024), and even wetland maintenance (Aheleroff et al., 2021). These applications highlight the extensive potential of XR-DTs for operational management and real-time data visualization across various physical domains.

Under the category of integrating XR with DT for interfaces, management and operation, robot arms and cranes have been the two main application domains. Robot arm related DTs were visualized and controlled through XR including AR (Hasan et al., 2022; Li et al., 2022; Stark et al., 2020; Suzuki et al., 2022; Yang et al., 2025; Yigitbas & Engels, 2023), VR (Havard et al., 2019; Kuts et al., 2022; Luu et al., 2025; Oyekan et al., 2019; Pizzagalli et al., 2021; Yang et al., 2022), and MR (Macci et al., 2022; Ravi et al., 2021; Tu et al., 2021; Tu et al., 2023). These studies demonstrate how immersive interfaces can significantly enhance the control, monitoring, and training functionalities associated with DT.

More closely related to this study, a smaller but growing body of work has explored XR-DT in building and infrastructure management. For example, MR-based DT environments have been used to visualize real-time indoor air quality (Siegele et al., 2021), while AR- DT systems have been applied to map and assess thermal comfort and HVAC control (Mohammadi et al., 2024). In addition, VR-DTs have been employed to display indoor temperature and users’ electrodermal activity (Jiang et al., 2020). Beyond building applications, VR-DTs have also been proposed for water and wastewater utility management (Carvalho, 2021), and MR-DTs have been introduced for industrial plant maintenance during the operational phase (Kunnen et al., 2020).

However, despite these promising developments, the total number of studies in this particular subdomain remains limited. Specifically, there is a notable lack of research that applies MR-DT to the management of adaptive building components—such as photovoltaic-integrated façades—which often involve high degrees of physical-digital synchrony and user interaction complexity. This gap forms the basis of the current investigation.

Methodology

The Overview

The methodology includes three main parts: introducing the physical APS; creating the digital twin; validating the digital twin. The detailed design process and technical challenges associated with the APS will be addressed in forthcoming publications. However, selected key aspects of the physical APS relevant to this study are briefly presented in Section 3.2. Section 3.3 introduces the development of the digital twin, with particular emphasis on its data architecture and operational logic. Section 3.4 outlines the user-based validation study involving thirty participants. By comparing the MR-DT with a conventional web-based DT interface, the validation process focuses on usability and user engagement, to assess the effectiveness of the proposed MR-DT.

The Physical APS

It should be noted that the primary focus of the present study is MR-DT rather than APS. Although the detailed mechanical design of the APS is beyond the scope of this paper, additional technical specifications are available in the authors’ related publications. As depicted in Figure 2a, the physical APS model is constructed from an aluminium alloy and mounted within a wooden window frame.

Figure 2.

The physical APS (a), close-up of illuminance sensors on the APS (b), The push-rod on the back of the APS (c).

The APS operates with dual-axis rotation. First, a door closer is affixed to the base of the wooden frame, and two hinges are installed, each with one leaf attached to the frame and the other to the APS. When the door closer extends or retracts, the entire APS framework pivots around these vertical hinges. For the second axis of rotation, the APS louver comprises seven slats. One slat is equipped with a compact device containing four photoresistor modules, while the remaining six slats are fitted with CIGS membranes. An electronic pushrod, consisting of a motor-driven rod and a control unit (Figure 2c), serves as a linear actuator to facilitate movement. The linear motion is converted into rotary motion via a sophisticated mechanical design within the APS. This mechanism enables seven slats to rotate around their horizontal axes. Electricity generated by CIGS membranes on these slats is stored in six AA batteries.

Rotational movements in both dimensions respond to real-time environmental conditions and are guided by predefined operational strategies. Various strategies have been developed for managing the APS, intentionally designed to achieve single or multiple objectives, such as maximizing the efficiency of converting sunlight into electricity. In this study, the APS aims to optimize sunlight capture by rotating either the entire system or its seven slats, positioning the CIGS membranes perpendicular to the incoming sunlight. The four photoresistor modules (Figure 2b) monitor the environment every two seconds, enabling the APS to determine the optimal rotation adjustments. The author employs an electrical relay to remotely shut down or restart the APS. As previously noted, due to the scope of this paper, further details on the APS's operational logic are not covered here.

In addition to the four photoresistor modules, this project incorporates a diverse array of sensors (Table 1). These include environmental sensors that measure parameters such as temperature, humidity, and illuminance, as well as sensors that monitor the APS's activities, such as rotational degrees in two dimensions. Two Arduino boards, serving as compact computing units, function as the central control system for the APS. While Arduino boards can be used to create data loggers for monitoring PV systems in accordance with IEC standards (Yang et al., 2022), in this study, all data is transmitted directly to the cloud via ESP8266 chipsets, cost-effective Wi-Fi microchip boards.

Table 1.

Main Sensors for Monitoring the Physical APS.

Sensor Module	Functionality
KY-018	Photoresistor, to obtain an analogy value of light intensity
BH1750	Measuring Illuminance Sensor
MPU6050	Measuring Rotation Degree of Slats (Rotation-A)
HW-526	Measuring Rotation Degree of the whole APS (Rotation-B)
CN3791 3065	Solar Charger module
ESP8266	WIFI Module, sending data to the cloud

The Digital Twin

The Digital Replica, Dataflow, and the MR Headset

A geometric digital model of the physical APS is first developed using SketchUp and exported in FBX format. The model is then imported into the Unity game engine for further development. In addition, Mixed Reality Toolkit (MRTK), an open-source cross-platform MR development framework provided by Microsoft, is employed to support spatial mapping and interface design.

At the conclusion of this stage, a digital replica is created and linked to real-time IoT data. This replica incorporates motion parameters and environmental variables that define its dynamic behaviour. As these parameters are continuously updated through IoT inputs, the digital replica synchronously reflects the real-time state of the physical APS, including both whole-system and individual slat rotations.

Infrastructure maintenance typically encompasses three stages: inspection, condition assessment, and maintenance (Mahmoodian et al., 2022). Accordingly, the MR-DT is designed to support these core functions. It provides multi-dimensional information on APS operation, including slat anomaly detection, electricity generation, and more. The system supports both remote monitoring and on-site verification modes. In addition, it enables a multi-user collaborative environment (Xie et al., 2020), allowing multiple users to observe the APS simultaneously through the MR interface, while control authority is restricted to a primary user. This facilitates remote collaboration among stakeholders for system evaluation and development.

The overall data flow of the MR-DT is illustrated in Figure 3. Sensor data is first collected and transmitted from Arduino boards to the ESP8266 gateway. The data is then forwarded via TCP to a router and subsequently transmitted using the MQTT protocol to the OneNet IoT platform developed by China Mobile Communications Corporation. After processing and management, the data is relayed to Tencent Cloud using the same MQTT protocol. Finally, the MR-DT retrieves the IoT data from the Tencent Cloud server via the HTTP protocol. Through this pipeline, real-time APS states are synchronised with the MR environment.

Figure 3.

The MR-DT data architecture.

Conversely, users can issue control commands through the MR interface to operate the physical APS. These commands are encoded as digital signals and transmitted through the reverse data flow path, where they are ultimately converted into control signals to actuate the electrical relay of the APS.

The MR-DT is deployed on the HoloLens 2, a mixed reality head-mounted display. It features a 2 K resolution display (2048 × 1536) with a 3:2 aspect ratio. Users can interact with holographic content through hand gestures and voice commands (Zhao et al., 2022; Zhao et al., 2023). Although HoloLens 2 is used in this study, the MR-DT framework is also compatible with other MR devices, such as Magic Leap headsets.

The Design of the Graphical User Interface (GUI)

After activation, the HoloLens 2 performs spatial mapping of the physical environment and establishes a XR space anchored to the real world. The MR-DT interface is then launched and rendered within this spatially registered environment. The GUI of the MR-DT serves as an essential interface between the physical and virtual APS, enabling seamless interaction between the two domains. Accordingly, the interface is designed to optimise usability and user experience.

The visualization of incoming sensor data is carefully designed and implemented. Numeric data is represented through dynamic, colour-changing renderings. Numerical values are encoded using dynamic colour transitions. For example, when the AA batteries reach sufficient charge levels, the corresponding virtual battery model changes colour from red to green, while its fill level increases to reflect the real-time state (Figure 4b). Users can resize and reposition virtual elements through hand gestures to suit their preferences.

Figure 4.

The screenshot of CPS (a); The graphical user interface of this MR-DT (b).

In addition, through a bidirectional data flow mechanism, users can control the physical APS via a virtual “Start/Stop” button within the MR environment. Overall, gesture-based interaction enables intuitive inspection and control of both the physical and digital representations of the APS.

The Validation Test

Following the development of the APS and the MR-DT, their functionality and stability were evaluated over a continuous week. In addition to the MR-DT, an identical DT was implemented in a web browser using HTML5. While the underlying data flow and user interface components (Figure 4) remained consistent, the information was presented through a web-based interface rather than in the MR environment. This Web-DT was developed as a benchmark for evaluating the usability and user experience of the MR-DT. Besides Web-DT and MR-DT, a traditional CPS was also tested (Figure 4a).

Thirty undergraduate students from an architecture programme (aged 18–20) participated in the experiment. They were randomly assigned to three groups of equal size, with one group using the CPS, another group using the Web-DT, and the other using the MR-DT, following a between-subject experimental design. All participants had prior exposure to smart architecture and XR technologies at a basic level, ensuring comparable familiarity across groups. The participant group was balanced in terms of gender. All participants provided informed consent for data collection and the use of anonymized data for research and publication purposes. Ethical approval was not required according to institutional guidelines for this type of study.

The MR-DT was accessed via a HoloLens, while the Web-DT was operated on a desktop with a 24-inch monitor at a resolution of 1920 × 1080 (Full HD). During the experiment, both systems were connected to the same wireless network provided by a cellular (4G/5G) mobile hotspot, ensuring consistent network conditions for data transmission. Real-time data exchange with the IoT platform was conducted under identical network settings for both interfaces.

The validation test was conducted over a single day under stable weather conditions within the digital laboratory, where the APS was deployed. Prior to the test, both the MR-DT and Web-DT were calibrated to ensure synchronized reception of identical data streams. The experiment consisted of two independent parts: task performance evaluation across different DT interfaces and assessment of user engagement.

In the first part, participants were instructed to interact with their assigned DT interface and complete three operational tasks. The first task required participants to launch the system, monitor APS status after reviewing the displayed parameters, and exit the interface (see Figure 5). Participants were encouraged to finish the task as soon as possible. Task completion time was defined as the duration from system launch to task completion.

Figure 5.

A screenshot of the scene of MR-DT in use (a); A test participant is operating the MR-DT system (b).

The second task evaluated system data correspondence. Upon receiving a synchronised signal from the experimenter, participants recorded three APS parameter values displayed in the interface. These values were compared with ground-truth data from the IoT platform to do a match check. The task was repeated five times for each interface. A trial was considered exact match if the recorded values are the same as the backend data. The exact-match ratio was calculated as the proportion of successful trials.

The third task simulated a system fault by physically disconnecting a rotation sensor. Participants were required to detect and report the anomaly as quickly as possible.

For the second section, participants completed a 5-point Likert scale questionnaire (Table 2). The questionnaire design is mainly based on previous pieces of literature (Georgiou & Kyza, 2017; Zhao et al., 2023). The questionnaire assessed two user engagement dimensions: usability and interest. Each item was rated on a five-point scale ranging from negative (−2), slightly negative (−1), neutral (0), slightly positive (+1), to positive (+2). The overall engagement score for each DT system was calculated as the average of the two dimensions.

Table 2.

The Questionnaire Regarding the Subjective Evaluation.

Indicators	ID	Questions
Usability	Q1	It is easy for me to operate the DT.
	Q2	The data presented in the DT is intuitive.
	Q3	The interaction of using this DT is satisfying.
	Q4	The operation of the DT is smooth.
	Q5	The GUI of DT is designed not confusing at all.
Interest	Q6	I like this experience because it is novel.
	Q7	I like this experience because it is interesting.
	Q8	This experience entertains my mind.
	Q9	This experience stimulates my mind.
	Q10	I want to continue this experience after the test.

Results and Discussions

Task Performance

Table 3 presents the results of the task performance evaluation. It should be emphasised that the current study focuses on the usability of the MR-DT rather than the intrinsic performance of the APS itself.

Table 3.

Objective Test Results of two DTs.

Indicators	CPS	Web-DT	MR-DT
Average time consumption (minutes)	5 ± 1.2	3.5 ± 0.8	5.2 ± 2.0
Forward data exact-match ratio (backend vs display; 5 trials)	100%	100%	100%
Rate of successful anomaly detection	60%	80%	90%

Regarding task completion time, participants using the MR-DT required approximately 1.7 min longer than those using the Web-DT to launch the system and interpret the displayed information. Several participants reported noticeable latency in the MR interface, particularly during hand-gesture interactions to exit MR-DT.

The forward data matching ratio was defined as the proportion of trials (out of five) in which the parameter values recorded by participants exactly matched the ground-truth data from the IoT backend. This metric primarily evaluates the forward data flow from physical sensors to the DT visualisation. All three systems achieved a 100% matching ratio. However, this result does not reflect delays in the reverse control flow (from MR commands to physical APS actuation), which was identified as a limitation of the experimental design in the current pilot study. The reverse latency, primarily caused by hand-gesture recognition delays and network transmission, was subjectively reported but not quantitatively captured in the task design. The underlying causes are difficult to isolate, as some cases may result from incorrect gesture inputs, while others may be associated with network congestion.

In terms of anomaly detection, both the MR-DT and Web-DT groups showed high success rates (90% and 80%, respectively), with only one or two participants failing to identify the simulated sensor disconnection. In contrast, the traditional CPS group exhibited a lower detection rate (60% compared to 80% and 90%).

Overall, the objective results indicate that the MR-DT holds a higher detection rate (90% vs 80%) in anomaly detection, despite its longer initiation time. The primary limitation observed was the time-consuming nature of the MR interface, which can largely be attributed to participants’ unfamiliarity with the HoloLens 2 (even though ten of them had prior experience). Furthermore, the relatively simple DT model in this pilot study may not have fully demonstrated the potential advantages of MR's intuitive visualization capabilities.

It should also be noted that the experimental tasks in this study were relatively simple and primarily focused on basic monitoring and data recording. Such tasks can be performed using conventional Web-DT, which may limit the extent to which the advantages of MR-DT in spatial understanding, intuitive interaction, and complex system inspection—are fully demonstrated. This may partly explain why the MR-DT did not show a clear advantage in usability.

Engagement Level

Table 4 presents the subjective evaluation results based on the 5-point Likert scale questionnaire. With respect to usability, the Web-DT received a slightly higher average score (1.52) than the MR-DT (1.28). However, when it comes to user interest and engagement, the MR-DT significantly outperformed the Web-DT, achieving an average interest score of 1.76 compared to 1.06.

Table 4.

Subjective Evaluation Results of Using two DTs and CPS.

Systems	Usability						Interest						Overall Score
Systems	Q1	Q2	Q3	Q4	Q5	Avg.	Q6	Q7	Q8	Q9	Q10	Avg.	Overall Score
CPS	−0.3	−0.9	−0.3	0.3	0.3	−0.45	−1.8	−1.9	−1.5	−1.6	−1.7	−1.7	−1.08
Web-DT	1.6	1.6	1.5	1.3	1.8	1.52	1.2	1.1	1.2	1.3	0.5	1.06	1.29
MR-DT	1.0	1.5	1.5	1.2	1.2	1.28	1.8	2.0	1.9	2.0	1.1	1.76	1.52

Consequently, the overall engagement score for the MR-DT (1.52) was 0.23 points higher than that of the Web-DT (1.29). This suggests that, from the users’ subjective perspective, the immersive and interactive nature of the MR-DT made the experience more engaging and preferable, despite its slightly lower usability rating in this pilot test.

It is worth noting that the traditional CPS received markedly negative scores in both usability and interest, highlighting the clear advantages of digital-twin-based interfaces over conventional CPS.

Conclusions

Solar energy harvesting remains a prominent research area within the built environment, with particular emphasis on adaptive façades (Juaristi et al., 2020). The integration of PV materials into façade components has emerged as a promising research direction. Given the complexity of their mechanical systems, DTs are becoming increasingly important for the operation and management of PV-integrated adaptive façades. When combined with MR, DT platforms become more intuitive and user-friendly, reducing the need for on-site inspection and supporting APS management tasks.

This study details the integration of MR to enhance the DT platform for the monitoring and operation of the APS. The resulting MR-DT also enables users to remotely monitor and control the APS. The validation involved thirty participants, and the results suggest that the MR-DT can assist users in operating the APS. Furthermore, subjective evaluations indicate that the MR-DT is preferred in terms of user engagement.

As this study represents a pilot implementation, the current system is relatively simplified and several limitations remain. Future work will focus on extending the APS into a waterproof version and deploying it on the building façade. Therefore, additional sensors, such as rainfall sensors, will be integrated. Consequently, the MR-DT will be expanded to incorporate more environmental and operational parameters to enable more comprehensive system management. The system will also be extended to façade scale deployment involving multiple APS units, allowing simultaneous visualization of large-scale façade information and rapid identification of anomalies.

Secondly, responsive DT for remote collaboration, inspection, and operation in the environment of mixed reality also will be more deeply taken into account (Baumgartner et al., 2022). In such scenarios, multiple users from different locations will be able to interact with and manage the virtual APS simultaneously, facilitating collaborative decision-making and system diagnosis. This will also enable more intuitive discussions among stakeholders through direct interaction within the MR environment. In addition, it may support improved post-deployment services for maintenance engineers following system commercialisation.

In future work, a range of computational techniques will be incorporated. Latency issues (Caiza & Sanz, 2022) will be addressed using fog computing approaches (Jesudian, 2018), while data security will be enhanced through blockchain integration (Pincheira et al., 2023). Reinforced learning algorithms will be applied to this project to train a neural network based on the operation data. The operational logic can then self-evolve to maximize the objectives across diverse external environments. By analysing real-world APS behaviour, the system will be able to learn underlying operational patterns and predict future performance trends. Moreover, large language models, such as ChatGPT and Gemini, will be integrated into the MR-DT to support system interaction and operation (Khelifi & Morris, 2024).

Finally, future studies will include more diverse participant groups to improve the generalisability of the findings. While the current study focuses on architecture students in a controlled environment, real-world deployment requires validation involving broader stakeholders. Future experiments will involve facility managers and engineers to better evaluate the usability and practical applicability of the proposed MR-DT system.

Footnotes

Abbreviations

Acknowledgements

Mr. Meihua Chen and Mr. Qiang Lu contributed on assembling the APS model.

Funding

This work is supported by Jiangsu Youth Research Funding under the grant of BK20210869.

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

ORCID iD

Shenghuan Zhao

Author Biography

Shenghuan Zhao is an architectural engineering researcher, a certified LEED AP BD+C professional, and also an international educator. He received his master and PhD from Politecnico di Milano, specializing in sustainable building technologies and digital architectural systems. Dr. Zhao's research integrates internet of things, extended reality, and AI-enhanced design tools to advance sustainability and intelligence in the built environment. He has published extensively in peer-reviewed journals, and actively contributes as an associate editor for Sustainable Buildings (eISSN: 2492-6035), as well as an editorial board member for Journal of Integrated Design and Process Science (eISSN: 1875-8959). His work bridges computational design, climate-responsive architecture, and occupant behaviors, reflecting a holistic vision for future-oriented, low-impact design strategies.

References

Adu-amankwa

N. A.

Rahimian

(2025). Blockchain-Enabled digital twins in the built environment : Socio-technical barriers to post- construction integration. Proceedings of Smart and Sustainable Built Environment Conference Series, Lille, France (pp. 106–115).

Aheleroff

Zhong

R. Y.

(2021). Digital twin as a service (DTaaS) in industry 4.0: An architecture reference model. Advanced Engineering. Informatics, 47(October 2020), 101225. https://doi.org/10.1016/j.aei.2020.101225

Alizadehsalehi

Yitmen

(2021). Digital twin-based progress through reality capture to extended reality technologies (DRX). Smart and Sustainable Built Environment. 12(1), 200–236. https://doi.org/10.1108/SASBE-01-2021-0016

Angelova

D. D.

Fernández

D. C.

Godoy

M. C.

(2024). A review on digital twins and its application in the modeling of photovoltaic installations. Energies, 17(5), 1227. https://doi.org/10.20944/preprints202401.1585.v1

Arowoiya

V. A.

Moehler

R. C.

Fang

(2024). Digital twin technology for thermal comfort and energy efficiency in buildings: A state-of-the-art and future directions. Energy and Built Environment, 5(5), 641–656. https://doi.org/10.1016/j.enbenv.2023.05.004

Artetxe

Uralde

Barambones

Calvo

Martin

(2023). Maximum power point tracker controller for solar photovoltaic based on reinforcement learning agent with a digital twin. Mathematics, 11(9). https://doi.org/10.3390/math11092166

Baumgartner

Brägger

Koebel

Scheidegger

Çöltekin

(2022). Visually annotated responsive digital twins for remote collaboration in mixed reality environments. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, V-4-2022, 329–336. https://doi.org/10.5194/isprs-annals-v-4-2022-329-2022

Biagini

Bongini

Marzi

(2024). From bim to digital twin. Iot data integration in asset management platform. Journal Of Information Technology in Construction, 29(May), 1103–1127. https://doi.org/10.36680/j.itcon.2024.049

Blanco-novoa

Fraga-lamas

Vilar-montesinos

M. A.

(2020). Towards the internet of augmented things : An open-source framework to interconnect IoT devices and augmented reality systems. Proceedings, 42(1). https://doi.org/10.3390/ecsa-6-06563

10.

Böke

Denz

Suwannapruk

Vongsingha

(2022). Active, passive and cyber- physical adaptive façade strategies : A comparative analysis through case studies. Journal of Facade Design and Engineering, 10(2), 1–18. https://doi.org/10.47982/jfde.2022.powerskin.01

11.

Böke

Knaack

Hemmerling

(2020a). Automated adaptive façade functions in practice - case studies on office buildings. Automation in Construction, 113(June 2019), 103113. https://doi.org/10.1016/j.autcon.2020.103113

12.

Böke

Knaack

Hemmerling

(2020b). Prototype of a cyber-physical façade system. Journal Of Building Engineering, 31(April), 101397. https://doi.org/10.1016/j.jobe.2020.101397

13.

Caiza

Sanz

(2022). Digital twin for monitoring an industrial process using augmented reality. In 17th Iberian Conference on Information Systems and Technologies (CISTI) (pp. 22–25). ITMA.

14.

Cantoral-ceballos

J. A.

(2022). Extended reality applications in industry 4 . 0 . – A systematic literature review. Telematics and Informatics, 73(September 2022). https://doi.org/10.1016/j.tele.2022.101863

15.

Carvalho

W. O.

(2021). BIM and AM to Manage Critical and Relevant Water and Wastewater Utilities Assets vol 3. Springer International Publishing. https://doi.org/10.1007/978-3-030-51295-8

16.

Catalano

, et al. (2022). A digital twin-driven and conceptual framework for enabling extended reality applications: A case study of a brake discs manufacturer. Procedia Computer Science, 200(2019), 1885–1893. https://doi.org/10.1016/j.procs.2022.01.389

17.

Coupry

Noblecourt

Richard

Baudry

Bigaud

(2021). BIM-Based digital twin and XR devices to improve maintenance procedures in smart buildings: A literature review. Application Science, 11(15). https://doi.org/10.3390/app11156810

18.

Davari

Shahinmoghadam

Motamedi

Poirier

(2022). Demystifying the Definition of Digital Twin for Built Environment. The 9th International Conference on Construction Engineering and Project Management.

19.

de Barros Lima

(2020). A web-based application to integrate building management system sensor data and building information model data to support facility management tasks . Eindhoven University of Technology.

20.

Dias

J. A.

Campos

N. P.

Boavida

(2025). Towards an IoT framework for the new generation of XR applications. IEEE Access, 13, 68726–68752. https://doi.org/10.1109/ACCESS.2025.3560547

21.

Donkers

(2022). Parallel Intelligence in Semantic Digital Twins : An Interactive Decision-Support System for Indoor. IEEE 2nd International Conference on Digital Twins and Parallel Intelligence (DTPI). IEEE.

22.

Dorian

E. G. R.

Björn

Henrik

Christof

(2020). A genetic algorithm approach as a self-learning and optimization tool for PV power simulation and digital twinning. Energies, 13. https://doi.org/10.3390/en13246712

23.

Edington

Dervilis

Ben Abdessalem

Wagg

(2023). A time-evolving digital twin tool for engineering dynamics applications. Mechanical System and Signal Processing, 188(November 2022), 109971. https://doi.org/10.1016/j.ymssp.2022.109971

24.

Eneyew

D. D.

Member

G. S.

Miriam

A. M.

(2022). Towards smart building digital twins: BIM and IoT data integration. IEEE Access, 10, 130487–130506. https://doi.org/10.1109/ACCESS.2022.3229370

25.

Fawad

Salamak

Chen

Koris

(2024). Development of an immersive digital twin framework to support infrastructure management : a case study of bridge asset health monitoring. 41st International Symposium on Automation and Robotics in Construction (ISARC 2024), Lille, France. https://doi.org/10.22260/ISARC2024/0105

26.

FILT3RS, “Solar diaphragms at IMA,” filt3rs.net.

27.

García

Á.

Bregon

Martínez-Prieto

M. A.

(2022). Towards a connected digital twin learning ecosystem in manufacturing: Enablers and challenges. Computers & Industrial Engineering, 171(September 2022). https://doi.org/10.1016/j.cie.2022.108463

28.

Georgiou

Kyza

E. A.

(2017). The development and validation of the ARI questionnaire: An instrument for measuring immersion in location-based augmented reality settings. International Journal Of Humuan Computer Studies, 98(February 2017), 24–37. https://doi.org/10.1016/j.ijhcs.2016.09.014

29.

Gordo-Gregorio

Alavi

Edwards

D. J.

Forcada

Guéna

(2025). An occupant-centric approach on digital twins for building management. Building Research & Information, 54(February 2025), 59–78. https://doi.org/10.1080/09613218.2025.2463676

30.

Grieves

(2005). Product lifecycle management: The new paradigm for enterprises. International Journal Of Product Development, 2(1–2), 71–84. https://doi.org/10.1504/IJPD.2005.006669

31.

Haag

Anderl

(2018). Digital twin – proof of concept. Manufacture Letter, 15(Part B), 64–66. https://doi.org/10.1016/j.mfglet.2018.02.006

32.

Haroon

Yucel

(2026). Design and evaluation of a mixed reality system for facility inspection and maintenance †. Buildings, 16(2). https://doi.org/10.3390/buildings16020425

33.

Hartmann

, et al. (2025). Digital-Twin-Based management of sewer systems: Research strategy for the KaSyTwin project. Water (Switzerland), 17(3), 1–13. https://doi.org/10.3390/w17030299

34.

Hasan

S. M.

Lee

Moon

Kwon

Jinwoo

Lee

(2022). Augmented reality and digital twin system for interaction with construction machinery. Journal Of Asian Architecture Building Enginerring, 21(2), 564–574. https://doi.org/10.1080/13467581.2020.1869557

35.

Havard

Jeanne

Lacomblez

Baudry

(2019). Digital twin and virtual reality: A co-simulation environment for design and assessment of industrial workstations. Production & Manufacturing Research, 7(1), 472–489. https://doi.org/10.1080/21693277.2019.1660283

36.

Liu

Chang

(2026). Integrating digital twin technologies for maintenance 4.0 in the building industry: A review and conceptual framework. Building Environment, 288(January 2026). https://doi.org/10.1016/j.buildenv.2025.113997

37.

Jain

, et al. (2020). A digital twin approach for fault diagnosis in distributed photovoltaic systems. IEEE Transactions on Power Electronics, 35(1), 940–956. https://doi.org/10.1109/TPEL.2019.2911594

38.

Jeršov

(2020). Development of Digital Twin in Extended Reality With Unreal Engine 4. Tallinn University of Technology.

39.

Jesudian

J. J. S. S.

(2018). An Integration of IOT and Mixed Reality Using Fog Computing Concepts. Texas State University.

40.

Jesús

Nuria

M-C.

Carlos

S-S.

(2022). BIPV modeling with artificial neural networks: Towards a BIPV digital twin. Energies, 15(11). https://doi.org/10.3390/en15114173

41.

Jiang

Tobia

Lawther

Branchaud

Bednarz

(2020). Double vision: Digital twin applications within extended reality. ACM SIGGRAPH 2020 Appy Hour, SIGGRAPH 2020 (pp. 10–12). https://doi.org/10.1145/3388529.3407313

42.

Jiang

Shi

Wang

Pan

(2023). Intelligent control of building fire protection system using digital twins and semantic web technologies. Automation Construction, 147(March 2023). https://doi.org/10.1016/j.autcon.2022.104728

43.

Jiang

Liu

Chen

Liu

Ding

(2024). Early-warning of unsafe hoisting operations: An integration of digital twin and knowledge graph. Development Of Built Environment, 19(October 2024). https://doi.org/10.1016/j.dibe.2024.100490

44.

Juaristi

Loonen

Isaia

Gómez-Acebo

Monge-Barrio

(2020). Dynamic climate analysis for early design stages: A new methodological approach to detect preferable adaptive opaque façade responses. Sustainable Cities and Society, 60(April), 102232. https://doi.org/10.1016/j.scs.2020.102232

45.

Xiang

Zhang

Zuo

(2019). A enhanced interaction framework based on VR, AR and MR in digital twin. In Procedia CIRP (Vol. 83, pp. 753–758). Elsevier B.V. https://doi.org/10.1016/j.procir.2019.04.103

46.

Khajavi

S. H.

Motlagh

N. H.

Jaribion

Werner

L. C.

Holmstrom

(2019). Digital twin: Vision, benefits, boundaries, and creation for buildings. IEEE Access, 7, 147406–147419. https://doi.org/10.1109/ACCESS.2019.2946515

47.

Khan

L. U.

Saad

Niyato

Han

Hong

C. S.

(2022). Digital-Twin-Enabled 6G: Vision, architectural trends, and future directions. IEEE Communications Magazine, 60(1), 74–80. https://doi.org/10.1109/MCOM.001.21143

48.

Khelifi

Morris

(2024). Mixed reality IoT smart environments with large language model agents. 2024 IEEE 4th Int. Conf. Human-Machine Syst. ICHMS 2024 (pp. 1–7), https://doi.org/10.1109/ICHMS59971.2024.10555610

49.

Krayem

Younes

Ghali

Jradi

(2026). A digital twin solution for optimizing productivity and energy performance in hybrid ventilated office space. Energy Conversion Management, 349(November 2025), 120880. https://doi.org/10.1016/j.enconman.2025.120880

50.

Kunnen

Adamenko

Pluhnau

Loibl

Nagarajah

(2020). System-based concept for a mixed reality supported maintenance phase of an industrial plant. In Procedia CIRP (pp. 15–20). Elsevier B.V. https://doi.org/10.1016/j.procir.2020.03.096

51.

Kuts

, et al. (2022). Digital twin as industrial robots manipulation validation tool. Robotics, 11(5), 1–24. https://doi.org/10.3390/robotics11050113

52.

Zheng

Pang

Lee

C. K. M.

(2022). AR-assisted digital twin-enabled robot collaborative manufacturing system with human-in-the-loop. Robotics and Computer Integrated Manufacturing, 76(July 2021), 102321. https://doi.org/10.1016/j.rcim.2022.102321

53.

Q. A.

Fang

(2025). An automatic ontology construction approach for BIM-IoT semantic integration in intelligent operation and maintenance of high-speed railway stations. Engineering Research Express, 7(4). https://doi.org/10.1088/2631-8695/ae1c92

54.

Yin

Xie

Pan

Wang

Brilakis

(2025). Development of A trustworthy AI-supported digital twin framework for road operation and maintenance. Smart Construction, 2025(December 2024), 1–28. https://doi.org/10.55092/sc20250001

55.

Luu

Nguyen

Tran

Ding

Kua

Hoang

(2025). Enhancing real-time robot teleoperation with immersive virtual reality in industrial IoT networks. The International Journal of Advanced Manufacturing Technology, 139, 6233–6257. https://doi.org/10.1007/s00170-025-16236-w

56.

Macci

Carf

Mastrogiovanni

(2022). Mixed reality as communication Medium for human-robot collaboration. 2022 International conference on robotics and automation (ICRA). IEEE (pp. 2796–2802).

57.

Mahmoodian

Shahrivar

Setunge

Mazaheri

(2022). Development of Digital Twin for Intelligent Maintenance of Civil Infrastructure. Sustain., 14(14). https://doi.org/10.3390/su14148664

58.

Mohammadi

Assaf

Assaad

R. H.

(2024). Real-time spatial-temporal mapping and visualization of thermal comfort and HVAC control by integrating immersive augmented reality technologies and IoT-enabled wireless sensor networks : Towards immersive human-building interactions. Journal Of Building Engineering., 94(February), 109887. https://doi.org/10.1016/j.jobe.2024.109887

59.

Natephra

Motamedi

(2019). Live data visualization of IoT sensors using augmented reality (AR) and BIM. ISARC 2019 (pp. 632–638).

60.

Noroozinejad Farsangi

Shehata

A. O.

Rashidi

Ghassempour

Mirjalili

(2024). Transitioning from BIM to digital twin to metaverse. Frontiers In Built Environment, 10(December), 1–7. https://doi.org/10.3389/fbuil.2024.1486423

61.

Orhon

(2016). A review on adaptive photovoltaic facades. In: Solar TR2016 International Solar Conference & Exhibition, pp.463–470.

62.

Oyekan

J. O.

, et al. (2019). The effectiveness of virtual environments in developing collaborative strategies between industrial robots and humans. Robotics Computer Integrated Manufacturing, 55(June 2018), 41–54. https://doi.org/10.1016/j.rcim.2018.07.006

63.

Pincheira

Vecchio

Antonelli

(2023). Smarttwin: A blockchain-based software framework for digital twins using IoT. In BLOCKCHAIN 2022 (pp. 67–77). Springer International Publishing. https://doi.org/10.1007/978-3-031-21229-1_7

64.

Pizzagalli

Kuts

Bondarenko

Otto

(2021). Evaluation of virtual reality interface interaction methods for digital twin industrial robot programming and control, a pilot study. ASME International Mechanical Engineering Congress and Exposition, Proceedings (IMECE), (pp. 1–9). https://doi.org/10.1115/IMECE2021-69408

65.

Ravi

K. S. D.

M. S.

Medina Ibáñez

Hall

D. M.

(2021). Real-time digital twin of on-site robotic construction processes in mixed reality. Proceedings of the International Symposium on Automation and Robotics in Construction (pp. 451–458). https://doi.org/10.22260/isarc2021/0062

66.

Saadatifar

Sawyer

A. O.

Byrne

(2024). Occupant-Centric digital twin: A case study on occupant engagement in thermal comfort decision-making. Architecture, 4(2), 390–415. https://doi.org/10.3390/architecture4020022

67.

Schmidt

Yigitbas

(2024). Development and Usability Evaluation of Transitional Cross-Reality Interfaces. Proc. ACM Hum.-Comput. Interact, Honolulu, USA, 8. https://doi.org/10.1145/3664637

68.

Shukla

A. K.

Shukla

K. N.

Dixit

(2026). IoT-enabled solar PV systems: Real-time monitoring and efficiency optimization. Results In Optics, 22(November 2025), 100949. https://doi.org/10.1016/j.rio.2025.100949

69.

Siegele

Penna

Riedl

(2021). Visualizing building energy measurement data in mixed reality applying B.I.M. AVR 2021 (pp. 247–253). https://doi.org/10.1007/978-3-030-87595-4_18

70.

Stark

Haffner

Drahoš

Leskovský

(2020). Using augmented reality and internet of things for control and monitoring of mechatronic devices. Electronics, 9(8). https://doi.org/10.3390/electronics9081272

71.

Stavropoulos

Papacharalampopoulos

Siatras

Mourtzis

(2021). An AR based digital twin for Laser based manufacturing process monitoring. Procedia CIRP, 102, 258–263. https://doi.org/10.1016/j.procir.2021.09.044

72.

Sun

Jayasinghe

Sidiq

Shahrivar

Mahmoodian

Setunge

(2025). Approach Towards the Development of Digital Twin for Structural Health Monitoring of Civil Infrastructure: A Comprehensive Review. Sensors, 25(1). https://doi.org/10.3390/s25010059

73.

Suzuki

Karim

Xia

Hedayati

Marquardt

(2022). Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces. Conference on Human Factors in Computing Systems - Proceedings, New Orleans, LA, USA. Association for Computing Machinery. https://doi.org/10.1145/3491102.3517719

74.

Tabadkani

Roetzel

H. X.

Tsangrassoulis

(2021). Design approaches and typologies of adaptive facades: A review. Automation in Construction, 121(November 2020), 103450. Elsevier B.V. https://doi.org/10.1016/j.autcon.2020.103450

75.

Tabadkani

Valinejad Shoubi

Soflaei

Banihashemi

(2019). Integrated parametric design of adaptive facades for user’s visual comfort. Automation In Construction, 106(May), 102857. https://doi.org/10.1016/j.autcon.2019.102857

76.

Autiosalo

Ala-laurinaho

Yang

Salminen

Tammi

(2023). TwinXR: Method for using digital twin descriptions in industrial eXtended reality applications. Frontiers In Virtual Reality, 4(1019080). https://doi.org/10.3389/frvir.2023.1019080

77.

Autiosalo

Jadid

Tammi

Klinker

(2021). A mixed reality interface for a digital twin based crane. Applied Science, 11(20), 1–19. https://doi.org/10.3390/app11209480

78.

Walczyk

Andrzej

(2024). Building information modeling and digital twins for functional and technical design of smart buildings with distributed IoT networks — Review and new challenges discussion. Future Internet, 16(7). https://doi.org/10.3390/fi16070225

79.

Wang

Song

Bao

Xiang

(2024). From BIM to digital twin in BIPV : A review of current knowledge. Sustainable Energy Technology. Assessments, 67(May), 103855. https://doi.org/10.1016/j.seta.2024.103855

80.

Xie

, et al. (2020). Interactive multi-user 3D visual analytics in augmented reality. International Symposium on Electronic Imaging 2020 (pp. 1–6). https://doi.org/10.2352/ISSN.2470-1173.2020.13.ERVR-363

81.

Xie

, et al. (2025). AIoT-powered building digital twin for smart firefighting and super real-time fire forecast. Advanced Engineering. Informatics, 65(Part A), 103117. https://doi.org/10.1016/j.aei.2025.103117

82.

Yang

, et al. (2022). Extended Reality Application Framework for a Digital-Twin-Based Smart Crane. Applied Science, 12(12). https://doi.org/10.3390/app12126030

83.

Yang

Zheng

Feng

Ala-Laurinaho

Tammi

(2025). A digital twin-driven industrial context-aware system: A case study of overhead crane operation. Journal Of Manufacturing. System, 78(2024), 394–409. https://doi.org/10.1016/j.jmsy.2024.12.006

84.

Yigitbas

Engels

(2023). Enhancing Robot Programming through Digital Twin and Augmented Reality. Proceedings of the 56th Hawaii International Conference on System Sciences. on-line.

85.

Yigitbas

Karch

(2025). Immersive smart home planning and automation with mixed reality and digital twin technologies. In PETRA ‘25 (pp. 137–145). Corfu Island. https://doi.org/10.1145/3733155.3733198

86.

Yildiz

Møller

Bilberg

(2020). Sciencedirect virtual factory : Digital twin based integrated factory simulations. In Procedia CIRP (pp. 216–221). Elsevier B.V. https://doi.org/10.1016/j.procir.2020.04.043

87.

Zhao

Dong

Tian

Chen

(2023). Comparing three XR technologies in reviewing performance-based building design: A pilot study of façade fenestrations. Computer Animated Virtual Worlds, 34(6), 1–20. https://doi.org/10.1002/cav.2139

88.

Zhao

Pan

Gao

Cheng

(2022). Integrating internet of things and mixed reality to teach performance-based architectural design: A case study of shading devices. Education Information Technology. 27, 9125–9143. https://doi.org/10.1007/s10639-022-10998-6 Integrating

89.

Zhu

Sun

(2025). A digital twin-based green construction management method for prefabricated buildings. Digital Twin, 2(3). https://doi.org/10.12688/digitaltwin.17830.1