Abstract
This paper discusses integrating Unmanned Aerial Vehicle (UAV) data, Artificial Intelligence (AI), and Mixed Reality (MR) as a hybrid system for monitoring mine site rehabilitation. UAV systems are reviewed and the application of MR and AI to mining is discussed. The most common UAVs are low-cost multi-rotor systems however, their capabilities may be limited for long-term autonomous operation. The application scope is broad but identifies many trial UAV applications and sensors that could be integrated with MR and AI to produce an autonomous system. A simplified model for autonomous UAV, AI, and MR integrated systems is suggested along with some core modules. Via collaborative, modular development, the integration of UAV data, AI, and MR may be possible. Such a system will provide a powerful tool for mine site rehabilitation and compliance sign-off via a visual tool that allows all stakeholders to comprehend processes and data and agree when final rehabilitation is achieved.
Introduction
A sustainable mining operation encompasses community, environment, economy, resource efficiency, and safety (Laurence 2011). Access to an intuitive tool that monitors these domains would be beneficial to mine operators, the public, and government stakeholders alike because efficient data acquisition, processing, display, and analysis of multiple and diverse data sets is essential to determining compliance within a sustainable mining operation.
An Unmanned Aerial Vehicle (UAV)/Artificial Intelligence (AI)/Mixed Reality (MR) hybrid system may provide such a tool and allow an operational baseline to be determined from a sustainability and compliance perspective. Data from diverse sources can be acquired, analysed, and monitored simultaneously.
A UAV/AR/MR hybrid may also provide a valuable opportunity to improve stakeholder engagement via a transparent and ongoing compliance reporting tool that reports progress on essential activities and issues that can be evaluated against agreed targets. Also, because orebodies are buried within the ground, subsidiary benefits of monitoring mine sites via UAV/AI/MR, as a digital twin, may be beneficial for developing case studies and providing a reconciliation tool for future projects.
This paper reviews the application of UAVs in the mining industry. It evaluates the integration of this technology with AI and MR with a view to providing an autonomous hybrid system for mine site rehabilitation surveillance. During the review, the paper classifies the types of UAVs applied to mine survey and surveillance activities.
The most common UAVs are low-cost multi-rotor systems however, their capabilities may be limited for long-term autonomous operation and this may not necessarily be the best format. The application scope is broad but identifies many trial UAV applications and sensors that could be integrated with MR and AI to produce an autonomous surveillance system. A simplified model for an autonomous UAV-AI-MR integrated system is suggested, along with several core modules, and hence, the integration of UAV data, AI, and MR can be considered achievable via collaborative, modular development.
Such a system would provide a powerful tool for mine site rehabilitation and compliance sign-off via a visual tool that allows all stakeholders to comprehend processes and data, and agree when final rehabilitation is achieved.
The power of the described system would be that it is an autonomous data acquisition, analysis and visualization system, that provides a compliance monitoring tool, that is accessible to and comprehendible by all stakeholders.
There are many UAV formats available to the industry and UAVs produce a great deal of data and information that can be included within MR. When this is combined with AI, an immensely powerful monitoring and reporting system can be developed. However, this is not a trivial engineering task and some of the barriers to the seamless integration of UAV AI MR are discussed along with a broad description of the tasks required to realize a UAV/AI/MR hybrid.
The research question asked is, ‘Can unmanned aerial vehicles, artificial intelligence, and mixed reality simulation be combined to provide an autonomous data acquisition tool to monitor and record mine closure compliance and improve stakeholder outcomes, and what form would such a system take?’
To answer this question, in this study, the three technologies of UAV, AI, and MR are introduced first and then a comprehensive review of the application of UAVs in different sectors of the mining industry is provided, discussing the advantages and limitations of the various techniques. Furthermore, the potential of integrating the UAV/AI/MR technologies that provide an autonomous monitoring system and the corresponding regulations are discussed and recommendations are made within the discussion section.
Mixed reality
Milgram and Kishino (1994) examined the concept of having both ‘virtual space’ and ‘reality’ combined within the same visual display environment. The view was that a Virtual Reality (VR) environment was one within which a person was immersed and able to interact with a completely synthetic world. Milgram and Kishino (1994) developed the virtuality continuum in Figure 1. Virtuality means a virtual object or thing as opposed to an actual object or thing. Milgram and Kishino (1994) considered the merging of real and virtual worlds and referred to them generically as Mixed Reality (MR) (Figure 1). Milgram and Kishino (1994) formulated a taxonomy of ways in which the ‘virtual’ and ‘real’ aspects of MR environments can be realized and provided a sound foundation for describing MR and its associated devices.
Virtuality continuum (modified from Milgram and Kishino 1994). Images are available in colour online.
In Figure 1, the left extreme of the continuum defines environments consisting solely of real objects and includes for example, what is observed via a conventional video display of a real-world scene. To the right are computer-generated images superimposed upon real-world images with the extreme right being 100% virtual and usually computer-generated. As shown in Figure 2, Milgram and Kishino (1994) depict the mixture of classes of objects presented in any specific display situation and remark that an additional example includes direct viewing of the same real scene, but not via any particular electronic display system. Milgram and Kishino (1994) comment that as indicated in Figure 2, the most straightforward way to view an MR environment, therefore, is one in which real-world and virtual-world objects are presented together within a single display, that is, anywhere between the extrema of the virtuality continuum. It is important to note that nowadays (2022) it is much easier to realize MR via advanced game engine technologies, associated plugins, and powerful handheld devices and visualization systems. However, the automatic generation of MR environments and virtual scenes is still relatively immature and more problematic. A concept sketch of an MR system for compliance monitoring is presented in Figure 3.
Aspects of viewing a mixed reality world (modified from Milgram and Kishino 1994). Images are available in colour online. Mixed Reality system monitoring ongoing compliance (modified from Milgram and Kishino 1994). Images are available in colour online.
In Figure 3, the acquisition, analysis, and presentation of data would ideally be fully autonomous with the human observer (stakeholder) acting as governor and focussing on issues highlighted by their observations and the artificial intelligence component. Surveys would be conducted at set time intervals T, within constraints defined to provide consistency and repeatability in data acquisition. Artificial intelligence and humans would also compare compliance against for example T 1, T 2 … , Tn, where T 1 would be the initial baseline survey of the state of the mining operation defined within some predefined boundaries and conditions as agreed by all stakeholders. Subsequent iterations T 2, … Tn would be an assessment of the progress towards compliance and sign-off within the agreement and targets determined by all stakeholders at T 1. The successful completion of the compliance stages could be rewarded by reducing the financial stake held by the government regulators. Equally, failure to comply could result in penalties.
Artificial intelligence
Artificial intelligence (AI) can be defined as the capability of a computer or computer-controlled system to conduct tasks with minimal human intervention in an intelligent manner. Indeed, AI promotes machines/robots to mimic human brain activities, such as perception, decision-making, and feedback (Russell and Norvig 2022). Due to significant progress in sensor technology, which has given rise to increased data quantity and variety, and the increase in processing power in recent decades, robust AI algorithms in different industries, specifically in the mining industry, have been utilized extensively for operational decision-making, performance evaluation, and automation (Ali and Frimpong 2020; Shirani Faradonbeh et al. 2022). AI and its related algorithms play a critical role in the fields of UAV and simulation within mining operations. Thanks to computer vision and image processing algorithms, it is possible to extract valuable information from UAV images resulting in innovative functions, such as autonomous mapping, object/individual detection, motion analysis, etc. For instance, UAVs equipped with a Vision Processing Unit (VPU) and neural computer chips can perform deep-learning calculations and feature detection locally, in semi-real-time, and without an internet connection (Al-Turjman and Zahmatkesh 2020).
Machine Learning (ML), as a subset of AI, utilizes statistical and mathematical techniques to allow computers to learn from data and improve their performance in solving tasks without being explicitly programmed for each task.
Deep Learning (DL), on the other hand, is a machine learning algorithm based on artificial neural networks (ANNs), which can deal with big data recorded by monitoring systems, such as UAVs. In contrast to common ML algorithms, whose performance will reach a plateau with an increase in data size, deep learning is scalable, and its performance keeps getting better as it is fed with more data (Pu et al. 2019). Different deep-learning frameworks include deep neural networks, convolutional neural networks (CNNs), and deep belief networks. CNNs, for example, have been used broadly for object detection, image classification, and recognition tasks (Al-Turjman and Zahmatkesh 2020). They have also been applied to blast fragmentation (Vu et al. 2021).
AI technology can be implemented in the pre-flight, during-flight, and post-flight stages of UAV surveys. In the pre-flight stage, AI algorithms can be used for path planning problems and software design. Currently, the UAV images are mostly analysed and interpreted during the post-flight stage, which is a time-consuming procedure and cannot provide real-time input for mixed reality (MR) simulations. However, cloud-based platforms and edge computing can provide UAVs with onboard data processing by utilizing deep-learning tools on the central processing unit (CPU) and graphics processing unit (GPU) (Oren and Verity 2020). Onboard AI accompanied by sensor technology makes it possible to build an autonomous system capable of perceiving, analysing, communicating, planning, and making decisions in real time. Such a system would be appropriate for integrating with the MR environment to provide remote monitoring technology (see Figure 3). The use of AI can increase the flexibility and adaptability of MR systems. The potential of AI technology in the MR environment is obvious from its success in image processing and its pertinent tasks. Given that MR begins with the camera images and terminates with augmentations projected onto the head-mounted display (HMD) or other types of displays, AI can add revolutionary features to MR applications. AI algorithms can introduce a paradigm shift in the performance of all the tasks related to MR. The MR environment, which is mostly dominated by fiducial markers, can potentially adapt to natural features. MR can be implemented without modifying the environment when it is powered by AI. DL can be used for a range of tasks, such as calibration, detection, tracking, etc. When creating virtual objects, DL can be used for developing adaptable augmentations. Furthermore, DL can be used to determine what data to display and how to display it optimally when rendering. By simulating a mining project in an MR environment using UAV images, in addition to monitoring the progress of the operation, several engineering assessments can be undertaken. For instance, using DL algorithms, it will be possible to detect the discontinuities and perform rock mass characterization more efficiently for stability analysis of the structures within the mine site (Janiszewski et al. 2020).
Benefits of UAVs
UAVs are a modern innovative technology with the potential to assist mining companies to improve operational efficiency. Some of the benefits of using UAVs in the mining industry include:
Monitoring – The priority of the mining industry is safety, and various kinds of monitoring play a significant role in increasing mine site safety. UAV-based geotechnical monitoring is a significant new type of monitoring that includes remote monitoring of mining pillars and stopes, continuous monitoring of slopes and highwalls, and the use of point cloud and image processing programs to determine rock mass movement and predict failure. UAV geotechnical monitoring could be especially useful in the underground environment, where traditional monitoring has serious access and safety issues (Donnelly et al. 2001; Iverson and Signer 2014; Jang and Topal 2020). Cost Efficiency – Geo-structure measurements, such as stockpiles or leach pads, are used to necessitate a team of surveyors travelling to the field with surveying equipment. That practice is labour-intensive and requires a lot of time along with a significant amount of error. The use of UAVs as a surveying tool necessitates careful planning, but it optimizes personnel use and produces results in real time with high precision, resulting in cost savings (Hugenholtz et al. 2015; Seo 2015). Development and Resource Replacement – The combination of UAVs and remote sensors allows mines to increase productivity and optimize equipment and personnel use. For example, a comprehensive land survey can create a three-dimensional contour map to locate geographic structures, buildings, and other assets. UAV deployments can be cost-effective compared to new tripod-based technologies such as GPS, total stations, and Light Detection and Ranging (LiDAR) surveys (Neigh et al. 2014). Safety and Security – The use of UAVs helps avoid unnecessary accidents by recognizing dangerous risks and identifying new sources of danger. With advanced remote sensors and continuous monitoring methods, UAVs help ensure continuous monitoring of increasingly complex mining projects. UAVs can continuously monitor the security risks of processing plants and other buildings and monitor geographic structures for unstable rock movements. It is also useful for escape and rescue operations (Sayab et al. 2018; Onifade and Genc 2019; Said et al. 2020). Dynamic 3D Models from UAV Data – A key benefit of modern UAVs is that high-resolution digital images can be readily captured of areas of interest and these images can be used to produce point cloud data from which 3D models and additional site information can be extracted. A point cloud is a set of data points in 3D space and a basic type of geospatial data, others being vector and raster data. Point clouds are typically generated by capturing images with LiDAR either on the ground or in the air, or via photogrammetry (Hirschmüller 2011). LiDAR and photogrammetry reproduce a 3D surface by continuously emitting and collecting light rays that indicate the location of a point in space or by taking multiple images and combining those images to obtain a point. Point clouds have proven to be extremely useful in the mining and private industries and can be automatically constructed from video images using semi-global matching (SGM) and structure from motion (SfM) methods. The SGM method is used to find the same pixel position in different image pairs using the camera's intrinsic and extrinsic orientations (Hirschmüller 2011). The SfM method creates a 3D structure from a 2D image. When capturing camera images or using a scanner, the key factor is to view the scene of interest from various locations and angles. The field of view is the area within the sensor's range where objects can be detected. Open-pit mines and underground mines of all sizes around the world use point cloud data surveying, designing, or ground surveying. Over the last few decades, point cloud acquisition devices have become affordable, lighter, smaller, and easier to use. With the rapid growth of point cloud technology, open-pit mines and underground mines regularly use point clouds to characterize bedrock (Lyons-Baral and Kemeny 2016).
UAV configurations
Classification of possible fixed-wing UAV system configurations, (Class 1 a to i), available to mine site data acquisition and monitoring.
Classification of possible fixed wing with rotors UAV system configurations, (Class 2 a to i), available to mine site data acquisition and monitoring.
Classification of possible rotary-winged UAV system configurations, (Class 3 a to f), available to mine site data acquisition and monitoring.
Classification of possible airship-based UAV system configurations available to mine site data acquisition and monitoring (modified from Gangadhar et al. 2021).
Mining applications
Since 2015, the publication of peer-reviewed literature on the application of UAVs has accelerated. Using the keywords UAV, UAV and Mining, for example, within a Web of Science search produces some forty papers. Researchers have applied UAVs to mining over the complete spectrum of the mining cycle and sustainability domains. Example projects are summarized in the following sections. Many are exceedingly high quality; however, the efficiency and sustainability of data acquisition and processing via UAV technology remain unclear. The reviewed papers are recent publications and included to assess:
What technology is available? How is it implemented? How autonomous is it?
Environmental
Evaluating the effect of the mining operation on the surrounding environment is of paramount significance. The UAV technology can facilitate this by providing a remotely controlled operation. In the following sub-sections, the application of UAVs for monitoring the tailing dams and anthropogenic topographic signatures are discussed.
Tailings dam monitoring
Giacomo et al. (2021) describe the application of UAVs to megafauna species identification, behaviour, population data, habitat characterization, and monitoring of environmental protection areas. The project assessed the vulnerability of marine megafauna to the Fundão tailings dam collapse in Brazil. A DJI Mavic 2 Zoom model (Class 3c, Aerodyne UAV Table 3 c) and a Virtual Reality Mask – First Person View (FPV) were used to collect data. Giacomo et al. (2021) report that permission was required from the Department of Airspace Control for some 15 manually flown campaigns that required a team of 5, consisting of a Remote Pilot, Co-pilot, UAV Observer, Fauna Observer, and a Logistic Operator. A team of five people to acquire data may be uneconomical on longer-term projects.
Castendyk et al. (2019) and Straight et al. (2021) report the application of UAVs to monitoring pit wastewater from the perspective of limited accessibility to mine pits over the long term for monitoring. Hence, Castendyk et al. (2019) and Straight et al. (2021) discuss the cost benefits associated with UAV data collection compared to traditional environmental surveys. Castendyk et al. (2019) propose that UAVs show potential to improve safety associated with sampling by eliminating the need for personnel to access the pit lake water surface. However, Castendyk et al. (2019) remark that autonomous sampling systems have limitations with respect to pit lake monitoring and introduce a system capable of collecting in situ water sample profiles to 100 m depth and detail a process to collect the water samples via UAV. Challenges reported by Castendyk et al. (2019) were that not all limnological methods used in environmental monitoring programs could be conducted and battery power and lift weight restricted flight times and loads. A modified Class 3d, Aerodyne UAV Table 3 c was used. Flight times may be a serious restriction for remote Australian sites.
Wang et al. (2019) report tailings as an environmental risk that cannot be underestimated and monitored effectively. Wang et al. (2019) proposed global monitoring via UAV-ground hyperspectral joint observation and processing, UAV hyperspectral imagery, ground hyperspectral data relating to water and waste residue and water quality testing. The study area was complex with valley-type tailings reservoirs. A Dajiang M600 (Class 3d Table 3) UAV was equipped with a spectral imager to collect data (Figure 4). Wang et al. (2019) report that the 300 × 1500-pixel images had some 270 bands. Spatial resolution was 0.56 m with some 15 classes contained within the scene (e.g. tailings slag, grass, grass_mountain, bare_soil, grass_tailing_slag, polluted water_1/2, dam body, mining road, pedestrian road, red roof, blue roof, baresoil_mountian, mixed_residue_water, and gravel block). Wang et al. (2019) conclude that, overall, the research project provided important and meaningful data for subsequent disaster and environmental protection of tailings reservoirs in the area. UAV data processing is assumed to be manually completed and little automation was applied to data acquisition and processing. The numerous data classes show the broad spectrum of data that can be acquired.
The studied tailing dam of the Axi gold mine in Xinjiang, China and the used Dajiang M600 UAV (modified from Wang et al. 2019). Images are available in colour online.
Anthropogenic topographic signatures
Jianping et al. (2015) applied a UAV to characterize open-pit mining features via high-resolution topography and the Slope Local Length of Autocorrelation (SLLAC) method. Two iron mines had Digital Surface Models (DSMs) derived via UAV and Structure from Motion (SfM) photogrammetric techniques. The SLLAC method allowed the mine to be automatically depicted allowing researchers to estimate the surface area (Figure 5). Jianping et al. (2015) conclude that the UAV information may be used in three key areas: (i) the availability of multi-temporal surveys to track the changes in the extent of the mine; (ii) to relate the extent of mining to the number of processes in the area (e.g. pollution, erosion, etc.), and to (iii) analyse the effects of the change related to changes in erosion. The UAV was a Skywalker X5, a small, fixed-wing UAV (Class 2g, Table 2), measuring 0.6 m in length, with a 1.2 m wingspan. Weighing less than 2.5 kg it can fly for up to 40 min and Jianping et al. (2015) do discuss some automatic characterization of open-pit features via the SLLAC method. The data classes collected would be relevant to autonomous MR environment development.
SLLAC maps constructed for the studied mines through UAV photogrammetry. The yellow line shows the extent of the extraction areas within the mine sites (after Jianping et al. 2015). Images are available in colour online.
Geotechnical
The stability assessment of the structures in geotechnical projects using conventional/terrestrial techniques is a time-consuming procedure and cannot provide a real-time evaluation to develop an early warning system. UAV photogrammetry combined with AI enables engineers to perform structural mapping and evaluate fracture networks efficiently. The following sub-sections will provide some examples.
Highwall structural mapping
Raval and Banerjee (2018) evaluated the suitability of a UAV-LiDAR system to overcome limitations inherent in existing mapping systems. Raval and Banerjee (2018) integrated a laser scanner with an inertial measurement unit (IMU) and a real-time kinematic global positioning system (RTK GPS) with a UAV (Figure 6). The system aimed to map the structural characteristics of pit walls and compare them to a Terrestrial Laser Scanner (TLS) and UAV photogrammetry images. Raval and Banerjee (2018) comment that raw data collected from individual sensors was filtered to remove noise and segmented to produce a surface scan model. The system was flown via an operator and a team of five people. Raval and Banerjee (2018) developed a useful guideline containing a data acquisition and processing workflow for mapping structural via the UAV-LiDAR system. Raval and Banerjee (2018) applied a Class 3d UAV of Table 3 to this project. No automation of data acquisition, processing or data visualization was applied in this pioneering project, however, issues that must be overcome to deploy UAVs on-site are discussed. Again, a team of five people to acquire data may be uneconomical on longer-term projects.
(a) The highwall scanning using the (b) UAV-LiDAR system in a coal mine in Australia (after Raval et al. 2018). Images are available in colour online.
Katuruza and Birch (2019) applied UAV technology to South African coal mines where geological mapping presents challenges in extracting highwall information. Katuruza and Birch (2019) used a UAV and photogrammetric techniques. A DJI Phantom (Class 3c (Table 3)) and structure from motion techniques were applied. According to Katuruza and Birch (2019) it took approximately 48 h to process the data and produce a 3D model and weathering data and lithological contact elevations for short-term planning were extracted. Results showed a good correlation between the resource model and the UAV data model. Katuruza and Birch (2019) describe key benefits as: high spatial accuracy of the UAV data; 3D point cloud geological contacts were well defined when compared to the geological model; UAV mission generates data quickly; processing times can be less than 48 h. Ideally, in an autonomous MR system, processing would be autonomous and semi-real-time.
Sofonia and Gray (2021) provide a case study using UAV and LiDAR technologies and proprietary data processing software to study rock mass characteristics in mining areas. Sofonia and Gray (2021) present a workflow aligned with the mine site's operational schedule. Scans were performed by trained UAV pilots and data was pre-processed underground to reduce surface processing time. Each scan produces a large and has a benefit of not exposing personnel to hazardous areas. Sofonia and Gray (2021) report LiDAR data sets contain additional information to X, Y, Z data in space, such as time, range, reflectance intensity, return number and ring number that provide, greater flexibility in filtering, slicing and visualizing point cloud data. More than 250 stope scans were recorded by the system and also drawpoint inspections. Sofonia and Gray (2021) conclude that the mine's adoption of mobile LiDAR technologies delivered assists in understanding risk relating to the transition from a well-understood mining area to a new one by providing insight into the new rock mass. The ability to safely scan previously inaccessible mine locations allows teams to validate modelling and improve stope design and importantly predict behaviour. The project required a person to operate the UAV in this case and some processing was completed underground to reduce surface processing. This application was not fully autonomous within the areas of data acquisition, processing, or display however, the application demonstrates the valuable data that can be acquired via UAV and sensor technologies.
Geology
Vasuki et al. (2014) introduced a semi-automated UAV mapping system that leveraged rock surface photogrammetry data generated from aerial photographs. The method used advanced automated image analysis techniques and human data interaction to rapidly map structures and calculate dip and dip directions. Vasuki et al. (2014) report that geological structures were detected from a primary photographic data set. Equivalent 3D structures were identified within a 3D surface model generated via structure from motion (SFM) methods. Location, dip and dip direction were then calculated. Vasuki et al. (2014) applied a Class 3e UAV (Figure 7) to this project and conclude that the method produced a geologically feasible fault map while minimizing interpretation time. The semi-automatic method improved the fault map digitization time from hours to minutes.
(a) Class 3e UAV used for mapping the structures and (b) the fault map developed from the semi-automated method (after Vasuki et al. 2014). Images are available in colour online.
Tscharf et al. (2015) applied UAVs to mining and archaeology and produced geo-accurate 3D reconstructions. Tscharf et al. (2015) remark that photogrammetric computer vision systems are well established and UAVs affordable. Automated multi-view processing pipelines allow easier spatial data acquisition and the creation of accurate 3D models. Tscharf et al. (2015) comment that UAVs can navigate slowly, hover and capture images at any position and record highly overlapping images. Multi-copter UAVs bridge a gap between terrestrial and traditional aerial image acquisition and are suited to safe data collection in complex environments. Tscharf et al. (2015) present an automated workflow for image processing and 3D reconstruction of complex geometries that allows georeferencing of UAV imagery based on GPS measurements. Ground control points (GCPs) are integrated directly and correctly for systematic distortions of the image block. The methods used by Tscharf et al. (2015) would be applicable to compliance monitoring via MR. The simplified pipeline presented in Figure 8 provides a conceptual foundation for automatically generating 3D mine models from UAV images. However, the architecture of a fully automated MR system would be significantly more complex as described by Elmokadem and Savkin (2021) and is discussed in a later section.
UAV image preparation for 3D model surface reconstruction (Tscharf et al. 2015). Images are available in colour online.
UAV classification (modified from Giordan et al. 2020).
UAV comparison (1 = low, 5 = high) (Giordan et al. 2020).
The main characteristics include:
photogrammetry and remote sensing to extract image information and produce 3D data. 3D modelling to reconstruct buildings or area 3D shape. civil and military surveillance. inspection of dangerous locations e.g. mine sites. disaster response and monitoring; forest and agriculture and geological investigations.
According to Giordan et al. (2020), thermal and multispectral sensors are emerging fields that UAVs can provide via high-resolution and repetitive data which is fundamental for monitoring purposes. Giordan et al. (2020) remark on the absence of a unique system for all problems and that a careful preventive evaluation of the mission characteristics defines the best solution and sensors to be carried. The ground control point for the geolocation of collected data is also important.
The main components of a UAV described by Giordan et al. (2020) are:
the aerial platform, including the airframe, navigation system, power system, and payload; the ground control station (GCS) allowing the pilot control from a remote location; and the communication system supporting communication between the other two components.
Given the diversity of UAV systems and their capabilities and capacities, careful planning of mine rehabilitation missions would be required to ensure the correct class of UAV was chosen for the mission.
Antoniou et al. (2020) utilized Immersive Virtual Reality (IVR) for mine data collection integrated with Geographic Information System (GIS) analysis. The focus was
three-dimensional (3D) high-resolution IVR scenario building, based on Structure from Motion photogrammetry (SfM) modelling; subsequent geological survey, mapping and data collection using IVR; data analysis, e.g. calculation of extracted volumes, as well as production of new maps in a GIS environment using input data directly from the IVR survey; presentation of new outcomes highlighting the mine as a key geological site.
Antoniou et al. (2020) remark that traditional surveys present difficulties when mapping complex sites because, although Terrestrial Laser Scanning (TLS) generates dense point cloud measurements of mine features, it is line-of-sight and consequently high cliffs, or complex morphology produces occlusions in surface models. These occlusions give rise to areas of unwanted distortion and missing information in the surface models that must be removed or corrected manually. Antoniou et al. (2020) used an IVR approach integrated with GIS and present a conceptual workflow as shown in Figure 9.
Overall conceptual workflow (modified from Antoniou et al. 2020). Images are available in colour online.
Antoniou et al. (2020) developed a head-mounted VR experience that allowed users to navigate, interact and perform measurements within it. A 3D reconstruction of the mine was created by applying UAV-SfM photogrammetry. A Class 3c UAV was applied to this project (Table 3). The camera was oriented both in Nadir and oblique settings. Metadata included information such as shutter speed, apertures, ISO, and GPS coordinates. Commercial manually operated photogrammetry software with SfM capability was applied. Several manual steps are described by Antoniou et al. (2020):
aligning all pictures using low-quality settings and generic preselection, all GCPs added to the model to obtain a scaled and georeferenced point cloud to improve the accuracy of the final model, dense Point Cloud building was generated using the aggressive depth filtering and medium quality settings, some points in the range of 0–3 from the vertical cliff were removed and all points not related to the final model area, and, involved processing of the Dense Cloud, production of the DSM and the Orthomosaic and the 3D Tiled Model, necessary to work with immersive Virtual Reality.
Antoniou et al. (2020) made a final step consisting of building the 3D Tiled model, which included both mesh and texture and produced a pre-mining model for comparison with the UAV-acquired model. Antoniou et al. (2020) delivered mine data that identified interesting geological features that are not easily appreciated via conventional means.
These applications would be valuable for mine site monitoring and rehabilitation. However, while the process was accurate it was not automated and required considerable human interaction to produce the intended results. This is an important aspect of UAV data processing that must be addressed for it to become efficient and autonomous.
Mining engineering
Mining engineering is a complex task and when the complete mining cycle is considered from a mine closure perspective, many potential UAV data sets may be collected, analysed, and included within a dynamic MR environment as a digital twin. The following sections provide some examples of data acquisition for open-pit mining, topography and geomorphic features, blasting, coal mine fires, rehabilitation, and landslides. All these aspects could be monitored and form part of the mine closure compliance plan presented in an MR environment.
Open-pit mining
Jianping et al. (2015) characterized open-pit mining features using high-resolution topography generated from UAVs and SfM photogrammetric techniques to produce Digital Surface Models (DSMs) for two iron mines. A Class 2g UAV was applied to this project (Table 2). Information was obtained via the Slope Local Length of Auto-Correlation (SLLAC) of Sofia and Marinello (2014), and Jianping et al. (2015) report that a landscape metric could be used as a baseline to:
enable the availability of multi-temporal surveys to track changes in the range of mines, associate the extent of the mine with the amount of process in the area (pollution, erosion, etc.), and, combine the two points to analyse the impact of changes associated with erosion changes.
By analysing the direction of the correlation length, researchers can also identify the direction of the terrace and understand the shape of the open pit. The structural environment and history or inheritance of a given escarpment can determine if it fails and how it fails; the orientation of the topographical surface or excavated area associated with the geological features are very important. Jianping et al. (2015) used digital topographical analysis based on UAV and SfM elevation data to establish (and test) a fast and low-cost way to characterize the topographical features of open-pit mining. The results emphasize that using very high-resolution images captured by the UAV and then processed with SfM technology makes it possible to obtain accurate DSMs in near real-time at a low cost. The detection of terrace directions could thus provide helpful information for use in geologic analyses for risk assessment (Jianping et al. 2015).
Topography and geomorphic features
Rossi et al. (2017) evaluate nadir and off-nadir imagery to reconstruct high-resolution topography and geomorphic features of quarries using UAV-based photogrammetry. A Class 3d UAV was applied to this project (Table 3). Two processing scenarios were applied, nadir images, combined with images taken at off-nadir angles, and an accurate set of ground control points (GCPs) for georeferencing and validation. To improve understanding some typical quarry shapes were created using both scenarios and compared to those surveyed by a total station as a separate benchmark technique. Some 786 images were acquired using an autonomous mode set at one shot per second. Three fights were made with the camera in nadir orientation to cover the entire area, and the acquisition timing resulted in a minimum of seven overlapping images for any single ground feature. The SfM technique created a 3D georeferenced point cloud, and the M3C2 methodology examined the similarity between point clouds supplied by Agisoft PhotoScan and Pix4D mapper. The conclusion of Rossi et al. (2017) was that UAV-based photogrammetry with the nadir and oblique imagery is a useful tool for quarry operators to manage and monitor mining activities on a regular basis. UAV surveys are also useful for assessing the environmental risk of hazardous and inaccessible areas and provide a reasonable measure of reliability to the geotechnical interpretation of spatially variable soil conditions.
Sayab et al. (2018) applied a UAV-based SfM-MVS (multi-view stereopsis) approach to create a high-resolution 3D model of an open-pit mine and demonstrated how the framework can be used to virtually distinguish various characteristics via computer, which is a challenge to mine safety because of the steepness of open-pit faces. Sayab et al. (2018) utilized a DJI's Phantom 3 Professional (P3P) quadcopter (Class 3c, Table 3) to acquire 158 overlapping oblique and nadir images that were processed with Agisoft to generate textured 3D surface models. Sixty-nine overlapping images were also acquired of the open-pit steep faces. The precision of the 3D model was assessed via the deployment of ground control points (GCPs). The 3D models were uploaded into CloudCompare, to extract planar surface orientation manually and automatically for detailed virtual structural analysis and segmentation of well-exposed surfaces, using the ‘Compass’ plugin. In the ArcGIS platform, the data was further statistically classified. Sayab et al. (2018) used CloudCompare's ‘Compass’ tool to digitally quantify the dip angle and dip direction.
Salvini et al. (2018) applied UAVs to slope identification, stability monitoring, and geological engineering investigations. A Class 3d UAV was applied to this project (Table 3). UAV data was used to create a detailed 3D model from high-resolution images that were processed using SfM techniques. Salvini et al. (2018) applied geometrical and radiometric information to determine rock mass characterization. Two large blocks were identified that posed a significant hazard. Salvini et al. (2018) obtained the geometric characteristic of the two blocks, including the orientation of the intersecting discontinuities and the volume of the meshed block. Via stereographic projection, the orientation of 154 discontinuity planes was selected on the point cloud, and based on the characteristics of discontinuity derived from surveys. Salvini et al. (2018) performed a preliminary stability analysis to demonstrate the potential application of remotely piloted aircraft systems (RPAS) information in engineering geology to identify and reduce possible geo-hazards.
Kong et al. (2021) proposed a high-resolution digital outcrop model (DOM) generation method (Figure 10) for rock exposures based on UAV photogrammetry integrated with the SfM techniques. A Class 3c UAV was applied to this project (Table 3). Kong et al. (2021) introduced a digital procedure based on mathematical algorithms to detect discontinuities, map traces and quantify discontinuity characterization. Kong et al. (2021) applied the method to two rock slopes from Greece and China, and statistically extracted/obtained the fundamental discontinuity parameters, i.e. the orientation, number of sets, trace length, set spacing, linear frequency, areal frequency, and areal intensity. After that, the calculated values were compared with carefully planned manual measurements, i.e. individual discontinuity measurements, scanline method, and window sampling method on the same sites. Kong et al. (2021) showed that the resulting deviations were reasonable and acceptable, which verified the reliability and accuracy of the method. The methodology used by Kong et al. (2021) was (1) Conventional discontinuity measurements: Scanline and Window sampling method, (2) Performing UAV-SfM photogrammetry. Digital photogrammetric survey, and SfM processing using Pix4Dmapper, and DOM construction, (3) Discontinuity detection and characterization, discontinuity detection from DOM, calculation of parameters, scanline sampling from DOM, and window sampling from DOM.
The combined UAV-SfM photogrammetry method proposed by Kong et al. (2021): (a) DJI Phantom 4 pro UAV, (b) the flight route planning to capture the complex geometry of vertical outcrop, (c) the low-density 3D point cloud generation, and (d) the dense 3D point cloud generation base on multi-view stereo (MVS) algorithm. Images are available in colour online.
Puniach et al. (2021) applied UAV-based orthomosaics to automatically determine the field of horizontal surface displacements caused by underground mining. It was concluded that a weighted normalized cross-correlation algorithm had the greatest potential to determine displacements based on UAV-derived orthomosaics. However, the detection of features and matching algorithms turned out to be less effective. Puniach et al. (2021) applied the method to various case studies to validate and assess its accuracy, comparing the results to displacements determined manually using UAV-derived orthomosaics and displacements determined independently using Terrestrial Laser Scanning. As a result, feature detection and matching algorithms for multi-temporal orthomosaics registration and displacement determination became less useful. The results indicated partial usefulness in the given task for some of the tested algorithms, such as KAZE, while others, such as the SURF algorithm, were less effective. However, Puniach et al. (2021) conclude that more sophisticated algorithms of this type can be expected to appear in the future which could change this conclusion. According to Puniach et al. (2021), using UAV-based photogrammetry to determine horizontal displacements allowed them to obtain the field of displacement with a level of accuracy that is sufficient in practice to generate model parameters and continuously monitor the phenomenon. Finally, Puniach et al. (2021) remark that the use of outlier removal methods based on knowledge of the phenomenon is critical, as the determined displacement field would be unsuitable for interpretation without them. This paper introduces some interesting concepts that would be relevant to an automated UAV MR system. A Class 3e UAV was applied to this project (Table 3).
Blasting
Bamford et al. (2020) investigated the application of unmanned aerial vehicle (UAV) systems to monitor and improve the process of blasting in open-pit mines. Bamford et al. (2020) conducted field experiments to evaluate parameters such as rock fragmentation, blast-induced damage on final pit walls, blast dynamics, and blast hole accuracy, including production and pre-split holes. UAV-based monitoring was applied in three stages: pre-blasting (Figure 11(a,b)), blasting (Figure 11(c)) and post-blasting (Figure 11(d)). During pre-blasting pit walls were mapped to collect structural data to predict in-situ block size distribution and to develop as-built pit wall digital elevation models (DEM) to assess blast-induced damage. This was followed by mapping the pattern of production blast holes used in the mine to assess the alignment of drill holes. To monitor the blasting process, Bamford et al. (2020) used a UAV equipped with a high-speed camera to investigate blast initiation, sequencing, misfired holes and stemming ejection. Post-blast, Bamford et al. (2020) monitored the blasted rockpile to estimate fragmentation and assess muck pile configuration, heave and throw. The collected aerial data provided detailed information and high spatial and temporal resolution on the quality of the blasting process and significant opportunities for process improvement. Bamford et al. (2020) used two UAV platforms: the DJI Matrice 600 Pro (Class 3d, Table 3) and DJI Inspire 2 (Class 3c, Table 3) to acquire data. Software was used to generate 3D coloured point clouds (orthophotos and DEMs). In addition, Bamford et al. (2020) performed a statistical analysis of data to generate a stochastic model of the jointed rock mass based on a 3D discrete fracture network (DFN). A DFN is a stochastic model that describes the geometrical characteristics of rock mass fractures or, in general, discontinuities. Following the DFN model, Bamford et al. (2020) conducted a deterministic estimation of in-situ block size distribution (IBSD) using the ray cast volume method. The ray cast volume method is based on a simple algorithm for calculating block size; the algorithm generates random points within the DFN model that are used as origins of rays, which are cast outwards until the rays intercept a fracture (Bamford et al. 2020).
(a) Pre-blast collection of the structures and construction of the DFN model, (b) pre-blast blast hole deviation surveying, (c) blasting monitoring, and (d) post-blast rock fragmentation analysis using UAV (modified from Bamford et al. 2020). Images are available in colour online.
Coal mine fires
Wang et al. (2015) applied a knowledge model for determining fissures via Landsat TM/ETM, unmanned aerial vehicle (UAV), and an infrared thermal imager to monitor Datong underground coal fires in the Majiliang, China, mining (Figure 12). Texture information, linear features, and brightness of the ground fissures (Wang et al. 2015) were extracted. Conclusions drawn include (1) Landsat thermal infrared band provides a target range for accurate coal fire positioning by macroscopically reflecting spatial distribution characteristics of solar temperature in coal fires, (2) UAVs provide flexible photography and Landsat thermal infrared band is an ideal source of data for coal fire monitoring. Images produced have a higher resolution, and the distribution information on the centimetre-level width of the cracks in the coal fire region can be extracted from images obtained by the unmanned aircraft's optical camera. The technique aids in the assessment of the underground coal fire's conditions and serves as a reference in filling cracks and controlling fires, and (3) The temperature image of the coal combustion region photographed by the infrared thermal imager can be used to create a surface temperature field model. The variation in the surface temperature field caused by coal fires can be studied on a small scale to draw conclusions about the more general issues of coal combustion characteristics and ignition depth, which then served as a reference for locating the coal fire region's ignition points (Wang et al. 2015). The UAV class applied to this project is unclear.
(a) UAV image, (b) ground fissure extraction based on the knowledge model, and (c) mosaics results of temperature for a coal mine (modified from Wang et al. 2015). Images are available in colour online.
Rehabilitation
According to Padró et al. (2019), open-pit mining may become unsustainable without the restoration of degraded sites and post-mining monitoring is a legal requirement involving financial provisions for environmental liabilities. Padró et al. (2019) applied a small UAV (Table 3, Class 3c) with extremely high (10 cm) spatial resolution multispectral imagery to help improve restoration documentation with detailed land-cover maps. Figure 13 shows the procedure of land-cover mapping in this study. Padró et al. (2019) applied field spectroradiometer measurements to radiometrically correct sensor data and for photogrammetry, remote sensing, and geographical information systems software was applied to acquired images. Padró et al. (2019) completed spectral data and land-cover classification validation via ground-truth plots which assisted in the detection and quantification of mine waste dumps, bare soil, and other land-cover extensions. A ground-truth plot would be essential for establishing a baseline in mine rehabilitation applications. Padró et al. (2019) report that plant formations and vegetation development were evaluated, allowing a quantitative, visual, and intuitive comparison with surrounding reference systems. The resulting protocol provides a pipeline solution intended for the implementation by public administrators and private companies for evaluating restorations in an expedient manner. The methodology of Padró et al. (2019) would be valuable for mine site rehabilitation in Australia.
The workflow for land-cover mapping using UAV photogrammetry (Padró et al. 2019). Images are available in colour online.
Johansen et al. (2019) report that mine sites must rehabilitate post-mining landforms however, ground-based monitoring can be labour-intensive and insufficient to catalogue land rehabilitation efforts on larger mine sites (>100 ha). Johansen et al. (2019) propose an alternative UAV method that can be used to map rehabilitation success and provide evidence across a range of scales. Johansen et al. (2019) comment that UAV surveys collect information on rehabilitation in a repeatable, flexible, and cost-effective manner. Johansen et al. (2019) present a method to automatically map indicators of safety, stability, and sustainability, by combining multispectral UAV imagery with geographic object-based image analysis. An empirical classification system was proposed to convert indicators into a status category based on criteria relating to land cover, landform, erosion, and vegetation structure (Johansen et al. 2019). Criteria included mapping tall trees; vegetation extent; senescent vegetation; extent of bare ground; and steep slopes. Converting land-cover indicators into appropriate mapping categories on a polygon basis indicates the level of rehabilitation success and variation, this method may be useful for baseline and ongoing rehabilitation compliance. A workflow is presented that provides a set of recommendations for future rehabilitation assessment and the Johansen et al. (2019) framework provides a method to determine whether rehabilitated land is appropriately conditioned for post-mining use. With the increasing use of UAVs for mine rehabilitation monitoring, the developed UAV-based framework has the potential to support many future applications, including management implications, legislative compliance, negotiated residual risk assessments and mine lease relinquishment. This is particularly important for sustainable mining operations. Two class 2g (Table 2) UAVs were applied carrying Red (R), Green (G), Blue (B), Red Edge (NE), and Near-Infrared (NIR) sensors.
McKenna et al. (2020) report that ecological remote sensing studies for mine site rehabilitation have increased. Early studies used Landsat sensors at the regional and landscape scales and recently, multiple earth observation and UAV-based sensors have contributed to understanding vegetation development, post-mining. A limitation identified by McKenna et al. (2020) was that less than half of the studies integrated ground-based field data with remote sensing data essential for calibrating remotely sensed ecological models. Another limitation identified by McKenna et al. (2020) was the assumption that increased vegetation cover, is a final measure of mine rehabilitation success. An increase in vegetation cover represents just one metric in a suite of criteria, and McKenna et al. (2020) show that there is a need for ongoing, scientific, and accurate monitoring of ecological data during mine site rehabilitation and automated UAV technologies may provide a robust method for data acquisition and analysis. Data acquired from this type of monitoring should be included within data sets available within MR rehabilitation systems. Remote sensing was the focus of this rehabilitation project.
Landslides
According to Giordan et al. (2020) Terrestrial LiDAR Scanning (TLS) and ground-based radar systems can survey landslides and other geo-hazards remotely, also, monitoring landslides via LIDAR and UAV technologies has increased following international initiatives (Figure 14). Ground surface Digital Elevation Models (DEMs) can be produced from point clouds, and vegetation removed to reveal high-resolution landslide models. LiDAR scanners with the addition of hyperspectral sensors can be added to UAVs and Giordan et al. (2020) suggest that instrumented UAVs are useful where the terrain may be remote, hazardous, and inaccessible (an operational mine site) and suggest that UAV photogrammetry, where point clouds and 3D models are produced from overlapping photography, meets this requirement. Giordan et al. (2020) also report on landslides that are commonly driven by rainfall, rock permeability and coastal erosion. Differentiating between active and inactive states can be difficult to determine. UAV surveys may be a solution or at least an important part of it and Giordan et al. (2020) state that the Georegistration of each survey in a landslide monitoring program is vital to compare individual surveys and calculate changes. A combination of TLS and SfM (using both terrestrial and UAV platforms) provides the best result in rock slope stability surveys when they are linked to an independent survey control network. Monitoring potential landslides would have an important application in mine slope stability monitoring when monitoring mine site rehabilitation.
Coastal cliff landslide surveying using a UAV equipped with a digital camera (Giordan et al. 2020). Images are available in colour online.
Shahmoradi J (2020) reviewed UAV technology and its applications in the mining industry. Applications include 3D mapping of the mine environment, ore control, rock discontinuities mapping, post-blast rock fragmentation measurements, and tailing stability monitoring. Shahmoradi J (2020) reviewed UAV types, specifications, and applications of commercially available UAVs. Two benefits are presented, (i) UAVs equipped with different types of sensors can conduct a quick inspection of an area, either in an emergency or hazard identification and (ii) inspection and unblocking of ore-passes can be achieved. Shahmoradi J (2020) provided a summary of UAVs and example sensors that can be integrated such as Infrared Sensors (IR), Ultrasonic Sensors (US), Red–Green–Blue (RGB) Sensors, Stereo Cameras, Laser Range Finders (LRFs) Ultra-Wideband Radar (UWB), Hyperspectral Sensors, Magnetic Sensors, Visible and Near-Infrared Spectral Range (VNIR), and Air Quality Sensors. Shahmoradi J (2020) described mining UAV challenges as, weather conditions that induce deviations in UAV's trajectories, ventilation flows that can be damaging to UAVs and energy consumption which is restrictive. Shahmoradi J (2020) remarked that UAVs commonly run-on batteries and consume energy for flight, hovering, wireless connection, data, and image processing. Underground mines present challenges from confined space, heat and humidity, dusty air, and poor lighting and UAVs are usually manually controlled. Shahmoradi J (2020) suggested that as a minimum, the designed UAV should be capable of fully autonomous navigation in a GPS-denied environment and fly in an environment with no lighting other than that provided by the UAV. The nature of underground mines requires the UAV to be collision tolerant and able to detect and avoid obstacles during flight. Shahmoradi J (2020) does not consider the processing aspect of data acquired by the UAVs or integration workflows when UAVs are combined with several other sensors. This process could be time-consuming if not fully automated and this must be addressed for the viability of autonomous UAV MR integration.
UAV-based sensing examples
Yang et al. (2020) applied a UAV platform equipped with visible infrared sensors for reconnaissance and commented that the UAV imagery characteristics required an automated visible infrared image analysis system to be developed. An automatic image analysis framework was proposed by Yang et al. (2020) that allowed the interpretation of visible light and infrared images acquired by UAV cameras mounted. The system involved many aspects of data processing however, Yang et al. (2020) focused on direct and indirect georeferencing and target tracking in their research. The project demonstrated the effectiveness of a multispectral image analysis system and UAV technologies as a reconnaissance platform. An automated image analysis system would be essential for ongoing and consistent mine rehabilitation monitoring.
Giordan et al. (2020) describe UAVs combined with sensors that allow the collection of various kinds of data combined with the positioning of datasets in three dimensions and consider this mandatory if UAV data are combined with other datasets and geocoded data. Sensors carried include (i) digital cameras, (ii) thermal detectors, (iii) multispectral cameras, (iv) LiDAR, and (v) sensors for air quality evaluation. The first four categories are used commonly in Engineering Geology.
Banerjee et al. (2020) discuss UAV-based hyperspectral imaging and suggest that near-earth spectroscopy via UAV-hyperspectral systems addresses the limitations of the resolution, signal-to-noise ratio, and cloud cover from traditional satellite or airborne platforms. A class 3c (Table 3) UAV was utilized in this project. Banerjee et al. (2020) addressed the challenges of sensor calibration, data acquisition, radiometric correction for illumination variation, mosaicking and geometric correction for UAV-hyperspectral imaging of heterogeneous environments. A pre-processing workflow addressed the specific issues of monitoring the heterogenous distribution of swamp vegetation and assessing its condition. Banerjee et al. (2020) report that the framework would be useful for environments where satellite or airborne hyperspectral observations are limited. Banerjee et al. (2020) summarize the various procedures and workflow required in the project and methods to determine ground truth relating to the study are described as the hyperspectral sensor, its detailed calibration, and its integration with a UAV system. The data and images collected by Banerjee et al. (2020) are comprehensive and the workflow and process developed by Banerjee et al. (2020) demonstrated that UAV-based monitoring tools may provide a safe and accurate environmental monitoring method for remote sites. However, the methods are not completely autonomous and may take some considerable time to complete. UAV data analysis and software.
UAV data analysis and software
UAVs have the potential to collect data very efficiently and produce vast amounts of data that must be interpreted and displayed. The following sections discuss point cloud technologies and Unity 3D and some of the challenges associated with efficiently importing data in MR environments. At present, automation of the acquisition, analysis and display processes is limited. Point clouds are described a quick to implement, but this may not be the case when they need to be viewed or analysed. Also, many 3D game engines have limited geospatial data support. Which again may limit their use; however, they are a good starting point.
Point cloud technologies
Lyons-Baral and Kemeny (2016) report that point cloud technology is valuable for geological and geotechnical data collection, interpretation, and analysis, and that mines regularly collect point cloud data via LiDAR. Photogrammetry surveying devices are affordable, and workflows are fast and efficient. Lyons-Baral and Kemeny (2016) comment that UAV airborne data acquisition options have expanded dramatically offering high-resolution coordinate scans, imagery, and other remote sensing. UAVs permit rapid data capture from narrow to broad areas and multiple perspectives reduce data ‘shadows.’ Lyons-Baral and Kemeny (2016) remark that the scanning range is at least 6 km and the altitude can be 5 km. Sampling rates can be at least one million Hz, and data processing time and difficulty have decreased via automatic software options for point cloud registration, photogrammetry, editing and geometry extraction, and level of detail rendering capabilities. Lyons-Baral and Kemeny (2016) describe case studies for digital outcrop modelling, sinkhole identification, rock mass characterization, identification of geological structure, predictive modelling, digital terrain engineering, failed rock mass monitoring and forensics. Lyons-Baral and Kemeny (2016) present future applications designed for overbreak prediction and other mine site-specific applications and conclude that point cloud applications are quick to implement nowadays. Lyons-Baral and Kemeny (2016) comment that more data sources will be combined into one picture, one interpretive model, and one analysis and utilize all the varied resources to create a complete geological picture. The applications described by Lyons-Baral and Kemeny (2016) appear powerful, however, the application of automation data acquisition and processing is not developed in any detail and there may be a significant human overhead required to complete any analysis. Automation of data acquisition and processing would benefit this type of project. A class 3d UAV (Table 3) was applied to this project.
Unity 3D
Unity 3D is a commonly used versatile game engine. It is a cross-platform game engine developed by Unity Technologies (Unity 2022). The game engine has been popular since 2005 and has been extended to support a variety of desktop, mobile, console and virtual reality platforms. The power of Unity is that it has a vast network of support and many add-ons and plugins that can be added to the basic Unity game environment. These include GIS data tools, point cloud tools, augmented reality, virtual reality, and mixed reality plugins. MR environments can be viewed on head-mounted displays, tables, PCs, or large-screen semi-immersive environments. There are other similar game engines such as Unreal (Unreal 2022) that have similar capabilities however, Unity is a popular choice for researchers at present.
Laksono and Aditya (2019) report that using real-world data in game engines is challenging because most have little geospatial data support. Unity was assessed for large-scale topographic data visualization from mixed data sources, and the level of detail 3D models was obtained via a terrestrial laser scanner and converted to FBX format (Laksono and Aditya 2019). Mapbox for Unity provided georeferencing support and OpenStreetMap (OSM) data for layers. Buildings were modelled via Mapbox topographic data, and building information was added to 3D features. Laksono and Aditya (2019) comment that Unity provides basic 3D navigation tools that enable users to gain a walk-through experience and virtually inspect the scene, and a UAV view helps users inspect objects from the air. A tool that would be useful in an MR system. The result is a multiplatform 3D visualization capable of displaying 3D models in LOD3, as well as providing user interfaces for exploring the scene. Using Unity to visualize mixed sources of topographic data creates many opportunities to optimize large-scale topographic data use. Unity may be a suitable platform on which to base a mine information modelling system for integration of UAV, artificial intelligence, and mixed reality. The concept and navigation are similar to those of LeBlanc-Smith et al. (1998). An example of open-pit mine terrain built from GIS data is shown in Figure 15.
Open-pit mine model mesh from Western Australia displayed within the Unity platform. Images are available in colour online.
Autonomous UAV survey data integration with mixed reality
The research papers discussed earlier confirm that the application of UAV technologies for data collection within the mining industry is widespread with numerous applications. However, as the research shows, the collection, processing, and interpretation of data can be time-consuming and requires significant human effort to travel to the site, set up the data acquisition mission, configure the UAV to collect the necessary data, fly the mission, record, and report the data, perform data analysis, and present the outcome via a 3D model. In some cases, five people are needed for missions.
UAV modes of operation (Elmokadem and Savkin 2021).
Elmokadem and Savkin (2021) consider fully autonomous UAV operation as a challenging and complex problem where a modular approach should be adopted for fault-tolerant solutions. Modules include components for collision-free navigation such as perception, localization and mapping, motion planning, obstacle avoidance, and control. Elmokadem and Savkin (2021) also consider autonomous navigation as crucial to ensure UAVs can move while avoiding collisions.
Elmokadem and Savkin (2021) describe navigation paradigms as deliberative (map-based), sensor-based (no maps), and hybrid (sensors and map-based).
Factors that must be managed include, overall navigation control structures, local path planning, local trajectory planning, reactive methods, UAV modelling and control, simultaneous localization, and mapping (SLAM), open-source projects, and research challenges.
Elmokadem and Savkin (2021) conclude that advances in UAV technologies have allowed progress towards achieving fully autonomous operations within control, motion planning, perception, localization, and mapping. Differences between adopted motion planning algorithms and control strategies are highlighted and a list of open-source projects is provided by Elmokadem and Savkin (2021) to aid researchers to deploy UAVs. For mine rehabilitation monitoring, a fully autonomous open-source mode may be the preferred option.
Several researchers report semi-automated workflows; however, these are limited to software processing and not the entire process. An example of system architecture is shown in Figure 16, and it is expected that the system would aim to be fully autonomous. The system includes an MR environment where analysis and decisions may be made, missions can be planned, data interpreted, and compliance assessed. The realization of the UAV MR integration requires a modular approach as there are many aspects and technical challenges to consider. Development of such systems and modules would be difficult and require inter-industry-academic collaboration. However, many of the technologies and software already exist but they are disseminated and should be brought together into one system and method as a hybrid solution.
A proposed modular automated UAV data acquisition and MR display system. Images are available in colour online.
Regulations affecting the operation of UAVs
Giordan et al. (2020) comment that national regulatory bodies have struggled to keep up with UAV developments and are likely to restrict or even prevent UAV surveys in some countries. Usually, there is a requirement for commercial users to have completed a training program. In Australia, UAVs (UAVs) are regulated by the Civil Aviation Safety Authority (CASA) if the user intends to fly a UAV commercially, there are extra rules that must be followed. There is a requirement to register UAVs and acquire a licence prior to flight. According to (Anon 2022) anyone can apply for a remote pilot licence (RePL) and this is only required for a remote pilot for an individual or business that holds a remotely piloted aircraft operator's certificate (ReOC) and to fly a UAV or remotely piloted aircraft (RPA) that weighs more than 25 kg but less than 150 kg over privately owned land. A RePL is not needed if the person is eligible to operate a micro (250 g or less) or excluded category (more than 250 g but no more than 2 kg or more than 2 kg but less than 25 kg over privately owned land) (Anon 2022). Hence, for most missions relating to mining projects, a licence would be required, because the UAV would be above 250 g in weight and depending on the flight path, may enter privately owned not owned by the operator or publicly owned land. It is unclear at this stage how fully autonomous UAVs fit within the licencing requirements.
Discussion
Integrating autonomous UAV survey data, AI capability and MR together within one system would provide an advanced compliance monitoring tool for mine site rehabilitation projects. Maintaining positive mine site sustainability outcomes and ongoing stakeholder engagement is essential over the longer term of mining projects. The goal is to establish a baseline of mutually agreed milestones and parameters that must be proven to achieve final landform rehabilitation to the satisfaction of all stakeholders. Integrating UAV data, images and AI with MR simulation and visualization could provide a tool to achieve this goal. However, developing a fully autonomous system that can be run remotely and reliably is a significant technical challenge and there are many aspects to be considered.
Within recent literature, it is clear the technologies have been applied across a wide range of projects ranging from geology, environmental, tailings dam monitoring, anthropogenic signatures, geotechnical, highwall structural mapping, mining rehabilitation and landslides with some success. However, the repeatability, economy, interoperability and long-term integration of these applications remain unclear and require further investigation.
Figure 16 shows a simplified diagram of a modular automated UAV AI MR system, and such a system has many components that must be developed to realize UAV AI MR integration. Various modules must be developed or refined for fully autonomous operation including,
Intuitive mission planning and execution modules. Collision-free autonomous navigation. i.e. perception, localization and mapping, motion planning, obstacle avoidance, and control. Data recording and reporting. Automated 3D model generation. Automated data and image analysis. Automated confirmation of compliance, or non-compliance. Automated generation of mixed reality visualization. Risk management and safety analysis for automated UAV. Automated calibration of sensors. Interoperability of existing software systems either proprietary or open source. Onboard or post-data acquisition data reprocessing.
The realization of a fully autonomous UAV AI MR system for mine closure compliance and indeed any form of mine site automated compliance monitoring via UAVs is a significant engineering challenge for the mining industry. The list above shows the broad scope and scale of the challenge. However, if the system is broken down into small manageable components within a collaborative industry research project and roadmap, that produces robust interoperable modules, it may be possible to realize fully autonomous UAV, AI and MR integration that provides a powerful compliance monitoring tool for the mining industry.
The significant issues that should be considered for developing a fully autonomous system are:
that open-source, interoperable, commercial off-the-shelf tools should be used to develop the system as far as possible. Proprietary or in-house software and systems should be avoided, where possible because it has the potential to limit the development of systems and can become costly concerning licencing, for example; that game engine technologies often have limited spatial data capacity, but they can already import geographical data via plugins. In the case of Unity, and the associated plugins, there is a licence fee involved and while these are inexpensive, the limitations of the licences, etc. must be explored; surveys will require a method for determining a ground truth prior to monitoring commencing. Hence, all surveys will require some type of ground control points to be established on-site to ensure data is located correctly within the MR environment and reflect the real world; flight range and capacity of UAVs must be considered because battery and payloads can be limited; interchangeability of sensors and UAVs over time must be a consideration; sensor calibration will be essential to ensure accurate data acquisition, and missions are repeatable; development of some industry standards to determine what sensor data is required for compliance, and how it should be collected so that it is consistent between sites and operations.
The UAV research projects reviewed within this paper primarily use smaller UAVs with multi-rotors and some fixed-wing examples. For fully autonomous missions on remote sites, larger remotely piloted aircraft (RPA) may be preferred, due to increased payload and flight duration. For example, a Triton Remotely Piloted Aircraft shown in Figure 17 (ADF 2020) may be the type of platform required. However, the capability, proprietary nature and cost of these aircraft must be considered against the industry's needs. The solution may lay somewhere in between smaller UAVs and RPAs. It may be that military spin-off companies for example could provide collaborative surveys to governments, stakeholders and mining companies who wish to collect MR compliance data for projects. The preferred model could be determined from an early scoping study, as indicated in Table 8.
Example of remotely piloted aircraft system (ADF 2020). Images are available in colour online. Example staging of a UAV, AI, MR integration project for mine site rehabilitation compliance.
Conclusions
This paper introduces the concept of integrating UAV data, AI and MR together as a hybrid system to provide a tool for monitoring mine site rehabilitation under the spirit of a sustainable mining ethos.
The concept of mixed reality is introduced and adapted to reflect a system intended to meet the mining industry's needs. The application of AI to mining operations is discussed. UAV systems and configurations are reviewed, and a simple classification is developed based on earlier works.
The review of literature conducted during this research indicates that the most common UAV used is the Class 3 multi-rotor system, this may simply be because they are readily available and low in cost, which makes them attractive to researchers. However, their capabilities may be limited and unsustainable for long-term autonomous operation, but this must be investigated much further in future work.
Recent peer-reviewed research papers were reviewed and identified many trial applications for UAVs and their associated sensors. Many sensors are available off the shelf and could be integrated with mixed reality and artificial intelligence to produce an autonomous system. The range of applications is broad and could be combined into one data-rich system for monitoring.
A simplified high-level schematic model of a fully autonomous UAV, AI, MR integrated system is presented, and some core modules are suggested from the literature and observation. From the literature review and the concepts presented in this paper, if a collaborative, modular development method is adopted, the integration of UAV data and imagery with AI and MR may be possible. Such a system will provide a powerful tool for mine site rehabilitation and compliance and help improve the mine compliance and sign-off by providing a visual tool that allows all stakeholders to comprehend the process and agree when final rehabilitation is achieved. The scale of such a development project is exceptionally large and may benefit from a collaboratively funded industry-academic research project.
Future work
This paper has investigated the integration of UAV survey data, AI, and MR as a concept. The scale of such a development project is exceptionally large and will benefit from a collaboratively funded industry-academic research project. The project should be approached in stages and include government, industry, academia, and military spin-off companies with capabilities to provide UAVs and RPAs of sufficient capability to collect long-range surveillance data. Table 8 shows an example timeframe.
Footnotes
Disclosure statement
No potential conflict of interest was reported by the author(s).