Research on 3D dimensional measurement of structures based on a dual monocular structured light system

Abstract

In recent years, machine vision has been increasingly applied in automated monitoring of civil engineering. However, the rapid and accurate acquisition of displacement and shape changes of structures remains a challenge. In the use of structured light to measure medium and large-sized objects, there generally exist shadow and insufficient field of view problems, especially for the traditional monocular structured light measurement method. To solve these, this article discusses a dual monocular structured light measurement method, which projects two-way stripe structured light and performs three-step phase shift, also proposes a new phase-unwrapping method of structured light, and discusses a point filtering algorithm of fusion and stitching of the point cloud data. The phase-unwrapping method and the point filtering algorithm proposed, respectively, improves the measuring speed, especially for large-sized objects, and accuracy, compared with gray-coding method and traditional ICP (iterative closest point) method. This article performs three-dimensional measurements of ball work pieces, and the experimental results show that the fused and stitched data is significantly better than the single-field data, and the diameter error is <0.15mm for 35mm ball workpieces, and 0.32mm for 45mm ball workpieces. In addition, this article compares the experimental results with gray-coding method and traditional ICP method.

Keywords

structured light dual monocular system phase unwrapping 3D measurement deep excavation pit

Introduction

In the past two decades, with the acceleration of new urbanization in China, the contradiction between the supply of land resources and the growing spatial demand has become increasingly prominent. Urban development has shown a trend toward three-dimensional (3D), high-density development, and the spatial dimension of urban service facilities has become taller and deeper. For deep excavation engineering in highly sensitive environments, the requirements for continuous monitoring data and compatibility with the construction status are gradually increasing. The safety operation and maintenance standards of adjacent existing municipal facilities are becoming more stringent. The defects of traditional manual monitoring systems cannot effectively meet the increasingly strict requirements of safety production. A dynamic, continuous, online, interference-free, and nonfailure automatic intelligent monitoring system is the development trend of the industry. The automated intelligent monitoring system of deep excavation pits mainly digitizes the traditional manual monitoring methods, breaks through the time limit of monitoring, realizes high-frequency continuous data acquisition, and solves the problem of timely monitoring. It also breaks through the limitations of space and environment to meet monitoring requirements under complex operating conditions. Currently, automated monitoring of excavation pits is moving from low-efficiency single-point measurement towards noncontact, large-scale, and fast dynamic and static combination monitoring, as well as toward the development of 3D laser scanning technology and close-range photogrammetry technologies.

Currently, monitoring the displacement and shape of structures during automated excavation pit monitoring is still a challenge. How to quickly and accurately obtain the displacement and shape changes of structures is an urgent issue that needs to be addressed. With the continuous development of machine vision technology and image acquisition equipment, structural displacement monitoring methods based on machine vision have emerged and have been validated in practical engineering applications (Dong et al., 2018; Feng et al., 2015; Xu and James, 2018). Due to its many advantages such as long-distance, noncontact, high precision, time-saving, labor-saving, and multipoint monitoring, it has attracted increasing attention from researchers and engineers.

Using structured light to perform 3D measurement of objects is an important noncontact measurement method (Liu et al., 2021; Nguyen et al., 2017; Song et al., 2019). Monocular structured light measurement (Tran and Ha, 2018) and binocular structured light measurement (Li et al., 2019; Yang et al., 2008) are two common measurement methods. Monocular structured light measurement is characterized by its relative simplicity, and its compactness and small size in structure. Lu et al. (2021) adopted monocular structured light measurement for 3D seam extraction of welding robot. However, for this method a reference plane is required which requires high accuracy for the camera and projector, and more importantly, the shadow formed at the edge of the measured object due to the projection of structured light pattern is difficult to solve. Binocular structured light measurement can overcome the problem of shadow. The key to this method is the pixel stereo matching, and some methods of pixel matching are proposed. Liu et al. (2014) proposed triangulation method and subpixel interpolation, and Chen et al. (2022b) performed quadratic polynomial fitting to constrain point cloud matching, Chen et al. (2022a) adopted epipolar-geometry and cross-ratio invariance method. However, binocular structured light measurement is still less effective especially for the work pieces lacking texture and is prone to sparse and less accurate point clouds. To solve the problems of shadow and insufficient field of view problems, a dual monocular structured light system is designed in this article for 3D measurement of medium- and large-sized objects. Meanwhile, this system avoids the problems such as occlusion and various distortions in stereo matching of the binocular structured light measurement system.

The 3D measurement of objects using monocular structured light or binocular structured light can be conducted through the phase measurement (Chen et al., 2010), which measures the continuous change in the gray level of the structured light stripe on the object and calculates the phase change to obtain the 3D of the object. Since the object space can be divided continuously, this method has a very high spatial resolution. The measurement can also be conducted through code measurement (Geng, 2011), which projects a special code pattern onto the object so that the object space is encoded by region. Since the coded pattern projected can only make a limited division of the object space, its spatial resolution is not high. For reasons of real-time and ease of operation, this article uses three-step phase shift in the N-step phase shift method for fast phase measurement.

To obtain a globally complete continuous phase, it is also necessary to unwrap the phase principal values of the spatial points to obtain a unique absolute phase value in the global range. Carsten et al. (2000) and Wan et al. (2020) proposed a multifrequency heterodyne method for phase-unwrapping in 3D complex objects measurement. This method is not sensitive to the surface color or reflection of the measured objects, and can obtain better stability and accuracy. However, this method needs to project multiple grating images of different frequencies, which has certain limitations on frequencies of fringe patterns, and has high requirements for phase accuracy. Wu et al. (2022), He et al. (2020), and Wu et al. (2019) presented a gray-coding plus phase-shifting method for phase-unwrapping, which uses multiple gray-code patterns to assign a unique codeword to each $2 π$ phase-change period. This method is algorithmically simple and easy to implement. However, the codeword is limited to $2^{M}$ , $M$ of which is the number of gray patterns used. Apparently, when this method is used for the measurement of large-sized objects, the number of codeword required will be large, which will result in a large number of gray-code patterns and poor real-time performance.

In this research, we propose a new method based on gray-coding method, to unwrap the phase. This method proposed improves the measuring speed, especially for large-sized objects. Compared gray-coding method, this method is more suitable for real-time measurement of objects with some requirements and large-sized objects.

In 3D reconstruction, we propose a point filtering algorithm of fusion and stitching of the point cloud data based on iterative closest point (ICP) method. This method is used to improve the measurement accuracy in the case of local shadow or noises during the measurement.

The rest of the article is organized as follows. The second section illustrates the dual monocular structured light measurement system, details the new method proposed for phase-unwrapping, and the algorithm of fusion and stitching of the point cloud data. The third section shows the experimental results conducted to validate the proposed system and method. And the fourth section summarizes this article.

Dual monocular structured light system

Introduction to the measurement system

The diagram of the dual monocular structured light measurement system used in this article is shown as in Figure 1. It consists of a computer, a projector, two CCD (charge coupled device) cameras on the left and right, an optical stage, a fixed support and a calibration plate. The distance from the left and right cameras to the projector is equal and is located on both sides of the projector. The optical stage and support serve to keep the spatial position of the camera projector constant. The parameters of the projector and cameras are listed in Table 1.

Table 1.

Parameters of projector and cameras.

	Instrumentation	Resolution
Spectral cameras	Hikvision MV-CE200-10GM(left)	5472×3648
	Hikvision MV-CE200-10GM(right)	5472×3648
Projector	GuangmiM3	1024×768

Structured light encoding

The diagram of dimensional measurement of object using structured light stripe phase is shown as in Figure 2. The distance from the camera center O_c to the projector center O_p is l, and the distance from O_p to the reference plane is d. The line O_pO_c is parallel to the reference plane. The CCD camera captures the object-modulated sinusoidal grating stripe from the projector.

Suppose a point on the object is B, its height h can be expressed as equation (1),

h = \frac{dp Δ ϕ}{2 π l + p Δ ϕ}

(1)

where p is the period of the sinusoidal grating stripe, and Δϕ is the phase difference between points D and E, is also the phase difference between point B on the object and the corresponding point on the reference plane. The key problem of height measurement is thus transformed into the measurement of the phase difference between the reference plane and after the object is placed on it. N grating stripe patterns that shift phase 2π/N at a time are projected sequentially onto the object, and the ith deformed grating fringe image intensity acquired by the CCD camera can be expressed as equation (2),

I_{i} (x, y) = A (x, y) + B (x, y) \cos [ϕ (x, y) + 2 π i / N]

(2)

where (x,y) represents the coordinate of a point on the object, A(x,y) is the average projection intensity, B(x,y) the modulation projection intensity, $\emptyset$ (x,y) the phase value, i the phase-shifting index, and N the total number of phase-shifting images. If N=3, the phase ϕ(x,y) can be obtained by equation (3),

ϕ (x, y) = \tan^{- 1} (\sqrt{3} (I_{1} - I_{2}) / (2 I_{0} - I_{1} - I_{2}))

(3)

where I₀(x,y), I₁(x,y), and I₂(x,y) are the ith grating fringe image intensity as in equation (2). Thus by the three-step phase shift, the phase values of the object and the reference plane are measured, respectively, by equation (3), and the values of $Δ \emptyset$ in equation (1) is calculated by subtracting these phase values to obtain the height values of each point of the measured object.

Phase unwrapping

According to the properties of the inverse tangent function, the solved phase principal values are truncated between (−π,π) and have unique values in a period range, and there are multiple period stripes in the whole measurement space range.

The phase-unwrapping process using this method is described in Figure 3. As shown in Figure 3(c), each period of encoded patterns corresponds to each $2 π$ phase-change period of fringe. And a period of encoded patterns is divided into four subregions every $π / 2$ phase-change period of fringe, that is region 1 corresponds to 0-to- $π / 2$ phase-change in a period of sinusoidal fringe, and region 2 to $π / 2$ -to- $π$ , and so on. Region 1, 2, 3, and 4, respectively, correspond to one bit of gray-code by their gray values. Thus, the codeword of region 1,2,3,4 is k₁k₂k₃k₄, and the codeword varies according to the rule of gray-code, as shown in Figure 3(d). That is, a gray-code in this method is spread over each $2 π$ phase-change period of fringe. When the size of the object exceeds the range that the gray-code of these four regions can express, the gray-codes of other patterns will be used. According to this method proposed as above, a unique absolute phase value of the spatial points in the global range can be obtained:

ϕ (x, y) = φ (x, y) + 2 π \cdot k_{n} k_{1} k_{2} k_{3} k_{4}

(4)

where ϕ is the absolute phase value, and φ the phase in 2π phase-change period of fringe, k₁k₂k₃k₄ the gray-code corresponding to a 2π phase-change period of fringe, and k_n the extended gray-code.

3D reconstruction and measurement

In the 3D measurement using stripe projection, it is difficult to obtain directly all the 3D information of the object measured from a single point of view in some cases. For example, the measured object itself is large in size, and cannot be collected the complete information from a single viewpoint; the measured object itself has quite complex surface shapes, such as large slopes, or the local specular reflection. These cases will lead to local shadows of the collected stripes, and incomplete 3D data. To solve these problems, multiple viewing angles can be selected to measure the object, or the relative motion between the object and the measurement system can be used to obtain 3D data at different angles, and finally obtain the complete 3D data of the object through coordinate system transformation and data fusion. Since the latter need to rely on relative motion to complete multiple measurements to achieve stitching, the stitching of data from multiple viewing angles used in this article is more conducive to real-time measurement.

Multiview measurement data are measured from different viewpoints, and the world coordinate system is different. So a conversion is necessary from local to global coordinate system, and then the data is fused and stitched to obtain the complete 3D measurement results. There are many ways to realize the localization of local space and the stitching of data, such as the multi-aperture scanning stitching technique (Chen et al., 2006), data stitching by marked points (Li and Wang, 2009), the iterative method of multi-aperture scanning stitching technique in cylindrical coordinate system (Guo and Chen, 2000), etc. The stitching of 3D surface shapes consists of two main parts: the transformation of coordinate systems of different viewpoints and the fusion of data.

In this research, the fusion and stitching of the point cloud dataset obtained by the left camera and one by the right camera is realized with ICP algorithm (Besl and McKay, 1992). Here the ICP algorithm is used to evaluate a rotation matrix R and a translation matrix t. And then by using $P_{L} = R \cdot P_{R} + t$ , the right point cloud dataset $P_{R}$ is aligned to the left point cloud dataset Q. However, as described above, there exist some problems in the process of fusion and stitching due to surface reflection, local shadow (or occlusion) and so on: (1) a pixel can be found in one point cloud set, but there is no matching point in another point cloud set; (2) there exists an abnormal value jump between a pixel and its neighboring pixel.

To solve these problems as above, a point filtering algorithm is proposed in this research before using the ICP algorithm.

The process of fusion and stitching is shown in Figure 4(a), and the principle and process of filtering points is shown in Figure 4(b). There may be amplitude jumps between points in left or right point cloud set, possibly due to local shadow (or occlusion), system noises, or normal fluctuations of the object measured itself. To distinguish these cases as above, the high and low thresholds, denoted as TH_H and TH_L, are introduced in the point filtering algorithm. The judgment of the points when their amplitude jumps exceed TH_H or smaller than TH_L is easy. However, when the amplitude jumps is between TH_H and TH_L, the point may be valid (due to normal fluctuations of the object measured itself), or invalid (due to noises or shadow).The judgment of the points in the case as above is made by the validity of their neighboring points.

Experimental verification

Image acquisition

In order to verify the dual monocular structured light system and the corresponding phase unwrapping and fusion and stitching methods described as above, two spheres with different diameters are selected for testing. In the measurement, d in equation (1), that is, the distance between the camera and the projector to the reference plane, is 1410 mm, and l, that is, the distance between the camera and the projector, is 270 mm. The grating frequency is 0.2 mm⁻¹. The diameters of spheres are, respectively, 35 and 45 mm. The parameters of projector and cameras are shown as in Table 1. The photographs of the system measuring workpieces are shown as in Figure 5.

Figure 6 shows the images of 35mm sphere grating modulation acquired by the left camera in the measurement system. From the equation (3), it is known that to acquire the phase values of the measured object and the reference plane, it is necessary to acquire three images for the left and right cameras, respectively, with sinusoidal grating phase shifts of 0, 2π/3, and 4π/3. From Figure 6, it can be seen that when the grating stripe is projected onto the sphere it is distorted due to the modulation by the height of the sphere surface. By demodulating these images carrying the height information of the object, the 3D data of the object can be obtained.

Image unwrapping

Figure 7 shows the process of absolute phase retrieval. Figure 7(a) and (b) are, respectively, two-dimensional and 3D wrapped phase images. To unwrap the phase principal values of the spatial points in the global range, the modified gray-code patterns are projected onto the sphere, which are shown in Figure 7(c). Figure 7(d) is the image of one cross-section of the wrapped phase and the corresponding codeword. Due to the disturbance of local shadow of the sphere, the phase around the 100th pixel cannot be obtained accurately, including the codeword of this location. Despite this case, the codeword of the neighboring period is not affected, and can be obtained accurately. By using the proposed phase-unwrapping method as above, it can be seen from Figure 7(e) that the valid information is perfectly unwrapped except for the shadow part.

3D reconstruction and results analysis

The 3D effect display of the measured 3D data of the sphere is shown in the Figure 8 as below. It is obvious from Figure 8(a) and (b) that the reconstruction effect of the left and right spheres can’t be reconstructed very well because of the shadows. By the left and right sphere data fusion and stitching using the methods as above, the effect of shadows on reconstruction has been removed, which is shown as in Figure 8(c).

Based on the reconstruction result of Figure 8(c), we measured the diameters of spheres. The measurement results and the comparison of proposed method with only ICP method are listed in Tables 2 and 3. And the time comparison of 3D reconstructions based on the method proposed with based on gray-coding method are listed in Table 4. The configuration of PC platform for reconstructions is as follows: CPU: Intel Core i3-8100 3.60GHz, RAM: 16.0GB.

Table 2.

Comparison of proposed method with only ICP (diameter of 35mm).

Units, mm	Measured dimension	Only ICP	Actual dimension
Left view	34.9097		35
Right view	35.1513		35
After fusion	35.0607	35.402	35

ICP: iterative closest point.

Table 3.

Comparison of proposed method with only ICP (diameter of 45mm).

Units, mm	Measured dimension	Only ICP	Actual dimension
Left View	45.3251		45
Right View	44.9251		45
After fusion	45.1015	46.033	45

ICP: iterative closest point.

Table 4.

Comparison of time cost of three-dimensional reconstructions.

Units, ms	Based on themethod proposed	Based on gray-codingmethod
35mm sphere	109+1240	245+1223
45mm sphere	109+1589	245+1582

From Table 2, it can be obtained that the relative error of sphere diameter varies from 0.26% (left view) and 0.43% (right view) to 0.17% after the stitching and fusion of the left and right view data. Compared with only ICP method, the error percentage of the proposed method is reduced from 1.15% to 0.17%. From Table 3, the relative error varies from 0.72% (left view) and 0.17% (right view) to 0.22%. Compared with only ICP method, the error percentage of the proposed method is reduced from 2.29% to 0.22%.

The time cost of 3D reconstructions in Table 4 has two parts. One is the time of image acquisition, and another is the time of reconstruction and fusion. The time cost of the latter is basically the same, and however, the time cost of the former is significantly shorter, as shown as in Table 4.

The measurement of larger sized objects were also conducted using the dual monocular structured light system and the methods described as above. Figure 9 is a 3D reconstruction of the head of a metal pressure vessel with a diameter of 201.00mm.

The measurement results of this head and other large-size metal heads are listed in Table 5. In addition, the depths of the heads based on 3D reconstruction were measured.

Table 5.

Measured diameters and depths of large-size metal heads.

Actual diameterand depth (mm)		Measureddiameter (mm)	Measureddepth (mm)
201	100	202.67	101.8
303	120	305.31	121.04
400	200	403.88	201.94
600	300	603.46	303.73

Conclusion

When using displacement measurement methods based on machine vision, various factors can affect the measurement accuracy, leading to measurement errors. In this article, we adopt a dual monocular structured light measurement system to solve shadow and insufficient field of view problems during the 3D measurement of objects using structured light. For reasons of high spatial resolution and fast phase measurement, three-step phase shift method is used to encode object.

In order to unwrap easily the phase principal values during the actual measurement, a code-based method is proposed. Compared with the multifrequency heterodyne method, the method proposed in this article has the advantages of gray-coding method’s simplicity and ease of implementation. And compared with gray-coding method, the method has faster measuring speed, especially for large-sized objects. However, this method requires certain methods to improve accuracy when dealing with objects with rich surface colors or high reflection, such as spraying powder on the surface of the objects.

Finally, an algorithm of fusion and stitching of the point cloud data is used to get the final data. The article experiments with this system and methods proposed using spheres with different diameters, and the results show that the accuracy is improved compared to single-channel structured light detection when dealing with workpieces with shadowed regions, and the problems of shadows, insufficient field of view and so on are also solved.

Footnotes

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Zhejiang Province Public Welfare Technology Application Research Project (grant number LGG22E050051), and the Science and Technology Innovation Funding of Hangzhou CBD Investment Group (grant number QT202201B001).

ORCID iD

Jianbo Luo

References

Besl

McKay

(1992) Method for registration of 3-D shapes. Proceedings of SPIE 1611: 586–606.

Carsten

Reinhold

Jan

(2000) 3D shape measurement of complex objects by combining photogrammetry and fringe projection. Optical Engineering 39(1): 20–25.

Chen

Zhong

Han

, et al. (2022a) Binocular visual dimension measurement method for rectangular workpiece with a precise stereoscopic matching algorithm. Measurement Science and Technology 34(3): 035010.

Chen

Guo

Cheng

(2006) Novel aperture connection method for measurement of surface with rotation axis. Proc. SPIE 4101, Laser Interferometry X: Techniques and Analysis, 193–202.

Chen

Lin

, et al. (2022b) Research on vision measurement method based on binocular structured light. 5th Optics Young Scientist Summit, SPIE 12448, 177–184.

Chen

Zuo

Wang

, et al. (2010) Survey on structured light pattern codification methods. Journal of Chinese Computer Systems 31(09): 1856–1863.

Dong

Jin

(2018) Identification of structural dynamic characteristics based on machine vision technology. Measurement 126: 405–416.

Feng

Ozer

, et al. (2015) A vision-based sensor for noncontact structural displacement measurement. Sensors 15(7): 16557–16575.

Geng

(2011) Structured-light 3D surface imaging: A tutorial. Advances in Optics and Photonics 3: 128–160.

10.

Guo

Chen

(2000) An iterative algorithm of multi aperture transformation and connection technique in cylindrical coordinates. Acta Optica Sinica 20(8): 1047–1052.

11.

Sui

Lyu

, et al. (2020) 3D reconstruction of objects with occlusion and surface reflection using a dual monocular structured light system. Applied Optics 59(29): 9259–9271.

12.

Jia

, et al. (2019) Multi-state objects depth acquisition based on binocular structured light. Optics and Lasers in Engineering 121: 521–528.

13.

Wang

(2009) Research on 3D reconstruction procedure of marked points for large workpiece measurement. 5th International Conference on Information Assurance and Security, 273–276.

14.

Liu

Zhou

Wei

, et al. (2014) Optimized stereo matching in binocular three-dimensional measurement system using structured light. Applied Optics 53: 6083–6090.

15.

Liu

Zhang

, et al. (2021) 3D snow sculpture reconstruction based on structured-light 3D vision measurement. Applied Sciences-Basel 11(8): 2076–3417.

16.

Fan

Hou

, et al. (2021) Automatic 3D seam extraction method for welding robot based on monocular structured light. IEEE Sensors Journal 21(14): 16359–16370.

17.

Nguyen

Tran

Nguyen

, et al. (2017) A high dynamic range, high precision framework for measuring machinery part using structured light. 2017 International Conference on System Science and Engineering, 670–675.

18.

Song

Tang

, et al. (2019) DOE-based structured-light method for accurate 3D sensing. Optics and Lasers in Engineering 120: 21–30.

19.

Tran

(2018) Non-contact gap and flush measurement using monocular structured multi-line light vision for vehicle assembly. International Journal of Control, Automation and Systems 16(5): 2432–2445.

20.

Wan

Cao

Kofman

(2020) High-accuracy 3D surface measurement using hybrid multi-frequency composite-pattern temporal phase unwrapping. Optics Express 28(26): 39165–39180.

21.

Guo

Zhang

(2019) High-speed three-dimensional shape measurement based on shifting gray-code light. Optics Express 27: 22631–22644.

22.

Guo

Zhang

(2022) Two-frequency phase-shifting method vs. gray-coded-based method in dynamic fringe projection profilometry: A comparative review. Optics and Lasers in Engineering 153: 106995.

23.

James

(2018) Review of machine-vision based methodologies for displacement measurement in civil structures. Journal of Civil Structural Health Monitoring 8: 91–110.

24.

Yang

Cheng

Chen

(2008) Flexible and accurate implementation of a binocular structured light system. Optics and Lasers in Engineering 46(5): 373–379.