DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-20 are pending for examination in the application filed 09/17/2024. Claims 1-14 are currently amended and claims 15-20 are new.
Priority
Acknowledgement is made of Applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been received in parent application EP22170876.1, filing date 04/29/2022. Acknowledgement is additionally made of the present application as a national stage entry of PCT/EP2023/061223, international filing date: 04/28/2023.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 09/17/2024 and 02/02/2026 have been considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3, 8, 11-12, 15, 17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Turkan (Turkan, Yelda, et al. "Tracking of secondary and temporary objects in structural concrete work." Construction Innovation 14.2 (2014): 145-167) in view of Braun (Braun, Alex, et al. "Improving progress monitoring by fusing point clouds, semantic data and computer vision." Automation in Construction 116 (2020): 103210).
Regarding claim 1, Turkan teaches a detection method for detecting a building part on a construction site, comprising the following steps: i) acquiring one or more scans of the construction site at an acquisition position (Figure 6: Wall recognition performance (%recognized) for different values of TD and at different construction stages. The curves show the average and standard variation of %recognized. [Abstract] This paper presents three different techniques for recognizing concrete construction secondary and temporary objects in TLS point clouds. Two of the techniques are tested using real-life data collected from a reinforced concrete building construction site);
ii) generating data points in a three-dimensional coordinate system the data points representing the construction site ([3 The Automated Object Recognition System] As a pre-processing step, the system requires converting the input 3D model into triangulated mesh format (e.g. STL, OBJ). It then follows a two-step process detailed below: 1. Registration of TLS point clouds with building 3D model 2. Recognition of 3D model objects in TLS point clouds);
iii) recognising data points in the three-dimensional coordinate system belonging to a formwork element on the construction site which is arranged in a forming position ([3 The Automated Object Recognition System] 2. Recognition of 3D model objects in TLS point clouds. [5.2 Recognition of Formwork and Rebar (Step 2)] Technique 2 (Bosché and Haas’ algorithm with larger values of TD was tested with values of TD ranging from 10mm and 60mm in order to identify whether rebar and formwork could be distinctly differentiated from completed objects…The experiment results are summarized in Figures 5 to 7 for columns, walls and floor slabs respectively. Overall, it appears, as expected, that recognition levels for objects in “formwork” state show very low %recognized for values of TD<30mm, but then clearly increase from 30mm onwards);
iv) determining that visible area of a building part to be erected in a stored building model, which is visible from the acquisition position according to a current construction progress ([5 Experiments] In order to evaluate the performance of the proposed secondary and temporary construction object recognition techniques, experiments were conducted using a 3D model obtained for the Engineering V Building site at the University of Waterloo, Canada (Figure 3) and eleven different TLS point clouds acquired on eleven different days over a seven month period. [3.2 Recognition of 3D Model Objects in TLS point clouds] At the end of the registration process, the project 3D model and TLS point clouds are optimally registered. Because it is known to which object points were matched at the last iteration, each model object can be assigned a corresponding as-built point cloud (sub-set from the complete as-built TLS point cloud). The analysis of the as-built point cloud can then lead to the recognition of the object itself using a surface-based recognition metric (Bosché, 2010));
v) assigning the data points belonging to the formwork element to the visible area of the building part to be erected when the formwork element is arranged in the forming position for erecting the building part on the construction site ([3.2 Recognition of 3D Model Objects in TLS point clouds] At the end of the registration process, the project 3D model and TLS point clouds are optimally registered. Because it is known to which object points were matched at the last iteration, each model object can be assigned a corresponding as-built point cloud (sub-set from the complete as-built TLS point cloud). The analysis of the as-built point cloud can then lead to the recognition of the object itself using a surface-based recognition metric (Bosché, 2010). This metric further requires the calculation of a virtual as-planned TLS point cloud using the 3D model and the scanner’s location of the registered point cloud. This point cloud can be segmented in a similar way as with the as-built point cloud, so that each model object can be assigned a corresponding as-planned point cloud (sub-set from the complete as-planned TLS point cloud);
and vi) detecting the building part on the construction site when specified data points belonging to the formwork element is assigned to the visible area of the building part to be erected ([3.2 Recognition of 3D Model Objects in TLS point clouds] At the end of the registration process, the project 3D model and TLS point clouds are optimally registered. Because it is known to which object points were matched at the last iteration, each model object can be assigned a corresponding as-built point cloud (sub-set from the complete as-built TLS point cloud). The analysis of the as-built point cloud can then lead to the recognition of the object itself using a surface-based recognition metric (Bosché, 2010). This metric further requires the calculation of a virtual as-planned TLS point cloud using the 3D model and the scanner’s location of the registered point cloud. This point cloud can be segmented in a similar way as with the as-built point cloud, so that each model object can be assigned a corresponding as-planned point cloud (sub-set from the complete as-planned TLS point cloud. [3.2 Recognition of 3D Model Objects in TLS point clouds] The analysis of the as-built point cloud can then lead to the recognition of the object itself using a surface-based recognition metric (Bosché, 2010)… After converting the as-planned, occluded and as-built point clouds for each object into planned, occluded and recognized (as-built) surfaces, the percentage of recognition %recognized is calculated as:
PNG
media_image1.png
256
799
media_image1.png
Greyscale
[5.2 Recognition of Formwork and Rebar (Step 2)] The experiment results are summarized in Figures 5 to 7 for columns, walls and floor slabs respectively. Overall, it appears, as expected, that recognition levels for objects in “formwork” state show very low %recognized for values of TD<30mm, but then clearly increase from 30mm onwards).
Turkan does not explicitly teach acquiring one or more camera images of the construction site with at least one digital camera at an acquisition position; detecting the building part on the construction site when a specified number of data points belonging to the formwork element is assigned to the visible area of the building part to be erected.
Braun, in the same field of endeavor of formwork analysis, teaches acquiring one or more camera images of the construction site with at least one digital camera at an acquisition position ([1.3 Contributions] A method is presented that makes use of the knowledge of construction methods and 4D data to adjust the detection thresholds (as-planned vs. as-built deviations allowed) according to their expected construction stage. This permits the detection of elements that are currently under construction and are, for example, covered by formwork. We introduce a method based on visibility analysis to identify elements that are detectable from the identified camera positions);
detecting the building part on the construction site when a specified number of data points belonging to the formwork element is assigned to the visible area of the building part to be erected ([3.6 Identified tasks during construction] Several tasks are required to construct in-situ concrete elements or similar elements. In concrete construction, formwork for in-situ concrete is the most common construction method. Several different methods are depicted in Fig. 1b) and d). All possible elements under construction are considered in order to detect formwork. In general, elements are counted as detected as soon as a certain amount of points per area [Pts/m2] with a distance of less than 2 cm are matched on the surface of the element. If the expected elements are not detected, the threshold for the maximum distance can be adjusted to take into account the fact that the formwork with a thickness of around 0.20 m might be currently in place. If this iteration brings positive results, the element can be marked as “under construction”.
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Turkan with the teachings of Braun to acquire camera images of the construction site because "The as-built status of a construction site is usually captured by laser scanners or cameras using SfM methods. Laser scanning…equipment is heavy and requires trained personnel…this method is more flexible and easier in its application, as camera equipment is standard, low-cost, and widely used on UAVs" [2.1 Scan vs BIM] and to detect the building part when a specified number of datapoints belonging to formwork is assigned because "Another reason for weak detection rates is building elements that are currently under construction. As those elements count towards the overall progress, they must not be missed, and play a crucial role in defining the exact state in the current process. In general challenges exist for all construction methods, whose geometry under construction differs largely from the final element geometry which requires the use of temporary construction objects" [4. Case Study].
Regarding claim 3, Turkan and Braun teach the method of claim 1. Turkan further teaches wherein in step iii) recognising of data points in the three-dimensional coordinate system that belong to the formwork element can be carried out on the basis of a segmentation and classification of segmented components of the scan ([3.2 Recognition of 3D Model Objects in TLS point clouds] The analysis of the as-built point cloud can then lead to the recognition of the object itself using a surface-based recognition metric (Bosché, 2010). This metric further requires the calculation of a virtual as-planned TLS point cloud using the 3D model and the scanner’s location of the registered point cloud. This point cloud can be segmented in a similar way as with the as-built point cloud, so that each model object can be assigned a corresponding as-planned point cloud (sub-set from the complete as-planned TLS point cloud).
Turkan does not explicitly teach the one camera image or the plurality of camera images.
Braun, in the same field of endeavor of formwork analysis, teaches the one camera image or the plurality of camera images ([1.3 Contributions] We introduce a method based on visibility analysis to identify elements that are detectable from the identified camera positions).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Turkan with the teachings of Braun to use camera images because "The as-built status of a construction site is usually captured by laser scanners or cameras using SfM methods. Laser scanning…equipment is heavy and requires trained personnel…this method is more flexible and easier in its application, as camera equipment is standard, low-cost, and widely used on UAVs" [2.1 Scan vs BIM]
Regarding claim 8, Turkan and Braun teach the method of claim 1. Turkan further teaches wherein one or more control points are provided on the construction site and the data points in step ii) are oriented and/or scaled according to the one or more control points ([3.1 Registration of TLS Point Clouds with Building 3D Model] An initial coarse registration is performed, for example by manually matching n pairs of points selected in the 3D model and in the scan using commercially available cloud processing software. Another potentially simpler and more accurate coarse registration approach based on plane extraction and matching is also proposed in (Bosché, 2011). A robust Iterative Closest Point (ICP) - based algorithm using the point-to-plane framework (Chen-Medioni, 1991; Rusinkiewicz and Levoy, 2001) is then employed to perform the fine registration of the TLS point clouds with the building 3D model).
PNG
media_image2.png
163
818
media_image2.png
Greyscale
PNG
media_image2.png
163
818
media_image2.png
Greyscale
Regarding claim 11, Turkan and Braun teach the method of claim 1. Turkan further teaches wherein the formwork element is a frame formwork element or a ceiling formwork element.
PNG
media_image3.png
214
650
media_image3.png
Greyscale
Regarding claim 12, Turkan and Braun teach the method of claim 1. Turkan further teaches a method for monitoring a construction progress of a building comprising the following steps: a) repeatedly executing a detection method for detecting a building part on a construction site according to claim 1 (See Claim 1 above. [5. Experiments] In order to evaluate the performance of the proposed secondary and temporary construction object recognition techniques, experiments were conducted using a 3D model obtained for the Engineering V Building site at the University of Waterloo, Canada (Figure 3) and eleven different TLS point clouds acquired on eleven different days over a seven month period);
b) marking a building part as completed if the building part has been detected according to step vi) of the detection method and the data points belonging to the formwork element arranged for erecting the building part on the construction site are no longer assigned to the visible area of the building part when the detection method is carried out again in step v) ([5.2 Recognition of Formwork and Rebar (Step 2)] Technique 2 (Bosché and Haas’ algorithm with larger values of TD ranging from 10mm and 60mm in order to identify whether rebar and formwork could be distinctly differentiated from completed objects. Five of the Engineering V Building TLS point clouds were used here which collectively contain data from 111 column, floor and wall objects in different construction states, namely: “built” (i.e. completed), “formwork” and “rebar”. The detailed numbers of objects per category are given in Table 2. Examples of the scans used are shown in Figure 3).
PNG
media_image3.png
214
650
media_image3.png
Greyscale
Regarding claim 15, Turkan and Braun teach the method of claim 1. Turkan further teaches wherein the building part on the construction site is in a shell construction erection phase.
PNG
media_image4.png
638
778
media_image4.png
Greyscale
Regarding claim 17, Turkan and Braun teach the method of claim 1. Turkan further teaches wherein the data points are generated in the form of a point cloud ([4.4 Summary] 1. Apply Technique 1 (i.e. using Bosché and Haas’s algorithm with a small value of TD, i.e. 20mm) to recognize completed 3D model objects. Upon completion, remove the recognized objects from the 3D model and the corresponding matched points from the point cloud).
Regarding claim 19, Turkan and Braun teach the method of claim 8. Turkan does not explicitly teach wherein the one or more control points are one or more markers.
Braun, in the same field of endeavor of formwork analysis, teaches wherein the one or more control points are one or more markers ([5.2 Limitations] In our approach, this is achieved by markers on site. However, a minor manual step is required in order to find the exact orientation and scaling. Only after combining this data with the aligned building information model is it possible to gather additional information from the images in relation to the building model).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Turkan with the teachings of Braun to use markers as control points for "the requirement for a well-aligned BIM" [5.2 Limitations].
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Turkan in view of Braun and Cvetkovic (EP3705661A1).
Regarding claim 4, Turkan and Braun teach the method of claim 1. Turkan does not explicitly teach wherein a distinction is made on the basis of an orientation and a formwork type of the formwork element as to whether the formwork element is arranged in a forming position or a storage position, and the formwork element is only taken into account in step iii) if it is arranged in the forming position.
Cvetkovic, in the same field of endeavor of formwork detection, teaches wherein a distinction is made on the basis of an orientation and a formwork type of the formwork element as to whether the formwork element is arranged in a forming position or a storage position, and the formwork element is only taken into account in step iii) if it is arranged in the forming position ([pg. 5 para. 2] Such a position recognition in three-dimensional space also offers the positive effect that formwork lying horizontally or on a stack can also be recognized. Formwork lying horizontally over a large area is basically to be equated with stationary, i.e. inactive formwork. As soon as a rest position is recognized, it can be calculated and compared with the digital model whether the formwork is still in use or can be transported away. If a flat, horizontally lying formwork is detected, it can be set to inactive in the digital system be set).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Turkan with the teachings of Cvetkovic to make a distinction on the orientation and type of formwork as in forming position or storage position so that the storage formwork "can be transported away" [pg. 5 para. 2].
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Turkan in view of Braun and Zhao (Zhao, Linlin, et al. Installation Quality Inspection for High Formwork Using Terrestrial Laser Scanning Technology. Symmetry 2022, 14, 377).
Regarding claim 9, Turkan and Braun teach the method of claim 1. Turkan does not explicitly teach wherein data points that do not belong to any formwork element are discarded.
Zhao, in the same field of endeavor of formwork image analysis, teaches wherein data points that do not belong to any formwork element are discarded ([3.2 Data Extraction, 3.2.1 Removal of Mixed Pixels] The scan data of a high formwork usually include multiple objects such as vertical poles, horizontal tubes, couplers, braces, bolts, ground, timber formwork, and so on. Construction sites include other unexpected noise, including construction equipment, workers, and etc. In order to retain useful data relating to poles and tubes, the others must be removed. Unfortunately, mixed pixels that are a type of false measurement are always included in the laser scan data. It occurs when a laser beam is split into two parts and falls on two different objects. Hence, the laser scanner obtains two reflective signals that are from two different objects, and then generates the mixed pixel measurements that cannot represent either of the two objects. As mixed pixels adversely influence the recognition of the poles and tubes, it is necessary to remove the mixed pixels before further processing of the point data. Since mixed pixels are located at greater distances from their neighbours, an algorithm based on the average distance from one point to its k-nearest neighbours was employed in this study. If the average distance is bigger than a threshold value, the point is considered to be a mixed pixel. Moreover, in this study, colour information of the point data was used to filter out the mixed pixels).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Turkan with the teachings of Zhao to discard data points that do not belong to any formwork element because "In order to retain useful data relating to poles and tubes, the others must be removed" [3.2.1 Removal of Mixed Pixels].
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Turkan in view of Braun and Chida (US20230008631A1).
Regarding claim 10, Turkan and Braun teach the method of claim 3. Turkan does not explicitly teach wherein people are detected in the camera image or the plurality of camera images and/or in the data points in the three-dimensional coordinate system, and at least faces of the people are anonymised.
Chida, in the same field of endeavor of image detection, teaches wherein people are detected in the camera image or the plurality of camera images and/or in the data points in the three-dimensional coordinate system, and at least faces of the people are anonymised ([0044] The internal-environment imaging device 72 images the face of the occupant 74 in the vehicle 7. The ECU 702 obtains face information of the occupant 74 from the image of the face of the occupant 74. [0084] When image data is obtained, the specific action detector 208 may automatically pixelate or blur out faces included in the image indicated by the image data).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Turkan with the teachings of Chida to detect and anonymize faces because "Then, image capturing with the external-environment imaging device 71 is less likely to cause some trouble" [0155].
Regarding claim 20, Turkan, Braun, and Chida teach the method of claim 10. Turkan does not explicitly teach wherein at least faces of the people are anonymized by pixelation.
Chida, in the same field of endeavor of image detection, teaches wherein at least faces of the people are anonymized by pixelation ([0044] The internal-environment imaging device 72 images the face of the occupant 74 in the vehicle 7. The ECU 702 obtains face information of the occupant 74 from the image of the face of the occupant 74. [0084] When image data is obtained, the specific action detector 208 may automatically pixelate or blur out faces included in the image indicated by the image data).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Turkan with the teachings of Chida to anonymize faces with pixelation because "Then, image capturing with the external-environment imaging device 71 is less likely to cause some trouble" [0155].
Claims 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Turkan in view of Braun and Lu (US20220118994A1).
Regarding claim 13, Turkan teaches a system for detecting a building part on a construction site comprising: at least one scanner for acquiring scans from the construction site, wherein the at least one scanner can be arranged at an acquisition position ([Introduction] This paper describes three techniques for recognizing concrete construction secondary and temporary objects in TLS point clouds, two of which are based on the object recognition system developed by Bosché and Haas (2008) that uses a “Scan-vs-BIM” framework (Guillemet et al., 2012). Two of the techniques are tested using real-life data. Figure 6: Wall recognition performance (%recognized) for different values of TD and at different construction stages. The curves show the average and standard variation of %recognized);
ii) generating data points in a three-dimensional coordinate system the data points representing the construction site ([3 The Automated Object Recognition System] As a pre-processing step, the system requires converting the input 3D model into triangulated mesh format (e.g. STL, OBJ). It then follows a two-step process detailed below: 1. Registration of TLS point clouds with building 3D model 2. Recognition of 3D model objects in TLS point clouds);
iii) recognising data points in the three-dimensional coordinate system belonging to a formwork element on the construction site which is arranged in a forming position ([3 The Automated Object Recognition System] 2. Recognition of 3D model objects in TLS point clouds. [5.2 Recognition of Formwork and Rebar (Step 2)] Technique 2 (Bosché and Haas’ algorithm with larger values of TD was tested with values of TD ranging from 10mm and 60mm in order to identify whether rebar and formwork could be distinctly differentiated from completed objects…The experiment results are summarized in Figures 5 to 7 for columns, walls and floor slabs respectively. Overall, it appears, as expected, that recognition levels for objects in “formwork” state show very low %recognized for values of TD<30mm, but then clearly increase from 30mm onwards);
iv) determining that visible area of a building part to be erected in a stored building model, which is visible from the acquisition position according to a current construction progress, wherein the building model is stored in memory ([5 Experiments] In order to evaluate the performance of the proposed secondary and temporary construction object recognition techniques, experiments were conducted using a 3D model obtained for the Engineering V Building site at the University of Waterloo, Canada (Figure 3) and eleven different TLS point clouds acquired on eleven different days over a seven month period. [3.2 Recognition of 3D Model Objects in TLS point clouds] At the end of the registration process, the project 3D model and TLS point clouds are optimally registered. Because it is known to which object points were matched at the last iteration, each model object can be assigned a corresponding as-built point cloud (sub-set from the complete as-built TLS point cloud). The analysis of the as-built point cloud can then lead to the recognition of the object itself using a surface-based recognition metric (Bosché, 2010));
v) assigning the data points belonging to the formwork element to the visible area of the building part to be erected when the formwork element is arranged in the forming position for erecting the building part on the construction site ([3.2 Recognition of 3D Model Objects in TLS point clouds] At the end of the registration process, the project 3D model and TLS point clouds are optimally registered. Because it is known to which object points were matched at the last iteration, each model object can be assigned a corresponding as-built point cloud (sub-set from the complete as-built TLS point cloud). The analysis of the as-built point cloud can then lead to the recognition of the object itself using a surface-based recognition metric (Bosché, 2010). This metric further requires the calculation of a virtual as-planned TLS point cloud using the 3D model and the scanner’s location of the registered point cloud. This point cloud can be segmented in a similar way as with the as-built point cloud, so that each model object can be assigned a corresponding as-planned point cloud (sub-set from the complete as-planned TLS point cloud);
and vi) detecting the building part on the construction site when specified data points belonging to the formwork element is assigned to the visible area of the building part to be erected ([5 Experiments] In order to evaluate the performance of the proposed secondary and temporary construction object recognition techniques, experiments were conducted using a 3D model obtained for the Engineering V Building site at the University of Waterloo, Canada (Figure 3) and eleven different TLS point clouds acquired on eleven different days over a seven month period. [3.2 Recognition of 3D Model Objects in TLS point clouds] At the end of the registration process, the project 3D model and TLS point clouds are optimally registered. Because it is known to which object points were matched at the last iteration, each model object can be assigned a corresponding as-built point cloud (sub-set from the complete as-built TLS point cloud). The analysis of the as-built point cloud can then lead to the recognition of the object itself using a surface-based recognition metric (Bosché, 2010). This metric further requires the calculation of a virtual as-planned TLS point cloud using the 3D model and the scanner’s location of the registered point cloud. This point cloud can be segmented in a similar way as with the as-built point cloud, so that each model object can be assigned a corresponding as-planned point cloud (sub-set from the complete as-planned TLS point cloud. [3.2 Recognition of 3D Model Objects in TLS point clouds] The analysis of the as-built point cloud can then lead to the recognition of the object itself using a surface-based recognition metric (Bosché, 2010)… After converting the as-planned, occluded and as-built point clouds for each object into planned, occluded and recognized (as-built) surfaces, the percentage of recognition %recognized is calculated as:
PNG
media_image1.png
256
799
media_image1.png
Greyscale
[5.2 Recognition of Formwork and Rebar (Step 2)] The experiment results are summarized in Figures 5 to 7 for columns, walls and floor slabs respectively. Overall, it appears, as expected, that recognition levels for objects in “formwork” state show very low %recognized for values of TD<30mm, but then clearly increase from 30mm onwards).
Turkan does not explicitly at least one digital camera for acquiring camera images from the construction site, wherein the at least one digital camera can be arranged at an acquisition position; i) controlling the at least one digital camera to capture one or more camera images from the construction site; detecting the building part on the construction site when a specified number of data points belonging to the formwork element is assigned to the visible area of the building part to be erected.
Braun, in the same field of endeavor of formwork analysis, at least one digital camera for acquiring camera images from the construction site, wherein the at least one digital camera can be arranged at an acquisition position; i) controlling the at least one digital camera to capture one or more camera images from the construction site ([1.3 Contributions] A method is presented that makes use of the knowledge of construction methods and 4D data to adjust the detection thresholds (as-planned vs. as-built deviations allowed) according to their expected construction stage. This permits the detection of elements that are currently under construction and are, for example, covered by formwork. We introduce a method based on visibility analysis to identify elements that are detectable from the identified camera positions. [3.2. Point of departure] Firstly, image acquisition for the generation of point clouds and camera position estimation is required. The authors provided several studies on image acquisition and proposed a UAV-based method as it is more flexible in comparison to fixed cameras);
detecting the building part on the construction site when a specified number of data points belonging to the formwork element is assigned to the visible area of the building part to be erected ([3.6 Identified tasks during construction] Several tasks are required to construct in-situ concrete elements or similar elements. In concrete construction, formwork for in-situ concrete is the most common construction method. Several different methods are depicted in Fig. 1b) and d). All possible elements under construction are considered in order to detect formwork. In general, elements are counted as detected as soon as a certain amount of points per area [Pts/m2] with a distance of less than 2 cm are matched on the surface of the element. If the expected elements are not detected, the threshold for the maximum distance can be adjusted to take into account the fact that the formwork with a thickness of around 0.20 m might be currently in place. If this iteration brings positive results, the element can be marked as “under construction”.
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Turkan with the teachings of Braun to acquire camera images of the construction site because "The as-built status of a construction site is usually captured by laser scanners or cameras using SfM methods. Laser scanning…equipment is heavy and requires trained personnel…this method is more flexible and easier in its application, as camera equipment is standard, low-cost, and widely used on UAVs" [2.1 Scan vs BIM] and to detect the building part when a specified number of datapoints belonging to formwork is assigned because "Another reason for weak detection rates is building elements that are currently under construction. As those elements count towards the overall progress, they must not be missed, and play a crucial role in defining the exact state in the current process. In general challenges exist for all construction methods, whose geometry under construction differs largely from the final element geometry which requires the use of temporary construction objects" [4. Case Study].
Turkan does not explicitly teach and a processing unit configured to carry out the following steps: i) controlling the at least one digital camera to capture one or more camera images.
Lu, in the same field of endeavor of multi-camera system analysis, teaches a processing unit configured to carry out the following steps: i) controlling the at least one digital camera to capture one or more camera images ([0113] In particular embodiments, processor 1002 includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor 1002 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 1004, or storage 1006; decode and execute them. [0002] One example of such sensors may include a stereo camera pair installed on the vehicle to obtain stereo views that depict the surroundings in 3D. Specifically, the stereo camera pair includes two cameras placed in parallel, each of which takes an image of the same object from their respective positions, and the resulting images can be compared and analyzed to generate a 3D view of the object that simulates human binocular vision).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Turkan with the teachings of Cole to use a processing unit to control the digital camera because "the respective sensing data captured by the first sensor or the second sensor includes at least one of: a measurement indicating a distance between the calibration target and the first sensor or the second sensor, or imaging data depicting a size or a shape of the calibration target from a point of view at the first sensor or the second sensor" [0012] and "Processor 1002 may then load the instructions from memory 1004 to an internal register or internal cache. To execute the instructions, processor 1002 may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor 1002 may write one or more results (which may be intermediate or final results) to the internal register or internal cache" [0114].
Regarding claim 14, Turkan, Braun, and Lu teach the system of claim 13. Turkan does not explicitly teach wherein at least two digital cameras are provided, which are arranged on a support frame, the digital cameras being spaced apart from each other by at least 50 cm.
Lu, in the same field of endeavor of multi-camera system analysis, teaches wherein at least two digital cameras are provided, which are arranged on a support frame, the digital cameras being spaced apart from each other by at least 50 cm ([0039] As used herein, the term “wide baseline” is used to refer to a distance between two stereo cameras that is similar to the width of the roof of the vehicle, e.g., at least three feet apart, etc. In contrast, the term “short baseline” is used to refer to a distance between two stereo cameras that is relatively smaller, e.g., less than three feet, etc. [0083] FIG. 7 is a block diagram illustrating an example configuration of a mounting device 108a or 108b used to hold a camera sensor 110a or 110b on a vehicle as shown in FIG. 2A, according to an embodiment of the present technology. The mounting device may include a mounting support 162 such as a rigid stem structure to support a holder 165 at one end of the stem, while the other end of the mounting support 162 may be fixed on the top of the vehicle 105. The holder 165 may serve as a mounting plate such that the camera sensor (e.g., 110a-b) can be fixed thereon).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Turkan with the teachings of Cole to use at least two digital cameras arranged on a support frame spaced apart by at least 50 cm because "When the stereo cameras or other sensors are installed on a vehicle, the movement of the vehicle may constantly cause vibration, jitter, or other slight movement that leads to position shifts of the stereo camera or other sensor, leaving the stereo camera or other sensor non-calibrated. This non-calibration issue may be even more prominent in the scenario when a wide-baseline stereo camera pair is installed on the vehicle. As used herein, the term “wide baseline” is used to refer to a distance between two stereo cameras that is similar to the width of the roof of the vehicle, e.g., at least three feet apart, etc". [0039].
Claims 5-7 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Turkan in view of Braun and Taylor (US20210192784A1).
Regarding claim 5, Turkan and Braun teach the method of claim 1. Turkan does not explicitly teach wherein at least two digital cameras are provided at a respective acquisition position, wherein the at least two digital cameras form a stereo camera and the digital cameras each simultaneously record camera images.
Taylor, in the same field of endeavor of construction site analysis, teaches wherein at least two digital cameras are provided at a respective acquisition position, wherein the at least two digital cameras form a stereo camera and the digital cameras each simultaneously record camera images ([0144] In this embodiment the first camera assembly 41 is arranged as pairs of stereoscopic cameras that capture images simultaneously to generate a point cloud representation of a field of view of the camera pair. In this embodiment the pair of cameras are offset by 10°).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Turkan with the teachings of Taylor to form a stereo camera and simultaneously record because "each with a field of view that capture a portion of the first face such that the individual fields of view are combined to form the first field of view, and images from each of the array of four sensors is combined to form a first composite image of the first face, and the first plane is fitted to a point cloud representation of the first face using the first composite image" [0112].
Regarding claim 6, Turkan, Braun, and Taylor teach the method of claim 5. Turkan does not explicitly teach wherein in step ii) the data points in the three-dimensional coordinate system are generated from two simultaneously captured camera images of the at least two digital cameras.
Taylor, in the same field of endeavor of construction site analysis, teaches wherein in step ii) the data points in the three-dimensional coordinate system are generated from two simultaneously captured camera images of the at least two digital cameras ([0144] In this embodiment the first camera assembly 41 is arranged as pairs of stereoscopic cameras that capture images simultaneously to generate a point cloud representation of a field of view of the camera pair…The two side cameras capture a portion of the side face 129 as shown in FIG. 8A and a point cloud representation of the side face is then generated. This point cloud data is then combined with the 3D model of the brick (obtained from the laser scanner 29) to generate a 6DOF position for the brick).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Turkan with the teachings of Taylor to generate the 3D data points using two simultaneously captured camera images because "The found shape match is then transformed into to the point cloud co-ordinates, and the X value of the match is represented as a plane. The intersection of the line (step 6) and this plane is then used to define the centroid location of the brick, and the rotations can be calculated from the orientations of the top and side planes to determine the pose in 6DOF" [0152].
Regarding claim 7, Turkan and Braun teach the method of claim 1. Turkan does not explicitly teach wherein the acquisition position is arranged on a construction site equipment, or on another building.
Taylor, in the same field of endeavor of construction site analysis, teaches wherein the acquisition position is arranged on a construction site equipment, or on another building ([0133] Perspective and side views of an embodiment of an automated bricklaying robot 11 are shown in FIGS. 2A and 2B. In this embodiment automated brick laying robot machine 11 has a base 13 in the form of a truck with a turntable in the form of a tower (or turret) 17 supported on a vertical yaw axis, and an articulated arm having a telescoping boom 19 supported on the tower 17 about a horizontal pitch axis about which the arm may be raised or lowered. The boom 19 has a telescoping stick 21, mounted on the end of the boom 19 about a horizontal pivot axis, and an end effector 23 in the form of an adhesive applying and brick laying head 23 mounted to the remote end of the stick 21. [0062] a) a first sensor apparatus that in use is mounted to the end effector and having a first location and a first field of view to image a first face of the object when gripped by the end effector; [0063] b) a second sensor apparatus that in use is mounted to the end effector and having a second location and a second field of view to image at least a portion of a second face of the object orthogonal to the first face when gripped by the end effector).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Turkan with the teachings of Taylor for the acquisition position to be arranged on construction site equipment so that "a first sensor apparatus that in use is mounted to the end effector and having a first location and a first field of view to image a first face of the object when gripped by the end effector; a second sensor apparatus that in use is mounted to the end effector and having a second location and a second field of view to image at least a portion of a second face of the object orthogonal to the first face when gripped by the end effector" [0129].
Regarding claim 16, Turkan and Braun teach the method of claim 1. Turkan does not explicitly teach wherein there are at least two digital cameras spaced apart at the acquisition position acquiring the one or more camera images of the construction site.
Taylor, in the same field of endeavor of construction site analysis, teaches wherein there are at least two digital cameras spaced apart at the acquisition position acquiring the one or more camera images of the construction site ([0144] In this embodiment the first camera assembly 41 is arranged as pairs of stereoscopic cameras that capture images simultaneously to generate a point cloud representation of a field of view of the camera pair. In this embodiment the pair of cameras are offset by 10°).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Turkan with the teachings of Taylor for at least two digital cameras spaced apart at the acquisition site to acquire the images of the construction site because "each with a field of view that capture a portion of the first face such that the individual fields of view are combined to form the first field of view, and images from each of the array of four sensors is combined to form a first composite image of the first face, and the first plane is fitted to a point cloud representation of the first face using the first composite image" [0112].
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Turkan in view of Braun, Taylor, and Butcher (US20240425332A1).
Regarding claim 18, Turkan, Braun, and Taylor teach the method of claim 7. Turkan does not explicitly teach wherein the acquisition position is arranged on a crane or a post.
Butcher, in the same field of endeavor of construction site image analysis, teaches wherein the acquisition position is arranged on a crane or a post ([0048] The controller may be in direct or wired data communication with a camera located for viewing the area below the hook of the crane (herein, “hook camera”), such as to control the hook camera to obtain image data captured by the hook camera).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Turkan with the teachings of Butcher for the acquisition position to be arranged on a crane because "construction site personnel may describe and relay what they see and hear at the construction site to the operator in the cab using handheld radios, i.e. walkie talkies. This method is less than ideal because there is often communication lost or miscommunication. The operator is more likely to make a mistake if they do not have correct and/or up-to-date information about the construction site. A mistake can result in a serious accident which may injure or kill personnel or damage property. Devices have been developed that monitor the construction site; containing monitoring equipment, such as a camera, a controller and a transmitter, and are often suspended from the hoist line proximal to the hook. The camera feed is processed in the controller and fed to the operator via a transmitter to enable them to monitor the construction site" [0004-0005].
Allowable Subject Matter
Claim 2 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 2, Turkan teaches the data points belonging to the formwork element are assigned to the building part according to their position in the three- dimensional coordinate system, the building part being detected on the construction site ([3.2 Recognition of 3D Model Objects in TLS point clouds] At the end of the registration process, the project 3D model and TLS point clouds are optimally registered. Because it is known to which object points were matched at the last iteration, each model object can be assigned a corresponding as-built point cloud (sub-set from the complete as-built TLS point cloud). The analysis of the as-built point cloud can then lead to the recognition of the object itself using a surface-based recognition metric (Bosché, 2010). This metric further requires the calculation of a virtual as-planned TLS point cloud using the 3D model and the scanner’s location of the registered point cloud. This point cloud can be segmented in a similar way as with the as-built point cloud, so that each model object can be assigned a corresponding as-planned point cloud (sub-set from the complete as-planned TLS point cloud. 3.2 Recognition of 3D Model Objects in TLS point clouds] The analysis of the as-built point cloud can then lead to the recognition of the object itself using a surface-based recognition metric (Bosché, 2010)… After converting the as-planned, occluded and as-built point clouds for each object into planned, occluded and recognized (as-built) surfaces, the percentage of recognition %recognized is calculated as:
PNG
media_image1.png
256
799
media_image1.png
Greyscale
[5.2 Recognition of Formwork and Rebar (Step 2)] The experiment results are summarized in Figures 5 to 7 for columns, walls and floor slabs respectively. Overall, it appears, as expected, that recognition levels for objects in “formwork” state show very low %recognized for values of TD<30mm, but then clearly increase from 30mm onwards).
The following limitation was not found to be taught in the art: wherein the visible area of the building part to be erected is subdivided into virtual cells, and the data points belonging to the formwork element are assigned to the cells of the building part according to their position in the three- dimensional coordinate system, the building part being detected on the construction site when at least one data point is assigned, respectively, to a specified number of cells of the building part.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571)272-4077. The examiner can normally be reached M-F 9-5. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666 /Molly Wilburn/Primary Examiner, Art Unit 2666
Prosecution Insights