Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claim Objections
Claim 95 is objected to because of the following informalities: “wherein aligning the first 3D model from the second 3D model is based on the matched architectural element” should be “wherein aligning the first 3D model from the second 3D model is based on the matched non-architectural element”. Appropriate correction is required.
Allowable Subject Matter
Claims 26-28, 94, 95 (with objection correction), 120 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 5, 11, 22, 53, 59, 62, 65, 87, 88, 106, 122, 243 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen U.S. Patent/PG Publication 20160314592 in view of Taguchi U.S. Patent/PG Publication 20140003705.
Regarding claim 1 (independent):
A method of combining disparate 3d models of a common building object, the method comprising: (Chen [0047] Referring now to FIG. 4, the operations performed, such as by the apparatus 200 of FIG. 2, for registration of interior and exterior three dimensional scans based on a semantic feature of a structure are illustrated.)
receiving a first plurality of images (Chen [0049] As shown in block 404 of FIG. 4, the apparatus 200 may include means, such as a processor 202, memory 204, a communications interface 206, or the like, configured to receive at least one interior scan of the structure.).
generating a first 3d model based on the first plurality of images, wherein the first 3d model is an interior model of the building object (Chen [0039] The UE 102 or scan server 104 may generate a composite structure model based on the registered exterior and interior scans. The model can be defined with standard 3D model format, such as the WaveFront Obj format or the PLY format. The UE 102 or scan server 104 may store the composite structure model in a memory such as the scan database 106, or output the composite structure model to a display of the UE.).
receiving a second plurality of images (Chen [0047] As shown in block 402 of FIG. 4, the apparatus 200 may include means, such as a processor 202, memory 204, a communications interface 206, or the like, configured to receive at least one external scan of a structure.).
generating a second 3d model based on the second plurality of images, wherein the second 3d model is an exterior model of the building object (Chen [0039] The UE 102 or scan server 104 may generate a composite structure model based on the registered exterior and interior scans. The model can be defined with standard 3D model format, such as the WaveFront Obj format or the PLY format. The UE 102 or scan server 104 may store the composite structure model in a memory such as the scan database 106, or output the composite structure model to a display of the UE.).
and aligning, in a common 3d coordinate system, the first 3d model with the second 3d model, (Chen [0057] As shown at block 412 of FIG. 4, the apparatus 200 may include means, such as a processor 202, or the like, configured to register the at least one exterior scan and the at least one interior scan.).
wherein the first 3d model and the second 3d model comprise (Chen [0060] The processor 202 may generate the composite structure model based on the registered exterior and interior scans. The processor 202 may store the composite structure model in a memory 204, such as the scan database 106. The processor 202 may output the composite structure model to a user interface 208 for display.)(Chen [0030] In an instance in which a heuristic method is utilized, the UE 102 or scan server 104 may detect plane primitives in the point clouds of the external and interior scans.)
Chen does not teach primitive based models. In a related field of endeavor, Taguchi teaches:
receiving a first plurality of images (Taguchi [0030] An example input 201 to the SLAM system is a set of one or more depth maps 202. A depth map describes a depth value (distance) of each pixel in the map, thus providing a 3D point cloud by back-projecting the pixels in the map according to their depth values. The SLAM system can optionally acquire a set of one or more 2D texture images 203 corresponding to the depth maps. The texture images can be gray-scale or have RGB color channels. The depth maps can be acquired by a 3D sensor, e.g., a passive stereo camera 204 or an active IR or laser scanner 205, while the texture images can be acquired by a camera, e.g., a single camera in the stereo camera 204 or a dedicated camera 206.)
generating a first 3d model based on the first plurality of images, wherein the first 3d model is an interior model of the building object (Taguchi [0030] A depth map describes a depth value (distance) of each pixel in the map, thus providing a 3D point cloud by back-projecting the pixels in the map according to their depth values.)(Taguchi [0089] The invention provides a registration method that uses both points and planes as primitives to register 3D data in two different coordinate systems. Using the registration method, a real-time SLAM system for 3D sensors that uses both point-to-point and plane-to-plane correspondences for registration is developed. Using both primitives enables faster and more accurate registration than only using points. Our SLAM system generates a 3D model as a set of planes, which provides more compact and semantic information of the scene than, strictly point-based representations.)(Taguchi Fig. 4)
receiving a second plurality of images; (Taguchi [0030] our registration method determines the motion of the sensors and reconstructs a scene by registering 3D point clouds acquired at different viewpoints.) generating a second 3d model based on the second plurality of images, wherein the second 3d model is (Taguchi [0030] An example input 201 to the SLAM system is a set of one or more depth maps 202. A depth map describes a depth value (distance) of each pixel in the map, thus providing a 3D point cloud by back-projecting the pixels in the map according to their depth values. The SLAM system can optionally acquire a set of one or more 2D texture images 203 corresponding to the depth maps. The texture images can be gray-scale or have RGB color channels. The depth maps can be acquired by a 3D sensor, e.g., a passive stereo camera 204 or an active IR or laser scanner 205, while the texture images can be acquired by a camera, e.g., a single camera in the stereo camera 204 or a dedicated camera 206.)
and aligning, in a common 3d coordinate system, the first 3d model with the second 3d model, (Taguchi [0029] FIG. 1 shows a schematic of the registration method 100. The registration method determines a 6-DOF rigid body transformation 105 between two different coordinate systems 101 and 102 by using two sets of at least three primitives selected from 3D data in the two different coordinate systems. The primitives include 3D points 110 and 3D planes 120. The three primitives in each set include at least one 3D plane, at least one 3D point, and a third primitive that can be either a point or a plane. Planes in the two sets are registered with planes, and points are registered with points. The registration method can be global and performed in real time.)
wherein the first 3d model and the second 3d model comprise primitive-based models (Taguchi [0089] Using both primitives enables faster and more accurate registration than only using points. Our SLAM system generates a 3D model as a set of planes, which provides more compact and semantic information of the scene than, strictly point-based representations.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to use primitive models as taught by Taguchi. The motivation for doing so would have been faster and more accurate registration (Taguchi [0089]). Therefore it would have been obvious to combine Taguchi with Chen to obtain the invention.
Regarding claim 5:
The method of claim 1, has all of its limitations taught by Chen in view of Taguchi. Chen further teaches wherein the first plurality of images comprises an interior of the building object and wherein the second plurality of images comprises an exterior of the building object (Chen [0002] There are currently several systems, such as laser scanners, for obtaining a three dimensional scan of the exterior of a structure, such as a building. [0003] A method and apparatus are provided in accordance with an example embodiment for registration of interior and exterior three dimensional scans based on a semantic feature of a structure.).
Regarding claim 11:
The method of claim 1, has all of its limitations taught by Chen in view of Taguchi. Taguchi further teaches further comprising generating the common 3d coordinate system by matching the first coordinate system with the second coordinate system (Taguchi [0029] FIG. 1 shows a schematic of the registration method 100. The registration method determines a 6-DOF rigid body transformation 105 between two different coordinate systems 101 and 102 by using two sets of at least three primitives selected from 3D data in the two different coordinate systems. The primitives include 3D points 110 and 3D planes 120. The three primitives in each set include at least one 3D plane, at least one 3D point, and a third primitive that can be either a point or a plane. Planes in the two sets are registered with planes, and points are registered with points. The registration method can be global and performed in real time.)
Therefore, it would have been obvious before the effective filing date of the claimed invention to match coordinate systems as taught by Taguchi. The motivation for doing so would have been to create a complete reconstruction of a scene (Taguchi [0005]). Therefore it would have been obvious to combine Taguchi with Chen to obtain the invention.
Regarding claim 22:
The method of claim 1, has all of its limitations taught by Chen in view of Taguchi. Chen further teaches further comprising:
identifying a first plurality of sides of the first 3D model (Chen [0030] In an instance in which a heuristic method is utilized, the UE 102 or scan server 104 may detect plane primitives in the point clouds of the external and interior scans.).
and identifying a second plurality of sides of the second 3D model, wherein aligning the first 3D model with the second 3D model comprises aligning the first plurality of sides with the second plurality of sides in the common 3d coordinate system such that the first plurality of sides are within a threshold number of 3D pixels of the second plurality of sides (Chen [0035] The UE 102 or scan server 104 may apply an iterative closest point (ICP) algorithm to find a rigid transformation between the exterior scans and interior scans. For each corresponding external and interior semantic feature, e.g. window pair W.sub.e and W.sub.i, the UE or scan server 104 may maximize the overlap between the semantic features and minimize the orientation difference of the plane to which the respective semantic feature corresponds, denoted N.sub.e and N.sub.i.).
Regarding claim 53:
The method of claim 1, has all of its limitations taught by Chen in view of Taguchi. Chen further teaches further comprising:
identifying a first plurality of elements of the first 3D model; identifying a second plurality of elements of the second 3D model; and correlating the first plurality of elements with the second plurality of elements, wherein aligning the first 3D model with the second 3D model is based on the correlated plurality of elements (Chen [0031] Semantic features, such as windows, doors, or the like, may be recessed from both exterior and interior walls.)(Chen [0035] The UE 102 or scan server 104 may apply an iterative closest point (ICP) algorithm to find a rigid transformation between the exterior scans and interior scans. For each corresponding external and interior semantic feature, e.g. window pair W.sub.e and W.sub.i, the UE or scan server 104 may maximize the overlap between the semantic features and minimize the orientation difference of the plane to which the respective semantic feature corresponds, denoted N.sub.e and N.sub.i.).
Regarding claim 59:
The method of claim 53, has all of its limitations taught by Chen in view of Taguchi. Chen further teaches further comprising:
identifying a third plurality of elements, wherein the third plurality of elements comprises elements common to the first plurality of elements and the second plurality of elements wherein aligning the first 3D model with the second 3D model is based on the third plurality of elements (Chen [0031] Semantic features, such as windows, doors, or the like, may be recessed from both exterior and interior walls.)(Chen [0035] The UE 102 or scan server 104 may apply an iterative closest point (ICP) algorithm to find a rigid transformation between the exterior scans and interior scans. For each corresponding external and interior semantic feature, e.g. window pair W.sub.e and W.sub.i, the UE or scan server 104 may maximize the overlap between the semantic features and minimize the orientation difference of the plane to which the respective semantic feature corresponds, denoted N.sub.e and N.sub.i.).
Regarding claim 62:
The method of claim 53, has all of its limitations taught by Chen in view of Taguchi. Chen further teaches further comprising:
identifying an aspect of an element of the first plurality of elements; identifying a corresponding aspect of a corresponding element of the second plurality of elements (Chen [0031] Semantic features, such as windows, doors, or the like, may be recessed from both exterior and interior walls.)
wherein aligning the first 3D model with the second 3D model comprises aligning the aspect of the element of the first plurality of elements with the corresponding aspect of the corresponding element of the second plurality of elements (Chen [0035] The UE 102 or scan server 104 may apply an iterative closest point (ICP) algorithm to find a rigid transformation between the exterior scans and interior scans. For each corresponding external and interior semantic feature, e.g. window pair W.sub.e and W.sub.i, the UE or scan server 104 may maximize the overlap between the semantic features and minimize the orientation difference of the plane to which the respective semantic feature corresponds, denoted N.sub.e and N.sub.i.).
Regarding claim 65:
The method of claim 1, has all of its limitations taught by Chen in view of Taguchi. Chen further teaches further comprising:
identifying a first plurality of elements of the first plurality of images; identifying a second plurality of elements of the second plurality of images; and correlating the first plurality of elements with the second plurality of elements, wherein aligning the first 3D model with the second 3D model is based on the correlated plurality of elements (Chen [0031] Semantic features, such as windows, doors, or the like, may be recessed from both exterior and interior walls.)(Chen [0035] The UE 102 or scan server 104 may apply an iterative closest point (ICP) algorithm to find a rigid transformation between the exterior scans and interior scans. For each corresponding external and interior semantic feature, e.g. window pair W.sub.e and W.sub.i, the UE or scan server 104 may maximize the overlap between the semantic features and minimize the orientation difference of the plane to which the respective semantic feature corresponds, denoted N.sub.e and N.sub.i.).
Regarding claim 87:
The method of claim 1, has all of its limitations taught by Chen in view of Taguchi. Chen further teaches further comprising:
matching an architectural element of the first 3D model with a corresponding architectural element of the second 3D model wherein aligning the first 3D model from the second 3D model is based on the matched architectural element (Chen [0031] Semantic features, such as windows, doors, or the like, may be recessed from both exterior and interior walls.)(Chen [0035] The UE 102 or scan server 104 may apply an iterative closest point (ICP) algorithm to find a rigid transformation between the exterior scans and interior scans. For each corresponding external and interior semantic feature, e.g. window pair W.sub.e and W.sub.i, the UE or scan server 104 may maximize the overlap between the semantic features and minimize the orientation difference of the plane to which the respective semantic feature corresponds, denoted N.sub.e and N.sub.i.).
Regarding claim 88:
The method of claim 1, has all of its limitations taught by Chen in view of Taguchi. Chen further teaches further comprising:
matching an architectural element of the first plurality of images with a corresponding architectural element of the second plurality of images; wherein aligning the first 3D model from the second 3D model is based on the matched architectural element (Chen [0031] Semantic features, such as windows, doors, or the like, may be recessed from both exterior and interior walls.)(Chen [0035] The UE 102 or scan server 104 may apply an iterative closest point (ICP) algorithm to find a rigid transformation between the exterior scans and interior scans. For each corresponding external and interior semantic feature, e.g. window pair W.sub.e and W.sub.i, the UE or scan server 104 may maximize the overlap between the semantic features and minimize the orientation difference of the plane to which the respective semantic feature corresponds, denoted N.sub.e and N.sub.i.).
Regarding claim 106:
The method of claim 1, has all of its limitations taught by Chen in view of Taguchi. Chen further teaches wherein the first plurality of images are associated with a first plurality of camera poses, wherein the second plurality of images are associated with a second plurality of camera poses, and wherein aligning the first 3D model with the second 3D model is based on relative locations of the first plurality of camera poses and the second plurality of camera poses (Chen [0027] The scan may be associated with location information, e.g. a two dimensional or three dimensional location, such as GPS, radio frequency identification (RFID), or the like. In some embodiments location data may be a calculated location based on an inertial measuring unit, image based feature tracking, RFID proximity, or the like. [0028] The UE 102 or image server 104 may receive one or more external scans of the structure and one or more interior scans of the structure from the scan capture device 108 and/or from the scan database 106. The scans may include a one or more structures, features of the one or more structures, or the like. The interior scans may be registered based on the associated location data, which may include inaccuracy and drift.)(Chen [0034] The UE 102 or scan server 104 may perform a rough registration of the interior scans to the external scan, based on the location data associated with external scans, such as GPS and the location data associated with the interior scans, such as IMU and image based feature tracking. Each of the detected semantic features may be treated a single three dimensional point for the rough registration.).
Regarding claim 122 (independent):
The claim is a parallel version of claim 1. As such it is rejected under the same teachings.
Regarding claim 243 (independent):
The claim is a parallel version of claim 1. As such it is rejected under the same teachings.
Claim(s) 101 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen U.S. Patent/PG Publication 20160314592 in view of Taguchi U.S. Patent/PG Publication 20140003705 in view of Fan U.S. Patent/PG Publication 10776936.
Regarding claim 101:
The method of claim 1, has all of its limitations taught by Chen in view of Taguchi. Chen in view of Taguchi does not teach scaling. In a related field of endeavor, Fan teaches:
wherein aligning the first 3D model with the second 3D model comprises scaling the first 3D model based on the second 3D model (Fan C5 L55- C6 L5 A scale difference between the first and the second 3D point cloud is removed based on a mean distance of points in a subset of first 3D point cloud and a corresponding subset of the second 3D point cloud (202). The first and the second 3D point cloud are arranged in a two-level structure, wherein a first level is a macro structure describing boundaries of the underlying object and a second level is a micro structure consisting of supervoxels comprising points of 3D point cloud having similar properties clustered together (204). Then a first graph is constructed from the first 3D point cloud and a second graph is constructed from the second 3D point cloud such that the supervoxels represent nodes of the graphs and adjacent relations of the supervoxels represents edges of the graphs (206). A graph matching process is carried out between the first and second graph for obtaining a transformation matrix (208) and the first and the second 3D point clouds are registered together by applying the transformation matrix (210).).
Therefore, it would have been obvious before the effective filing date of the claimed invention to use scaling with registration as taught by Fan. The motivation for doing so would have been to reduce registration errors. Further, the rationale for doing so would have been it is applying a known technique to a known device ready for improve to yield predictable results, where Chen and Taguchi use registration, and Fan uses registration but with the technique of scaling, where there are predictable results since the end result if still registration. Therefore it would have been obvious to combine Fan with Chen in view of Taguchi to obtain the invention.
Conclusion
For the prior art referenced and the prior art considered pertinent to Applicant’s disclosure but not relied upon, see PTO-892 “Notice of References Cited”.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON PRINGLE-PARKER whose telephone number is (571) 272-5690 and e-mail is jason.pringle-parker@uspto.gov. The examiner can normally be reached on 8:30am-5:00pm est Monday-Friday. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, King Poon can be reached on (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, seehttp://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON A PRINGLE-PARKER/
Primary Examiner, Art Unit 2617
