DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 10/14/2025 has been entered.
Response to Amendment
This is in response to applicant’s amendment/response filed on 09/17/2025, which has been entered and made of record. Claims 1, 7, and 11-12 have been amended. Claim 3 has been cancelled. Claims 1-2 and 5-12 are pending in the application.
Response to Arguments
Applicant's arguments filed on 09/17/2025 have been fully considered but they are not persuasive.
Applicants state that “it is respectfully submitted that any combination of Patkar and Metzler fails to teach "(ii) correcting the first image using the first line, correcting the second image using a second line corresponding to the contour in the second image, and performing matching by searching for similar points between the corrected first image and the corrected second image," as required by the above-noted features of claim 1. In view of the above, Applicant respectfully submits that any combination of Patkar and Metzler fails to disclose, suggest, or otherwise render obvious the above-noted features of claim 1. Accordingly, claim 1 is patentable over any combination of Patkar and Metzler. Claims 2 and 8-10 are patentable over any combination of Patkar and Metzler based at least on their dependency from claim 1. Claims 11 and 12 recite features generally corresponding to the above-noted features of claim 1. Accordingly, Applicant respectfully submits that any combination of Patkar and Metzler fails to disclose, suggest, or otherwise render obvious these corresponding features of claims 11 and 12 for reasons similar to those discussed above with respect to claim 1, and as such, claims 11 and 12 are patentable over any combination of Patkar and Metzler”. The examiner disagrees. Prior art teach correcting the first image using the first line, correcting the second image using a second line corresponding to the contour in the second image (Patkar et al.: par 0042, “ given a set of images depicting a number of 3D points from different viewpoints, bundle adjustment may include refining 3D coordinates describing the scene geometry and camera poses of the camera(s) used to acquire the images. The process refines the various parameters noted according to an optimality criterion. In one aspect, the optimality criterion may involve the corresponding image projections of feature points and/or image projections of edge”, par 0065-0067, “the system may correct orthogonality between the first dominant axis and the second dominant axis” ….refine image based on scene geometry), and performing matching by searching for a similar points between the corrected first image and the corrected second image (Patkar et al.: par 0036-0041, “The system may detect feature points using any of a variety of known feature point detection techniques. In block 215, the system may match feature points across images. For example, the system determines which feature point in a first image matches, or is the same as, a feature point found in another, different image. In one aspect, the system may consider pairs of images and determine a matching of feature points for the pairs…. Pairwise matches between feature points may be organized into tracks. In one example, each track may specify a list of image-feature point pairs, where each feature point in a track is identified as a same feature point across the images listed in the track…. the system may determine a set of feature points that are matched across the images. The set of feature points may be the entirety of the feature points within the images or a subset of the feature points detected within the images… the system may determine a set of edges that are matched across the images. The set of edges may be the entirety of the edges detected within the images or a subset of the edges detected within the images” …..match feature point between two images, Metzler et al.: par 0090, “In order to further resolve ambiguity in the conglomeration of the first and second 3D-model 4a,4b in an additional degree of freedom, there is a second pair of line sections 2b, comprising a third line section 2b defined in the first 2D-visual-image 1a and a corresponding fourth line section 2b in the second 2D visual picture 1b shown, which are located at the top corner of the room. In many instances of the present invention, there can be an automatic matching of pairs of the line features 2a,2b in-between the first and second 2D-visual image, e.g. as often only one matching is geometrically resolvable and/or makes technical sense”, par 0098, “Such can e.g. also comprise an automatic snapping to an image detected visual edge feature. Also the intersection point 6am is manually corrected. The line segment 2c in the 2D-visual-image got the geometric restriction 5a applied, according to which it is substantially perpendicular to the line sections 2a and 2b”, par 0101, “In particular the line segments 2 can therein be matched to corresponding plan features 2p and 2h of the plan 9. For example, the feature 2p can be an Edge 2 as defined in the images at FIG. 8a and FIG. 8b and the feature 2h can be a height reference or horizontal reference 2 as indicated in those images”). So the rejection of claim 1 would be maintained. Same reason for independent claims 11 and 12 and dependent claims 2 and 8-10.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2 and 8-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2016/0098858 to Patkar et al. in view of U.S. PGPubs 20190162534 to Metzler et al..
Regarding claim 1, Patkar et al. teach a three-dimensional model generation method executed by a computer (abstract, par 0006), the three-dimensional model generation method comprising:
obtaining a first image generated by shooting a subject from a first viewpoint and a second image generated by shooting the subject from a second viewpoint (par 0032-0035, “ system 100 may receive a plurality of 2-dimensional (2D) images 160. 2D images 160 may be of a particular scene. 2D images 160 may be taken using one camera or using different cameras … The cameras, for example, do not require additional sensors. The images may be captured by a same camera or by two or more different cameras”);
detecting, in at least the first image, a first line composed of a series of edges and corresponding to a contour of the subject (par 0038-0039, “the system may detect edges within the images. The system may implement an edge detection technique to detect one or more edges within the received images. Any of a variety of known edge detection technique may be used. In another aspect, any of a variety of known line detection techniques may be used to identify edges. In block 225, the system may match detected edges across the images”); and
generating first three-dimensional points representing the contour, in a three-dimensional space in the computer, based on the first line (par 0021-0022, “This disclosure relates to 3-dimensional (3D) model generation using edges. In accordance with the inventive arrangements described within this disclosure, 2-dimensional (2D) images may be processed to identify edges contained therein. The edges may be processed to identify same edges across different ones of the 2D images. Camera poses may be estimated using a cost function that depends, at least in part, upon the edges ….3D models may be generated using 3D points, triangular meshes, 3D voxels, or the like. In one aspect, generating the 3D model may include one or more additional operations such as determining axes, determining planes based upon the edges, removing false planes, and rendering the planes in 3D space. The planes may represent textureless regions” par 0050-0052, “he camera poses define the position and orientation of the camera(s). In one aspect, using the determined camera poses, a 3D model may be generated …the system may perform dense reconstruction. Dense reconstruction generally refers to the creation of a 3D model, e.g., a dense point cloud, from the collected images of a scene using the camera poses. The dense point cloud includes the set of feature points and the set of edges”),
wherein the generating the first three-dimensional points includes:(i) correcting, based on the first line, positions of third three-dimensional points that correspond to the contour, the third three-dimensional points being among second three-dimensional points generated by performing matching by searching for similar points between the first image and the second image, the second three- dimensional points representing the subject; or (ii) correcting the first image using the first line, correcting the second image using a second line corresponding to the contour in the second image(par 0042, “ given a set of images depicting a number of 3D points from different viewpoints, bundle adjustment may include refining 3D coordinates describing the scene geometry and camera poses of the camera(s) used to acquire the images. The process refines the various parameters noted according to an optimality criterion. In one aspect, the optimality criterion may involve the corresponding image projections of feature points and/or image projections of edge”, par 0065-0067, “the system may correct orthogonality between the first dominant axis and the second dominant axis” ….refine image based on scene geometry), and performing matching by searching for similar points between the corrected first image and the corrected second image (par 0036-0041, “The system may detect feature points using any of a variety of known feature point detection techniques. In block 215, the system may match feature points across images. For example, the system determines which feature point in a first image matches, or is the same as, a feature point found in another, different image. In one aspect, the system may consider pairs of images and determine a matching of feature points for the pairs…. Pairwise matches between feature points may be organized into tracks. In one example, each track may specify a list of image-feature point pairs, where each feature point in a track is identified as a same feature point across the images listed in the track…. the system may determine a set of feature points that are matched across the images. The set of feature points may be the entirety of the feature points within the images or a subset of the feature points detected within the images… the system may determine a set of edges that are matched across the images. The set of edges may be the entirety of the edges detected within the images or a subset of the edges detected within the images” …..match detected edges/lines across images (first image and second image)(par 0039)).
But Patkar et al. keep silent for teaching generating first three-dimensional points representing the contour, in a three-dimensional space in the computer, based on the first image, the second image, and the first line.
PNG
media_image1.png
322
440
media_image1.png
Greyscale
In related endeavor, Metzler et al. teach generating first three-dimensional points representing the contour, in a three-dimensional space in the computer, based on the first image, the second image, and the first line (par 0100-0101, “By FIG. 8a to FIG. 8d, there is another embodiment of the surveying of a true-to-size 3D-model according to the invention illustrated. As shown before, FIG. 8a as well as FIG. 8b are showing 2D-visual-images, in which line sections 2 are overlaid, by which the corresponding 3D-partial models are conglomerated to a 3D-model 4 as illustrated in FIG. 8d. In the shown example, the 2D-visual-images are overlaid by some first line-segments 2a,2b,2c as well as by some elucidations to aid the description of the invention—like the texts next to the line-segments, the heading, etc.—which are not necessarily mandatory to be shown in practical embodiments …. the feature 2p can be an Edge 2 as defined in the images at FIG. 8a and FIG. 8b and the feature 2h can be a height reference or horizontal reference 2 as indicated in those images. Another option which is here shown, but which could also be considered independently of the 3D-plan-view is a 2D-view 10 in which information regarding the field of view of the device can be graphically defined and provided, in particular as additional information for the conglomeration of the 3D-partial models 4a,4b, whereby additional geometric restriction for the combination of the 3D-partial-models 4a,4b can be derived which aid the process of combining them in a true to size manner. In this example, the (at least roughly) indicated marking 10v refers to a selected wall of the floor-plan 10 and/or 9, which indicates the wall which shown in the picture 1b of FIG. 8b in the floor-plan, and can provide a linking information in-between the 3D-partial-models (and/or pictures 1a, 1b), and the floor plan—which information can be used to derive the true to size 3D-model for the floor-plan on basis of the 3D-measurements of the 3D-partial-models which are combined according to the information derived by the line segments indicated in corresponding 2D-pictures which were captured”), wherein the generating the first three-dimensional points includes:(i) correcting, based on the first line, positions of third three-dimensional points that correspond to the contour, the third three-dimensional points being among second three-dimensional points generated by performing matching by searching for similar points between the first image and the second image, the second three- dimensional points representing the subject; or (ii) correcting the first image using the first line, correcting the second image using a second line corresponding to the contour in the second image, and performing matching by searching for similar points between the corrected first image and the corrected second image (par 0090, “In order to further resolve ambiguity in the conglomeration of the first and second 3D-model 4a,4b in an additional degree of freedom, there is a second pair of line sections 2b, comprising a third line section 2b defined in the first 2D-visual-image 1a and a corresponding fourth line section 2b in the second 2D visual picture 1b shown, which are located at the top corner of the room. In many instances of the present invention, there can be an automatic matching of pairs of the line features 2a,2b in-between the first and second 2D-visual image, e.g. as often only one matching is geometrically resolvable and/or makes technical sense”, par 0098, “Such can e.g. also comprise an automatic snapping to an image detected visual edge feature. Also the intersection point 6am is manually corrected. The line segment 2c in the 2D-visual-image got the geometric restriction 5a applied, according to which it is substantially perpendicular to the line sections 2a and 2b”, par 0101, “In particular the line segments 2 can therein be matched to corresponding plan features 2p and 2h of the plan 9. For example, the feature 2p can be an Edge 2 as defined in the images at FIG. 8a and FIG. 8b and the feature 2h can be a height reference or horizontal reference 2 as indicated in those images”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Metzler et al. to include generating first three-dimensional points representing the contour, in a three-dimensional space in the computer, based on the first image, the second image, and the first line as taught by Kempinski to combine of the first 3D-partial-model with the second 3D-partial-model to form the 3D-model of the whole 3D-scene which is desired to be measured to improve a deriving of a true-to-size 3D-model of a 3D-scene to be measured or surveyed, in particular in a way which is sufficiently accurate for construction work, robust and easy to be handled.
Regarding claim 2, Patkar et al. as modified by Metzler et al. teach all the limitation of claim 1, and further teach wherein the first line is represented in units smaller than pixel units in the first image (Patkar et al.: Figs 6-8, par 0069-0071, “The dense point cloud further illustrates the edges 605 matched across the images and positioned within the dense point cloud after the three axes have been determined”, Metzler et al.: par 0040, “In particular, the edge- or corner feature can be derived by an automatic pattern recognition unit. For example, an automatic fine snapping can be derived with a sub-pixel resolution of the 2D-picture, e.g. when the defining comprises an automatic identifying of a potential candidate for the at least one visual feature for aligning one of the line segments by an artificial intelligence computing unit or the like. There can be an automated fine-snapping of the at least one line section to such an automatically identified 2D-visual-image feature, e.g. for fine aligning of the least one line section within one of the 2D-visual images, without requiring an operator to establish a highly accurate alignment by hand. For example, line segments can be derived in sub-pixel accuracy from the 2D-images by edge-extraction and/or line-fitting algorithms”, par 0096, “The detection unit can therein comprise image processing means for deriving and/or identifying image features such as e.g. edges, corners, lines, intersections, etc., preferably in sub-pixel resolution of the 2D-visual-image. In another embodiment according to the invention, the operator can manually define the line-sections 2 in the 2D-visual-image according to the invention, preferably aided by the automated image processing features like e.g. an automatic snapping function for snapping an line-section 2, handle 7, point 6a, face 6b, etc. to a image-detected edge-feature in the 2D-visual-image 1, which is close to the location where the operator places the line-section 2”).
Regarding claim 8, Patkar et al. as modified by Metzler et al. teach all the limitation of claim 1, and Patkar et al. further teach wherein the generating the first three-dimensional points removing, from the first three- dimensional points, three-dimensional points located at positions whose distance from the first viewpoint is less than a predetermined distance (par 0072-0073, “Planes 810 and 815 illustrate examples where a plurality of planes are located within a predetermined distance of one another and may be merged into a single plane. Planes are merged, as discussed, prior to removing or identifying false planes …..FIG. 9 is another overhead plan view of the dense point cloud of FIG. 6 after removing false planes. As illustrated, false planes, including false planes 805, have been removed from the overhead plan view leaving what are considered true planes that may be rendered as interior walls or doors in this example. In the example of FIG. 9, false plane removal may be performed on a per segment basis. As pictured, some segments 820 were determined to be false planes and removed while others were determined to be true and remain”).
Regarding claim 9, Patkar et al. as modified by Metzler et al. teach all the limitation of claim 1, and Patkar et al. further teach wherein in the generating the first three-dimensional points an edge detected from an image having a predetermined resolution or higher between the first image and the second image, is not used (par 0022, “ generating the 3D model may include one or more additional operations such as determining axes, determining planes based upon the edges, removing false planes, and rendering the planes in 3D space. The planes may represent textureless regions”, par 0059-0060, “only those segments of a plane determined to be false using the criteria described above may be removed. This allows a portion of a plane, e.g., one or more segments, to remain as a true plane, while one or more other segments of the plane determined to be false may be removed ….once the true planes are determined and false planes removed, a more accurate representation of walls in the scene is obtained. In that case, the system may correct, or adjust, the point cloud with respect to the determined planes to better align both the representations, e.g., the planes and the dense point cloud”, par 0108, “ the processor may be programmed to initiate executable operations including determining planes for edges of the set of edges, identifying false planes from among the determined planes, and removing the false planes”).
Regarding claim 10, Patkar et al. as modified by Metzler et al. teach all the limitation of claim 1, and further teach wherein the detecting includes detecting, in the second image, the second line corresponding to the contour of the subject, the second line being composed of a series of edges (Patkar et al.: par 0033, “System 100 further may determine a set edges from the plurality of edges that are matched across the 2D images 160. System 100 further may estimate camera poses 165 for 2D images 160, using a cost function that depends upon the set of edges. The system may also generate a 3D model 170 using the camera poses”, par 0036-0042, “In block 215, the system may match feature points across images. For example, the system determines which feature point in a first image matches, or is the same as, a feature point found in another, different image. In one aspect, the system may consider pairs of images and determine a matching of feature points for the pairs. Each pair of images may considered for purposes of matching feature points. ….the system may create tracks for the edges. A track may specify one or more pairs of images and a matched edge for each image. For example, a track for edges may specify a list of image-edge pairs, where each edge in a track is identified as a same edge across the images listed in the track. In some cases, the same edge may be not be found or located in each image….In block 230, the system may perform bundle adjustment. In general, bundle adjustment refers to a process where 3D coordinates describing scene geometry and camera poses for the cameras used to capture the images are determined. For example, given a set of images depicting a number of 3D points from different viewpoints, bundle adjustment may include refining 3D coordinates describing the scene geometry and camera poses of the camera(s) used to acquire the images. The process refines the various parameters noted according to an optimality criterion. In one aspect, the optimality criterion may involve the corresponding image projections of feature points and/or image projections of edges”, par 0060, “once the true planes are determined and false planes removed, a more accurate representation of walls in the scene is obtained. In that case, the system may correct, or adjust, the point cloud with respect to the determined planes to better align both the representations, e.g., the planes and the dense point cloud”, par 0065-0067, “the system ensures that the axes are perpendicular to one another by applying a correction to the second axis as necessary and computing the third axis as the cross-product of the first axis and the second axis, thereby ensuring that all three axes are perpendicular to one another” …..match detected edges/lines (first line and second line) across images (first image and second image)(par 0039), Metzler et al.: par 0090, “In order to further resolve ambiguity in the conglomeration of the first and second 3D-model 4a,4b in an additional degree of freedom, there is a second pair of line sections 2b, comprising a third line section 2b defined in the first 2D-visual-image 1a and a corresponding fourth line section 2b in the second 2D visual picture 1b shown, which are located at the top corner of the room. In many instances of the present invention, there can be an automatic matching of pairs of the line features 2a,2b in-between the first and second 2D-visual image, e.g. as often only one matching is geometrically resolvable and/or makes technical sense”, par 0098, “Such can e.g. also comprise an automatic snapping to an image detected visual edge feature. Also the intersection point 6am is manually corrected. The line segment 2c in the 2D-visual-image got the geometric restriction 5a applied, according to which it is substantially perpendicular to the line sections 2a and 2b”, par 0101, “In particular the line segments 2 can therein be matched to corresponding plan features 2p and 2h of the plan 9. For example, the feature 2p can be an Edge 2 as defined in the images at FIG. 8a and FIG. 8b and the feature 2h can be a height reference or horizontal reference 2 as indicated in those images”).
Regarding claim 11, Patkar et al. teach a three-dimensional model generation device comprising: a processor; and a memory storing a program executable by the processor to cause the three-dimensional model generation device to (abstract, par 0007). The remaining limitations of the claim are similar in scope to claim 1 and rejected under the same rationale.
Regarding claim 12, Patkar et al. teach non-transitory computer readable medium storing a program executable by a processor to cause an information processing apparatus to (par 0008, par 0025). The remaining limitations of the claim are similar in scope to claim 1 and rejected under the same rationale.
Allowable Subject Matter
Claims 5-7 objected to as being dependent upon a rejected base, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The cited prior art fails to teach the combination of elements recited in claim 5, including " wherein the generating the first three-dimensional points includes: specifying a plane by performing principal component analysis on the third three- dimensional points; and correcting the positions of the third three-dimensional points to bring the third three-dimensional points closer to the plane".
The following is a statement of reasons for the indication of allowable subject matter: The cited prior art fails to teach the combination of elements recited in claim 7, including " wherein the generating the first three-dimensional points includes generating an approximate line by performing a least squares method on the third three-dimensional points; and correcting the positions of the third three-dimensional points to bring the third three-dimensional points closer to the approximate line".
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jin Ge whose telephone number is (571)272-5556. The examiner can normally be reached 8:00 to 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571)272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JIN . GE
Examiner
Art Unit 2619
/JIN GE/Primary Examiner, Art Unit 2619