Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Applicant’s Arguments
Applicant’s arguments filed 03/16/2026 have been fully considered, but for claims 1-5, 11-15, and 21-25, they are not deemed to be persuasive based on the new reference of SZASZ et al (US 2020/0202624 Al) or ILIC et al (US 2017 /0085733 Al); specifically, the Ilic teaches the claimed “receiving user selection of one or more additional sampled surface points on the object to increase the collection be including the one or more additional sampled surface points an updated collection; determining a fit of the model point cloud to the point cloud of the object based on the updated collection of sample points of the object” (Ilic, [0044] - When information about depth of the points relative to the camera is included, this point cloud may serve as a depth map. Estimates of positional uncertainty may be associated with the coordinates. As more image frames are acquired, processing to update the point cloud with new information in the additional image frames may improve certainty for the coordinates; [0057] - a user may observe differences between the displayed model of the object and the actual object. Such differences may indicate to the user that further data is required for portions of the object. The user may allow the portable electronic device to continue to acquire image frames, providing more information from which processing may improve the quality of the object model. When the image quality appears suitable, the user may input a command that stops the portable electronic device from collecting further images. However, it should be appreciated that criteria to stop image collection may also be applied automatically) (see also Szasz, [0041] - FIG. 2.A illustrates the generation of a point cloud 100 and mesh representation 400 based on a 2D image, according to various embodiments described herein. As illustrated in FIG. 2A, analysis of each image (e.g., images 130a and 130b) may result in the identification of points 140 through 144, which may collectively be referred to as point cloud 200, which is a plurality of points 200 identified from respective images of the object 135. From these identified plurality of points 200, methods, systems, and computer program products according to the present inventive concepts update a mesh representation 400 of the object 135 in block 1100; [0049] - The process described herein may be repeated for multiple scans of the object 135, iteratively and continuously updating and refining the mesh representation 400 of the object 135; [0051] - As described herein, the process of adjusting the mesh representation 400 may include incremental adjustment based on a series of received point clouds 200 from scanned images). Accordingly, the claimed invention as represented in the claims 1, 11, and 21 (and its depend claims 2-5, 12-15, and 22-25) does not represent a patentable distinction over the art of record.
For claims 6-7, 16-17, and 26-27, applicant’s arguments are persuasive; specifically, the cited references do not teach “wherein obtaining the user selection of the origin point comprises providing an anatomy model augmented reality (AR) element overlaying a portion of a view to the patient anatomy, the view showing a registration probe, and the anatomy model AR element being provided at a fixed position relative to a probe tip of the probe, wherein user movement of the probe repositions the anatomy model AR element and wherein the user selection comprises: the user positioning and orienting the anatomy model AR element in the view to overlay the patient anatomy by touching the patient anatomy with the probe tip, and providing input to select the origin point as a position of the probe tip touching the patient anatomy.”
For claims 8-10, 18-20, 28-30, Applicant argues that “which depend from claims 6, 16, and 26 respectively, are allowable for their dependency on allowable base claims”; however, the claims 8-10, 18-20, 28-30 do not depend on claim 6, 16, and 26, therefore, applicant’s arguments on these claims are deemed to moot.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 11-12, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over LURIE et al (2017/0046833) in view of PANDEY et al (Visually bootstrapped generalized ICP); and further in view of SZASZ et al (US 2020/0202624 Al) or ILIC et al (US 2017 /0085733 Al).
As per claim 1, Lurie teaches the claimed "computer implemented method" comprising: "registering a model point cloud to a point cloud of an object," the registering comprising: "obtaining a user selection of an origin point for the model point cloud, the origin point being a sampled surface point on the object and being a first point included in an established collection of sample points of the object, the collection forming the point cloud of the object" (Lurie, [0060] - An initial sparse point cloud 151- a representative set of 3D points (Xi) on the surface of the organ is generated; [0127] - The identification of correspondences may be performed manually or via one of various automatic processes) (Noted: the original point of Lurie's model can be selected manually based on the sampled point of the organ); "obtaining one or more other sampled surface points on the object and including the obtained one or more other sampled surface points in the collection" (Lurie, [0077]-[ 0079] - The outliers may be filtered using RANdom SAmpling and Consensus (RANSAC), which simultaneously estimates both the relative transformation (camera poses) between a pair of keyframes as well as their shared (inlier) correspondences. RANSAC is an iterative process that employs a hypothesize-and-verify scheme. In each iteration, a random subset of correspondences is sampled and that subset is used to compute a hypothesis. In case of rigid transformation, a subset containing as few as 5 correspondences may be used to compute an essential matrix as a hypothesis which encodes the relative transformation. A consensus set is subsequently computed by finding all correspondences that agree with the essential matrix (validation). If the size of the consensus set is sufficient (e.g., above 20 correspondences), this keyframe pair may be deemed geometrically consistent); "determining an initial pose of the model point cloud based on the collection of sample points of the object" (Lurie, [0077]-[ 0079] - The outliers may be filtered using RANdom SAmpling and Consensus (RANSAC), which simultaneously estimates both the relative transformation (camera poses) between a pair of keyframes as well as their shared (inlier) correspondences. RANSAC is an iterative process that employs a hypothesize-and-verify scheme. In each iteration, a random subset of correspondences is sampled and that subset is used to compute a hypothesis. In case of rigid transformation, a subset containing as few as 5 correspondences may be used to compute an essential matrix as a hypothesis which encodes the relative transformation. A consensus set is subsequently computed by finding all correspondences that agree with the essential matrix (validation). If the size of the consensus set is sufficient (e.g., above 20 correspondences), this keyframe pair may be deemed geometrically consistent). Lurie does not provide, but the Ilic teaches the claimed “receiving user selection of one or more additional sampled surface points on the object to increase the collection be including the one or more additional sampled surface points an updated collection; determining a fit of the model point cloud to the point cloud of the object based on the updated collection of sample points of the object” (Ilic, [0044] - When information about depth of the points relative to the camera is included, this point cloud may serve as a depth map. Estimates of positional uncertainty may be associated with the coordinates. As more image frames are acquired, processing to update the point cloud with new information in the additional image frames may improve certainty for the coordinates; [0057] - a user may observe differences between the displayed model of the object and the actual object. Such differences may indicate to the user that further data is required for portions of the object. The user may allow the portable electronic device to continue to acquire image frames, providing more information from which processing may improve the quality of the object model. When the image quality appears suitable, the user may input a command that stops the portable electronic device from collecting further images. However, it should be appreciated that criteria to stop image collection may also be applied automatically) (see also Szasz, [0041] - FIG. 2.A illustrates the generation of a point cloud 100 and mesh representation 400 based on a 2D image, according to various embodiments described herein. As illustrated in FIG. 2A, analysis of each image (e.g., images 130a and 130b) may result in the identification of points 140 through 144, which may collectively be referred to as point cloud 200, which is a plurality of points 200 identified from respective images of the object 135. From these identified plurality of points 200, methods, systems, and computer program products according to the present inventive concepts update a mesh representation 400 of the object 135 in block 1100; [0049] - The process described herein may be repeated for multiple scans of the object 135, iteratively and continuously updating and refining the mesh representation 400 of the object 135; [0051] - As described herein, the process of adjusting the mesh representation 400 may include incremental adjustment based on a series of received point clouds 200 from scanned images). Thus, it would have been obvious, in view of Szasz and Ilic, to configure Lurie's method as claimed by further generating a fit model from an initial model. The motivation is to refining the initial model by repeating the scanning to provide more information for enhancing refined details to build the refined model.
Claim 2 adds into claim 1 "wherein the model point cloud comprises an anatomy model point cloud and wherein the object comprises a patient anatomy" (Lurie, [0019] - The image processing program configures the processor to preprocess a plurality of images, the plurality of images comprising images captured by an endoscope, wherein the plurality of images includes images of at least a portion of an organ. The processor is further configured to generate a three-dimensional (3D) point cloud representing points on a surface of the organ based on the set of color-adjusted images, define a mesh representing the surface of the organ based on the 3D point cloud, and generate a texture of the surface of the organ based on the set of color- adjusted images).
Claims 11-12, 21-22 claim a computer system and a computer program product based on the method of claims 1-2; therefore, they are rejected under a similar rationale.
Claims 3-5, 8-10, 13-15, 18-20, 23-25 and 28-30 are rejected under 35 U.S.C. 103 as being unpatentable over LURIE et al (2017/0046833) in view of SZASZ et al (US 2020/0202624 Al) or ILIC et al (US 2017 /0085733 Al), and further in view of PANDEY et al (Visually bootstrapped generalized ICP).
Claim 3 adds into claim 2 "wherein the performing processing comprises, based on the determined registration accuracy being less than a preconfigured threshold level of accuracy: iterating, one or more times, the receiving user selection of one or more additional sampled surface point” (Ilic, [0044] - When information about depth of the points relative to the camera is included, this point cloud may serve as a depth map. Estimates of positional uncertainty may be associated with the coordinates. As more image frames are acquired, processing to update the point cloud with new information in the additional image frames may improve certainty for the coordinates; [0057] - a user may observe differences between the displayed model of the object and the actual object. Such differences may indicate to the user that further data is required for portions of the object. The user may allow the portable electronic device to continue to acquire image frames, providing more information from which processing may improve the quality of the object model. When the image quality appears suitable, the user may input a command that stops the portable electronic device from collecting further images. However, it should be appreciated that criteria to stop image collection may also be applied automatically) (see also Szasz, [0041] - FIG. 2.A illustrates the generation of a point cloud 100 and mesh representation 400 based on a 2D image, according to various embodiments described herein. As illustrated in FIG. 2A, analysis of each image (e.g., images 130a and 130b) may result in the identification of points 140 through 144, which may collectively be referred to as point cloud 200, which is a plurality of points 200 identified from respective images of the object 135. From these identified plurality of points 200, methods, systems, and computer program products according to the present inventive concepts update a mesh representation 400 of the object 135 in block 1100; [0049] - The process described herein may be repeated for multiple scans of the object 135, iteratively and continuously updating and refining the mesh representation 400 of the object 135; [0051] - As described herein, the process of adjusting the mesh representation 400 may include incremental adjustment based on a series of received point clouds 200 from scanned images). Noted that “the determining a fit, and the determining the registration accuracy" which Lurie does not teach, but Pandey teaches these features (Pandey, Algorithm 2 Standard ICP Algorithm - step 4: while not converted (i.e., the object and the model are not aligned) do... to step 14: converted) (It is conventional knowledge that the more collected information (i.e., samples of the object and the model point cloud) is the better estimation for aligning the object and the model point cloud (i.e., a maximum likelihood estimate (MLE) of the transformation "T" that best aligns the two scans in Pandey's Algorithm 2) from the ICP algorithm (Pandey, B. ICP Framework - Once the point correspondences are established, the ICP cost function is formulated as a maximum likelihood estimate (MLE) of the transformation "T" that best aligns the two scans) - see Pandey, IV. Conclusion - In the experiments performed with real world data, we have shown that the bootstrapped generalized ICP algorithm is more robust and gives accurate results even when the overlap between the two scans reduces to less than 50%). Thus, it would have been obvious, in view of Pandey, Szasz, and Ilic, to configure Lurie's method as claimed by further obtaining more sample data of the object for the model point cloud. The motivation is to provide a more accurate set of point correspondences to the generalized ICP algorithm by taking advantage of high dimensional image feature descriptors.
Claim 4 adds into claim 3 "wherein the iterating halts based on the determined registration accuracy being at least the preconfigured threshold level of accuracy" (Pandey, Algorithm 2 Standard ICP Algorithm - step 4: while not converged (i.e., the object and the model are not aligned) do... to step 14: converted) (the iterating stops when the object and the model point cloud are aligned (within a defined accuracy of the MLE estimater)). Thus, it would have been obvious, in view of Pandey, Szasz, and Ilic, to configure Lurie's method as claimed by stopping the iterating when the aligning of the object and the model point cloud is satisfied. The motivation is to reduce the processing cost by increasing the efficiency of the ICP algorithm.
Claim 5 adds into claim 4 "wherein based on halting the iterating, the determined fit of the anatomy model point cloud to the point cloud of the patient anatomy provides a registration of the anatomy model point cloud to the point cloud of the patient anatomy" (Pandey, Table I - The resulting error statistics are tabulated in Table / where we see that the bootstrapped GICP is able to provide sub 25 cm translational error at 15 scans apart, while GICP alone begins to fail after only 5 scans of displacement), and wherein the method further comprises "determining and digitally presenting to a surgeon one or more indications of surgical guidance" (Lurie, [0178] - Optical coherence tomography (OCT) is a promising complement to WLC due to its ability to image in depth, which allows it to distinguish cancerous from healthy tissue (i.e., based on the number of subsurface layers). OCT data that are co-registered to a 3D reconstruction of the bladder wall may enable complete staging of a tumor or identification of surgical margins, a visualization that could help a surgeon prepare for surgery or track tumor recurrence). Thus, it would have been obvious, in view of Pandey, Szasz, and Ilic, to configure Lurie's method as claimed by further generating a fit model from an initial model. The motivation is to provide a more accurate set of point correspondences to the generalized ICP algorithm by taking advantage of high dimensional image feature descriptors.
Claim 8 adds into claim 2 "wherein determining the fit of the anatomy model point cloud to the point cloud of the patient anatomy based on the updated collection of sample points of the patient anatomy comprises: performing a rough fitting of the anatomy model point cloud to the point cloud of the patient anatomy using the updated collection of sample points of the patient anatomy" (Pandey, II. METHODOLOGY - Once we have augmented the 3D point cloud with the high dimensional feature descriptors, we then use them to align the scans in a two step process. In the first step, we establish putative point correspondence in the high dimensional feature space and then use these correspondences within a RANSAC framework to obtain a coarse initial alignment of the two scans. In the second step, we refine this coarse alignment using a generalized ICP framework) (Pandey, A. RANSAC Framework - In the first part of our algorithm, we estimate a rigid body transformation that approximately aligns the two scans using putative visual correspondences); and "based on performing the rough fitting, performing a fine fitting of the anatomy model point cloud to the point cloud of the patient anatomy using the updated collection of sample points of the patient anatomy" (Pandey, B. ICP Framework - In GICP the point correspondences are established by considering the Euclidean distance between the two point clouds XA and XB. Once the point correspondences are established, the ICP cost function is formulated as a maximum likelihood estimate (MLE) of the transformation "T" that best aligns the two scans). Thus, it would have been obvious, in view of Pandey, Szasz, and Ilic, to configure Lurie's method as claimed by further generating a fit model from an initial model. The motivation is to provide a more accurate set of point correspondences to the generalized ICP algorithm by taking advantage of high dimensional image feature descriptors.
Claim 9 adds into claim 8 "wherein performing the rough fitting comprises applying a random sample consensus (RANSAC) algorithm and/or performing the fine fitting comprises applying an iterative closest point (ICP) algorithm" (Pandey, II. METHODOLOGY - Once we have augmented the 3D point cloud with the high dimensional feature descriptors, we then use them to align the scans in a two step process. In the first step, we establish putative point correspondence in the high dimensional feature space and then use these correspondences within a RANSAC framework to obtain a coarse initial alignment of the two scans. In the second step, we refine this coarse alignment using a generalized ICP framework). Thus, it would have been obvious, in view of Pandey, Szasz, and Ilic, to configure Lurie's method as claimed by further generating a fit model from an initial model. The motivation is to provide a more accurate set of point correspondences to the generalized ICP algorithm by taking advantage of high dimensional image feature descriptors.
Claim 10 adds into claim 2 "wherein determining the initial pose of the anatomy model point cloud comprises performing a rough fitting of the anatomy model point cloud to the point cloud of the patient anatomy by applying a random sample consensus (RANSAC) algorithm" (Pandey, II. METHODOLOGY - Once we have augmented the 3D point cloud with the high dimensional feature descriptors, we then use them to align the scans in a two step process. In the first step, we establish putative point correspondence in the high dimensional feature space and then use these correspondences within a RANSAC framework to obtain a coarse initial alignment of the two scans. In the second step, we refine this coarse alignment using a generalized ICP framework) (Pandey, A. RANSAC Framework - In the first part of our algorithm, we estimate a rigid body transformation that approximately aligns the two scans using putative visual correspondences) and, "based on performing the rough fitting, performing a fine fitting of the anatomy model point cloud to the point cloud of the patient anatomy by applying an iterative closest point (ICP) algorithm" (Pandey, B. ICP Framework - In GICP the point correspondences are established by considering the Euclidean distance between the two point clouds XA and XB. Once the point correspondences are established, the ICP cost function is formulated as a maximum likelihood estimate (MLE) of the transformation "T" that best aligns the two scans). Thus, it would have been obvious, in view of Pandey, Szasz, and Ilic, to configure Lurie's method as claimed by further generating a fit model from an initial model. The motivation is to provide a more accurate set of point correspondences to the generalized ICP algorithm by taking advantage of high dimensional image feature descriptors.
Claims 13-15, 18-20, 23-25 and 28-30 claim a computer system and a computer program product based on the method of claims 3-5, 8-10; therefore, they are rejected under a similar rationale.
Claims 6-7, 16-17, and 26-27 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
In claim 6, and its dependent claim 7, (similar for claims 16-17 and 26-27) “wherein obtaining the user selection of the origin point comprises providing an anatomy model augmented reality (AR) element overlaying a portion of a view to the patient anatomy, the view showing a registration probe, and the anatomy model AR element being provided at a fixed position relative to a probe tip of the probe, wherein user movement of the probe repositions the anatomy model AR element and wherein the user selection comprises: the user positioning and orienting the anatomy model AR element in the view to overlay the patient anatomy by touching the patient anatomy with the probe tip, and providing input to select the origin point as a position of the probe tip touching the patient anatomy.”
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHU K NGUYEN whose telephone number is (571)272-7645. The examiner can normally be reached M-F 8-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel F. Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHU K NGUYEN/ Primary Examiner, Art Unit 2616
Prosecution Insights