Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Priority
Acknowledgment is made of applicant’s foreign priority claim, for U.S. Application No. 19/058,224, based on a foreign application filed on 02/22/2024.
Status of Claims
Claims 1–15 are pending in the application. Claims 1, 2, 4, 7, 8, 9, 10, 11, 14, 15 are rejected.
Claims 3, 5, 6, 12, 13 are objected to.
Allowable Subject Matter
Claims 3, 5, 6, 12, 13 are objected to as being dependent upon a rejected base claim(s), but would be allowable if rewritten in independent form including all of the limitations of the base claim(s) and any intervening claim(s).
Overview of Grounds of Rejection
Ground of Rejection
Claim(s)
Statute(s)
Reference(s)
Ground 1
1, 2, 4, 7, 8, 9, 10, 11, 14, 15
§ 103
Polchin (US20230363830A1) in view of Srimohanarajah et al. (US20210056699A1)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
(Please see the cited paragraphs, sections, pages, or surrounding text in the references for the paraphrased content.)
Ground of Rejection 1
Claims 1, 2, 4, 7, 8, 9, 10, 11, 14, 15 are rejected under 35 U.S.C. § 103 as being unpatentable over Polchin (US20230363830A1) in view of Srimohanarajah et al. (US20210056699A1).
As per Claim 1, Polchin teaches the following portion of Claim 1, which recites: “A method for carrying out patient registration on a medical visualization system, comprising:”
Polchin teaches an integrated surgical navigation and visualization system providing “patient registration, surgical navigation, and visualization.” Polchin, ¶ [0010].
Polchin teaches the following portion of Claim 1, which recites:
“capturing an image of a body part of a patient by a camera of the medical visualization system,” Polchin, abstract, e.g. a camera capturing an image of a patient include their one or more body parts.
Polchin teaches that the “surface of the live patient is captured” by “taking a snapshot at each pose while keeping the relevant part of the patient’s anatomy in the field of view of the digital surgical microscope camera(s).” Polchin, ¶ [0251].
Polchin alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Srimohanarajah, they collectively teach all of the limitation(s).
Polchin and Srimohanarajah teach the following portion of Claim 1, which recites: “estimating, with a trained machine learning method and/or a method of computer vision, a three-dimensional surface profile of the body part in a coordinate system of the camera using the captured image as a starting point,”
Polchin teaches sending captured images to a “photogrammetry module,” performing “Feature detection in each image using... SIFT,” extracting “a surface point per matched stereo pixel pair,” and “Patient anatomy 3D model generation” as the “surface extraction step for the live patient data.” Polchin, ¶ [0257].
Polchin does not clearly state that the resulting surface is represented specifically in the camera coordinate system. Srimohanarajah et al. teaches a camera-based 3D scanner producing a “3D surface or as a 3D point cloud” wherein “The 3D scan data is obtained in a 3D scan coordinate space.” Srimohanarajah et al., ¶¶ [0068], [0084].
Polchin teaches the following portion of Claim 1, which recites: “determining or estimating a scaling factor of the three-dimensional surface profile,”
Polchin teaches “Scale and reference frame origin and orientation specification” and that “Scale is found via readily detected features of known dimension such as two April tags placed a known distance away from each other.” Polchin, ¶ [0257].
Polchin teaches the following portion of Claim 1, which recites: “fitting preoperatively acquired three-dimensional patient data that are available in a patient coordinate system to the estimated three-dimensional surface profile,”
Polchin teaches that a surface extracted from preoperative 3D scan data “is used to align to the similar surface extracted from the live patient data.” Polchin, ¶ [0247]. It further teaches that the live surface is “matched (also known as registered, or aligned)” to the preoperative surface using, for example, “iterative closest point (ICP).” Polchin, ¶¶ [0281], [0284].
Polchin teaches the following portion of Claim 1, which recites: “determining a transformation rule between the coordinate system of the camera and the patient coordinate system using a resultant fit result as a starting point,”
Polchin teaches that the surface registration produces “the transformation patientRefFrm_T_patientData” and subsequently determines:
“dsmCam_T_patientData = dsmCam_T_navCam * navCam_T_navTarget * navTarget_T_patientData”
where “navTarget_T_patientData is the transformation output... determined during the patient registration step.” Polchin, ¶¶ [0281], [0234].
Polchin teaches the following portion of Claim 1, which recites: “wherein the determining of the transformation rule is performed taking into account the determined and/or estimated scaling factor,”
Polchin teaches that during surface registration “a search over a small range of scale is sometimes useful,” thereby permitting scale to be incorporated into the patient-registration transformation used in determining the camera-to-patient-data transformation. Polchin, ¶ [0285].
Polchin teaches the following portion of Claim 1, which recites: “and providing the determined transformation rule.”
Polchin teaches that “The output is a transformation matrix” and identifies “dsmCam_T_patientData” as the “final single matrix needed to render the patient data.” Polchin, ¶¶ [0286], [0234].
Before the effective filing date of the claimed invention, a POSITA would have been motivated to combine Polchin with Srimohanarajah et al. because both concern camera-based registration of a patient's three-dimensional surface with preoperative patient data. Applying Srimohanarajah et al.'s 3D scan coordinate space to Polchin's camera-derived 3D patient surface would provide a defined camera-associated reference frame for subsequent surface registration and transformation calculations, improving coordinate-space integration and yielding the predictable result of a camera-referenced 3D patient surface suitable for registration with preoperative patient data.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
As per Claim 2, Polchin teaches Claim 2, which recites: “The method according to claim 1, wherein the scaling factor of the estimated three-dimensional surface profile is determined using settings of the camera and/or of optical elements of the medical visualization system at the time of image capture as a starting point.”
Polchin teaches that the patient-surface reconstruction procedure is performed at a “single zoom and working distance setting” and further teaches:
“The zoom value is incorporated by changing the field of view (essentially scaling the image) about the principal point of the camera. The amount of scale is determined by a separate calibration step which maps field of view to zoom motor counts.” - Polchin, ¶ [0257].
Thus, Polchin teaches determining image/surface scale using camera optical settings, including zoom and working distance, closely corresponding to the additional limitation of Claim 2.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
As per Claim 4, Polchin teaches Claim 4, which recites: “The method according to claim 1, wherein the scaling factor of the estimated three-dimensional surface profile is determined using as a starting point at least one marker that is arranged on the patient and that is captured by means of the camera or by the environment camera of the medical visualization system.”
Polchin teaches “One or more calibration targets are mounted rigidly to the patient anatomy” and that the calibration target “may need to appear in at least a small number of the snapshots.” Polchin, ¶ [0256]. Polchin further teaches that “Scale is found via readily detected features of known dimension such as two April tags placed a known distance away from each other.” Polchin, ¶ [0257].
Thus, Polchin teaches determining the claimed scaling factor using a patient-mounted marker captured by the camera.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
As per Claim 7, Polchin teaches Claim 7, which recites: “The method according to claim 1, wherein a camera pose of the camera is determined in a reference coordinate system, wherein the transformation rule is determined between the reference coordinate system and the patient coordinate system.”
Polchin teaches that “the pose of the DSM camera relative to the navigation targets may be determined” and that, using the resulting information, “the pose of the digital surgical microscope camera(s) relative to the patient data may be calculated.” Polchin, ¶ [0077].
Polchin further teaches a patient-registration transformation “patientTarget_T_patientData”, representing the pose of the patient anatomy relative to the patient reference target. Polchin, ¶¶ [0192]-[0195].
Thus, Polchin teaches determining the camera pose in a reference coordinate system and determining the transformation between that reference coordinate system and the patient coordinate system.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
As per Claim 8, Polchin teaches Claim 8, which recites: “The method according to claim 1, wherein a change in a camera pose of the camera is followed by a renewed capture of an image of the body part of the patient and renewed fitting and the renewed determination of the transformation rule.”
Polchin teaches “moving the robot about the patient in many poses taking a snapshot at each pose”, thereby teaching a change in camera pose followed by renewed capture of an image of the patient. Polchin, ¶ [0251]. Polchin further teaches matching the captured live-patient surface to the preoperative surface using “iterative closest point (ICP)”, with the process producing a “transformation matrix” between the datasets. Polchin, ¶¶ [0281], [0284], [0286].
Although Polchin does not directly state that the fitting and transformation are recomputed after each camera-pose change, before the effective filing date of the claimed invention, a POSITA would have found it obvious to repeat the disclosed fitting and transformation determination after acquiring a new image from a changed camera pose in order to update registration using the newly acquired patient-surface information. This would merely repeat Polchin’s known registration steps for newly acquired data and would predictably maintain or improve registration accuracy.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
As per Claim 9, Polchin teaches Claim 9, which recites: “The method according to claim 1, wherein the trained machine learning method and/or the method of computer vision comprises at least one first method and at least one second method, wherein the at least one first method determines distinguished points on the body part in the captured image, and wherein the at least one second method estimates the three-dimensional surface profile using the determined distinguished points as a starting point.”
Polchin teaches “Feature detection in each image using a feature description mechanism such as SIFT” and “Correlation of features in images taken from ‘nearby’ poses,” followed by “Patient anatomy 3D model generation for the relatively sparse set of data represented by the feature extraction step. This is the surface extraction step for the live patient data.” Polchin, ¶ [0257].
Thus, Polchin teaches a first computer-vision method determining distinguished image features/points and a second method estimating the 3D patient surface from those detected features.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
As per Claim 10, Polchin teaches Claim 10, which recites: “The method according to claim 1, wherein the camera is an environment camera of the medical visualization system.”
Polchin teaches that the integrated surgical visualization system includes a “navigation camera (‘navigation localizer’ or ‘localizer’)” and that the “digital surgical microscope camera(s) and navigation camera(s) are used to gather live patient data to enable patient registration and navigation.” Polchin, ¶¶ [0059], [0250].
Thus, Polchin reasonably teaches the claimed camera as an environment/navigation camera of the medical visualization system.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
As per Claim 11, Polchin teaches Claim 11, which recites: “The method according to claim 1, wherein at least one further image of the body part of the patient is captured in at least one other camera pose of the camera or by means of a further camera of the medical visualization system arranged in the at least one other camera pose, wherein fitting is implemented with the captured at least one further image being taken into account.”
Polchin teaches “moving the robot about the patient in many poses taking a snapshot at each pose” and processing the resulting images by “Correlation of features in images taken from ‘nearby’ poses” to generate the “Patient anatomy 3D model.” Polchin, ¶¶ [0251], [0257]. The resulting live-patient surface is then “matched (also known as registered, or aligned)” to the preoperative patient surface. Polchin, ¶ [0281].
Thus, Polchin teaches capturing further images from other camera poses and taking those images into account in the fitting.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
As per Claim 14, Polchin teaches Claim 14, which recites: “The method according to claim 11, wherein the camera and/or the at least one further camera for capturing the images is arranged in at least two camera poses by means of a robotic stand of the medical visualization system.”
Polchin teaches a “digital surgical microscope (DSM) head 110 mounted on a robotic arm 120” and further teaches capturing the patient surface by “moving the robot about the patient in many poses taking a snapshot at each pose.” Polchin, ¶¶ [0053], [0251].
Thus, Polchin teaches arranging the camera in at least two camera poses by means of a robotic stand/arm of the medical visualization system.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Claim 15 does not include any additional limitations that would significantly distinguish it from claim 1. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Conclusion
The prior art made of record and relied upon in this action is as follows:
Patent Literature:
Polchin (US20230363830A1) — “Auto-navigating digital surgical microscope”
Srimohanarajah et al. (US20210056699A1) — “Patient registration systems, devices, and methods for a medical procedure”
Non-Patent Literature (NPL):
(none) Note: A PDF copy of each NPL reference is attached with this Office Action. URLs are included for applicant convenience. If a link becomes unavailable in the future, the citation information may be used to locate the reference or access archived versions via the Wayback Machine.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure and is listed as follows:
Patent Literature:
Siemionow et al. (US20220401148A1) — “Method and apparatus for registering a neurosurgical patient and determining brain shift during surgery using machine learning and stereooptical three-dimensional depth camera with a surface-mapping system”
Prakash et al. (US20140368816A1) — “Optical device”
Non-Patent Literature (NPL):
(none) Any inquiry concerning this communication or earlier communications from the examiner should be directed to ADEEL BASHIR whose telephone number is (571) 270-0440. The examiner can normally be reached Monday-Thursday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on (571) 276-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ADEEL BASHIR/
Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights