DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-12 are pending.
Prior art:
D1: US20230260158A1 Stopp et al
D2: JP2001304809A Azuma
D3: WO2016066287A1 Kosmecki et al
D4: US20130010081A1 Tenney et al
D5: US20190346659A1 Stegmann et al
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1, 3, 7-8 and 11-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over D1 in view of D2 and further in view of D5.
Regarding claim 1, D1 teaches a method for determining an optical axis of a main observer camera of a medical microscope arrangement in a reference coordinate system, the method comprising:
(D1, "The computer-implemented medical method according to the present invention for calibrating an optical system of a surgical microscope comprises the following steps", [0012]; "Modern microscopes used within an O.R. are often spatially tracked with respect to a patient, which not only allows a visualisation of the microscope’s optical axis and focal point with respect to the patient’s anatomy", [0002]; a method for calibrating/determining the optical axis of a surgical microscope's optical system)
capturing a capture region with a main observer camera;
(D1, "The surgical microscope comprises an optical system which guides an optical path of an image received by the optical system towards an optical receiver such as a CCD-sensor which transmits data describing the received image.", [0018]; capturing a region via the main observer camera of the surgical microscope)
capturing the capture region at least partly and an environment of the capture region with an environment camera; and
(D1, "A detection system 5 having a monoscopic camera 10 is rigidly attached to the housing of the surgical microscope 2 and to the optical system 1 thereof.", [0095]; "the camera assigned to the detection system may have a larger field of view than the microscope camera and/or wherein the camera of the detection system has a visual axis that is angled with respect to the visual axis of the microscope camera.", [0034]; a separate environment camera (detection system camera) captures an overlapping and broader field of view (the environment))
generating and providing at least one item of descriptive information describing the optical axis in the reference coordinate system,
(D1, "d) calibration data is determined based on the first position data, the second position data and the transformation data, which describes a second calibration of the optical system", [0012]; generating descriptive calibration data concerning the alignment/optical axis in the assigned coordinate system)
wherein the environment camera is configured to track objects,
(D1, "In a (for example second) exemplary step and in a similar manner, the spatial position of the object is simultaneously determined by a detection system that does not form part of the microscope’s optical system, but rather represents an entity separate from the surgical microscope.", [0019]; the environment camera is utilized to simultaneously track the spatial position of an object)
wherein a first coordinate system of the environment camera or a second coordinate system of an optical marker forms the reference coordinate system,
(D1, "Based on the data received therefrom, the object’s spatial position is then determined within a co-ordinate system assigned to the detection system and/or an optical system thereof.", [0019]; using the coordinate system assigned to the detection system (environment camera) as the reference coordinate system)
wherein a test object is captured at at least one working distance with the main observer camera and the environment camera,
(D1, Fig. 2; "In the situation shown in FIG. 2 , the distal portion 7 of the instrument 3 is located within the field of view of the microscope cameras 9 such that the spatial position thereof can be determined within co-ordinate system 4 based on data derived from the images received by the microscope cameras 9. In a similar manner, the spatial position of the proximal portion 8 can be determined within co-ordinate system 6 based on the image data received by camera 10.", [0096]; capturing a test object (instrument 3) simultaneously with both cameras at a given working distance.)
wherein at the at least one working distance a neutral point of a zoom system of the main observer camera is determined in the first coordinate system of the environment camera by capturing and evaluating image representations of the test object at different magnifications of the zoom system at the at least one working distance, and
(D1, "In case the focus and/or zoom of the microscope cameras 9 is altered, for example upon request of a surgeon operating the microscope 2, a correct calibration of the microscope cameras 9 can be verified by comparing the spatial position of the object as indicated by the 2D-marker at the distal portion which is detected by camera 10, with the spatial position of the instrument 3 as indicated by the ring markers at the distal portion 7 which are detected by cameras 9.", [0097]; D2, "Next, while moving the microscope at a predetermined magnification interval from the highest magnification to the lowest magnification, the object to be photographed is photographed with a television camera at each magnification, and the outline of the object to be photographed is extracted from the photographed image.", [0005], p3; "Finally, the difference between the average position of the edge point sequence in the X-axis direction and the Y-axis direction for each magnification stored in the work memory 9 and the average position of the edge point sequence in the X-axis direction and the Y-axis direction at the lowest magnification is calculated.", p5; D1 teaches comparing positions when zoom is altered but does not expressly disclose the explicit determination of a "neutral point" by evaluating images across a sequence of different magnifications. D2 fills this gap by evaluating an average (neutral) position at different magnifications to find the optical axis shift)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of D2 into the system or method of D1 in order to precisely track and mathematically compensate for inherent optical axis center shifts across the zoom lens's full magnification range. The combination of D1 and D2 also teaches other enhanced capabilities.
The combination of D1 and D2 further teaches:
wherein the optical axis of the main observer camera is determined proceeding from the neutral point determined in the first coordinate system of the environment camera at the at least one working distance; and
(D1, "a deviation of the position detected by the microscope would indicate that the microscope’s calibration is incorrect. In such case measures can be taken to correct the microscope’s calibration such that it correctly detects the object’s spatial position in alignment with the detection system.", [0021]; D2, "Then, the difference is stored in the work memory 9 as the shift amount of the optical axis at each magnification with reference to the optical axis at the lowest magnification (step 17).", p5; the combination of D1 and D2 teaches that the optical axis shift and recalibration are determined proceeding from the comparative reference point found across magnifications)
The combination of D1 and D2 does not expressly disclose but D5 teaches:
wherein the neutral point is a point and/or region in the image representations which does not move across different magnification levels.
(D5, Fig.1 beam path 01, optical zoom lens 03, image sensor 04; "an invariant point of the zoom optical unit is determined in the image recorded by the image sensor in order to ascertain the displacement of the localizable region caused by the change in the magnification factor. The invariant point is that point in the image recorded by the image sensor which is not displaced in the image by the change in the magnification factor", [0032]; "AA localizable region of the object to be imaged is selected in an image recorded by the image sensor in a step of the second method according to the invention. The localizable region represents a region of interest (ROI) or a point of interest (POI)", [0026] ; invariant point = neutral point, defined as point/ROI in image representations not displaced (does not move across magnification levels); D5 does not teach away D1/D2 because D5 is in the same field, i.e., microscope zoom optical unit deviation correction, solves same problem of image displacement during zooming as D1 medical microscope main observer camera neutral point determination and D2 shift amount across magnifications, D5 does not discourage, or render incompatible D1's reference-coordinate-system method or D2's averaging/difference calculation, but complements them by providing explicit method to isolate magnification-induced displacement using the invariant point)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate D5 into D1 and D2 in order to explicitly define and determine the neutral/invariant point that does not move across different magnification levels to improve accuracy of optical axis determination and to distinguish magnification-induced displacement from mechanical deviation. The combination of D1 and D2 also teaches other enhanced capabilities.
Regarding claim 3, the combination of D1, D2 and D5 teaches its/their respective base claim(s).
The combination further teaches the method as claimed in claim 1, further comprising:
capturing a test object at at least two working distances with the main observer camera and the environment camera;
(D1, "after the setting of the surgical microscope has been changed, particularly after the viewing direction of the surgical microscope and/or the viewing distance of the surgical microscope has changed;", [0036]; capturing and recalculating based on the object when the viewing distance (working distance) changes)
determining at the at least two working distances the neutral point of the zoom system of the main observer camera in each case in the first coordinate system of the environment camera by capturing and evaluating image representations of the test object at different magnifications of the zoom system at each of the working distances;
(D1, "In case the focus and/or zoom of the microscope cameras 9 is altered", [0097]; D2, "the deviation amount of the optical axis for each magnification is measured and stored in advance", [0013], p5; the combination of D1 and D2 teaches tracking and evaluating the optical shift/neutral point at multiple different magnifications and working distances (focus settings))
determining the optical axis of the main observer camera proceeding from the neutral point determined in the first coordinate system of the environment camera; and
generating and providing the at least one item of the descriptive information describing the determined optical axis in the reference coordinate system.
(D1, D2, see comments on claim 1)
Regarding claim 7, the combination of D1, D2 and D5 teaches its/their respective base claim(s).
The combination further teaches the method as claimed in claim 1,
wherein neutral points determined for at least two working distances are connected by a line of best fit or a polynomial function, and
(D2, "An approximation curve is calculated by performing function approximation on these data groups", p5; connecting the determined data points across levels using a function approximation (polynomial function/line of best fit))
wherein the at least one item of descriptive information describing the optical axis includes parameters of the line of best fit or polynomial function.
(D2, "based on the approximation curve, the deviation amount of the optical axis at a magnification other than a preset magnification is set.", p5; the optical axis deviation info is defined by the parameters of the calculated approximation curve)
Regarding claim 8, the combination of D1, D2 and D5 teaches its/their respective base claim(s).
The combination further teaches the method as claimed in claim 1, wherein positions on the optical axis for non-measured working distances are estimated by at least one of interpolation and extrapolation.
(D2, "An approximation curve is calculated by performing function approximation on these data groups, and based on the approximation curve, the deviation amount of the optical axis at a magnification other than a preset magnification is set.", p5; calculating (estimating) the deviation for non-measured values by interpolating/extrapolating the generated approximation curve)
Regarding claim 11, the combination of D1, D2 and D5 teaches its/their respective base claim(s).
The combination further teaches the method as claimed in claim 1, wherein a position of the neutral point at a working distance is monitored at least in a manner of random sampling during use of the zoom system.
(D1; "it is also conceivable to perform the method as described herein on a regular basis, for example constantly and as long as the microscope camera is used for tracking purposes, or at predefined time intervals, for example every few seconds or minutes", [0042]; "the microscope camera may be re-calibrated only if its detection misalignment with respect to the detection system increases a predefined threshold", [0043]; "in predefined time intervals during the operation of the surgical microscope; after the setting of the surgical microscope has been changed, particularly after the viewing direction of the surgical microscope and/or the viewing distance of the surgical microscope has changed; and/or after the setup of the optical system, particularly the zoom-setting of the optical system and/or the focus-setting of the optical system has changed.", [0036]; monitoring the optical system's calibration state, functionally the position of the neutral point, during active use of the surgical microscope, including specifically after the "zoom-setting of the optical system … has changed" , which corresponds to monitoring during use of the zoom system. The monitoring is performed at predefined time intervals or only when a deviation threshold is exceeded, both of which satisfy "at least in a manner of random sampling," i.e., non-continuous, periodic or conditional spot-checking rather than exhaustive continuous monitoring)
Regarding claim 12, the combination of D1, D2 and D5 teaches a medical microscope arrangement, comprising:
a main observer camera having a zoom system, and being configured to capture a capture region;
an environment camera configured to at least partly capture the capture region and an environment of the capture region,
wherein the environment camera is configured to track objects, and
wherein a first coordinate system of the environment camera or a second coordinate system of an optical marker forms a reference coordinate system,
(D1, see comments on claim 1)
an actuator system configured to move at least the main observer camera, and
(D1, “The surgical microscope 2 is carried by a movable trolley (not indicated in FIG. 2)”, [0094]; D2; "reference numeral 3 denotes a microscope driving motor for driving the microscope 1 ... In the microscope 1, the magnification of the optical system can be changed by rotating the microscope driving motor 3 in either the forward or reverse direction.", [0008], p4; D1 teaches a movable microscope trolley, but does not expressly disclose a specifically named actuator system that moves the camera's optical/zoom components. D2 explicitly teaches a microscope driving motor to accomplish this. Incorporating D2 into D1 would fully automate the zoom adjustments and physical repositioning routines of the tracking system.)
a control device configured to:
(D1, "The surgical microscope 2 is carried by a movable trolley (not indicated in FIG. 2) which may comprise a computer 13 having at least one processor for performing the method steps outlined in FIG. 1 ", [0094]; the hardware control device)
instigate capturing of a test object at at least one working distance with the main observer camera and the environment camera, and
(D1, see comments on claim 1)
control the actuator system and the zoom system of the main observer camera such that at the at least one working distance a neutral point of the zoom system of the main observer camera can be determined in the first coordinate system of the environment camera by capturing and evaluating image representations of the test object at different magnifications of the zoom system,
(D1, D2, see comments on claim 1; D2, see comments above)
wherein the neutral point is a point and/or region in the image representations which does not move across different magnification levels;
(D5, see comments on claim 1)
determine an optical axis of the main observer camera proceeding from the neutral point determined at the at least one working distance in the first coordinate system of the environment camera; and
(D1, D2, see comments on claim 1)
generate and provide at least one item of descriptive information describing the determined optical axis in the reference coordinate system.
(D1, see comments on claim 1)
Claim(s) 2 and 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over D1 in view of D2 and further in view of D5 and D3.
Regarding claim 2, the combination of D1, D2 and D5 teaches its/their respective base claim(s).
The combination does not expressly disclose but D3 teaches the method as claimed in claim 1, further comprising:
determining the optical axis at at least one further working distance proceeding from the neutral point determined at the at least one working distance based on a pixel coordinate of the main observer camera that corresponds to the neutral point at the at least one working distance.
(D3, "Therefore, a calibration of the projective properties of the microscopic camera system must be performed. Due to the possible changes of zoom and focus settings of the microscope during the surgical procedure, the camera properties need to be recorded for all microscope settings.", [0006]; for a medical microscope, calibration (identifying camera properties like the optical axis/origin) must be recorded for all settings, specifically including focus settings which correspond to different working distances. This functionally teaches determining the axis for further distances based on the established calibration/projective properties)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of D3 (recording camera calibration for all focus settings/working distances) into the modified system or method of D1 (calibrating an optical system of a surgical microscope) and D2 in order to ensure accurate, reliable, and high-precision calibration across the entire operating range of a variable-focus microscope. The combination of D1, D2, D5 and D3 also teaches other enhanced capabilities.
Regarding claim 9, the combination of D1, D2 and D5 teaches its/their respective base claim(s).
The combination of D1, D2, D5 and D3 teaches the method as claimed in claim 1, wherein the test object is displayed as a virtual test object on a display device.
(D3, "comprising a monitor for displaying the real microscopic image superimposed with a virtual microscopic image", [claim 11]; "the coordinates in the "video" reference coordinate system can be projected onto the image plane of the microscope... and used to generate virtual microscopic images", [0069]; using virtual images/objects projected on a display device within the microscope system, which can serve as the test object for alignment)
Claim(s) 4 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over D1 in view of D2 and further in view of D5 and D4.
Regarding claim 4, the combination of D1, D2 and D5 teaches its/their respective base claim(s).
The combination does not expressly disclose but D4 teaches the method as claimed in claim 1, further comprising:
controlling an actuator system of the medical microscope arrangement to capture the capture region such that a position of a model-based optical axis of the main observer camera at the at least one working distance is arranged at a distinguished position of the test object,
(D4, "The method comprises positioning the movable arm about the target to multiple image-capture locations such that the target is within the field of view of the camera system", [0009]; controlling a robotic actuator (movable arm) to position the camera relative to a target (test object) at specific locations/working distances. By positioning the arm based on the coordinate system to look at a target, it functionally arranges the camera axis at a distinguished position of the object)
wherein during the evaluating a difference between the distinguished position and the neutral point determined for the at least one working distance is determined and taken into account when generating the at least one item of the descriptive information.
(D4, "the transformation between the movable arm's coordinate system and the camera system's coordinate system is determined... by iteratively selecting a value for the transformation... and comparing (A) a product of the selected value... and coordinates of a calibration feature... and (B) a product... of the corresponding calibration feature at another image-capture location to determine a value for the transformation that minimizes a difference between (A) and (B).", [0014]; evaluating differences between expected positions (distinguished positions/model-based) and captured feature points (neutral points/coordinates) to calculate a transformation that is used for descriptive mapping information)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of D4 (robotic actuator control for positioning and evaluation) into the modified system or method of D1 (calibration of a surgical microscope's optical system) and D2 in order to achieve higher precision, consistency, and efficiency, while automatically registering the surgical microscope to the robotic coordinate system. The combination of D1, D2, D5 and D4 also teaches other enhanced capabilities.
Regarding claim 10, the combination of D1, D2 and D5 teaches its/their respective base claim(s).
The combination of D1, D2, D5 and D4 teaches the method as claimed in claim 1, wherein the test object includes at least one of a checkered pattern and a ChArUco pattern.
(D4, "the features are calibration features, which may be selected from one or more of a group consisting of checkerboard corners", [0005]; "FIG. 7A illustrates a checkerboard calibration target 700 including a 7×9 pattern", [0095]; using a checkered/checkerboard pattern as the calibration test object)
Allowable Subject Matter
Claim(s) 5-6 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening Claim(s).
The following is a statement of reasons for the indication of allowable subject matter:
Claim(s) 5-6 recite(s) limitation(s) related to superimposing image to determine the neutral point; and evaluating an optical flow to determine the neutral point. There are no explicit teachings to the above limitation(s) found in the prior art cited in this office action and from the prior art search.
Response to Arguments
Applicant's arguments filed on 7/20/2026 with respect to one or more of the pending claims have been fully considered but are moot in view of the new ground(s) of rejection.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center. for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 9/19/2026
Prosecution Insights