DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1 and 13 are amended.
Response to Arguments
Applicant’s arguments with respect to claims 1 and 13 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1, 4, 8 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Tripathi et
al. (US 20200008899 Al) in view of Garcia (US 20170287169 A1).
Regarding claim 1, Tripathi discloses in at least figures 8 and 25-26, a method for operating
(procedure 2500 figs. 25-26) a stereoscopic medical microscope (stereoscopic visualization camera 300
may be referred to as digital stereoscopic microscope paragraph [0043]),
wherein deteriorated and/or invalid calibration data are recognized (If misalignment is detected,
a re-calibration procedure can be initiated by the processor 1562 paragraph [0247])
wherein for this purpose mutually corresponding image representations (right and left images
paragraph [0296]) of at least one feature (pixel values paragraph [0296]) arranged in capture regions
(target site 700 fig. 8) of two cameras (right image sensor 746 and left image sensor 748 fig. 8) of a stereo camera system (collective image sensor 744 fig. 8) of the medical microscope (stereoscopic visualization camera 300 fig. 8) are captured by means of the two cameras (right image sensor 746 and left image sensor 748 fig. 8), the captured image representations (right and left images paragraph [0296]) are evaluated (the information processor module 1408 may determine images are not aligned by executing a program 1560 that overlays right and left images paragraph [0296]) by means of feature-based image processing (the differences in pixel values is determined, where greater differences over large areas of pixels are indicative of misaligned images paragraph [0296]), and
wherein at least one measure is carried out depending on an evaluation result (a calibration
routine is carried out after determining misalignment paragraph [0296]).
Tripathi does not disclose, wherein the at least one feature is recognized in the captured image
representations and a misalignment and/or a decalibration of the two cameras with respect to one another are/is recognized on the basis of the at least one feature recognized.
However Garcia discloses in at least figure 4, wherein the at least one feature (object point Ps fig. 4) is recognized (the other camera 12 of the stereoscopic system, which camera is illustrated in a similar manner to the camera 11, forms in an identical manner to the camera 11an image point P′I of the object point Ps paragraph [0059]) in the captured image representations (Image planes I1 and I2 fig. 4) and a misalignment and/or a decalibration (the yaw calibration deviation paragraph [0056]) of the two cameras (cameras 11 and 12 fig. 4) with respect to one another (ΔL formed between the cameras cause the yaw calibration deviation paragraph [0056]) are/is recognized on the basis of the at least one feature recognized (the yaw angle elementary variation dL between the two cameras 11 and 12 of the system is then determined as a function of an elementary depth deviation dZ measured from the image point Ps paragraph [0060]).
Therefore, it would be obvious for one skilled in the art before the effective filling date of the
claimed invention to detect the decalibration with respect to each camera as taught by Garcia in the procedure of Tripathi. It appears to be important to be able to correct the stereoscopic system yaw calibration deviation which is “convergent” or “divergent” (due to the vibrations, the initial calibration and/or the thermal effects) and which leads to positive or negative inclination profiles, respectively (paragraph [0055]).
Regarding claim 4, the combination Tripathi and Garcia discloses all the limitations of claim
1 and Tripathi further discloses, wherein for the purpose of checking a focus position calibration (the
processor 1562 measures and verifies optimal focus paragraph [0334]), a sharpness (sharpness signal is
generated by graphics processing unit 1564 paragraph [0334]) of the at least one feature (the sharpness
signal can be determined by an image analysis program paragraph [0334]) in the image representations
(left and right images paragraph [0334]) respectively captured by the cameras (stereoscopic
visualization camera 300 adjusts focus for left and right images paragraph [0333]) is determined,
wherein a difference between the sharpnesses determined (the sharpness signal for the left and
right images can increase on one side and decrease on the other paragraph [0335]) is determined and
provided as evaluation result (if the signals related to both the right and left images are relatively high
and approximately equal the optical elements are properly focused paragraph [0335]).
Regarding claim 8, the combination Tripathi and Garcia discloses all the limitations of claim
1 and Tripathi further discloses, wherein capturing the image representations (right and left images
paragraph [0296]), recognizing the deteriorated and/or invalid calibration data (misalignment paragraph
[0296]) and carrying out the at least one measure (a calibration routine is carried out after determining
misalignment paragraph [0296]) are/is carried out in an automated manner (the instructions may be
received from the information processor module 1408 paragraph [0296]) are/is carried out during
regular operation (the instructions are received after determining right and left images are misaligned
paragraph [0296]) of the medical microscope (stereoscopic visualization camera 300 may be referred to
as digital stereoscopic microscope paragraph [0043)}).
Regarding claim 13, Tripathi discloses in at least figures 8, 14 and 25-26, a medical microscope
(stereoscopic visualization camera 300 may be referred to as digital stereoscopic microscope paragraph
[0043]), comprising:
a stereo camera system (collective image sensor 744 fig. 8) having two cameras (right image sensor 746 and left image sensor 748 fig. 8), configured for capturing mutually corresponding image
representations (right and left images paragraph [0296]) of capture regions (target site 700 fig. 8) of the two cameras (right image sensor 746 and left image sensor 748 fig. 8),
and an image processing device (information processor module 1408 fig. 14), configured for
processing the captured image representations (right and left images paragraph [0296]),
wherein the image processing device (information processor module 1408 fig. 14) is
furthermore configured to recognize deteriorated and/or invalid calibration data (misalignment
paragraph [0296]) and for this purpose to instigate capturing of mutually corresponding image
representations (right and left images paragraph [0296]) of at least one feature (pixel values paragraph
[0296]) arranged in the capture regions (target site 700 fig. 8) of the two cameras (right image sensor 746 and left image sensor 748 fig. 8),
to evaluate (the information processor module 1408 may determine images are not aligned by
executing a program 1560 that overlays right and left images paragraph [0296]) the captured mutually
corresponding image representations (right and left images paragraph [0296]) by means of feature based image processing (the information processor module 1408 may determine images are not aligned
by executing a program 1560 that overlays right and left images and determines differences in pixel
values, where greater differences over large areas of pixels are indicative of misaligned images
paragraph [0296]),
and to instigate at least one measure is carried out depending on an evaluation result (a
calibration routine is carried out after determining misalignment paragraph [0296]).
Tripathi does not disclose, wherein the at least one feature is recognized in the captured image
representations and a misalignment and/or a decalibration of the two cameras with respect to one another are/is recognized on the basis of the at least one feature recognized.
However Garcia discloses in at least figure 4, wherein the at least one feature (object point Ps fig. 4) is recognized (the other camera 12 of the stereoscopic system, which camera is illustrated in a similar manner to the camera 11, forms in an identical manner to the camera 11an image point P′I of the object point Ps paragraph [0059]) in the captured image representations (Image planes I1 and I2 fig. 4) and a misalignment and/or a decalibration (the yaw calibration deviation paragraph [0056]) of the two cameras (cameras 11 and 12 fig. 4) with respect to one another (ΔL formed between the cameras cause the yaw calibration deviation paragraph [0056]) are/is recognized on the basis of the at least one feature recognized (the yaw angle elementary variation dL between the two cameras 11 and 12 of the system is then determined as a function of an elementary depth deviation dZ measured from the image point Ps paragraph [0060]).
Therefore, it would be obvious for one skilled in the art before the effective filling date of the
claimed invention to detect the decalibration with respect to each camera as taught by Garcia in the procedure of Tripathi. It appears to be important to be able to correct the stereoscopic system yaw calibration deviation which is “convergent” or “divergent” (due to the vibrations, the initial calibration and/or the thermal effects) and which leads to positive or negative inclination profiles, respectively (paragraph [0055]).
Claims 2-3 are rejected under 35 U.S.C. 103 as being unpatentable Tripathi et al. (US
20200008899 Al) in view of Garcia (US 20170287169 A1) as applied to claim 1 above and in further
view of Brailovskiy (US 10489912 B1).
Regarding claim 2, The combination Tripathi and Garcia discloses all the limitations of claim
1.
Tripathi does not disclose, wherein during the evaluating on the basis of the at least one feature
recognized an offset at least in one direction between the mutually corresponding image
representations captured is determined and provided as evaluation result.
However Brailovskiy discloses in at least figure 4, wherein during the evaluating (compute
calibration parameters 410 fig. 4) on the basis of the at least one feature (feature points 404 fig. 4)
recognized an offset (offset due to misalignment col. 14 lines 24-25) at least in one direction between
(direction of the offset col. 6 lines 16-17) the mutually corresponding image representations captured
(images for each camera 402 fig. 4) is determined and provided as evaluation result (subsequent images
captured by the stereo camera pair can be analyzed to determine whether the model or calibration
parameters removed the offset due to the misalignment col. 14 lines 25-28).
Therefore, it would be obvious for one skilled in the art before the effective filling date of the
claimed invention to align the corresponding images with offset feature points as taught by Brailovskiy
in the stereoscopic microscope of Tripathi. The calibration parameters can account for the offset
between the images (col. 14 lines 21-28).
Regarding claim 3, The combination Tripathi and Garcia discloses all the limitations of claim
1.
Tripathi does not disclose, wherein during the evaluating on the basis of the at least one feature
recognized a rotation between the mutually corresponding image representations captured is
determined and provided as evaluation result.
However Brailovskiy discloses in at least figure 4,wherein during the evaluating (compute
calibration parameters 410 fig. 4) on the basis of the at least one feature (feature points 404 fig. 4)
recognized a rotation (rotation along the x or y axis col. 4 lines 23-28) between the mutually
corresponding image representations captured (images for each camera 402 fig. 4) is determined and
provided as evaluation result (if one of the cameras is determined to be angled at 0.5 degrees with
respect to the other camera, images captured by one of the cameras can be rotated by 0.5 degrees to
align the images col. 12 lines 5-8).
Therefore, it would be obvious for one skilled in the art before the effective filling date of the
claimed invention to align the corresponding images with rotation as taught by Brailovskiy in the
stereoscopic microscope of Tripathi. Feature points can be mapped to rotation invariant detectors so
that the they math in different portions of the image (col. 10 lines 28-36).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Tripathi (US 20220343539
Al) in view of Garcia (US 20170287169 A1) as applied to claim 1 above and in further view of Tesar
(US 20150297311 Al).
Regarding claim 5, The combination Tripathi and Garcia discloses all the limitations of claim
1.
Tripathi does not disclose wherein during the capturing of the image representations, the at
least one feature is injected into a respective capture region and/or a respective beam path of the
cameras of the stereo camera system by means of at least one injection device of the medical
microscope.
However Tesar discloses in at least figure 1, wherein during the capturing of the image
representations (left and right images paragraph [0243]), the at least one feature (calibration pattern
paragraph [0242]) is injected into a respective capture region (a light source and imaging optics may be
employed to project an image of a reticle or other calibration pattern having known features and
dimensions onto the surgical site being imaged paragraph [0242]) and/or a respective beam path of the
cameras (one or more cameras paragraph [0242]) of the stereo camera system (stereo camera pair
paragraph [0243]) by means of at least one injection device (light source and imaging optics paragraph
[0242]) of the medical microscope (surgical microscope paragraph [0242]).
Therefore, it would be obvious for one skilled in the art before the effective filling date of the
claimed invention to project a calibration pattern as taught by Tesar in the microscope of Tripathi. The
calibration pattern has known features to provide depth information paragraph [0242]).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Tripathi (US 20220343539
Al) in view of Garcia (US 20170287169 A1) as applied to claim 1 above and in further view of
Dumont et al. (US 20150317780 Al).
Regarding claim 6, The combination Tripathi and Garcia discloses all the limitations of claim
1.
Tripathi does not disclose, wherein the at least one feature, for the purpose of capturing, is
arranged in the capture regions and/or in an intermediate image plane of the medical microscope in an
automated manner by means of a feature actuator system and/or wherein the stereo camera system,
for the purpose of capturing, is moved in an automated manner by means of an actuator system, such
that the at least one feature is arranged in the capture regions.
However Dumont discloses in at least fig. 1, wherein the at least one feature (laser projection by
the laser projector 106 paragraph [0026]), for the purpose of capturing (capture of images paragraph
[0026]), is arranged (camera 102 can be arranged to view the laser plane projected by laser projector
106 and camera 108 is proximate to the laser projection paragraph [0024]) in the capture regions (each
camera captures high definition video paragraph [0024]) and/or in an intermediate image plane of the
medical microscope (measuring device 100 fig. 1, medical microscope taught above by Tripathi) in an
automated manner (the processing system 112 may control the generation of the laser projection by the
laser projector 106 paragraph [0026]) by means of a feature actuator system (laser projector 106 fig. 1)
and/or wherein the stereo camera system, for the purpose of capturing, is moved in an automated
manner by means of an actuator system, such that the at least one feature is arranged in the capture
regions.
Therefore, it would be obvious for one skilled in the art before the effective filling date of the
claimed invention to project a feature in an automated manner as taught by Dumont in the microscope
of Tripathi. The processor computes the laser triangulation calibration based on the results of
photogrammetric processing paragraph [0026]).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Tripathi (US 20220343539
Al) in view of Garcia (US 20170287169 A1) as applied to claim 1 above and in further view of
Mihal et al. (US 10453220 Bl).
Regarding claim 7, The combination Tripathi and Garcia discloses all the limitations of claim
1 and Tripathi further discloses, wherein capturing the image representations (right and left images
paragraph [0296]), recognizing the deteriorated and/or invalid calibration data (misalignment paragraph
[0296]) and carrying out the at least one measure (a calibration routine is carried out after determining
misalignment paragraph [0296]) are/is carried out in an automated manner (the instructions may be
received from the information processor module 1408 paragraph [0296]).
Tripathi does not explicitly disclose calibration during starting up and/or shutting down and/or
during an idle state of the medical microscope.
However, Mihal discloses in at least figure 10 calibration (camera misalignment values are
calculated col. 17 lines 19-20) and during starting up and/or shutting down and/or during an idle state
(the device is idle col. 17 line 20) of the medical microscope (the camera is used for stereo depth
perception col. 1 lines 7-10 and the medical microscope taught above by Tripathi).
Therefore, it would be obvious for one skilled in the art before the effective filling date of the
claimed invention to calibrate during an idle mode as taught by Mihal in the microscope of Tripathi. The
idle calibration maintains up to date values for the current camera misalignment (col. 17 lines 20-21).
Claims 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Tripathi (US
20220343539 Al) in view of Garcia (US 20170287169 A1) as applied to claim 1 above and in further
view of Hilsebecher et al. (US 20160335755 Al).
Regarding claim 9, The combination Tripathi and Garcia discloses all the limitations of claim
1.
Tripathi does not disclose, wherein the evaluation result is compared with at least one
predefined limit value, wherein the at least one measure is selected depending on a comparison result.
However, Hilsebecher discloses in at least figure 3, wherein the evaluation result (the relative
yaw angle paragraph [0056]) is compared with (the relative yaw angle is greater than a tolerance value
paragraph [0056]) at least one predefined limit value (a tolerance value paragraph [0056]), wherein the
at least one measure (recalibration paragraph [0056]) is selected depending on a comparison result (if
the change of the relative yaw angle is greater than a tolerance value. The recalibration takes place
mechanically, electronically, and algorithmically, preferably algorithmically paragraph [0056)).
Therefore, it would be obvious for one skilled in the art before the effective filling date of the
claimed invention to use a predefined reference angle as taught by Hilsebecher in the microscope of
Tripathi. The predefined value provides an indication for when the cameras are in misalignment.
Regarding claim 10, the combination of Tripathi, Garcia and Hilsebecher discloses all the
limitations of claim 9.
Tripathi does not disclose, wherein a first limit value has been predefined or is predefined,
wherein as measure calibration data are determined and/or adapted if the evaluation result is greater
than or equal to the predefined first limit value.
However Hilsebecher further discloses, wherein a first limit value has been predefined or is
predefined (prespecified angle paragraph [0020]), wherein as measure calibration data (the cameras are
calibrated using the change in relative yaw angle paragraph [0056]) are determined and/or adapted (the
yaw angles are corrected to recalibrate the system paragraph [0057] if the evaluation result is greater
than (the relative yaw angle is greater than a tolerance value paragraph [0056]) or equal to the
predefined first limit value (prespecified angle paragraph [0020)]).
Therefore, it would be obvious for one skilled in the art before the effective filling date of the
claimed invention to use a predefined reference angle as taught by Hilsebecher in the microscope of
Tripathi. The predefined value provides an indication for when the cameras are in misalignment.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Tripathi (US 20220343539
Al) in view of Garcia (US 20170287169 A1) and Hilsebecher et al. (US 20160335755 Al) as applied
to claim 10 above and in further view of Amthor et al. (US 20200379236 Al) and Nobori et al. (US
20180316906 Al).
Regarding claim 11, the combination of Tripathi, Garcia and Hilsebecher discloses all the
limitations of claim 10.
Tripathi does not disclose, wherein the evaluation result is stored in a maintenance database
operated for anticipatory maintenance if the evaluation result lies below the predefined first limit value.
However, Amthor discloses in at least figure 6, wherein the evaluation result is stored in a
maintenance database operated for anticipatory maintenance (the calibration results of the microscope
system are stored in a database to optimize preventative servicing measure paragraph [0048]).
Therefore, it would be obvious for one skilled in the art before the effective filling date of the
claimed invention to store the calibration results in a database as taught by Amthor for the microscope
of Tripathi. The calibration data can be evaluated at a later time when stored in a database (paragraph
[0048]).
Amthor does not disclose, if the evaluation result lies below the predefined first limit value.
Additionally, Nobori discloses in at least figure 10, if the evaluation result (evaluation value J
paragraph [0171]) lies below (the evaluation value J is less than a first threshold and a condition
paragraph [0171]) the predefined first limit value (first threshold paragraph [0171]).
Therefore, it would be obvious for one skilled in the art before the effective filling date of the
claimed invention to have evaluation value less than the predefined limit as taught by Nobori recorded
in the database of Amthor, in the microscope of Tripathi. The calibration process continues unit the
value is within the threshold (paragraph [0171]).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Tripathi (US 20220343539
Al) in view of Garcia (US 20170287169 A1) and Hilsebecher et al. (US 20160335755 Al) as applied
to claim 9 above in further view of Bostrom (US 20150097948 Al).
Regarding claim 12, the combination of Tripathi, Garcia and Hilsebecher discloses all the
limitations of claim 9.
Tripathi does not disclose, wherein a second limit value has been predefined or is predefined,
wherein as measure a service message is generated and displayed and/or sent if the evaluation
result is greater than or equal to the predefined second limit value.
However, Bostrom discloses in at least figure 4, wherein a second limit value has been
predefined or is predefined (calibration is performed when a difference between a measured position
and expected position is determined a first limit value correction is added or subtracted to the
measured value through algorithm adjustment paragraph [0021] if the deviation of the measurement is
over a second limit threshold value the deviation is to large and manual calibration is required
paragraph [0021]),
wherein as measure a service message is generated and displayed (the maintenance alert may
also be issued if a deviation between a measured position of said calibration markers and an expected
position of said calibration markers paragraph [0021]) and/or sent is greater than or equal to
(calibration markers in the reference coordinate system exceeds a threshold value paragraph [0021]) the
predefined second limit value (calibration is performed when a difference between a measured position
and expected position is determined a first limit value correction is added or subtracted to the
measured value through algorithm adjustment paragraph [0021] if the deviation of the measurement is
over a second limit threshold value the deviation is to large and manual calibration is required
paragraph [0021]).
Therefore, it would be obvious for one skilled in the art before the effective filling date of the
claimed invention to have a maintenance alert when the calibration exceeds a threshold value as taught
by Bostrom in the microscope of Tripathi. The maintenance alert can initiate when manual calibration is
required paragraph [00221]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Heidemann (US 20170188015 A1) discloses a calibration plate and method for 3D calibration with a projector and at least one camera.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW R WRIGHT whose telephone number is (703)756-5822. The examiner can normally be reached Mon-Thurs 7:30-5 Friday 8-12.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pinping Sun can be reached at 1-571-270-1284. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW R WRIGHT/Examiner, Art Unit 2872
/PINPING SUN/Supervisory Patent Examiner, Art Unit 2872