DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1-11 were previously pending and subject to final action filed 01/28/2026. In the response filed on 06/28/2026, claim 1 were amended. Therefore, claims 1-11 are currently pending and subject to the non-final action below.
Response to Arguments
Applicant’s arguments, see page 1, filed 06/28/2026, with respect to claim 9 under claim objection have been fully considered and are persuasive. The claim objection of claim 9 has been withdrawn.
Applicant’s arguments, see pages 1-2, filed 06/28/2026, with respect to claims 1-11 under 35 U.S.C. 101 have been fully considered and are persuasive. The 101 rejection of claims 1-11 has been withdrawn.
Applicant's arguments filed 06/28/2026, with respect to 35 U.S.C. 103 of claims 1-11 have been fully considered but are moot because the arguments do not apply to the new combinations of references being used in the current rejection.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-2 and 4-8 are rejected under 35 U.S.C. 103 as being unpatentable over KUMAR (US PGPUB: 20230190136 A1, Filed Date: May. 27, 2020) in view of Leiderman (US PGPUB: 20220104884 A1, Filed Date Feb. 08, 2019) in view of NIR (US PGPUB: US 20180271603 A1, Pub Date: Sep. 27, 2018) In view of Kim (US 10991081 B1, Filed Date: Dec. 31, 2020).
Regarding independent claim 1, KUMAR teaches: A system, comprising: a camera positionable to capture image data corresponding to a treatment site for display on an image display; (KUMAR – [0008] calculating one or more measurements for the selected area in the video stream; and displaying the measurements on a displaying device intraoperatively to be used by an operator during the arthroscopic procedure [0018] [0067] In some embodiments, the arthroscopic imaging device comprises a digital camera.)
at least one processor and at least one memory, the at least one memory storing instructions executable by said at least one processor to: (KUMAR – [0067] The camera control unit may send the video stream to the one or more computer processors.)
analyze image data to determine an estimated tool width in pixels of a first surgical instrument positioned at the treatment site; (KUMAR − [0067] the video stream may be obtained from a digital camera specialized for an arthroscopic procedure. [0083] the reference sizing module 314 estimates a size of tooltip using a trained machine learning algorithm. In some embodiments, reference sizing module 314 provides a mask outlining the tip of the tool. In some embodiments, reference sizing module 314 also provides a width of the tool in pixels. Providing the size of the tooltip width in pixels.)
by comparing the estimated tool width to the known physical width of the first surgical instrument. (KUMAR − [0083] pathology sizing module 315 uses information from reference sizing module 314 to compare the recognized pathology from pathology recognition module 313 with a tool tip from reference sizing module 314. Examiner Notes: Comparing estimated tool tip from reference sizing model to the pathology sizing module 315)
and generating a real-time overlay for display on the image display, the real-time overlay representing the determined distance. (KUMAR − [0018] In some embodiments, the operations further comprise generating an overlay video stream by blending the labels or the measurements with the received video stream; and displaying the overlaid video stream on a displaying device.)
the depth coordinate [Z] being a function of the f parameter, the known physical width and the estimated tool width in pixels; (KUMAR − [0067] the video stream may be obtained from a digital camera specialized for an arthroscopic procedure. [0071] In embodiments, the output 130 may comprise a point-to-point distance measurement. A standard surgical probe may be used to register a start location. In some embodiments, a size of the probe or a part thereof is known. The system may identify the probe and store a location of the probe. The point-to-point distance may then be calculated by aggregating a displacement of the probe location from a first point to a second point. FIG. 2B shows an example of a point-to-point measurement between a start location 203 and an end location 204, with a traced line 205 that is straight. FIG. 2C shows an example of a point-to-point measurement, where the traced line 206 is curved. [0083] the reference sizing module 314 estimates a size of tooltip using a trained machine learning algorithm. In some embodiments, reference sizing module 314 provides a mask outlining the tip of the tool. In some embodiments, reference sizing module 314 also provides a width of the tool in pixels. Providing the size of the tooltip width in pixels.)
KUMAR teaches a depth coordinate but but does not explicitly teach: a depth coordinate Z of a tip of the first surgical instrument
However, Leiderman teaches: estimate a depth coordinate Z of a tip of the first surgical instrument (Leiderman − [0123] estimate the depth of the tip (i.e., the Z coordinate) to locate it in three-dimensional space.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of KUMAR and Leiderman as each invention same field of image analysis. One of ordinary skill in the art would have been motivated to make these modification in order to improves accuracy of surgical tool position and location.
KUMAR does not explicitly teach: determining in real time a distance between the tip of the first surgical instrument and a point of interest at the treatment site
However, NIR teaches: and based on the depth coordinate Z determined by comparing the estimated tool width to the known physical width of the surgical instrument, determining in real time a distance between the tip of the first surgical instrument and a point of interest at the treatment site, (NIR – [0277] The movement of an endoscope or other surgical tool can be parallel to the Z-axis [0474] Fig. 14 In this embodiment, as shown in FIG. 14, measurements are made of the distance X (150) from the tip (195) of the endoscope (100) to the pivot point of the endoscope (190), where said pivot point is at or near the surface of the skin (1100) of a patient (1000). Measurements are also made of the distance Y (250) from the tip of the endoscope (195) to the object in the center of the scene of view (200).) The Z-axis is the depth coordinate.
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of KUMAR, Leiderman and NIR as each invention same field of image analysis. One of ordinary skill in the art would have been motivated to make these modification in order to improves accuracy of surgical tool position and location.
KUMAR does not explicitly teach: in which the f parameter of the camera is the only required intrinsic camera parameter,
However, Kim teaches: using a projective camera model in which the f parameter of the camera is the only required intrinsic camera parameter, (Kim – Fig. 7A [Col. 10 ll. 6-7] FIG. 7A illustrates a pinhole camera model of taking the photograph 120. [Col. 10 ll. 24-36] Accordingly, a focal length f becomes the only intrinsic parameter of the pinhole camera that needs to be estimated/computed in the process. [Co. 10 ll. 35-40] Accordingly, using an ideal pinhole camera model has an advantage to reduce the number of camera's intrinsic parameters that need to be estimated.)
Accordingly, it would have been obvious to one of ordinary skill in the art to apply the pinhole camera model of Kim to the size-based depth determination of Kumar, as modified by Leiderman, such that the depth coordinate Z is determined as a function of focal length f, the known physical dimension of the surgical tool, and the corresponding estimated tool width in pixels, in order to determine depth while reducing the number of intrinsic camera parameters required for the determination.
Regarding dependent claim 2, depends on claim 1, KUMAR teaches: wherein the instructions are further executable by said at least one processor to estimate X and Y coordinates of the first surgical instrument tip (KUMAR − [0083] the reference sizing module 314 estimates a size of tooltip using a trained machine learning algorithm. In some embodiments, reference sizing module 314 provides a mask outlining the tip of the tool. In some embodiments, reference sizing module 314 also provides a width of the tool in pixels. Providing the size of the tooltip width in pixels.)
KUMAR does not explicitly teach: the first surgical instrument tip based on the Z coordinate.
However, Leiderman teaches: wherein the instructions are further executable by said at least one processor to estimate X and Y coordinates of the first surgical instrument tip based on the Z coordinate. (Leiderman − [0123] the contemplated embodiments determine XY coordinates of a surgical instrument's tip in two-dimensional space, estimate the depth of the tip (i.e., the Z coordinate) to locate it in three-dimensional space, and perform computation in real-time to provide a surgeon with the position of the surgical instrument's tip in three-dimensional space.)
Accordingly, it would have been obvious to one of ordinary skill in the art to apply the pinhole camera model of Kim to the size-based depth determination of Kumar, as modified by Leiderman, such that the depth coordinate Z is determined as a function of focal length f, the known physical dimension of the surgical tool, and the corresponding estimated tool width in pixels, in order to determine depth while reducing the number of intrinsic camera parameters required for the determination.
Regarding dependent claim 4, depends on claim 1, KUMAR teaches: wherein the camera is a monocular camera. (KUMAR − [0009] imaging device that is monocular)
Regarding dependent claim 5, depends on claim 1, KUMAR teaches: wherein the camera is a stereoscopic camera. (KUMAR − [0009] imaging device that is stereoscopic)
Regarding dependent claim 6, depends on claim 1, KUMAR teaches: wherein the only known intrinsic parameter of the camera is the f parameter. (KUMAR − [0009] imaging device that is monocular or stereoscopic [0094] In some embodiments, an imaging device comprises a lens of small focal length. Examiner Notes: Lens focal length is an intrinsic parameter of the camera.)
Regarding independent claim 7, KUMAR teaches: A method comprising:
capturing image data of a treatment site using a camera positioned in a body cavity; (KUMAR − [0071] The measurement of the point-to-point distance may be estimated based, at least in part on, the size of the probe and a location of the probe (e.g., the distance of the probe from the lens of a camera). FIG. 2A shows an example of a technique for initiating a start location for a measurement using an arbitrary probe 201. The probe and a tip 202 of the probe is recognized by the system)
analyzing the image data to determine an estimated tool width in pixels of a first surgical instrument positioned at the treatment site; (KUMAR − [0067] the video stream may be obtained from a digital camera specialized for an arthroscopic procedure. [0083] the reference sizing module 314 estimates a size of tooltip using a trained machine learning algorithm. In some embodiments, reference sizing module 314 provides a mask outlining the tip of the tool. In some embodiments, reference sizing module 314 also provides a width of the tool in pixels. Providing the size of the tooltip width in pixels.)
by comparing the estimated tool width to the known physical width of the first surgical instrument. (KUMAR − [0083] pathology sizing module 315 uses information from reference sizing module 314 to compare the recognized pathology from pathology recognition module 313 with a tool tip from reference sizing module 314. Examiner Notes: Comparing estimated tool tip from reference sizing model to the pathology sizing module 315)
displaying images from the camera in real time on an image display; (KUMAR − [0008] calculating one or more measurements for the selected area in the video stream; and displaying the measurements on a displaying device intraoperatively to be used by an operator during the arthroscopic procedure.)
and generating a real-time overlay for display on the image display, the real-time overlay representing the determined distance. (KUMAR − [0018] In some embodiments, the operations further comprise generating an overlay video stream by blending the labels or the measurements with the received video stream; and displaying the overlaid video stream on a displaying device.)
the depth coordinate Z being a function of the f parameter, the known physical width and the estimated tool width in pixels; (KUMAR − [0067] the video stream may be obtained from a digital camera specialized for an arthroscopic procedure. [0071] In embodiments, the output 130 may comprise a point-to-point distance measurement. A standard surgical probe may be used to register a start location. In some embodiments, a size of the probe or a part thereof is known. The system may identify the probe and store a location of the probe. The point-to-point distance may then be calculated by aggregating a displacement of the probe location from a first point to a second point. [0083] the reference sizing module 314 estimates a size of tooltip using a trained machine learning algorithm. In some embodiments, reference sizing module 314 provides a mask outlining the tip of the tool. In some embodiments, reference sizing module 314 also provides a width of the tool in pixels. Providing the size of the tooltip width in pixels.)
KUMAR teaches a depth coordinate Z but does not explicitly teach: a depth coordinate Z of a tip
However, Leiderman teaches: estimating a depth coordinate Z of a tip of the first surgical instrument (Leiderman − [0123] estimate the depth of the tip (i.e., the Z coordinate) to locate it in three-dimensional space.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of KUMAR and Leiderman as each invention same field of image analysis. One of ordinary skill in the art would have been motivated to make these modification in order to improves accuracy of surgical tool position and location.
KUMAR does not explicitly teach: in which the f parameter of the camera is the only required intrinsic camera parameter,
However, Kim teaches: using a projective camera model in which the f parameter of the camera is the only required intrinsic camera parameter, (Kim – Fig. 7A [Col. 10 ll. 6-7] FIG. 7A illustrates a pinhole camera model of taking the photograph 120. [Col. 10 ll. 24-36] Accordingly, a focal length f becomes the only intrinsic parameter of the pinhole camera that needs to be estimated/computed in the process. [Co. 10 ll. 35-40] Accordingly, using an ideal pinhole camera model has an advantage to reduce the number of camera's intrinsic parameters that need to be estimated.)
Accordingly, it would have been obvious to one of ordinary skill in the art to apply the pinhole camera model of Kim to the size-based depth determination of Kumar, as modified by Leiderman, such that the depth coordinate Z is determined as a function of focal length f, the known physical dimension of the surgical tool, and the corresponding estimated tool width in pixels, in order to determine depth while reducing the number of intrinsic camera parameters required for the determination.
Regarding dependent claim 8, depends on claim 7, KUMAR teaches: further comprising estimating X and Y coordinates of the first surgical instrument tip based on the Z coordinate (KUMAR − [0083] the reference sizing module 314 estimates a size of tooltip using a trained machine learning algorithm. In some embodiments, reference sizing module 314 provides a mask outlining the tip of the tool. In some embodiments, reference sizing module 314 also provides a width of the tool in pixels. Providing the size of the tooltip width in pixels.)
KUMAR does not explicitly teach: the first surgical instrument tip based on the Z coordinate
However, Leiderman teaches: further comprising estimating X and Y coordinates of the first surgical instrument tip based on the Z coordinate. (Leiderman − [0123] the contemplated embodiments determine XY coordinates of a surgical instrument's tip in two-dimensional space, estimate the depth of the tip (i.e., the Z coordinate) to locate it in three-dimensional space, and perform computation in real-time to provide a surgeon with the position of the surgical instrument's tip in three-dimensional space.)
Accordingly, it would have been obvious to one of ordinary skill in the art to apply the pinhole camera model of Kim to the size-based depth determination of Kumar, as modified by Leiderman, such that the depth coordinate Z is determined as a function of focal length f, the known physical dimension of the surgical tool, and the corresponding estimated tool width in pixels, in order to determine depth while reducing the number of intrinsic camera parameters required for the determination.
Claim(s) 3 and 9-11 are rejected under 35 U.S.C. 103 as being unpatentable over KUMAR , Leiderman, NIR and Kim as applied to claims 2 and 8 above, and further in view of NIXON (USPGPUB: 20090171371 A1, Filed Date: Dec. 26, 2007).
Regarding dependent claim 3, depends on claim 2, KUMAR teaches: wherein the instructions are further executable by said at least one processor to: analyze image data to determine an estimated tool width in pixels of a second surgical instrument positioned at the treatment site; (KUMAR − [0067] the video stream may be obtained from a digital camera specialized for an arthroscopic procedure. [0083] the reference sizing module 314 estimates a size of tooltip using a trained machine learning algorithm. In some embodiments, reference sizing module 314 provides a mask outlining the tip of the tool. In some embodiments, reference sizing module 314 also provides a width of the tool in pixels. Providing the size of the tooltip width in pixels.)
by comparing the estimated tool width of the second surgical instrument to the known physical width of the second surgical instrument, the tip of the second surgical instrument being the point of interest at the treatment site; (KUMAR − [0083] pathology sizing module 315 uses information from reference sizing module 314 to compare the recognized pathology from pathology recognition module 313 with a tool tip from reference sizing module 314. Examiner Notes: Comparing estimated tool tip from reference sizing model to the pathology sizing module 315)
KUMAR does not explicitly teach: estimate a depth coordinate Z of a tip of the first surgical instrument
However, Leiderman teaches estimate a depth coordinate Z of a tip of the second surgical instrument (Leiderman − [0123] estimate the depth of the tip (i.e., the Z coordinate) to locate it in three-dimensional space.)
and estimate X and Y coordinates of the second surgical instrument tip based on the estimated Z coordinate of the second surgical instrument tip, (Leiderman − [0123] the contemplated embodiments determine XY coordinates of a surgical instrument's tip in two-dimensional space, estimate the depth of the tip (i.e., the Z coordinate) to locate it in three-dimensional space, and perform computation in real-time to provide a surgeon with the position of the surgical instrument's tip in three-dimensional space.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of KUMAR and Leiderman as each invention same field of image analysis. One of ordinary skill in the art would have been motivated to make these modification in order to improves accuracy of surgical tool position and location.
KUMAR does not explicitly teach: and estimate a distance between the first surgical instrument tip and the second instrument tip using the estimated X,Y and Z coordinates for the first instrument tip and the second instrument tip.
However, NIXON teaches: and estimate a distance between the first surgical instrument tip and the second instrument tip using the estimated X,Y and Z coordinates for the first instrument tip and the second instrument tip. (NIXON − [0012] determining the distance between the first and second tools using the sensed positions of the first and second tools; [0070] as shown in FIG. 11 so that the distance determination processor 402 can compute a distance between the two points.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of KUMAR, Leiderman, NIR, Kim and NIXON as each invention same field of image analysis. One of ordinary skill in the art would have been motivated to make these modification in order to improves accurate measurement of surgical tools position and location.
Regarding dependent claim 9, depends on claim 8, KUMAR teaches: further including: analyzing the image data to determine an estimated tool width in pixels of a second surgical instrument positioned at the treatment site; (KUMAR − [0067] the video stream may be obtained from a digital camera specialized for an arthroscopic procedure. [0083] the reference sizing module 314 estimates a size of tooltip using a trained machine learning algorithm. In some embodiments, reference sizing module 314 provides a mask outlining the tip of the tool. In some embodiments, reference sizing module 314 also provides a width of the tool in pixels. Providing the size of the tooltip width in pixels.)
by comparing the estimated second tool width of the second surgical instrument to the known physical width of the second surgical instrument; (KUMAR − [0083] pathology sizing module 315 uses information from reference sizing module 314 to compare the recognized pathology from pathology recognition module 313 with a tool tip from reference sizing module 314. Examiner Notes: Comparing estimated tool tip from reference sizing model to the pathology sizing module 315)
KUMAR does not explicitly teach: estimating a depth coordinate Z of a tip of the first surgical instrument
However, Leiderman teaches estimating a depth coordinate Z of a tip of the second surgical instrument (Leiderman − [0123] estimate the depth of the tip (i.e., the Z coordinate) to locate it in three-dimensional space.)
and estimate X and Y coordinates of the second surgical instrument tip based on the estimated Z coordinate of the second surgical instrument tip, (Leiderman − [0123] the contemplated embodiments determine XY coordinates of a surgical instrument's tip in two-dimensional space, estimate the depth of the tip (i.e., the Z coordinate) to locate it in three-dimensional space, and perform computation in real-time to provide a surgeon with the position of the surgical instrument's tip in three-dimensional space.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of KUMAR and Leiderman as each invention same field of image analysis. One of ordinary skill in the art would have been motivated to make these modification in order to improves accuracy of surgical tool position and location.
KUMAR does not explicitly teach: and estimating a distance between the first surgical instrument tip and the second instrument tip using the estimated X,Y and Z coordinates for the first instrument tip and the second instrument tip.
However, NIXON teaches: and estimating a distance between the first surgical instrument tip and the second instrument tip using the estimated X,Y and Z coordinates for the first instrument tip and the second instrument tip. (NIXON − [0012] determining the distance between the first and second tools using the sensed positions of the first and second tools; [0070] as shown in FIG. 11 so that the distance determination processor 402 can compute a distance between the two points.)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of KUMAR, Leiderman, NIR, Kim and NIXON as each invention same field of image analysis. One of ordinary skill in the art would have been motivated to make these modification in order to improves accurate measurement of surgical tools position and location.
Regarding dependent claim 10, depends on claim 9, KUMAR teaches: further including displaying images from the camera on an image display, (KUMAR − [0036] FIG. 2A shows an example of initiating a start location for a measurement using a probe, according to some embodiments.)
KUMAR does not explicitly teach: and displaying an overlay on the image display showing the distance estimated.
However, NIXON teaches: and displaying an overlay on the image display showing the distance estimated. (NIXON − [0078] FIG. 15 illustrates, as an example, a display screen 1500 of the monitor 104, on which, an image of the tool 1200 and information of a distance traversed by the tip of the tool 1200 is show)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of KUMAR, Leiderman, NIR, Kim and NIXON as each invention same field of image analysis. One of ordinary skill in the art would have been motivated to make these modification in order to improves accurate measurement of surgical tools position and location.
Regarding dependent claim 11, depends on claim 10, KUMAR does not explicitly teach: further including displaying an overlay on the image identifying points between which the distance is measured.
However, NIXON teaches: further including displaying an overlay on the image identifying points between which the distance is measured. (NIXON − [0078] FIG. 15 illustrates, as an example, a display screen 1500 of the monitor 104, on which, an image of the tool 1200 and information of a distance traversed by the tip of the tool 1200 is show)
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the teaching of KUMAR, Leiderman, NIR, Kim and NIXON as each invention same field of image analysis. One of ordinary skill in the art would have been motivated to make these modification in order to improves accurate measurement of surgical tools position and location.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CARL E BARNES JR whose telephone number is (571)270-3395. The examiner can normally be reached Monday-Friday 9am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Hong can be reached at (571) 272-4124. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CARL E BARNES JR/Examiner, Art Unit 2178
/STEPHEN S HONG/Supervisory Patent Examiner, Art Unit 2178
