DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments filed May 1, 2026 argue that limitation that have been newly added to claim 1 by the instant amendment are not taught by Yang in view of Tang. The examiner agrees that some of these new limitations are not taught by the combined teachings of Yang and Tang, and therefore the rejections has been withdrawn. However, a new ground of rejection regarding these new limitations is set forth below. Some of the new limitations are taught by the combined teachings of Yang and Tang, as explained below.
Specifically, Applicant argues that the combined teachings of Yang and Tang do not teach the following new limitations:
''projecting, by the projector, a first pattern of dots and a second pattern of dots within the tracking volume, wherein the first pattern of dots is captured at a first orientation, wherein the second pattern of dots is captured at a second orientation different than the first orientation," ''capturing, by the image capture unit, at least two second images of a plurality of dots comprising (i) a portion of the first pattern of dots and (ii) a portion of the second pattern of dots," and ''determining three-dimensional positions of the plurality of dots from the at least two captured second images."
The examiner agrees that the combined teachings of Yang and Tang do not explicitly disclose this combination of new limitations. While the combined teachings of Yang and Tang disclose projecting structured light dot patterns into a tracking volume, capturing images of the structured light dot patterns and determining three-dimensional positions of the dot patterns from the captured images, the combined teachings of Yang and Tang do not explicitly disclose projecting a first pattern of dots and a second pattern of dots, capturing the first and second patterns of dots from first and second orientations that are different and 'determining three-dimensional positions of the plurality of dots from the two captured second images. For this reason, the rejection of claim 1 under 35 U.S.C. 103 as being unpatentable over Yang in view of Tang is withdrawn. However, a new ground of rejection is set forth below.
Regarding the new limitation of ''capturing, by the image capture unit, at least two first images of the medical instrument and the one or more markers within the tracking volume" and ''determining a three-dimensional position of the one or more markers within the tracking volume from the captured at least two first images of the one or more markers", the examiner disagrees with Applicant’s contention that the combined teachings of Yang and Tang do not teach these new limitations.
Yang discloses capturing multiple images of the medical instrument having markers thereon in order to perform tracking within the tracking volume, which is the field of view (FOV) of the imaging system. In Yang, the tracking volume is the volume within the field of view (FOV) of the stereoscopic tracking system employing the two tracking cameras (Fig. 2, para. [0137], medical instrument 40 has fiducial markers 45 attached thereto and is tracked by the stereoscopic tracking system). Since the stereoscopic tracking system uses two cameras, at least two first images are captured. In addition, since tracking of the medical instrument 40 is being performed, this indicates that multiple images are captured over time, which also means that at least two first images are captured. Para. [0137] describes the tracking system 20 determining the “position and orientation” of the medical instrument 40 based on images of the markers 45 acquired by the cameras of the stereoscopic tracking system, which means that the three-dimensional position of the markers 45 is determined. Similar findings by the examiner were set forth in the nonfinal Office Action and were not disputed in Applicant’s response to the nonfinal Office Action.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation (BRI) using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The BRI of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
In the following, some of the terms in the claims have been given BRIs in light of the specification. These BRIs are used for purposes of searching for prior art and examining the claims, but cannot be incorporated into the claims. Should Applicant believe that different interpretations are appropriate, Applicant should point to the portions of the specification that clearly support a different interpretation.
The BRI of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f), is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f):
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f), is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f), because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
Image capture unit in claims 1 and 15.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f), they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
The image capture unit recited in claim 1 is interpreted based on the present specification as one or more cameras and equivalents.
The image capture unit recited in claim 15 is interpreted based on the present specification as one or more cameras and one or more processors of, or in communication with, the camera(s) for performing the steps recited in claim 15, and equivalents.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 6-9, 12, 15-17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publ. Appl. No. 2019/0350658 A1 to Yang et al. (hereinafter referred to as “Yang”) in view of U.S. Pat. No. 11,880,992 B2 to Tang et al. (hereinafter referred to as “Tang”) and further in view of U.S. Publ. Appl. No. 2025/0049293 A1 to Shelton IV et al. (hereinafter referred to as “Shelton”).
Regarding claim 1, Yang discloses a system comprising:
a projector configured to project patterns of dots within a tracking volume (The BRI for this limitation is that some type of projection device projects structured light patterns comprising dots onto a surface. In Yang, structured light is used by the surface detection subsystem 10 to measure surface topology, such as the surface topology of the medical instrument and of the patient’s skin, para. [0136] and Fig. 2. The surface detection subsystem 10 can include “structured light imaging systems, which project surface topography detection light onto a region of interest”. Para. [0180] and Fig. 5C disclose using structured light to measure surface topology of anatomical features of the patient. Yang does not explicitly disclose that the structured light pattern is a pattern of dots);
a medical instrument having one or more markers (Fig. 2, para. [0137], medical instrument 40 has fiducial markers 45 attached thereto, para. [0137]), the medical instrument being positioned within the tracking volume (Para. [0137], Fig. 2, the tracking volume is the volume within the field of view (FOV) of the stereoscopic tracking system employing the two tracking cameras);
an image capture unit configured to capture imagery of the medical instrument and the one or more markers and configured to capture imagery of the pattern of dots within the tracking volume (para. [0137] and Fig. 2, the tracking system 20 comprises the stereoscopic tracking system that employes the two cameras for tracking the medical instrument 40 based on captured images of the markers 45); and
a computing device comprising a memory configured to store instructions and a processor to execute the instructions to perform operations (Fig. 2, processor 110 and memory 115, para. [0145]) comprising:
capturing, by the image capture unit, at least two first images of the medical instrument and the one or more markers within the tracking volume (para. [0137], the tracking system 20 under the control of the processor 110 initiates capture by the stereoscopic tracking system of images of the medical instrument 40 and the markers 45 affixed to the medical instrument 40; since the stereoscopic tracking system uses two cameras, at least two first images are captured; in addition, since tracking of the medical instrument 40 is being performed, this indicates that at least two first images are captured);
determining a three-dimensional position of the one or more markers from the captured images of the one or more markers (para. [0137] describes the tracking system 20 determining the “position and orientation” of the medical instrument 40 based on images of the markers 45 acquired by the cameras of the stereoscopic tracking system, which means that the three-dimensional position of the markers 45 is determined);
projecting, by the projector, a first pattern of dots and a second pattern of dots within the tracking volume, wherein the first pattern of dots is captured at a first orientation, wherein the second pattern of dots is captured at a second orientation different than the first orientation (Paras. [0137] and [0180], the structured light detection system of the surface detection subsystem 10, under the control of the processor 110, projects structured light patterns onto the medical instrument and/or onto an anatomical feature of the patient within the tracking volume. Since the projection of patterns of structured light occurs during tracking, multiple structured light patterns are projected. These multiple structured light patterns projected at different times constitute at least first and second light patterns. However, Yang does not explicitly disclose that the structured light patterns are dot patterns. Yang also does not explicitly disclose that the projected structured light patterns are captured at different, respective orientations);
capturing, by the image capture unit, at least two second images of a plurality of dots comprising (i) a portion of the first pattern of dots and (ii) a portion of the second pattern of dots (para. [0137], the structured light detection system of the surface detection subsystem 10, under the control of the processor 110, initiates capture of the structured light patterns by two tracking cameras of the stereoscopic tracking system 20 in order to detect the topology of the surface upon which the structured light is projected. Since the stereoscopic tracking system 20 uses two cameras, at least two images of first and second structured light patterns are acquired. Also, as indicated above, since tracking of the medical instrument 40 is being performed, this indicates that at least two images of the structured light patterns are captured over time during tracking); and
determining three-dimensional positions of the plurality of dots from the captured at least two second images (since the structure light patterns are being used by the structured light detection system of the surface detection subsystem 10 to determine surface topology of a subject or object, three-dimensional positions of features of the structured light patterns are being determined. As is well known in the art, structured light is used to determine three-dimensional surface topologies of a subject and therefore the three-dimensional positions of the features of the structured light patterns are necessarily determined in order to use their positions to determine the three-dimensional surface topologies of a subject).
As indicated above, Yang does not explicitly disclose that the structured light pattern is a pattern of dots. Tang, in the same field of endeavor, discloses using structured light projection and detection of dot patterns to perform 3D reconstruction of surfaces (Col. 11, lines 49-58; Col. 13, lines 12-30).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the surface detection subsystem 10 of Yang to measure surface topology by using structured light patterns comprising dots as taught by Tang. Yang is silent as to the type of structured light pattern that is used, but it is common in structured light imaging systems to use patterns of lines, grids, strips or dots. Therefore, it would have been obvious to one of ordinary skill in the art to use dot patterns as the structured light patterns in Yang as taught by Tang since there are a finite number of structured light patterns that are used for measuring surface topology and there would be a reasonable expectation of success using dot patterns since dot patterns have been successfully used in the past. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (using suitable light sources such as lasers or LEDs and/or suitable lens arrangements in the surface detection subsystem 10 of Yang to project dot light patterns and to detect the reflected dot patterns).
Yang also does not explicitly disclose that the projected structured light patterns are captured at different, respective orientations. Shelton, in the same field of endeavor, discloses using different structured light patterns that “may be cycled among a variety of grids having different line spacings or orientations” for different frame sequences (Para. [0617]). Examples provided in Shelton include projecting and capturing one light pattern over a particular number of image frames and then projecting and capturing a different light pattern over a particular number of subsequent image frames (Paras. [0616]-[0617]). Capturing the different structured light patterns having different “spacings or orientations” during different cycles constitutes capturing first and second light patterns at first and second orientations, respectively, that are different from one another.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the surface detection subsystem 10 of Yang to project and capture structured light patterns of different orientations during different cycles as taught by Shelton. A person of ordinary skill in the art would have been motivated to make the modification to “add three-dimensional tissue variances” and “provide additional tissue characteristics and/or metadata of the imaged tissue”, as taught by Shelton. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (using suitable image acquisition devices and circuitry to project and capture different dot patterns having different orientations over different cycles).
Regarding claim 2, Yang discloses determining the three-dimensional position of the one or more markers and the three-dimensional positions of the plurality of dots in a same coordinate system (para. [0157] discloses that during navigation/tracking the tracking system 20 and the surface detection subsystem 10 are in a common coordinate system relative to a reference frame that is attached to the patient and that “[s]ince the reference frame remains in a fixed position relative to the patient, recording position/orientation information from tracked instruments or the surface detection system relative to the reference frame position/orientation…is sufficient to compensate for these apparent motions”; this indicates that the three-dimensional positions of the markers and of the structured light pattern are in the same coordinate system).
As indicated above, Yang does not explicitly disclose that the structured light pattern is a pattern of dots. As indicated above, Tang discloses using structured light projection and detection of dot patterns to perform 3D reconstruction of surfaces (Col. 11, lines 49-58; Col. 13, lines 12-30).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the surface detection subsystem 10 of Yang to measure surface topology by using structured light patterns comprising dots as taught by Tang. As indicated above, Yang is silent as to the type of structured light pattern that is used, but it is common in structured light imaging systems to use line, grid, strip and dot patterns. Therefore, it would have been obvious to one of ordinary skill in the art to use dot patterns as the structured light patterns in Yang since there are a finite number of structured light patterns that are used for measuring surface topology and there would be a reasonable expectation of success using dot patterns since dot patterns have been successfully used in the past. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (using suitable light sources such as lasers or LEDs and/or suitable lens arrangements in the surface detection subsystem 10 of Yang to project dot light patterns and to detect the reflected dot patterns).
Regarding claim 3, Yang discloses determining a position of patient anatomy using the three-dimensional positions of the structured light patterns (Yang discloses using the detected surface topology of the detected structured light patterns to determine the surface topology of anatomy. Determining surface topology constitutes determining three-dimensional positions. Para. [0136]: “[t]he surface detection system 10 may be any suitable system for detecting, measuring, imaging, or otherwise determining the surface topography of one or more objects (such as, but not limited to, an exposed cranial tissue region of a subject 50, such the skin, the skull, or underlying tissues)...Non-limiting examples of suitable optical devices include…structured light imaging systems, which project surface topography detection light onto a region of interest, and detect surface topography light that is scattered or reflected from the region of interest”). As indicated above, Yang but does not disclose that the light pattern is a pattern of dots.
As indicated above, Tang discloses using structured light projection and detection of dot patterns to perform 3D reconstruction of surfaces (Col. 11, lines 49-58; Col. 13, lines 12-30).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the surface detection subsystem 10 of Yang to measure surface topology by using structured light patterns comprising dots as taught by Tang for the reasons discussed above in the rejection of claims 1-2. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (using suitable light sources such as lasers or LEDs and/or suitable lens arrangements in the surface detection subsystem 10 of Yang to project dot light patterns and to detect the reflected dot patterns).
Regarding claim 4, Yang discloses determining the 3D positions of the structured light patterns by using the captured structured light patterns because Yang discloses determining the surface topology of the structured light reflected from the surface upon which it is projected (Para. [0136]: “[t]he surface detection system 10 may be any suitable system for detecting, measuring, imaging, or otherwise determining the surface topography of one or more objects (such as, but not limited to, an exposed cranial tissue region of a subject 50, such the skin, the skull, or underlying tissues)...Non-limiting examples of suitable optical devices include…structured light imaging systems, which project surface topography detection light onto a region of interest, and detect surface topography light that is scattered or reflected from the region of interest”).
However, Yang does not explicitly disclose that the structured light patterns are patterns of dots. As indicated above, Tang discloses using structured light projection and detection of dot patterns to perform 3D reconstruction of surfaces (Col. 11, lines 49-58; Col. 13, lines 12-30).
For the reasons discussed above in the rejections of claims 1-3, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the surface detection subsystem 10 of Yang to measure surface topology by using structured light patterns comprising dots as taught by Tang. In addition, a person of ordinary skill in the art would have been motivated to make the modification to reduce the number of pixels making up the dots that have to be processed as taught by Tang. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (using suitable light sources such as lasers or LEDs and/or suitable lens arrangements in the surface detection subsystem 10 of Yang to project dot light patterns and to detect the reflected dot patterns).
Regarding claim 6, Yang does not explicitly disclose matching the plurality of dots across the captured images. Tang discloses matching the dots across the captured images of the portion of the pattern of dots (Col. 3, lines 55-67 and Col. 31, lines 1-34 discuss performing block matching, which in the context of structured-light dot pattern images, means matching dots of a captured dot-pattern image with corresponding dots of another captured dot-pattern image).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the surface detection subsystem 10 of Yang to perform block matching of dots across multiple images as taught by Tang. A person of ordinary skill in the art would have been motivated to make the modification to match the dots of multiple images during tracking to ensure that the medical instrument and/or the anatomical feature surface being measured are properly tracked. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (using a block matching algorithm or similar algorithm to match dots across multiple images).
Regarding claim 7, Yang does not explicitly disclose projecting the first pattern of dots in time intervals. Tang discloses projecting the first pattern of dots at time intervals corresponding to every third video frame (Col. 37, lines 36-52).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the surface detection subsystem 10 of Yang to measure surface topology by using structured light patterns comprising dots projected in time intervals as taught by Tang. A person of ordinary skill in the art would have been motivated to make the modification to reduce the number of pixels making up the dots that have to be processed as taught by Tang and to reduce the need to use filtering to reduce noise in the RGB images caused by the dot pattern images and vice versa . The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (using suitable light sources such as lasers or LEDs and/or suitable lens arrangements in the surface detection subsystem 10 of Yang along with suitable timing control circuitry to project dot light patterns in time intervals and to detect the reflected dot patterns).
Regarding claim 8, Yang does not explicitly disclose that the capturing of the at least two second images of the structured light patterns is synchronized with the time intervals, or that the first structured light pattern is geometrically changed between subsequent projections. As indicated above, Shelton discloses using different structured light patterns that “may be cycled among a variety of grids having different line spacings or orientations” for different frame sequences (Para. [0617]). Changing the structured line pattern grids to have different line spacings or orientations during different frame sequences constitutes changing the structured light pattern geometrically between subsequent projections.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the surface detection subsystem 10 of Yang to project and capture structured light patterns of different geometries during different cycles as taught by Shelton. A person of ordinary skill in the art would have been motivated to make the modification to “add three-dimensional tissue variances” and “provide additional tissue characteristics and/or metadata of the imaged tissue”, as taught by Shelton. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (using suitable image acquisition devices and circuitry to project and capture different dot patterns having different orientations over different cycles).
Regarding claim 9, as indicated above in the rejections of claims 1 and 8, Shelton discloses using different structured light patterns that “may be cycled among a variety of grids having different line spacings or orientations” for different frame sequences (Para. [0617]). Examples provided in Shelton include projecting and capturing one light pattern over a particular number of image frames and then projecting and capturing a different light pattern over a particular number of subsequent image frames (Paras. [0616]-[0617]). Therefore, the timing intervals of the projections of the different structured light patterns is synchronized with the timing intervals of the image frame captures in Shelton. Shelton also discloses that the structured light patterns can be dot patterns (para. [0208]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the surface detection subsystem 10 of Yang to project and capture structured light patterns comprising a plurality of dots during different cycles with the image projection and the image capturing being synchronized as taught by Shelton. A person of ordinary skill in the art would have been motivated to make the modification to “add three-dimensional tissue variances” and “provide additional tissue characteristics and/or metadata of the imaged tissue”, as taught by Shelton. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (using suitable image acquisition devices and circuitry to project and capture different dot patterns different cycles).
Regarding claim 12, Yang is silent regarding the timing of capturing the two first images of the markers and the two second images of the structured light patterns. Tang discloses capturing at least two first RGB images during a first time period (Fig. 7, Col. 7, lines 36-52, two first RGB images are captured in the frames 302b and 302c other than every third frame) and capturing at least two dot-pattern images in a second time period (Fig. 7, Col. 7, lines 36-52, two second dot pattern images are captured every third frame 302a and 302d), where the first and second time periods are different.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the surface detection subsystem 10 of Yang to measure surface topology by using structured light patterns comprising dots projected in time intervals as taught by Tang and capturing the dot-pattern images in synchronization with the time intervals as taught by Tang while capturing marker images in different time intervals. A person of ordinary skill in the art would have been motivated to make the modification to reduce the number of pixels making up the dots that have to be processed as taught by Tang and to prevent interference between the marker image data and the structured light image data. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (using suitable light sources such as lasers or LEDs and/or suitable lens arrangements in the surface detection subsystem 10 of Yang along with suitable timing control circuitry to project and detect dot light patterns in certain time intervals and to capture the marker images during time intervals that are different from the time intervals during which the structured light images are captured).
Regarding claims 15-17, the rejection of claims 1-3 apply mutatis mutandis to claims 15-17, respectively.
Regarding claim 20, the rejection of claim 1 applies mutatis mutandis to claim 20.
Claims 5, 11, 14 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Tang and Shelton as applied to claims 1-4, 6-8, 9, 12, 15-17 and 20 and further in view of U.S. Pat. No. 10,016,243 B2 to Esterberg (hereinafter referred to as “Esterberg”).
Regarding claim 5, the combined teachings of Yang, Tang and Shelton do not explicitly disclose that determining the three-dimensional positions of the dots comprises determining a centroid. Esterberg, in the same field of endeavor, discloses systems and methods for assisted surgical navigation in which dot patterns are projected onto the surgical field. The dot-pattern image is captured and the centroid of each dot is determined (Col. 2, line 59-Col. 3, line 6).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system of Yang as modified by Tang and Shelton further based on the teachings of Esterberg such that the centroids of the dots are determined and used to determine the 3D positions of the dots. A person of ordinary skill in the art would have been motivated to make the modification to determine the 3D positions of the dots with greater precision by ensuring that the centers of the dots are precisely located. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (using a known centroid determination algorithm to calculate the centroids of pixels comprising the dots).
Regarding claim 11, the combined teachings of Yang, Tang and Shelton do not explicitly teach that the pattern of dots comprises a pseudorandom pattern. Esterberg discloses that the pattern of dots can comprise a pseudorandom pattern of dots (Col. 12, lines 3-6 and Col. 15, lines 8-11).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system of Yang as modified by Tang and Shelton further based on the teachings of Esterberg such that the pattern of dots being projected is a pseudorandom pattern as taught by Esterberg. A person of ordinary skill in the art would have been motivated to make the modification to allow correspondence matching between the dots of the pattern generated by the projector and the dots reflected from the surface being measured. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (varying the lens arrangement and/or the light source driver of the projector of Yang to pseudorandomize the dot patterns being projected).
Regarding claim 14, the combined teachings of Yang, Tang and Shelton do not explicitly teach that the projector is positioned remote from a housing that contains the image capture unit and that is movable within the tracking volume.
Esterberg discloses that the projector can be positioned remote from the housing that contains the image capture unit (Col. 19, lines 19-22, the dot matrix projector 805 can be remote from the HMD housing in which the image capture unit is mounted, such as on a ceiling mount). Esterberg also discloses that the structured light projector (Fig. 2, laser matrix projector 220, Col. 12, lines 43-54) can be affixed to a head mount (Fig. 2, headset 105), which means that it is movable within the tracking volume as the surgeon’s head moves.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system of Yang as modified by Tang and Shelton further based on the teachings of Esterberg such that the dot pattern projector is mounted remotely from the image capture unit and movable within the tracking volume as taught by Esterberg. A person of ordinary skill in the art would have been motivated to make the modification to reduce the size and complexity of the housing that houses the image capture unit. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (remotely mounting the projector while allowing it to be movable within the tracking volume).
Regarding claim 21, some of the limitations in this claim are recited in claims 5 and 6. Accordingly, the rejections of claims 5 and 6 apply mutatis mutandis to claim 21. Regarding the triangulation limitation of claim 21, the combined teachings of Yang, Tang and Shelton do not explicitly teach this limitation. However, Esterberg teaches this limitation (Col. 12, lines 6-13, Col. 16, lines 13-28 and Col. 16, lines 33-37).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system of Yang as modified by Tang and Shelton further based on the teachings of Esterberg such that the dot pattern matching across the captured images to determine 3D positions of the plurality of dots comprises triangulating centroid locations of the plurality of dots based on geometries of the cameras as taught by the combined teachings of Tang and Esterberg. A person of ordinary skill in the art would have been motivated to make the modification to improve the accuracy of determining the 3D profiles of anatomical features. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (implementing software in the system of Yang to perform dot pattern centroid determination and matching).
Claims 10 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Tang, Shelton and Esterberg as applied to claims 5, 11 and 14 and further in view of an article entitled “Surgical Structured Light for 3D Minimally Invasive Surgical Imaging”, by Reiter et al., published September 14-18, 2014 in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014) (hereinafter referred to as “Reiter”).
Regarding claim 10, the combined teachings of Yang, Tang, Shelton and Esterberg do not explicitly teach using a first camera at an orientation to capture images of the markers and using a second camera at an orientation that is different from the orientation of the first camera to capture images of the structured light dot patterns.
Reiter, in the same field of endeavor, discloses using a first white light camera positioned at an orientation to capture images of the surgical site including the human anatomy and the medical instrument and a second pattern camera at a different orientation to capture images of projected structured light patterns (Figs. 1 and 2 show the cameras at different orientations and Section II, System Overview discusses the system setup).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system of Yang as modified by Tang, Shelton and Esterberg further based on the teachings of Reiter to use a first white light camera in the system of Yang to capture images of the medical instrument and the one or more markers and a second pattern camera to capture the images of the structured light patterns of dots. A person of ordinary skill in the art would have been motivated to make the modification to improve depth perception and facilitate 3D reconstruction of the surgical site. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (using separate cameras to capture images of the medical instrument and of the projected structured light patterns).
Regarding claim 22, the combined teachings of Yang, Tang, Shelton and Esterberg do not explicitly teach that the at least two first images and the at least two second images are captured in the same portion of the electromagnetic spectrum.
Reiter discloses that the images that are captured by the white light camera and both the pattern camera are all in the white light spectrum (Fig. 2, the LED projector projects narrowband blue light that is patterned by dispersion mask into the scene and the dichroic beam splitter passes the blue light to the pattern camera and the remaining light other than the blue light to the white light camera). Therefore, all images are captured in the visible light electromagnetic spectrum.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system of Yang as modified by Tang, Shelton and Esterberg further based on the teachings of Reiter to capture all images in the white light spectrum as taught by Reiter. A person of ordinary skill in the art would have been motivated to make the modification to reduce the complexity of the system in a way that allows the blue light to be removed without the surgeon realizing a strong difference in the color of the image with the blue light removed as taught by Reiter. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (implementing a system setup in Yang that is similar to the system setup disclosed in Reiter).
Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Tang and Shelton as applied to claims 1-3, 6-8, 9, 12, 15-17 and 20 and further in view of U.S. Pat. No. 11,350,066 B1 to Jaime Rene De La Cruz Vazquez (hereinafter referred to as “Vazquez”).
Regarding claim 23, the combined teachings of Yang, Tang and Shelton do not explicitly teach that each pattern of dots defines a respective point cloud of surfaces within the tracking volume projected on by the projector. Vazquez, in the same field of endeavor, discloses projecting structured light dot patterns onto a projection surface and capturing the projected dot patterns, changing the pose of the projector and camera, and repeating the project/capture process (Col. 4, lines 17-44), where each pattern of dots defines a respective point cloud of surfaces within the tracking volume projected on by the projector (Col. 7, lines 8-23: “[f]or each pose, the dots in point cloud 500 represent the points of the structured light pattern 404”).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system of Yang as modified by Tang and Shelton further based on the teachings of Vazquez such that each pattern of dots defines a respective point cloud of surfaces as taught by Vazquez. A person of ordinary skill in the art would have been motivated to make the modification to take advantage of the dimensional precision of point cloud data and its ability to be encoded/compressed into different formats for transmission. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (using encoding/decoding software in the system of Yang to encode/decode image data representing the dot patterns as point cloud data).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL J SANTOS whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL J. SANTOS/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights