DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 09/24/2025, 10/23/2025, and 12/10/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements have been considered by the examiner.
Status of Claims
Applicant’s Amendments filed on 09/24/2025 has been entered and made of record.
Currently pending Claim(s)
1, 3, 5–9, 17, 25–28, 31–33, 37, 41, and 43
Independent Claim(s)
1
Amended Claim(s)
1, 6, 7, 15, 26, 27, 31, 37 and 43
Response to Amendment
This office action is responsive to Applicant’s Arguments/Remarks Made in an Amendment received on 09/24/2025.
In view of the claim amendments [Remarks] filed on 09/24/2025, the claim objection for claim 43 has been withdrawn.
In view of the claim amendments [Remarks] filed on 09/24/2025 with respect to 35 U.S.C. 112(b) claim rejections to claims 1, 7, 26, 27, 31, and 37 have been carefully considered and the claim rejections under 35 U.S.C. 112(b) are withdrawn.
Applicant’s Arguments/Remarks with respect to the 101 rejection, on pages 7-11, have been fully considered but are not persuasive. Applicant states, on the bottom of page 8-bottom of page 9, as follows:
PNG
media_image1.png
136
898
media_image1.png
Greyscale
PNG
media_image2.png
756
840
media_image2.png
Greyscale
However, the Examiner respectfully disagrees. Independent claim 1 is interpreted under 35 U.S.C. 101 because the “determining” and “overlaying” limitations simply consist of observing an occlusion of a previously received landmark on a video image and overlaying the landmark above the occlusion or obstruction which is practically capable of being performed in the human mind with the assistance of pen and paper. Although the disclosed invention may be “rooted in computer technology to overcome a problem specifically arising in graphical user interfaces,” the claimed invention provides no meaningful limitations for the improvement to be realized. Therefore, the claim does not amount to significantly more than the abstract idea.
Regarding Applicant’s Arguments for Step 2A, Prong Two, the Examiner respectfully disagrees. For example, although Applicant argues, on the top of page 10, that independent claim 1 provides “improved landmark stability, particularly when tool movements (or other objects) can impede or block the visibility of the landmark,” this is not disclosed in the independent claim. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims (See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993)). In addition, Applicant states, on page 10, the following:
PNG
media_image3.png
548
762
media_image3.png
Greyscale
The Examiner respectfully disagrees. The claim elements of “receiving a video stream from an arthroscopic imaging device,” “receiving one or more sets of coordinates of one or more landmarks,” and “displaying said overlay on more or more display devices intraoperatively to be used by an operator during said arthroscopic procedure” are simply necessary data gathering and outputting and are insignificant extra-solution activity. The “receiving” limitations are necessary to perform the abstract idea or mental process limitations and the “displaying” limitation is outputting the results of the abstract idea. Furthermore, the additional elements in the dependent claims do not place the abstract idea into a practical application. These additional statements fail to impose meaningful limits because, for example, providing low-latency with a camera-base system is a generic computer component.
Applicant’s Reply (September 24, 2025) includes substantive amendments to the claims. This Office action has been updated with new grounds of rejection addressing those amendments. Further Applicant’s Arguments/Remarks with respect to independent claims 1, on pages 11–17, have been considered but are moot because the arguments do not apply to any of the references being used in the current rejection and the arguments are now rejected by newly cited art ‘Winne et al. ("Overlay visualization in endoscopic ENT surgery.")’ as explained in the body of rejection below.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 15 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 15 recites the limitation "said radiological images" in line 1. There is insufficient antecedent basis for this limitation in the claim. For examination purposes, this limitation will be read as “radiological images” instead.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3, 5-9, 12, 15, 17, 25-28, 31-33, 37, 41, and 43 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., abstract idea – mental process) without significantly more.
Step (1) Are the claims directed to a process, machine, manufacture, or composition of matter;
Step (2A) Prong One: Are the claims directed to a judicially recognized exception, i.e., a law of nature, a natural phenomenon, or an abstract idea;
Prong Two: If the claims are directed to a judicial exception under Prong One, then is the judicial exception integrated into a practical application;
Step (2B) If the claims are directed to a judicial exception and do not integrate the judicial exception, do the claims provide an inventive concept.
Claim 1:
Step 1:
Claim 1 recites a system for assisting a minimally invasive procedure. Therefore, the claim is directed to the statutory category of machine.
Step 2A:
Prong One:
Claim 1 recites “determining an occlusion of at least one of said one or more landmarks by at least one object in said video stream” and “overlaying said one or more landmarks on said video stream, wherein the overlaying comprises displaying the one or more landmarks over the at least one object.” Under its broadest reasonable interpretation in light of the specification, these limitations encompass the mental process of determining an occlusion or blocking of a landmark and overlaying the landmark in a video stream which is practically capable of being performed in the human mind with the assistance of pen and paper.
Prong Two:
This judicial exception is not integrated into a practical application. The additional elements of “receiving a video stream from an arthroscopic imaging device,” “receiving one or more sets of coordinates of one or more landmarks,” and “displaying said overlay on one or more display devices intraoperatively to be used by an operator during said arthroscopic procedure” amount to no more than mere necessary data gathering and outputting. Thus, they are insignificant extra-solution activity. Even when viewed in combination, these additional elements do not integrate the abstract idea into a practical application and the claim is thus directed to the abstract idea.
Step 2B:
Claim 1 do not include additional elements that are sufficient to amount to significantly more than the judicial exception. “Receiving a video stream from an arthroscopic imaging device,” “receiving one or more sets of coordinates of one or more landmarks,” and “displaying said overlay on one or more display devices intraoperatively to be used by an operator during said arthroscopic procedure” amount to no more than mere data gathering and outputting. These elements, individually and in combination, are well-understood, routine, conventional activity. As such, the claim is ineligible.
Claim 3:
Step 1:
Claim 3 recites a system for assisting a minimally invasive procedure. Therefore, the claim is directed to the statutory category of machine.
Step 2A:
Prong One:
Claim 3 recites “wherein said operations further comprise identifying and labeling one or more elements in said video stream using at least one of a trained computer algorithm and one or more modules, wherein said one or more elements comprise one or more of an anatomical structure, a surgical tool, an operational procedure or action, or a pathology.” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of identifying and labelling elements in a video which is practically capable of being performed in the human mind with the assistance of pen and paper.
Prong Two:
This judicial exception is not integrated into a practical application. The additional elements of “using at least one of a trained computer algorithm” and “one or more modules” provide mere instructions to implement an abstract idea on a generic computer component with its functionality being recited at a high level of generality. Thus, they are insignificant extra-solution activity. Even when viewed in combination, these additional elements do not integrate the abstract idea into a practical application and the claim is thus directed to the abstract idea.
Step 2B:
Claim 3 do not include additional elements that are sufficient to amount to significantly more than the judicial exception. “Using at least one of a trained computer algorithm” and “one or more modules” provide mere instructions to implement an abstract idea of a generic computer component with its functionality being recited at a high level of generality. These elements, individually and in combination, are well-understood, routine, conventional activity. As such, the claim is ineligible.
Claim 5:
Step 1:
Claim 5 recites a system for assisting a minimally invasive procedure. Therefore, the claim is directed to the statutory category of machine.
Step 2A:
Prong One:
Claim 5 merely narrow the previously recited abstract idea limitation. For the reasons described above, this judicial exception is not meaningfully integrated into a practical application, or significantly more than the abstract idea. The claim disclose similar limitations described for the independent claim above and do not provide anything more than the mental process that is practically capable of being performed in the human mind with the assistance of pen and paper.
Prong Two:
This judicial exception is not integrated into a practical application nor includes additional elements that are sufficient to amount to significantly more. Thus, the claim is ineligible.
Claim 6:
Step 1:
Claim 6 recites a system for assisting a minimally invasive procedure. Therefore, the claim is directed to the statutory category of machine.
Step 2A:
Prong One:
Claim 6 recites “wherein said system recommends one or more landmarks based at least partially on said identified elements.” Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of recommending landmarks based on identified elements from the video which is practically capable of being performed in the human mind with the assistance of pen and paper.
Prong Two:
This judicial exception is not integrated into a practical application nor includes additional elements that are sufficient to amount to significantly more. Thus, the claim is ineligible.
Claim 7:
Step 1:
Claim 7 recites a system for assisting a minimally invasive procedure. Therefore, the claim is directed to the statutory category of machine.
Step 2A:
Prong One:
Claim 7 recites “identifying said one or more sets of coordinates for said one or more landmarks.” Under its broadest reasonable interpretation in light of the specification, these limitations encompasses the mental process of determining coordinates for landmarks which is practically capable of being performed in the human mind with the assistance of pen and paper.
Prong Two:
This judicial exception is not integrated into a practical application. The additional elements of “storing the one or more sets of coordinates of one or more landmarks,” “changing a view of said display to omit said overlaid landmark from being displayed,” and “reverting said view to said previous display” and “re-overlaying said one or more landmarks” amount to no more than mere necessary data gathering and outputting. Thus, they are insignificant extra-solution activity. Even when viewed in combination, these additional elements do not integrate the abstract idea into a practical application and the claim is thus directed to the abstract idea.
Step 2B:
Claim 7 do not include additional elements that are sufficient to amount to significantly more than the judicial exception. “Storing the one or more sets of coordinates of one or more landmarks,” “changing a view of said display to omit said overlaid landmark from being displayed,” and “reverting said view to said previous display” and “re-overlaying said one or more landmarks” amount to no more than mere data gathering and outputting. These elements, individually and in combination, are well-understood, routine, conventional activity. As such, the claim is ineligible.
Claims 8-9, 12, and 15:
Step 1:
Claims 8-9, 12, and 15 recite a system for assisting a minimally invasive procedure. Therefore, the claims are directed to the statutory category of machine.
Step 2A:
Prong One:
Claims 8-9, 12, and 15 merely narrow the previously recited abstract idea limitations. For the reasons described above, these judicial exceptions are not meaningfully integrated into a practical application, or significantly more than the abstract idea. The claims disclose similar limitations described for the independent claim above and do not provide anything more than the mental process that is practically capable of being performed in the human mind with the assistance of pen and paper.
Prong Two:
These judicial exception are not integrated into a practical application nor includes additional elements that are sufficient to amount to significantly more. Thus, the claims are ineligible.
Claim 17:
Step 1:
Claim 17 recites a system for assisting a minimally invasive procedure. Therefore, the claim is directed to the statutory category of machine.
Step 2A:
Prong One:
Claim 17 merely narrows the previously recited abstract idea limitation. For the reasons described above, this judicial exception is not meaningfully integrated into a practical application, or significantly more than the abstract idea. The claim disclose similar limitations described for the independent claim above and do not provide anything more than the mental process that is practically capable of being performed in the human mind with the assistance of pen and paper.
Prong Two:
This judicial exception is not integrated into a practical application. The additional elements of “an arthroscope” is simply used as a tool to perform the generic function of providing video. Thus, this is insignificant extra-solution activity. This additional element do not integrate the abstract idea into a practical application and the claim is thus directed to the abstract idea.
Step 2B:
Claim 17 do not include additional elements that are sufficient to amount to significantly more than the judicial exception. “An arthroscope” is simply used as a tool to perform the generic function of providing video. This element is well-understood, routine, conventional activity. As such, the claim is ineligible.
Claims 25-28:
Step 1:
Claims 25-28 recite a system for assisting a minimally invasive procedure. Therefore, the claims are directed to the statutory category of machine.
Step 2A:
Prong One:
Claims 25-28 merely narrow the previously recited abstract idea limitation. For the reasons described above, this judicial exception is not meaningfully integrated into a practical application, or significantly more than the abstract idea. The claims disclose similar limitations described for the independent claim above and do not provide anything more than the mental process that is practically capable of being performed in the human mind with the assistance of pen and paper.
Prong Two:
This judicial exception is not integrated into a practical application. The additional elements of “one or more computer processors,” “one or more camera control units,” and “a digital camera” are simply used as tools to perform the generic computer function of receiving video. Thus, they are insignificant extra-solution activity. These additional elements do not integrate the abstract idea into a practical application and the claims are thus directed to the abstract idea.
Step 2B:
Claims 25-28 do not include additional elements that are sufficient to amount to significantly more than the judicial exception. “One or more computer processors,” “one or more camera control units,” and “a digital camera” are simply used as tools to perform the generic computer function of providing video. These elements are well-understood, routine, conventional activity. As such, the claims are ineligible.
Claims 31-33, 37:
Step 1:
Claims 31-33 and 37 recite a system for assisting a minimally invasive procedure. Therefore, the claims are directed to the statutory category of machine.
Step 2A:
Prong One:
Claims 31-33 and 37 merely incorporate generic tools to the previously recited abstract idea limitation. For the reasons described above, this judicial exception is not meaningfully integrated into a practical application, or significantly more than the abstract idea. The claims disclose similar limitations described for the independent claim above and do not provide anything more than the mental process that is practically capable of being performed in the human mind with the assistance of pen and paper.
Prong Two:
This judicial exception is not integrated into a practical application. The additional elements of “a camera control unit,” “a light source,” and “a digital camera” are simply used as tools to perform the generic function of obtaining video. Thus, they are insignificant extra-solution activity. These additional elements do not integrate the abstract idea into a practical application and the claims are thus directed to the abstract idea.
Step 2B:
Claims 31-33 and 37 do not include additional elements that are sufficient to amount to significantly more than the judicial exception. “A camera control unit,” “a light source,” and “a digital camera” are simply used as tools to perform the generic function of obtaining video. These elements are well-understood, routine, conventional activity. As such, the claims are ineligible.
Claim 41:
Step 1:
Claim 41 recites a system for assisting a minimally invasive procedure. Therefore, the claim is directed to the statutory category of machine.
Step 2A:
Prong One:
Claim 41 merely incorporates generic tools to the previously recited abstract idea limitation. For the reasons described above, this judicial exception is not meaningfully integrated into a practical application, or significantly more than the abstract idea. The claim disclose similar limitations described for the independent claim above and do not provide anything more than the mental process that is practically capable of being performed in the human mind with the assistance of pen and paper.
Prong Two:
This judicial exception is not integrated into a practical application. The additional element of “a touchscreen device” is simply used as a tool to perform the generic computer function of obtaining input. Thus, it is insignificant extra-solution activity. This additional element do not integrate the abstract idea into a practical application and the claim is thus directed to the abstract idea.
Step 2B:
Claim 41 do not include additional elements that are sufficient to amount to significantly more than the judicial exception. “A touchscreen device” is simply used as a tool to perform the generic computer function of obtaining input. This element is well-understood, routine, conventional activity. As such, the claim is ineligible.
Claim 43:
Step 1:
Claim 43 recites a system for assisting a minimally invasive procedure. Therefore, the claim is directed to the statutory category of machine.
Step 2A:
Prong One:
Claim 43 recites “wherein said one or more landmarks are tracked during said minimally invasive procedure, further wherein said tracking of one or more landmarks is associated with said set of coordinates of said one or more landmarks relative to at least an anatomical structure.” Under its broadest reasonable interpretation in light of the specification, these limitations encompasses the mental process tracking landmarks with coordinates which is practically capable of being performed in the human mind with the assistance of pen and paper.
Prong Two:
This judicial exception is not integrated into a practical application nor includes additional elements that are sufficient to amount to significantly more. Thus, the claim is ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 12, 15, and 41 are rejected under 35 U.S.C. 103 as being unpatentable over Winne et al. (Winne, Christian, et al. "Overlay visualization in endoscopic ENT surgery." International journal of computer assisted radiology and surgery 6.3 (2011): 401-406) (hereafter, “Winne”) in view of Chabanas et al. (US 2013/0096373 A1) (hereafter, “Chabanas”).
Regarding claim 1, Winne discloses A system for assisting a minimally invasive procedure by allowing computer-implemented arbitrary landmark placement [Winne, to demonstrate the benefit of the overlay visualization in ENT surgery, an assistance system was developed, pg. 402, left column, Materials and methods, first paragraph], the system comprising one or more computer processors and one or more non-transitory computer-readable storage media storing instructions that are operable, when executed by said one or more computer processors, to cause said one or more computer processors to perform operations [Winne, it is based on a surgical navigation platform for ENT surgery and is composed of a control computer (Penta GmbH, Germany), the optical positional measurement system Polaris (NDI, ON, Canada) and user interface modules such as a touch screen, pg. 402, left column, Materials and methods, first paragraph] comprising: receiving a video stream [from an arthroscopic imaging device] [this base system was extended by integrating a Storz HOPKINS II 4 x 180 mm endoscope in conjunction with a Storz TRICAM SL camera ... the objective of this work was to support the surgeon in locating target structures with overlay visualization within intraoperative endoscopic video images, pg. 402, left column, Materials and Methods, second paragraph; left column, Introduction, second paragraph]; determining an occlusion of at least one of said one or more landmarks by at least one object in said video stream [the preoperatively defined target structure and intraoperatively used instruments are marked within the endoscopic video image. This enables the surgeon to assess the position of the target region though it is hidden behind other tissue and not visible in the video image, pg. 403, right column, Overlay visualization, second paragraph]; overlaying said one or more landmarks on said video stream, wherein the overlaying comprises displaying the one or more landmarks over the at least one object [Figure 3 & 4b; a guiding line from the pointer tip to the target structure is additionally superimposed within the endoscopic video image ... the occlusion of image by overlaid data has to be minimized in order not to constrain the surgeon. Therefore, the target region and the pointer are marked in the form of silhouettes, pg. 403, right column, Overlay visualization, third paragraph; pg. 404, Overlay visualization, left column, first paragraph]; and displaying said overlay on one or more display devices intraoperatively to be used by an operator [during an arthroscopic procedure] [Figure 4b; after successful patient image registration and endoscope calibration, virtual images were computed and superimposed on the live endoscopic video images (Fig. 4b) ... composed of a control computer (PentaGmbH, Germany), pg. 405, left column, Intraoperative evaluation, first paragraph].
Winne fails to explicitly disclose [receiving a video stream] from an arthroscopic imaging device; receiving one or more sets of coordinates of one or more landmarks; and [displaying said overlay on one or more display devices intraoperatively to be used by an operator] during an arthroscopic procedure.
However, Chabanas [receiving a video stream] from an arthroscopic imaging device [during surgery, a real arthroscope 1 produces a real arthroscopic image 2, para 0065]; receiving one or more sets of coordinates of one or more landmarks [from the anatomical landmarks, a bone reference coordinate system Rbone=(OF, XF, YF, ZF) is determined, para 0053]; and [displaying said overlay on one or more display devices intraoperatively to be used by an operator] during an arthroscopic procedure [displayed side by side to the real arthroscopic image 2, during a surgical procedure, para 0068].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne by incorporating the teachings of Chabanas with landmarks and coordinates to safely access areas during surgery, as recognized by Chabanas.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Chabanas with Winne to obtain the invention as specified in claim 1.
Regarding claim 12, which claim 1 is incorporated, Winne fails to explicitly disclose wherein said one or more sets of coordinates of said one or more landmarks is generated from one or more medical images of a subject.
However, Chabanas discloses wherein said one or more sets of coordinates of said one or more landmarks is generated from one or more medical images of a subject [from a particular bone surface S and the 3D image, it is possible to determine anatomical landmarks (points and axis) ... intra-operatively, if the patient images are registered to the patient coordinate system using one of the numerous image-guided surgery registration methods, then the characteristic elements built in Rbone are determined in the intra-operative patience coordinate system, para 0053, 0072].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne by incorporating the teachings of Chabanas determine characteristic elements intra-operatively in the patient coordinate system, as recognized by Chabanas.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Chabanas with Winne to obtain the invention as specified in claim 12.
Regarding claim 15, which claim 1 is incorporated, Winne fails to explicitly disclose wherein said radiological images are associated with a shoulder, a knee, or a hip of said subject.
However, Chabanas teaches wherein said radiological images are associated with a shoulder, a knee, or a hip of said subject [hip: Figure 3 & 4; from the patient 3D image, an external 3D surface model S of the bone showing the cortical surface or the cartilage surface is reconstructed ... on the pelvis side, the bone reference coordinate system is constructed from 4 anatomical landmarks: the pubis symphysis PS, the left and right antero-superior iliac spines ASISL and ASISR, and the center of the acetabulum sphere A. Our protocol requires the acquisition of a few CT slices at the level of the iliac spines ... surgical target areas are defined in the patient images, para 0051, 0054, 0066].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne by incorporating the teachings of Chabanas to place the instruments at precise locations and avoid dangerous areas, as recognized by Chabanas.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Chabanas with Winne to obtain the invention as specified in claim 15.
Regarding claim 41, which claim 1 is incorporated, Winne discloses further comprising an input, wherein said input comprises one or more of: a push-button, a touchscreen device, a foot pedal, a gesture recognition system, or a voice recognition system [touchscreen device (the examiner interprets the one or more modules to require only one limitation): is composed of a control computer ... and user interface modules such as a touch screen, pg. 402, left column, Materials and methods, first paragraph].
Claims 3 and 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Winne ("Overlay visualization in endoscopic ENT surgery") in view of Chabanas (US 2013/0096373 A1), as applied above, and further in view of Barral et al. (US 2019/0069957 A1) (hereafter, “Barral”).
Regarding claim 3, which claim 1 is incorporated, neither Winne nor Chabanas appear to explicitly disclose wherein said operations further comprise identifying and labeling one or more elements in said video stream using at least one of a trained computer algorithm and one or more modules, wherein said one or more elements comprise one or more of an anatomical structure, a surgical tool, an operational procedure or action, or a pathology.
However, Barral discloses wherein said operations further comprise identifying and labeling one or more elements in said video stream using at least one of a trained computer algorithm and one or more modules [processing apparatus 107 may identify anatomical features in the video using a machine learning algorithm, and generate an annotated video where the anatomical features from the video are accentuated, para 0026], wherein said one or more elements comprise one or more of an anatomical structure, a surgical tool, an operational procedure or action, or a pathology [pathology (the examiner interprets one or more elements to require only one limitation): processing apparatus 107 may identify diseased portions (e.g., tumor, lesions, etc.) and healthy portions (e.g., an organ that looks “normal” relative to a set of established standards), para 0026].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas by incorporating the teachings of Barral with landmarks and coordinates to safely access areas during surgery, as recognized by Barral.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Barral with Winne and Chabanas to obtain the invention as specified in claim 3.
Regarding claim 5, which claim 3 is incorporated, Winne discloses wherein said one or more modules comprise video stream decomposition, tool recognition, anatomy recognition, tool tracking, gesture recognition, landmark point registration, or anatomy and landmark tracking [tool tracking (the examiner interprets the one or more modules to require only one limitation): Figure 1a; an endoscope tracker was constructed and rigidly coupled to the camera module attached to the endoscope, pg. 402, left column, Materials and methods, second paragraph].
Regarding claim 6, which claim 3 is incorporated, Winne fails to explicitly disclose wherein said system recommends one or more landmarks based at least partially on said identified elements.
However, Chabanas discloses wherein said system recommends one or more landmarks based at least partially on said identified elements [it is possible to determine anatomical landmarks (points and axis) that represent critical geometrical reference elements ... such intra-operative data can be landmark points collected using a navigation system during the surgical procedure, para 0053, 0073].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne by incorporating the teachings of Chabanas by recommending landmarks to consider critical areas in the procedure, as recognized by Chabanas.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Barral with Winne and Chabanas to obtain the invention as specified in claim 6.
Claims 7-9, 31-33, 37, and 43 are rejected under 35 U.S.C. 103 as being unpatentable over Winne ("Overlay visualization in endoscopic ENT surgery") in view of Chabanas (US 2013/0096373 A1), as applied above, and further in view of Dumpe et al. (WO 2020/047051 A1) (hereafter, “Dumpe”).
Regarding claim 7, which claim 1 is incorporated, Winne fails to explicitly disclose wherein said operations further comprise: storing the one or more sets of coordinates of one or more landmarks; changing a view of said display to omit said overlaid landmark from being displayed; reverting said view to said previous display; identifying said one or more set of coordinates for said one or more landmarks; and re-overlaying said one or more landmarks.
However, Chabanas discloses wherein said operations further comprise: identifying said one or more set of coordinates for said one or more landmarks [from the anatomical landmarks, a bone reference coordinate system Rbone=(OF, XF, YF, ZF) is determined, para 0053].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne by incorporating the teachings of Chabanas with landmarks and coordinates to safely access areas during surgery, as recognized by Chabanas.
Neither Winne nor Chabanas appears to explicitly disclose wherein said operations further comprise: storing the one or more sets of coordinates of one or more landmarks; changing a view of said display to omit said overlaid landmark from being displayed; reverting said view to said previous display; identifying said one or more set of coordinates for said one or more landmarks; and re-overlaying said one or more landmarks.
However, Dumpe discloses wherein said operations further comprise: storing the one or more sets of coordinates of one or more landmarks [the CASS 200 can compare and register the location data of bony landmarks collected by the surgeon, para 0057]; changing a view of said display to omit said overlaid landmark from being displayed [as an alternative or supplement to the display 225, one or more members of the surgical staff may wear an Augmented Reality (AR) Head Mounted Device (HMD). For example, ... the surgeon 211 is wearing an AR HMD 255 that may, for example, overlay pre-operative image data on the patient, para 0059]; reverting said view to said previous display [the display 225 can also include information about the anatomy of the surgical target region including the location of landmarks ... the surgeon can cycle through these images ... instructing the system to advance or rewind the display, para 0102, 0151]; and re-overlaying said one or more landmarks [as an alternative or supplement to the display 225, one or more members of the surgical staff may wear an Augmented Reality (AR) Head Mounted Device (HMD). For example, ... the surgeon 211 is wearing an AR HMD 255 that may, for example, overlay pre-operative image data on the patient, para 0059].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas by incorporating the teachings of Dumpe to display the information in real-time, as recognized by Dumpe.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Dumpe with Winne and Chabanas to obtain the invention as specified in claim 7.
Regarding claim 8, which claim 7 is incorporated, neither Winne nor Chabanas appears to explicitly disclose wherein said operator activates said changing and said reverting steps
However, Dumpe discloses wherein said operator activates said changing and said reverting steps [the surgeon can cycle through these images, para 0151].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas by incorporating the teachings of Dumpe for the operator can see specific information as needed, as recognized by Dumpe.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Dumpe with Winne and Chabanas to obtain the invention as specified in claim 8.
Regarding claim 9, which claim 7 is incorporated, Winne discloses wherein said changing a view step is activated automatically based on a change in an identified anatomical structure or pathology [the line changes its color every millimeter to enable estimation of the distance and the depth position of the target region, pg. 403, right column, Overlay visualization, third paragraph].
Regarding claim 31, which claim 1 is incorporated, neither Winne nor Chabanas appears to explicitly disclose further comprising a camera control unit configured to control a light source, capture digital information produced by said digital camera.
However, Dumpe discloses further comprising a camera control unit configured to control a light source, capture digital information produced by said digital camera [a combination of IR and visible light cameras can be used in an array. Various illumination sources, such as an IR LED light source, can illuminate the scene, para 0053].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas by incorporating the teachings of Dumpe to illuminate the scene for imaging purposes, as recognized by Dumpe.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Dumpe with Winne and Chabanas to obtain the invention as specified in claim 31.
Regarding claim 32, which claim 31 is incorporated, neither Winne nor Chabanas appear to explicitly disclose wherein said camera control unit converts said digital information produced by said digital camera into said video stream.
However, Dumpe discloses wherein said camera control unit converts said digital information produced by said digital camera into said video stream [components may stream data to the Surgical Computer 250 in real-time or near real-time during surgery, 0107].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas by incorporating the teachings of Dumpe for real-time video during surgery, as recognized by Dumpe.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Dumpe with Winne and Chabanas to obtain the invention as specified in claim 32.
Regarding claim 33, which claim 31 is incorporated, neither Winne nor Chabanas appears to explicitly disclose wherein said camera control unit record said digital information produced by said digital camera in a memory device.
However, Dumpe discloses wherein said camera control unit record said digital information produced by said digital camera in a memory device [the powered impaction device includes feedback sensors that gather data during instrument use, and send data to a computing device such as a controller within the device or the Surgical Computer 250. This computing device can then record the data for later analysis and use, para 0072].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas by incorporating the teachings of Dumpe for future analysis, as recognized by Dumpe.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Dumpe with Winne and Chabanas to obtain the invention as specified in claim 33.
Regarding claim 37, which claim 31 is incorporated, neither Winne nor Chabanas appears to explicitly disclose wherein said camera control unit is configured to overlay said output from with said one or more computer processors with said video stream.
However, Dumpe discloses wherein said camera control unit is configured to overlay said output from with said one or more computer processors with said video stream [registration process that registers the CAS 200 to the relevant anatomy of the patient can also involve the use of anatomical landmarks, such as landmarks on a bone or cartilage ... display 225 overlays image information collected from various modalities ... collected pre-operatively or intra-operatively, para 0056, 0059].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas by incorporating the teachings of Dumpe to provide various view of patient’s anatomy in real-time, as recognized by Dumpe.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Dumpe with Winne and Chabanas to obtain the invention as specified in claim 37.
Regarding claim 43, which claim 1 is incorporated, neither Winne nor Chabanas appears to explicitly disclose wherein said one or more landmarks are tracked during said minimally invasive procedure, further wherein said tracking of one or more landmarks is associated with said set of coordinates of said one or more landmarks relative to at least an anatomical structure.
However, Dumpe discloses wherein said one or more landmarks are tracked [tracking system 620 can be calibrated to provide a localized 3D coordinate system within which the implant host 601 and one or more surgical tools can be spatially tracked, para 0182] during said minimally invasive procedure [surgical navigation systems such as the CASS 200 often employ various forms of computing technology to perform a wide variety of standard and minimally invasive surgical procedures and techniques, para 0047], further wherein said tracking of one or more landmarks is associated with said set of coordinates of said one or more landmarks [tracking system 620 can be calibrated to provide a localized 3D coordinate system, para 0182] relative to at least an anatomical structure [track and navigate the placement of instruments and implants relative to the body of a patient, para 0047].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas by incorporating the teachings of Dumpe to accurately determine location and orientation, as recognized by Dumpe.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Dumpe with Winne and Chabanas to obtain the invention as specified in claim 43.
Claims 17 is rejected under 35 U.S.C. 103 as being unpatentable over Winne ("Overlay visualization in endoscopic ENT surgery") in view of Chabanas (US 2013/0096373 A1), as applied above, and further in view of Murphy et al. (US 2013/0211232 A1) (hereafter, “Murphy”).
Regarding claim 17, which claim 1 is incorporated, neither Winne nor Chabanas appears to explicitly disclose wherein said video stream is provided by an arthroscope during an arthroscopic procedure.
However, Murphy discloses wherein said video stream is provided by an arthroscope during an arthroscopic procedure [during registration and surgery, real time images of a surgical site are obtained ... the arthroscope may include a video camera to capture real-time video images of the surgical site, para 0020].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas by incorporating the teachings of Murphy to provide image guidance to the surgeon during the procedure and provide accurate visualization with respect to bone structures, as recognized by Murphy.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Murphy with Winne and Chabanas to obtain the invention as specified in claim 17.
Claims 25 and 27-28 are rejected under 35 U.S.C. 103 as being unpatentable over Winne ("Overlay visualization in endoscopic ENT surgery") in view of Chabanas (US 2013/0096373 A1), as applied above, and further in view of Charron et al. (US 10,543,046 B2) (hereafter, “Charron”).
Regarding claim 25, which claim 1 is incorporated, neither Winne nor Chabanas appear to explicitly disclose wherein said one or more computer processors receive said video stream from one or more camera control units using a wired media connection.
However, Charron discloses wherein said one or more computer processors receive said video stream from one or more camera control units using a wired media connection [the video capture device 706 may be a camera ... that captures digital video data in real-time and has an interface to transmit the video to a computer processor 705 in real-time ... the electronic communication may be facilitated by, for example, one or more busses, or other wired or wireless connections, Col 4, line 17; Col 9, line 46-48].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas by incorporating the teachings of Charron for real-time transmission, as recognized by Charron.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Charron with Winne and Chabanas to obtain the invention as specified in claim 25.
Regarding claim 27, which claim 25 is incorporated, neither Winne nor Chabanas appears to explicitly disclose wherein a latency between receiving said video stream from the one or more camera control units and overlaying said output and said video stream is no more than a time between two consecutive frames from said digital camera.
However, Charron discloses wherein a latency between receiving said video stream from the one or more camera control units and overlaying said output and said video stream is no more than a time between two consecutive frames from said digital camera [the video capture device 706 has a low latency analog to digital converter that converts the image viewed through the lens of the video capture device 706 to frames of digital video in real-time. The digital video data is transmitted in real-time via an electronic interface to the computer processor 705 with low latency, Col 5, line 6-11].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas by incorporating the teachings of Charron because for real-time transmission with low latency, as recognized by Charron.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Charron with Winne and Chabanas to obtain the invention as specified in claim 27.
Regarding claim 28, which claim 1 is incorporated, neither Winne nor Chabanas appear to explicitly disclose wherein said one or more computer processors receive said video stream from one or more camera control units using a network connection.
However, Charron discloses wherein said one or more computer processors receive said video stream from one or more camera control units using a network connection [the video capture device 706 may be a camera ... that captures digital video data in real-time and has an interface to transmit the video to a computer processor 705 in real-time ... the electronic communication may be facilitated by, for example, one or more busses, or other wired or wireless connections ... the processors may be ... loosely coupled, e.g., by being connected by a wide-area network, Col 4, line 17-20; Col 9, line 46-51].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas by incorporating the teachings of Charron for real-time transmission, as recognized by Charron.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Charron with Winne and Chabanas to obtain the invention as specified in claim 28.
Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over Winne ("Overlay visualization in endoscopic ENT surgery") in view of Chabanas (US 2013/0096373 A1) and further in view of Charron (US 10,543,046 B2), as applied above, and Miller et al. (US 10,806,325 B2) (hereafter, “Miller”).
Regarding claim 26, which claim 25 is incorporated, neither Winne nor Chabanas appears to explicitly disclose wherein a latency between receiving said video stream from the one or more camera control units and overlaying said output and said video stream is at most 40 milliseconds (ms) to accommodate a digital camera with about 24 frames per second (fps).
However, Charron discloses wherein a latency between receiving said video stream from the one or more camera control units and overlaying said output and said video stream is at most 40 milliseconds (ms) [each processed image frame may be produced and displayed in preferably less than 0.1 seconds, or more preferably less than 0.05 seconds or 0.02 seconds, after the corresponding image frame was captured by the video capture device 706, Col 5, line 12-16].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas by incorporating the teachings of Charron for real-time transmission, as recognized by Charron.
Neither Winne nor Chabanas nor Charron appear to explicitly disclose to accommodate a digital camera with about 24 frames per second (fps).
However, Miller discloses to accommodate a digital camera with about 24 frames per second (fps) [for minimally-invasive surgical procedures a 15 fps rate may provide adequate fidelity while 20-30 fps rate or higher may provide an optimal fidelity, Col 14, line 61-64].
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify Winne in view of Chabanas and further in view of Charron by incorporating the teachings of Miller for optimal fidelity, as recognized by Miller.
Further, one skilled in the art could have combined the elements as described above with known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Miller with Winne and Chabanas and Charron to obtain the invention as specified in claim 26.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 2010/0249507 A1 to Prisco et al. discloses landmark directional guidance for an endoscopic device.
Interactive Guidance by Image Overlay in Robot Assisted Coronary Artery Bypass to Mourgues et al. discloses a guidance system with an overlay on endoscopic images.
US 2012/0226150 A1 to Balicki et al. discloses a visual tracking and annotation system for surgical intervention.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TOLUWANI MARY-JANE IJASEUN whose telephone number is (571)270-1877. The examiner can normally be reached Monday - Friday 7:30AM-4PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached on (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TOLUWANI MARY-JANE IJASEUN/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676
Prosecution Insights