DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 07/10/2026 has been entered.
Response to Arguments
Applicant’s arguments with respect to claims 1, 4-15, 17-18 have been considered but are moot as discussed in new ground of rejection below.
Claims 2-3, 16 have been canceled.
Applicant argues that no new matter is added (see page 7 in the amendment filed 07/10/2026 and Applicant’s remarks on page 8 of the amendment filed 04/02/2026). This argument is respectfully traversed.
Regarding the negative limitation “….without an external sensor” previously added and currently cited in claim 1 (lines 24-25) and claim 11 (lines 14-15), the originally-filed specification does not identify any "descri[ption of] a reason to exclude the relevant limitation." Santarus, Inc. v. Par Pharmaceutical, Inc., 694 F.3d 1344, 1351 (Fed. Cir. 2012) ("Negative claim limitations are adequately supported when the specification describes a reason to exclude the relevant limitation. Such written description support need not rise to the level of disclaimer. In fact, it is possible for the patentee to support both the inclusion and exclusion of the same material."). See also MPEP § 2173.05(i) ("Any negative limitation or exclusionary proviso must have basis in the original disclosure ....The mere absence of a positive recitation is not basis for an exclusion."). See also page 5 in the Patent Board Decision issued in application No. 12/870,150.
Applicant argues Sauer does not teach or suggest an initial registration performed based on “detecting, in the intracorporeal image, an anatomical landmark and/or an anatomical orientation and to determine a corresponding anatomical landmark and/or a corresponding anatomical orientation in the 3D image data” because Sauer describes registration of an instrument to the patient using “various methods of three-dimensional texture mapping and image recognition” but the only concrete example of an initial registration is the registration of a physical “tracked instrument” (page 8). This argument is respectfully traversed.
MPEP 2111.04 states “Claim scope is not limited by claim language that suggests or makes optional but does not require steps to be performed, or by claim language that does not limit a claim to a particular structure. However, examples of claim language, although not exhaustive, that may raise a question as to the limiting effect of the language in a claim are:
(A) "adapted to" or "adapted for" clauses;”
Claim 1 recites “a registration unit adapted to detect, in the intracorporeal image, an anatomical landmark and/or an anatomical orientation and to determine a corresponding anatomical landmark and/or a corresponding anatomical orientation in the 3D image data, and to register the intracorporeal image with the 3D image data, at least initially, and to register the endoscope relative to the patient;”
According to MPEP 2111.04, the limitation “….detect, in the intracorporeal image, an anatomical landmark and/or an anatomical orientation and to determine a corresponding anatomical landmark and/or a corresponding anatomical orientation in the 3D image data, and to register the intracorporeal image with the 3D image data, at least initially, and to register the endoscope relative to the patient” recited in claim 1 or any limitation after “adapted to” are “language that suggests or makes optional but does not require the steps to be performed, or does not limit the claim to a particular structure”.
In addition, Sauer discloses the live image from the endoscope can be mapped onto the surface structures rendered in the virtual scene. The resulting image would consist of a live image overlaid directly onto the virtual image in order to provide an integrated three-dimensional view of the local area of the endoscope (paragraph 0018). Sauer also discloses a three-dimensional (3D) model of a patient’s anatomy is received and linking, synchronizing the images from 3D model and images from endoscope, and combining the images together (paragraph 0017). The image models (3D models) are correctly registered to their counterparts (e.g., anatomic models correspond closed to the real anatomy of the patient and instruments models are correctly registered to the real instrument)… (see paragraphs 0022-0023). Thus, Sauer obviously discloses “ “a registration unit” (part of computer 103 – figure 1) adapted to detect, in the intracorporeal image, an anatomical landmark and/or an anatomical orientation and to determine a corresponding anatomical landmark and/or a corresponding anatomical orientation in the 3D image data, and to register the intracorporeal image with the 3D image data, at least initially, and to register the endoscope relative to the patient” (read on part of computer 103 adapted to, at least initially/starting, to detect in the images from the endoscope, landmarks/information/marker of in image of patient’s internal anatomy or anatomical orientation corresponding to the information/landmarks/markers or anatomic orientation in the 3D image to register/link/map the image of endoscope with the 3D image data as discussed in the final rejection, page 3 discussed above and see also figures 1, 3, paragraphs 0017-0018, 0022-0024).
In addition to Sauer and to provide a clearly support including “at least initially register”, newly cited Inglis reference also disclose the teaching of “a registration unit adapted to detect, in the intracorporeal image, an anatomical landmark and/or an anatomical orientation and to determine a corresponding anatomical landmark and/or a corresponding anatomical orientation in the 3D image data, and to register the intracorporeal image with the 3D image data, at least initially, and to register the endoscope relative to the patient” (-see include, but are not limited to, the disclosure in figures 1, 2A-2B, 7-9, paragraphs 0008, 0039-0040, 0049-0051, 0053).
Applicant further argues Plewe does not teach initial registration step performed as recited in claim 1. Instead, Plewe teaches using “EM” tracking to register the instrument to the patient’s anatomy, the EM field generators which must be located outside the patient. In contrast to the combined art, claim 1 combined an internal image-based initial registration with an internal tracking system, which allows a completely internal surgical assistance system. A person skilled in the art would not arrive at the claimed invention by combining Sauer and Plew, because the registration described in Sauer is fundamentally tied to a global coordinate system established by external tracking hardware (e.g., external magnetic or optical sensors), and this strict reliance on an external tracking infrastructure for the initial registration is not replaced by the teachings of Plewe, which the Office Action relies upon solely for the subsequent continuous tracking aspect and not initial registration (page 9). This argument is respectfully traversed.
In response to Applicant's arguments against the references individually (i.e., Plew does not teach an initial registration step performed as recited in claim 1), one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). In this case, the rejection does not rely on Plewe for teaching of “initial registration”. Instead, the teaching of “initial registration” or “at least initially” is already taught by Sauer reference as discussed in the final rejection and/or above. Plewe is relied on for the teaching of control unit is adapted to determine movement of the endoscope via an image analysis of moving anatomical structure in the intracorporeal image…(see the final rejection, page 4).
With respect to Applicant’s argument that Sauer is fundamentally tied a global coordinate system established by external tracking hardware and this strict reliance on an external tracking infrastructure for the initial registration, Examiner notices that Sauer discloses images from the endoscope can also be collected and can then be stored in computer 103. By linking and synchronizing the images and then combining some or all of the data types together, as well as data related to the location of the surgical instruments, a physician can select different views and different perspective in order to guide the performance of surgical tasks and, in more particularly, the position of an endoscope or other surgical instrument while inside a patient’s body (paragraph 0017). The physician has both a global view of the patient, a graphical presentation within that global view of the precise location and view of the endoscope, as well as a live optical local view of a patient’s internal anatomy from the endoscope. Endoscope 304 and medical instrument 307 are inserted through incision to position 306. Various tracking methods are used to present a graphical image of the endoscope 304 and instrument 307 (inserted inside the patient’s body (see paragraph 0018). In order to track devices such as flexible endoscopes and medical devices that are inserted into the patient’s body, magnetic tracking systems may be preferable. By generating a magnetic field and detecting the location of the sensor coil, the position and orientation of objects inside a patient’s body can be determined (paragraphs 0021, 0023).
Since discloses endoscopes and instruments are inserted into patient’s body and images from endoscope and/or of the instrument are images inside patient’s body/patient’s internal anatomy, Sauser does not tied on only external tracking hardware. In fact, Saure discloses tracking images of endoscope and/or instrument inside patient’s body.
In addition, discussing the question of obviousness of claimed subject matter involving a combination of known elements, KSR Int'l v. Teleflex, Inc., 127 S. Ct. 1727 (2007), explains: When a work is available in one field of endeavor, design incentives and other market forces can prompt variations of it, either in the same field or a different one. If a person of ordinary skill can implement a predictable variation, § 103 likely bars its patentability. For the same reason, if a technique has been used to improve one device, and a person of ordinary skill in the art would recognize that it would improve similar devices in the same way, using the technique is obvious unless its actual application is beyond his or her skill. Sakraida [v. AG Pro, Inc., 425 U.S. 273 (1976)] and Anderson ’s-Black Rock, Inc. v. Pavement Salvage Co., 396 U.S. 57, 163 USPQ
673 (1969). In this case, all claim limitations are known ben prior art and benefit/motivation for combination of the teaching is provided. Therefore, it would have been obvious to one of ordinary skill to combine the teachings in Plewe with Sauer as discussed in the rejection.
With respect to claim 11, Applicant argues claim 11 corresponds in substance to the language of claim 1 discussed above and therefore is allowable (pages 9-10, bridge paragraph), Examiner’s response to Applicant’s argument is similar to Examiner’s response in claim 1 above.
Applicant further argues new claim 18 depends from claim 1 and recites determining the endoscope movement data based on weighted values of movements detected by the inertial measurement unit and the image analysis. The prior art does not teach or suggest claim 18. Thus, Applicant respectfully request favorable consideration and allowance of claim 18 (see page 10).
In response, Examiner notices that the teaching of determining endoscope movement data and translation data detected by inertial measurement unit is well-known in the art (see for example, Draxinger et al. (US 2007, 0060792: paragraph 0060). It would have been obvious to one of ordinary skill in the art to try weighting rotation components, translation components, complementarily weighting rotation components in result of the image analysis with any percentage/ratios such as more than 80%, less then 20% as recited in claim 18 to achieve a desired choice or desired number.
For the reasons given above, rejections of claims 1, 4-15, 17-18 are discussed below.
Examiner’s note
According to MPEP 2111.04, I. “Claim scope is not limited by claim language that suggests or makes optional but does not require steps to be performed, or by claim language that does not limit a claim to a particular structure. However, examples of claim language, although not exhaustive, that may raise a question as to the limiting effect of the language in a claim are:[AltContent: rect]
(A) "adapted to" or "adapted for" clauses;
(B) "wherein" clauses; and
(C) "whereby" clauses.”
Thus, the limitation cited in the claims following “adapted to” could be considered as language that suggests or makes optional but does not require steps to be performed or does not limit a claim to a particular structure.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 1, 4-15, 17-18 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Independent claims 1 and 11 recite limitation “…. via an image analysis…without an external tracking sensor.” which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention (see also discussion in “response to arguments” above).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, and 6-15 are rejected under 35 U.S.C. 103 as being unpatentable over Sauer et al. (US 2006/0281971, hereinafter “Sauer”) in view of Plewe et al. (US 2021/0196312, hereinafter “Plewe”), and further in view of Inglis et al. (US 20210137350).
As to claim 1, Sauer discloses a surgical assistance system for use in a surgical intervention on a patient, the surgical assistance system comprising:
a display device (e.g. display 301, Fig.3) for displaying a visual content;
an endoscope (endoscope 108, Fig.1) comprising an imaging recording head (image forming distal end, [0004]) adapted to create an intracorporeal image of the patient (“images from the endoscope…can be stored”, [0017]);
a data-providing unit (part of computer 103, Fig.1) adapted to provide digital 3D image data of the patient that is digital (three dimensional preoperative CT/MRI images, [0008], digitally stored on computer 103, [0017]);
a control unit (part of computer 103, Fig.1) adapted to process the intracorporeal image and the 3D image data (both endoscope image and preoperative image processed to be combined and displayed, [0018]);
a registration unit (part of computer 103, Fig.1) adapted to detect, in the intracorporeal image, at least one an anatomical landmark and/or an anatomical orientation and to determine a corresponding anatomical landmark and/or a corresponding anatomical orientation in the 3D image data, and to register the intracorporeal image with the 3D image data, and to register the endoscope relative to the patient (computer 103 registers the endoscope image with the preoperative images, [0022], using image recognition techniques involving comparing and matching coordinates of anatomical landmarks in both images, [0023]) and see also discussion in “response to arguments” above; and
a tracking system (optical or sensor tracking arrangements, [0020]-[0021]) adapted to continuously detect a position, an orientation and a movement of the endoscope, and to provide the control unit with endoscope movement data (computer collects the position and orientation data of the endoscope, [0022],[0028]);
the control unit (part of computer 103) being further adapted to generate a correlation display with at least a display of the intracorporeal image and at least a display of a view of the 3D image data (computer generates a display as shown in Fig.3, with a display of the live endoscope view 311 and a display of the preoperative image 303), in which the intracorporeal image and the view of the 3D image data are correlated with respect to the endoscope by transferring the endoscope movement data to at least one virtual position and/or orientation of a virtual recording head in the view of the 3D image data for a correlated movement (positional information and movement of the endoscope and instruments are represented by the positions and movement of graphical representations 304,307 with respect to preoperative image, Fig.3, [0016],[0018]), and the control unit is adapted to visually output the correlation display on the display device (as illustrated in Fig.3).
Sauer, as set forth above, discloses, as the tracking system, a marker based optical tracking system or a magnetic based tracking system, which includes embedded magnetic sensor coils ([0021]) as an endoscope-internal movement tracking system, and thus fails to disclose an image based optical tracking system, wherein the control unit is adapted to determine the movement of the endoscope via an image analysis of moving anatomical structures in the intracorporeal image, without an external tracking sensor, the endoscope-internal movement sensor comprising an inertial measurement unit in the endoscope. However, Plewe teaches, in a similar surgical assistance system, that image-based techniques, wherein endoscope movement is determined with image analysis of moving anatomical structure (e.g. optical flow, motion detection, etc.) can be used in addition to or as an alternative to optical or magnetic tracking without external tracking sensor ([0165]-[0167],[0170]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have used image-based techniques (e.g. optical flow, motion detection, etc.) as an equivalent alternative means for tracking the position/orientation/movement of the endoscope instead of optical or magnetic tracking, as taught by Plewe. Use of such image-based tracking system as an alternative would not require an “external tracking sensor”.
Additionally and/or alternatively, Inglis discloses a registration unit adapted to detect, in intracorporeal image, an anatomical landmark and/or an anatomical orientation and to determine a corresponding anatomical landmark and/or a corresponding anatomical orientation in 3D image data, and to register the intracorporeal image with the 3D image data, at least initially, and to register the endoscope relative to patient (processor/controller is adapted to detect in the image (34) from the endoscope inside the patient, landmarks/orientation/feature corresponding to the landmarks/orientation/feature in three-dimensional anatomical model 38 (hereinafter referred to as 3D image data), and register/map/pin/mix the image from the endoscope with the 3D image data in model 38, at least initially or at time prior to mapping/mixing, etc., and register/pin the endoscope to the patient – see include, but are not limited to, in figures 1, 2A-2B, 7-9, paragraphs 0008, 0039-0040, 0049-0051, 0053);
a tracking system adapted to continuously detect a position, an orientation, and a movement of the endoscope, and provide the control unit with endoscope movement data (tracking system with sensors adapted to continuously detect/track the position, an orientation, movement of the endoscope 12 in the patient and provide the control unit (processor/controller) with endoscope movement data – see include, but are not limited to, figures 2A-2B, 3-5, 7, paragraphs 0031, 0039, 0051-0052);
the tracking system is configured as an endoscope-internal tracking system and the endoscope has an endoscope-internal movement sensor (e.g., orientation sensor 756 – figure 7) comprising an inertial measurement unit (IMU) in the endoscope (712) for detecting the movement of the endoscope (see include, but are not limited to, figure 7, paragraphs 0078-0079), and
for the tracking system, the control unit adapted to determine the movement of the endoscope via an image analysis of moving anatomical structures in the intracorporeal image without an external tracking sensor (for tracking system, the control unit/processor adapted to determine movement of the endoscope (712) via an image analysis of moving anatomical structures in the image (34) from the endoscope inside the patient’s body and not using external tracking sensor – see include, but are not limited to, figures 2A-3, 7, paragraphs 0051, 0078-0080).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify Sauer with the teachings including a tracking system is configured as an endoscope-internal tracking system and the endoscope has an endoscope-internal movement sensor comprising an inertial measurement unit (IMU) in the endoscope for detecting the movement of the endoscope as taught by Inglis in order to yield predictable result of accurately detecting static orientation, dynamic movement, orientation of the distal tip of endoscope (paragraph 0079).
As to claim 4, Sauer in view of Plewe and Inglish discloses the surgical assistance system according to claim 1. Sauer further discloses the tracking system comprises an endoscope-internal movement sensor provided in the endoscope and adapted to determine the movement of the endoscope as a second movement determination (embedded magnetic sensor coils, [0021] as part of the magnetic tracking system), and thus fails to disclose: 1) that the endoscope-internal movement sensor comprises an acceleration sensor and/or a rotation rate sensor; 2) that, as an additional tracking system, the control unit is adapted to determine the movement of the endoscope via an image analysis of moving anatomical structures in the intracorporeal image as a first movement determination; and 3) that the control unit is adapted to calculate a final movement determination based on an adjusted mean value or a weighted value calculated from first movement determination and the second movement determination.
However, Plewe teaches, in a similar surgical assistance system, that 1) an acceleration sensor or gyroscope (rotation rate sensor) (both of which constitute an inertial measurement unit) can be used to determine position/orientation/movement of the endoscope ([0169]) as an alternative to a magnetic tracking system ([0167]); 2) image-based techniques, wherein endoscope movement is determined with image analysis of moving anatomical structure (e.g. feature tracking, optical flow, motion detection, etc.) can be used in addition to or as an alternative to optical or magnetic tracking ([0165]-[0167],[0170]); and 3) the control unit can use a combination of input data (i.e. first movement determination from the image-based tracking and second determination data from the magnetic/acceleration sensors), wherein a final movement determination is based on an adjusted mean or weighted combination of tracking data ([0170]).
See also disclosures in Inglis (paragraphs 0046, 0051, 0079-0081).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have substituted an acceleration sensor or gyroscope for the magnetic tracking sensor of Sauer since they are known in the art to be equivalent alternative means for determining endoscope position/orientation/movement, as taught by Plewe and/or Inglis. Furthermore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have used more than one tracking system (e.g. additionally used an image-based tracking system) and determined a final movement using the combined tracking data in a weighted fashion to provide a more reliable determined result (Plewe: [0170]).
As to claim 6, Sauer in view of Plewe and Inglis discloses the surgical assistance system according to claim 1, wherein the registration unit performs a re-registration beyond an initial registration at at least one further point in time in order to further increase an accuracy of a correlation (updates the registration/mapping, see for example, Saure: [0023]; Inglis: paragraphs 0049-0051, 0053-0054).
As to claim 7, Sauer in view of Plewe and Inglis discloses the surgical assistance system according to claim 1. Sauer also discloses manual manipulation of the endoscope/instruments by the surgeon and thus fails to disclose that the endoscope is robot-guided via a robot arm and the tracking system determines the position and/or the orientation and/or the movement of the endoscope via an associated position and/or orientation and/or movement of the robot arm and thus via the robot. However, Plewe teaches in the art of surgical assistance devices, to use a robotic system including a robot-guided arm to advance and navigate the endoscope into the patient ([0045]-[0046]). Plewe further teaches that when a robotic arm is used, as the tracking system, robotic command and/or kinematics data can be used to determine the position/orientation/movement of the endoscope ([0168]). Given the teachings of Plewe, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have provided a robotic system including a robot arm for guiding the endoscope to a target site for more controlled and precise navigation of the endoscope over manual navigation. Furthermore, when using a robotic system for guidance, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have use, as the tracking system, robotic command and/or kinematic data from the robot arm to determine the position/orientation/movement of the endoscope as an alternative equivalent means for tracking the endoscope (note that Plewe teaches such robotic tracking as an alternative to using a magnetic tracking system, [0167]-[0168]).
As to claim 8, Sauer in view of Plewe and Inglis discloses the surgical assistance system according to claim 1, wherein the control unit is adapted to display in the correlation display in the display of the view of the 3D image data and/or of the intracorporeal image, in addition to the endoscope, pre-planned medical instruments and/or implants stored in a storage unit in a correct position in the correlation display (computer configured to display a stored (preplanned) graphical representation of the surgical instrument 307, Fig.3, in its tracked (correct) relative position with respect to the endoscope and preoperative image, see Sauer: [0016]-[0018], Fig.3. See also: Inglis: figures 1-4, 7).
As to claim 9, Sauer in view of Plewe and Inglis discloses the surgical assistance system according to claim 1, wherein the control unit is adapted to display a planned path and/or annotations in the correlation display in order to guide a surgeon to a surgical field (computer is configured to display a optical axis 309 and view angle cone 310, which either can constitute a planned path and/or annotations, for guiding either of the endoscope 304 or surgical instrument 307 to the surgical field (within cone 310), - see include, but are not limited to, Sauer: Fig.3, [0018]; Inglis: figures 2A-5).
As to claim 10, Sauer in view of Plewe and Inglis discloses the surgical assistance system according to claim 1, wherein the control unit is adapted to generate the view of the 3D image data, as: a 3D scene (three dimensional model, [0016]); or two-dimensional cross-sections relative to a picture-coordinate system of the endoscope and/or along an image axis of the endoscope (display of a 3D image data on a 2D display screen, see include, but are not limited to, Sauer: Fig.3, is essentially a 2D cross-section); or a virtual endoscope image (virtual global view, Sauer: [0018]) or see Inglis: figures 2a-5.
As to claim 11, Sauer discloses an image display method for a central, correlated display of two different images, the image display method comprising the steps of:
reading in 3D image data of a patient (three dimensional preoperative CT/MRI image, [0008]);
creating or detecting an intracorporeal image through an endoscope (images from endoscope are collected, [0017]);
detecting at least one landmark and/or anatomical orientation in the intracorporeal image (computer registers the endoscope image with the preoperative image using image recognition techniques, [0022], involving detecting coordinates of anatomical landmarks from the intracorporeal image, [0023]);
determining at least one corresponding landmark and/or at least one corresponding orientation in the 3D image data (relevant anatomical structures may be identified in the preoperative image, [0023];
registering the 3D image data to the intracorporeal image via the at least one detected corresponding landmark and/or the at least one corresponding orientation (registration by matching the landmark coordinates in both images, [0022],[0023] and discussion in claim 1 and “response to arguments”);
generating a correlation display with at least one display of the intracorporeal image and at least one display of a view of the 3D image data (computer generates a display as shown in Fig.3, with a display of the live endoscope view 311 and a display of the preoperative image 303);
continuously detecting a position and an orientation and a movement of the endoscope (computer collects the position and orientation and movement data of the endoscope, [0022],[0028]);
transferring a detected movement of the endoscope to at least one virtual position and/or at least one virtual orientation of a virtual recording head in the view of the 3D image data (positional information and movement of the endoscope and instruments are represented by the positions and movement of graphical representations 304,307 with respect to preoperative image, Fig.3, [0016],[0018]); and
generating at least one updated correlation display with the intracorporeal image and an updated view of the 3D image data and outputting the at least one updated correlation display to the display device (images and views are updated in real time, [0028]).
Sauer, as set forth above, discloses, as the tracking system, a marker based optical tracking system or a magnetic based tracking system, which includes embedded magnetic sensor coils ([0021]) as an endoscope-internal movement tracking system, and thus fails to disclose an image based optical tracking system, wherein the control unit is adapted to determine the movement of the endoscope via an image analysis of moving anatomical structures in the intracorporeal image, without an external tracking sensor. However, Plewe teaches, in a similar surgical assistance system, that image-based techniques, wherein endoscope movement is determined with image analysis of moving anatomical structure (e.g. optical flow, motion detection, etc.) can be used in addition to or as an alternative to optical or magnetic tracking ([0165]-[0167],[0170]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have used image-based techniques (e.g. optical flow, motion detection, etc.) as an equivalent alternative means for tracking the position/orientation/movement of the endoscope instead of optical or magnetic tracking, as taught by Plewe. Use of such image-based tracking system as an alternative would not require an “external tracking sensor”.
Additionally and/or alternatively, Inglis discloses creating or detecting an intracorporeal image through an endoscope (creating or detecting an image (34) from an endoscope – see figures 2A-2B, paragraphs 0031, 0052);
detecting at least one landmark and/or anatomical orientation in the incorporeal image (detecting at least one landmark and/or anatomical orientation in the image from the endoscope – see for example, paragraph 0052);
determining at least one corresponding landmark and/or at least one corresponding orientation in the 3D image data (determining at least one landmark/feature or orientation in the three-dimensional anatomical model 38 – see include, but are not limited to, figures 2A-4, paragraphs 0037, 0039-0040, 0049-0053);
registering the 3D image data to the intracorporeal image, at least initially, via at least one corresponding landmark and/or the at least one corresponding orientation (registering/mapping/projecting/mixing at the three-dimensional image data to the image from the endoscope inside the patient, at least initially or at time prior to mapping/mixing, via at least one corresponding landmark in the image (34) from the endoscope and/or the at least one corresponding orientation – see include, but are not limited to, in figures 1, 2A-2B, 7-9, paragraphs 0008, 0039-0040, 0049-0051, 0053);
generating a correlation display with at least one display of the intracorporeal image (34) and at least one display of a view of the 3D image data (38) – see include, but are not limited to, figures 2A-2B, 3, paragraphs 0039, 0049-0050)
continuously detecting a position, an orientation, and a movement of the endoscope by a tracking system that is configured as an endoscope tracking system and by an internal movement sensor, and via an image analysis of moving anatomical structures in the intracorporeal image within an external tracking sensor, – see include, but are not limited to, figures 2A-2B, 3-5, 7, paragraphs 0031, 0039, 0051-0052, 0078-0079 and discussion in the rejection of claim 1 above);
transferring a detected movement of the endoscope to at least one virtual position and generating at least one updated correlation display with the intracorporeal image and updated view of the 3D image data and outputting the at least one updated correlation display to the display device – tracking movement of tip of endoscope and update the movement and/or orientation and outputting the update with 3D image data of model 38 for displaying to operator– see include, but are not limited to, figures 2A-3, 8, paragraphs 0051, 0054-0056, 0078-0080).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify Sauer with the teachings including registering 3D image data to intracorporeal image, at least initially, via the at least one corresponding landmark and/or corresponding orientation, the endoscope has an endoscope-internal movement sensor as taught by Inglis in order to yield predictable result of accurately detecting static orientation, dynamic movement, orientation of the distal tip of endoscope (paragraph 0079).
As to claim 12, Sauer in view of Plewe and Inglis discloses the method according to claim 11. Sauer further discloses the steps of: determining a deformation of a real anatomical structure based on at least one landmark and the at least one corresponding landmark, and transferring the deformation to a virtual anatomical structure in the 3D image data in order to adjust and correct the 3D image data (if an anatomical structure is removed (deformation), the virtual model image (preoperative) can be updated to reflect this deformation, [0024]). See also Inglis: figure 2a-4, paragraphs 0052-0053).
As to claim 13, Sauer in view of Plewe and Inglis discloses a non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method steps of claim 11 (see description of claim 11 above; all steps are performed by programming on a computer, Sauer: [0017],[0018]; Inglis: figure 7).
As to claim 14, Sauer in view of Plewe and Inglis discloses a medical sterile space (operating room, see Sauer: Fig.1, [0006],[0017],[0028]; Inglis: figure 1) comprising a surgical the surgical assistance system according to claim 1 (the surgical assistance system 100, as set forth with respect to claim 1 above, is used in the operating room, see Sauer: Fig.1; Inglis: figure 1).
As to claim 15, Sauer in view of Plewe and Inglis discloses the method according to claim 11, wherein a handling portion and/or the imaging recording head of the endoscope comprises the endoscope-internal movement sensor (embedded sensor coils, as set forth above; see also Inglis: figures 2a-2b, 7 paragraphs 0008, 0059, 0078-0079).
Claims 5 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Sauer et al. (US 2006/0281971, hereinafter “Sauer”) and Plewe et al. (US 2021/0196312, hereinafter “Plewe”), Inglis (US 20210137350) as applied to claim 1 above, and further in view of Garbey et al. (US 2014/0171787, hereinafter “Garbey”).
As to claims 5 and 17, Sauer discloses, as set forth above with respect to claim 1, that the control unit (computer 103) displays the at least one medical instrument in the view of the 3D image data in the correct position (Fig.3, displays instrument 307 correctly positioned with respect to the preoperative image based on tracking system data, [0018]), so that the at least one medical instrument is displayed in the intracorporeal image and in the view of the 3D image data (Fig.3, 307 shown in both views). However, Sauer uses either an optical or magnetic tracking system ([0020],[0021]), or other alternative means of tracking as taught by Plewe ([0165]-[0170]) to determine the position of the instrument and thus fails to disclose that the control unit is adapted to recognize at least one medical instrument in the intracorporeal image in a correct position based on a database stored in the storage unit with geometrically predefined structures of medical instruments. Garbey teaches, in the surgical assistance system art, that a surgical instrument can be tracked by recognizing the position and orientation of an instrument in an image based on stored geometrical features associated with the instrument ([0054]-[0056]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have tracked the instrument using the image, based on predefined stored instrument features, as taught by Garbey, as an equivalent alternative to the tracking system of Sauer/Plewe. One would be motivated to use the image tracking in Sauer to eliminate the need for additional structures (physical sensors, sensor equipment) when an image of the instrument, which is already being obtained, could be used to track it.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Sauer et al. (US 2006/0281971, hereinafter “Sauer”) and Plewe et al. (US 2021/0196312, hereinafter “Plewe”), Inglis (US 20210137350) as applied to claim 1 above, and further in view of Draxinger et al. (US 20070060792).
Regarding claim 18, Sauer in view of Plewe and Inglis discloses the surgical assistance system according to claim 1, wherein the control unit is adapted to calculate a combined value from the movement of the endoscope detected by the inertial measurement unit and the movement of the endoscope determined via the image analysis as the endoscope movement data (processor/controller is adapted to calculate or track or determine a combined value/information from movement of the endoscope detected by the IMU and movement of the endoscope determine via image analysis of movement/position, etc., obtained from the combination of sensors- the see include, but are not limited to, Inglis: figures 2A-4, paragraphs 0079-0080). However, Sauer in view of Plewe and Inglis does not explicitly disclose calculating a combined value by weighting rotation components in a result of the inertial measurement unit with more than 80%, translation components in the result of the inertial measurement unit with less than 20%, and by complementarily weighting rotation components in a result of the image analysis with less than 20%, and translation components in the result of the image analysis with more than 80%.
Draxinger discloses control unit is adapted to calculate a combined value from movement of endoscope detected by inertial measurement unit by weighting rotation components and translation components in a result the inertial measurement unit (inertial sensor system 32 acquires the position and orientation of the endoscope by virtue of the fact that it has sensors which detect the rotations and translations in space, the inertial sensor system 32 being capable of detecting all the six translation and rotational degrees of freedom of the endoscope – see paragraph 0060).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Sauer in view of Plewe and Inglis with the teaching of weighting rotation component and translation components in a result of the inertial measurement unit as taught by Draxinger in order to yield predictable result of improving accuracy for tracking the movement of the endoscope.
Draxinger does not explicitly disclose specific weights with more than 80% by rotation components, less than 20% by translation components, less than 20% by complementarily weighting rotation. It would have been obvious to one of ordinary skill to try any desired number, value, ratio, or percentage by weight rotation components, translation components, complementarily weighting rotation such as weighting rotation components in a result of the inertial measurement unit with more than 80%, translation components in the result of the inertial measurement unit with less than 20%, and by complementarily weighting rotation components in a result of the image analysis with less than 20%, and translation components in the result of the image analysis with more than 80% as recited in the claim in order to achieve a desired choice.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ohnishi et al. (US 7641609) discloses endoscope device and navigation method for endoscope device for displaying and registering/mapping both information of image of endoscope and 3D image data (see for example, figures 9, 12-23).
Nishide et al. (US 20220198742) discloses processor for endoscope, program, information processing method and information processing system for displaying endoscopic image, coordinates of the intracorporeal site on the three-dimensional medical image (see include, paragraph 0260).
Prior et al. (US 20220047259) discloses preoperative or intraoperative 3D images may be registered with the endoscopic images and displaying markings – see paragraph 0061).
Gliner et al. (US 20210186620) discloses 2D pathfinder visualization (see also para. 0010).
Tata et al. (US 20210127949) discloses user interface for steerable endoscope with inertial measurement unit (see paragraphs 0059-0060).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AN SON P HUYNH whose telephone number is (571)272-7295. The examiner can normally be reached 9:00 am-6:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANH TUAN T. NGUYEN can be reached at 571-272-4963. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AN SON P HUYNH/Primary Examiner, Art Unit 3795
August 25, 2026
