Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4, and 6-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sauer et al. (US 2006/0281971, hereinafter “Sauer”) in view of Plewe et al. (US 2021/0196312, hereinafter “Plewe”).
As to claim 1, Sauer discloses a surgical assistance system for use in a surgical intervention on a patient, the surgical assistance system comprising:
a display device (e.g. display 301, Fig.3) for displaying a visual content;
an endoscope (endoscope 108, Fig.1) comprising an imaging recording head (image forming distal end, [0004]) adapted to create an intracorporeal image of the patient (“images from the endoscope…can be stored”, [0017]);
a data-providing unit (part of computer 103, Fig.1) adapted to provide digital 3D image data of the patient that is digital (three dimensional preoperative CT/MRI images, [0008], digitally stored on computer 103, [0017]);
a control unit (part of computer 103, Fig.1) adapted to process the intracorporeal image and the 3D image data (both endoscope image and preoperative image processed to be combined and displayed, [0018]);
a registration unit (part of computer 103, Fig.1) adapted to detect, in the intracorporeal image, at least one an anatomical landmark and/or an anatomical orientation and to determine a corresponding anatomical landmark and/or a corresponding anatomical orientation in the 3D image data, and to register the intracorporeal image with the 3D image data, at least initially, and to register the endoscope relative to the patient (computer 103 registers the endoscope image with the preoperative images, [0022], using image recognition techniques involving comparing and matching coordinates of anatomical landmarks in both images, [0023]); and
a tracking system (optical or sensor tracking arrangements, [0020]-[0021]) adapted to continuously detect a position, an orientation and a movement of the endoscope, and to provide the control unit with endoscope movement data (computer collects the position and orientation data of the endoscope, [0022],[0028]);
the control unit (part of computer 103) being further adapted to generate a correlation display with at least a display of the intracorporeal image and at least a display of a view of the 3D image data (computer generates a display as shown in Fig.3, with a display of the live endoscope view 311 and a display of the preoperative image 303), in which the intracorporeal image and the view of the 3D image data are correlated with respect to the endoscope by transferring the endoscope movement data to at least one virtual position and/or orientation of a virtual recording head in the view of the 3D image data for a correlated movement (positional information and movement of the endoscope and instruments are represented by the positions and movement of graphical representations 304,307 with respect to preoperative image, Fig.3, [0016],[0018]), and the control unit is adapted to visually output the correlation display on the display device (as illustrated in Fig.3).
Sauer, as set forth above, discloses, as the tracking system, a marker based optical tracking system or a magnetic based tracking system, which includes embedded magnetic sensor coils ([0021]) as an endoscope-internal movement tracking system, and thus fails to disclose an image based optical tracking system, wherein the control unit is adapted to determine the movement of the endoscope via an image analysis of moving anatomical structures in the intracorporeal image, without an external tracking sensor. However, Plewe teaches, in a similar surgical assistance system, that image-based techniques, wherein endoscope movement is determined with image analysis of moving anatomical structure (e.g. optical flow, motion detection, etc.) can be used in addition to or as an alternative to optical or magnetic tracking ([0165]-[0167],[0170]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have used image-based techniques (e.g. optical flow, motion detection, etc.) as an equivalent alternative means for tracking the position/orientation/movement of the endoscope instead of optical or magnetic tracking, as taught by Plewe. Use of such image-based tracking system as an alternative would not require an “external tracking sensor”.
As to claims 4 and 16, Sauer discloses, as set forth above with respect to claim 1, that
the tracking system comprises an endoscope-internal movement sensor provided in the endoscope and adapted to determine the movement of the endoscope as a second movement determination (embedded magnetic sensor coils, [0021] as part of the magnetic tracking system), and thus fails to disclose: 1) that the endoscope-internal movement sensor comprises an acceleration sensor and/or a rotation rate sensor (i.e. inertial measurement unit); 2) that, as an additional tracking system, the control unit is adapted to determine the movement of the endoscope via an image analysis of moving anatomical structures in the intracorporeal image as a first movement determination; and 3) that the control unit is adapted to calculate a final movement determination based on an adjusted mean value or a weighted value calculated from first movement determination and the second movement determination. However, Plewe teaches, in a similar surgical assistance system, that 1) an acceleration sensor or gyroscope (rotation rate sensor) (both of which constitute an inertial measurement unit) can be used to determine position/orientation/movement of the endoscope ([0169]) as an alternative to a magnetic tracking system ([0167]); 2) image-based techniques, wherein endoscope movement is determined with image analysis of moving anatomical structure (e.g. feature tracking, optical flow, motion detection, etc.) can be used in addition to or as an alternative to optical or magnetic tracking ([0165]-[0167],[0170]); and 3) the control unit can use a combination of input data (i.e. first movement determination from the image-based tracking and second determination data from the magnetic/acceleration sensors), wherein a final movement determination is based on an adjusted mean or weighted combination of tracking data ([0170]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have substituted an acceleration sensor or gyroscope for the magnetic tracking sensor of Sauer since they are known in the art to be equivalent alternative means for determining endoscope position/orientation/movement, as taught by Plewe. Furthermore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have used more than one tracking system (e.g. additionally used an image-based tracking system) and determined a final movement using the combined tracking data in a weighted fashion to provide a more reliable determined result (Plewe: [0170]).
As to claim 6, wherein the registration unit performs a re-registration beyond an initial registration at at least one further point in time in order to further increase an accuracy of a correlation (updates the registration, [0023]).
As to claim 7, Sauer, as set forth with respect to claim 1 above, disclose manual manipulation of the endoscope/instruments by the surgeon and thus fails to disclose that the endoscope is robot-guided via a robot arm and the tracking system determines the position and/or the orientation and/or the movement of the endoscope via an associated position and/or orientation and/or movement of the robot arm and thus via the robot. However, Plewe teaches in the art of surgical assistance devices, to use a robotic system including a robot-guided arm to advance and navigate the endoscope into the patient ([0045]-[0046]). Plewe further teaches that when a robotic arm is used, as the tracking system, robotic command and/or kinematics data can be used to determine the position/orientation/movement of the endoscope ([0168]). Given the teachings of Plewe, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have provided a robotic system including a robot arm for guiding the endoscope to a target site for more controlled and precise navigation of the endoscope over manual navigation. Furthermore, when using a robotic system for guidance, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have use, as the tracking system, robotic command and/or kinematic data from the robot arm to determine the position/orientation/movement of the endoscope as an alternative equivalent means for tracking the endoscope (note that Plewe teaches such robotic tracking as an alternative to using a magnetic tracking system, [0167]-[0168]).
As to claim 8, wherein the control unit is adapted to display in the correlation display in the display of the view of the 3D image data and/or of the intracorporeal image, in addition to the endoscope, pre-planned medical instruments and/or implants stored in a storage unit in a correct position in the correlation display (computer configured to display a stored (preplanned) graphical representation of the surgical instrument 307, Fig.3, in its tracked (correct) relative position with respect to the endoscope and preoperative image, [0016]-[0018], Fig.3).
As to claim 9, wherein the control unit is adapted to display a planned path and/or annotations in the correlation display in order to guide a surgeon to a surgical field (computer is configured to display a optical axis 309 and view angle cone 310, which either can constitute a planned path and/or annotations, for guiding either of the endoscope 304 or surgical instrument 307 to the surgical field (within cone 310), Fig.3, [0018]).
As to claim 10, wherein the control unit is adapted to generate the view of the 3D image data, as: a 3D scene (three dimensional model, [0016]); or two-dimensional cross-sections relative to a picture-coordinate system of the endoscope and/or along an image axis of the endoscope (display of a 3D image data on a 2D display screen, Fig.3, is essentially a 2D cross-section); or a virtual endoscope image (virtual global view, [0018]).
As to claim 11, Sauer discloses an image display method for a central, correlated display of two different images, the image display method comprising the steps of:
reading in 3D image data of a patient (three dimensional preoperative CT/MRI image, [0008]);
creating or detecting an intracorporeal image through an endoscope (images from endoscope are collected, [0017]);
detecting at least one landmark and/or anatomical orientation in the intracorporeal image (computer registers the endoscope image with the preoperative image using image recognition techniques, [0022], involving detecting coordinates of anatomical landmarks from the intracorporeal image, [0023]);
determining at least one corresponding landmark and/or at least one corresponding orientation in the 3D image data (relevant anatomical structures may be identified in the preoperative image, [0023];
registering the 3D image data to the intracorporeal image via the at least one detected corresponding landmark and/or the at least one corresponding orientation (registration by matching the landmark coordinates in both images, [0022],[0023]);
generating a correlation display with at least one display of the intracorporeal image and at least one display of a view of the 3D image data (computer generates a display as shown in Fig.3, with a display of the live endoscope view 311 and a display of the preoperative image 303);
continuously detecting a position and an orientation and a movement of the endoscope (computer collects the position and orientation and movement data of the endoscope, [0022],[0028]);
transferring a detected movement of the endoscope to at least one virtual position and/or at least one virtual orientation of a virtual recording head in the view of the 3D image data (positional information and movement of the endoscope and instruments are represented by the positions and movement of graphical representations 304,307 with respect to preoperative image, Fig.3, [0016],[0018]); and
generating at least one updated correlation display with the intracorporeal image and an updated view of the 3D image data and outputting the at least one updated correlation display to the display device (images and views are updated in real time, [0028]).
Sauer, as set forth above, discloses, as the tracking system, a marker based optical tracking system or a magnetic based tracking system, which includes embedded magnetic sensor coils ([0021]) as an endoscope-internal movement tracking system, and thus fails to disclose an image based optical tracking system, wherein the control unit is adapted to determine the movement of the endoscope via an image analysis of moving anatomical structures in the intracorporeal image, without an external tracking sensor. However, Plewe teaches, in a similar surgical assistance system, that image-based techniques, wherein endoscope movement is determined with image analysis of moving anatomical structure (e.g. optical flow, motion detection, etc.) can be used in addition to or as an alternative to optical or magnetic tracking ([0165]-[0167],[0170]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have used image-based techniques (e.g. optical flow, motion detection, etc.) as an equivalent alternative means for tracking the position/orientation/movement of the endoscope instead of optical or magnetic tracking, as taught by Plewe. Use of such image-based tracking system as an alternative would not require an “external tracking sensor”.
As to claim 12, Sauer further discloses the steps of: determining a deformation of a real anatomical structure based on at least one landmark and the at least one corresponding landmark, and transferring the deformation to a virtual anatomical structure in the 3D image data in order to adjust and correct the 3D image data (if an anatomical structure is removed (deformation), the virtual model image (preoperative) can be updated to reflect this deformation, [0024]).
As to claim 13, Sauer discloses a non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method steps of claim 11 (see description of claim 11 above; all steps are performed by programming on a computer, [0017],[0018]).
As to claim 14, Sauer discloses a medical sterile space (operating room, Fig.1, [0006],[0017],[0028]) comprising a surgical the surgical assistance system according to claim 1 (the surgical assistance system 100, as set forth with respect to claim 1 above, is used in the operating room, Fig.1).
As to claim 15, wherein a handling portion and/or the imaging recording head of the endoscope comprises the endoscope-internal movement sensor (embedded sensor coils, as set forth above).
Claim(s) 5 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sauer et al. (US 2006/0281971, hereinafter “Sauer”) and Plewe et al. (US 2021/0196312, hereinafter “Plewe”), as set forth above with respect to claim 1, and further in view of Garbey et al. (US 2014/0171787, hereinafter “Garbey”).
As to claims 5 and 17, Sauer discloses, as set forth above with respect to claim 1, that the control unit (computer 103) displays the at least one medical instrument in the view of the 3D image data in the correct position (Fig.3, displays instrument 307 correctly positioned with respect to the preoperative image based on tracking system data, [0018]), so that the at least one medical instrument is displayed in the intracorporeal image and in the view of the 3D image data (Fig.3, 307 shown in both views). However, Sauer uses either an optical or magnetic tracking system ([0020],[0021]), or other alternative means of tracking as taught by Plewe ([0165]-[0170]) to determine the position of the instrument and thus fails to disclose that the control unit is adapted to recognize at least one medical instrument in the intracorporeal image in a correct position based on a database stored in the storage unit with geometrically predefined structures of medical instruments. Garbey teaches, in the surgical assistance system art, that a surgical instrument can be tracked by recognizing the position and orientation of an instrument in an image based on stored geometrical features associated with the instrument ([0054]-[0056]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have tracked the instrument using the image, based on predefined stored instrument features, as taught by Garbey, as an equivalent alternative to the tracking system of Sauer/Plewe. One would be motivated to use the image tracking in Sauer to eliminate the need for additional structures (physical sensors, sensor equipment) when an image of the instrument, which is already being obtained, could be used to track it.
Response to Arguments
Rejections and objections from the previous Office Action that have not been repeated in this Office Action should be considered as addressed or corrected, and thus hereby withdrawn.
Applicant's arguments filed April 2, 2026 have been fully considered but they are not persuasive.
With respect to claim 1, which has been amended to include the rejected subject matter of previously claims 2 and 3, Applicant argues that Sauer describes a system “that cannot function without the external hardware” (the “external hardware” referring to components of the tracking system that are external to the tracked objects, i.e. endoscope). The Examiner respectfully disagrees. Nowhere in the Sauer reference explicitly states or even suggests that an optical or magnetic tracking system (both of which have components external to the endoscope) are necessarily required or are critical to Sauer’s system. In fact, Sauer offers these two “well-known” tracking systems as merely examples of tracking systems that CAN BE used to track the endoscope and surgical instruments ([0019]). Applicant fails to evidence why Sauers system can not function without tracking systems with external components.
Furthermore, Applicant contends that the magnetic coils (within the endoscope) “are not endoscope-internal movement sensors”. Obviously, the Examiner respectfully disagrees. The magnetic coils are embedded in the endoscope, making them “internal”, and provide signals indicative of position/orientation/movement, making them movement sensors. The term “endoscope-internal movement sensor” neither requires the sensor as being totally “internal”, nor as detecting movement “inertially”, as Applicant desires (see second full paragraph of Applicant’s Remarks, page 10).
Regarding the Plewe reference, Applicant attempts to dismiss the teachings of Plewe as being “theoretical possibilities, without providing a specific technical teaching for the concrete implementation of such combination”. This is not well taken. Although the Examiner and those of ordinary skill would not characterize Plewe’s explicit teachings of alternative means, including their combination, to track the endoscope, as “theoretical possibilities”, even IF considered as such, such “theoretical possibility” of using such tracking means as an alternative to those of Sauer are still taught, contemplated and suggested by Plewe, and would make obvious the “theoretical possibility” of using the tracking system encompassed by the scope of Applicant’s claimed tracking system.
Furthermore, the Examiner does not agree with Applicant’s characterization of the Plewe reference as not teaching “completely dispensing with external trackers by continuously combining data from endoscope-internal movement sensor and image analysis as the primary tracking principle”. No matter what Plewe considers as a “primary” technique, inertial movement sensors (within the endoscope) and image analysis movement sensing techniques are both contemplated, and Plewe explicitly teaches that both forms of input can be used in combination.
Regarding Applicant’s characterization of the combination of Sauer and Plewe as relying on impermissible hindsight bias and “essentially destroying the navigation concept described in Sauer”, the Examiner respectfully disagrees. As previously pointed out, Sauer merely uses well known tracking techniques (Sauer: [0019]) which is exemplified by two particular ones to explain the invention. Nothing in Sauer suggests that those two particular tracking techniques are critical to the invention or that use of any other tracking technique would do anything more than provide position/orientation/movement data that is required by the system.
For these reasons, the rejections are being maintained.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN P LEUBECKER whose telephone number is (571)272-4769. The examiner can normally be reached Generally, M-F, 5:30-2:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anhtuan T Nguyen can be reached at 571-272-4963. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN P LEUBECKER/Primary Examiner, Art Unit 3795
Prosecution Insights