DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 21 and 24-30 are objected to because of the following informalities:
Claim 21 recites the limitation “the intracorporeal image” should read “the real time intracorporeal image”.
Claims 24-30 recite the limitation “the step” should read “a step”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 20 and 26-27 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 20 recites the limitation "the data set" in line 2. There is insufficient antecedent basis for this limitation in the claim.
Claim 26 recites the limitation “the step of assigning a selection of data recorded before the surgical procedure and/or during the surgical procedure with the second data modality to the stored data set by a user interface and the control unit”. It is unclear what is meant by the limitation. The examiner question whether the stored data is the real time intracorporeal image data, the imported data, or the real time recorded data? The examiner interpret the limitation as assigning preoperative image data to the real time intracorporeal image data.
Claim 27 recites the limitation “the step of displaying a position of the stored data set in a scan or an image of the patient.”. It is unclear what is meant by the limitation. The examiner question whether the stored data is the real time intracorporeal image data, the imported data, or the real time recorded data? The examiner interpret the limitation as assigning preoperative image data to the real time intracorporeal image data.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 16-31 are rejected under 35 U.S.C. 103 as being unpatentable over Polchin et al. (WO 2022/072835) in the view of Jagga et al. (US 2018/0042681).
Regarding claim 16, Polchin teaches a surgical assistance system for use in a surgical procedure on a patient, the surgical assistance system comprising (para. 0091; pipeline 400 for the integrated surgical navigation and visualization system):
a surgical robot (para. 0053; The integrated surgical navigation and visualization system 101C of FIG. IB and/or the legacy surgical visualization system 101B of FIG. 1A may include a digital surgical microscope (DSM) head 110 mounted on a robotic arm 120);
a robot base as a local connection point of the surgical robot and a robot arm movably connected to the robot base and including at least one robot arm segment (paras. 0053 and 0064; The integrated surgical navigation and visualization system 101C of FIG. IB and/or the legacy surgical visualization system 101B of FIG. 1A may include a digital surgical microscope (DSM) head 110 mounted on a robotic arm 120. the integrated surgical navigation and visualization system 101 may comprise a navigation system integrated into the DSM head 102, which may be mounted on robotic arm 120. The cart 154 may support the robotic arm 120 as well as a boom-mounted 3D stereoscopic display (e.g., 3D stereoscopic display 170) and a mast-mounted touchscreen 171 for user input.);
a visualization system connected to the robot arm and adapted to generate and provide at least one real-time intracorporeal image (paras. 0064 and 0092; the integrated surgical navigation and visualization system 101 may comprise a navigation system integrated into the DSM head 102, which may be mounted on robotic arm 120. The cart 154 may support the robotic arm 120 as well as a boom-mounted 3D stereoscopic display (e.g., 3D stereoscopic display 170) and a mast-mounted touchscreen 171 for user input. Pipeline 400 may begin with image acquisition of a surgical site (block 402) (e.g., as part of an image data stream). The surgical site image acquisition may occur at or be performed by a surgical site image acquisition module. An example image acquisition module of a fully featured stereoscopic digital surgical microscope, including light source(s), zoom and focus optics, image sensors and all supporting electronics, software, firmware and hardware. This image acquisition module may generate surgical site image data stream 410, which may be communicated to microscope processing unit 420 and the associated surgical site image processing module 430.);
a navigation system adapted to determine, with spatial reference to the patient, at least one location of the visualization system and thus a location of the at least one real-time intracorporeal image (paras. 0077 and 0220-0221; At step 114C, the pose of the DSM camera relative to the navigation targets may be determined (e.g., in real time or near-real-time). For example, the navigation component of the integrated surgical navigation and visualization system may be used to view the targets on the clamp in real-time or near real-time to provide the latest pose of the digital surgical microscope camera(s) relative to the navigation targets. the camera registration result describing the location of the reference frame of the digital surgical microscope camera(s) relative to the reference frame of the navigation camera; dsmCam T navTarget is the pose of the digital surgical microscope camera(s) relative to the navigation target and is determined during camera registration via camera calibration and/or photogrammetry; and navTarget T navCam is the pose of the navigation target as seen by the navigation camera and is solved per frame using an algorithm such as OpenCV::findChessboardCorners to find critical features of the navigation target in every frame (or a subset of frames, depending on available compute capability) in concert with OpenCV::solvePnP to take those image locations and along with the navigation camera calibration information determine the pose information of the navigation target. Step 700B refers to the calculation of the relative pose between the digital surgical microscope camera(s) and the patient data and may be a final single matrix result needed to render a representation of the patient data from the same orientation and position that the digital surgical microscope camera(s) are relative to the live patient. This enables augmented reality for surgery.); and
the control unit being configured to (para. 0117):
import data recorded in at least one second data modality different from the visualization system into a data information system of the control unit via a data interface (paras. 0117 and 0230-0233; The user may proceed to use the software application on the integrated surgical navigation and visualization system to import patient information and patient image data at step 970 from patient data central storage. In some aspects, the patient data central storage may comprise one or more of a picture archiving and communication system (PACS), a hospital information system (HIS), or a radiology information system (RIS), collectively referred to as PACS/HIS/RIS 980. The patient information and patient image data may be provided over a communications interface such as hospital ethernet as formatted patient data 990. The patient information and/or patient image data may be formatted using one or more options (e.g., Digital Imaging Communication in Medicine (DICOM), Health Level (HL7), etc.). The registration step performs the calculation of patientRefFrm T patientData. The patient preop data is ingested and managed, the patient is prepared and the cal/navigation targets applied and scanned for surface data which is then aligned to the preop data. d. Patient scan data i. Data ingestion and reconstruction [00231] The data from the patient scan(s) are ingested into the system and in the case of 3D data placed in a data format convenient for 2D rendering as well as 3D (volumetric) rendering of the data. ii. Surface extraction from 3D scan data. A surface generally corresponding to the patient’ s skin is extracted from the 3D scan data. This surface is used to align to the similar surface extracted from the live patient data. iii. Registration of different modalities.),
assign at least a position to imported data in the coordinate system and store the imported data together with the position in the data information system (paras. 0242-0248; In this step all or some of the captured portion of the surface of the live patient data is matched (also known as registered, or aligned) to all or some of the captured portion of the surface of the patient in preop data. This process results in the transformation patientRefFrm_T_patientData),
generate a view of the at least one real-time intracorporeal image together with the imported data that are spatially arranged within the at least one real-time intracorporeal image (paras. 0071 and 0104-0108; A higher level of complexity for navigation achieved in various embodiments described herein is to overlay a rendering of such a two-dimensional “slice” of the patient scan data over the live microscope image, and have visible features of the rendering align to their corresponding features in the live view to some level of accuracy and to enable movement of the “slice” along the optical axis, for purposes of “x-ray vision” to view structures beneath the current physical surface of the patient’s anatomy. An even higher level of complexity for navigation also achieved in various embodiments described herein is to provide a three-dimensional rendering (albeit to the two-dimensional display) of the patient scan data over the live view, and have corresponding features align. III. Determining the Relative Pose of an Object and the Patient Anatomy), and
output the view via a monitor (paras. 0104-0108; For example, the overlay may be displayed at a distance along the optical axis of the digital surgical microscope, with an on/off option available.).
However, Polchin fails to explicitly teach a control unit adapted to assign and store a location in a coordinate system of the patient to the at least one real-time intracorporeal image.
Jagga, in the same field of endeavor, teaches a control unit adapted to assign and store a location in a coordinate system of the patient to the at least one real-time intracorporeal image (paras. 0060 and 0062; At any point during the procedure, the surgeon 101 may wish to capture or freeze the current view(s) in the user interface. This may be done through the selection of a button provided by a toolbar 415 in the user interface, for example using a foot pedal or other input device. The captured view(s) thus correspond to a selected position and orientation of the surgical tool 220. The capture view(s) may be stored (e.g., in a memory associated with the user interface or the navigation system 200). A waypoint 305 may be created automatically when the view(s) are captured or in response to further input (e.g., further selection of a button on the toolbar 415). The waypoint 305 may thus be created intraoperatively to store (e.g., in a memory associated with the user interface or the navigation system 200) the position and orientation of the surgical tool 220 at the position and orientation at which the captured view(s) are captured. In some examples, the waypoint 305 may be stored in association with the stored captured view(s). In some examples, only the waypoint 305 may be stored but not the captured view(s); instead, the captured view(s) may be regenerated for later viewing, from pre-operative images, using the stored position and orientation information of the waypoint 305).
It would have been obvious to one in the ordinary skill in the art before the effective filling date of the claimed invention to have modified the system of Polchin to incorporate the waypoint selection and storage of Jagga. This modification will help to provide navigational feedback by serving as a home point to which the surgical tool should return to, by serving as a reference point or landmark (e.g., giving the surgeon a sense of direction and relative distance) and/or by providing information about direction and/or distance to another point of interest as disclosed within Jagga in para. 0055.
Regarding claim 17, Polchin teaches the surgical assistance system according to claim 16, wherein the at least one second data modality comprises at least two second data modalities, and the imported data are recorded in the at least two second data modalities (para. 0233; Different modalities such as CT and MRI are used for generating patient scan data. The registration described thus far aligns the data from the live patient scan and the data from one modality scan (typically CT.) To use an additional modality, that modality may need to also be registered to the live patient data. The registration step includes optionally use of a modality alignment module which aligns other modalities either directly to the live patient data or by registering to the already-aligned modality. e. Patient preparation).
Regarding claim 18, Polchin teaches the surgical assistance system according to claim 16, wherein the data recorded with the at least one second data modality are image data and/or measurement data of the patient and/or operating data of the surgical robot (para. 0233; Different modalities such as CT and MRI are used for generating patient scan data. The registration described thus far aligns the data from the live patient scan and the data from one modality scan (typically CT.) To use an additional modality, that modality may need to also be registered to the live patient data. The registration step includes optionally use of a modality alignment module which aligns other modalities either directly to the live patient data or by registering to the already-aligned modality. e. Patient preparation).
Regarding claim 19, Polchin teaches the surgical assistance system according to claim 16, however fails to explicitly teach the control unit is configured to permanently store a real-time-recorded data set of the surgical assistance system in the data information system, the real-time-recorded data set comprises at least one real-time, navigated position of the surgical robot and the at least one real-time intracorporeal image and its location and the imported data, each in the coordinate system of the patient, the control unit is configured to recall the real-time-recorded data set of the surgical assistance system from the data information system, and when the real-time-recorded data set is recalled, a posture or position of the surgical robot is reset and the at least one real-time intracorporeal image and the imported data are output again.
Jagga, in the same field of endeavor, teaches the control unit is configured to permanently store a real-time-recorded data set of the surgical assistance system in the data information system, the real-time-recorded data set comprises at least one real-time, navigated position of the surgical robot and the at least one real-time intracorporeal image and its location and the imported data, each in the coordinate system of the patient (paras. 0059 and 0062; the image of the surgical field provided by the user interface may include a real-time view, such as a real-time view of the actual surgical field (e.g., as captured by an endoscope or an exoscope), a real-time view of a virtual surgical field (e.g., a virtual view that tracks the procedure in real-time), a real-time view of a preoperative image (e.g., a view of the preoperative image that tracks the procedure in real-time), or a combination thereof. At any point during the procedure, the surgeon 101 may wish to capture or freeze the current view(s) in the user interface. This may be done through the selection of a button provided by a toolbar 415 in the user interface, for example using a foot pedal or other input device. The captured view(s) thus correspond to a selected position and orientation of the surgical tool 220. The capture view(s) may be stored (e.g., in a memory associated with the user interface or the navigation system 200). The examiner notes that the live surgical field is recorded in a storage of the system which includes the real time intracorporeal image, preoperative image, and pose information),
the control unit is configured to recall the real-time-recorded data set of the surgical assistance system from the data information system (para. 0074; Even if the depiction of the waypoint is removed from the user interface, the waypoint and its stored position and orientation remain stored in memory (e.g., until manually deleted), and may be recalled to the user interface at a later time, for example), and
when the real-time-recorded data set is recalled, a posture or position of the surgical robot is reset and the at least one real-time intracorporeal image and the imported data are output again (para. 0063; After viewing the captured view(s) (or optionally without viewing the captured view(s)), the surgeon 101 may continue the procedure and move the surgical tool 220. After the surgical tool 220 has been moved, feedback (e.g., visual, audio and/or tactile feedback) may be provided to the surgeon to indicate the stored position and orientation of the waypoint 305 (e.g., to guide return of the surgical tool 220 to the waypoint 305 and/or to serve as a reference point). In some examples, such feedback may be provided automatically (e.g., in the form of a persistent visual representation of the waypoint 305, such as in FIG. 4) as soon as the waypoint 305 is created. In some examples, such feedback may be provided only in response to user input (e.g., selection of a button on the toolbar 415).).
It would have been obvious to one in the ordinary skill in the art before the effective filling date of the claimed invention to have modified the system of Polchin to incorporate the waypoint selection and storage of Jagga. This modification will help to provide navigational feedback by serving as a home point to which the surgical tool should return to, by serving as a reference point or landmark (e.g., giving the surgeon a sense of direction and relative distance) and/or by providing information about direction and/or distance to another point of interest as disclosed within Jagga in para. 0055.
Regarding claim 20, Polchin teaches the surgical assistance system according to claim 19, further comprising a user interface configured to selectively select elements of the data set for output or to deselect elements of the data set from the output (paras. 0264-0266; The patient data is displayed based on surgeon preference in various manners: Sharing the main visualization display with the live view of the surgical site. On its own display near the surgeon’ s field of view. Overlaid onto the live view of the surgical site, with options for the user to specify on/off and opacity. When a “Display tip and vector” option is selected in the software, the tip of each calibrated and verified tool (including the digital surgical microscope camera(s)) in view of the navigation camera is represented in the patient space by a uniquely colored annotation such as a dot.).
Regarding claim 21, Polchin teaches the surgical assistance system according to claim 16, wherein the control unit is adapted to output at least the intracorporeal image of the visualization system and/or the data recorded in the second data modality in an ordered manner depending on their data modality, and/or depending on a size of a field of view and/or depending on a viewing angle of a field of view and/or depending on a recording time (para. 0233; Different modalities such as CT and MRI are used for generating patient scan data. The registration described thus far aligns the data from the live patient scan and the data from one modality scan (typically CT.) To use an additional modality, that modality may need to also be registered to the live patient data. The registration step includes optionally use of a modality alignment module which aligns other modalities either directly to the live patient data or by registering to the already-aligned modality. e. Patient preparation).
Regarding claim 22, Polchin teaches a method for data visualization during a surgical procedure on a patient using a surgical assistance system, the method comprising the steps of (para. 0091; pipeline 400 for the integrated surgical navigation and visualization system):
navigating a visualization system connected to a robot arm of a surgical robot with spatial reference to a patient by a control unit (paras. 0250; The auto-navigating digital surgical microscope is placed to enable the navigation device to view the navigation target or targets XX mounted rigidly to the patient anatomy and to enable the digital surgical microscope camera(s) to view the areas of the patient anatomy relevant to the procedure.),
creating and providing at least one real-time intracorporeal image by the visualization system (paras. 0064 and 0092; the integrated surgical navigation and visualization system 101 may comprise a navigation system integrated into the DSM head 102, which may be mounted on robotic arm 120. The cart 154 may support the robotic arm 120 as well as a boom-mounted 3D stereoscopic display (e.g., 3D stereoscopic display 170) and a mast-mounted touchscreen 171 for user input. Pipeline 400 may begin with image acquisition of a surgical site (block 402) (e.g., as part of an image data stream). The surgical site image acquisition may occur at or be performed by a surgical site image acquisition module. An example image acquisition module of a fully featured stereoscopic digital surgical microscope, including light source(s), zoom and focus optics, image sensors and all supporting electronics, software, firmware and hardware. This image acquisition module may generate surgical site image data stream 410, which may be communicated to microscope processing unit 420 and the associated surgical site image processing module 430.);
determining a location of the visualization system, and thus a location of the at least one real- time intracorporeal image, with spatial reference to the patient, by a navigation system (paras. 0077 and 0220-0221; At step 114C, the pose of the DSM camera relative to the navigation targets may be determined (e.g., in real time or near-real-time). For example, the navigation component of the integrated surgical navigation and visualization system may be used to view the targets on the clamp in real-time or near real-time to provide the latest pose of the digital surgical microscope camera(s) relative to the navigation targets. the camera registration result describing the location of the reference frame of the digital surgical microscope camera(s) relative to the reference frame of the navigation camera; dsmCam T navTarget is the pose of the digital surgical microscope camera(s) relative to the navigation target and is determined during camera registration via camera calibration and/or photogrammetry; and navTarget T navCam is the pose of the navigation target as seen by the navigation camera and is solved per frame using an algorithm such as OpenCV::findChessboardCorners to find critical features of the navigation target in every frame (or a subset of frames, depending on available compute capability) in concert with OpenCV::solvePnP to take those image locations and along with the navigation camera calibration information determine the pose information of the navigation target. Step 700B refers to the calculation of the relative pose between the digital surgical microscope camera(s) and the patient data and may be a final single matrix result needed to render a representation of the patient data from the same orientation and position that the digital surgical microscope camera(s) are relative to the live patient. This enables augmented reality for surgery.); and
importing data that is recorded with at least one second data modality different from the visualization system into the data information system by a data interface of the control unit (paras. 0117 and 0230-0233; The user may proceed to use the software application on the integrated surgical navigation and visualization system to import patient information and patient image data at step 970 from patient data central storage. In some aspects, the patient data central storage may comprise one or more of a picture archiving and communication system (PACS), a hospital information system (HIS), or a radiology information system (RIS), collectively referred to as PACS/HIS/RIS 980. The patient information and patient image data may be provided over a communications interface such as hospital ethernet as formatted patient data 990. The patient information and/or patient image data may be formatted using one or more options (e.g., Digital Imaging Communication in Medicine (DICOM), Health Level (HL7), etc.). The registration step performs the calculation of patientRefFrm T patientData. The patient preop data is ingested and managed, the patient is prepared and the cal/navigation targets applied and scanned for surface data which is then aligned to the preop data. d. Patient scan data i. Data ingestion and reconstruction [00231] The data from the patient scan(s) are ingested into the system and in the case of 3D data placed in a data format convenient for 2D rendering as well as 3D (volumetric) rendering of the data. ii. Surface extraction from 3D scan data. A surface generally corresponding to the patient’ s skin is extracted from the 3D scan data. This surface is used to align to the similar surface extracted from the live patient data. iii. Registration of different modalities.),
assigning a position in the coordinate system to the imported data and storing the position and the imported data by the control unit (paras. 0242-0248; In this step all or some of the captured portion of the surface of the live patient data is matched (also known as registered, or aligned) to all or some of the captured portion of the surface of the patient in preop data. This process results in the transformation patientRefFrm_T_patientData),
generating a view of the at least one real-time intracorporeal image together with the imported data which, due to their position, are spatially arranged within the at least one real-time intracorporeal image, via the control unit (paras. 0071 and 0104-0108; A higher level of complexity for navigation achieved in various embodiments described herein is to overlay a rendering of such a two-dimensional “slice” of the patient scan data over the live microscope image, and have visible features of the rendering align to their corresponding features in the live view to some level of accuracy and to enable movement of the “slice” along the optical axis, for purposes of “x-ray vision” to view structures beneath the current physical surface of the patient’s anatomy. An even higher level of complexity for navigation also achieved in various embodiments described herein is to provide a three-dimensional rendering (albeit to the two-dimensional display) of the patient scan data over the live view, and have corresponding features align. III. Determining the Relative Pose of an Object and the Patient Anatomy), and
outputting the view via a monitor (paras. 0104-0108; For example, the overlay may be displayed at a distance along the optical axis of the digital surgical microscope, with an on/off option available.).
However, Polchin fails to explicitly teach assigning a location in a coordinate system of the patient to the at least one real-time intracorporeal image and storing the location and image in a data information system of the control unit by means of the control unit.
Jagga, in the same field of endeavor, teaches assigning a location in a coordinate system of the patient to the at least one real-time intracorporeal image and storing the location and image in a data information system of the control unit by means of the control unit (paras. 0060 and 0062; At any point during the procedure, the surgeon 101 may wish to capture or freeze the current view(s) in the user interface. This may be done through the selection of a button provided by a toolbar 415 in the user interface, for example using a foot pedal or other input device. The captured view(s) thus correspond to a selected position and orientation of the surgical tool 220. The capture view(s) may be stored (e.g., in a memory associated with the user interface or the navigation system 200). A waypoint 305 may be created automatically when the view(s) are captured or in response to further input (e.g., further selection of a button on the toolbar 415). The waypoint 305 may thus be created intraoperatively to store (e.g., in a memory associated with the user interface or the navigation system 200) the position and orientation of the surgical tool 220 at the position and orientation at which the captured view(s) are captured. In some examples, the waypoint 305 may be stored in association with the stored captured view(s). In some examples, only the waypoint 305 may be stored but not the captured view(s); instead, the captured view(s) may be regenerated for later viewing, from pre-operative images, using the stored position and orientation information of the waypoint 305).
It would have been obvious to one in the ordinary skill in the art before the effective filling date of the claimed invention to have modified the system of Polchin to incorporate the waypoint selection and storage of Jagga. This modification will help to provide navigational feedback by serving as a home point to which the surgical tool should return to, by serving as a reference point or landmark (e.g., giving the surgeon a sense of direction and relative distance) and/or by providing information about direction and/or distance to another point of interest as disclosed within Jagga in para. 0055.
Regarding claim 23, Polchin teaches the method for data visualization according to claim 22, wherein: the at least one second data modality comprises at least two second data modalities, and the imported data are recorded in the at least two second data modalities (para. 0233; Different modalities such as CT and MRI are used for generating patient scan data. The registration described thus far aligns the data from the live patient scan and the data from one modality scan (typically CT.) To use an additional modality, that modality may need to also be registered to the live patient data. The registration step includes optionally use of a modality alignment module which aligns other modalities either directly to the live patient data or by registering to the already-aligned modality. e. Patient preparation).
Regarding claim 24, Polchin teaches the method for data visualization according to claim 22, however, fails to explicitly teach permanently storing a real-time-recorded data set in the data information system by an operating interface and the control unit; and recalling the real-time-recorded data set from the data information system by an operating interface and the control unit, wherein: the real-time-recorded data set contains at least one real-time navigated position of the surgical robot and the at least one real-time intracorporeal image and its location and the imported data, in each case in the coordinate system of the patient, and wherein when real-time-recorded data set is recalled, a posture or position of the surgical robot is reset, and the at least one real-time intracorporeal image and the imported data are output again.
Jagga, in the same field of endeavor, teaches permanently storing a real-time-recorded data set in the data information system by an operating interface and the control unit (paras. 0059 and 0062; the image of the surgical field provided by the user interface may include a real-time view, such as a real-time view of the actual surgical field (e.g., as captured by an endoscope or an exoscope), a real-time view of a virtual surgical field (e.g., a virtual view that tracks the procedure in real-time), a real-time view of a preoperative image (e.g., a view of the preoperative image that tracks the procedure in real-time), or a combination thereof. At any point during the procedure, the surgeon 101 may wish to capture or freeze the current view(s) in the user interface. This may be done through the selection of a button provided by a toolbar 415 in the user interface, for example using a foot pedal or other input device. The captured view(s) thus correspond to a selected position and orientation of the surgical tool 220. The capture view(s) may be stored (e.g., in a memory associated with the user interface or the navigation system 200). The examiner notes that the live surgical field is recorded in a storage of the system which includes the real time intracorporeal image, preoperative image, and pose information); and recalling the real-time-recorded data set from the data information system by an operating interface and the control unit (para. 0074; Even if the depiction of the waypoint is removed from the user interface, the waypoint and its stored position and orientation remain stored in memory (e.g., until manually deleted), and may be recalled to the user interface at a later time, for example), wherein: the real-time-recorded data set contains at least one real-time navigated position of the surgical robot and the at least one real-time intracorporeal image and its location and the imported data, in each case in the coordinate system of the patient (paras. 0059 and 0062; the image of the surgical field provided by the user interface may include a real-time view, such as a real-time view of the actual surgical field (e.g., as captured by an endoscope or an exoscope), a real-time view of a virtual surgical field (e.g., a virtual view that tracks the procedure in real-time), a real-time view of a preoperative image (e.g., a view of the preoperative image that tracks the procedure in real-time), or a combination thereof. At any point during the procedure, the surgeon 101 may wish to capture or freeze the current view(s) in the user interface. This may be done through the selection of a button provided by a toolbar 415 in the user interface, for example using a foot pedal or other input device. The captured view(s) thus correspond to a selected position and orientation of the surgical tool 220. The capture view(s) may be stored (e.g., in a memory associated with the user interface or the navigation system 200). The examiner notes that the live surgical field is recorded in a storage of the system which includes the real time intracorporeal image, preoperative image, and pose information),
wherein when real-time-recorded data set is recalled, a posture or position of the surgical robot is reset, and the at least one real-time intracorporeal image and the imported data are output again (para. 0063; After viewing the captured view(s) (or optionally without viewing the captured view(s)), the surgeon 101 may continue the procedure and move the surgical tool 220. After the surgical tool 220 has been moved, feedback (e.g., visual, audio and/or tactile feedback) may be provided to the surgeon to indicate the stored position and orientation of the waypoint 305 (e.g., to guide return of the surgical tool 220 to the waypoint 305 and/or to serve as a reference point). In some examples, such feedback may be provided automatically (e.g., in the form of a persistent visual representation of the waypoint 305, such as in FIG. 4) as soon as the waypoint 305 is created. In some examples, such feedback may be provided only in response to user input (e.g., selection of a button on the toolbar 415).).
It would have been obvious to one in the ordinary skill in the art before the effective filling date of the claimed invention to have modified the system of Polchin to incorporate the waypoint selection and storage of Jagga. This modification will help to provide navigational feedback by serving as a home point to which the surgical tool should return to, by serving as a reference point or landmark (e.g., giving the surgeon a sense of direction and relative distance) and/or by providing information about direction and/or distance to another point of interest as disclosed within Jagga in para. 0055.
Regarding claim 25, Polchin teaches the method for data visualization according to claim 24, however fails to explicitly teach further comprising the step of recalling the real-time-recorded data set from the data information system, wherein the step of recalling the real-time-recorded data set is triggered when the surgical assistance system moves to a position of the real-time-recorded data set via the control unit.
Jagga, in the same field of endeavor, teaches the step of recalling the real-time-recorded data set from the data information system, wherein the step of recalling the real-time-recorded data set is triggered when the surgical assistance system moves to a position of the real-time-recorded data set via the control unit (para. 0063; After viewing the captured view(s) (or optionally without viewing the captured view(s)), the surgeon 101 may continue the procedure and move the surgical tool 220. After the surgical tool 220 has been moved, feedback (e.g., visual, audio and/or tactile feedback) may be provided to the surgeon to indicate the stored position and orientation of the waypoint 305 (e.g., to guide return of the surgical tool 220 to the waypoint 305 and/or to serve as a reference point). In some examples, such feedback may be provided automatically (e.g., in the form of a persistent visual representation of the waypoint 305, such as in FIG. 4) as soon as the waypoint 305 is created. In some examples, such feedback may be provided only in response to user input (e.g., selection of a button on the toolbar 415).
It would have been obvious to one in the ordinary skill in the art before the effective filling date of the claimed invention to have modified the system of Polchin to incorporate the waypoint selection and storage of Jagga. This modification will help to provide navigational feedback by serving as a home point to which the surgical tool should return to, by serving as a reference point or landmark (e.g., giving the surgeon a sense of direction and relative distance) and/or by providing information about direction and/or distance to another point of interest as disclosed within Jagga in para. 0055.
Regarding claim 26, Polchin teaches the method for data visualization according to claim 24, further comprising the step of assigning a selection of data recorded before the surgical procedure and/or during the surgical procedure with the second data modality to the stored data set by a user interface and the control unit (paras. 0098 and 0249; a patient data acquisition device (CT, MRI, etc.) 640 may be used to scan the relevant anatomy of patient 250 to generate acquired patient data 650. The acquired patient data may be optionally stored in a patient data central storage 660. The patient data may be sent (e.g., from the central storage 670) to the navigation processor 680. The same process of patient scan data registration to the live patient pose relative to the navigation target(s) is used to register new or updated data such as intraoperative MRI that might be generated during the procedure. m. Navigation).
Regarding claim 27, Polchin teaches the method for data visualization according to claim 24, further comprising the step of displaying a position of the stored data set in a scan or an image of the patient (paras. 0071 and 0233; A higher level of complexity for navigation achieved in various embodiments described herein is to overlay a rendering of such a two-dimensional “slice” of the patient scan data over the live microscope image, and have visible features of the rendering align to their corresponding features in the live view to some level of accuracy and to enable movement of the “slice” along the optical axis, for purposes of “x-ray vision” to view structures beneath the current physical surface of the patient’s anatomy. An even higher level of complexity for navigation also achieved in various embodiments described herein is to provide a three-dimensional rendering (albeit to the two-dimensional display) of the patient scan data over the live view, and have corresponding features align. III. Determining the Relative Pose of an Object and the Patient Anatomy).
Regarding claim 28, Polchin teaches the method for data visualization according to claim 24, further comprising the step of creating a history of data sets (paras. 0098 and 0117; a patient data acquisition device (CT, MRI, etc.) 640 may be used to scan the relevant anatomy of patient 250 to generate acquired patient data 650. The acquired patient data may be optionally stored in a patient data central storage 660. The user may proceed to use the software application on the integrated surgical navigation and visualization system to import patient information and patient image data at step 970 from patient data central storage. In some aspects, the patient data central storage may comprise one or more of a picture archiving and communication system (PACS), a hospital information system (HIS), or a radiology information system (RIS), collectively referred to as PACS/HIS/RIS 980. The patient information and patient image data may be provided over a communications interface such as hospital ethernet as formatted patient data 990. The patient information and/or patient image data may be formatted using one or more options (e.g., Digital Imaging Communication in Medicine (DICOM), Health Level (HL7), etc.).).
Regarding claim 29, Polchin teaches the method for data visualization according to claim 24, further comprising the step of measuring differences between images and/or scans of a data modality and a location taken at different recording times (paras. 0232-0233; A surface generally corresponding to the patient’ s skin is extracted from the 3D scan data. This surface is used to align to the similar surface extracted from the live patient data. iii. Registration of different modalities. Different modalities such as CT and MRI are used for generating patient scan data. The registration described thus far aligns the data from the live patient scan and the data from one modality scan (typically CT.) To use an additional modality, that modality may need to also be registered to the live patient data. The registration step includes optionally use of a modality alignment module which aligns other modalities either directly to the live patient data or by registering to the already-aligned modality. e. Patient preparation).
Regarding claim 30, Polchin teaches the method for data visualization according to claim 24, further comprising the steps of: marking an area and/or a coordinate within the at least one real-time intracorporeal image and/or in the data recorded in the second data modality different from the visualization system (para. 0301; During planning phase 1060, useful features may be added to the patient data space to aid the surgeon in the execution of the surgical procedure. These features include but are not limited to surgical opening “cut by numbers” patterns, approach vectors (e.g., trajectory plans), and approach waypoints at which the digital surgical microscope can be posed repeatably to establish and evaluate progress.); assigning a function and/or a property and/or a parameter (paras. 0301-0304; During planning phase 1060, useful features may be added to the patient data space to aid the surgeon in the execution of the surgical procedure. These features include but are not limited to surgical opening “cut by numbers” patterns, approach vectors (e.g., trajectory plans), and approach waypoints at which the digital surgical microscope can be posed repeatably to establish and evaluate progress. an example trajectory plan applicable to the integrated surgical navigation and visualization system, according to an example embodiment of the present disclosure. A trajectory plan can be optionally added in the patient data space 270. The trajectory may comprise a path in patient data space along which the user desires the procedure to proceed. For example, a cranial neurosurgeon might plan a trajectory toward an aneurysm that avoids critical parts of the brain and favors more readily traversed regions. If the trajectory is complex, it may be split into separate smaller trajectories which are more readily represented and achieved (e.g., for piecewise linearly). Also or alternatively, waypoints may be added by the user in the patient data space showing desired camera poses relative to the patient. With the connection of robot space, camera space, and patient space allowed in systems and methods described herein, such waypoints can be visited at any time during the procedure. Furthermore, such opening, trajectory and waypoint planning can be updated and/or augmented at any time during the procedure.); and storing the marking and assignment in the data information system (paras. 0301-0304; an example trajectory plan applicable to the integrated surgical navigation and visualization system, according to an example embodiment of the present disclosure. A trajectory plan can be optionally added in the patient data space 270. The trajectory may comprise a path in patient data space along which the user desires the procedure to proceed. For example, a cranial neurosurgeon might plan a trajectory toward an aneurysm that avoids critical parts of the brain and favors more readily traversed regions. If the trajectory is complex, it may be split into separate smaller trajectories which are more readily represented and achieved (e.g., for piecewise linearly). Also or alternatively, waypoints may be added by the user in the patient data space showing desired camera poses relative to the patient. With the connection of robot space, camera space, and patient space allowed in systems and methods described herein, such waypoints can be visited at any time during the procedure. Furthermore, such opening, trajectory and waypoint planning can be updated and/or augmented at any time during the procedure.).
Regarding claim 31, Polchin teaches the method for data visualization according to claim 24, however fails to explicitly teach wherein the step of recalling the real-time-recorded data set is triggered when the robot arm moves to a position of said real-time-recorded data set.
Jagga, in the same field of endeavor, teaches recalling the real-time-recorded data set is triggered when the robot arm moves to a position of said real-time-recorded data set (para. 0063; After viewing the captured view(s) (or optionally without viewing the captured view(s)), the surgeon 101 may continue the procedure and move the surgical tool 220. After the surgical tool 220 has been moved, feedback (e.g., visual, audio and/or tactile feedback) may be provided to the surgeon to indicate the stored position and orientation of the waypoint 305 (e.g., to guide return of the surgical tool 220 to the waypoint 305 and/or to serve as a reference point). In some examples, such feedback may be provided automatically (e.g., in the form of a persistent visual representation of the waypoint 305, such as in FIG. 4) as soon as the waypoint 305 is created. In some examples, such feedback may be provided only in response to user input (e.g., selection of a button on the toolbar 415).
It would have been obvious to one in the ordinary skill in the art before the effective filling date of the claimed invention to have modified the system of Polchin to incorporate the waypoint selection and storage of Jagga. This modification will help to provide navigational feedback by serving as a home point to which the surgical tool should return to, by serving as a reference point or landmark (e.g., giving the surgeon a sense of direction and relative distance) and/or by providing information about direction and/or distance to another point of interest as disclosed within Jagga in para. 0055.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZAINAB M ALDARRAJI whose telephone number is (571)272-8726. The examiner can normally be reached Monday-Thursday7AM-5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Carey Michael can be reached at (571) 270-7235. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZAINAB MOHAMMED ALDARRAJI/ Patent Examiner, Art Unit 3797
