Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/08/2026 has been entered.
Response to Amendment
Applicant's amendments and remarks filed 06/08/2026 have been entered and considered but are not found convincing. Claims 1,4,9,11-17,36, 39 have been amended. Claims 2, 18-35 were cancelled. In summary, claims 1, 3-17, 36-39 are pending in this application.
Claim Rejections - 35 USC § 103
Applicant's arguments with respect to independent claim 1 have been considered but are moot because the rejection has been modified to address the newly added limitations. The Examiner now relies on the new reference Diolaiti.
Claim Objections
Claim 38 is objected to because of the following informalities:
Claim 38 recites the limitation “ the image” in line 1. It should be “the image of the medical environment”. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
1. Claims 1, 4-6, 10,14, 17, 36-37, 39 are rejected under 35 U.S.C. 103 as being unpatentable over AZIZIAN et al, WO2019/006028 (“AZIZIAN”) in view of Ryan et al, U.S Patent Application Publication No.2018/0049622 (“Ryan”) further in view of Fuerst et al., U.S Patent Application Publication No.20210121233 (“Fuerst”) further in view of Diolaiti et al, IDS, U.S Patent Application Publication No.20090326556 (“Diolaiti”)
Regarding independent claim 1, AZIZIAN teaches a system ([0020] Referring to FIG.1 A of the drawings, a teleoperational medical system for use in, for example, medical procedures including diagnostic, therapeutic, or surgical procedures, is generally indicated by the reference numeral 10. As will be described, the teleoperational medical systems of this disclosure are under the teleoperational control of a surgeon. In alternative embodiments, a teleoperational medical system may be under the partial control of a computer programmed to perform the procedure or sub-procedure. In still other alternative embodiments, a fully automated medical system, under the full control of a computer”) comprising:
a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions, when executed by the processor, ([0023] The teleoperational medical system 10 also includes a control system 20. The control system 20 includes at least one memory 24 and at least one processor 22, and typically a plurality of processors, for effecting control between the medical instrument system 14, the operator input system 16, and other auxiliary systems 26 which may include, for example, imaging systems, audio systems, fluid delivery systems, display systems, illumination systems, steering control systems, irrigation systems, and/or suction systems. The control system 20 can be used to process the images of the surgical environment from the imaging system 15 for subsequent display to the surgeon S through the surgeon's console 16. The control system 20 also includes programmed instructions (e.g., a computer-readable medium storing the instructions) to implement some or all of the methods described in accordance with aspects disclosed herein”) cause the system to:
receive an image of a medical environment ([0022],[0038] “At a process 108, the three-dimensionally mapped endoscopic image dataset is used to generate and display an image from a vantage point that may assist the surgeon. For example, an initial vantage point may be the viewpoint from the distal end of the endoscope within the patient anatomy (i.e., the conventional endoscopic view from the endoscope). The initial vantage point image may be constructed based on the three dimensional point cloud. FIG. 3 illustrates a display 150 visible by a surgeon at the operator console or on another display screen. The display 150 includes an endoscopic image 152 of a surgical environment including instruments 156, 157, 158. The image 152 is generated from the mapped endoscopic image dataset. Optionally, the display 150 includes an anatomic model image 154 generated from the anatomic model dataset. In this example, the endoscopic image 152 is overlaid, superimposed, blended or otherwise mapped onto the anatomic model image 154. The display 150 also includes instrument illustrations 156', 157', 158' overlaid or superimposed on the anatomic model image 154. The instrument illustrations 156', 157', 158' may be generated based on instrument pose and relative scale, known or determined from kinematic chain, position or shape sensor information, vision-based tracking or a combination of those”);
identify a medical component in the image of the medical environment, the medical component disposed in a first configuration ([0038] At a process 108, the three-dimensionally mapped endoscopic image dataset is used to generate and display an image from a vantage point that may assist the surgeon. For example, an initial vantage point may be the viewpoint from the distal end of the endoscope within the patient anatomy (i.e., the conventional endoscopic view from the endoscope). The initial vantage point image may be constructed based on the three dimensional point cloud. FIG. 3 illustrates a display 150 visible by a surgeon at the operator console or on another display screen. The display 150 includes an endoscopic image 152 of a surgical environment including instruments 156, 157, 158. The image 152 is generated from the mapped endoscopic image dataset. Optionally, the display 150 includes an anatomic model image 154 generated from the anatomic model dataset. In this example, the endoscopic image 152 is overlaid, superimposed, blended or otherwise mapped onto the anatomic model image 154. The display 150 also includes instrument illustrations 156', 157', 158' overlaid or superimposed on the anatomic model image 154. The instrument illustrations 156', 157', 158' may be generated based on instrument pose and relative scale, known or determined from kinematic chain, position or shape sensor information, vision-based tracking or a combination of those. Various tool tracking techniques have been described, for example in U.S. Patent Application No. 11/865,014, filed September 30, 2007, disclosing "METHODS AND SYSTEMS FOR ROBOTIC INSTRUMENT TOOL TRACKING WITH ADAPTIVE FUSION OF KINEMATICS INFORMATION AND F AGE INFORMATION" and U.S. Patent Application No. 11/130,471, filed May 16, 2005. disclosing "Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery," which are incorporated by reference herein their entirety. The display 150 thus provides the surgeon with additional environmental context beyond that provided by the endoscopic image 152 alone. For example, the anatomic model image 154 illustrates additional organs, bones, and tumors not visible on the endoscopic image 152, and the instrument illustrations 156', 157', 158' provide information about instrument trajectories not visible on the endoscopic image. In some embodiments, the endoscopic image 152 may be distinguishable from the anatomic model image 154 by a border illustration, color differences (e.g., the endoscopic image in color, and the anatomic model image in grayscale), in brightness, or other distinguishing characteristic”);
receive kinematic information about the medical component (see at least [0022]; [0038] At a process 108, the three-dimensionally mapped endoscopic image dataset is used to generate and display an image from a vantage point that may assist the surgeon. For example, an initial vantage point may be the viewpoint from the distal end of the endoscope within the patient anatomy (i.e., the conventional endoscopic view from the endoscope). The initial vantage point image may be constructed based on the three dimensional point cloud. FIG. 3 illustrates a display 150 visible by a surgeon at the operator console or on another display screen. The display 150 includes an endoscopic image 152 of a surgical environment including instruments 156, 157, 158. The image 152 is generated from the mapped endoscopic image dataset. Optionally, the display 150 includes an anatomic model image 154 generated from the anatomic model dataset. In this example, the endoscopic image 152 is overlaid, superimposed, blended or otherwise mapped onto the anatomic model image 154. The display 150 also includes instrument illustrations 156', 157', 158' overlaid or superimposed on the anatomic model image 154. The instrument illustrations 156', 157', 158' may be generated based on instrument pose and relative scale, known or determined from kinematic chain, position or shape sensor information, vision-based tracking or a combination of those. Various tool tracking techniques have been described, for example in U.S. Patent Application No. 11/865,014, filed September 30, 2007, disclosing "METHODS AND SYSTEMS FOR ROBOTIC INSTRUMENT TOOL TRACKING WITH ADAPTIVE FUSION OF KINEMATICS INFORMATION AND F AGE INFORMATION" and U.S. Patent Application No. 11/130,471, filed May 16, 2005. disclosing "Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery," which are incorporated by reference herein their entirety. The display 150 thus provides the surgeon with additional environmental context beyond that provided by the endoscopic image 152 alone. For example, the anatomic model image 154 illustrates additional organs, bones, and tumors not visible on the endoscopic image 152, and the instrument illustrations 156', 157', 158' provide information about instrument trajectories not visible on the endoscopic image. In some embodiments, the endoscopic image 152 may be distinguishable from the anatomic model image 154 by a border illustration, color differences (e.g., the endoscopic image in color, and the anatomic model image in grayscale), in brightness, or other distinguishing characteristics”); and
generate virtual guidance based on the kinematic information, the virtual guidance including a virtual movement of the medical component from the first configuration to a second configuration ([see at least [0038] At a process 108, the three-dimensionally mapped endoscopic image dataset is used to generate and display an image from a vantage point that may assist the surgeon. For example, an initial vantage point may be the viewpoint from the distal end of the endoscope within the patient anatomy (i.e., the conventional endoscopic view from the endoscope). The initial vantage point image may be constructed based on the three dimensional point cloud. FIG. 3 illustrates a display 150 visible by a surgeon at the operator console or on another display screen. The display 150 includes an endoscopic image 152 of a surgical environment including instruments 156, 157, 158. The image 152 is generated from the mapped endoscopic image dataset. Optionally, the display 150 includes an anatomic model image 154 generated from the anatomic model dataset. In this example, the endoscopic image 152 is overlaid, superimposed, blended or otherwise mapped onto the anatomic model image 154. The display 150 also includes instrument illustrations 156', 157', 158' overlaid or superimposed on the anatomic model image 154. The instrument illustrations 156', 157', 158' may be generated based on instrument pose and relative scale, known or determined from kinematic chain, position or shape sensor information, vision-based tracking or a combination of those. Various tool tracking techniques have been described, for example in U.S. Patent Application No. 11/865,014, filed September 30, 2007, disclosing "METHODS AND SYSTEMS FOR ROBOTIC INSTRUMENT TOOL TRACKING WITH ADAPTIVE FUSION OF KINEMATICS INFORMATION AND F AGE INFORMATION" and U.S. Patent Application No. 11/130,471, filed May 16, 2005. disclosing "Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery," which are incorporated by reference herein their entirety. The display 150 thus provides the surgeon with additional environmental context beyond that provided by the endoscopic image 152 alone. For example, the anatomic model image 154 illustrates additional organs, bones, and tumors not visible on the endoscopic image 152, and the instrument illustrations 156', 157', 158' provide information about instrument trajectories not visible on the endoscopic image. In some embodiments, the endoscopic image 152 may be distinguishable from the anatomic model image 154 by a border illustration, color differences (e.g., the endoscopic image in color, and the anatomic model image in grayscale), in brightness, or other distinguishing characteristics”; [0041] Optionally, the display 160 includes an anatomic model image 164 generated from the anatomic model dataset. In this example, the endoscopic image 162 is overlaid, superimposed, blended or otherwise mapped onto the anatomic model image 164. The display 160 also includes an instrument illustration 166 overlaid or superimposed on the anatomic model image 164. The instrument illustration 166 may be generated based on instrument pose and scaling known or determined from kinematic chain, position or shape sensor information, vision-based tracking or a combination of those. The display 160 thus provides the surgeon with additional environmental context beyond that provided by the endoscopic image 162 alone. For example, the anatomic model image 164 illustrates additional organs and bones not visible on the endoscopic image 162, and the instrument illustration 166 provides information about instrument trajectories not visible on the endoscopic image. From the second vantage point, the surgeon is able to observe a more expansive anatomical area while still maintaining awareness of the direct view from the endoscope. The display of the anatomic model image 164 is optional because in some examples, the endoscopic image 162 alone, presenting the three-dimensionally mapped endoscopic image dataset from the second vantage point, may provide the surgeon with sufficient spatial perspective.) AZIZIAN is understood to be silent on the remaining limitations of claim 1.
In the same field of endeavor, Ryan teaches receive kinematic information about the medical component (see at least [0079] Referring to FIG. 5, the one or more cameras 500, 506 of the sensor suites (400, 422, 210, and 306) and the one or more visual markers 502, 504 are used to visually track a distinct object (e.g., a surgical tool, a desired location within an anatomical object, etc.) and determine attitude and position relative to the user 106. In one embodiment, each of the one or more markers is distinct and different from each other visually. Standalone object recognition and machine vision technology can be used for marker recognition. Alternatively, the present invention also provides for assisted tracking using IMUs 408 on one or more objects of interest, including but not limited to the markers 502, 504. Please note that the one or more cameras 500, 506 can be remotely located from the user 106 and provide additional data for tracking and localization.”);
generate virtual guidance based on the kinematic information and the indicator of guidance type, the virtual guidance including a virtual guiding the medical component (see at least [0096] FIG. 7 depicts an alternate view of the MXUI previously shown in FIG. 6, wherein a virtual target 700 and a virtual tool 702 are presented to the user 106 for easy use in achieving the desired version and inclination. In this embodiment, further combinations of virtual reality are used to optimize the natural feeling experience for the user by having a virtual target 700 with actual tool 702 fully visible or a virtual tool (not shown) with virtual target fully visible. Other combinations of real and virtual imagery can optionally be provided. Presentation of data can be in readable form 704 or in the form of imagery including but not limited to 3D representations of tools or other guidance forms.”; [0107] FIG. 23 depicts an exemplary embodiment of a MXUI shown to the user 106 via the display device 104 during resection of the femoral neck of a hip replacement procedure with a virtual resection guide 2300. A sagittal saw 2302 is shown having a plurality of fiducials 2304 defining a marker, allows the pose of the sagittal, saw 2302 to be tracked. Resection of the femoral neck can be guided either by lining up the actual saw blade 2306 with the virtual resection guide 2300 in the case where the drill is not tracked or by lining up a virtual saw blade (not shown) with the virtual resection guide 2300 in the case where the saw 2302 is tracked. As with the tracked drill shown in FIG. 20, the angles of the saw 2302 may be displayed numerically if the saw 2302 is tracked. These angles could be displayed relative to the pelvic reference frame or the femoral reference frame.[0108] FIG. 24 depicts an exemplary embodiment of a MXUI shown to the user 106 via the display device 104 during positioning of the acetabular shell of a hip replacement procedure wherein a virtual target 2400 for the acetabular impactor assembly 1100 and a virtual shell 2402 are shown. Placement of the acetabular impactor assembly 1100 is guided by manipulating it to align with the virtual target 2400. The posterior/lateral quadrant of the shell portion of the virtual target may be displayed in a different color or otherwise visually differentiated from the rest of the shell 2402 to demarcate to the user 106 a target for safe placement of screws into the acetabulum. The numerical angle of the acetabular impactor and the depth of insertion relative to the reamed or un-reamed acetabulum are displayed numerically as virtual text 2404. A magnified stereoscopic image (not shown) similar to 2202 centered on the tip of the impactor may be displayed showing how the virtual shell interfaces with the acetabulum of the virtual pelvis 2102.” ” Where provide virtual guidance based on the posture and position of the medical component (tool), including the presentation of data in human-readable format (704) and a 3D representation of the tool. In addition, the virtual target (700) is a virtual image of the medical component and is arranged at a position different from that of the virtual tool (700), and corresponds to the arrangement of the second setting)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to the method of using a mixed reality surgical navigation system for a medical procedures of AZIZIAN with generating virtual guidance based on the posture and position of the medical component as seen in Ryan because this modification would assist in the performance of a medical procedure ([0005] of Ryan) Both AZIZIAN and Ryan are understood to be silent on the remaining limitations of claim 1.
In the same field of endeavor, Fuerst teaches receive kinematic information about the medical component ([0032] A surgical robotic kinematics processor 106 can generate kinematics parameters that define movement restraints and movement abilities of a physical surgical robotic arm. For example, each robotic arm can have various interconnected members, each of which can move rotate or travel in one or more defined directions, planes, or axis. Movements can be bound, for example, by angles or distances of travel. Thus, the kinetic parameters can define how a surgical robotic arm and surgical robotic tool can or cannot move, which in turn, determines a reach and movement capabilities of a tool in a workspace.”);
receive an indicator of guidance type ([0048] In one embodiment, the VR simulation 100 can be communicated or output through a display, for example, on a user console 120. A view generator can generate one or more views 118 of the VR simulation, for example, a stadium view or first person view. Such view can be used by a user to further generate inputs to the optimizer 20. For example, a user can use the user console 120 to modify or replace the virtual patient, a workspace, virtual surgical equipment, and/or a virtual operating room. The VR simulation can be optimized at block 110 based on the adjustments made by the user. The view of the simulation can then be re-generated for the user, thus providing adjustments and optimization of the workflow based on user generated inputs. In this manner, the user can tailor the workflow and layout optimization to suit a particular patient and procedure type. This can streamline setup of robotic arms for different patients, different procedures, and help selection of operating rooms to facilitate such procedures.”); and
generate virtual guidance based on the kinematic information and the indicator of guidance type, the virtual guidance including a virtual guiding movement of the medical component ([0040] In one embodiment, the kinematics parameters can be generated that define movement constraints of a surgical robotic arm to be used in the procedure type. For example, the kinematics parameters can define a direction that a member of the robot moves, a maximum or minimum distance or angle that each member can travel, the mass of each member, the stiffness of each member, and/or speed and force in which members move. Thus, the arranging of the virtual surgical robotic arm or the virtual tool can further be based on the kinematics parameters of the virtual robotic arm so that the virtual surgical robotic arm can perform the surgical procedures type within the surgical workspace, with reference to the kinematics parameters. [0050] In one embodiment, as shown in FIG. 4, an optimization process 110 can optimize a room layout 152 and surgical robotic arm configuration 154, based on one or more inputs 138. The inputs can include changes to one or more of the following factors: the procedure type, the surgical workspace, the virtual patient, the tool position, and kinematics parameters.” [0052] Changes in a patient model, surgical tool position, robot kinematics, or surgical workspace can initiate an adjustment to the setup of the robotic arm. For example, if a size of a virtual patient changes, this can cause a change in a location of the workspace. The trocar position is adjusted to maximize the reach of a tool within the changed location of the workspace. The port location and position of the surgical robotic arm can be adjusted accordingly. It should be understood that ‘setup’, ‘configuration’, and ‘arrangement’ as discussed with relation to the robotic arm and tool can describe the manner in which members of the robotic arm and/or tool is positioned, either absolutely or relative to one another. Thus, the ‘setup’ or ‘configuration’ of the robotic arm can describe a particular pose of the robotic arm and a position of the tool which is attached to the robotic arm. [0053] In one embodiment, changes in the location or orientation of virtual surgical equipment (e.g., user console, scanning equipment, control tower) or staff personnel models can initiate a rearrangement of the virtual surgical equipment. Similarly, changes to a procedure type can initiate changes in the surgical workspace in a virtual patient which, in turn, can cause a change in default tool positions (e.g., a default trocar position), port locations, and surgical robotic arm positions. Similarly, changes in a virtual patient (for example, taller or heavier build) can initiate a change in the surgical workspace of the virtual patient, which can cause a ripple effect in tool position, port location, and surgical robotic arm configuration. Similarly, a change in platform position (e.g., height, angle or slope of table) can similarly initiate changes in the tool position, port location, and robotic arm configuration, given that the virtual patient is laid upon the platform during the virtual procedure [0054] Referring to FIG. 4, the optimization process 110 can generate an optimized a room layout 152 and surgical robotic arm configuration 154 supports tool positions to have adequate reach within a defined workspace. The optimization process can have a hierarchy of operations based on a ranked order of importance. Blocks 140, 142, 144, and 146 can be performed sequentially in the order shown.”; [0061] Referring to FIG. 5, one example is shown of an optimized room layout in plan view. The location of any of the virtual objects, such as but not limited to: virtual patient 226, surgical robotic platform 222, surgical robotic arms and attached tools 224, a virtual control tower 228, a user console 232, and other virtual surgical equipment 230 can be arranged in a virtual surgical environment 220. The arrangement of the virtual objects can be described with coordinates such as, but not limited to, an x, y, and/or z axis. The orientation can describe a direction of any of the objects.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to the method of using a mixed reality surgical navigation system for a medical procedures of AZIZIAN and Ryan with generating virtual room operating based on based on procedure type as seen in Fuerst because this modification would plan a surgical robotic workflow that describes an arrangement of surgical robotic arms in advance of a procedure ([0005] ) AZIZIAN, Ryan, Fuerst are understood to be silent on the remaining limitations of claim 1.
In the same field of endeavor, Diolaiti teaches receive an indicator of guidance type, the indicator of guidance type comprising an indicator of a mode of operation (see at least [0051] In this example, each of the input devices 108, 109 may be selectively associated with one of the devices 211, 231, 241, 200 so that the associated device may be controlled by the input device through its controller and manipulator. For example, by placing switches 258, 259 respectively in tool following modes "T2" and "T1", the left and right input devices 108, 109 may be respectively associated with the first and second surgical tools 231, 241, which are telerobotically controlled through their respective controllers 233, 243 (preferably implemented in the processor 102) and manipulators 232, 242 so that the Surgeon may perform a medical procedure on the Patient while the entry guide 200 is locked in place”; [0074] Rather than setting the viewing point to a fixed point at all times, the viewing point may also be automatically changed depending upon the control mode (i.e., one of the modes described in reference to FIG. 2) that is operative at the time. As an example, FIG. 18 illustrates a method for automatically changing the auxiliary viewing mode depending upon the control mode currently operative in the medical robotic system 100. In particular, using this method, a first auxiliary viewing mode is performed in 1802 when the medical robotic system 100 is determined in 1801 to be in a tool following mode, a second auxiliary viewing mode is performed in 1804 when the medical robotic system 100 is determined in 1803 to be in an entry guide positioning mode, and a third auxiliary viewing mode is performed in 1806 when the medical robotic system 100 is determined in 1805 to be in a camera positioning mode. The viewing modes for each control mode are selected so as to be most beneficial to the surgeon for performing actions during that mode. For example, in the tool following and camera positioning modes, either or both the surgical tools 231, 241 and camera 211 is being moved at the time and therefore, an auxiliary view of the articulatable camera 211 and articulatable surgical tools 231, 241 extending out of the distal end of the entry guide 200, such as depicted in FIGS. 12 and 13, is useful to avoid collisions between links that are out of the field of view of the camera 211. On the other hand, in the entry guide positioning mode, the articulatable camera 211 and the articulatable surgical tools 231, 241 are locked in position relative to the entry guide 200 and therefore, an auxiliary view providing information on other things such as depicted in FIGS. 16 and 17 may be useful.”) ; and generate virtual guidance based on the kinematic information and the indicator of guidance type, the virtual guidance including a virtual animation guiding movement of the medical component from the first configuration to a second configuration, the second configuration corresponding to the mode of operation ([0052] When the camera 211 or the entry guide 200 is to be repositioned by the Surgeon, either one or both of the left and right input devices 108, 109 may be associated with the camera 211 or entry guide 200 so that the Surgeon may move the camera 211 or entry guide 200 through its respective controller (213 or 203) and manipulator (212 or 202). In this case, the disassociated one(s) of the surgical tools 231, 241 is locked in place relative to the entry guide 200 by its controller. For example, by placing switches 258, 259 respectively in camera positioning modes "C2" and "C1", the left and right input devices 108, 109 may be associated with the camera 211, which is telerobotically controlled through its controller 213 (preferably implemented in the processor 102) and manipulator 212 so that the Surgeon may position the camera 211 while the surgical tools 231, 241 and entry guide 200 are locked in place by their respective controllers 233, 243, 203. If only one input device is to be used for positioning the camera, then only one of the switches 258, 259 is placed in its camera positioning mode while the other one of the switches 258, 259 remains in its tool following mode so that its respective input device may continue to control its associated surgical tool.[0053] On the other hand, by placing switches 258, 259 respectively in entry guide positioning modes "G2" and "G1", the left and right input devices 108, 109 may be associated with the entry guide 200, which is telerobotically controlled through its controller 203 (preferably implemented in the processor 102) and manipulator 202 so that the Surgeon may position the entry guide 200 while the surgical tools 231, 241 and camera 211 are locked in place relative to the entry guide 200 by their respective controllers 233, 243, 213. As with the camera positioning mode, if only one input device is to be used for positioning the entry guide, then only one of the switches 258, 259 is placed in its entry guide positioning mode while the other one of the switches 258, 259 remains in its tool following mode so that its respective input device may continue to control its associated surgical tool. [0054] The selective association of the input devices 108, 109 to other devices in this example may be performed by the Surgeon using the GUI 170 or the voice recognition system 160 in a conventional manner. Alternatively, the association of the input devices 108, 109 may be changed by the Surgeon depressing a button on one of the input devices 108, 109 or depressing the foot pedal 105, or using any other well known mode switching technique.”; [0081] Rather than overlaying the captured image, rendering may also include using the auxiliary view to augment the image captured by the camera 211 by displaying only the portions of the instruments 231, 241 that are not seen in the captured image (i.e., the dotted line portion of the instruments 231, 241 in FIG. 17) in proper alignment and adjacent the captured image in a mosaic fashion.”; [0074] Rather than setting the viewing point to a fixed point at all times, the viewing point may also be automatically changed depending upon the control mode (i.e., one of the modes described in reference to FIG. 2) that is operative at the time. As an example, FIG. 18 illustrates a method for automatically changing the auxiliary viewing mode depending upon the control mode currently operative in the medical robotic system 100. In particular, using this method, a first auxiliary viewing mode is performed in 1802 when the medical robotic system 100 is determined in 1801 to be in a tool following mode, a second auxiliary viewing mode is performed in 1804 when the medical robotic system 100 is determined in 1803 to be in an entry guide positioning mode, and a third auxiliary viewing mode is performed in 1806 when the medical robotic system 100 is determined in 1805 to be in a camera positioning mode. The viewing modes for each control mode are selected so as to be most beneficial to the surgeon for performing actions during that mode. For example, in the tool following and camera positioning modes, either or both the surgical tools 231, 241 and camera 211 is being moved at the time and therefore, an auxiliary view of the articulatable camera 211 and articulatable surgical tools 231, 241 extending out of the distal end of the entry guide 200, such as depicted in FIGS. 12 and 13, is useful to avoid collisions between links that are out of the field of view of the camera 211. On the other hand, in the entry guide positioning mode, the articulatable camera 211 and the articulatable surgical tools 231, 241 are locked in position relative to the entry guide 200 and therefore, an auxiliary view providing information on other things such as depicted in FIGS. 16 and 17 may be useful. [0082] In addition to, or in lieu of, overlaying the captured image over the auxiliary view or augmenting the captured image with the auxiliary view, rendering may also include providing other useful information in the auxiliary view. As an example, FIG. 16 illustrates an auxiliary side view of an articulatable camera 211 with a frustum 1601 rendered on the auxiliary view so as to be displayed on the auxiliary display 140 as emanating from, and moving with, the camera tip 311. Note that although the frustum 1601 is shown in the figure as a truncated cone, it may also appear as a truncated pyramid to correspond to the captured image that is shown on the monitor 104. The sides of the frustum 1601 indicate a viewing range of the camera 211 and the base 1602 of the frustum 1601 displays an image 1650 that was captured by the camera 211. Note that for simplification purposes, the surgical tools 231, 241 normally in the auxiliary view have been removed for this example. As another example, FIG. 14 shows a semi-translucent sphere or bubble 1401 (preferably colored red) which is displayed by the method as part of the rendering process when a warning threshold is reached so as to indicate to the operator that the highlighted portions 1402, 1403 of the surgical tool 241 and camera 211 are dangerously close to colliding. In this case, the highlighted portions 1402, 1403 are preferably centered within the sphere. As yet another example, FIG. 14 also shows a marker or other indicator 1410 indicating an optimal position for the camera tip 311 for viewing the end effectors of the surgical tools 231, 241 as they are being used to perform a medical procedure. The optimal position may be determined, for example, by finding a location where the tips of the end effectors are equidistant from a center of the captured image.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to the method of using a mixed reality surgical navigation system for a medical procedures of AZIZIAN, Ryan and Fuerst with selecting an indicator of guidance type of Diolaiti because this modification would change the auxiliary viewing mode depending upon the control mode ([0074 of Diolaiti).
Thus, the combination of AZIZIAN, Ryan, Fuerst and Diolaiti teaches
a system comprising: a processor; and a memory having computer readable instructions stored thereon, the computer readable instructions, when executed by the processor, cause the system to: receive an image of a medical environment; identify a medical component in the image of the medical environment, the medical component disposed in a first configuration; receive kinematic information about the medical component; receive an indicator of guidance type, the indicator of guidance type comprising an indicator of a mode of operation; and generate virtual guidance based on the kinematic information and the indicator of guidance type, the virtual guidance including a virtual animation guiding movement of the medical component from the first configuration to a second configuration, the second configuration corresponding to the mode of operation.
Regarding claim 4, AZIZIAN, Ryan, Fuerst and Diolaiti teach the system of claim 1,wherein the medical component is a robot-assisted manipulator assembly (see at least [0020][0022],[0027][0038] of AZIZIAN, [0037] of Diolaiti “As with other parts of the medical robotic system 100, the illustration of the robotic arm assembly 130 is simplified in FIG. 1. In one example of the medical robotic system 100, the robotic arm assembly 130 includes a setup arm and an entry guide manipulator. The setup arm is used to position the entry guide 200 at the entry aperture 150 so that it properly enters the entry aperture 150. The entry guide manipulator is then used to robotically insert and retract the entry guide 200 into and out of the entry aperture 150. It may also be used to robotically pivot the entry guide 200 in pitch, roll and yaw about a pivot point located at the entry aperture 150. An example of such an entry guide manipulator is the entry guide manipulator 202 of FIG. 2 and an example of the four degrees-of-freedom movement that it manipulates the entry guide 200 with is shown in FIG. 5.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 5, AZIZIAN, Ryan, Fuerst and Diolaiti teach the system of claim 1, wherein receiving the image includes receiving the image from a mobile device(see at least: [0022],[0038] of AZIZIAN “At a process 108, the three-dimensionally mapped endoscopic image dataset is used to generate and display an image from a vantage point that may assist the surgeon. For example, an initial vantage point may be the viewpoint from the distal end of the endoscope within the patient anatomy (i.e., the conventional endoscopic view from the endoscope). The initial vantage point image may be constructed based on the three dimensional point cloud. FIG. 3 illustrates a display 150 visible by a surgeon at the operator console or on another display screen. The display 150 includes an endoscopic image 152 of a surgical environment including instruments 156, 157, 158.; [0102] of Ryan “FIG. 18 depicts an exemplary embodiment of a MXUI shown to the user 106 via the display device 104 (e.g., the AR headset 3600) showing the calibration assembly 1500 being used for various calibration steps. First, the hip impactor assembly 1100 can be screwed into the appropriate hole of the plate 1502 so that the shoulder 1206 is seated squarely without play against the surface of the plate 1502. The cameras 3904 of the AR headset 3600 can then capture images which processed by an algorithm to determine the relationship between the shoulder of the impactor on which the acetabular shell will seat and the marker 1104 of the hip impactor assembly 1100.”; [0113] FIG. 28 depicts a flowchart showing how the system 10 and its display device 104 (e.g., the AR headset 3600) can be used in conjunction with the C-arm 2700 in a surgical procedure. The camera 3904 (e.g., a high definition camera or the like) incorporated in the AR headset 3600 can be used to capture the image displayed on the C-arm monitor (2800)….”; [0041] of Fuerst “ In one embodiment, the VR simulation can include virtual equipment 115 such as a surgical robotic platform, a control tower, a user console, scanning equipment (e.g., mobile X-ray machine (C-arm) or ultrasound imaging machine), stands, stools, trays, and other equipment that would be arranged in a surgical operating room. The virtual equipment can be generated based on computer models that define parameters such as shape and size of the various equipment.) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 6, AZIZIAN, Ryan, Fuerst and Diolaiti teach the system of claim 1, wherein receiving the image includes receiving the image from a camera system mounted in the medical environment (see at least: [0025]; [0027],[0030],[0038] of AZIZIAN “Endoscopic imaging systems (e.g., systems 15, 28) may be provided in a variety of configurations including rigid or flexible endoscopes. Rigid endoscopes may include a rigid tube, housing a relay lens system, for transmitting an image from a distal end to a proximal end of the endoscope. Flexible endoscopes may transmit images using one or more flexible optical fibers. Digital image based endoscopes have a "chip on the tip" design in which a distal digital sensor such as a one or more charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) device acquire image data and acquired image data can be transferred over a wired or wireless interface. Endoscopic imaging systems may provide two- or three- dimensional images to the viewer. Two-dimensional images may provide limited depth perception. Three-dimensional stereo endoscopic images may provide the viewer with more accurate depth perception. Stereo endoscopic instruments employ stereo cameras to capture stereo images of the patient anatomy. An endoscopic instrument may be a fully sterilizable assembly with the endoscope cable, handle and shaft all rigidly coupled and hermetically sealed.”; [0070] of Ryan “Referring to FIGS. 1, 2A-B, and 3, a sensory augmentation system 10 of the present invention is provided for use in medical procedures. The system 10 includes one or more visual markers (100, 108, 110), a processing unit 102, a sensor suite 210 having one or more tracking camera(s) 206, and a display device 104 having a display generator 204 that generates a visual display on the display device 104 for viewing by the user 106. The display device 104 is attached to a user 106 such that the display device 104 can augment his visual input. In one preferred embodiment, the display device 104 is attached to the user's 106 head. Alternatively, the display device 104 is located separately from the user 106, while still augmenting the visual scene. In one embodiment, each of the markers (100, 108, and 110) is distinct and different from each other visually so they can be individually tracked by the camera(s) 206) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 10, AZIZIAN, Ryan, Fuerst and Diolaiti teach the system of claim 1, wherein receiving kinematic information about the medical component includes receiving sensor information from the medical component (see at least: [0038] of AZIZIAN; [0079] of Ryan “Referring to FIG. 5, the one or more cameras 500, 506 of the sensor suites (400, 422, 210, and 306) and the one or more visual markers 502, 504 are used to visually track a distinct object (e.g., a surgical tool, a desired location within an anatomical object, etc.) and determine attitude and position relative to the user 106. In one embodiment, each of the one or more markers is distinct and different from each other visually. Standalone object recognition and machine vision technology can be used for marker recognition. Alternatively, the present invention also provides for assisted tracking using IMUs 408 on one or more objects of interest, including but not limited to the markers 502, 504. Please note that the one or more cameras 500, 506 can be remotely located from the user 106 and provide additional data for tracking and localization”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 14, AZIZIAN, Ryan, Fuerst and Diolaiti teach the system of claim 1, wherein the virtual animation includes an auxiliary component (see at least: [0038] of AZIZIAN; [0094]; [0096] of Ryan “FIG. 7 depicts an alternate view of the MXUI previously shown in FIG. 6, wherein a virtual target 700 and a virtual tool 702 are presented to the user 106 for easy use in achieving the desired version and inclination. In this embodiment, further combinations of virtual reality are used to optimize the natural feeling experience for the user by having a virtual target 700 with actual tool 702 fully visible or a virtual tool (not shown) with virtual target fully visible. Other combinations of real and virtual imagery can optionally be provided. Presentation of data can be in readable form 704 or in the form of imagery including but not limited to 3D representations of tools or other guidance forms.”; [0061] of Fuerst Referring to FIG. 5, one example is shown of an optimized room layout in plan view. The location of any of the virtual objects, such as but not limited to: virtual patient 226, surgical robotic platform 222, surgical robotic arms and attached tools 224, a virtual control tower 228, a user console 232, and other virtual surgical equipment 230 can be arranged in a virtual surgical environment 220. The arrangement of the virtual objects can be described with coordinates such as, but not limited to, an x, y, and/or z axis. The orientation can describe a direction of any of the objects.”; [0080] of Diolaiti “Rendering may also include overlaying the image captured by the camera 211 over the auxiliary view when the viewing point of the auxiliary image is the same as or directly behind that of the camera 211. As an example, FIG. 17 illustrates a captured image 1700 of the camera 211 rendered as an overlay to an auxiliary view of surgical tools 231, 241 which has been generated from a viewing point of (or right behind) the camera 211. In this example, the auxiliary view of the surgical tools 231, 241 being displayed on the auxiliary display screen 140 (and/or the auxiliary display screen 140') includes portions (e.g., 1731, 1741) in the overlaying captured image 1700 and portions (e.g., 1732, 1742) outside of the overlaying captured image 1700. Thus, the portions of the surgical tools 231, 241 outside of the captured image 1700 provide the Surgeon with additional information about their respective links or articulating arms that are out of the field of view of the camera 211. Highlighting of the instrument portions (e.g., 1732, 1742) outside of the captured image 1700 may also be done for identification purposes or to indicate a warning or alerting condition as described above. Overlaying the captured image 1700 onto the auxiliary view also has the advantage in this case of showing an anatomic structure 360 which is in front of the surgical tools 231, 241 that would not otherwise normally be in the auxiliary view. Although this example shows the captured image 1700 overlaying the auxiliary view on the auxiliary display screen 140, in another rendering scheme, the auxiliary view may overlay the captured image that is being displayed on the monitor 104.[0081] Rather than overlaying the captured image, rendering may also include using the auxiliary view to augment the image captured by the camera 211 by displaying only the portions of the instruments 231, 241 that are not seen in the captured image (i.e., the dotted line portion of the instruments 231, 241 in FIG. 17) in proper alignment and adjacent the captured image in a mosaic fashion.”) In addition ,the same motivation is used as the rejection for claim 1.
Regarding claim 17, AZIZIAN, Ryan, Fuerst and Diolaiti teach the system of claim 1, wherein the virtual animation is overlayed on the received image of the medical environment including the medical component in the first configuration. (see at least: [0038] of AZIZIAN; [0094]-[0096] of Ryan “FIG. 7 depicts an alternate view of the MXUI previously shown in FIG. 6, wherein a virtual target 700 and a virtual tool 702 are presented to the user 106 for easy use in achieving the desired version and inclination. In this embodiment, further combinations of virtual reality are used to optimize the natural feeling experience for the user by having a virtual target 700 with actual tool 702 fully visible or a virtual tool (not shown) with virtual target fully visible. Other combinations of real and virtual imagery can optionally be provided. Presentation of data can be in readable form 704 or in the form of imagery including but not limited to 3D representations of tools or other guidance forms.” where real medical component (tool 702) in a first setting (current position) and a virtual image (virtual target 700)); ; [0080] of Diolaiti “ Rendering may also include overlaying the image captured by the camera 211 over the auxiliary view when the viewing point of the auxiliary image is the same as or directly behind that of the camera 211. As an example, FIG. 17 illustrates a captured image 1700 of the camera 211 rendered as an overlay to an auxiliary view of surgical tools 231, 241 which has been generated from a viewing point of (or right behind) the camera 211. In this example, the auxiliary view of the surgical tools 231, 241 being displayed on the auxiliary display screen 140 (and/or the auxiliary display screen 140') includes portions (e.g., 1731, 1741) in the overlaying captured image 1700 and portions (e.g., 1732, 1742) outside of the overlaying captured image 1700. Thus, the portions of the surgical tools 231, 241 outside of the captured image 1700 provide the Surgeon with additional information about their respective links or articulating arms that are out of the field of view of the camera 211. Highlighting of the instrument portions (e.g., 1732, 1742) outside of the captured image 1700 may also be done for identification purposes or to indicate a warning or alerting condition as described above. Overlaying the captured image 1700 onto the auxiliary view also has the advantage in this case of showing an anatomic structure 360 which is in front of the surgical tools 231, 241 that would not otherwise normally be in the auxiliary view. Although this example shows the captured image 1700 overlaying the auxiliary view on the auxiliary display screen 140, in another rendering scheme, the auxiliary view may overlay the captured image that is being displayed on the monitor 104. [0081] Rather than overlaying the captured image, rendering may also include using the auxiliary view to augment the image captured by the camera 211 by displaying only the portions of the instruments 231, 241 that are not seen in the captured image (i.e., the dotted line portion of the instruments 231, 241 in FIG. 17) in proper alignment and adjacent the captured image in a mosaic fashion. [0082] In addition to, or in lieu of, overlaying the captured image over the auxiliary view or augmenting the captured image with the auxiliary view, rendering may also include providing other useful information in the auxiliary view. As an example, FIG. 16 illustrates an auxiliary side view of an articulatable camera 211 with a frustum 1601 rendered on the auxiliary view so as to be displayed on the auxiliary display 140 as emanating from, and moving with, the camera tip 311. Note that although the frustum 1601 is shown in the figure as a truncated cone, it may also appear as a truncated pyramid to correspond to the captured image that is shown on the monitor 104. The sides of the frustum 1601 indicate a viewing range of the camera 211 and the base 1602 of the frustum 1601 displays an image 1650 that was captured by the camera 211. Note that for simplification purposes, the surgical tools 231, 241 normally in the auxiliary view have been removed for this example. As another example, FIG. 14 shows a semi-translucent sphere or bubble 1401 (preferably colored red) which is displayed by the method as part of the rendering process when a warning threshold is reached so as to indicate to the operator that the highlighted portions 1402, 1403 of the surgical tool 241 and camera 211 are dangerously close to colliding. In this case, the highlighted portions 1402, 1403 are preferably centered within the sphere. As yet another example, FIG. 14 also shows a marker or other indicator 1410 indicating an optimal position for the camera tip 311 for viewing the end effectors of the surgical tools 231, 241 as they are being used to perform a medical procedure. The optimal position may be determined, for example, by finding a location where the tips of the end effectors are equidistant from a center of the captured image.”). In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 36, AZIZIAN, Ryan, Fuerst and Diolaiti teach he system of claim 1, further comprising a display system configured to display the virtual guidance (see at least: [0038] of AZIZIAN “At a process 108, the three-dimensionally mapped endoscopic image dataset is used to generate and display an image from a vantage point that may assist the surgeon. For example, an initial vantage point may be the viewpoint from the distal end of the endoscope within the patient anatomy (i.e., the conventional endoscopic view from the endoscope). The initial vantage point image may be constructed based on the three dimensional point cloud. FIG. 3 illustrates a display 150 visible by a surgeon at the operator console or on another display screen. The display 150 includes an endoscopic image 152 of a surgical environment including instruments 156, 157, 158. The image 152 is generated from the mapped endoscopic image dataset.”; [0070] of Ryan “Referring to FIGS. 1, 2A-B, and 3, a sensory augmentation system 10 of the present invention is provided for use in medical procedures. The system 10 includes one or more visual markers (100, 108, 110), a processing unit 102, a sensor suite 210 having one or more tracking camera(s) 206, and a display device 104 having a display generator 204 that generates a visual display on the display device 104 for viewing by the user 106. The display device 104 is attached to a user 106 such that the display device 104 can augment his visual input. In one preferred embodiment, the display device 104 is attached to the user's 106 head. Alternatively, the display device 104 is located separately from the user 106, while still augmenting the visual scene. In one embodiment, each of the markers (100, 108, and 110) is distinct and different from each other visually so they can be individually tracked by the camera(s) 206.”;Fig. 5 of Fuerst, Fig.2,14-20 of Diolaiti ) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 37, AZIZIAN, Ryan, Fuerst and Diolaiti teach the system of claim 1, further comprising a robot-assisted manipulator assembly configured for operating a medical instrument in the medical environment (see at least: [0022], [0027],[0038] of AZIZIAN; [0003]; [0102] of Ryan ; [0037] of Fuerst “Thus, the kinematics modeling can benefit from physical feedback as well as computer-generated restraints to generate a kinematics model that resembles real-life kinematics of the robotic arm. Beneficially, by using real data from a physical robotic arm, the kinematics parameters can provide accuracy in determining the arrangement of the virtual robotic arms relative to the workspace and virtual patient. Accordingly, based on the movement constraints and capabilities of a physical surgical robotic arm and/or tool, the virtual robotic arm can be arranged relative to the virtual patient in the VR simulation to enable proper movements within the workspace of the virtual patient, as required by a particular surgical procedure.’; Figs 1-18 of Diolaiti) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 39, AZIZIAN, Ryan, Fuerst and Diolaiti teach the system of claim 1, wherein the mode of operation comprises at least one of a set-up mode, a sterile preparation mode, an instrument exchange mode, a troubleshooting mode, a servicing mode, an inspection mode, a cleaning mode, or a storage mode (see at least: [0038] of Fuerst “In one embodiment, an optimization procedure 110 is performed to arrange the virtual objects (e.g., the patient, the robotic arm and tool, the platform, the user console, the control tower, and other surgical equipment) based on one or more inputs and changes to inputs. For example, inputs can workspace, patient geometry, operating room, procedure type, platform height, etc. Changes to inputs can include a modification to a virtual patient, a change to a procedure, a change to the operating room, a change to platform height, and/or any change that would modify a location or size of a workspace in a virtual patient. In some cases, a user interface can allow a user to manually move a virtual object in a VR simulation while wearing a headset having a head-up display. The user can be immersed in the VR simulation. Based on one or more inputs, the optimization procedure can rearrange the virtual objects and re-optimize.”;[0074] of Diolaiti Rather than setting the viewing point to a fixed point at all times, the viewing point may also be automatically changed depending upon the control mode (i.e., one of the modes described in reference to FIG. 2) that is operative at the time. As an example, FIG. 18 illustrates a method for automatically changing the auxiliary viewing mode depending upon the control mode currently operative in the medical robotic system 100. In particular, using this method, a first auxiliary viewing mode is performed in 1802 when the medical robotic system 100 is determined in 1801 to be in a tool following mode, a second auxiliary viewing mode is performed in 1804 when the medical robotic system 100 is determined in 1803 to be in an entry guide positioning mode, and a third auxiliary viewing mode is performed in 1806 when the medical robotic system 100 is determined in 1805 to be in a camera positioning mode. The viewing modes for each control mode are selected so as to be most beneficial to the surgeon for performing actions during that mode. For example, in the tool following and camera positioning modes, either or both the surgical tools 231, 241 and camera 211 is being moved at the time and therefore, an auxiliary view of the articulatable camera 211 and articulatable surgical tools 231, 241 extending out of the distal end of the entry guide 200, such as depicted in FIGS. 12 and 13, is useful to avoid collisions between links that are out of the field of view of the camera 211. On the other hand, in the entry guide positioning mode, the articulatable camera 211 and the articulatable surgical tools 231, 241 are locked in position relative to the entry guide 200 and therefore, an auxiliary view providing information on other things such as depicted in FIGS. 16 and 17 may be useful.) In addition, the same motivation is used as the rejection for claim 1.
2. Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over AZIZIAN et al, WO2019/006028 (“AZIZIAN”) in view of Ryan et al, U.S Patent Application Publication No.2018/0049622 (“Ryan”) further in view of Fuerst et al., U.S Patent Application Publication No.20210121233 (“Fuerst”) further in view of Diolaiti et al, IDS, U.S Patent Application Publication No.20090326556 (“Diolaiti”) further in view of further in view of DONHOWE et al, IDS, WO-2014106253 (“DONHOWE”)
Regrading claim 3, AZIZIAN, Ryan, Fuerst and Diolaiti the system of claim 1, wherein the computer readable instructions, when executed by the processor, further cause the system to: provide an implementation compared to the virtual guidance ([0079] of Fuerst “ At operation 278, a compatibility determination can be performed, describing how compatible a physical OR is for a surgical robotic system and/or procedure. The determination can show how much capacity the OR has to support the virtual surgical robotic system. In one embodiment, the determination can be based on whether the virtual equipment can be reliably moved in and out of the virtual OR, which, in turn, reflects whether the physical equipment can reliably be moved in and out of the physical OR. Compatibility score also be based on comparing the sizes of virtual equipment (e.g., the virtual robotic arm, platform, user console, and control tower), with entrances or pathways of the virtual OR.”).) AZIZIAN, Ryan, Fuerst and Diolaiti are understood to be silent on the remaining limitations of claim 3.
In the same field of endeavor, DONHOWE teaches wherein the computer readable instructions, when executed by the processor, further cause the system to: provide an evaluation of an implementation compared to the virtual guidance (see at least page 16, least paragraph- page 17, first and second paragraph “At 408, a planned deployment location for the interventional instrument is located. The planned deployment location may be marked on the model of the plurality of passageways. The planned deployment location can be selected based upon the instrument operational capability information, the target structure information, the patient anatomy information, or a combination of the types of information. The selected deployment location may be at a point in an anatomic passageway nearest to the target structure. However, in many patients a nearest point deployment location may be impossible for the distal end of the interventional instrument to reach because the instrument has insufficient bend capability within the size and elasticity constraints of the selected anatomic passageway. A more suitable deployment location may be at a point on an anatomic passageway wall where the interventional instrument has an approach angle to the passageway wall that is within the bending capability of the instrument. For example, if the interventional instrument has an inflexible distal end that permits little or no bending, a suitable deployment location may be at a carina near the target structure. At the carina the interventional instrument may be deployed at an approximately 90° approach angle to the passageway wall with minimal bending of the distal end of the instrument. As another example, the navigation planning module may select a deployment location such that the approach angle is between approximately 30° and 90°. When selecting a deployment location, the planning system also confirms that the interventional tool is capable of extending from the catheter a sufficient distance to reach the target structure to perform the interventional procedure. As described, the planned deployment location may be located based on the analysis of the instrument operational capability, the target structure, and the patient anatomy. Alternatively or in combination with the system assessment, the planned deployment location may be identified by a clinician and communicated to the navigation planning module to locate or mark the clinician-identified planned deployment location in the model. When the navigation planning module receives the clinician-identified planned deployment location, the module may compare it with the system-identified deployment location. A visual or audible feedback cue may be issued if the clinician-identified deployment location is objectionable (e.g., "The chosen biopsy needle is not long enough to reach the target from this deployment location.")
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of using a mixed reality surgical navigation system for a medical procedure of AZIZIAN, Ryan, Fuerst and Diolaiti with comparing different the planned placement positions as seen in DONHOWE because this modification would provide appropriate visual or audible feedback (see page 17, second paragraph of DONHOWE).
Thus, the combination of AZIZIAN, Ryan, Fuerst , Diolaiti and DONHOWE teach wherein the computer readable instructions, when executed by the processor, further cause the system to: provide an evaluation of an implementation compared to the virtual guidance.
3. Claims 7-9, 38 are rejected under 35 U.S.C. 103 as being unpatentable over AZIZIAN et al, WO2019/006028 (“AZIZIAN”) in view of Ryan et al, U.S Patent Application Publication No.2018/0049622 (“Ryan”) further in view of Fuerst et al., U.S Patent Application Publication No.20210121233 (“Fuerst”) further in view of Diolaiti et al, IDS, U.S Patent Application Publication No.20090326556 (“Diolaiti”) further in view of DUINDAM et al, IDS, WO-2018129532 (“DUINDAM”)
Regarding claim 7, AZIZIAN, Ryan, Fuerst , Diolaiti teach the system of claim 1, wherein the image has an image frame of reference and the medical component has a component frame of reference and wherein the computer readable instructions, when executed by the processor, further cause the system to register the image frame of reference to the component frame of reference (see [0043] of AZIZIAN “Optionally, the display 170 includes an anatomic model image 174 generated from the anatomic model dataset. In this example, the endoscopic image 172 is overlaid, superimposed, blended or otherwise mapped onto the anatomic model image 174. The display 170 also includes an illustration 176 of the instrument 156 overlaid or superimposed on the anatomic model image 174. The illustrated instrument 176 may be generated based on pose and/or poses of instrument 156 known or determined from kinematic chain, position or shape sensor information, vision-based tracking or a combination of those. The display 170 thus provides the surgeon with additional environmental context beyond that provided by the endoscopic image 172 alone. For example, the anatomic model image 174 illustrates additional organs and bones not visible on the endoscopic image 172, and the instrument illustration 176 provides information about instrument trajectories not visible on the endoscopic image. Optionally, the display 170 includes an illustration 178 of the surgical environment external of the patient anatomy. In this example, display 170 includes the instrument illustration 176 which shows the instrument extending through an image of the patient's anatomic wall 180. The image of the patient's anatomic wall may be part of the anatomic model image 174. External to the patient's anatomic wall 174 in the external environment illustration 178 is an illustration 182 of the proximal end of the instrument 156 extending outside of the patient anatomy. The proximal end of the instrument 156 is coupled to a teleoperational manipulator arm 184 (e.g., manipulator arm 51). The external environment illustration 178 may be generated based on instrument and manipulator arm positions known or determined from kinematic chain, position or shape sensor information, visual tracking based on a camera external to the patient anatomy or a combination of those. For example, with the teleoperational manipulator, the patient, the endoscopic system, and the anatomic model all registered to a common surgical coordinate system, a composite virtual image of the surgical environment beyond the vantage point endoscopic image can be generated based on the anatomic model dataset and known kinematic and/or structural relationships of the components of the teleoperational system. The display 170 thus provides the surgeon with additional environmental context beyond that provided by the endoscopic image 172 alone. From the third vantage point of FIG. 5, the surgeon is able to observe a more expansive intra- and extra-anatomical area while still maintaining awareness of the view from the distal end of the endoscope.”; [0092] of Ryan;[0090-0091] of Diolaiti ) In addition, the same motivation is used as the rejection for claim 1. AZIZIAN, Ryan, Fuerst and Diolaiti are understood to be silent on the remaining limitations of claim 7.
In the same field of endeavor, DUINDAM teaches wherein the image has an image frame of reference and the medical component has a component frame of reference and wherein the computer readable instructions, when executed by the processor, further cause the system to register the image frame of reference to the component frame of reference (see at least [0099] as shown in Fig. 9 “ Referring back to operation 912 of FIG. 9, the computing system performing segmentation identifies these fiducial features (e.g., the axial support structure 1006, the control ring 1008, the tip ring 1010, the tool lumen 101 1, the medical tool 1012, the shape sensor lumen 1014, the shape sensor 1016) in the intraoperative image data to identify one or more portions of the medical instrument and uses the position, orientation, and/or pose of the medical instrument determined from the shape data obtained above to register the instrument reference frame to the intraoperative image reference frame. The registration may rotate, translate, or otherwise manipulate by rigid or non-rigid transforms points associated with the segmented shape and points associated with the sensed shape data. This registration between the instrument and intraoperative image frames of reference may be achieved, for example, by using an ICP technique or another point cloud registration technique. Alternatively, registration may be performed by matching and registering feature points within instrument and image point clouds where point correspondences are determined from shape similarity in some feature space. In some embodiments, the segmented shape of the medical instrument is registered to the shape data in the shape sensor frame and the associated transform (a vector applied to each of the points in the segmented shape to align with the shape data in the instrument reference frame) may then be applied to the entire three-dimensional image and/or to subsequently obtained three- dimensional images during the medical procedure. The transform may be a 6DOF transform, such that the shape data may be translated or rotated in any or all of X, Y, and Z and pitch, roll, and yaw.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to the method of using a mixed reality surgical navigation system for a medical procedures of AZIZIAN, Ryan, Fuerst and Diolaiti with registering the instrument reference frame to the image reference frame as seen in DUINDAM because this modification would use registered real-time images and prior-time anatomic images during an image-guided procedure ([0002] of DUINAM)
Thus, the combination of AZIZIAN, Ryan, Fuerst, Diolaiti and DUINAM teach wherein the image has an image frame of reference and the medical component has a component frame of reference and wherein the computer readable instructions, when executed by the processor, further cause the system to register the image frame of reference to the component frame of reference.
Regarding claim 8, AZIZIAN, Ryan, Fuerst, Diolaiti and DUINAM teach the system of claim 7, wherein registering the image frame of reference to the component frame of reference includes identifying a fiducial portion of the medical component in both the image frame of reference and the component frame of reference (see at least : [0043] of AZIZIAN “.. For example, with the teleoperational manipulator, the patient, the endoscopic system, and the anatomic model all registered to a common surgical coordinate system, a composite virtual image of the surgical environment beyond the vantage point endoscopic image can be generated based on the anatomic model dataset and known kinematic and/or structural relationships of the components of the teleoperational system. The display 170 thus provides the surgeon with additional environmental context beyond that provided by the endoscopic image 172 alone. From the third vantage point of FIG. 5, the surgeon is able to observe a more expansive intra- and extra-anatomical area while still maintaining awareness of the view from the distal end of the endoscope.”; [0092],[0105], [018] of Ryan; [0090-0091] of Diolaiti; [0099] as shown in Fig. 9 of DUINDAM “ Referring back to operation 912 of FIG. 9, the computing system performing segmentation identifies these fiducial features (e.g., the axial support structure 1006, the control ring 1008, the tip ring 1010, the tool lumen 101 1, the medical tool 1012, the shape sensor lumen 1014, the shape sensor 1016) in the intraoperative image data to identify one or more portions of the medical instrument and uses the position, orientation, and/or pose of the medical instrument determined from the shape data obtained above to register the instrument reference frame to the intraoperative image reference frame. The registration may rotate, translate, or otherwise manipulate by rigid or non-rigid transforms points associated with the segmented shape and points associated with the sensed shape data. This registration between the instrument and intraoperative image frames of reference may be achieved, for example, by using an ICP technique or another point cloud registration technique. Alternatively, registration may be performed by matching and registering feature points within instrument and image point clouds where point correspondences are determined from shape similarity in some feature space. In some embodiments, the segmented shape of the medical instrument is registered to the shape data in the shape sensor frame and the associated transform (a vector applied to each of the points in the segmented shape to align with the shape data in the instrument reference frame) may then be applied to the entire three-dimensional image and/or to subsequently obtained three- dimensional images during the medical procedure. The transform may be a 6DOF transform, such that the shape data may be translated or rotated in any or all of X, Y, and Z and pitch, roll, and yaw.” ) In addition, the same motivation is used as the rejection for claim 7.
Regarding claim 9, AZIZIAN, Ryan, Fuerst, Diolaiti and DUINAM teach he system of claim 7, wherein the computer readable instructions, when executed by the processor, further cause the system to display the virtual guidance in the image frame of reference (see at least: [0043] of AZIZIAN; [0094], [0105] [0108] of Ryan “In one exemplary embodiment of the present invention and referring to FIG. 6, the system 10 is used for hip replacement surgery wherein a first marker 600 is attached via a fixture 602 to a pelvis 604 and a second marker 606 is attached to an impactor 608. The user 106 can see the mixed reality user interface image (“MXUI”) shown in FIG. 6 via the display device 104. The MXUI provides stereoscopic virtual images of the pelvis 604 and the impactor 604 in the user's field of view during the hip replacement procedure ;[0090-0091] of Diolaiti; [0112] of DUINDAM “ Referring to operation 918, optionally, the computing system may display a representation of the preoperative model as updated in operation 916. FIG. 1 1 depicts a composite image 1 100 that includes a representation of the preoperative model according to examples of the present disclosure displayed on a display 1 10 such as that of FIG. 1. Image 1 100 may be substantially similar to image 800 in many regards. The composite image 1 100 includes a surface model 1 102 of a passageway (e.g., a bronchial passage) in which the medical instalment is located generated from the preoperative model. An internal perspective may be provided to the operator O to facilitate image guided medical procedures. The internal perspective presents a view of the model 1102 from the perspective of the distal tip of the medical instrument. For reference, a rendering 1104 of the medical instrument may be displayed. If a medical tool has been extended through the medical instrument, a rendering 1106 of the medical tool may be displayed. [0113] The composite image 1100 further includes an image of the target 1108. The target 1108 may be rendered as an opaque object, while other tissues are rendered to be semi- transparent, such that the tumor can been seen through other tissues. For example, when the target 1108 is not co-located with a wall of the model 1102, the model 1102 may be rendered semi-transparently to permit a perspective view of the target 1108. The computing system may calculate a position and orientation of the distal tip of the instrument model 1104 and may display a trajectory vector 1110 extending from the distal tip of the instrument model 1104 to the target 1108. Other information and elements may be presented in the display 110 to the operator O, such a physiological information or control elements. For example, the display 110 may include the image 600 of FIG. 6 in a window 1112.[0114] Optionally, the operator O and/or teleoperational system may use the updated preoperative model to perform an image-guided medical procedure. For example, the operator O or teleoperational system may navigate the medical instrument within the patient anatomy by steering the medical instrument based on the image 1100. Because the medical instrument is registered to the image 1100, movement of the medical instrument with respect to the patient P can be visualized by displaying corresponding movements of the displayed medical instrument 1104 within the patient anatomy represented in the image 1100. Once the medical instrument is positioned near the target, the operator O may advance a medical tool, such as the medical tool 226 of FIG. 2, through the medical instrument. The operator O may use the medical tool to perform a procedure such as surgery, biopsy, ablation, illumination, irrigation, or suction on the target, and may visualize the movement and operation of the medical tool using the rendering 1106 thereof during the procedure.”) In addition, the same motivation is used as the rejection for claim 7.
Regarding claim 38, AZIZIAN, Ryan, Fuerst, Diolaiti teach the system of claim 37, wherein the image has an image frame of reference registered to a manipulator frame of reference of the robot-assisted manipulator assembly (see [0043] of AZIZIAN “Optionally, the display 170 includes an anatomic model image 174 generated from the anatomic model dataset. In this example, the endoscopic image 172 is overlaid, superimposed, blended or otherwise mapped onto the anatomic model image 174. The display 170 also includes an illustration 176 of the instrument 156 overlaid or superimposed on the anatomic model image 174. The illustrated instrument 176 may be generated based on pose and/or poses of instrument 156 known or determined from kinematic chain, position or shape sensor information, vision-based tracking or a combination of those. The display 170 thus provides the surgeon with additional environmental context beyond that provided by the endoscopic image 172 alone. For example, the anatomic model image 174 illustrates additional organs and bones not visible on the endoscopic image 172, and the instrument illustration 176 provides information about instrument trajectories not visible on the endoscopic image. Optionally, the display 170 includes an illustration 178 of the surgical environment external of the patient anatomy. In this example, display 170 includes the instrument illustration 176 which shows the instrument extending through an image of the patient's anatomic wall 180. The image of the patient's anatomic wall may be part of the anatomic model image 174. External to the patient's anatomic wall 174 in the external environment illustration 178 is an illustration 182 of the proximal end of the instrument 156 extending outside of the patient anatomy. The proximal end of the instrument 156 is coupled to a teleoperational manipulator arm 184 (e.g., manipulator arm 51). The external environment illustration 178 may be generated based on instrument and manipulator arm positions known or determined from kinematic chain, position or shape sensor information, visual tracking based on a camera external to the patient anatomy or a combination of those. For example, with the teleoperational manipulator, the patient, the endoscopic system, and the anatomic model all registered to a common surgical coordinate system, a composite virtual image of the surgical environment beyond the vantage point endoscopic image can be generated based on the anatomic model dataset and known kinematic and/or structural relationships of the components of the teleoperational system. The display 170 thus provides the surgeon with additional environmental context beyond that provided by the endoscopic image 172 alone. From the third vantage point of FIG. 5, the surgeon is able to observe a more expansive intra- and extra-anatomical area while still maintaining awareness of the view from the distal end of the endoscope.”; [0094], [0105] [0108] of Ryan; [0025] of Fuers “A surgical robotic arm can have movable, jointed, and/or motorized members with multiple degrees of freedom that can hold various tools or appendages at distal ends. Example systems include the da Vinci® Surgical System which can be used for minimally invasive surgery (e.g., urologic surgical procedures, general laparoscopic surgical procedures, gynecologic laparoscopic surgical procedures, general non-cardiovascular thoracoscopic surgical procedures and thoracoscopically assisted cardiotomy procedures). A “virtual surgical robotic arm” can be a computer generated representation of a robotic arm rendered over the captured video of a user setup. The virtual surgical robotic arm can be a complex 3D model of the real robotic arm. Alternatively or additionally a virtual surgical robotic arm can include visual aids such as arrows, tool tips, or other representation relating to providing pose information about a robotic arm such as a geometrically simplified version of the real robotic arm.” [0034] Alternatively or additionally, the kinematics processor can generate the kinematic parameters based on robot control data (e.g. motor/actuator commands and feedback, servo commands, position data, and speed data) generated by robot controls 112. The robot control data can be generated during an actual surgical procedure using a real surgical robotic arm and/or surgical tool 108, or a simulated surgical procedure (e.g., a test run), also using a real surgical robotic arm and/or tool.”) In addition, the same motivation is used as the rejection for claim 1. AZIZIAN, Ryan, Fuerst, Diolaiti are understood to be silent on the remaining limitations of claim 38.
In the same field of endeavor, DUINDAM teaches wherein the image has an image frame of reference registered to a manipulator frame of reference of the robot-assisted manipulator assembly (see at least[0099] as shown in Fig. 9 of DUINDAM “ Referring back to operation 912 of FIG. 9, the computing system performing segmentation identifies these fiducial features (e.g., the axial support structure 1006, the control ring 1008, the tip ring 1010, the tool lumen 101 1, the medical tool 1012, the shape sensor lumen 1014, the shape sensor 1016) in the intraoperative image data to identify one or more portions of the medical instrument and uses the position, orientation, and/or pose of the medical instrument determined from the shape data obtained above to register the instrument reference frame to the intraoperative image reference frame. The registration may rotate, translate, or otherwise manipulate by rigid or non-rigid transforms points associated with the segmented shape and points associated with the sensed shape data. This registration between the instrument and intraoperative image frames of reference may be achieved, for example, by using an ICP technique or another point cloud registration technique. Alternatively, registration may be performed by matching and registering feature points within instrument and image point clouds where point correspondences are determined from shape similarity in some feature space. In some embodiments, the segmented shape of the medical instrument is registered to the shape data in the shape sensor frame and the associated transform (a vector applied to each of the points in the segmented shape to align with the shape data in the instrument reference frame) may then be applied to the entire three-dimensional image and/or to subsequently obtained three- dimensional images during the medical procedure. The transform may be a 6DOF transform, such that the shape data may be translated or rotated in any or all of X, Y, and Z and pitch, roll, and yaw.” …[0104] Referring next to operation 914, the preoperative image reference frame of the preoperative model is registered to the intraoperative image reference frame. As described above, the computing system may register the preoperative image reference frame to the instrument reference frame in operation 908 and register the instrument reference frame to the intraoperative image reference frame in operation 912. Accordingly, registration of the preoperative image reference frame to the intraoperative image reference frame may be performed using the common frame of reference i.e., the instrument reference frame”). In addition, the same motivation is used as the same rejection for claim 7.
Thus, the combination of AZIZIAN, Ryan, Fuerst, Diolaiti and DUINAM teaches wherein the image has an image frame of reference registered to a manipulator frame of reference of the robot-assisted manipulator assembly.
4. Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over AZIZIAN et al, WO2019/006028 (“AZIZIAN”) in view of Ryan et al, U.S Patent Application Publication No.2018/0049622 (“Ryan”) further in view of Fuerst et al., U.S Patent Application Publication No.20210121233 (“Fuerst”) further in view of Diolaiti et al, IDS, U.S Patent Application Publication No.20090326556 (“Diolaiti”) further in view of further in view of KOENIG et al, Patent Application Publication No.2018/0079090 (“KOENIG”)
Regarding claim 11, AZIZIAN, Ryan, Fuerst, Diolaiti teach the system of claim 1, wherein the second configuration (see at least [0022];[0027] of AZIZIAN FIG. IB is a perspective view of one embodiment of a teleoperational assembly 12 which may be referred to as a patient side cart. The patient side cart 12 shown provides for the manipulation of three surgical tools 30a, 30b, 30c (e.g., instrument systems 14) and an imaging device 28 (e.g., endoscopic imaging system 15), such as a stereoscopic endoscope used for the capture of images of the site of the procedure. The imaging device may transmit signals over a cable 56 to the control system 20. Manipulation is provided by teleoperative mechanisms having a number of joints. The imaging device 28 and the surgical tools 30a-c can be positioned and manipulated through incisions in the patient or through a natural orifice (e.g., oral cavity) so that a kinematic remote center is maintained at the entry to minimize the size of the incision or to avoid damage to the natural orifice boundaries. Images of the surgical environment within the patient anatomy can include images of the distal ends of the surgical tools 30a-c when they are positioned within the field-of-view of the imaging device 28. [0028] The patient side cart 22 includes a drivable base 58. The drivable base 58 is connected to a telescoping column 57, which allows for adjustment of the height of the arms 54. The arms 54 may include a rotating joint 55 that both rotates and moves up and down. Each of the arms 54 may be connected to an orienting platform 53. The orienting platform 53 may be capable of 360 degrees of rotation. The patient side cart 22 may also include a telescoping horizontal cantilever 52 for moving the orienting platform 53 in a horizontal direction. [0038] ; [0096] of Ryan “FIG. 7 depicts an alternate view of the MXUI previously shown in FIG. 6, wherein a virtual target 700 and a virtual tool 702 are presented to the user 106 for easy use in achieving the desired version and inclination. In this embodiment, further combinations of virtual reality are used to optimize the natural feeling experience for the user by having a virtual target 700 with actual tool 702 fully visible or a virtual tool (not shown) with virtual target fully visible. Other combinations of real and virtual imagery can optionally be provided. Presentation of data can be in readable form 704 or in the form of imagery including but not limited to 3D representations of tools or other guidance forms”; Fig. 5 of Fuerst, Fig. 2 of ) In addition, the same motivation is used as the rejection for claim 1. AZIZIAN, Ryan, Fuerst and Diolaiti are understood to be silent on the remaining limitations of claim 11.
In the same field of endeavor, KOENIG teaches wherein the second configuration is a stowage configuration (see at least [0066] The various links in the robotic arm may be arranged in any number of predetermined configurations for different purposes. For instance, a robotic arm (e.g., a variation with offset axes for spherical roll, spherical pitch, and instrument rotation, as described above with reference to FIG. 1F) may be arranged in a compact, folded configuration, such as for stowage under a surgical table, storage, and/or transport. The folded arm configuration may also incorporate the folding, retraction, or other compact storage of components coupled to the robotic arm, such as a table adapter coupling the robotic arm to a surgical patient table, cart, or other surface ..”); [0067] FIGS. 7A and 7B illustrate exemplary variations of robotic arms (similar to robotic arm 500 described above with reference to FIGS. 5A-5C) arranged in an exemplary folded configuration underneath a surgical patient table. …In some variations, for example, the stowage configuration of an arm shown in FIGS. 7A and 7B may occupy a volume of generally between about 8 and about 12 inches high (along the vertical height of the table), between about 8 and about 12 inches wide (along the width of the table), and between about 18 and 22 inches long (along the longitudinal length of the table). In one exemplary variations, for example, the stowage configuration of an arm may occupy a volume of about 10 inches high, about 10 inches wide, and about 20 inches long.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to the method of using a mixed reality surgical navigation system for a medical procedures of AZIZIAN, Ryan, Fuerst and Diolaiti with applying stowable configuration as seen in KOENIG because this modification would arrange in a compact, folded configuration ([0066])
Thus, the combination of AZIZIAN, Ryan, Fuerst, Diolaiti, KOENIG wherein the second configuration is a stowage configuration.
5. Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over AZIZIAN et al, WO2019/006028 (“AZIZIAN”) in view of Ryan et al, U.S Patent Application Publication No.2018/0049622 (“Ryan”) further in view of Fuerst et al., U.S Patent Application Publication No.20210121233 (“Fuerst”) further in view of Diolaiti et al, IDS, U.S Patent Application Publication No.20090326556 (“Diolaiti”) further in view of Lau et al, U.S Patent Application Publication No.2021/0153966 (“Lau”)
Regarding claim 12, AZIZIAN, Ryan, Fuerst, Diolaiti teach the system of claim 1, wherein the second configuration is a draping configuration (see at least [0022];[0027] , [0038]of AZIZIAN FIG. IB is a perspective view of one embodiment of a teleoperational assembly 12 which may be referred to as a patient side cart. The patient side cart 12 shown provides for the manipulation of three surgical tools 30a, 30b, 30c (e.g., instrument systems 14) and an imaging device 28 (e.g., endoscopic imaging system 15), such as a stereoscopic endoscope used for the capture of images of the site of the procedure. The imaging device may transmit signals over a cable 56 to the control system 20. Manipulation is provided by teleoperative mechanisms having a number of joints. The imaging device 28 and the surgical tools 30a-c can be positioned and manipulated through incisions in the patient or through a natural orifice (e.g., oral cavity) so that a kinematic remote center is maintained at the entry to minimize the size of the incision or to avoid damage to the natural orifice boundaries. Images of the surgical environment within the patient anatomy can include images of the distal ends of the surgical tools 30a-c when they are positioned within the field-of-view of the imaging device 28. [0028] The patient side cart 22 includes a drivable base 58. The drivable base 58 is connected to a telescoping column 57, which allows for adjustment of the height of the arms 54. The arms 54 may include a rotating j oint 55 that both rotates and moves up and down. Each of the arms 54 may be connected to an orienting platform 53. The orienting platform 53 may be capable of 360 degrees of rotation. The patient side cart 22 may also include a telescoping horizontal cantilever 52 for moving the orienting platform 53 in a horizontal direction”; [0096] of Ryan; [0029] of Fuerst, Figs 1-2 of Dio ). In addition, the same motivation is used as the rejection for claim 1. AZIZIAN, Ryan, Fuerst, Diolaiti rare understood to be silent on the remaining limitations of claim 12.
In the same field of endeavor, Lau teaches wherein the second configuration is a draping configuration (see at least [0151] Additionally, the adjustable arm support 250 may also be draped. The adjustable arm support 250 can be particularly challenging as the adjustable arm support 250 can support a plurality of robotic arms 210 which can linearly translate relative to the length of the adjustable arm support 250. To maintain sterility, the robotic arms 210 and the adjustable arm support 250 can be draped simultaneously and with a single drape 300. However, draping both the plurality of robotic arms and the adjustable arm support adds complexity to the shape and design of the drape configuration, in particular in designing the drape to maintain sterility during the draping process. The drape 300 should be able to accommodate the motion of the plurality of robotic arms 210 linearly relative to the length of the adjustable arm support 250, as well motion of each robotic arm 210 in several degrees of motion. As will be described more below, the drape 300 can be multiple times longer (e.g., at least two, three, or four times longer) than the adjustable arm support 250 to accommodate the motion of the plurality of robotic arms 210 relative to the surface of the adjustable arm support 250.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to the method of using a mixed reality surgical navigation system for a medical procedures of AZIZIAN, Ryan, Fuerst, Diolaiti with applying drape configuration as seen in Lau because this modification would maintain sterility ([0151] of Lau)
Thus, the combination of AZIZIAN, Ryan, Fuerst, Diolaiti, Lau teaches wherein the second configuration is a draping configuration.
6. Claims 13 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over AZIZIAN et al, WO2019/006028 (“AZIZIAN”) in view of Ryan et al, U.S Patent Application Publication No.2018/0049622 (“Ryan”) further in view of Fuerst et al., U.S Patent Application Publication No.20210121233 (“Fuerst”) further in view of Diolaiti et al, IDS, U.S Patent Application Publication No.20090326556 (“Diolaiti”) further in view of White et al, U.S Patent Application Publication No.2018/0082480 (“White”)
Regarding claim 13, AZIZIAN, Ryan, Fuerst, Diolaiti teach the system of claim 1, wherein the virtual animation (see at least: [0038] of AZIZIAN; 0094]-[0096] of Ryan; [0052],[0074],[0081-0082] of Diolaiti). AZIZIAN, Ryan, Fuerst, Diolaiti are understood to be silent on the remaining limitations of claim 13.
In the same field of endeavor, White teaches wherein the virtual animation includes a procedure step (see at least abstract “A system and method for using augmented reality device for use during a surgical procedure are described. A system may include an augmented reality device to present a virtual indication, such as a virtual indication of a surgical instrument, a force vector, a direction, or the like. The augmented reality device may present a virtual aspect of a procedure, such as a virtual animation of a step of a procedure. The augmented reality device may present a virtual object, indication, aspect, etc., within a surgical field while permitting the surgical field or aspects of the surgical field to be viewed through the augmented reality display (e.g., presenting virtual objects mixed with real objects). “; [0042] The contents of some cases may require assembly before a surgical procedure begins. The operator may be prompted on the AR display with instructions for opening the case and locating pieces for the item to be assembled. The AR display may present three-dimensional representations of the pieces or the location of the pieces, such displaying a three-dimensional representation of the piece above where the piece is physically located. Through object recognition or scanning identification tags on a piece (e.g., a bar code, a QR code), the AR device may determine that all necessary pieces have been located. The AR display may present instructions for assembling the item. The instruction may include a list of textual steps, animations, video, or the like. The instructions may be animated steps using the three-dimensional representations of the pieces. When the operator completes the assembly, the AR display may provide a three-dimensional representation of the assembled item such that the operator may confirm the physical assembled item resembles the representation. The operator may then be provided instructions on the AR display for the placement of the assembled item with the surgical procedure room.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of using a mixed reality surgical navigation system for a medical procedures of AZIZIAN, Ryan, Fuerst, Diolaiti with present virtual animation of a step of a procedure as seen in White because this modification would improve surgical outcomes and customize surgery for a patient ([0004] of White)
Thus, the combination of AZIZIAN, Ryan, Fuerst, Diolaiti, White teaches wherein the virtual animation includes a procedure step.
Regarding claim 15, AZIZIAN, Ryan, Fuerst, Diolaiti teach the system of claim 14, wherein the virtual animation includes the medical component and the auxiliary component (see at least [0038] of AZIZIAN ; [0096], [018] of Ryan, Fig.5 of Fuerst, [0052],[0074],[0081-0082] of Diolaiti). AZIZIAN, Ryan, Fuerst, Diolaiti, are understood to be silent on the remaining limitations of claim 15.
In the same field of endeavor, White teaches wherein the virtual animation includes a set-up procedure for the medical component and the auxiliary component (see at least abstract “A system and method for using augmented reality device for use during a surgical procedure are described. A system may include an augmented reality device to present a virtual indication, such as a virtual indication of a surgical instrument, a force vector, a direction, or the like. The augmented reality device may present a virtual aspect of a procedure, such as a virtual animation of a step of a procedure. The augmented reality device may present a virtual object, indication, aspect, etc., within a surgical field while permitting the surgical field or aspects of the surgical field to be viewed through the augmented reality display (e.g., presenting virtual objects mixed with real objects).”;[0046]; [0051],[0054] The technique 500 may include displaying patient or procedure information, for example in a heads-up portion of the AR display. In an example, a series of virtual components may be presented, such as to provide instruction for assembly of the surgical instrument. The series of virtual components may be displayed in progression automatically, such as in response to detecting (e.g., automatically using a detection device or from a user input) that an operation corresponding to one of the series of virtual components is complete. The system may be able to automatically detect the different instruments being used during a procedure by shape recognition, RFID tags, bar codes, or similar tagging or recognition methods.”; [0042] The contents of some cases may require assembly before a surgical procedure begins. The operator may be prompted on the AR display with instructions for opening the case and locating pieces for the item to be assembled. The AR display may present three-dimensional representations of the pieces or the location of the pieces, such displaying a three-dimensional representation of the piece above where the piece is physically located. Through object recognition or scanning identification tags on a piece (e.g., a bar code, a QR code), the AR device may determine that all necessary pieces have been located. The AR display may present instructions for assembling the item. The instruction may include a list of textual steps, animations, video, or the like. The instructions may be animated steps using the three-dimensional representations of the pieces. When the operator completes the assembly, the AR display may provide a three-dimensional representation of the assembled item such that the operator may confirm the physical assembled item resembles the representation. The operator may then be provided instructions on the AR display for the placement of the assembled item with the surgical procedure room.”) In addition, the same motivation is used as the rejection for claim 13.
Thus, the combination of AZIZIAN, Ryan, Fuerst, Diolaiti, and White teaches wherein the virtual animation includes a set-up procedure for the medical component and the auxiliary component.
7. Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over AZIZIAN et al, WO2019/006028 (“AZIZIAN”) in view of Ryan et al, U.S Patent Application Publication No.2018/0049622 (“Ryan”) further in view of Fuerst et al., U.S Patent Application Publication No.20210121233 (“Fuerst”) further in view of Diolaiti et al, IDS, U.S Patent Application Publication No.20090326556 (“Diolaiti”) further in view of LEVIN et al, U.S Patent Application Publication No.2023/0363821 (“LEVIN”)
Regarding claim 16, AZIZIAN, Ryan, Fuerst, Diolaiti teach the system of claim 1, wherein the virtual animation includes the medical component (see at least [0038] of AZIZIAN ;[0108] of Ryan “FIG. 24 depicts an exemplary embodiment of a MXUI shown to the user 106 via the display device 104 during positioning of the acetabular shell of a hip replacement procedure wherein a virtual target 2400 for the acetabular impactor assembly 1100 and a virtual shell 2402 are shown. Placement of the acetabular impactor assembly 1100 is guided by manipulating it to align with the virtual target 2400. The posterior/lateral quadrant of the shell portion of the virtual target may be displayed in a different color or otherwise visually differentiated from the rest of the shell 2402 to demarcate to the user 106 a target for safe placement of screws into the acetabulum. The numerical angle of the acetabular impactor and the depth of insertion relative to the reamed or un-reamed acetabulum are displayed numerically as virtual text 2404. A magnified stereoscopic image (not shown) similar to 2202 centered on the tip of the impactor may be displayed showing how the virtual shell interfaces with the acetabulum of the virtual pelvis 2102.”; [0052],[0074],[0081-0082] of Diolaiti). AZIZIAN, Ryan, Fuerst, Diolaiti are understood to be silent on the remaining limitation of claim 16.
In the same field of endeavor, LEVIN teaches wherein the virtual animation includes a patient and the medical component(see at least [0143] At optional step 306, an animation of a patient on a patient bed in the procedure room may be displayed. In some embodiments, the patient may be displayed in the animation already having an automated (robotic) medical device mounted thereon, or in close proximity, thereto. [0162] According to some embodiments, at optional step 418, the method may include displaying an animation of a virtual instrument being steered by a virtual robot to the next checkpoint or to the target. According to some embodiments, the animation may include the selected medical instrument. According to some embodiments, the animation may be stored in a memory module, for example memory module 108. According to some embodiments, the animation may be generated in real-time using algorithm(s). According to some embodiments, the animation may include a virtual medical instrument being steered by a virtual automated medical device from one checkpoint to the next along the trajectory. According to some embodiments, the method may include displaying the advancement of the instrument until target is reached. According to some embodiments, the animation of the virtual instrument being advanced may be shown on a cross-sectional view of the virtual patient, which may correspond to the image-view presented on the display.”; [0200] FIG. 7F is a screenshot of GUI 70 and an animation window 72, which shows an animation of a patient lying on the patient bed with an automated device 770, for example automated device 60 shown in FIG. 6A, attached to the patient's body. According to some embodiments, as shown in FIG. 7F, the automated device 770 may be attached to the patient's body using an attachment apparatus 775, such as the mounting base/attachment frame disclosed in abovementioned U.S. Pat. No. 11,103,277 or U.S. Patent Application Publication No. 2021/228,311.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of using a mixed reality surgical navigation system for a medical procedures of AZIZIAN, Ryan, Fuerst, Diolaiti with displaying virtual animation including the patient and the medical instrument as seen in LEVIN because this modification would increase efficiency and safety of the medical procedures involved ([0007] of LEVIN)
Thus, the combination of AZIZIAN, Ryan, Fuerst, Diolaiti and LEVIN teaches wherein the virtual animation includes a patient and the medical component.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SARAH LE whose telephone number is (571)270-7842. The examiner can normally be reached Monday: 8AM-4:30PM EST, Tuesday: 8 AM-3:30PM EST, Wednesday: 8AM-2:30PM EST, Thursday and Friday off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SARAH LE/Primary Examiner, Art Unit 2614
Prosecution Insights