DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 7 is objected to because of the following informalities: “a detected activation in the field of view” should be changed to “a detected activation of a tool in the field of view” to clarify the meaning of the “activation”. The only explicit references to any activation in the described embodiments are found in paragraphs 31 and 34, both of which describe activated tools. Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-6, 8-10 and 12-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by U.S. Pat. Appl. Pub. No. 20200054399 to Duindam et al. (hereinafter “Duindam”)
Regarding claim 1, Duindam teaches a medical system (Duindam, par. 37, “teleoperated medical system 100”) comprising:
a display system (Duindam, par. 44, “Display system 110”) for displaying a display area (Duindam, par. 44, “Display system 110 may also display an image of the surgical site and medical instruments captured by the visualization system. ... By true presence, it is meant that the presentation of an image is a true perspective image simulating the viewpoint of a physician that is physically manipulating medical instrument 104.”); and
a control system (Duindam, par. 43, “control system 112.”), wherein the control system includes a processing unit including one or more processors (Duindam, par. 43, “The visualization system may be implemented as hardware, firmware, software or a combination thereof which interact with or are otherwise executed by one or more computer processors, which may include the processors of a control system 112.”), and wherein the processing unit is configured to:
generate an image (Duindam, par. 43, “a concurrent or real-time image of a surgical site”, par. 84, “Local views 530 may include a virtual distal view 533 that displays a rendering of the 3D model from the perspective of the distal end of the catheter.”) of a field of view for display in the display area (Duindam, par. 43, “medical instrument 104 may have a visualization system (discussed in more detail below), which may include a viewing scope assembly that records a concurrent or real-time image of a surgical site and provides the image to the operator or physician O through one or more displays of medical system 100, such as one or more displays of display system 110.”);
generate a user interface component (Anything in a graphical user interface (GUI) is a user interface element. See Duindam at title. A graphical element, e.g., cross hair 150, uncertainty zone 1345, or semi solid sphere 1340, displayed via a GUI. See Duindam at par. 120) for display in the display area (Duindam, par. 67, “The live camera feed and/or other real-time navigation information may be displayed to the clinician via a graphical user interface.”);
determine a priority record (A database structure that prioritizes one or more graphic elements, e.g., when the target is within a threshold distance “where the appearance of virtual distal view 1300 is modified to display the passageway walls as transparent [or semi-transparent such] that a target is visible.” See Duindam at par. 120) for an element in the image of the field of view (Duindam, par. 120, “target”);
determine a display characteristic for the user interface component (A display characteristic includes any feature, quality, appearance, layout, size and so forth, that is programmatically or deterministically created by the computer executing the GUI, such as graphical objects being rendered in a semi solid, transparent or semi-transparent appearance based on what they represent as configured by the user and otherwise determined by the program itself. See, e.g., Duindam at par. 120, “transparent” or “semi-transparent”) based on the priority record of the element in the field of view (Duindam, par. 120, “transparent” or “semi-transparent”); and
PNG
media_image1.png
612
500
media_image1.png
Greyscale
Duindam, Figure 13B showing distal view 1300 that includes target indicator 1320, hazard distance indicator 1330, target 1340, uncertainty zone 1345 and cross hair 1350.
display the user interface component in accordance with the display characteristic in the display area overlayed on a displayed image of the field of view (Duindam, par. 120, “When the catheter is positioned within a predetermined threshold distance from the target, the appearance and/or configuration of virtual distal view 1300 can transition to provide directional guidance towards the target. In particular, the target may be embedded within the passageway wall relative to the distal end of the catheter and thus not visible as shown in FIG. 13A. FIG. 13B displays an example of the virtual distal view when the distal end of the catheter is positioned within a threshold distance to the target, as indicated by the distance to target indicator 1320. In this example, 3.1 cm is within a threshold where the appearance of virtual distal view 1300 is modified to display the passageway walls as transparent (or semi-transparent such) that a target is visible. In other examples, the threshold distance can vary based on procedure, type of tool, target size, target location, etc. As shown in FIG. 13B, the target is shown as a semi solid sphere 1340 while an uncertainty zone 1345 is shown as a semi-transparent sphere. Additionally, a cross hair 1350 can be provided which indicates the direction the catheter is facing.”).
Regarding claim 2, Duindam teaches the medical system of claim 1, wherein the user interface component includes at least one of a menu (Duindam, par. 72, “As depicted in FIG. 4, graphical user interface 400 includes an upper portion that displays one or more global views 410, a middle portion that displays one or more compact views 420, and a lower portion that displays one or more local views 430.”), an alert (Duindam, par. 80, “Message bar 542 displays one or more messages which may be used to inform the clinician of the next steps in the procedure (e.g., “Confirm registration on controller screen to start navigating to target”)”), an indicator (Duindam, par. 119, “virtual distal view 1300 can include a target distance indicator 1320 and a hazard distance indicator 1330. Target distance indicator 1320 can provide the distance from the distal end of the catheter to the target.”), a digital tool (Providing a measured distance with the target distance indicator 1320 makes it a tool.), or an auxiliary image (Duindam, 84, “Local views 530 may include a virtual distal view 533 that displays a rendering of the 3D model from the perspective of the distal end of the catheter.”).
Regarding claim 3, Duindam teaches the medical system of claim 1, wherein the user interface component includes modified image data based on an image analysis of image content (Duindam, par. 116, “supplemental guidance information 1210 may be generated by identifying features in the images using image processing techniques, by determining the location of the camera in the anatomy using shape data from the catheter, and/or the like.”).
Regarding claim 4, Duindam teaches the medical system of claim 1, wherein the priority record includes a ranking indicator of priority (An arrow pointing towards a planned route, an arrow that appears when approaching a bifurcation and otherwise being hidden and a color-coded reticle indicating a branch to steer towards or avoid, are all indicators of decision ranking, e.g., which path is prioritized for a successful outcome by following the planned route or which path is prioritized to avoid, or a distance-based target indicating arrow prioritizing one path over another. See Duindam at par. 116).
Regarding claim 5, Duindam teaches the medical system of claim 1, wherein the priority record is determined for an anatomic structure in the image of the field of view (The target is prioritized for view by being displayed as semi solid within an uncertainty zone that is semi-transparent and walls that can be made transparent. See Duindam at par. 120 and Fig. 13B).
Regarding claim 6, Duindam teaches the medical system of claim 1, wherein the priority record is determined for a tool in the image of the field of view (Duindam, par. 116, “identifying features in the images using image processing techniques, by determining the location of the camera in the anatomy using shape data from the catheter, and/or the like. As depicted in FIGS. 12A-12B, supplemental guidance information 1210 is displayed using a reticle that includes a circle with an arrow. As the camera approaches a bifurcation, the arrow points towards the branch that corresponds to the planned route.”).
Regarding claim 8, Duindam teaches the medical system of claim 1, wherein determining the priority record for the element in the image of the field of view includes distinguishing a plurality of elements in the image of the field of view, including the element in the field of view (The distal view 1300 includes distance indicator 1320 and hazard indicator 1330, which are distinguished from the semi solid sphere 1340, i.e., the target, the uncertainty zone 1345 and the cross hair 1350 by being arranged outside of the virtual distal view, i.e., the rectangular border of the virtual view is a solid line, which distinguishes the virtual view from the indicators. The graphical shapes within the virtual view are distinguished from each other via their distinct shapes. See Duindam at FIG. 13B).
Regarding claim 9, Duindam teaches the medical system of claim 8, wherein distinguishing the plurality of elements includes identifying a boundary between the plurality of elements (The virtual view is presented with a clear boundary, which is maintained as the user navigates throughout the planned trajectory. The uncertainty zone 1345 is a semi-transparent sphere, which permits the semi solid sphere 1340 indicating the target position to be visible . The zone is a boundary that segments an anatomical region likely to be the target from another anatomical region likely not to be the target. See Duindam at FIG. 13B).
Regarding claim 10, Duindam teaches the medical system of claim 8, wherein distinguishing the plurality of elements includes receiving a graphical segmentation of the image of the field of view (The virtual view is presented with a clear boundary, which is maintained as the user navigates throughout the planned trajectory. The uncertainty zone 1345 is a semi-transparent sphere, which permits the semi solid sphere 1340 indicating the target position to be visible . The zone is a boundary that segments an anatomical region likely to be the target from another anatomical region likely not to be the target. See Duindam at FIG. 13B).
Regarding claim 12, Duindam teaches the medical system of claim 1, wherein determining a priority record for the element in the image of the field of view includes receiving a priority assignment (Selecting specific targets within the constructed 3D model prioritizes those targets over the non-selected candidate targets. See Duindam at par. 68).
Regarding claim 13, Duindam teaches the medical system of claim 1, wherein determining a priority record for the element in the image of the field of view includes receiving an indication of priority based on an image analysis of the image of the field of view (Duindam, par. 116, “supplemental guidance information 1210 may be generated by identifying features in the images using image processing techniques, by determining the location of the camera in the anatomy using shape data from the catheter, and/or the like. As depicted in FIGS. 12A-12B, supplemental guidance information 1210 is displayed using a reticle that includes a circle with an arrow. As the camera approaches a bifurcation, the arrow points towards the branch that corresponds to the planned route.” An arrow is an indication of the more important path to follow to comply with the surgical plan.).
Regarding claim 14, Duindam teaches the medical system of claim 13, wherein the image analysis includes an identification of a model priority record (Duindam, par. 116, “supplemental guidance information 1210 may be generated by identifying features in the images using image processing techniques, by determining the location of the camera in the anatomy using shape data from the catheter, and/or the like. As depicted in FIGS. 12A-12B, supplemental guidance information 1210 is displayed using a reticle that includes a circle with an arrow. As the camera approaches a bifurcation, the arrow points towards the branch that corresponds to the planned route.” The arrow indicates the more important path to follow to comply with the surgical plan, which is determined from the 3D model that registers image data to target and other anatomical feature locations. See Duindam at par. 68.) from a model co-registered with the image of the field of view (Duindam, par. 68, “One or more of these steps may be performed on the same robotic catheter system used to perform the biopsy, on a different medical instrument system, on a standalone processor, such as a workstation dedicated to pre-operative planning, and/or the like. The plan for the biopsy procedure may be saved (e.g., as one or more digital files) and transferred to the robotic catheter system used to perform the biopsy procedure. The saved plan may include the 3D model, identification of airways, target locations, paths to target locations, and/or the like.”).
Regarding claim 15, Duindam teaches the medical system of claim 1,
wherein the field of view includes a plurality of elements including the element, which is a first element (target indicated at 1340. See Duindam at Fig. 13B), and a second element (Live remote view 1510. See id.),
wherein the first element has a higher priority value than the second element (The target has a higher priority because it is given a special indicator (1320). See Duindam at Fig. 13B.), and
wherein displaying the user interface component (e.g., cross hair 1350. See Duindam at Fig. 5C) with the display characteristic in the display area includes displaying the user interface component overlayed on the second element (The crosshair is displayed at the front of the stack of graphical elements overlaying the virtual view. See Duindam at Figs. 5C and 15C) based on the first element's higher priority value (Duindam, par. 120, “Cross hair 1350 could be automatically centered in the view such that the view is always showing the distal view from the catheter centered in the image. In one example, cross hair 1350 would always be displayed showing the center forward facing direction of the catheter. In alternative examples, cross hair 1350 would automatically appear as the catheter approaches the target within a pre-defined threshold.”).
Regarding claim 16, Duindam teaches the medical system of claim 1, wherein the processing unit is further configured to determine a procedure characteristic (e.g., drive force 543. See Duindam at par. 81) and wherein determining the display characteristic is further based on the procedure characteristic (Duindam, par. 85, “graphical user interface 500 may automatically transition from the configuration depicted in FIG. 5B to the alternative configuration depicted in FIG. 5C in response to detecting that the catheter is in proximity of the target location within a threshold and/or is aligned with the target location within a threshold.”).
Regarding claim 17, Duindam teaches the medical system of claim 16, wherein the procedure characteristic is a type of procedure being performed in the field of view (Duindam, Fig. 5C, “continue with Ebus”, Fig. 5D, “Next Step Biopsy”).
Regarding claim 18, Duindam teaches the medical system of claim 16, wherein the procedure characteristic is associated with an instrument in the image of the field of view (Duindam. 5D, “Next Step Biopsy”).
Regarding claim 19, Duindam teaches the medical system of claim 1, wherein the display characteristic includes a display of a transparency level or blurring level of the user interface component (Duindam, par. 120, “3.1 cm is within a threshold where the appearance of virtual distal view 1300 is modified to display the passageway walls as transparent (or semi-transparent such) that a target is visible. In other examples, the threshold distance can vary based on procedure, type of tool, target size, target location, etc.”).
Regarding claim 20, Duindam teaches the medical system of claim 1, wherein the display characteristic includes an outline of edge demarcation of a structure in the field of view obstructed by the user interface component (Fig. 15A of Duindam, for example, shows a live probe image overlaying a live remote image, where the live probe image has a defined border or edge that demarcates or sets the boundaries of the graphical object that represents the live remote image).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 7 is rejected again, but now under 35 U.S.C. 103 as being obvious over Duindam in view of U.S. Pat. Appl. Pub. No. 20220336097 to Shelton, IV et al. (hereinafter “Shelton”).
Regarding claim 7, Duindam teaches the medical system of claim 1, wherein the priority record is dynamic based on a detected activation (Duindam, par. 67, “A catheter of the robotic catheter system provides a conduit for tools such as endoscopes, endobronchial ultrasound (EBUS) probes, therapeutic tools, and/or biopsy tools to be delivered to locations within the airways where one or more targets of the lung biopsy, such as lesions, nodules, tumors, and/or the like, are present. When the catheter is driven through anatomy, typically an endoscope is installed such that a clinician, such as surgeon O, can monitor a live camera feed of a distal end of the catheter. The live camera feed and/or other real-time navigation information may be displayed to the clinician via a graphical user interface.”, par. 125, “The clinician may adjust the position of the catheter while monitoring live remote image 1510 and/or live probe image 1520.”. The catheter is robotic and therefore activated when commanded to move, thereby changing the live view and the associated graphical navigation information.), but does not teach that which is explicitly taught by Shelton.
Shelton teaches a priority record is dynamic based on a detected activation in the field of view of a tool (Shelton, par. 105, “the augmented reality modeler 86 may receive images of the instrument in a surgical field, such as from the camera 79 or sensor 90, and create the augmented reality environment to fit within a display image of the surgical field of view.”, par. 161, “As illustrated in FIG. 16, the method 6040 includes detecting 6041 an attempt by the user to activate the surgical instrument 21. If 6042 the needed parameter is received, a low display-priority value is assigned 6043. If 6042, however, the trocar-anvil connection is not detected, a high display-priority value is assigned 6044 to the parameters.”, par. 162, “the parameter is a sensor parameter, which can be an internal sensor of the surgical instrument 21, or any other sensor, configured to measure a parameter needed for proper operation of the surgical procedure.”).
Duindam discloses a user interface that supports surgical operations based on priorities of the particular surgical procedure being carried out in a live feed displayed in the user interface along with dynamically-displayed navigation and other surgery-related data to assist a surgeon in performing a surgical operation. Thus, Duindam shows that it was known in the art before the effective filing date of the claimed invention to change the layout and presentation of a user interface for surgical robotic operations based on priorities, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, providing an intuitive graphical user interface to support robotic surgeries. Shelton discloses a user interface embodied in a head-worn VR/MR device. Like Duindam, Shelton discloses monitoring a distance to a critical structure (FIG. 14), capturing an image of an instrument in a surgical field (par. 105) and assigning a low or high priority values depending on a distance threshold comparison using image analysis (pars. 153, 157), and changing the size, highlighting or transparency of a display element dynamically based on its significance level (par. 213). Thus, Shelton shows that it was known in the art before the effective filing date of the claimed invention to prioritize particular display elements based on a detection of a user activating an instrument but preventing the activation in the event an anatomical parameter, e.g., tissue tension, is not available, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, providing an intuitive graphical user interface to support robotic surgeries.
A person of ordinary skill in the art would have been motivated to combine a priority based on a detected activation of a surgical instrument from an image of the instrument in a surgical field as seen by a camera, as disclosed by Shelton with the dynamic user interface disclosed by Duindam, to thereby modify the display elements to prioritize a warning in the event a user activates an instrument when a preset condition or parameter is not satisfied. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of improving patient safety.
Claim 11 is rejected under 35 U.S.C. 103 as being obvious over Duindam in view of U.S. Pat. Appl. Pub. No. 20190183591 to Johnson et al. (hereinafter “Johnson”).
Regarding claim 11, Duindam teaches the medical system of claim 8, but does not teach that which is explicitly taught by Johnson.
Johnson teaches receiving an indication of a selection from a priority menu for the element (Johnson, par. 59, “Generally, a user (e.g., surgeon, other surgical staff) may populate various panels of a display with different applications, by selecting, for example, which applications are visible and where within an application layout they are placed on the multi-panel display.”, par. 60, “the GUI layout may be dynamic throughout a surgical procedure. For example, the importance of various content may differ depending on surgeon needs, what is happening during a surgical procedure, etc. Accordingly, as it may become more useful to have content from different applications or data sources be presented in a larger viewing area, the layout may change from time to time.”, par. 114, “tool widgets may summarize high-priority information such as tool type, tool state, tool settings, and/or tool "lives" remaining (e.g., number of firings left in a cartridge, etc.). Tool widgets may be overlaid over an endoscopic image, adjacent or proximate the endoscopic image, and/or or in any other suitable portion of the displayed GUI.”, par. 133, “In some variations, a tool widget may display an alert or a notification in response to the detection of a trigger event occurring during a workflow. ... Any suitable trigger event for prompting an alert of a notification may be defined. In some variations, alerts or notifications may be prioritized for display in order of urgency.”, par. 140, “The layout (e.g., content, shape, interactive form, etc.) populated of the with quick any access suitable menu may reconfigurable to be populated with any suitable combination of applications and/or general GUI functionalities. Similar to the panels in the multi-panel display, a user may customize the content of the quick access menu based on user input ... a user may customize a form factor of the quick access menu (e.g., circular or ring-shaped, list such as the quick access menu 2520 variation shown in FIG. 25, rectangular or other grid of selectable icons or descriptors, drop-down list, or any suitable format).”).
Duindam is analogous to the claimed invention for the reasons provided above. Johnson discloses a customizable menu for a dynamic GUI of reconfigurable panels that supports a surgical procedure. The dynamic GUI includes display of an endoscope image along with a quick access menu (par. 145) and various other tools and widgets in a customized layout for the surgeon and/or the procedure (FIG. 15). Each item in the quick access menu is a priority, hence its inclusion in the menu. The multi-panel GUI itself also acts as a menu of priorities as the panels’ appearance and arrangement is customized based on what the user thinks is important to them, at least in some examples. Thus, Johnson shows that it was known in the art before the effective filing date of the claimed invention to determine GUI display element priorities based on a selection or indication from a menu of priorities, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, providing an intuitive graphical user interface to support robotic surgeries.
A person of ordinary skill in the art would have been motivated to combine the multi-panel layout and appearance customization as well as the customizable quick-access menu disclosed by Johnson with the dynamic user interface disclosed by Duindam, to thereby enable the user of the GUI to customize panels or tiles in the GUI based on their priorities and display a customized menu which is selected by a user to select different configurations of the GUI for different priorities based on corresponding selectable descriptors or menu items in the customized menu. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of giving the user more flexibility to customize the interface to their individual needs and priorities.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
U.S. Pat. Appl. Pub. No. 20220331047 is pertinent to at least claim 1 by disclosing, for example, “A method for presenting surgical data onto a livestream of a surgical field on a display during a surgical procedure, the method comprising detecting, by a control module, surgical data, assigning, by the control module, display priority values to the surgical data, determining, by the control module, a display arrangement of the surgical data on the display based on the display priority values, and presenting onto the livestream visual representations of the surgical data in accordance with the display arrangement.” (par. 111).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN P POTTS whose telephone number is (571)272-6351. The examiner can normally be reached M-F, 9am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN P POTTS/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights