DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 1/22/2026 has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see pg. 11, filed 1/22/2026, with respect to the status of the claims and the status of the interview on December 23, 2025 are hereby acknowledged.
Applicant’s arguments, see Remarks pg. 11-12, filed 1/22/2026, with respect to the rejection(s) of claim(s) 1-23 under 35 U.S.C. 103 have been fully considered. The examiner notes that the applicant’s arguments are directed to the independent claims and the newly amended limitations not previously presented. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of newly found prior art in order to address the newly amended limitations.
The examiner will rely, in part, on newly found prior art but also rely on the prior art of record to Segev of record. With respect to the prior art of record, in Remarks pg. 11-12, filed 1/22/2026, applicant argues the following:
Applicant respectfully submits that the relied upon references do not disclose or suggest a determination that superimposed data of a second video stream above an area of a first video stream is to no longer be superimposed while at least the portion of the first video stream is to remain at least partially obscured. Instead, DiMaio describes how overlays are removed to return an initial display of the GUI in the MaM mode responsive to the surgeon selecting a menu button. Nowhere does DiMaio describe that a determination is made in which the overlay of the image volume 2532 is to no longer be overlaid above an area of the display while the area remains at least partially obscured.
The examiner respectfully disagrees. In response to applicant’s argument, the test for obviousness is not whether the features of a secondary reference may be bodily incorporated into the structure of the primary reference; nor is it that the claimed invention must be expressly suggested in any one or all of the references. Rather, the test is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981). Additionally, on the issue of obviousness, the Supreme Court stated the analysis of a rejection on obviousness grounds need not seek out precise teachings directed to the specific subject matter of the challenged claim, for a court can take account of the inferences and creative steps that a person of ordinary skill in the art would employ. See KSR International Co. v. Teleflex Inc., 550 U.S. 398, 418, 82 USPQ2d 1385 (2007). The obvious analysis cannot be confined by a formalistic conception of the words teaching, suggestion, and motivation. Id. at 419. Further, the Court stated that common sense teaches, however, that familiar items may have obvious uses beyond their primary purposes, and in many cases a person of ordinary skill will be able to fit the teachings of multiple patents together like pieces of a puzzle. Id. at 420.
Applicant’s arguments do not appear to consider the significant teaching value of Segev relating to the display of PIP content during a surgical procedure. For example, Segev para 190-218 teaches displaying elements on a user interface are performed automatically. For example, Segev teaches that a location of a PIP on the field of view of HMD 102 may be dynamically changed according to the location of the automatically-detected area-of-interest in the main image. Even if Segev teaches that the surgeon is able to predefine PIP content, a person of ordinary skill in the art would understand that Segev’s disclosure enables for the automatic presentation and removal of PIP content at appropriate times during a surgery and without the need for the surgeon to manually display the PIP during the surgery. For example, Segev para 387 teaches PIP content is automatically removed (i.e., Once a marking becomes obsolete, to prevent these from obstructing the display of the live image, system 100 may be configured to cause the markings to fade after a predetermined time period. In one embodiment, a symbol automatically fades according to a default setting (e.g. after ten seconds). In another embodiment, a dedicated preplanning application menu allows surgeon 120 to disable (i.e. turn off) each of the markings.). All things considered, the examiner will rely on the prior art of record in order to address the newly amended limitations and in an effort to advance prosecution, the examiner will rely on newly found prior art in order to evidence what the combined teachings of the references would have suggested to those of ordinary skill in the art.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1-2, 5-6, 9-10, 12, 15, 17-18, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over DiMaio; Simon P. et al. US 20090036902 A1 (hereafter DiMaio) and in further view of SEGEV; Eran et al. US 20210382559 A1 (hereafter Segev) and in further view of Omori; Shigeru US 20090192519 A1 (hereafter Omori) and in further view of Casas; Carlos Quiles US 20200053335 A1 (hereafter Casas) and in further view of Johnson; Norbert et al. US 20200246081 A1 (hereafter Johnson).
Regarding claim 1, “a method performed by a video controller, the method comprising: receiving a first video stream captured by an endoscope of a surgical system; receiving a second video stream that comprises surgical data; displaying, on a display, the surgical data of the second video stream superimposed above an area of the first video stream such that the surgical data at least partially obscures a portion of the first video stream within the area; determining, without user intervention, that the surgical data of the second video stream is no longer to be superimposed above the area, while at least the portion of the first video stream is to remain at least partially obscured; wherein determining that the surgical data of the second video stream is no longer to be superimposed is based on a content analysis of at least the second video stream; and responsive to determining that the surgical data of the second video stream is no longer to be superimposed above the area of the first video stream, continuing to display the first video stream unobscured” DiMaio teaches para 110-115 teleoperated surgical system will support at least two types of video sources comprising endoscopes or endoscopic cameras and surgical data obtained from ultrasound probes, medical images, models, surgical plans, and other application data. With respect to “displaying, on a display, the surgical data of the second video stream superimposed above an area of the first video stream such that the surgical data at least partially obscures a portion of the first video stream within the area; determining, without user intervention, that the surgical data of the second video stream is no longer to be superimposed above the area, while at least the portion of the first video stream is to remain at least partially obscured; wherein determining that the surgical data of the second video stream is no longer to be superimposed is based on a content analysis of at least the second video stream; and responsive to determining that the surgical data of the second video stream is no longer to be superimposed above the area of the first video stream, continuing to display the first video stream unobscured” DiMaio [0119] After being captured by the ultrasound (US) image capture module 1422, the LapUS data is transmitted to the image fusion block 1414. The image fusion block 1414 fuses the ultrasound images with the 3D endoscopic images that are then coupled into the overlay block 1418. The overlay block 1418 selectively overlays the graphical user interface and the medical image volume onto the fused ultrasound and endoscopic images. The combined image data including the overlaid graphics and images onto the fused images is coupled to the rendering block 1424 for rendering onto the hardware display 1430. With respect to “determining, without user intervention, that the surgical data of the second video stream is no longer to be superimposed above the area, while at least the portion of the first video stream is to remain at least partially obscured; wherein determining that the surgical data of the second video stream is no longer to be superimposed is based on a content analysis of at least the second video stream; and responsive to determining that the surgical data of the second video stream is no longer to be superimposed above the area of the first video stream, continuing to display the first video stream unobscured” DiMaio does not disclose displaying overlaying content without user intervention but DiMaio does disclose elements relevant to the recited limitation (e.g., para 144-152, 175-176 – disclosing that when two video streams are simultaneously displayed, the surgeon operates the functionality to select and deselect whether an overlay is to remain; see also [0154] By overlaying a GUI over live images from the endoscope and further overlaying ultrasound images captured by the ultrasound instrument onto the live images, the SAW fuses graphical objects with physical objects in a physical coordinate frame. DiMaio para 130-131 overlays are determined based on surgeon control such that overlays will appear and disappear based on surgeon inputs.)
Whereas DiMaio’s teachings also do not use the term “surgical data” as claimed (i.e., receiving a second video stream that comprises surgical data), a person of ordinary in the art would have reasonably inferred that DiMaio reads on the limitation (i.e., para 110-115 teleoperated surgical system will support at least two types of video sources comprising endoscopes or endoscopic cameras and surgical data obtained from ultrasound probes, medical images, models, surgical plans, and other application data), however, the limitation is further obvious in view of the prior art teachings to Segev para [0196] PIP (picture-in-picture) system mode. The display of picture-in-picture (PIP) allows displaying preoperative images, live video feeds, snapshots of previously acquired video, GIFs, and data such as vital signs, and other data, in a window overlaid on the current background image displayed via HMD 102. See also Segev para 96-97 determines which images to stream to surgeon 120 based on the current system mode, as well as one or more inputs received from the surgeon via a user interface. See [0176] In addition to controlling operational settings by surgeon 120 wearing HMD 102, touchscreen 108 allows to control modes, settings and preferences that cannot be controlled by surgeon 120 wearing HMD 102. Touchscreen 108 may display any of the video feeds that are available to system 100, either in their raw form or in the format seen by the wearer of HMD 102, e.g. with overlays, PIPs, etc.). With respect to the deficiency of DiMaio regarding “without user intervention” as discussed above, Segev para 190-218 teaches displaying elements on a user interface are performed automatically. For example, Segev teaches that a location of a PIP on the field of view of HMD 102 may be dynamically changed according to the location of the automatically-detected area-of-interest in the main image. Even if Segev teaches that the surgeon is able to predefine PIP content, a person of ordinary skill in the art would understand that Segev’s disclosure enables for the automatic presentation and removal of PIP content at appropriate times during a surgery and without the need for the surgeon to manually display the PIP during the surgery. For example, Segev para 387 teaches PIP content is automatically removed (i.e., Once a marking becomes obsolete, to prevent these from obstructing the display of the live image, system 100 may be configured to cause the markings to fade after a predetermined time period. In one embodiment, a symbol automatically fades according to a default setting (e.g. after ten seconds). In another embodiment, a dedicated preplanning application menu allows surgeon 120 to disable (i.e. turn off) each of the markings.).
The prior art further evidences the inferences that a person of ordinary skill in the art would have drawn based on the teachings of DiMaio and Segev. In an analogous art, Omori para 80-104 discloses a surgical system comprising a video feed from an endoscopic device and wherein a display a user interface displays a plurality of camera feeds and is able to automatically, and without user intervention, determine an unexpected failure/malfunction in the display of the camera feeds in order to automatically change the displayed images to enable the operator to continue with the surgical procedure.
In an analogous art, Casas and Johnson evidence a motivation for modifying the teachings of DiMaio with the teachings of Segev and Omori wherein Casas teaches an invention comprising stereoscopic cameras provide real-time images of the scene including the patient wherein a stereoscopic video display is used by the surgeon, who sees a graphical representation of the preoperative or intraoperative images blended with the video images in a stereoscopic manner through a see-through display (Abstract, para 59-62, 66, 113-116 teaching, inter alia, “3D volume blended with the 3D surface is tracked in space to the location and orientation of the stereoscopic display 126, by way of tracking means 136...Alternatively, the tracked position of the head of the surgeon 128 is used to define the location and orientation of the virtual cameras offering the stereoscopic view (or the virtual camera offering a 2D view) of the 3D volume image. Thus, allowing for an instant registration of the surgeon's 128 direct vision of the portion of the patient 118 and the 3D volume image, that may be adjusted with the available real-time user interface means 132...display system 126 is a projector which projects the processed images 124 over the target portion of the patient 118. In order to achieve a seamless integration with the surgeon's 128 view of the patient 118 during surgery, tracking means 136 are used to accurately track in real time the position of the projector and of the head of the surgeon 128, relative to the common coordinate system. Computer means 100, taking into account the location, orientation and lens-characteristics of the projector, send the blended images that are projected over the patient 118, so that the images appear to the surgeon 128 as a property of the target portion of the patient 118, with the desired adjustments in transparency, color, contrast, etc….” Stated differently, Casas teaches a stereoscopic video stream is combined with a second video stream comprising preoperative imaging and intraoperative imaging content such that based on a determination of a surgeon’s direct vision, the second video stream content will be displayed to obscure the video stream comprising the patient when desired by the surgeon based on a determination of whether the surgical data (i.e., data comprising preoperative or intraoperative images) is to be overlayed within the direct vision of the surgeon. See also Johnson teaching embodiments analogous to the teachings of Casas wherein the subject matter comprises the presentation of surgical data along with a video stream of the patient to the surgeon performing surgery on a patient (para 51, 87, 117-118, 125, 149-158 “…whether one or multiple representations of trajectories are displayed, the augmentation graphics used may appear overlaid over a patient anatomy to give a surgeon an accurate view of how the trajectory intersects a patient's anatomy. In some embodiments, portions of a surgical tool and/or its trajectories appear on a virtual display screen viewed through a display screen of a head mounted display (e.g., in spatial correspondence with a model of patient anatomy such that the trajectory representation, portion of a surgical tool, and model appear to have the same relative position and orientation as physical reality). For example, a virtual display screen may appear as panel A or panel B of FIG. 15. Overlaid augmentation graphics may appear as the sketch in FIG. 16 illustrates….”
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of DiMaio for capturing and displaying camera images of a surgical site on at least one display device at a surgeon console comprising a teleoperated surgical system supporting at least two types of video sources comprising endoscopes or endoscopic cameras and surgical data obtained from ultrasound probes, medical images, models, surgical plans, and other application data in order to provide a surgeon with a multi-view perspective comprising a plurality of surgical data on a single interface by further incorporating known elements of Segev’s invention comprising a visual display of picture-in-picture (PIP) video content which allows for automatically, and without user intervention, displaying of preoperative images, live video feeds, snapshots of previously acquired video, GIFs, and data such as vital signs, and other data, in a window overlaid on the current background image because the modification would improve microsurgical procedures, wherein a surgeon's hands are typically occupied with the surgical tools, and he thus requires a wide range of hands-free methods to interface with the surgical microscope and control system parameters as disclosed by Omori for addressing failures that occur with the video feeds displayed from surgical equipment wherein Casas’ stereoscopic cameras providing real-time images of the scene including the patient wherein a stereoscopic video display is used by the surgeon, who sees a graphical representation of the preoperative or intraoperative images blended with the video images in a stereoscopic manner through a see-through display and further disclosed in Johnson for displaying a surgical site and utilizing a position tracking system by detecting one or more real-world features and (ii) the real-world feature representations contained in a patient model to transform the patient model to a present perspective view of a wearer of HMD wherein some or all of the transformed patient model can then be displayed on a display screen of HMD using augmentation graphics, for example, to provide the surgeon with an augmentation graphical overlay that is precisely oriented and scaled on the surgical site or other target location on a patient.
Regarding claim 2, “wherein the surgical data comprises a graphical user interface (GUI) that includes at least one of a notification associated with a surgical procedure, image data captured by one or more cameras of the surgical system, or a user interface (UI) item for allowing a user to interact with the GUI through the display” is further rejected on obviousness grounds as discussed in the rejection of claim 1 wherein DiMaio para 107, 124, 130, 145-151 allowing the surgeon to interact with the SAW graphical user interface (GUI). In this mode, each MTM operates as a 3D mouse, such that it can be used to position a graphical cursor overlaid on the stereo display console, while gripper open/close motions are used to emulate click and drag operations. In this way, the surgeon is able to interact with graphical objects and menus displayed by the SAW application. This mode is called a masters-as-mice (MaM) mode. See also Segev’s para 193-196 GUI operation.
Regarding claim 5, “wherein determining that the second video stream is no longer being superimposed comprises performing the content analysis upon a blend of the first and second video streams to determine whether the area of the first video stream over which the second video stream is displayed exceeds a threshold area” is further rejected on obviousness grounds as discussed in the rejection of claims 1-2 wherein DiMaio para 107, 123, 134, 147, 152-154 – determining that a surgeon has interacted with graphical objects and displayed menus to determine that a closing motion has been initiated. Segev relating to the display of PIP content during a surgical procedure. For example, Segev para 190-218 teaches displaying elements on a user interface are performed automatically. For example, Segev teaches that a location of a PIP on the field of view of HMD 102 may be dynamically changed according to the location of the automatically-detected area-of-interest in the main image. Even if Segev teaches that the surgeon is able to predefine PIP content, a person of ordinary skill in the art would understand that Segev’s disclosure enables for the automatic presentation and removal of PIP content at appropriate times during a surgery and without the need for the surgeon to manually display the PIP during the surgery. For example, Segev para 387 teaches PIP content is automatically removed (i.e., Once a marking becomes obsolete, to prevent these from obstructing the display of the live image, system 100 may be configured to cause the markings to fade after a predetermined time period. In one embodiment, a symbol automatically fades according to a default setting (e.g. after ten seconds). In another embodiment, a dedicated preplanning application menu allows surgeon 120 to disable (i.e. turn off) each of the markings.).
Regarding claim 6, “wherein the video controller is a first video controller, wherein the second video stream is received from a second video controller of the surgical system” is further rejected on obviousness grounds as discussed in the rejection of claims 1-2 wherein Segev Fig. 1b elements 118I and 118H; See also DiMaio para 118-119 – [0118] In the video processing/ visualization pathways, image data from cameras 1426 and LapUS 1428 is captured by their respective image capture modules, stereo image capture module 1420 and ultrasound (US) image capture module 1422. Video image data from the endoscopic cameras 1426 is further rectified in the rectification block 1416 before being coupled into the stereo processor block 1412 for processing from 2D to 3D images. The 3D images of block 1412 are then transmitted to the tool tracking subsystem 1410 and used in conjunction with the kinematic data provided by collaborative robot block 1404 to monitor the surgical tools.
Regarding the device claims 9-10, 12, and 15 and the system claims 17-18 and 21, the claims are grouped and rejected with the method claims 1-2, 5-6 because the steps of the method claims are met by the disclosure of the apparatus and methods of the reference(s) as discussed in the rejection of claims 1-2, 5-6 and because the steps of the method are easily converted into elements of computer implemented system and methods by one of ordinary skill in the art. With respect to a video control device of claim 9, Casas Fig. 1 and para 178-198 disclose a computing system with components interpreted as video controllers for receiving and processing the disclosed and claimed video signals. With respect to independent power supplies as recited in claim 10, Segev para 105 discloses multiple processors and power supplies.
Claim(s) 3, 13, 19 are rejected under 35 U.S.C. 103 as being unpatentable over DiMaio; Simon P. et al. US 20090036902 A1 (hereafter DiMaio) and in further view of SEGEV; Eran et al. US 20210382559 A1 (hereafter Segev) and in further view of Omori; Shigeru US 20090192519 A1 (hereafter Omori) and in further view of Casas; Carlos Quiles US 20200053335 A1 (hereafter Casas) and in further view of Johnson; Norbert et al. US 20200246081 A1 (hereafter Johnson) and in further view of Itkowitz; Brandon D. et al. US 20110282140 A1 (hereafter Itkowitz).
Regarding claim 3, “wherein the display comprises a plurality of lines of pixels, wherein displaying the surgical data of the second video stream superimposed above the area of the first video stream comprises: producing a blended video stream by blending the first video stream with the second video stream; storing, in a line buffer, selected pixels for a set of lines of the plurality of lines based on the blended video stream; and providing, from the line buffer, the stored selected pixels to the display” is further rejected on obviousness grounds as discussed in the rejection of claims 1-2, however, whereas DiMaio does not use the terms “pixels” and “buffer” wherein Segev’s para 106, 370, 380-382 teaches Image processor 118K processes images received from any of camera system 112, memory 118D, or a GPU, such as by applying a zoom in, zoom out, digital filtering, sharpening, smoothing, color corrections, fusion of several image sources, embedding one image source in another image source in a picture-in-picture form, and the like. See also Johnson para 29, 43 disclose the display comprises pixels. See also Casas para 35, 128 discussing blending images and blending images comprises applying an algorithm to the pixels but Casas does not reference a set of lines.
In an analogous art, Itkowitz teaches capturing and displaying camera images of a surgical site on at least one display device at a surgeon console comprising a teleoperated surgical system supporting at least two types of video sources and teaches a sequence of combined video images wherein a first video image is superimposed over a real-time video of the surgical site wherein the superimposing utilizes superimposing a “storing, in a line buffer, selected pixels for a set of lines of the plurality of lines based on the blended video stream; and providing, from the line buffer, the stored selected pixels to the display” See Itkowitz para 12, 67, 75-80, 109.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of DiMaio, Segev, Omori, Casas, and Johnson’s inventions for capturing and displaying multiple camera images of a surgical site on at least one display device at a surgeon console comprising a visual display of picture-in-picture (PIP) video or overlaid content which allows displaying preoperative images, live video feeds, snapshots of previously acquired video, GIFs, and data such as vital signs, and other data, in a window overlaid/superimposed on the current background image by further incorporating elements of Itkowitz teaches capturing and displaying camera images of a surgical site on at least one display device at a surgeon console comprising a teleoperated surgical system supporting at least two types of video sources and teaches a sequence of combined video images wherein a first video image is superimposed over a real-time video of the surgical site wherein the superimposing utilizes a buffer for storing selected pixels for a set of lines of the plurality of lines based on the blended video stream and providing, from the buffer, the stored selected pixels to the display because the modification would improve microsurgical procedures utilizing graphical user interface providing a wide range of hands-free methods to interface with the surgical microscope and control system parameters and facilitate configuring video feeds for an application to be the appropriate pixel dimensions (e.g., full screen, half screen, etc.).
Regarding the device claims 13 and the system claim 19, the claims are grouped and rejected with the method claim 1-3, 5-6 because the steps of the method claims are met by the disclosure of the apparatus and methods of the reference(s) as discussed in the rejection of claims 1-3, 5-6 and because the steps of the method are easily converted into elements of computer implemented system and methods by one of ordinary skill in the art.
Claim(s) 4, 8, 14, 16, 20, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over DiMaio; Simon P. et al. US 20090036902 A1 (hereafter DiMaio) and in further view of SEGEV; Eran et al. US 20210382559 A1 (hereafter Segev) and in further view of Omori; Shigeru US 20090192519 A1 (hereafter Omori) and in further view of Casas; Carlos Quiles US 20200053335 A1 (hereafter Casas) and in further view of Johnson; Norbert et al. US 20200246081 A1 (hereafter Johnson) and in further view of Shelton, IV; Frederick E. et al. US 20190200844 A1 (hereafter Shelton).
Regarding claim 4, “wherein the area is a first area, wherein, while the surgical data of the second video stream is superimposed above the area of the first video stream, the surgical system is in a blended mode in which a blended video stream of the first and second video streams is displayed, wherein the method further comprises presenting a notification that provides a recommendation for a user of the surgical system to switch to a failover mode in which the first video stream is superimposed above a second area of the blended video stream, wherein the first video stream is continued to be displayed responsive to receiving user input via a user input device for switching from the blended mode to the failover mode” the combination of DiMaio and Segev as discussed in the rejection of claims 1-2 render obvious the limitation except for “wherein the area is a first area, wherein, while the second video stream is superimposed above the area of the first video stream, the surgical system is in a blended mode in which a blended video stream of the first and second video streams is displayed.” See also Casas teaches an invention comprising stereoscopic cameras provide real-time images of the scene including the patient wherein a stereoscopic video display is used by the surgeon, who sees a graphical representation of the preoperative or intraoperative images blended with the video images in a stereoscopic manner through a see-through display (Abstract, para 59-62, 66, 113-116; Casas para 35, 128 discussing blending images and blending images comprises applying an algorithm to the pixels but Casas does not reference a set of lines); See also Johnson teaching embodiments analogous to the teachings of Casas wherein the subject matter comprises the presentation of surgical data along with a video stream of the patient to the surgeon performing surgery on a patient (para 51, 87, 117-118, 125, 149-158).
In an analogous art, Shelton discloses the deficiency of DiMaio and Segev wherein Shelton para [1209] and [0561] teaches providing notification relating to failures teaches capturing and para [1048] teaches the first video stream is superimposed above a second area of the blended video stream, wherein the first video stream is continued to be displayed responsive to receiving user input via a user input device for switching from the blended mode.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of DiMaio, Segev, Casas, and Johnson’s inventions for capturing and displaying multiple camera images of a surgical site on at least one display device at a surgeon console comprising a visual display of picture-in-picture (PIP) video content which allows displaying preoperative images, live video feeds, snapshots of previously acquired video, GIFs, and data such as vital signs, and other data, in a window overlaid/superimposed on the current background image by further incorporating elements of Shelton teaches capturing and displaying camera images of a surgical site on at least one display device at a surgeon console comprising a teleoperated surgical system supporting at least two types of video sources and teaches a sequence of combined video images wherein a first video image is superimposed over a real-time video of the surgical site wherein the superimposing a first video stream is superimposed above a second area of the blended video stream, wherein the first video stream is continued to be displayed responsive to receiving user input via a user input device for switching from the blended mode in order to provide a surgeon with a redundant view of an operation to prevent losing visuals during failures.
Regarding claim 8, “wherein displaying the surgical data of the second video stream superimposed above the area of the first video stream comprises providing a first blended video stream that includes the first and second video streams to the display, wherein continuing to display the first video stream comprises: producing a second blended video stream by retrieving a first set of one or more video frames of the first video stream from a frame buffer and blending the first set of one or more video frames with a second set of one or more video frames of the first blended video stream; and providing the second blended video stream to the display” the combination of DiMaio, Segev, Omori, Casas and Johnson as discussed in the rejection of claims 1-2 and 4 render obvious the limitation except for “retrieving a first set of one or more video frames of the first video stream from a frame buffer and blending the first set video frames with a second set of one or more video frames of the first blended video stream; and providing the second blended video stream to a display.” See Segev’s para 106-107, 370, 380-382 teaches Image processor 118K processes images received from any of camera system 112, memory 118D, or a GPU, such as by applying a zoom in, zoom out, digital filtering, sharpening, smoothing, color corrections, fusion of several image sources, embedding one image source in another image source in a picture-in-picture form, and the like.
In an analogous art, Shelton the deficiency of DiMaio, Segev, Casas and Johnson wherein Shelton para [1209] and [0561] teaches providing notification relating to failures teaches capturing and para [1048] teaches retrieving a first set of one or more video frames of the first video stream from a frame buffer and blending the first set of video frames with a second set of one or more video frames of the first blended video stream; and providing the second blended video stream to a display.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of DiMaio, Segev, Omori, Casas and Johnson’s inventions for capturing and displaying multiple camera images of a surgical site on at least one display device at a surgeon console comprising a visual display of picture-in-picture (PIP) video content which allows displaying preoperative images, live video feeds, snapshots of previously acquired video, GIFs, and data such as vital signs, and other data, in a window overlaid/superimposed on the current background image by further incorporating elements of Shelton teaches capturing and displaying camera images of a surgical site on at least one display device at a surgeon console comprising a teleoperated surgical system supporting at least two types of video sources and teaches a sequence of combined video images wherein a first video image is superimposed over a real-time video of the surgical site wherein the superimposing a first video stream is superimposed above a second area of the blended video stream, wherein the first video stream is continued to be displayed responsive to receiving user input via a user input device for switching from the blended mode in order to provide a surgeon with a redundant view of an operation to prevent losing visuals during failures.
Regarding the device claims 14, 16 and the system claims 20 and 22, the claims are grouped and rejected with the method claims 4 and 8 because the steps of the method claims are met by the disclosure of the apparatus and methods of the reference(s) as discussed in the rejection of claims 4 and 8 and because the steps of the method are easily converted into elements of computer implemented system and methods by one of ordinary skill in the art.
Claim(s) 7, 11, 23 are rejected under 35 U.S.C. 103 as being unpatentable over DiMaio; Simon P. et al. US 20090036902 A1 (hereafter DiMaio) and in further view of SEGEV; Eran et al. US 20210382559 A1 (hereafter Segev) and in further view of Casas; Carlos Quiles US 20200053335 A1 (hereafter Casas) and in further view of Johnson; Norbert et al. US 20200246081 A1 (hereafter Johnson) and in further view of Shelton, IV; Frederick E. et al. US 20220104822 A1 (hereafter Shelton '822).
Regarding claim 7, DiMaio, Casas, and Johnson are silent with respect to “wherein the first video controller is a field-programmable gate array (FPGA), and the second video controller is a graphics processing unit (GPU).” Segev’s para 106, 370, 380-382 teaches Image processor 118K processes images received from any of camera system 112, memory 118D, or a GPU, such as by applying a zoom in, zoom out, digital filtering, sharpening, smoothing, color corrections, fusion of several image sources, embedding one image source in another image source in a picture-in-picture form, and the like. In an analogous art, Shelton the deficiency of DiMaio and Segev wherein Shelton para 23, 160, 534-535, 548, 1193 teaches devices employed to identify anatomical structures of the body using a variety of sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices wherein the invention utilizes FPGA and GPU’s to perform the computer methods.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of DiMaio, Segev, Casas, and Johnson’s inventions for capturing and displaying multiple camera images of a surgical site on at least one display device at a surgeon console comprising a visual display of picture-in-picture (PIP) video content which allows displaying preoperative images, live video feeds, snapshots of previously acquired video, GIFs, and data such as vital signs, and other data, in a window overlaid/superimposed on the current background image by further incorporating elements of Shelton ‘822 teaches devices employed to identify anatomical structures of the body using a variety of sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices wherein the invention utilizes FPGA and GPU’s to perform the computer methods in order to provide a purchased functional tier to enable the FGPA and GPU to be updatable and/or may make certain transforms available to the surgical visualization system.
Regarding the device claim 11 and the system claim 23, the claims are grouped and rejected with the method claims 1-3, 5-7 because the steps of the method claims are met by the disclosure of the apparatus and methods of the reference(s) as discussed in the rejection of claims 1-3, 5-7 and because the steps of the method are easily converted into elements of computer implemented system and methods by one of ordinary skill in the art.
CONCLUSION
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALFONSO CASTRO whose telephone number is (571)270-3950. The examiner can normally be reached on Monday to Friday from 10am to 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Nathan Flynn can be reached. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALFONSO CASTRO/Primary Examiner, Art Unit 2421