DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see pg. 17, filed 6/11/2026, with respect to the status of the claims and the status of the interview on June 2, 2026, are hereby acknowledged.
Applicant’s arguments, see Remarks pg. 17-18, filed 6/11/2026, with respect to the rejection(s) of claim(s) 1-23 under 35 U.S.C. 103 have been fully considered. The examiner notes that the applicant’s arguments are directed to the independent claims and the newly amended limitations not previously presented. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of newly found prior art in order to address the newly amended limitations.
Whereas the applicant argues that “[n]owhere, however, do any of the relied upon sections of these relied upon references disclose or suggest responsive to determining that the surgical data of the second video stream is no longer to be superimposed above the area of the first video stream, while at least the portion of the first video stream is to remain at least partially obscured, producing a second blended video stream that includes one or more video frames of the first video stream superimposed above the first blended video stream, and providing, to the display, the second blended video stream...,” the examiner will rely, in part, on newly found prior art and will no longer rely on the teachings of Casas and Johnson of record.
The examiner will rely on newly found prior art and reiterates that the test for obviousness is not whether the features of a secondary reference may be bodily incorporated into the structure of the primary reference; nor is it that the claimed invention must be expressly suggested in any one or all of the references. Rather, the test is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981). Additionally, on the issue of obviousness, the Supreme Court stated the analysis of a rejection on obviousness grounds need not seek out precise teachings directed to the specific subject matter of the challenged claim, for a court can take account of the inferences and creative steps that a person of ordinary skill in the art would employ. See KSR International Co. v. Teleflex Inc., 550 U.S. 398, 418, 82 USPQ2d 1385 (2007). The obvious analysis cannot be confined by a formalistic conception of the words teaching, suggestion, and motivation. Id. at 419. Further, the Court stated that common sense teaches, however, that familiar items may have obvious uses beyond their primary purposes, and in many cases a person of ordinary skill will be able to fit the teachings of multiple patents together like pieces of a puzzle. Id. at 420.
All things considered, the examiner will rely on the prior art of record to DiMaio, Segev, Omori and newly found prior art to NAMIKI; Hirotaka US 20120239058 A1 (hereafter Namiki) and Shelton, IV; Frederick E. et al. US 20230025827 A in order to address the newly amended limitations.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1-2, 4-7, 9-12, 14-15, 17-18, and 20-21, 23 are rejected under 35 U.S.C. 103 as being unpatentable over DiMaio; Simon P. et al. US 20090036902 A1 (hereafter DiMaio) and in further view of SEGEV; Eran et al. US 20210382559 A1 (hereafter Segev) and in further view of Omori; Shigeru US 20090192519 A1 (hereafter Omori) and in further view of NAMIKI; Hirotaka US 20120239058 A1 (hereafter Namiki) and in further view of Shelton, IV; Frederick E. et al. US 20230025827 A1 (hereafter Shelton).
Regarding claim 1, “a method performed by a video controller, the method comprising: receiving a first video stream captured by an endoscope of a surgical system; receiving a second video stream that comprises surgical data; providing, to a display, a first blended video stream that includes the surgical data of the second video stream superimposed above an area of the first video stream such that the surgical data at least partially obscures a portion of the first video stream within the area; determining, without user intervention, that the surgical data of the second video stream is no longer to be superimposed above the area, while at least the portion of the first video stream is to remain at least partially obscured, wherein determining that the surgical data of the second video stream is no longer to be superimposed is based on a content analysis of at least the second video stream; and responsive to determining that the surgical data of the second video stream is no longer to be superimposed above the area of the first video stream, while at least the portion of the first video stream is to remain at least partially obscured, producing a second blended video stream that includes one or more video frames of the first video stream superimposed above the first blended video stream, and providing, to the display, the second blended video stream” DiMaio teaches para 110-115 teleoperated surgical system will support at least two types of video sources comprising endoscopes or endoscopic cameras and surgical data obtained from ultrasound probes, medical images, models, surgical plans, and other application data. With respect to “providing, to a display, a first blended video stream that includes the surgical data of the second video stream superimposed above an area of the first video stream such that the surgical data at least partially obscures a portion of the first video stream within the area; determining, without user intervention, that the surgical data of the second video stream is no longer to be superimposed above the area, while at least the portion of the first video stream is to remain at least partially obscured, wherein determining that the surgical data of the second video stream is no longer to be superimposed is based on a content analysis of at least the second video stream; and responsive to determining that the surgical data of the second video stream is no longer to be superimposed above the area of the first video stream, while at least the portion of the first video stream is to remain at least partially obscured, producing a second blended video stream that includes one or more video frames of the first video stream superimposed above the first blended video stream, and providing, to the display, the second blended video stream” DiMaio [0119] After being captured by the ultrasound (US) image capture module 1422, the LapUS data is transmitted to the image fusion block 1414. The image fusion block 1414 fuses the ultrasound images with the 3D endoscopic images that are then coupled into the overlay block 1418. The overlay block 1418 selectively overlays the graphical user interface and the medical image volume onto the fused ultrasound and endoscopic images. The combined image data including the overlaid graphics and images onto the fused images is coupled to the rendering block 1424 for rendering onto the hardware display 1430. With respect to “determining, without user intervention, that the surgical data of the second video stream is no longer to be superimposed above the area, while at least the portion of the first video stream is to remain at least partially obscured, wherein determining that the surgical data of the second video stream is no longer to be superimposed is based on a content analysis of at least the second video stream; and responsive to determining that the surgical data of the second video stream is no longer to be superimposed above the area of the first video stream, while at least the portion of the first video stream is to remain at least partially obscured, producing a second blended video stream that includes one or more video frames of the first video stream superimposed above the first blended video stream, and providing, to the display, the second blended video stream” DiMaio does not disclose displaying overlaying content without user intervention as claimed but DiMaio does disclose elements relevant to the recited limitation (e.g., para 144-152, 175-176 – disclosing that when two video streams are simultaneously displayed, the surgeon operates the functionality to select and deselect whether an overlay is to remain; see also [0154] By overlaying a GUI over live images from the endoscope and further overlaying ultrasound images captured by the ultrasound instrument onto the live images, the SAW fuses graphical objects with physical objects in a physical coordinate frame. DiMaio para 130-131 overlays are determined based on surgeon control such that overlays will appear and disappear based on surgeon inputs.)
Whereas DiMaio’s teachings also do not use the term “surgical data” as claimed (i.e., receiving a second video stream that comprises surgical data), a person of ordinary in the art would have reasonably inferred that DiMaio reads on the limitation (i.e., para 110-115 teleoperated surgical system will support at least two types of video sources comprising endoscopes or endoscopic cameras and surgical data obtained from ultrasound probes, medical images, models, surgical plans, and other application data), however, the limitation is further obvious in view of the prior art teachings to Segev para [0196] PIP (picture-in-picture) system mode. The display of picture-in-picture (PIP) allows displaying preoperative images, live video feeds, snapshots of previously acquired video, GIFs, and data such as vital signs, and other data, in a window overlaid on the current background image displayed via HMD 102. See also Segev para 96-97 determines which images to stream to surgeon 120 based on the current system mode, as well as one or more inputs received from the surgeon via a user interface. See [0176] In addition to controlling operational settings by surgeon 120 wearing HMD 102, touchscreen 108 allows to control modes, settings and preferences that cannot be controlled by surgeon 120 wearing HMD 102. Touchscreen 108 may display any of the video feeds that are available to system 100, either in their raw form or in the format seen by the wearer of HMD 102, e.g. with overlays, PIPs, etc.). With respect to the deficiency of DiMaio regarding “without user intervention” as discussed above, Segev para 190-218 teaches displaying elements on a user interface are performed automatically. For example, Segev teaches that a location of a PIP on the field of view of HMD 102 may be dynamically changed according to the location of the automatically-detected area-of-interest in the main image. Even if Segev teaches that the surgeon is able to predefine PIP content, a person of ordinary skill in the art would understand that Segev’s disclosure enables for the automatic presentation and removal of PIP content at appropriate times during a surgery and without the need for the surgeon to manually display the PIP during the surgery. For example, Segev para 387 teaches PIP content is automatically removed (i.e., Once a marking becomes obsolete, to prevent these from obstructing the display of the live image, system 100 may be configured to cause the markings to fade after a predetermined time period. In one embodiment, a symbol automatically fades according to a default setting (e.g. after ten seconds). In another embodiment, a dedicated preplanning application menu allows surgeon 120 to disable (i.e. turn off) each of the markings.).
The prior art further evidences the inferences that a person of ordinary skill in the art would have drawn based on the teachings of DiMaio and Segev. In an analogous art, Omori para 80-104 discloses a surgical system comprising a video feed from an endoscopic device and wherein a display a user interface displays a plurality of camera feeds and is able to automatically, and without user intervention, determine an unexpected failure/malfunction in the display of the camera feeds in order to automatically change the displayed images to enable the operator to continue with the surgical procedure.
In an analogous art, Namiki and Shelton further render obvious the applicant’s claimed invention and also evidence a motivation for modifying the teachings of DiMaio with the teachings of Segev and Omori wherein Namiki teaches surgical system including an imaging system for acquiring image signals and determine, based on the image information, whether an abnormal state has occurred in the surgical system and automatically switch to an abnormality control mode of executing abnormality-handling processing (Abstract, para 39-40 and 43-66, 89-120 giving a plurality of embodiments for determining, based on the image information, whether an abnormal state has occurred in the surgical system and automatically switch to an abnormality control mode of executing abnormality-handling processing; discussing monitoring display image data, colors of displayed image, fogging of camera lens, noise in image, image frame abnormality, frame loss, 3D display abnormality, abnormal detection patterns, etc). More importantly, Namiki teaches embodiments wherein upon detecting a malfunction, wide visual field using an endoscope is obtained first before stopping of other image obtaining tools attached to control arms (e.g., surgical tools or cameras as discussed in para 32 and 51 include multiple cameras)
In an analogous art, Shelton teaches an invention for redundant communication channels of imaging feeds in order to avoid interruptions in the viewing of video streams received from image providing surgical devices due to malfunctions and detected abnormalities (Abstract, para 199-210) and further teaches video streams for overlaying onto another video stream (para 216-218). Shelton teaches that if an issue associated with a pathway for transmitting a portion of the video stream is detected, combining or merging of the video stream portions may be suspended, and the video stream portion that has successfully been transmitted to the display system may be displayed (para 199). More importantly, with respect to “determining, without user intervention, that the surgical data of the second video stream is no longer to be superimposed above the area, while at least the portion of the first video stream is to remain at least partially obscured, wherein determining that the surgical data of the second video stream is no longer to be superimposed is based on a content analysis of at least the second video stream; and responsive to determining that the surgical data of the second video stream is no longer to be superimposed above the area of the first video stream, while at least the portion of the first video stream is to remain at least partially obscured, producing a second blended video stream that includes one or more video frames of the first video stream superimposed above the first blended video stream, and providing, to the display, the second blended video stream” Shelton para 237-244 teaches generating customized overlay content displayed and is also able to analyze video frames of a primary surgical imaging or video feed to identify missing information for the primary surgical imaging or video feed. The computing system may identify region(s) of poor quality (e.g., artifacts, blurry, obstructed view, etc.) and provides additional imaging to supplement the primary surgical video feed. Shelton teaches fuse imaging or videos obtained from different sources and use the primary imaging for supplementary image and provides and overlay a portion of imaging data from one imaging feed onto a corresponding portion of the imaging data from another imaging feed. All things considered, applicant’s limitation (i.e., “…determining, without user intervention, that the surgical data of the second video stream is no longer to be superimposed above the area, while at least the portion of the first video stream is to remain at least partially obscured, wherein determining that the surgical data of the second video stream is no longer to be superimposed is based on a content analysis of at least the second video stream; and responsive to determining that the surgical data of the second video stream is no longer to be superimposed above the area of the first video stream, while at least the portion of the first video stream is to remain at least partially obscured, producing a second blended video stream that includes one or more video frames of the first video stream superimposed above the first blended video stream, and providing, to the display, the second blended video stream) taught and rendered obvious based on the additional teachings of Shelton.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of DiMaio for capturing and displaying camera images of a surgical site on at least one display device at a surgeon console comprising a teleoperated surgical system supporting at least two types of video sources comprising endoscopes or endoscopic cameras and surgical data obtained from ultrasound probes, medical images, models, surgical plans, and other application data in order to provide a surgeon with a multi-view perspective comprising a plurality of surgical data on a single interface by further incorporating known elements of Segev’s invention comprising a visual display of picture-in-picture (PIP) video content which allows for automatically, and without user intervention, displaying of preoperative images, live video feeds, snapshots of previously acquired video, GIFs, and data such as vital signs, and other data, in a window overlaid on the current background image because the modification would improve microsurgical procedures, wherein a surgeon's hands are typically occupied with the surgical tools, and he thus requires a wide range of hands-free methods to interface with the surgical microscope and control system parameters as disclosed by Omori for addressing failures that occur with the video feeds displayed from surgical equipment wherein Namiki teaches surgical system including an imaging system for acquiring image signals and determine, based on the image information, whether an abnormal state has occurred in the surgical system and automatically switch to an abnormality control mode of executing abnormality-handling processing and further disclosed in Shelton teaches an invention for redundant communication channels of imaging feeds in order to avoid interruptions in the viewing of video streams received from image providing surgical devices due to malfunctions and detected abnormalities and discloses the benefit of generating customized overlay content for display based on analyzing video frames of a primary surgical imaging or video feed to identify missing information for the primary surgical imaging or video feed (e.g., identify region(s) of poor quality (e.g., artifacts, blurry, obstructed view, etc.)) and provides additional imaging to supplement the primary surgical video feed, for example, by fusing imaging or videos obtained from different sources and use the primary imaging for supplementary image and provides and overlay a portion of imaging data from one imaging feed onto a corresponding portion of the imaging data from another imaging feed.
Regarding claim 2, “wherein the surgical data comprises a graphical user interface (GUI) that includes at least one of a notification associated with a surgical procedure, image data captured by one or more cameras of the surgical system, or a user interface (UI) item for allowing a user to interact with the GUI through the display” is further rejected on obviousness grounds as discussed in the rejection of claim 1 wherein DiMaio para 107, 124, 130, 145-151 allowing the surgeon to interact with the SAW graphical user interface (GUI). In this mode, each MTM operates as a 3D mouse, such that it can be used to position a graphical cursor overlaid on the stereo display console, while gripper open/close motions are used to emulate click and drag operations. In this way, the surgeon is able to interact with graphical objects and menus displayed by the SAW application. This mode is called a masters-as-mice (MaM) mode. See also Segev’s para 193-196 GUI operation. See also Shelton para 206 providing notification to user. See also Namiki providing a notification to operator to switch to a second mode after an abnormality/malfunction is detected herein surgical system including an imaging system for acquiring image signals and determine, based on the image information, whether an abnormal state has occurred in the surgical system and automatically switch to an abnormality control mode of executing abnormality-handling processing (Abstract, para 39-40 and 43-66, 89-120 giving a plurality of embodiments for determining, based on the image information, whether an abnormal state has occurred in the surgical system and automatically switch to an abnormality control mode of executing abnormality-handling processing; discussing monitoring display image data, colors of displayed image, fogging of camera lens, noise in image, image frame abnormality, frame loss, 3D display abnormality, abnormal detection patterns, etc).
Regarding claim 4, “wherein the area is a first area, wherein, while the surgical data of the second video stream is superimposed above the area of the first video stream, the surgical system is in a blended mode in which the first blended video stream of the first video stream and the second video stream is displayed, wherein the method further comprises presenting a notification that provides a recommendation for a user of the surgical system to switch to a failover mode in which the one or more video frames of the first video stream is superimposed above a second area of the first blended video stream, wherein the first video stream is continued to be displayed responsive to receiving user input via a user input device for switching from the blended mode to the failover mode” the combination of DiMaio and Segev as discussed in the rejection of claims 1-2 render obvious the limitation except for “wherein the area is a first area, wherein, while the second video stream is superimposed above the area of the first video stream, the surgical system is in a blended mode in which a blended video stream of the first and second video streams is displayed.” See also Namiki teaches providing a notification to operator to switch to a second mode after an abnormality/malfunction is detected herein surgical system including an imaging system for acquiring image signals and determine, based on the image information, whether an abnormal state has occurred in the surgical system and automatically switch to an abnormality control mode of executing abnormality-handling processing (Abstract, para 39-40 and 43-66, 89-120 giving a plurality of embodiments for determining, based on the image information, whether an abnormal state has occurred in the surgical system and automatically switch to an abnormality control mode of executing abnormality-handling processing; discussing monitoring display image data, colors of displayed image, fogging of camera lens, noise in image, image frame abnormality, frame loss, 3D display abnormality, abnormal detection patterns, etc). See also Shelton para 206 providing indication to user. Shelton teaches an invention for redundant communication channels of imaging feeds in order to avoid interruptions in the viewing of video streams received from image providing surgical devices due to malfunctions and detected abnormalities (Abstract, para 199-210) and further teaches video streams for overlaying onto another video stream (para 216-218). Shelton teaches that if an issue associated with a pathway for transmitting a portion of the video stream is detected, combining or merging of the video stream portions may be suspended, and the video stream portion that has successfully been transmitted to the display system may be displayed (para 199). Shelton para 237-244 teaches generating customized overlay content displayed and is also able to analyze video frames of a primary surgical imaging or video feed to identify missing information for the primary surgical imaging or video feed. The computing system may identify region(s) of poor quality (e.g., artifacts, blurry, obstructed view, etc.) and provides additional imaging to supplement the primary surgical video feed. Shelton teaches fuse imaging or videos obtained from different sources and use the primary imaging for supplementary image and provides and overlay a portion of imaging data from one imaging feed onto a corresponding portion of the imaging data from another imaging feed.
Regarding claim 5, “wherein determining that the second video stream is no longer to be superimposed comprises performing the content analysis upon the first blended stream to determine whether the area of the first video stream over which the second video stream is displayed exceeds a threshold area” is further rejected on obviousness grounds as discussed in the rejection of claims 1-2 wherein DiMaio para 107, 123, 134, 147, 152-154 – determining that a surgeon has interacted with graphical objects and displayed menus to determine that a closing motion has been initiated. Segev relating to the display of PIP content during a surgical procedure. For example, Segev para 190-218 teaches displaying elements on a user interface are performed automatically. For example, Segev teaches that a location of a PIP on the field of view of HMD 102 may be dynamically changed according to the location of the automatically-detected area-of-interest in the main image. Even if Segev teaches that the surgeon is able to predefine PIP content, a person of ordinary skill in the art would understand that Segev’s disclosure enables for the automatic presentation and removal of PIP content at appropriate times during a surgery and without the need for the surgeon to manually display the PIP during the surgery. For example, Segev para 387 teaches PIP content is automatically removed (i.e., Once a marking becomes obsolete, to prevent these from obstructing the display of the live image, system 100 may be configured to cause the markings to fade after a predetermined time period. In one embodiment, a symbol automatically fades according to a default setting (e.g. after ten seconds). In another embodiment, a dedicated preplanning application menu allows surgeon 120 to disable (i.e. turn off) each of the markings.). See also Shelton teaches an invention for redundant communication channels of imaging feeds in order to avoid interruptions in the viewing of video streams received from image providing surgical devices due to malfunctions and detected abnormalities (Abstract, para 199-210) and further teaches video streams for overlaying onto another video stream (para 216-218). Shelton teaches that if an issue associated with a pathway for transmitting a portion of the video stream is detected, combining or merging of the video stream portions may be suspended, and the video stream portion that has successfully been transmitted to the display system may be displayed (para 199). More importantly, Shelton para 237-244 teaches generating customized overlay content displayed and is also able to analyze video frames of a primary surgical imaging or video feed to identify missing information for the primary surgical imaging or video feed. The computing system may identify region(s) of poor quality (e.g., artifacts, blurry, obstructed view, etc.) and provides additional imaging to supplement the primary surgical video feed. Shelton teaches fuse imaging or videos obtained from different sources and use the primary imaging for supplementary image and provides and overlay a portion of imaging data from one imaging feed onto a corresponding portion of the imaging data from another imaging feed. Shelton para 210 the computing system may extract one or more portions (e.g., a portion that includes an area of interest) from a surgical video stream and overlay the extracted portion onto another surgical video stream, as described herein. Shelton para 226 area for placing overlay content. Shelton [0228] The size of the overlay region may be determined based on the size of an area on the surgical instrument suitable for content overlay, the size of a fiducial marker in real life, and the size of the fiducial marker in a video frame of the surgical video stream (e.g., a ratio between the real-life size and the imaged size). For example, the computing system may identify the fiducial marker in the video frames of the surgical video stream and determine the respective location, size, and/or orientation of the fiducial marker(s) in respective video frames. For a given video frame, or a group of video frames, the computing system may determine the size, location and/or orientation of the overlay region based on the location, size and/or orientation of the fiducial marker captured therein.
Regarding claim 6, “wherein the video controller is a first video controller, wherein the second video stream is received from a second video controller of the surgical system” is further rejected on obviousness grounds as discussed in the rejection of claims 1-2 wherein Segev Fig. 1b elements 118I and 118H; See also DiMaio para 118-119 – [0118] In the video processing/ visualization pathways, image data from cameras 1426 and LapUS 1428 is captured by their respective image capture modules, stereo image capture module 1420 and ultrasound (US) image capture module 1422. Video image data from the endoscopic cameras 1426 is further rectified in the rectification block 1416 before being coupled into the stereo processor block 1412 for processing from 2D to 3D images. The 3D images of block 1412 are then transmitted to the tool tracking subsystem 1410 and used in conjunction with the kinematic data provided by collaborative robot block 1404 to monitor the surgical tools. See also Shelton para 206 teaching a plurality of processing modules for obtaining video streams. See also Shelton obtaining video streams and overlay content from different sources wherein Shelton para 237-244 teaches generating customized overlay content displayed and is also able to analyze video frames of a primary surgical imaging or video feed to identify missing information for the primary surgical imaging or video feed. The computing system may identify region(s) of poor quality (e.g., artifacts, blurry, obstructed view, etc.) and provides additional imaging to supplement the primary surgical video feed. Shelton teaches fuse imaging or videos obtained from different sources and use the primary imaging for supplementary image and provides and overlay a portion of imaging data from one imaging feed onto a corresponding portion of the imaging data from another imaging feed.
Regarding claim 7, DiMaio and Namiki are silent with respect to “wherein the first video controller is a field-programmable gate array (FPGA), and the second video controller is a graphics processing unit (GPU).” Segev’s para 106, 370, 380-382 teaches Image processor 118K processes images received from any of camera system 112, memory 118D, or a GPU, such as by applying a zoom in, zoom out, digital filtering, sharpening, smoothing, color corrections, fusion of several image sources, embedding one image source in another image source in a picture-in-picture form, and the like. In an analogous art, Shelton teaches the deficiency of DiMaio and Segev wherein Shelton para 189-190 teaches devices employed to identify anatomical structures of the body using a variety of sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices wherein the invention utilizes FPGA and GPU’s to perform the computer methods. See also Shelton as discussed in para 188-191, 195, 201 disclosing FPGA.
Regarding the device claims 9-10, 12, 14-15 and the system claims 17-20, the claims are grouped and rejected with the method claims 1-2, 4-5 because the steps of the method claims are met by the disclosure of the apparatus and methods of the reference(s) as discussed in the rejection of claims 1-2, 4-5 and because the steps of the method are easily converted into elements of computer implemented system and methods by one of ordinary skill in the art. With respect to a video control device of claim 9 and the power supply of claim 10, Nakima Fig. 2 para 32, 58, 84-86 teaches an image processing circuit which is coupled to endoscope and sensors/cameras for obtaining images with a plurality of independent power units. See also Segev para 105 discloses multiple processors and power supplies relating to independent power supplies as recited in claim 10.
Regarding the device claim 11 and the system claim 23, the claims are grouped and rejected with the method claims 1-2, 4-7, 9-10 because the steps of the method claims are met by the disclosure of the apparatus and methods of the reference(s) as discussed in the rejection of claims 1-2, 4-7, 9-10 and because the steps of the method are easily converted into elements of computer implemented system and methods by one of ordinary skill in the art. With respect to a video control device of claim 9 and the power supply of claim 10, Nakima Fig. 2 para 32, 58, 84-86 teaches an image processing circuit which is coupled to endoscope and sensors/cameras for obtaining images with a plurality of independent power units. See also Segev para 105 discloses multiple processors and power supplies relating to independent power supplies as recited in claim 10.
Regarding claim 21, “wherein the first blended video stream is displayed within an entire area of a screen of the display while the first blended video stream is provided, wherein the second area is smaller than the entire area of the screen of the display” is further rejected on obviousness grounds as discussed in the rejection of claims 1-2, 4-7, 9-12, 14-15, 17-18, 20 wherein DiMaio [0119] After being captured by the ultrasound (US) image capture module 1422, the LapUS data is transmitted to the image fusion block 1414. The image fusion block 1414 fuses the ultrasound images with the 3D endoscopic images that are then coupled into the overlay block 1418. The overlay block 1418 selectively overlays the graphical user interface and the medical image volume onto the fused ultrasound and endoscopic images. (e.g., para 144-152, 175-176 – disclosing that when two video streams are simultaneously displayed, the surgeon operates the functionality to select and deselect whether an overlay is to remain; see also [0154] By overlaying a GUI over live images from the endoscope and further overlaying ultrasound images captured by the ultrasound instrument onto the live images, the SAW fuses graphical objects with physical objects in a physical coordinate frame. DiMaio para 130-131 overlays are determined based on surgeon control such that overlays will appear and disappear based on surgeon inputs.) Segev para [0196] PIP (picture-in-picture) system mode. The display of picture-in-picture (PIP) allows displaying preoperative images, live video feeds, snapshots of previously acquired video, GIFs, and data such as vital signs, and other data, in a window overlaid on the current background image displayed via HMD 102. See also Segev para 96-97 determines which images to stream to surgeon 120 based on the current system mode, as well as one or more inputs received from the surgeon via a user interface. See [0176] In addition to controlling operational settings by surgeon 120 wearing HMD 102, touchscreen 108 allows to control modes, settings and preferences that cannot be controlled by surgeon 120 wearing HMD 102. Touchscreen 108 may display any of the video feeds that are available to system 100, either in their raw form or in the format seen by the wearer of HMD 102, e.g. with overlays, PIPs, etc.). See also Shelton para 210-235 discussing overlay embodiments comprising wherein the first blended video stream is displayed within an entire area of a screen of the display while the first blended video stream is provided, wherein the second area is smaller than the entire area of the screen of the display.
Claim(s) 3, 13, 19 are rejected under 35 U.S.C. 103 as being unpatentable over DiMaio; Simon P. et al. US 20090036902 A1 (hereafter DiMaio) and in further view of SEGEV; Eran et al. US 20210382559 A1 (hereafter Segev) and in further view of Omori; Shigeru US 20090192519 A1 (hereafter Omori) and in further view of NAMIKI; Hirotaka US 20120239058 A1 (hereafter Namiki) and in further view of Shelton, IV; Frederick E. et al. US 20230025827 A1 (hereafter Shelton) and in further view of Shelton, IV; Frederick E. et al. US 20190200844 A1 (hereafter Shelton ‘844) and in further view of Itkowitz; Brandon D. et al. US 20110282140 A1 (hereafter Itkowitz).
Regarding claim 3, “wherein the display comprises a plurality of lines of pixels, wherein the method further comprises: producing the blended video stream by blending the first video stream with the second video stream; and storing, in a line buffer, selected pixels for a set of lines of the plurality of lines based on the blended video stream; wherein the first blended video stream is provided to the display by providing, from the line buffer, the stored selected pixels to the display” is further rejected on obviousness grounds as discussed in the rejection of claims 1-2, however, whereas DiMaio does not use the terms “pixels” and “buffer” wherein Segev’s para 106, 370, 380-382 teaches Image processor 118K processes images received from any of camera system 112, memory 118D, or a GPU, such as by applying a zoom in, zoom out, digital filtering, sharpening, smoothing, color corrections, fusion of several image sources, embedding one image source in another image source in a picture-in-picture form, and the like. See also Shelton para 44 and 182 the image sensor may also comprise a CCD image sensor. The CMOS or CCD sensor may comprise an array of individual light sensing elements (pixels) and para 91, 117 teaching a prefetch buffer.
In an analogous art, with respect to “storing, in a line buffer, selected pixels for a set of lines of the plurality of lines based on the blended video stream; wherein the first blended video stream is provided to the display by providing, from the line buffer, the stored selected pixels to the display,” Shelton ‘844 para 704-711 discloses processing of the captured frames may include performance of simple operations, such as histogram calculations, 2D filtering, and arithmetic operations on arrays of pixels to the performance of more complex tasks, such as object detection, 3D filtering, and the like. Shelton ‘844 also teaches utilizing a frame grabber in combination with a buffer wherein [075-0711] teaches the derived information can be overlaid onto the livestream. In one aspect, the still frames and/or the information resulting from processing the still frames can be communicated to a cloud 104 for data aggregation and further analysis. [0707] In various aspects, the frame grabber 3200 may include a digital video decoder and a memory for storing the acquired still frames, such as, for example, a frame buffer. The frame grabber 3200 may also include a bus interface through which a processor can control the acquisition and access the data and a general purpose I/O for triggering image acquisition. As such, a person of ordinary skill in the art would reasonably infer based on the teachings of Shelton ‘844 that a frame grabber comprising a buffer stores the pixels of a frame.
With respect to “storing, in a line buffer, selected pixels for a set of lines of the plurality of lines based on the blended video stream,” in an analogous art, Itkowitz teaches capturing and displaying camera images of a surgical site on at least one display device at a surgeon console comprising a teleoperated surgical system supporting at least two types of video sources and teaches a sequence of combined video images wherein a first video image is superimposed over a real-time video of the surgical site wherein the superimposing utilizes superimposing a “storing, in a line buffer, selected pixels for a set of lines of the plurality of lines based on the blended video stream; and providing, from the line buffer, the stored selected pixels to the display” See Itkowitz para 12, 67, 75-80, 109.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of DiMaio, Segev, Omori, Namiki and Shelton’s inventions for capturing and displaying multiple camera images of a surgical site on at least one display device at a surgeon console comprising a visual display of picture-in-picture (PIP) video or overlaid content which allows displaying preoperative images, live video feeds, snapshots of previously acquired video, GIFs, and data such as vital signs, and other data, in a window overlaid/superimposed frames comprising pixels on the current background image by further incorporating elements of Shelton ‘844 also teaches utilizing a frame grabber in combination with a buffer wherein Itkowitz recognizes a benefit for capturing and displaying camera images of a surgical site on at least one display device at a surgeon console comprising a teleoperated surgical system supporting at least two types of video sources and teaches a sequence of combined video images wherein a first video image is superimposed over a real-time video of the surgical site wherein the superimposing utilizes a buffer for storing selected pixels for a set of lines of the plurality of lines based on the blended video stream and providing, from the buffer, the stored selected pixels to the display because the modification would improve microsurgical procedures utilizing graphical user interface providing a wide range of hands-free methods to interface with the surgical microscope and control system parameters and facilitate configuring video feeds for an application to be the appropriate pixel dimensions (e.g., full screen, half screen, etc.).
Regarding the device claims 13 and the system claim 19, the claims are grouped and rejected with the method claim 1-3, 5-6 because the steps of the method claims are met by the disclosure of the apparatus and methods of the reference(s) as discussed in the rejection of claims 1-3, 5-6 and because the steps of the method are easily converted into elements of computer implemented system and methods by one of ordinary skill in the art.
Claim(s) 8, 16, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over DiMaio; Simon P. et al. US 20090036902 A1 (hereafter DiMaio) and in further view of SEGEV; Eran et al. US 20210382559 A1 (hereafter Segev) and in further view of Omori; Shigeru US 20090192519 A1 (hereafter Omori) and in further view of NAMIKI; Hirotaka US 20120239058 A1 (hereafter Namiki) and in further view of Shelton, IV; Frederick E. et al. US 20230025827 A1 (hereafter Shelton) and in further view of Shelton, IV; Frederick E. et al. US 20190200844 A1 (hereafter Shelton ‘844).
Regarding claim 8, “wherein the one or more video frames of the first video stream comprises a first set of one or more video frames of the first video stream, wherein producing the second blended video stream comprises retrieving the first set of one or more video frames of the first video stream from a frame buffer and blending the first set of one or more video frames with a second set of one or more video frames of the first blended video stream” the combination of DiMaio, Segev, Omori, Namiki and Shelton as discussed in the rejection of claims 1-2 and 4 render obvious the limitation except for a “buffer” as discussed in “retrieving a first set of one or more video frames of the first video stream from a frame buffer and blending the first set video frames with a second set of one or more video frames of the first blended video stream; and providing the second blended video stream to a display.” See Segev’s para 106-107, 370, 380-382 teaches Image processor 118K processes images received from any of camera system 112, memory 118D, or a GPU, such as by applying a zoom in, zoom out, digital filtering, sharpening, smoothing, color corrections, fusion of several image sources, embedding one image source in another image source in a picture-in-picture form, and the like. See also Shelton not disclosing a “buffer” but teaching para 237-244 teaches generating customized overlay content displayed and is also able to analyze video frames of a primary surgical imaging or video feed to identify missing information for the primary surgical imaging or video feed. The computing system may identify region(s) of poor quality (e.g., artifacts, blurry, obstructed view, etc.) and provides additional imaging to supplement the primary surgical video feed. Shelton teaches fuse imaging or videos obtained from different sources and use the primary imaging for supplementary image and provides and overlay a portion of imaging data from one imaging feed onto a corresponding portion of the imaging data from another imaging feed. Shelton para 210 the computing system may extract one or more portions (e.g., a portion that includes an area of interest) from a surgical video stream and overlay the extracted portion onto another surgical video stream, as described herein. Shelton para 226 area for placing overlay content. Shelton [0228] The size of the overlay region may be determined based on the size of an area on the surgical instrument suitable for content overlay, the size of a fiducial marker in real life, and the size of the fiducial marker in a video frame of the surgical video stream (e.g., a ratio between the real-life size and the imaged size). For example, the computing system may identify the fiducial marker in the video frames of the surgical video stream and determine the respective location, size, and/or orientation of the fiducial marker(s) in respective video frames. For a given video frame, or a group of video frames, the computing system may determine the size, location and/or orientation of the overlay region based on the location, size and/or orientation of the fiducial marker captured therein.
In an analogous art, Shelton ‘844 teaches the deficiency of DiMaio, Segev, Namiki and Shelton wherein Shelton ‘844 para [0704-0711] teaches frame buffer for supplanting images and [1209] and [0561] teaches providing notification relating to failures teaches capturing and para [1048] teaches retrieving a first set of one or more video frames of the first video stream from a frame buffer and blending the first set of video frames with a second set of one or more video frames of the first blended video stream; and providing the second blended video stream to a display.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of DiMaio, Segev, Omori, Namiki and Shelton’s inventions for capturing and displaying multiple camera images of a surgical site on at least one display device at a surgeon console comprising a visual display of picture-in-picture (PIP) video content which allows displaying preoperative images, live video feeds, snapshots of previously acquired video, and data such as vital signs, and other data, in a window overlaid/superimposed on the current background image by further incorporating elements of Shelton ‘844 for utilizing stored video images related to a surgery and further capturing and displaying camera images of a surgical site on at least one display device at a surgeon console comprising a teleoperated surgical system supporting at least two types of video sources and in order to provide a surgeon with a redundant view of an operation to prevent losing visuals during failures and utilizing stored images for supplementing blended images and correct missing data.
Regarding the device claim 16 and the system claim 22, the claims are grouped and rejected with the method claims 8 because the steps of the method claims are met by the disclosure of the apparatus and methods of the reference(s) as discussed in the rejection of claims 8 and because the steps of the method are easily converted into elements of computer implemented system and methods by one of ordinary skill in the art.
CONCLUSION
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALFONSO CASTRO whose telephone number is (571)270-3950. The examiner can normally be reached on Monday to Friday from 10am to 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Nathan Flynn can be reached. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALFONSO CASTRO/Primary Examiner, Art Unit 2421
Prosecution Insights