DETAILED ACTION
This office action is responsive to applicant’s communications filed 03/16/2026.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Application is acknowledged as a National Stage application of PCT/EP2022/087529. Priority to PCT/EP2022/087529 with a priority date of 12/22/2022 is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Information Disclosure Statement
The IDS dated 01/23/2026 has been considered and placed in the application file.
Response to Arguments
Applicant's arguments, filed 03/16/2026, regarding the rejection of the amended independent claims under 35 U.S.C. 103 (including limitations incorporated from dependent claim 2) have been fully considered but they are not persuasive.
Firstly, applicant argues that the combination of Gur with Sun would not have been obvious because Gur teaches transmitting a stream of encoded information extracted from an ROI video, while Sun teaches transmitting a full ROI video stream. Applicant suggests that transmitting a full video stream would negate one of the main advantages of Gur, which is compressing data for low-bandwidth situations such as aviation instrumentation.
In response to applicant's argument, the test for obviousness is not whether the features of a secondary reference may be bodily incorporated into the structure of the primary reference; nor is it that the claimed invention must be expressly suggested in any one or all of the references. Rather, the test is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981).
One of ordinary skill in the art could have recognized the increases in bandwidth since the publication of Gur in 2007, or the possibility of other applications for the invention in which limited bandwidth is of a lesser concern – for instance, the classroom setting of Sun, or the medical setting of Wang. Thus, one of ordinary skill in the art could have taken the relevant concepts from Gur, such as the real-time recognition and analysis of instrument data, with the relevant concepts of Sun, such as the ROI video streaming, to produce an obvious combination in a different context than the original invention of Gur, where adding the features of Sun would not confer the previously discussed disadvantage.
Secondly, applicant argues that Gur does not “identify one or more areas in a sequence of images of an input video stream that show one or more of a control element, a status indicator, a graphical display, and/or a numerical display” because it identifies predetermined shapes in the instruments rather than the instruments themselves.
However, the broadest reasonable interpretation of this claim does not require any specific means of identifying the regions of interest, merely that they are identified. This interpretation is consistent with the specification, which lists several embodiments describing methods of ROI identification, but none of them are explicitly required. Therefore, Gur reads on the claim language.
Additionally, examiner notes that, although not relied upon in the associated claim rejection, the secondary reference of Wang does also teach this limitation:
[0041] “In step 308, the surgical video analysis device 12 identifies one or more objects of interest or regions of interest appearing in at least a subset of the plurality of frames based on the comparison of the video to the historical set of surgical procedure images and the associated contextual information. The surgical video analysis device 12 advantageously identifies multiple objects in the surgical video. The objects, or regions, of interest can include, for example, one or more of a surgical instruments used in the surgical procedure, an anatomical structure, a fluid, or a structural abnormality. In one example, the objects in surgery video are identified using the fully convolutional network (FCN), which learns representations and make the decisions based on local spatial features.”
Therefore, the rejection of the amended independent claims is maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-6, 8, 10-11, 14-17, 19, 25, and 28 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gur et al. (US 20070236366 A1, hereinafter “Gur”) in view of Sun et al. (“Region of interest extraction and virtual camera control based on panoramic video capturing”, IEEE Transactions on Multimedia, Vol. 5 no. 5 (31 Oct 2005), pp. 981-990. https://doi.org/10.1109/TMM.2005.854388; hereinafter “Sun”) and Wang et al. (US 20210182568 A1, hereinafter “Wang”).
Regarding claim 1, Gur teaches: A data processing method of providing information about a device (Abstract “A method and system are provided for acquiring data from an instrument panel or the like by obtaining images of the panel and optically identifying readings of the instruments in the image.”; fig. 3 shows steps), the method comprising:
acquiring an input video stream captured using a camera (fig. 1 camera 20; [0197] “Referring to FIG. 1, in a first embodiment of the present invention, the data acquisition system, generally designated with the reference numeral 100, comprises at least one camera 20 operatively connected to a processing means such as a computer 30, which may be capable of processing images and other data, the system 100 being powered by a suitable power supply 60… The camera 20 is located at a suitable location such as to be able to optically capture the instruments 12 that of interest, and typically that form part of an instrument panel 10.”; [0207] “The camera 20 may provide a video digitizer unit (not shown) for processing successive video frames, and the digitizer unit sends digitized images 110 to the computer 30 via input 21.”), the input video stream comprising a sequence of images showing one or more regions of interest (ROI), wherein each region of interest comprises a representation of a part of the device (fig. 3 step 310; [0075] “The panel may comprise a plurality of said displays, and said processing system is adapted for dividing the said image into a corresponding plurality of regions of interest (ROI), each said ROI comprising one said display, wherein said processing system processes said first image of each said readout to provide a corresponding plurality of coded data streams representative of said images.”) comprising one or more of a control element, a status indicator, a graphical display showing a graph, and/or a numerical display indicator showing a value with one or more digits (Gur fig. 3 steps 310 and 320: instrument panel is divided into regions of interest corresponding to individual instruments; [0213] “Thus, the computer 30 works on the frame buffer in which the image 110 has been downloaded, and divides the image into a plurality of regions of interest (ROI), each corresponding to an instrument or switch being monitored, for example, using the reference markers 113 on the instrument panel.”; [0219] describes recognizing the status of a ROI comprising a dial or a switch; [0227] describes recognizing a display comprising an “alphanumeric character output” on the instrument panel; [0228] describes a variety of instrument types which can be monitored);
identifying one or more areas in the sequence of images of the input video stream that show the one or more of the control element, the status indicator, the graphical display, and/or the numerical display in the one or more regions of interest (fig. 3 step 320; [0211] “In step 320, the system 100, in particular computer 30, enables the part 112 of the image 110 corresponding to each instrument 12 of interest to be separated from the main digital image 110. Typically, such a part 112 comprises the region of interest (ROI) for the particular instrument or switch being monitored. In particular, the portion 114 of this part 112 that is indicative of the reading of the instrument 12, as it appears in the captured image(s) 110, is identified and converted into a digital value that is correlated to this reading, as will be described in greater detail herein.”); and
generating [a visual output] from the input video stream for each of the identified one or more areas (fig. 8 steps 450-460; [0342] to [0345] describes how the digital data generated from the previously obtained ROI video is used to generate a virtual copy of the instrument panel containing a display for each instrument reading corresponding to each ROI of the input video).
Gur does not explicitly teach that the visual output generated from the input video stream is an ROI video stream.
Sun teaches generating an ROI video stream from the input video stream for each of the identified one or more areas (fig. 1 shows input and output video; Abstract “We present a system for automatically extracting the region of interest (ROI) and controlling virtual cameras’ control based on panoramic video… To generate conventional video, a region of interest can be cropped from the panoramic video.”; pg. 983 col. 1 “After the above processing, the output digital ROI video can be recorded or distributed, for example over the web.”).
Gur and Sun are both analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input video. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur with the teachings of Sun to enable it to include actual video footage of each ROI in the output video, rather than just a representation of digital data gathered from the ROI. The motivation would have been to provide additional information to a user; additionally, the invention could be applied towards an instrument or indicator that is too intricate for its output to be easily quantified, and needs to be observed directly, or to another usage of the invention in which bandwidth is not as much of a concern.
The combination of Gur in view of Sun does not explicitly teach that the device is used in a medical environment.
Wang teaches identifying a region of interest from a device used in a medical environment ([0041] “In step 308, the surgical video analysis device 12 identifies one or more objects of interest or regions of interest appearing in at least a subset of the plurality of frames based on the comparison of the video to the historical set of surgical procedure images and the associated contextual information. The surgical video analysis device 12 advantageously identifies multiple objects in the surgical video. The objects, or regions, of interest can include, for example, one or more of a surgical instruments used in the surgical procedure, an anatomical structure, a fluid, or a structural abnormality.”).
Wang and the combination of Gur and Sun are both analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input video. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun with the teachings of Wang to apply it to the field of medicine; generating and analyzing regions of interest corresponding to medical device instrumentation. The motivation would have been to broaden the invention’s applicability since Gur teaches the method and system can be used in many other applications ([0378]) and Sun teaches the method integrates detection, tracking and image recording process to mimic human camera control (section V11) and is done efficiently in both compressed and uncompressed video systems.
Regarding claim 3, the combination of Gur in view of Sun and Wang teaches the method of claim 1, further comprising removing distortion from an ROI video stream, to provide a planar view onto the corresponding region of interest (Gur [0291] “The edge of the ROI image may fail to conform to a perfect circle in the image thereof for various reasons, including, for example, camera induced distortions such as originate when the camera optical axis is not exactly orthogonal to the particular dial being imaged, and/or perspective effects due to the relative wide angle lenses that are required given the proximity of the camera to the instrument and the relatively large field of view often required.”; [0292] to [0294] describe a method of transforming the ROI image to compensate for the camera distortion).
Regarding claim 4, the combination of Gur in view of Sun and Wang teaches the method of claim 1, further comprising providing an ROI video stream as a virtual video source (Sun pg. 981 col. 2 “Cropping a ROI from the panoramic video results in a “virtual camera” that can be panned and zoomed.”).
Gur, Sun, and Wang are analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input video; it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the additional teachings of Sun to enable the output ROI video to be adjusted using a “virtual camera”. The motivation would have been to allow the position of the ROI video output to be adjusted independently of the cameras, reducing the dependence on hardware.
Regarding claim 5, the combination of Gur in view of Sun and Wang teaches the method of claim 1. Gur also teaches a wide-angle lens for video imaging ([291]). However, Sun further teaches wherein the sequence of images in the input video stream are 360° images (Sun fig. 2 shows 360° camera system; pg. 989 col. 1 “Provided there is only one speaker in the scene, this method can be applied to a panoramic view of up to 360° using the system shown in Fig. 2(a).”).
Gur, Sun, and Wang are analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input video; it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the additional teachings of Sun to use a 360-degree panoramic video input. The motivation would have been to enable the invention to observe devices or instrumentation located throughout a room, rather than just a single instrument panel as taught by Gur.
Regarding claim 6, the combination of Gur in view of Sun and Wang teaches the method of claim 1, wherein the identifying the one or more areas comprises identifying the device in the input video stream; obtaining information that defines the corresponding region of interest from a database; and identifying the area based on the obtained information (Gur [0214] “Optionally, the laboratory calibration may be such that the computer 20 may be programmed with a library of control panel configurations in a data base, and the appropriate configuration chosen according to the specific type of panel 10. Such a choice may be made manually, for example. Alternatively, an optical character recognition program may be adapted for comparing a datum image of the control panel with each configuration in the data base, and the best match with respect thereto is then chosen.”).
Regarding claim 8, the combination of Gur in view of Sun and Wang teaches the method of claim 1, wherein the identifying the one or more areas comprises identifying the one or more areas using the position of the corresponding device relative to the camera (Gur [0209]-[0210] describes how fiducial markers located on the instrument panel can be used to track and compensate for the movement of the instrument panel relative to the camera).
Regarding claim 10, the combination of Gur in view of Sun and Wang teaches the method of claim 1, further comprising acquiring user input data indicating a type of the device, wherein an area in the images of the input video stream is identified based on the indicated type of the device (Gur [0214] “Optionally, the laboratory calibration may be such that the computer 20 may be programmed with a library of control panel configurations in a data base, and the appropriate configuration chosen according to the specific type of panel 10. Such a choice may be made manually, for example.”; the type and location of devices on the control panel is dependent on the type of panel).
Regarding claim 11, the combination of Gur in view of Sun and Wang teaches the method of claim 1, wherein the identifying the one or more areas in the sequence of images of the input video stream comprises identifying at least one of the one or more areas using artificial intelligence (Wang [0041] “In step 306, the surgical analysis device 12 applies the machine learning model to the plurality of frames of the videos(s) to compare the plurality of frames of the obtained video to the historical set of surgical procedure images and correlated sets of contextual data obtained in step 300. In step 308, the surgical video analysis device 12 identifies one or more objects of interest or regions of interest appearing in at least a subset of the plurality of frames based on the comparison of the video to the historical set of surgical procedure images and the associated contextual information. The surgical video analysis device 12 advantageously identifies multiple objects in the surgical video. The objects, or regions, of interest can include, for example, one or more of a surgical instruments used in the surgical procedure, an anatomical structure, a fluid, or a structural abnormality. In one example, the objects in surgery video are identified using the fully convolutional network (FCN), which learns representations and make the decisions based on local spatial features.”).
Gur, Sun, and Wang are analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input video; it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the additional teachings of Wang to locate the regions of interest using a machine learning model. The motivation would have been to take advantage of more recent developments in image recognition, increasing the accuracy of ROI detection.
Regarding claim 14, the combination of Gur in view of Sun and Wang teaches the method of claim 1, further comprising tracking a region of interest in the input video stream over time (Sun pg. 981 col. 2 “We integrate ROI detection and tracking for virtual camera control. The ROI location is then processed using a Kalman filter to steer a virtual camera for display or recording. The Kalman filter output smoothes the ROI motion to mimic the response of a human camera operator (as discussed later).”, pg. 986-987 section V “Tracking Using a Kalman Filter” explains in additional detail).
Gur, Sun, and Wang are analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input video; it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the additional teachings of Sun to track a moving region of interest. The motivation would have been to enable the invention to function on devices or instrumentation that are moving relative to the camera, rather than being restricted to stationary installations, increasing its applicability.
Regarding claim 15, the combination of Gur in view of Sun and Wang teaches the method of claim 1, wherein the method further comprises transforming a status of the control element into a graphical representation and embedding the graphical representation in the corresponding ROI video stream (Gur fig. 8, digital data produced from visual ROI images is converted into a graphical representation and displayed on a virtual instrument panel; [0342] to [0345] “In step 450, the computer 230 displays in display 220 a virtual image of the panel 10, which is typically stored in the memory of the computer. This image may be, for example, a photographic image of the panel, or a graphic or virtual representation thereof. In either case, virtual windows 212 are provided for the actual dials or other markers that indicate the reading of instruments, or switches, digital readouts or displays, and so on, and are left blank at this stage. Indicia representing the scales of each instrument are provided to enable the viewer to read the data from the position of the dial on the display, or the position of a control lever, etc.
In step 460, the computer can then display an image of an indicator in each window 212 in display 220, such that a dial appears at an angle with respect to a known datum corresponding to the received digital value, such as for example a datum that is related to marker 130, when part 112 refers to a dial-type instrumentation.
Alternatively, for windows 212 corresponding to instruments providing a digital readout, the corresponding digital value received by computer 230 is converted to an image of the digits corresponding to this data. Similarly, changes in position of a level, knob, and so on, or different types of display can also be shown in the appropriate window 212 in a manner similar to that originally displayed in panel 10.
Thus, images corresponding to the digital data, corrected from the position of the corresponding datums, are superimposed over an image of this instrument 12, i.e. at the appropriate window 212, and optionally also of the rest of the instrument panel 10, Thus, the display 220 can display a virtual image of the control panel 10, having virtual windows 212 corresponding to each instrument 12. Any changes in the readings of the real instruments 12 are then simulated in the appropriate window 212 of display 220.”;
also see [0219] and [0228] for a description of how the visual images for each ROI are first converted to digital data).
Regarding claim 16, the combination of Gur in view of Sun and Wang teaches the method of claim 1, wherein the method further comprises transforming the status indicated by the status indicator (Gur [0043] teaches that the parameter being measured by an instrument may include “at least one of airspeed, altitude, pitch, roll, yaw, turn rate, vertical speed, horizontal situation (compass heading), engine rpm, oil status, fuel status, oil temperature, Mach number, chronological time.”, where several possible parameters are status indicators) into a graphical representation and embedding the graphical representation in the corresponding ROI video stream (Gur fig. 8, digital data produced from visual ROI images is converted into a graphical representation and displayed on a virtual instrument panel; [0342] to [0345] describe this process in full detail (see claim 15 for full text of this section), also see [0219] and [0228] for a description of how the visual images for each ROI are first converted to digital data).
Regarding claim 17, the combination of Gur in view of Sun and Wang teaches the method of claim 1, wherein the generating the ROI video stream comprises generating the ROI video stream from at least one of two or more raw video streams obtained from two or more camera modules comprising the camera, wherein the acquired input video stream is a synthetization of the two or more raw video streams (Sun pg. 982 col. 1 “In our work, we use the FlyCam [10] system to capture panoramic video. FlyCam stitches video from multiple cameras to create a high-resolution output.”; fig. 2 shows camera comprised of multiple camera modules; fig. 1 shows panoramic image synthesized from multiple camera module inputs).
Gur, Sun, and Wang are analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input video; it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the additional teachings of Sun to generate the input panoramic video using multiple camera feeds. The motivation would have been to be able to generate 360 degree panoramic video input, as taught by Sun, without the use of specialized camera hardware.
Regarding claim 19, the combination of Gur in view of Sun and Wang teaches the method of claim 1, further comprising generating an output video stream from at least one ROI video stream (Sun pg. 983 col. 1 section III.B “General System Architecture”: “The video is further processed through ROI detection, Kalman filtering, and virtual camera control. After the above processing, the output digital ROI video can be recorded or distributed, for example over the web.”; fig. 11 shows output video stream).
Gur, Sun, and Wang are analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input video; it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the additional teachings of Sun to enable it to include actual video footage of each ROI in the output video, rather than just a representation of digital data gathered from the ROI. The motivation would have been to provide additional information to a user; additionally, the invention could be applied towards an instrument or indicator that is too intricate for its output to be easily quantified, and needs to be observed directly.
Regarding claim 25, it is rejected with the same rationales, references, and motivations to combine as claim 1, with the additional limitation of a computer (Gur fig. 1 element 30, fig. 2 element 230) comprising:
a processor (Gur [0091] “According to a further aspect of the invention, a computer readable medium storing instructions for programming a processing means of a system to perform a data acquisition method and/or a data display method according to another aspects of the invention.”; [0329] “a processor such as a computer 230…”);
a non-transient memory device (Gur [0413] “The present invention also relates to a computer readable medium storing instructions for programming a processor means of the data acquisition system of the invention to perform a data acquisition method of the invention.”; [0414] “The present invention also relates to a computer readable medium storing instructions for programming a processor means of a data display system of the invention to perform the data display method of the invention.”; [0415] “Such computer readable media may include, for example, optical discs, magnetic discs, magnetic tapes, RAM memory, and so on.”); and
logic stored in the non-transient memory device, the logic being executable by the processor to perform a method (Gur [0413] “The present invention also relates to a computer readable medium storing instructions for programming a processor means of the data acquisition system of the invention to perform a data acquisition method of the invention.”; [0414] “The present invention also relates to a computer readable medium storing instructions for programming a processor means of a data display system of the invention to perform the data display method of the invention.”).
Regarding claim 28, the combination of Gur in view of Sun and Wang teaches the method according to claim 1, further comprising:
extracting the one or more areas of the input video stream that show the one or more regions of interest from the input video stream (Sun pg. 981 abstract “We present a system for automatically extracting the region of interest (ROI) and controlling virtual cameras’ control based on panoramic video.”); and
providing the extracted one or more areas of the input video stream as the ROI video stream for each of the one or more areas (Sun pg. 983 col. 1 section III.B “General System Architecture”: “The video is further processed through ROI detection, Kalman filtering, and virtual camera control. After the above processing, the output digital ROI video can be recorded or distributed, for example over the web.”; fig. 11 shows output video stream).
Gur, Sun, and Wang are analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input video; it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the additional teachings of Sun to enable it to include actual video footage of each ROI as the output video, rather than just a representation of digital data gathered from the ROI. The motivation would have been to provide additional information to a user; additionally, the invention could be applied towards an instrument or indicator that is too intricate for its output to be easily quantified, and needs to be observed directly.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gur (US 20070236366 A1) in view of Sun (“Region of interest extraction and virtual camera control based on panoramic video capturing”) and Wang (US 20210182568 A1) as applied to claim 6 above, and further in view of Lee (US 20180082360 A1).
Regarding claim 7, the combination of Gur in view of Sun and Wang teaches the method of claim 6, but does not explicitly teach wherein identifying the device comprises identifying an identification tag of the device in the input video stream.
Lee teaches wherein identifying the device comprises identifying an identification tag of the device in the input video stream ([0069] “In operation S430, the recognizer 330 may recognize products displayed in the corresponding store in the internal image of the offline store. For example, the recognizer 330 may recognize a product in an image by recognizing a unique identification code, such as a barcode, a quick read (QR) code, etc., printed on the product, or a text such as a brand name.”).
Lee and the combination of Gur in view of Sun and Wang are both analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the teachings of Lee to enable it to recognize a ROI based on an identification code such as a barcode or QR code. The motivation would have been to add an extra safeguard to ensure accurate ROI detection.
Claim(s) 9, 12-13, 18, and 20-23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gur (US 20070236366 A1) in view of Sun (“Region of interest extraction and virtual camera control based on panoramic video capturing”) and Wang (US 20210182568 A1) as applied to claims 1 and 19 above, and further in view of Warner et al. (US 20160048636 A1, hereinafter “Warner”).
Regarding claim 9, the combination of Gur in view of Sun and Wang teaches the method of claim 1, but does not explicitly teach further comprising acquiring user input data indicating a spatial position in the input video stream, wherein an identified area in the images of the input video stream is an area including the position indicated by the user input data.
Warner teaches acquiring user input data indicating a spatial position in the input video stream, wherein an identified area in the images of the input video stream is an area including the position indicated by the user input data (Warner fig. 4; [0055] “In order to select a region of interest in the video output of a data source, a user may utilize a pixel-selection tool to define the region of interest, for example, the visualized ultrasound data. Controller 101 may then capture only the defined region of interest for inclusion in the corresponding display information element.”).
Warner and the combination of Gur in view of Sun and Wang are both analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the teachings of Warner to allow a user to manually select a region of interest in the input video. The motivation would have been to give a user the option of having finer control of the selection of a region of interest, rather than relying on automated systems.
Regarding claim 12, the combination of Gur in view of Sun and Wang teaches the method of claim 1, but does not explicitly teach wherein the identifying the one or more areas in the sequence of images of the input video stream comprises identifying at least one of the one or more areas based on an aspect of a medical procedure that is currently being performed.
Warner teaches wherein the identifying the one or more areas in the sequence of images of the input video stream comprises identifying at least one of the one or more areas based on an aspect of a medical procedure that is currently being performed ([0056] “In one example, a defined region of interest may vary from a data source 119 as a case progresses, so that the data source 119 itself may adaptively define the proper selection region in order to capture the information of interest. For example, a user may indicate that he or she wants to capture the ultrasound picture region, and the picture region to pixel mapping may be continuously updated by the ultrasound system 130 if the screen layout of the ultrasound system 130 changes. In this way, a data source 119 may be hard-coded to intelligently provide an appropriate video output for aggregated display on a single screen.”).
Warner and the combination of Gur in view of Sun and Wang are both analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the teachings of Warner to enable it to select a ROI to fit the current step of an ongoing medical procedure. The motivation would have been to improve the readability and ease of use for medical personnel performing a complex procedure by displaying only what they need to see at a specific time.
Regarding claim 13, the combination of Gur in view of Sun and Wang and further in view of Warner teaches the method of claim 12, wherein the identifying the at least one of the one or more areas in the sequence of images of the input video stream comprises identifying at least one of the one or more areas based on a workflow step of the medical procedure that is currently being performed (Warner, same citation and motivation to combine as claim 12).
Regarding claim 18, the combination of Gur in view of Sun and Wang teaches the method of claim 1, but does not explicitly teach further comprising adding a graphical representation of device information to the ROI video stream
Warner teaches adding a graphical representation of device information to the ROI video stream (Warner figs. 2-4, each display element is labeled with its source or target, including video stream; [0052] to [0053] describes how a video display element may be a region of interest of a larger source video).
Warner and the combination of Gur in view of Sun and Wang are both analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the teachings of Warner to display information about the source or target of each ROI subdivision of the output video. The motivation would have been to improve the readability and ease of use by allowing users to immediately read, at a glance, what each displayed device component is.
Regarding claim 20, the combination of Gur in view of Sun and Wang teaches the method of claim 19, but does not explicitly teach further comprising arranging the at least one ROI video stream in the output video stream based on user preferences.
Warner teaches arranging the at least one ROI video stream in the output video stream based on user preferences (Warner fig. 3; [0031] “User interface 117 enables the arbitrary placement of any display information element, from any source, in any arrangement to be simultaneously displayed on display 110. To that end, controller 101 may include WYSIWYG software stored in non-transitory memory 107 that when executed by processor 105 enables a user to create one or more display configurations, where the display configurations include an arrangement of display information elements. Each configuration of display information elements may be saved in memory 107 as a display macro for later use. The pre-configuration of a display macro using WYSIWYG software is described further herein and with regard to FIG. 3.”; [0052]-[0053] teaches that a display information element may comprise a region of interest video).
Warner and the combination of Gur in view of Sun and Wang are both analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the teachings of Warner to provide an interface to enable a user to customize and rearrange the output display screen as desired. The motivation would have been to improve the readability and ease of use by allowing users to alter the output display in a way that best suits their needs.
Regarding claim 21, the combination of Gur in view of Sun and Wang teaches the method of claim 19, but does not explicitly teach further comprising arranging the at least one ROI video stream in the output video stream based on a medical procedure that is currently performed.
Warner teaches arranging the at least one ROI video stream in the output video stream based on a medical procedure that is currently performed (Warner [0032] “Multiple display macros may be utilized during a single medical procedure. For example, a user may establish a set of display macros for a single procedure, with each display macro emphasizing data from a particular, different data source. In some examples, each display macro may be related to an event during a medical procedure so that the display 110 automatically switches to an appropriate display macro when the corresponding event occurs. In other examples, user interface 117 may enable a user to manually switch between display macros, and even further may enable a user to adjust an arrangement of display information elements for a given display macro in real-time during the medical procedure.”).
Warner and the combination of Gur in view of Sun and Wang are both analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the teachings of Warner to modify the arrangement of the output display to fit the current step of an ongoing medical procedure. The motivation would have been to improve the readability and ease of use for medical personnel performing a complex procedure by displaying only what they need to see at a specific time.
Regarding claim 22, the combination of Gur in view of Sun and Wang and further in view of Warner teaches the method of claim 21, further comprising arranging the at least one ROI video stream in the output video stream based on a workflow step of the medical procedure that is currently performed (Warner, same citation and motivation to combine as claim 21).
Regarding claim 23, the combination of Gur in view of Sun and Wang teaches the method of claim 19, as well as using artificial intelligence to detect a region of interest (Wang, see claim 11).
The combination of Gur in view of Sun and Wang does not explicitly teach further comprising using artificial intelligence to arrange the at least one ROI video stream in the output video stream.
Warner teaches a means of automatically arranging the at least one ROI video stream in the output video stream (see claims 21-22).
Warner and the combination of Gur in view of Sun and Wang are both analogous to the claimed invention because they pertain to the same issue of determining a region of interest from an input image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Gur in view of Sun and Wang with the teachings of Warner to use artificial intelligence to modify the arrangement of the output display to fit the current step of an ongoing medical procedure. The motivation would have been to fully automate the process of creating display macros as taught by Warner (see claim 11) instead of doing it manually, saving the user time.
References Cited
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wolf et al. (US 20210012868 A1) teaches a method of identifying a region of interest containing a medical instrument (see [0303]); the method may take into account the type and stages of the medical procedure (see [0115], [0338], [0522]).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENJAMIN STATZ whose telephone number is (571)272-6654. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571)272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BENJAMIN TOM STATZ/Examiner, Art Unit 2611
/TAMMY PAIGE GODDARD/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights