Detailed Action
1. Claims 1-21 are pending in this Application.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
3. Claims 1-5,7-15, 17-19 and 21are rejected under 35 U.S.C. 103 as being unpatentable over PELISSIER et al. ( hereafter PELISSIER), US 20170105701A, pub. 04/20/2017, in view of DORN et al,(hereafter DORN), US 20220174367 A1, pub. 06/02/2022.
As to claim 1, PELISSIER teaches An assistive system for providing remote assistance to a local operator of a medical imaging device having an electronic imaging device controller and a controller display (Abstract, Fig. 1, [0061], A user operates apparatus 100A to perform an ultrasound scan on a patient. An expert uses apparatus 100B to review the ultrasound procedure information in real time as it happens, and he expert provides real-time guidance and feedback from apparatus 100B back to the user at apparatus 100A), the assistive system comprising: an assistor electronic device including an processor, an assistor display, and at least one user input device; (Fig. 1 unit 100B, [0071], [0072], [0074], [0076]) a video router (Fig. 1 unit 106 network, [0070], [0074] and [0097]) configured to:
receive controller video from the electronic imaging device controller; (Figs. 1 and 3 pars. [0070] and [0077], Apparatus 100A transmits ultrasound imaging data and/or video through a data communication network 106 to remote apparatus 100B);
receive overlay content or video from the assistor electronic device; (Fig. 2 unit 212, Figs.1, 4 and 6, [00 71],[ 0072], [0097], [0045]- [0046], [0165]-[0166], the remote expert at apparatus 100B may add graphical elements to an image and provide this feedback to the user of apparatus 100A in a substantially real-time screen sharing mode) ;
output the controller video to the assistor electronic device; and (Figs.1 and 3, [0070], [0077], Apparatus 100A transmits ultrasound imaging data and/or video through a data communication network 106 to remote apparatus 100B);
output annotated video to the controller display, whereby the annotated video is displayed on the controller display; ( Fig.1, Fig.2 unit 212; Figs.4 and 6, [0071], [0072], [0097], [0145]- [0146], [0165]- [0166, the remote expert at apparatus 100B may add graphical elements to an image and provide this feedback to the user of apparatus 100A in a substantially real-time screen sharing mode) wherein the assistor electronic device is programmed to:
provide a graphical user interface operative to display the controller video on the assistor display( Fig.6 , par [0145], [146], Fig. 6 shows an example remote user interface with drag and drop graphical elements that a remote user can select to provide feedback. In the example, the expert drags and drops a targeting icon 121 onto patient video stream 404 to indicate the desired location of the probe. The remote interface 118 sends a plurality of messages for a drag and drop event so that a user at apparatus 100A sees where the expert has dragged the icon in real time while the icon is being dragged by the expert at remote interface) receive graphical annotations of the controller video from the at least one user input device, the assistor electronic device further programmed to generate the overlay content or video comprising at least the graphical annotations; and (Fig.6 , par [0145], [146], the remote interface 118 sends a plurality of messages for a drag and drop event so that a user at apparatus 100A sees where the expert has dragged the icon in real time while the icon is being dragged by the expert at remote interface)) wherein the annotated video comprises the controller video Fig.6 , par [0145], [146], the remote expert may add graphical elements to an image. These graphical elements may be provided to the user of apparatus 100A in a substantially real-time screen sharing mode. The graphical elements may include arrows, circles, or other shapes to indicate a point or region of interest).
It is noted that PELISSIER does not specifically teach “video annotation is performed by the video router”, although PELISSIER teaches video annotation as discus above (Fig.6 , par [0145], [0146]
On the other hand Dorn teaches video annotation is performed by the video route
(Fig.1 units 200, 210, Fig.2 units 100A,100B ,[0109]-[0113], annotation processor 210 executing on a cloud server 200 of the cloud system in as showmen in Figs. 1 and 2) discloses, for example, a network device (cloud server 200 in fig. 2) performing a video annotation provided by a user (producer 100A in fig. 2) and consumed by another user (consumer 100B in fig. 2).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify PELISSIER by incorporating the cloud-based execution of a video annotation processor as taught by Dorn.
A person of ordinary skill in the art would have been motivated to combine these teachings to provide users of PELISSIER with massive scalability, accelerated computing speed, and enhanced clinical collaboration.
As to claim 2, PELISSIER teaches the assistor electronic device is programmed to generate the overlay content or video as the annotated video comprising the controller video overlaid with the graphical annotations, and the video router is configured to output the annotated video received from the assistor electronic device to the controller display (Fig.1 Fig.2 unit 212, Figs. 4 and 6, the remote expert at apparatus 100B may add graphical elements to an image and provide this feedback to the user of apparatus 100A in a substantially real-time screen sharing mode).
As to claim 3, the combination of PELISSIER and Dorn teaches the assistor electronic device is programmed to generate the overlay content or video comprising the graphical annotations without the controller video PELISSIER: Generating the annotated video by combining the controller video (e.g. ultrasound imaging data(see in [0070] and [0077] ) and the overlay content, such as a symbolic graphical element or an overlay video, (e.g. graphical annotations, e.g. arrows, circles or other shapes, by remote expert (see [0145], [0165] and [0166]) at a digital signal processor in the video router (Dorn: annotation processor 210 executing on a cloud server 200 of the cloud system in Fig. 1; Fig. 2, (see par. 109-113)) .
As to claim 4, PELISSIER teaches the overlay content or video received from the assistor electronic device is overlay content in which the graphical annotations are represented as vectorized or symbolic graphical elements ([0145], [146], [0145], [0165] [0166], [0070] and [0077] Generating the annotated video by combining the controller video (e.g. ultrasound imaging data(see in [0070] and [0077] ) and the overlay content, such as a symbolic graphical element or an overlay video, (e.g. graphical annotations, e.g. arrows, circles or other shapes, by remote expert (see [0145], [0165] and [0166) and the digital signal processor is programmed to generate the annotated video by converting the vectorized or symbolic graphical elements to a screen overlay and fusing the screen overlay with the controller video to produce the annotated video (Fig.6 , [0145], [146], the remote expert may add graphical elements to an image. These graphical elements may be provided to the user of apparatus 100A in a substantially real-time screen sharing mode. The graphical elements may include arrows, circles, or other shapes to indicate a point or region of interest. Generating the annotated video by combining the controller video (e.g. ultrasound imaging data(see in [0070] and [0077] ))
As to claim 5, PELISSIER teaches the overlay content or video received from the assistor electronic device is overlay video in which the graphical annotations are represented as overlay video, and the digital signal processor is programmed to generate the annotated video by fusing the overlay video with the controller video to produce the annotated video (Fig.6 , [0145], [146], the remote expert may add graphical elements to an image. These graphical elements may be provided to the user of apparatus 100A in a substantially real-time screen sharing mode. The graphical elements may include arrows, circles, or other shapes to indicate a point or region of interest. Generating the annotated video by combining the controller video (e.g. ultrasound imaging data(see in [0070] and [0077]).
As to claim 7, PELISSIER teaches a local electronic device operable by the local operator, wherein the video routers configured to receive the overlay content or video from the assistor electronic device via the local electronic device and to output the controller video to the assistor electronic device via the local electronic device (Fig.1, Fig.2 unit 212,, Figs., 3,4 and 6 [0065, [0070] [0072], l[0077], [0097], [0045], [0165] and [0166]; fig. 1, user interface 102 may comprise a personal computer; the remote expert at apparatus 100B may add graphical elements to an image and provide this feedback to the user of apparatus 100A in a substantially real- time screen sharing mode; apparatus 100A transmits ultra- sound imaging data and/or video through a data communication network 106 to remote apparatus 100B).
As to claim 8, PELISSIER teaches the video router comprises a keyboard, video, monitor witch (Abstract, [0076] If the remote expert considers that the user's technique for performing the ultrasound scan could be improved then the remote expert may provide feedback to the user by way of remote user interface 118. To achieve this, the expert may activate physical controls (such as buttons, joysticks, trackballs, switches, computer mice or the like); interact with a tough screen of remote interface 118; issue voice commands; use a stylus or other pointing device to operate controls presented by a graphical user interface of remote interface 118 or the like.
Claim 9 is rejected the same as claim 1 except claim 9 is directed to a method claim. All the limitations of claim 9 addressed in calim1. Thus the analysis applied to claim 1 above is also applicable to claim 9.
As to claim 10, PELISSIER teaches splitting the controller video into a plurality of feeds; transmitting one of the feeds to the assistor electronic device; and transmitting another of the feeds to a screen overlay processor that also receives the graphical annotations from the assistor electronic device and that generates the controller video overlaid with the graphical annotation ([0070] –[0071], and [0077], When remote feedback is desired apparatus 100A is caused to transmit one or both of the ultrasound imaging data and the video stream through a data communication network 106 to remote apparatus 100B. Apparatus 100A may send other data to apparatus 100B instead of or in addition to the video stream. Additional data may include positional measurements of probe 103, positional measurements of the patient, measurements of forces applied to the patient by probe 103, or a secondary video, for example. Generating the annotated video by combining the controller video (e.g. ultrasound imaging data, and the overlay content, such as a symbolic graphical element or an overlay video, (e.g. graphical annotations, e.g. arrows, circles or other shapes, by remote expert (see [0145], [0165] and [0166])
As to claim 11, PELISSIER teaches splitting comprises employing a Keyboard, Video, & Mouse switch to split the feed into the plurality of feeds([0076] If the remote expert considers that the user's technique for performing the ultrasound scan could be improved then the remote expert may provide feedback to the user by way of remote user interface 118. To achieve this, the expert may activate physical controls (such as buttons, joysticks, trackballs, switches, computer mice or the like); interact with a tough screen of remote interface 118; issue voice commands; use a stylus or other pointing device to operate controls presented by a graphical user interface of remote interface 118 or the like.)
Claim 12 is rejected the same as claim 2 except claim 12 is directed to a method claim. The analysis applied to claim 1 above is also applicable to claim 12.
As to claim 13, PELISSIER teaches the controller video overlaid with the graphical annotations is generated by a screen overlay component that comprises a digital signal processing component (Fig.1 Fig.2 unit 212, Figs. 4 and 6, the remote expert at apparatus 100B may add graphical elements to an image and provide this feedback to the user of apparatus 100A in a substantially real-time screen sharing mode)).
As to claim 14, PELISSIER teaches the controller video is an analog signal, the method further comprising: converting the controller video to digital controller video that is processed by the DSP component using an analog-to-digital converter ; and converting the controller video overlaid with the graphical annotations to an analog signal that is displayed by the controller display using a digital-to-analog converter (Abstract Fig.1 An apparatus and method for providing remote expert feedback to an operator of an ultrasound imaging machine. The remote feedback consists of graphical icons that are dragged and dropped onto a video stream of the patient being scanned or the ultrasound image being captured and shared in real-time with the local operator to help improve their scanning technique. The apparatus inherently includes both digital-to-analog and analog-to-digital converter converter).
Claim 15 is rejected the same as claim 7 except claim 15 is directed to a method claim . The analysis applied to claim 7 above is also applicable to claim 15.
As to claim 17, PELISSIER teaches the controller display includes a touchscreen, and the method further includes: receiving, via the touchscreen, one or more graphical annotations from the local operator, wherein the controller video is further overlaid with the graphical annotations from the local operator (Fig.6, [0072], [0139] and [0146] FIG. 6 shows an example remote user interface with drag and drop graphical elements that a remote user can select to provide feedback. In the example, the expert drags and drops a targeting icon 121 onto patient video stream 404 to indicate the desired location of the probe. In some embodiments remote interface 118 sends a plurality of messages for a drag and drop event so that a user at apparatus 100A sees where the expert has dragged the icon in real time while the icon is being dragged by the expert at remote interface 11).
Claim 18 is rejected the same as claim 1 except claim 18 is directed to system claim. All the limitations of claim 18 are addressed in calim1. Thus the analysis applied to claim 1 above is also applicable to claim 18.
Claim 19 is rejected the same as claim 4 except claim 19 is directed to a system claim. All the limitation of claim 19 is addressed in claim 4. Thus analysis applied to claim 4 above is also applicable to claim 19.
Claim 21 is rejected the same as claim 7 except claim 21 is directed to a system claim . The analysis applied to claim 7 above is also applicable to claim 21.
3. Claims 6,16 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over PELISSIER, US 20170105701A, in view of DORN, US 20220174367 A1, further in view of
LIN SHENG YUEH ( hereafter LIN), TW I310513 B, pub 06/01/2009
As to claim 6, PELISSIER teaches the video router includes an overlay 0070] –[0071], [0077]).
However modified , PELISSIER does not specifically teach “overlay disable switch”
On the other hand LIN teaches disable switch operable by the local operator to cause the video router to stop combining the overlay content ( page3, when it is determined in step S220 that the digital video signal fails to pass the HDCP standard annotation, the digital video signal representing the current image may be a simple image or a static image signal such as a text, rather than a dynamic image signal, so By automatically switching the LCD monitor to the movie playback mode, several other display modes are provided for the user to manually switch (S25〇)).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify PELISSIER by incorporating a well -known method of manually switching a video annotation display as taught by Lin.
A person of ordinary skill in the art would have been motivated to combine these teachings to provide users of PELISSIER with complete control over the display state, which eliminates automatic switching errors, unintended distractions, and timing mismatches during live presentations.
Claim 16 is rejected the same as claim 6 except claim 16 is directed to a method claim . The analysis applied to claim 6 above is also applicable to claim 16.
Claim 20 is rejected the same as claim 6 except claim 20 is directed to a system claim . The analysis applied to claim 6 above is also applicable to claim 20.
Prior art not used in rejections but pertinent to the claims or disclosure.
a. “Real-Time Video Annotations for Augmented Reality”, Department of Engineering University of Cambridge, Cambridge, pub. 2005, to Edward Rosten et al.. disclosed:
Augmented reality (AR) provides an intuitive user interface to present information in the context of the real world. A common application is to overlay screen-aligned annotations for real world objects to create in-situ information displays for users. While the referenced object’s location is fixed in the view the annotating labels should be placed in such a way as to not interfere with other content of interest such as other labels or objects in the real world. We present a new approach to determine and track areas with less visual interest based on feature density and to automatically compute label layout from this information. The algorithm works in under 5ms per frame, which is fast enough that it can be used with existing AR systems. Moreover, it provides flexible constraints for controlling label placement behaviour to the application designer. The resulting overlays are demonstrated with a simple hand-held augmented reality system for information display in a lab environment (see Abstract)
Contact Information
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Mekonen Bekele whose telephone number is (469) 295-9077.The examiner can normally be reached on Monday -Friday from 9:00AM to 6:50 PM Eastern Time.
If attempt to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Eng, George can be reached on (571) 272-7495.The fax phone number for the organization where the application or proceeding is assigned is 571-237-8300. Information regarding the status of an application may be obtained from the patent Application Information Retrieval (PAIR) system. Status information for published application may be obtained from either Private PAIR or Public PAIR.
Status information for unpublished application is available through Privet PAIR only.
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have question on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866.217-919 (tool-free)
/MEKONEN T BEKELE/Primary Examiner, Art Unit 2699
