DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 11, 14 and 17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
REGARDING CLAIM 11, the claim recites “perform AI inference on the frame of the main video corresponding to a time point when or after the AI inference result is generated” in lines 3-4. The claim language is unclear as to what frame is referred to. The parent claim 1 recites “a second frame of the main video stream, wherein the second frame of the main video stream is a frame corresponding to a time point when or after the AI inference result is generated.” As currently drafted, claim 11 is unclear as to whether the claim is intended to refer the same previously introduced frame (i.e. the second frame) or to an entirely different frame. Use of the antecedent “the” indicates “the frame” is previously introduced, but then the claim specifies “the frame” corresponds to “a time point”, which has not been previously introduced. Accordingly, the claim is rendered indefinite. Furthermore, nothing in the specification lends any support for clarifying the meaning of the claim language. As currently drafted, claim is interpreted as a frame at any time point when or after the first AI inference result.
REGARDING CLAIMS 14 and 17, the claims include substantially the same language and issues as claim 11 and are rendered indefinite for the same reasons as claim 11 set forth above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 6-8, and 11-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Kadtan et al. (US 2025/0069409 A1) in view of
Quist et al. (US 2022/0406061 A1).
REGARDING CLAIM 1, Kadtan discloses:
A computing device comprising: (Kadtan, Fig. 8A and ¶159: autonomous vehicle; ¶163: controllers, including system on chips and GPUs as components of vehicle 800, including onboard computing devices to process sensor signals; ¶168: vehicle 800 used to implement selective video tracking described in Fig. 4)
A first processor, configured to continuously receive a main video stream (Kadtan, ¶98 discloses performing dynamic adjustment based on object movement detected from live streaming video at a given time, with neural network detecting movement of identified objects; ¶101: one or more streams of videos received from communication interface 415; ¶164: controllers provide signals in response to received sensor data from cameras; ¶¶173-174 discloses cameras within vehicle 800 providing sensor data, used for object detection and classification and object tracking; ¶178: various cameras described in FIG. 8B may be used to capture one or more video streams on which embodiments of selective video tracking described in FIGS. 1-4 may be implemented; ¶210: processors 810 including real-time camera engine including dedicated processor subsystem for handling real-time camera management; also ¶¶214-215: system on chip (SoC) receiving video and input from cameras, freeing CPUs 806 from routine data management tasks, vs. ¶217: computer vision algorithms executed on CPUs; also see ¶222: CPUs 818 coupled to SoCs 804, and performing functions other than SoCs; Also note Kadtan discloses a plurality of computer architectures read on by the separation of two processors receiving video, including ¶¶250-251 disclosing multiple computers performing computer vision on images, including one with neural network, MCU, separate from a computer performing standard computer vision); and
A second processor, configured to; (Kadtan, Fig. 4 and ¶111: processor 410 performing processes, including video tracking module 430 – see Fig. 4; CPUs 818 coupled to SoCs 804, and performing functions other than SoCs; Also note Kadtan discloses a plurality of computer architectures read on by the separation of two processors receiving video, including ¶¶250-251 disclosing multiple computers performing computer vision on images, including one with neural network, MCU, separate from a computer performing standard computer vision; Also ¶359 discloses operations of neural network inferencing performed on one or more GPGPUs; Fig. 17 and ¶369-372 discloses computer system having multiple processors separate from I/O hub driving display controller, and performing selective video tracking of Fig. 4)
Perform [resizing] on the main video stream to generate a [resized] video stream (Kadtan, ¶70: video frame divided into portions, referred to as ROIs; ¶¶90-91: system may redefine or update ROIs to focus on regions with movement; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network, where ¶256 discloses vehicle implements selective video tracking from Fig. 4); and
Perform artificial intelligence (AI) inference on a first frame of the [resized] video stream to generate an AI inference result for a second frame of the main video stream, wherein the second frame of the main video stream is a frame corresponding to a time point when or after the AI inference result is generated (Kadtan, ¶65: tracking module 116 may identify the same object as having the same object ID from a current inference and an immediately preceding inference, and compare location parameters of the object, e.g., coordinates of the same object within a ROI, to determine whether the object has moved between the two inferences; ¶68: according to a dynamically adjusted inference rate, detection and classification module 114 may use a neural network to perform object identification on only a portion of a video frame at a given time, e.g., one or more ROI(s) that contain at least one moving objects in recent inferences, instead of using the neural network to perform object identification on the entire video frame at a fixed rate; ¶69: a visualized output may indicate which objects within a video frame have recently moved, which ROI(s) on the video frame contains moving objects during the latest inferences, and/or which ROI(s) on the video frame contains no moving objects during the latest inferences; Fig. 2B and ¶93: bounding boxes overlayed on video frame around identified objects; Also ¶106 discloses presenting a checkmark or cross sign on a particular region based on whether object identification was performed); ¶103: object detection may be performed using one or more neural networks on one or more ROIs; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network)
Although disclosing changing the size of images individually processed in the video prior to performing AI processing, Kadtan does not explicitly teach scaling the video stream prior to the AI processing. However, scaling a video prior to AI processing is known in the art to reduce latency issues.
Quist discloses:
Perform scaling on the main video stream to generate a scaled video stream and perform artificial intelligence (AI) inference on a first frame of the scaled video stream to generate an AI inference result for a second frame of the frame of the(Quist, ¶165: video processing module receives raw unprocessed video data 930, providing a second video data stream to video analysis apparatus 825 for machine analysis, where the second video data stream is downsampled to enable low latency processing through the video analysis apparatus; ¶¶166-167: arthroscopic video analysis apparatus 825 outputs overlay information 933 regarding the anatomical regions in the second video stream 932, where overlay information 933 can be associated with analytical information, determined using machine learning models, that highlights or emphasizes anatomical structures of interest, pathologies or defects, or particular relevant measurements for the surgery related to the anatomical structures in the arthroscopic field of view, and overlay information is video formatted and correlated by position to anatomical areas in the first video stream)
Both Kadtan and Quist are directed to AI inference systems on streaming videos for allowing real-time contextual information related to the video. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, by further using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system by allowing faster processing of video data by downsampling the video to prevent latency issues (see Quest, ¶165), while maintaining the high resolution video for viewing.
.
REGARDING CLAIM 18, the device of claim 1 performs the method of claim 18 and as such claim 18 is rejected based on the same rational as claim 1 set forth above.
REGARDING CLAIM 2, Kadtan modified by Quist further discloses:
Wherein the second processor performs image reduction on the main video stream to generate the scaled video stream. (Quist, ¶165: video processing module receives raw unprocessed video data 930, providing a second video data stream to video analysis apparatus 825 for machine analysis, where the second video data stream is downsampled to enable low latency processing through the video analysis apparatus; ¶¶166-167: arthroscopic video analysis apparatus 825 outputs overlay information 933 regarding the anatomical regions in the second video stream 932, where e overlay information 933 can be associated with analytical information, determined using machine learning models, that highlights or emphasizes anatomical structures of interest, pathologies or defects, or particular relevant measurements for the surgery related to the anatomical structures in the arthroscopic field of view, and overlay information is video formatted and correlated by position to anatomical areas in the first video stream)
Both Kadtan and Quist are directed to AI inference systems on streaming videos for allowing real-time contextual information related to the video. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, by further using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system by allowing faster processing of video data by downsampling the video to prevent latency issues (see Quest, ¶165), while maintaining the high resolution video for viewing.
REGARDING CLAIM 6, Kadtan further discloses:
Wherein the second processor performs image analysis on the first frame of the [resized] video stream to determine a designated object in the first frame of the [resized] video stream, and wherein the second processor generates the AI inference result with respect to the designated object in the second frame of the(Kadtan, ¶63 discloses that inference is not performed every frame, but at intervals of frames, such as every 4 frames, where Fig. 2B and ¶93: bounding boxes displayed over identified objects across frames – i.e. inference is provided from first of set of analyzed frames across all intervale of frames, such as every 4 frames, which indicates that second frame is a next frame; ¶¶65: tracking module 116 may identify the same object as having the same object ID from a current inference and an immediately preceding inference, and compare location parameters of the object, e.g., coordinates of the same object within a ROI, to determine whether the object has moved between the two inferences across different video frames; ¶68: according to a dynamically adjusted inference rate, detection and classification module 114 may use a neural network to perform object identification on only a portion of a video frame at a given time, e.g., one or more ROI(s) that contain at least one moving objects in recent inferences, instead of using the neural network to perform object identification on the entire video frame at a fixed rate; ¶103: object detection may be performed using one or more neural networks on one or more ROIs; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network)
Wherein the second processor performs image analysis on the first frame of the scaled video stream to determine a designated object in the first frame of the scaled video stream, and wherein the second processor generates the AI inference result with respect to the designated object in the main video stream (Quist, ¶165: video processing module receives raw unprocessed video data 930, providing a second video data stream to video analysis apparatus 825 for machine analysis, where the second video data stream is downsampled to enable low latency processing through the video analysis apparatus; ¶¶166-167: arthroscopic video analysis apparatus 825 outputs overlay information 933 regarding the anatomical regions in the second video stream 932, where overlay information 933 can be associated with analytical information, determined using machine learning models, that highlights or emphasizes anatomical structures of interest, pathologies or defects, or particular relevant measurements for the surgery related to the anatomical structures in the arthroscopic field of view, and overlay information is video formatted and correlated by position to anatomical areas in the first video stream; Also ¶175: video compositing overlaying information generated by video analysis apparatus on video stream)
Both Kadtan and Quist are directed to AI inference systems on streaming videos for allowing real-time contextual information related to the video. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, by further using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system by allowing faster processing of video data by downsampling the video for AI processing to prevent latency issues (see Quest, ¶165), while maintaining the high resolution video for viewing.
REGARDING CLAIM 7, Kadtan further discloses:
Generate texts or graphics according to the AI inference result and superimpose the texts or the graphics at a position in association with the designated object in the second frame of the (Kadtan, ¶63 discloses that inference is not performed every frame, but at intervals of frames, such as every 4 frames, where Fig. 2B and ¶93: FIG. 2B illustrates an example of identified objects within each user-defined region of interests within a video frame, according to at least one embodiment. In at least one embodiment, for example as shown in FIG. 2B, bounding boxes around identified objects such as 205a-c. 206a-c may be displayed, e.g., by visualization module 120 in FIG. 1, overlaying video frame 200)
Kadtan modified by Quist further discloses:
A third processor, configured to: generate texts or graphics according to the AI inference result and superimpose the texts or the graphics at a position in association with the designated object in the second frame of the main video stream (Quist, ¶51: features of objects tracked using fiducials; ¶156: one or more analytics displayed as overlays used to generate anatomical region-specific bitmask to apply processing specific to that anatomical region; ¶163 discloses plurality of processors executing programmed instructions; ¶166: arthroscopic video analysis apparatus 825 outputs overlay information determined using machine learning models, which is associated with analytical information that highlights or emphasizes anatomical structures of interest; Fig. 9 and ¶175: post-processing module 926 receives overlay information from arthroscopic video analysis apparats 825, where the overlay information 933 comprises binary bitmasks, the post-processing module 926 may generate the third video stream 934 as a high resolution endoscopic video stream where specific anatomical regions have been post-processed via the mask overlays in the overlay information 933)
Both Kadtan and Quist are directed to AI inference systems on streaming videos for allowing real-time contextual information related to the video. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, by further using the technique utilizing the multiple modules for downsampling the real-time video and using AI for generating overlay inferencing as provided by Quist, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system by allowing faster processing of video data by downsampling the video for AI processing to prevent latency issues (see Quest, ¶165), while maintaining the high resolution video for viewing by utilizing the split functions for allowing better management of processing by distribution among components for easier implantation and more efficient processing.
REGARDING CLAIM 8, Kadtan modified by Quist further discloses:
Wherein the third processor determines the position at which the texts or the graphics are superimposed in the second frame of the main video stream according to the first frame of the scaled video stream and an image reduction ratio of the first frame (Examiner notes that there is no clarification of the manner in which the position is determined, i.e. no meaningful limiting step that indicates how the system determines a position according to the scaled video stream and image reduction ratio other than merely stating that the position is determined according to the scaled video stream and image reduction ratio; Under the broadest reasonable interpretation, this merely means that the position determination is performed on a scaled video stream that has a reduction ratio, and not any other particular utilization of the ratio in calculation etc., as the Specification is also completely without description of any other particular use; Quist discloses an image downsampling which inherently includes a “reduction ratio”, as downsampling reduces the data from an original data amount, and performs the object location determination for use with the overlay then superimposed on the original frame – see Quist, ¶51: features of objects tracked using fiducials; ¶156: one or more analytics displayed as overlays used to generate anatomical region-specific bitmask to apply processing specific to that anatomical region; ¶163 discloses plurality of processors executing programmed instructions; ¶166: arthroscopic video analysis apparatus 825 outputs overlay information determined using machine learning models, which is associated with analytical information that highlights or emphasizes anatomical structures of interest; Fig. 9 and ¶175: post-processing module 926 receives overlay information from arthroscopic video analysis apparats 825, where the overlay information 933 comprises binary bitmasks, the post-processing module 926 may generate the third video stream 934 as a high resolution endoscopic video stream where specific anatomical regions have been post-processed via the mask overlays in the overlay information 933)
Both Kadtan and Quist are directed to AI inference systems on streaming videos for allowing real-time contextual information related to the video. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, by further using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system by allowing faster processing of video data by downsampling the video to prevent latency issues (see Quest, ¶165), while maintaining the high resolution video for viewing.
REGARDING CLAIM 11, Kadtan further discloses:
Wherein the second processor is further configured to: perform AI inference on the frame of the main video corresponding to a time point when or after the AI inference result is generated to generate a new AI inference result for a third frame of the main video stream, wherein the third frame of the main video stream is a frame corresponding to a time point when or after the new AI inference result is generated. (Kadtan, ¶63 discloses that inference performed at intervals of frames, such as every 4 frames, where Fig. 2B and ¶93: bounding boxes displayed over identified objects across frames – i.e. inference is provided from first of set of analyzed frames across all intervale of frames, such as every 4 frames, which indicates that second frame is a next frame; ¶¶65: tracking module 116 may identify the same object as having the same object ID from a current inference and an immediately preceding inference, and compare location parameters of the object, e.g., coordinates of the same object within a ROI, to determine whether the object has moved between the two inferences across different video frames; ¶68: according to a dynamically adjusted inference rate, detection and classification module 114 may use a neural network to perform object identification on only a portion of a video frame at a given time, e.g., one or more ROI(s) that contain at least one moving objects in recent inferences, instead of using the neural network to perform object identification on the entire video frame at a fixed rate; ¶103: object detection may be performed using one or more neural networks on one or more ROIs; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network)
REGARDING CLAIM 12, Kadtan discloses:
A computing device comprising: (Kadtan, Fig. 8A and ¶159: autonomous vehicle; ¶163: controllers, including system on chips and GPUs as components of vehicle 800, including onboard computing devices to process sensor signals; ¶168: vehicle 800 used to implement selective video tracking described in Fig. 4)
A first processor, configured to continuously receive a main video stream (Kadtan, ¶98 discloses performing dynamic adjustment based on object movement detected from live streaming video at a given time, with neural network detecting movement of identified objects; ¶101: one or more streams of videos received from communication interface 415; ¶164: controllers provide signals in response to received sensor data from cameras; ¶¶173-174 discloses cameras within vehicle 800 providing sensor data, used for object detection and classification and object tracking; ¶178: various cameras described in FIG. 8B may be used to capture one or more video streams on which embodiments of selective video tracking described in FIGS. 1-4 may be implemented; ¶210: processors 810 including real-time camera engine including dedicated processor subsystem for handling real-time camera management; also ¶¶214-215: system on chip (SoC) receiving video and input from cameras, freeing CPUs 806 from routine data management tasks, vs. ¶217: computer vision algorithms executed on CPUs; also see ¶222: CPUs 818 coupled to SoCs 804, and performing functions other than SoCs; Also note Kadtan discloses a plurality of computer architectures read on by the separation of two processors receiving video, including ¶¶250-251 disclosing multiple computers performing computer vision on images, including one with neural network, MCU, separate from a computer performing standard computer vision)
A second processor, configured to: (Kadtan, Fig. 4 and ¶111: processor 410 performing processes, including video tracking module 430 – see Fig. 4; CPUs 818 coupled to SoCs 804, and performing functions other than SoCs; Also note Kadtan discloses a plurality of computer architectures read on by the separation of two processors receiving video, including ¶¶250-251 disclosing multiple computers performing computer vision on images, including one with neural network, MCU, separate from a computer performing standard computer vision; Also ¶359 discloses operations of neural network inferencing performed on one or more GPGPUs; Fig. 17 and ¶369-372 discloses computer system having multiple processors separate from I/O hub driving display controller, and performing selective video tracking of Fig. 4)
Perform [resizing] on the main video stream to generate a[resized] video stream; (Kadtan, ¶70: video frame divided into portions, referred to as ROIs; ¶¶90-91: system may redefine or update ROIs to focus on regions with movement; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network, where ¶256 discloses vehicle implements selective video tracking from Fig. 4)
Perform artificial intelligence (AI) inference on a first frame of the [resized] video stream to generate an AI inference result (Kadtan, ¶65: tracking module 116 may identify the same object as having the same object ID from a current inference and an immediately preceding inference, and compare location parameters of the object, e.g., coordinates of the same object within a ROI, to determine whether the object has moved between the two inferences; ¶68: according to a dynamically adjusted inference rate, detection and classification module 114 may use a neural network to perform object identification on only a portion of a video frame at a given time, e.g., one or more ROI(s) that contain at least one moving objects in recent inferences, instead of using the neural network to perform object identification on the entire video frame at a fixed rate; ¶103: object detection may be performed using one or more neural networks on one or more ROIs; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network)); and
An on-screen display controller (Kadtan, ¶366: I/O hub 1707 can enable a display controller, which may be included in one or more processor(s) 1702, to provide outputs to one or more display device(s) 1710A; Also ¶454: display controller 2511 to drive graphics processor output to one or more coupled displays, either as separate module coupled to graphics processor or integrated within graphics processor; Also ¶366: I/O hub 1707 can enable a display controller, which may be included in one or more processor(s) 1702, to provide outputs to one or more display device(s) 1710A – see Fig. 17), configured to:
Superimpose texts or graphs on a second frame of the (Kadtan, ¶69: a visualized output may indicate which objects within a video frame have recently moved, which ROI(s) on the video frame contains moving objects during the latest inferences, and/or which ROI(s) on the video frame contains no moving objects during the latest inferences; Fig. 2B and ¶93: bounding boxes overlayed on video frame around identified objects; Also ¶106 discloses presenting a checkmark or cross sign on a particular region based on whether object identification was performed)
Although disclosing changing the size of images individually processed in the video prior to performing AI processing, Kadtan does not explicitly teach scaling the video stream prior to the AI processing. However, scaling a video prior to AI processing is known in the art to reduce latency issues.
Quist discloses:
Perform scaling on the main video stream to generate a scaled video stream and perform artificial intelligence (AI) inference on a first frame of the scaled video stream to generate an AI inference result for a second frame of the main video stream, wherein the second frame of the main video stream is a frame corresponding to a time point when or after the AI inference result is generated (Quist, ¶165: video processing module receives raw unprocessed video data 930, providing a second video data stream to video analysis apparatus 825 for machine analysis, where the second video data stream is downsampled to enable low latency processing through the video analysis apparatus; ¶¶166-167: arthroscopic video analysis apparatus 825 outputs overlay information 933 regarding the anatomical regions in the second video stream 932, where overlay information 933 can be associated with analytical information, determined using machine learning models, that highlights or emphasizes anatomical structures of interest, pathologies or defects, or particular relevant measurements for the surgery related to the anatomical structures in the arthroscopic field of view, and overlay information is video formatted and correlated by position to anatomical areas in the first video stream)
Both Kadtan and Quist are directed to AI inference systems on streaming videos for allowing real-time contextual information related to the video. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, by further using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system by allowing faster processing of video data by downsampling the video to prevent latency issues (see Quest, ¶165), while maintaining the high resolution video for viewing.
REGARDING CLAIM 19, the device of claim 12 performs the method of claim 19 and as such claim 19 is rejected based on the same rational as claim 2 set forth above.
REGARDING CLAIM 13, Kadtan further discloses:
Wherein the second processor performs image analysis on the first frame of the [resized] video stream to determine a designated object in the first frame of the [resized] video stream, and wherein the second processor generates the AI inference result with respect to the designated object in the second frame of the(Kadtan, ¶63 discloses that inference is not performed every frame, but at intervals of frames, such as every 4 frames, where Fig. 2B and ¶93: bounding boxes displayed over identified objects across frames – i.e. inference is provided from first of set of analyzed frames across all intervale of frames, such as every 4 frames, which indicates that second frame is a next frame; ¶¶65: tracking module 116 may identify the same object as having the same object ID from a current inference and an immediately preceding inference, and compare location parameters of the object, e.g., coordinates of the same object within a ROI, to determine whether the object has moved between the two inferences across different video frames; ¶68: according to a dynamically adjusted inference rate, detection and classification module 114 may use a neural network to perform object identification on only a portion of a video frame at a given time, e.g., one or more ROI(s) that contain at least one moving objects in recent inferences, instead of using the neural network to perform object identification on the entire video frame at a fixed rate; ¶103: object detection may be performed using one or more neural networks on one or more ROIs; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network)
Wherein the second processor performs image analysis on the first frame of the scaled video stream to determine a designated object in the first frame of the scaled video stream, and wherein the second processor generates the AI inference result with respect to the designated object in the main video stream (Quist, ¶165: video processing module receives raw unprocessed video data 930, providing a second video data stream to video analysis apparatus 825 for machine analysis, where the second video data stream is downsampled to enable low latency processing through the video analysis apparatus; ¶¶166-167: arthroscopic video analysis apparatus 825 outputs overlay information 933 regarding the anatomical regions in the second video stream 932, where overlay information 933 can be associated with analytical information, determined using machine learning models, that highlights or emphasizes anatomical structures of interest, pathologies or defects, or particular relevant measurements for the surgery related to the anatomical structures in the arthroscopic field of view, and overlay information is video formatted and correlated by position to anatomical areas in the first video stream; Also ¶175: video compositing overlaying information generated by video analysis apparatus on video stream)
Both Kadtan and Quist are directed to AI inference systems on streaming videos for allowing real-time contextual information related to the video. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, by further using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system by allowing faster processing of video data by downsampling the video for AI processing to prevent latency issues (see Quest, ¶165), while maintaining the high resolution video for viewing.
REGARDING CLAIM 14, Kadtan further discloses:
Wherein the second processor is further configured to: perform AI inference on the frame of the main video corresponding to a time point when or after the AI inference result is generated to generate a new AI inference result for a third frame of the main video stream, wherein the third frame of the main video stream is a frame corresponding to a time point when or after the new AI inference result is generated. (Kadtan, ¶63 discloses that inference performed at intervals of frames, such as every 4 frames, where Fig. 2B and ¶93: bounding boxes displayed over identified objects across frames – i.e. inference is provided from first of set of analyzed frames across all intervale of frames, such as every 4 frames, which indicates that second frame is a next frame; ¶¶65: tracking module 116 may identify the same object as having the same object ID from a current inference and an immediately preceding inference, and compare location parameters of the object, e.g., coordinates of the same object within a ROI, to determine whether the object has moved between the two inferences across different video frames; ¶68: according to a dynamically adjusted inference rate, detection and classification module 114 may use a neural network to perform object identification on only a portion of a video frame at a given time, e.g., one or more ROI(s) that contain at least one moving objects in recent inferences, instead of using the neural network to perform object identification on the entire video frame at a fixed rate; ¶103: object detection may be performed using one or more neural networks on one or more ROIs; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network)
REGARDING CLAIM 15, Kadtan discloses:
An edge device comprising (Kadtan, Fig. 8A and ¶159: autonomous vehicle)
a computing device comprising: (Kadtan, Fig. 8A and ¶159: autonomous vehicle; ¶163: controllers, including system on chips and GPUs as components of vehicle 800, including onboard computing devices to process sensor signals; ¶168: vehicle 800 used to implement selective video tracking described in Fig. 4)
A first processor, configured to continuously receive a main video stream (Kadtan, ¶98 discloses performing dynamic adjustment based on object movement detected from live streaming video at a given time, with neural network detecting movement of identified objects; ¶101: one or more streams of videos received from communication interface 415; ¶164: controllers provide signals in response to received sensor data from cameras; ¶¶173-174 discloses cameras within vehicle 800 providing sensor data, used for object detection and classification and object tracking; ¶178: various cameras described in FIG. 8B may be used to capture one or more video streams on which embodiments of selective video tracking described in FIGS. 1-4 may be implemented; ¶210: processors 810 including real-time camera engine including dedicated processor subsystem for handling real-time camera management; also ¶¶214-215: system on chip (SoC) receiving video and input from cameras, freeing CPUs 806 from routine data management tasks, vs. ¶217: computer vision algorithms executed on CPUs; also see ¶222: CPUs 818 coupled to SoCs 804, and performing functions other than SoCs; Also note Kadtan discloses a plurality of computer architectures read on by the separation of two processors receiving video, including ¶¶250-251 disclosing multiple computers performing computer vision on images, including one with neural network, MCU, separate from a computer performing standard computer vision)
A second processor, configured to: (Kadtan, Fig. 4 and ¶111: processor 410 performing processes, including video tracking module 430 – see Fig. 4; CPUs 818 coupled to SoCs 804, and performing functions other than SoCs; Also note Kadtan discloses a plurality of computer architectures read on by the separation of two processors receiving video, including ¶¶250-251 disclosing multiple computers performing computer vision on images, including one with neural network, MCU, separate from a computer performing standard computer vision; Also ¶359 discloses operations of neural network inferencing performed on one or more GPGPUs; Fig. 17 and ¶369-372 discloses computer system having multiple processors separate from I/O hub driving display controller, and performing selective video tracking of Fig. 4)
Perform [resizing] on the main video stream to generate a[resized] video stream; (Kadtan, ¶70: video frame divided into portions, referred to as ROIs; ¶¶90-91: system may redefine or update ROIs to focus on regions with movement; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network, where ¶256 discloses vehicle implements selective video tracking from Fig. 4)
Perform artificial intelligence (AI) inference on a first frame of the [resized] video stream to generate an AI inference result (Kadtan, ¶65: tracking module 116 may identify the same object as having the same object ID from a current inference and an immediately preceding inference, and compare location parameters of the object, e.g., coordinates of the same object within a ROI, to determine whether the object has moved between the two inferences; ¶68: according to a dynamically adjusted inference rate, detection and classification module 114 may use a neural network to perform object identification on only a portion of a video frame at a given time, e.g., one or more ROI(s) that contain at least one moving objects in recent inferences, instead of using the neural network to perform object identification on the entire video frame at a fixed rate; ¶103: object detection may be performed using one or more neural networks on one or more ROIs; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network); and
An on-screen display controller (Kadtan, ¶366: I/O hub 1707 can enable a display controller, which may be included in one or more processor(s) 1702, to provide outputs to one or more display device(s) 1710A; Also ¶454: display controller 2511 to drive graphics processor output to one or more coupled displays, either as separate module coupled to graphics processor or integrated within graphics processor; Also ¶366: I/O hub 1707 can enable a display controller, which may be included in one or more processor(s) 1702, to provide outputs to one or more display device(s) 1710A – see Fig. 17), configured to:
Superimpose texts or graphs on a second frame of the (Kadtan, ¶69: a visualized output may indicate which objects within a video frame have recently moved, which ROI(s) on the video frame contains moving objects during the latest inferences, and/or which ROI(s) on the video frame contains no moving objects during the latest inferences; Fig. 2B and ¶93: bounding boxes overlayed on video frame around identified objects; Also ¶106 discloses presenting a checkmark or cross sign on a particular region based on whether object identification was performed)
a screen monitory, configured to: display the processed second frame (Kadtan, ¶112: video tracking module 430 may receive an input 440 such as an incoming video stream (e.g., video 102 in FIG. 1) via a data interface 415 and generate an output 450 which may be any of visualized bounding boxes overlaying a video frame; Fig. 8A, HMI display 834 and ¶165 and ¶253; Also ¶461: overlay planes for display device and composition of multiple layers of video)
Although disclosing changing the size of images individually processed in the video prior to performing AI processing, Kadtan does not explicitly teach scaling the video stream prior to the AI processing. However, scaling a video prior to AI processing is known in the art to reduce latency issues.
Quist discloses:
Perform scaling on the main video stream to generate a scaled video stream and perform artificial intelligence (AI) inference on a first frame of the scaled video stream to generate an AI inference result for a second frame of the main video stream, wherein the second frame of the main video stream is a frame corresponding to a time point when or after the AI inference result is generated (Quist, ¶165: video processing module receives raw unprocessed video data 930, providing a second video data stream to video analysis apparatus 825 for machine analysis, where the second video data stream is downsampled to enable low latency processing through the video analysis apparatus; ¶¶166-167: arthroscopic video analysis apparatus 825 outputs overlay information 933 regarding the anatomical regions in the second video stream 932, where overlay information 933 can be associated with analytical information, determined using machine learning models, that highlights or emphasizes anatomical structures of interest, pathologies or defects, or particular relevant measurements for the surgery related to the anatomical structures in the arthroscopic field of view, and overlay information is video formatted and correlated by position to anatomical areas in the first video stream)
Both Kadtan and Quist are directed to AI inference systems on streaming videos for allowing real-time contextual information related to the video. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, by further using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system by allowing faster processing of video data by downsampling the video to prevent latency issues (see Quest, ¶165), while maintaining the high resolution video for viewing.
REGARDING CLAIM 20, the device of claim 15 performs the method of claim 20 and as such claim 20 is rejected based on the same rational as claim 15 set forth above.
REGARDING CLAIM 16, Kadtan further discloses:
Wherein the second processor performs image analysis on the first frame of the [resized] video stream to determine a designated object in the first frame of the [resized] video stream, and wherein the second processor generates the AI inference result with respect to the designated object in the second frame of the(Kadtan, ¶63 discloses that inference is not performed every frame, but at intervals of frames, such as every 4 frames, where Fig. 2B and ¶93: bounding boxes displayed over identified objects across frames – i.e. inference is provided from first of set of analyzed frames across all intervale of frames, such as every 4 frames, which indicates that second frame is a next frame; ¶¶65: tracking module 116 may identify the same object as having the same object ID from a current inference and an immediately preceding inference, and compare location parameters of the object, e.g., coordinates of the same object within a ROI, to determine whether the object has moved between the two inferences across different video frames; ¶68: according to a dynamically adjusted inference rate, detection and classification module 114 may use a neural network to perform object identification on only a portion of a video frame at a given time, e.g., one or more ROI(s) that contain at least one moving objects in recent inferences, instead of using the neural network to perform object identification on the entire video frame at a fixed rate; ¶103: object detection may be performed using one or more neural networks on one or more ROIs; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network)
Wherein the second processor performs image analysis on the first frame of the scaled video stream to determine a designated object in the first frame of the scaled video stream, and wherein the second processor generates the AI inference result with respect to the designated object in the main video stream (Quist, ¶165: video processing module receives raw unprocessed video data 930, providing a second video data stream to video analysis apparatus 825 for machine analysis, where the second video data stream is downsampled to enable low latency processing through the video analysis apparatus; ¶¶166-167: arthroscopic video analysis apparatus 825 outputs overlay information 933 regarding the anatomical regions in the second video stream 932, where overlay information 933 can be associated with analytical information, determined using machine learning models, that highlights or emphasizes anatomical structures of interest, pathologies or defects, or particular relevant measurements for the surgery related to the anatomical structures in the arthroscopic field of view, and overlay information is video formatted and correlated by position to anatomical areas in the first video stream; Also ¶175: video compositing overlaying information generated by video analysis apparatus on video stream)
Both Kadtan and Quist are directed to AI inference systems on streaming videos for allowing real-time contextual information related to the video. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, by further using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system by allowing faster processing of video data by downsampling the video for AI processing to prevent latency issues (see Quest, ¶165), while maintaining the high resolution video for viewing.
REGARDING CLAIM 17, Kadtan further discloses:
Wherein the second processor is further configured to: perform AI inference on the frame of the main video corresponding to a time point when or after the AI inference result is generated to generate a new AI inference result for a third frame of the main video stream, wherein the third frame of the main video stream is a frame corresponding to a time point when or after the new AI inference result is generated. (Kadtan, ¶63 discloses that inference performed at intervals of frames, such as every 4 frames, where Fig. 2B and ¶93: bounding boxes displayed over identified objects across frames – i.e. inference is provided from first of set of analyzed frames across all intervale of frames, such as every 4 frames, which indicates that second frame is a next frame; ¶¶65: tracking module 116 may identify the same object as having the same object ID from a current inference and an immediately preceding inference, and compare location parameters of the object, e.g., coordinates of the same object within a ROI, to determine whether the object has moved between the two inferences across different video frames; ¶68: according to a dynamically adjusted inference rate, detection and classification module 114 may use a neural network to perform object identification on only a portion of a video frame at a given time, e.g., one or more ROI(s) that contain at least one moving objects in recent inferences, instead of using the neural network to perform object identification on the entire video frame at a fixed rate; ¶103: object detection may be performed using one or more neural networks on one or more ROIs; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network)
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Kadtan et al. (US 2025/0069409 A1) in view of
Quist et al. (US 2022/0406061 A1) and in further view of
Kaskela et al. (US 2023/0196662 A1)
REGARDING CLAIM 3, the limitations included from claim 1 are rejected based on the same rationale as claim 1 set forth above. Further regarding claim 3, Kadtan further discloses:
Wherein the processor performs image [focus] on a predetermined region of the main video stream to generate the[resized] video stream (Kadtan, ¶70: video frame divided into portions, referred to as ROIs; ¶¶90-91: system may redefine or update ROIs to focus on regions with movement; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network, where ¶256 discloses vehicle implements selective video tracking from Fig. 4)
It is furthermore pointed out that Quist is relied upon for the separation of the main video from the scaled video for processing as claimed (Quist, ¶165: video processing module receives raw unprocessed video data 930, providing a second video data stream to video analysis apparatus 825 for machine analysis, where the second video data stream is downsampled to enable low latency processing through the video analysis apparatus; ¶¶166-167: arthroscopic video analysis apparatus 825 outputs overlay information 933 regarding the anatomical regions in the second video stream 932, where overlay information 933 can be associated with analytical information, determined using machine learning models, that highlights or emphasizes anatomical structures of interest, pathologies or defects, or particular relevant measurements for the surgery related to the anatomical structures in the arthroscopic field of view, and overlay information is video formatted and correlated by position to anatomical areas in the first video stream)
Both Kadtan and Quist are directed to AI inference systems on streaming videos for allowing real-time contextual information related to the video. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, by further using the technique for separating the video processing to a different analysis module/apparatus by using a second video stream as provided by Quist, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system by allowing faster processing of video data by offsetting the neural network processing to a separate dedicated process allowing for better utilization of resources and parallel computing (see Quiest ¶55).
The only element not taught by Kadtan and Quist is the magnification or upscaling (see ¶24 of applicant’s Specification filed 12/8/2024) of the predetermined region of video.
Kaskela discloses:
Wherein the processor performs image magnification on a predetermined region of the main video stream to generate the scaled video stream (Kaskela, Fig. 1 and ¶46: low-resolution rendered image 106 can be processed using an upscaler 108 to generate an upscaled image 110 that represents content of low resolution rendered image 106 at a resolution that equals (or is at least closer to) a target output resolution; ¶49: this upscaled image 110 can be provided as input to a neural network 112 to determine one or more blending factors or blending weights)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, using the technique for separating the machine learning to a separate video stream for machine learning processing Quist, by utilizing the magnification of a region for AI processing as provided by Kaskela, using known electronic interfacing and programming techniques. The modification results in an improved AI video processing by ensuring only a particular region of data is utilized to reduce unnecessary processing of video for more efficient utilization of resources, while also ensuring a focused region is of a better resolution for improved image results from any subsequent AI inference.
Claim(s) 4-5 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Kadtan et al. (US 2025/0069409 A1) in view of
Quist et al. (US 2022/0406061 A1) and in further view of
Woo (US 2025/0139408 A1)
REGARDING CLAIM 4, the limitations included from claim 1 re rejected based on the same rationale as claim 1 set forth above. Further regarding claim 4, Kadtan modified by Quist further discloses:
Wherein the second frame of the main video stream is a(Kadtan, ¶65: tracking module 116 may identify the same object as having the same object ID from a current inference and an immediately preceding inference, and compare location parameters of the object, e.g., coordinates of the same object within a ROI, to determine whether the object has moved between the two inferences; ¶68: according to a dynamically adjusted inference rate, detection and classification module 114 may use a neural network to perform object identification on only a portion of a video frame at a given time, e.g., one or more ROI(s) that contain at least one moving objects in recent inferences, instead of using the neural network to perform object identification on the entire video frame at a fixed rate; ¶103: object detection may be performed using one or more neural networks on one or more ROIs; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network)
Quist further discloses the overlay onto the main video after AI inference is generated, disclosing:
Wherein the second frame of the main video stream is a(Quist, ¶165: video processing module receives raw unprocessed video data 930, providing a second video data stream to video analysis apparatus 825 for machine analysis, where the second video data stream is downsampled to enable low latency processing through the video analysis apparatus; ¶¶166-167: arthroscopic video analysis apparatus 825 outputs overlay information 933 regarding the anatomical regions in the second video stream 932, where overlay information 933 can be associated with analytical information, determined using machine learning models, that highlights or emphasizes anatomical structures of interest, pathologies or defects, or particular relevant measurements for the surgery related to the anatomical structures in the arthroscopic field of view, and overlay information is video formatted and correlated by position to anatomical areas in the first video stream; Also ¶175: video compositing overlaying information generated by video analysis apparatus on video stream)
Both Kadtan and Quist are directed to AI inference systems on streaming videos for allowing real-time contextual information related to the video. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, by further using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system by allowing faster processing of video data by downsampling the video for AI processing to prevent latency issues (see Quest, ¶165), while maintaining the high resolution video for viewing.
The only aspect not explicitly taught by Kadtan modified by Quist is that the system ensures the timing of the video overlay based on a frame-by-frame timing for presenting the data. It was known, however, by one of ordinary skill in the art at the time of filing, that AI inference processing can be timed based on frame-to-frame data.
Woo discloses:
Wherein the second frame of the main video stream is a current frame of the main video stream at the time point when the AI inference result is generated (Woo, ¶23: system determines a size of recurring time window based on rate or frequency of receiving each frame of input, e.g. frame per second, where each occurrence of the recurring time window is a processing cycle in which a frame is processed using machine learning model; ¶26 discloses technique to ensure that input is processed by machine learning within an identified time window – see Fig. 2; ¶28: processing input such as video; ¶40: Each hardware processing unit 110 is configured to perform machine learning computations including inference operations of the assigned sub-models or models for processing each frame of the input data 137b and generate output data 167b after processing the frame of input data 137b using the model(s)/sub-model(s); Also ¶51: The time window can be determined by the system 100 based on the rate of receiving each frame of input data, e.g., frames of input per second; ¶¶53-54 and 82 discusses inference window for video frames; also ¶103 discloses using multiple time windows)
Kadtan, Quist and Woo are directed to AI inference systems on streaming videos for allowing real-time processing It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, by further using the technique for scheduling AI processing based on video frame windows as provided by Woo, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system processing by addressing time window limits avoids performance issues, lower performance or errors (see Woo, ¶24)
REGARDING CLAIM 5, the limitations included from claim 1 are rejected based on the same rationale as claim 1 set forth above. Further regarding claim 4, Kadtan modified by Quist further discloses:
Wherein the second frame of the main video is an (Kadtan, ¶63 discloses that inference is not performed every frame, but at intervals of frames, such as every 4 frames, where Fig. 2B and ¶93: bounding boxes displayed over identified objects across frames – i.e. inference is provided from first of set of analyzed frames across all intervale of frames, such as every 4 frames, which indicates that second frame is a next frame; ¶¶65: tracking module 116 may identify the same object as having the same object ID from a current inference and an immediately preceding inference, and compare location parameters of the object, e.g., coordinates of the same object within a ROI, to determine whether the object has moved between the two inferences across different video frames; ¶68: according to a dynamically adjusted inference rate, detection and classification module 114 may use a neural network to perform object identification on only a portion of a video frame at a given time, e.g., one or more ROI(s) that contain at least one moving objects in recent inferences, instead of using the neural network to perform object identification on the entire video frame at a fixed rate; ¶103: object detection may be performed using one or more neural networks on one or more ROIs; ¶114: video tracking module 430 includes ROI creation submodule 431, object detection submodule 432, and tracking movement submodule 433, where 432 and 433 may be implemented via artificial neural network )
Quist further discloses the overlay onto the main video after AI inference is generated, disclosing:
Wherein the second frame of the main video is an (Quist, ¶165: video processing module receives raw unprocessed video data 930, providing a second video data stream to video analysis apparatus 825 for machine analysis, where the second video data stream is downsampled to enable low latency processing through the video analysis apparatus; ¶¶166-167: arthroscopic video analysis apparatus 825 outputs overlay information 933 regarding the anatomical regions in the second video stream 932, where overlay information 933 can be associated with analytical information, determined using machine learning models, that highlights or emphasizes anatomical structures of interest, pathologies or defects, or particular relevant measurements for the surgery related to the anatomical structures in the arthroscopic field of view, and overlay information is video formatted and correlated by position to anatomical areas in the first video stream; Also ¶175: video compositing overlaying information generated by video analysis apparatus on video stream)
Both Kadtan and Quist are directed to AI inference systems on streaming videos for allowing real-time contextual information related to the video. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, by further using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system by allowing faster processing of video data by downsampling the video for AI processing to prevent latency issues (see Quest, ¶165), while maintaining the high resolution video for viewing.
The only aspect not explicitly taught by Kadtan modified by Quist is that the system ensures the timing of the video overlay based on a frame-by-frame timing for presenting the data. It was known, however, by one of ordinary skill in the art at the time of filing, that AI inference processing can be timed based on frame-to-frame data.
Woo discloses:
Wherein the second frame of the main video is an immediate next frame of the main video stream after the time point when the AI inference result is generated (Woo, ¶23: system determines a size of recurring time window based on rate or frequency of receiving each frame of input, e.g. frame per second, where each occurrence of the recurring time window is a processing cycle in which a frame is processed using machine learning model; ¶26 discloses technique to ensure that input is processed by machine learning within an identified time window – see Fig. 2; ¶28: processing input such as video; ¶40: Each hardware processing unit 110 is configured to perform machine learning computations including inference operations of the assigned sub-models or models for processing each frame of the input data 137b and generate output data 167b after processing the frame of input data 137b using the model(s)/sub-model(s); Also ¶51: The time window can be determined by the system 100 based on the rate of receiving each frame of input data, e.g., frames of input per second; ¶¶53-54 and 82 discusses inference window for video frames; also ¶103 discloses using multiple time windows)
Examiner notes that combination of references, where Kadtan modified by Quist teach the display of overlay across frames, including at least a next frame from a previous analyzed frame, along with the timing of AI analysis within a single time frame of Woo, teaches the limitations of the claim. Kadtan, Quist and Woo are directed to AI inference systems on streaming videos for allowing real-time processing It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, by further using the technique for scheduling AI processing based on video frame windows as provided by Woo, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system processing by addressing time window limits avoids performance issues, lower performance or errors (see Woo, ¶24)
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Kadtan et al. (US 2025/0069409 A1) in view of
Quist et al. (US 2022/0406061 A1) and in further view of
Das et al. (US 2026/0120572 A1).
REGARDING CLAIM 9, the limitations of claim 6 are rejected based on the same rationale as claim 6 set forth above. Further regarding claim 9, Kadtan modified by Quist further disclose:
A fourth processor, configured to: generate a voice signal (Kadtan, ¶208: processor(s) 810 may further include a set of embedded processors that may serve as an audio processing engine which may be an audio subsystem that enables full hardware support for multi-channel audio over multiple interfaces, and a broad and flexible range of audio I/O interfaces. In at least one embodiment, an audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM)
Kadtan modified by Quist does not explicitly disclose the generation of a voice signal in association with the designated object in the second frame of the main video stream according to the AI inference result. Generating a voice signal in association with a designated object as claimed, however, was known at the time of filing.
Das discloses:
generate a voice signal in association with the designated object in the second frame of the main video stream according to the AI inference result (Das, Abstract: processing real-time data to detect and track vehicle and road user movements, predicting movements, and analyzing risk assessment to determine conditions, and upon identifying a potential crash or unsafe condition, a warning message or alert of an impending crash or unsafe condition is generated and transmitted to the drivers and/or vulnerable road users so as to prevent a collision; ¶¶43-44: computer 101 including processing engine configured to process captured real-time data on vehicle movements and road users, where real-time data is captured using cameras/video systems; ¶48: real-time video; /video systems; ¶50: processing engine 201 processes the captured real-time data using computer vision and machine learning algorithms; ¶¶51-52: prediction engines detecting and tracking vehicles and predicting trajectories; ¶62: warning engine generates warning message or alert of impending potential crash or unsafe condition in response to identifying potential crash or unsafe condition, and transfers to driver in form of audio message; ¶77-78 discusses Neural network video analysis including drawing bounding boxes and processing to obtain object tracking; ¶139: warning message or alert may be “potential collision with turning vehicle”; ¶141: such a warning/alert is an audio message, such as a voice alert or a distinct sound played inside a vehicle or on a mobile device)
Both Kadtan and Das are directed to AI inference systems for real-time video processing in vehicles. Kadtan, Quist and Das are directed to real-time video processing using machine learning for generating contextual information to a user. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, by further utilizing the audio alerts provided by Das, using known electronic interfacing and programming techniques. The modification results in an improved user interface for presenting contextual information related to video analysis by including both video and audio indicators of machine learning results, allowing for more in-depth understanding of results, better accommodating different users and their preferences, and also providing more attention-grabbing indicators to better attract user attention for improved usability.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Kadtan et al. (US 2025/0069409 A1) in view of
Quist et al. (US 2022/0406061 A1) and in further view of
Chung et al. (US 2025/0349269 A1).
REGARDING CLAIM 10, the limitations of claim 6 are rejected based on the same rationale as claim 6 set forth above. Further regarding claim 10, Kadtan modified by Quist further disclose:
A fifth processor, configured to: a display of the second frame of the main video stream according to the AI inference result (Kadtan, Fig. 2B and ¶93: bounding boxes displayed over identified objects across frames; ¶366: I/O hub 1707 can enable a display controller, which may be included in one or more processor(s) 1702, to provide outputs to one or more display device(s) 1710A; Also ¶454: display controller 2511 to drive graphics processor output to one or more coupled displays, either as separate module coupled to graphics processor or integrated within graphics processor; Also ¶366: I/O hub 1707 can enable a display controller, which may be included in one or more processor(s) 1702, to provide outputs to one or more display device(s) 1710A – see Fig. 17)
Also Quist discloses:
a display of the second frame of the main video stream according to the AI inference result (Quist, ¶165: video processing module receives raw unprocessed video data 930, providing a second video data stream to video analysis apparatus 825 for machine analysis, where the second video data stream is downsampled to enable low latency processing through the video analysis apparatus; ¶¶166-167: arthroscopic video analysis apparatus 825 outputs overlay information 933 regarding the anatomical regions in the second video stream 932, where overlay information 933 can be associated with analytical information, determined using machine learning models, that highlights or emphasizes anatomical structures of interest, pathologies or defects, or particular relevant measurements for the surgery related to the anatomical structures in the arthroscopic field of view, and overlay information is video formatted and correlated by position to anatomical areas in the first video stream)
Both Kadtan and Quist are directed to AI inference systems on streaming videos for allowing real-time contextual information related to the video. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, by further using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, using known electronic interfacing and programming techniques. The modification results in an improved real-time video inference system by allowing faster processing of video data by downsampling the video to prevent latency issues (see Quest, ¶165), while maintaining the high resolution video for viewing.
The only limitation not explicitly taught by Kadtan and Quist is generating a control signal in association with a display according to the AI inference result
Chung discloses:
A fifth processor, configured to: generate a control signal in association with a display according to the AI inference result (Chung, ¶6: a method of controlling backlights for a local dimming display; ¶¶57-58: dimming display system including computer and ambient luminance sensor, and local dimming display having dimming regions; ¶60: ambient luminance sensor determines ambient luminance data of local environment, such as in vehicle passenger cabin, and sends captured ambient luminance data to computer for processing; ¶61: computer includes processor and trained backlight extraction model 26; ¶68 discloses training framework including deep neural network as model for backlight extraction; ¶74: a backlight extraction output 218 is generated, and includes or at least indicates backlight extraction values to be used for controlling the local dimming regions of the local dimming display)
Both Kadtan and Chung are directed to AI inference systems to control the display of images within a vehicle. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and method for performing real-time machine learning analysis on a video stream as provided by Kadtan, using the technique for downsampling the real-time video prior to using AI for generating overlay inferencing as provided by Quist, by further utilizing the dimming control for presentation of an image on a display as provided by Chung, using known electronic interfacing and programming techniques. The modification results in an improved display of images to a user by ensuring proper lighting control better match the presented data to a user based on changing environment conditions (e.g. within a vehicle).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM A BEUTEL whose telephone number is (571)272-3132. The examiner can normally be reached Monday-Friday 9:00 AM - 5:00 PM (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DANIEL HAJNIK can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM A BEUTEL/Primary Examiner, Art Unit 2616
Prosecution Insights