DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5, 7-8, 12-13, 16 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Maharana et al (U.S. Patent Application Publication 2024/0114180 A1) in view of Träff et al (U.S. Patent Application Publication 2016/0353052 A1) in view of SOMMER et al (U.S. Patent Application Publication 2023/0419674 A1).
Regarding claim 1, Maharana discloses a system for reducing latency and bandwidth usage comprising:
a reality device comprising a camera to operably capture a set of FIGS.11A to 11C show vehicle 1100; paragraphs [0222]-[0223], FIG. 11B illustrates an example of camera locations and fields of view for autonomous vehicle 1100 of FIG. 11A, ... camera types for cameras may include, but are not limited to, digital cameras that may be adapted for use with components and/or systems of vehicle 1100. ... In at least one embodiment, camera types may be capable of any image capture rate, such as 60 frames per second (fps), 1220 fps, 240 fps, etc.; paragraph [0291], cameras may be used to capture image data around an entire periphery of vehicle 1100); and
one or more processors (Paragraph [0217], controller(s) 1136 may include one or more onboard (e.g., integrated) computing devices that process sensor signals ...; paragraph [0310], FIG. 11D is a diagram of a system 1176 for communication between cloud-based server(s) and autonomous vehicle 1100 of FIG. 11A; paragraph [0058], FIG. 1 is a block diagram illustrating part of an architecture 100 for video streaming between a video sender 102 and a video receiver 116; paragraph [0069], a video sender 102 may be a computing system or any other computing device comprising one or more video input 104 devices and/or one or more sender neural networks 108. In at least one embodiment, a video sender 102 generates or captures video data 128 using one or more video input devices 104 in conjunction with a processor 106 ...) operable to:
select one or more frames of the set of Paragraph [0076], a video sender 102 can select a reference frame for inclusion in a set of frames having all reference frames that is cached for transmission to a video receiver 116) based on network quality metrics (Paragraph [0059], ... require continuous use of large bandwidths to provide sufficient information with each sent reference frame) and a pending frame queue (Paragraphs [0075]-[0076], a reference frame 110, such as a video frame, is a first image in a sequence of images and can be a first frame in a sequence of video frames ... a video sender 102 can select a reference frame for inclusion in a set of frames having all reference frames that is cached ... Thus, a sequence of images of reference frame 1, reference frame 2 and etc. within the processor 106 can be interpreted as a pending frame queue); and
transmit the selected one or more frames to an Paragraph [0076], a video sender 102 can select a reference frame for inclusion in a set of frames having all reference frames that is cached for transmission to a video receiver 116) for performing a task on behalf of the vehicle (Paragraph [0094], video receiver 116 may include a receiver neural network 124 having data values and software instructions that, when executed by a receiver processor 126, reconstruct or otherwise infer one or more reference frames 134 to be used with one or more prediction frames to cause video to be displayed by one or more video output devices 118).
However, Maharana does not specifically disclose capture a set of consequent frames;
select one or more frames of the set of consequent frames based on user focus areas of a user; and
transmit the selected one or more frames to an edge server.
In additional, Träff discloses (Paragraph [0030], in one embodiment, the starting scene condition requires movement occurring in the scene shot by the camera during capturing the frames of the preliminary frame sequence. Such movement can be, for example, rotational and/or translational movement or any other type of movement of an object lying within or forming a part of the scene. A moving object within the scene may be, for example, a human being, an animal, a vehicle, a plant, or any other object affecting the general nature of the captured moment ...) capture a set of consequent frames (FIG. 4; paragraphs [0057]-[0058], starts by sequentially capturing, in step 401 which is repeated as long as an image capturing user input is received in step 402 ... “Sequential” capturing of frames by a digital camera shooting a scene refers to continuous operation where consequent frames, following each other in time, are captured, the frames representing the same scene at consequent moments);
select one or more frames of the set of consequent frames based on user focus areas of a user (Paragraph [0041], selection criterion may be used for selecting the starting frame of the displayable frame sequence. Such further criterion may be, for example, a user input received before or during capturing the frames of the preliminary frame sequence, which user input may relate, for example, to selecting a desired focus point or area corresponding to a specific target in the scene. For example, when user of the camera has indicated a preferred focus area or point ...).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana incorporate the teachings of Träff, and applying the method for obtaining a preliminary frame sequence and selecting a starting frame from the preliminary frame sequence taught by Träff to capture consequent frames by a time interval and provide additional condition for selecting a frame from a set of consequent frames based on the focus point of the user. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana according to the relied-upon teachings of Träff to obtain the invention as specified in claim.
It’s noted that Maharana discloses that the device of the vehicle transmits the selected frame to the server (As shown in FIG. 11D). However, Maharana does not specifically disclose transmit the selected frames to an edge server.
In additional, SOMMER discloses transmit the selected frames to an edge server (Paragraph [0006], the document generally describes technology to generating and selecting best digital images of a person, such as a guest in a retail environment or store, from a digital image data stream (e.g., video feed from a security camera) as they move throughout a physical environment or physical space. The disclosed technology includes techniques for more efficiently generating best digital images of people in the physical environment, which can permit for the generation of these images to be pushed to edge computing devices that leverage low computational resources (e.g., low powered processors, small amount of RAM) ... As shown in FIG. 1, paragraphs [0038]-[0042]).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana in view Träff incorporate the teachings of SOMMER, and applying the system for generating best images of a person in a retail environment taught by SOMMER to implement the server taught by Maharana as an edge server to perform computational operations for the selected frame by the reality device. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana in view Träff according to the relied-upon teachings of SOMMER to obtain the invention as specified in claim.
Regarding claim 2, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 1), and Maharana further disclose wherein the one or more processors are further operable to compress one or more of the selected one or more frames (FIG. 1; paragraph [0079], during encoding, at least one sender neural network 108 to such image data 128 to assemble, based at least in part on differences of an attribute 110B of an object 110A, a set of reference frames 132—e.g., in a cache 122. This function can encode the set of reference images in the video stream to generate an encoded video stream; paragraph [0153], a generative adversarial network (GAN) can be used for training one or more neural networks discussed throughout herein to perform video compression and decompression in order to facilitate video streaming, such as video conferencing, according to at least one embodiment. In at least one embodiment, one or more neural networks herein is usable for video streaming, such as video conferencing, as described above in conjunction with FIGS. 1-5) before transmitting to the edge server (Paragraph [0076], a video sender 102 can select a reference frame for inclusion in a set of frames having all reference frames that is cached for transmission to a video receiver 116) based on the network quality metrics and the pending frame queue (see claim 1).
Regarding claim 5, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 1), and Maharana further disclose wherein the network quality metrics comprise round-trip time, bandwidth (Paragraph [0059], ... require continuous use of large bandwidths to provide sufficient information with each sent reference frame), packet loss, network congestion, error rate, or a combination thereof.
Regarding claim 7, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 1), and Maharana further disclose wherein the one or more processors are further operable to receive object detection data from the edge server (FIG. 11D; paragraph [0314], deep-learning infrastructure of server(s) 1178 ... in at least one embodiment, deep-learning infrastructure may receive periodic updates from vehicle 1100, such as a sequence of images and/or objects that vehicle 1100 has located in that sequence of images (e.g., via computer vision and/or other machine learning object classification techniques)).
Regarding claim 8, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 7), and Maharana further disclose wherein the one or more processors are further operable to autonomously drive the vehicle based on the object detection data (FIG. 11D; paragraph [0314], deep-learning infrastructure of server(s) 1178 ... deep-learning infrastructure may run its own neural network to identify objects and compare them with objects identified by vehicle 1100 and, if results do not match and deep-learning infrastructure concludes that AI in vehicle 1100 is malfunctioning, then server(s) 1178 may transmit a signal to vehicle 1100 instructing a fail-safe computer of vehicle 1100 to assume control, notify passengers, and complete a safe parking maneuver).
Regarding claim 12, Maharana discloses a method for reducing latency and bandwidth usage comprising:
selecting one or more frames of a set of FIGS.11A to 11C show vehicle 1100; paragraphs [0222]-[0223], FIG. 11B illustrates an example of camera locations and fields of view for autonomous vehicle 1100 of FIG. 11A, ... camera types for cameras may include, but are not limited to, digital cameras that may be adapted for use with components and/or systems of vehicle 1100. ... In at least one embodiment, camera types may be capable of any image capture rate, such as 60 frames per second (fps), 1220 fps, 240 fps, etc.; paragraph [0291], cameras may be used to capture image data around an entire periphery of vehicle 1100) and skipping rest of the set of Paragraph [0076], a video sender 102 can select a reference frame for inclusion in a set of frames having all reference frames that is cached for transmission to a video receiver 116) based on network quality metrics (Paragraph [0059], ... require continuous use of large bandwidths to provide sufficient information with each sent reference frame) and a pending frame queue (Paragraphs [0075]-[0076], a reference frame 110, such as a video frame, is a first image in a sequence of images and can be a first frame in a sequence of video frames ... a video sender 102 can select a reference frame for inclusion in a set of frames having all reference frames that is cached ... Thus, a sequence of images of reference frame 1, reference frame 2 and etc. within the processor 106 can be interpreted as a pending frame queue); and
transmitting the selected one or more frames to an Paragraph [0076], a video sender 102 can select a reference frame for inclusion in a set of frames having all reference frames that is cached for transmission to a video receiver 116) for performing a task on behalf of a vehicle (Paragraph [0094], video receiver 116 may include a receiver neural network 124 having data values and software instructions that, when executed by a receiver processor 126, reconstruct or otherwise infer one or more reference frames 134 to be used with one or more prediction frames to cause video to be displayed by one or more video output devices 118).
However, Maharana does not specifically disclose a set of consequent frames captured by a device;
selecting one or more frames of the set of consequent frames based on user focus areas of a user; and
transmitting the selected one or more frames to an edge server.
In additional, Träff discloses (Paragraph [0030], in one embodiment, the starting scene condition requires movement occurring in the scene shot by the camera during capturing the frames of the preliminary frame sequence. Such movement can be, for example, rotational and/or translational movement or any other type of movement of an object lying within or forming a part of the scene. A moving object within the scene may be, for example, a human being, an animal, a vehicle, a plant, or any other object affecting the general nature of the captured moment ...) a set of consequent frames captured by a device (FIG. 4; paragraphs [0057]-[0058], starts by sequentially capturing, in step 401 which is repeated as long as an image capturing user input is received in step 402 ... “Sequential” capturing of frames by a digital camera shooting a scene refers to continuous operation where consequent frames, following each other in time, are captured, the frames representing the same scene at consequent moments);
selecting one or more frames of the set of consequent frames based on user focus areas of a user (Paragraph [0041], selection criterion may be used for selecting the starting frame of the displayable frame sequence. Such further criterion may be, for example, a user input received before or during capturing the frames of the preliminary frame sequence, which user input may relate, for example, to selecting a desired focus point or area corresponding to a specific target in the scene. For example, when user of the camera has indicated a preferred focus area or point ...).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana incorporate the teachings of Träff, and applying the method for obtaining a preliminary frame sequence and selecting a starting frame from the preliminary frame sequence taught by Träff to capture consequent frames by a time interval and provide additional condition for selecting a frame from a set of consequent frames based on the focus point of the user. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana according to the relied-upon teachings of Träff to obtain the invention as specified in claim.
It’s noted that Maharana discloses that the device of the vehicle transmits the selected frame to the server (As shown in FIG. 11D). However, Maharana does not specifically disclose transmitting the selected one or more frames to an edge server.
In additional, SOMMER discloses transmitting the selected one or more frames to an edge server (Paragraph [0006], the document generally describes technology to generating and selecting best digital images of a person, such as a guest in a retail environment or store, from a digital image data stream (e.g., video feed from a security camera) as they move throughout a physical environment or physical space. The disclosed technology includes techniques for more efficiently generating best digital images of people in the physical environment, which can permit for the generation of these images to be pushed to edge computing devices that leverage low computational resources (e.g., low powered processors, small amount of RAM) ... As shown in FIG. 1, paragraphs [0038]-[0042]).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana in view Träff incorporate the teachings of SOMMER, and applying the system for generating best images of a person in a retail environment taught by SOMMER to implement the server taught by Maharana as an edge server to perform computational operations for the selected frame by the reality device. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana in view Träff according to the relied-upon teachings of SOMMER to obtain the invention as specified in claim.
Regarding claim 13, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 12), and Maharana further disclose wherein the method further comprises compressing one or more of the selected one or more frames (FIG. 1; paragraph [0079], during encoding, at least one sender neural network 108 to such image data 128 to assemble, based at least in part on differences of an attribute 110B of an object 110A, a set of reference frames 132—e.g., in a cache 122. This function can encode the set of reference images in the video stream to generate an encoded video stream; paragraph [0153], a generative adversarial network (GAN) can be used for training one or more neural networks discussed throughout herein to perform video compression and decompression in order to facilitate video streaming, such as video conferencing, according to at least one embodiment. In at least one embodiment, one or more neural networks herein is usable for video streaming, such as video conferencing, as described above in conjunction with FIGS. 1-5) before transmitting to the edge server (Paragraph [0076], a video sender 102 can select a reference frame for inclusion in a set of frames having all reference frames that is cached for transmission to a video receiver 116) based on the network quality metrics and the pending frame queue (see claim 12).
Regarding claim 16, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 12), and Maharana further disclose wherein the network quality metrics comprise round-trip time, bandwidth (Paragraph [0059], ... require continuous use of large bandwidths to provide sufficient information with each sent reference frame), packet loss, network congestion, error rate, or a combination thereof.
Regarding claim 18, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 12), and Maharana further disclose wherein the method further comprises:
receiving object detection data from the edge server (FIG. 11D; paragraph [0314], deep-learning infrastructure of server(s) 1178 ... in at least one embodiment, deep-learning infrastructure may receive periodic updates from vehicle 1100, such as a sequence of images and/or objects that vehicle 1100 has located in that sequence of images (e.g., via computer vision and/or other machine learning object classification techniques)); and
autonomously driving the vehicle based on the object detection data (Paragraph [0314], deep-learning infrastructure of server(s) 1178 ... deep-learning infrastructure may run its own neural network to identify objects and compare them with objects identified by vehicle 1100 and, if results do not match and deep-learning infrastructure concludes that AI in vehicle 1100 is malfunctioning, then server(s) 1178 may transmit a signal to vehicle 1100 instructing a fail-safe computer of vehicle 1100 to assume control, notify passengers, and complete a safe parking maneuver).
Claims 3 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Maharana et al (U.S. Patent Application Publication 2024/0114180 A1) in view of Träff et al (U.S. Patent Application Publication 2016/0353052 A1) in view of SOMMER et al (U.S. Patent Application Publication 2023/0419674 A1) in view of CHAURASIA et al (U.S. Patent Application Publication 2020/0265649 A1).
Regarding claim 3, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 1).
However, Maharana does not specifically disclose wherein the reality device further comprises an eye-tracking sensor operable to track user eye movements, and a head-tracking sensor operable to track user head movements; and
the one or more processors are further operable to:
determine whether the user focus areas in a corresponding frame of the set of consequent frames are off-interest to the user based on the user eye movements and the user head movements; and
in response to determining that the user focus areas are off-interest to the user, skip the corresponding frame.
In additional, CHAURASIA discloses wherein the reality device (Paragraph [0064], FIG. 2 illustrates a block diagram 200 depicting the computer implemented system for displaying contents on an AR device 102 of FIG. 1) further comprises an eye-tracking sensor operable to track user eye movements (Paragraphs [0072]-[0073], the user input controller 210 comprises an eye gaze tracking module 212 ... The eye gaze tracking module 212 is configured to track the vision of the user by identifying an eye gaze, direction of eye movement, dilation of pupils, a number of eye blinks, facial expression, widening of eye size, pupil and iris movement, and/or other vision related parameters), and a head-tracking sensor operable to track user head movements (Paragraph [0072], the user input controller 210 comprises a head movement tracking module 214; paragraph [0074], the head movement tracking module 214 is configured to track direction of head movement, an angle of rotation of head, speed of rotation of head, and other head related parameters); and
the one or more processors are further operable to:
determine whether the user focus areas in a corresponding frame of the set of consequent frames (FIG. 5; paragraph [0124], the flow diagram 500 starts from step 502 of monitoring and storing a video/image of objects in a user's field of view ) are off-interest to the user (Paragraph [0120], the user starts watching the object continuously at time t1 (i.e., at 16:10:40). At a point 2, the user gazes at the object for threshold time, and it becomes an “object of interest” ... it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to understand that the image of object is “off-interest to the user” if the user gazes at the object below the threshold time) based on the user eye movements and the user head movements (Paragraph [0124], the user input controller 210 is configured to track the user's eyes and movement); and
in response to determining that the user focus areas are off-interest to the user, skip the corresponding frame (Paragraph [0125], at step 502, the process comprises determining an object of interest based on a threshold gaze time and other parameters. In an embodiment, the determination module 224 determines an object of interest based on a threshold gaze time and other parameters. It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to understand that the image of object is skip if the user gazes at the object below the threshold time “off-interest to the user”).
It's noted that CHAURASIA describes “set of consequent frames”. But CHAURASIA describes the method for capturing video/image of objects and determining an object of interest from the captured video/image of objects based on the automated tracking the user's eyes and movement. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana in view Träff in view of SOMMER incorporate the teachings of CHAURASIA, and applying the method for determining the an object of interest based on a threshold gaze time taught by CHAURASIA to determine and skip the “off-interest” of frames from the set of consequent frames based on the user eye movements and the user head movements. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana in view Träff in view of SOMMER according to the relied-upon teachings of CHAURASIA to obtain the invention as specified in claim.
Regarding claim 14, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 12).
However, Maharana does not specifically disclose wherein the method further comprises:
determining whether the user focus areas in a corresponding frame of the set of consequent frames are off-interest to the user based on user eye movements and user head movements tracked by the reality device; and
in response to determining that the user focus areas are off-interest to the user, skipping the corresponding frame.
In additional, CHAURASIA discloses (Paragraph [0064], FIG. 2 illustrates a block diagram 200 depicting the computer implemented system for displaying contents on an AR device 102 of FIG. 1) wherein the method further comprises:
determining whether the user focus areas in a corresponding frame of the set of consequent frames (FIG. 5; paragraph [0124], the flow diagram 500 starts from step 502 of monitoring and storing a video/image of objects in a user's field of view ) are off-interest to the user (Paragraph [0120], the user starts watching the object continuously at time t1 (i.e., at 16:10:40). At a point 2, the user gazes at the object for threshold time, and it becomes an “object of interest” ... it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to understand that the image of object is “off-interest to the user” if the user gazes at the object below the threshold time) based on user eye movements (Paragraphs [0072]-[0073], the user input controller 210 comprises an eye gaze tracking module 212 ... The eye gaze tracking module 212 is configured to track the vision of the user by identifying an eye gaze, direction of eye movement, dilation of pupils, a number of eye blinks, facial expression, widening of eye size, pupil and iris movement, and/or other vision related parameters) and user head movements tracked (Paragraph [0072], the user input controller 210 comprises a head movement tracking module 214; paragraph [0074], the head movement tracking module 214 is configured to track direction of head movement, an angle of rotation of head, speed of rotation of head, and other head related parameters) by the reality device (Paragraph [0124], the user input controller 210 is configured to track the user's eyes and movement); and
in response to determining that the user focus areas are off-interest to the user, skipping the corresponding frame (Paragraph [0125], at step 502, the process comprises determining an object of interest based on a threshold gaze time and other parameters. In an embodiment, the determination module 224 determines an object of interest based on a threshold gaze time and other parameters. It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to understand that the image of object is skip if the user gazes at the object below the threshold time “off-interest to the user”).
It's noted that CHAURASIA describes “set of consequent frames”. But CHAURASIA describes the method for capturing video/image of objects and determining an object of interest from the captured video/image of objects based on the automated tracking the user's eyes and movement. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana in view Träff in view of SOMMER incorporate the teachings of CHAURASIA, and applying the method for determining the an object of interest based on a threshold gaze time taught by CHAURASIA to determine and skip the “off-interest” of frames from the set of consequent frames based on the user eye movements and the user head movements. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana in view Träff in view of SOMMER according to the relied-upon teachings of CHAURASIA to obtain the invention as specified in claim.
Claims 4 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Maharana et al (U.S. Patent Application Publication 2024/0114180 A1) in view of Träff et al (U.S. Patent Application Publication 2016/0353052 A1) in view of SOMMER et al (U.S. Patent Application Publication 2023/0419674 A1) in view of Zhou et al (U.S. Patent No. 9,143,693 B1).
Regarding claim 4, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 1), and Maharana further disclose wherein the one or more processors are further operable to:
select one or more representative frames of the set of consequent frames and skip rest of the set of consequent frames (FIG. 1; paragraph [0076], a video sender 102 can select a reference frame for inclusion in a set of frames having all reference frames ...; paragraph [0090], a video sender 102 may transmit or otherwise communicate a video stream 130 having a set of reference frames 110 assembled at a video sender 102 and may transmit or otherwise communicate one or more features 112 in a video stream 130, over a network 114 at different times.), wherein the representative frames represent a representative scene and change in the consequent frames (Paragraph [0083], one or more sender neural networks 108 can be used by a video sender 102 to generate or otherwise infer features 112 that are associated with at least one attribute 110B of provided video data 128. In at least one embodiment, such video data may represent one or more videos that may include different content, such as a face, which may be determined as an object for pursuing with respect to criteria forming a basis for a reference frame ...; paragraph [0089], a video sender 102 or any other entity, such as a streaming video or video conferencing provider, trains one or more sender neural networks 108 using any type of training framework usable to train one or more neural networks 108 to infer different attributes from features 112, to select reference frames 110 for a cache 122).
However, Maharana does not specifically disclose determine whether the pending frame queue is beyond a pending threshold; and
in response to determining that the pending frame queue is beyond the pending threshold, select one or more representative frames of the set of consequent frames.
In additional, Zhou discloses determine whether the pending frame queue is beyond a pending threshold (Col 3, lines 47-67, the live view interface may include a display, such as may be found on a smartphone, tablet, television, head-mountable display (HMD), or heads-up display (HUD). The live view interface may be configured to display a sequence of image frames and/or a representation of the sequence of image frames ... The image buffer may be configured to temporarily store, queue ... The image buffer may have a static or dynamic buffer size); and
in response to determining that the pending frame queue is beyond the pending threshold (Col 3, lines 47-67, if the number of buffered frames reaches a buffer threshold), select one or more representative frames of the set of consequent frames (Col 3, lines 47-67, ).
It's noted that Zhou describes “set of consequent frames”. But Zhou describes the method for controlling the sequence of image frames in the image buffer. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana in view Träff in view of SOMMER incorporate the teachings of Zhou, and applying the method for controlling the sequence of image frames in the image buffer taught by Zhou to determine the size of the image queue and select one or more image frames from the image queue when the number of buffered frames reaches a buffer threshold. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana in view Träff in view of SOMMER according to the relied-upon teachings of Zhou to obtain the invention as specified in claim.
Regarding claim 15, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 12), and Maharana further disclose wherein the method further comprises:
selecting one or more representative frames of the set of consequent frames and skipping rest of the set of consequent frames (FIG. 1; paragraph [0076], a video sender 102 can select a reference frame for inclusion in a set of frames having all reference frames ...; paragraph [0090], a video sender 102 may transmit or otherwise communicate a video stream 130 having a set of reference frames 110 assembled at a video sender 102 and may transmit or otherwise communicate one or more features 112 in a video stream 130, over a network 114 at different times), wherein the representative frames represent a representative scene and change in the consequent frames (Paragraph [0083], one or more sender neural networks 108 can be used by a video sender 102 to generate or otherwise infer features 112 that are associated with at least one attribute 110B of provided video data 128. In at least one embodiment, such video data may represent one or more videos that may include different content, such as a face, which may be determined as an object for pursuing with respect to criteria forming a basis for a reference frame ...; paragraph [0089], a video sender 102 or any other entity, such as a streaming video or video conferencing provider, trains one or more sender neural networks 108 using any type of training framework usable to train one or more neural networks 108 to infer different attributes from features 112, to select reference frames 110 for a cache 122).
However, Maharana does not specifically disclose determining whether the pending frame queue is beyond a pending threshold; and
in response to determining that the pending frame queue is beyond the pending threshold, selecting one or more representative frames of the set of consequent frames.
In additional, Zhou discloses determining whether the pending frame queue is beyond a pending threshold (Col 3, lines 47-67, the live view interface may include a display, such as may be found on a smartphone, tablet, television, head-mountable display (HMD), or heads-up display (HUD). The live view interface may be configured to display a sequence of image frames and/or a representation of the sequence of image frames ... The image buffer may be configured to temporarily store, queue ... The image buffer may have a static or dynamic buffer size); and
in response to determining that the pending frame queue is beyond the pending threshold (Col 3, lines 47-67, if the number of buffered frames reaches a buffer threshold), selecting one or more representative frames of the set of consequent frames (Col 3, lines 47-67, ).
It's noted that Zhou describes “set of consequent frames”. But Zhou describes the method for controlling the sequence of image frames in the image buffer. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana in view Träff in view of SOMMER incorporate the teachings of Zhou, and applying the method for controlling the sequence of image frames in the image buffer taught by Zhou to determine the size of the image queue and select one or more image frames from the image queue when the number of buffered frames reaches a buffer threshold. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana in view Träff in view of SOMMER according to the relied-upon teachings of Zhou to obtain the invention as specified in claim.
Claims 6 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Maharana et al (U.S. Patent Application Publication 2024/0114180 A1) in view of Träff et al (U.S. Patent Application Publication 2016/0353052 A1) in view of SOMMER et al (U.S. Patent Application Publication 2023/0419674 A1) in view of KANG et al (U.S. Patent Application Publication 2013/0235935 A1).
Regarding claim 6, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 1), and Maharana further disclose wherein the one or more processors are further operable to:
select one or more representative frames of the set of consequent frames and skip rest of the set of consequent frames (FIG. 1; paragraph [0076], a video sender 102 can select a reference frame for inclusion in a set of frames having all reference frames ...; paragraph [0090], a video sender 102 may transmit or otherwise communicate a video stream 130 having a set of reference frames 110 assembled at a video sender 102 and may transmit or otherwise communicate one or more features 112 in a video stream 130, over a network 114 at different times), wherein the representative frames represent a representative scene and change in the consequent frames (Paragraph [0083], one or more sender neural networks 108 can be used by a video sender 102 to generate or otherwise infer features 112 that are associated with at least one attribute 110B of provided video data 128. In at least one embodiment, such video data may represent one or more videos that may include different content, such as a face, which may be determined as an object for pursuing with respect to criteria forming a basis for a reference frame ...; paragraph [0089], a video sender 102 or any other entity, such as a streaming video or video conferencing provider, trains one or more sender neural networks 108 using any type of training framework usable to train one or more neural networks 108 to infer different attributes from features 112, to select reference frames 110 for a cache 122).
However, Maharana does not specifically disclose determine whether the network quality metrics are greater than a network traffic threshold; and
in response to determining that the network quality metrics are greater than the network traffic threshold, select one or more representative frames of the set of consequent frames.
In additional, KANG discloses determine whether the network quality metrics are greater than a network traffic threshold (Paragraph [0026], the network bandwidth is determined to exceed the highest threshold); and
in response to determining that the network quality metrics are greater than the network traffic threshold (Paragraph [0026], the network bandwidth is determined to exceed the highest threshold, selecting a first image type having a highest compression rate), select one or more representative frames of the set of consequent frames (Paragraph [0038], FIG. 1 shows an image sequence configuration at the time of image compression; paragraph [0026], the network bandwidth is determined to exceed the highest threshold, selecting a first image type having a highest compression rate; and transmitting a frame related to a selected image type).
It's noted that KANG describes “set of consequent frames”. But KANG describes the method for providing image data for each image type based on the network bandwidth. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana in view Träff in view of SOMMER incorporate the teachings of KANG, and applying the method for providing image data for each image type buffer taught by KANG to determine the network bandwidth and select one or more image frames from the image sequence when the determined network bandwidth are greater than the network traffic threshold. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana in view Träff in view of SOMMER according to the relied-upon teachings of KANG to obtain the invention as specified in claim.
Regarding claim 17, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 12), and Maharana further disclose wherein the method further comprises:
selecting one or more representative frames of the set of consequent frames and skipping rest of the set of consequent frames (FIG. 1; paragraph [0076], a video sender 102 can select a reference frame for inclusion in a set of frames having all reference frames ...; paragraph [0090], a video sender 102 may transmit or otherwise communicate a video stream 130 having a set of reference frames 110 assembled at a video sender 102 and may transmit or otherwise communicate one or more features 112 in a video stream 130, over a network 114 at different times), wherein the representative frames represent a representative scene and change in the consequent frames (Paragraph [0083], one or more sender neural networks 108 can be used by a video sender 102 to generate or otherwise infer features 112 that are associated with at least one attribute 110B of provided video data 128. In at least one embodiment, such video data may represent one or more videos that may include different content, such as a face, which may be determined as an object for pursuing with respect to criteria forming a basis for a reference frame ...; paragraph [0089], a video sender 102 or any other entity, such as a streaming video or video conferencing provider, trains one or more sender neural networks 108 using any type of training framework usable to train one or more neural networks 108 to infer different attributes from features 112, to select reference frames 110 for a cache 122).
However, Maharana does not specifically disclose determining whether the network quality metrics are greater than a network traffic threshold; and
in response to determining that the network quality metrics are greater than the network traffic threshold, selecting one or more representative frames of the set of consequent frames.
In additional, KANG discloses determine determining whether the network quality metrics are greater than a network traffic threshold (Paragraph [0026], the network bandwidth is determined to exceed the highest threshold); and
in response to determining that the network quality metrics are greater than the network traffic threshold (Paragraph [0026], the network bandwidth is determined to exceed the highest threshold, selecting a first image type having a highest compression rate), selecting one or more representative frames of the set of consequent frames (Paragraph [0038], FIG. 1 shows an image sequence configuration at the time of image compression; paragraph [0026], the network bandwidth is determined to exceed the highest threshold, selecting a first image type having a highest compression rate; and transmitting a frame related to a selected image type).
It's noted that KANG describes “set of consequent frames”. But KANG describes the method for providing image data for each image type based on the network bandwidth. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana in view Träff in view of SOMMER incorporate the teachings of KANG, and applying the method for providing image data for each image type buffer taught by KANG to determine the network bandwidth and select one or more image frames from the image sequence when the determined network bandwidth are greater than the network traffic threshold. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana in view Träff in view of SOMMER according to the relied-upon teachings of KANG to obtain the invention as specified in claim.
Claims 9-11 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Maharana et al (U.S. Patent Application Publication 2024/0114180 A1) in view of Träff et al (U.S. Patent Application Publication 2016/0353052 A1) in view of SOMMER et al (U.S. Patent Application Publication 2023/0419674 A1) in view of Schlacter et al (U.S. Patent Application Publication 2025/0026376 A1).
Regarding claim 9, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 7).
However, Maharana does not specifically disclose wherein the object detection data comprises box cords of detected objects in the selected frames, confidence of each corresponding box cord, and object information of each detected object.
In additional, Schlacter discloses (Abstract, embodiments herein include an autonomy system of an automated vehicle performing operations for detecting and avoiding lazy or unpredictable vehicles on a roadway. The autonomy system generates bounding boxes for tracking traffic vehicles recognized in sensor data ...) wherein the object detection data (FIG. 1; paragraph [0020], the autonomy system 150 applies to image data generated by cameras of the truck 102 ...; paragraph [0028], the perception module of the autonomy system 150 may receive input sensor data from the various sensors, such as the one or more cameras ... the perception module of the autonomy system 150 may include, communicate with, or otherwise execute software for performing object tracking and/or object classification functions allowing the autonomy system 150 to perform object detection and classification operations) comprises box cords of detected objects in the selected frames (Paragraph [0023], using the image data, the autonomy system 150 may recognize, detect, or otherwise identify instances that the bounding box 142 or expanded box 144 of the traffic vehicle 140), confidence of each corresponding box cord (FIG. 3; paragraph [0055], the artificial intelligence model 310 can generate a respective prediction (e.g., classification, object location, object size/bounding box) for each cell extracted from the input image. As such, each cell can correspond to a respective prediction, presence, and location of an object within its respective area of the input image. The artificial intelligence model 310 may also generate one or more respective confidence values indicating a level of confidence that the predictions are correct. If an object represented in the image spans multiple cells, the cell with the highest prediction confidence can be utilized to detect the object), and object information of each detected object (Paragraph [0023], the autonomy system may, for example, detect instances when the traffic vehicle 140 veers into (or appears to veer into) the current travel lane of the truck 102).
It's noted that Schlacter describes “set of consequent frames”. But Schlacter describes the method for detecting object in the sequence of image frames. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana in view Träff in view of SOMMER incorporate the teachings of Schlacter, and applying the autonomy system of an automated vehicle performing operations for detecting and avoiding lazy or unpredictable vehicles on a roadway buffer taught by Schlacter to detect object in the sequence image. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana in view Träff in view of SOMMER according to the relied-upon teachings of Schlacter to obtain the invention as specified in claim.
Regarding claim 10, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 7).
However, Maharana does not specifically disclose wherein the one or more processors are further operable to superimpose the object detection data onto a real-world view.
In additional, Schlacter discloses (Abstract, embodiments herein include an autonomy system of an automated vehicle performing operations for detecting and avoiding lazy or unpredictable vehicles on a roadway. The autonomy system generates bounding boxes for tracking traffic vehicles recognized in sensor data ...) wherein the one or more processors are further operable to superimpose the object detection data (FIG. 1; paragraph [0020], the autonomy system 150 applies to image data generated by cameras of the truck 102 ... The autonomy system 150 places or overlays bounding boxes 142 around the objects detected in the image data. For instance, a bounding box 142 includes a rectangular box representation that the autonomy system 150 generates and applies (or “draws”) around an object in the image data, as identified by the computer vision engine. Each bounding box 142 represents the location and scale of potential or recognized objects within the imagery of the image data, such as traffic vehicles 140, pedestrians, bicycles, street signs, or any other relevant features in the environment 100 ...; paragraph [0028], the perception module of the autonomy system 150 may receive input sensor data from the various sensors, such as the one or more cameras ... the perception module of the autonomy system 150 may include, communicate with, or otherwise execute software for performing object tracking and/or object classification functions allowing the autonomy system 150 to perform object detection and classification operations) onto a real-world view (Paragraph [0027], one or more cameras (not shown) mounted around the truck 102 and coupled to the autonomy system 150. The cameras capture imagery of the roadway environment 100 surrounding the truck 102 within the cameras' field-of-view (e.g., perception radius 130) and generate image data for the imagery. The camera sends the image data generated to the perception module of the autonomy system 150).
It's noted that Schlacter describes “set of consequent frames”. But Schlacter describes the method for detecting object in the sequence of image frames. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana in view Träff in view of SOMMER incorporate the teachings of Schlacter, and applying the autonomy system of an automated vehicle performing operations for detecting and avoiding lazy or unpredictable vehicles on a roadway buffer taught by Schlacter to detect object in the sequence image and overlays bounding boxes around the objects detected in the image data. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana in view Träff in view of SOMMER according to the relied-upon teachings of Schlacter to obtain the invention as specified in claim.
Regarding claim 11, the combination of Maharana in view Träff in view of SOMMER in view of Schlacter discloses everything claimed as applied above (see claim 10). As discussed in claim 10, Schlacter discloses wherein the object detection data are superimposed onto a vision of a user or a current frame (FIG. 1; paragraph [0020], the autonomy system 150 applies to image data generated by cameras of the truck 102 ... The autonomy system 150 places or overlays bounding boxes 142 around the objects detected in the image data; paragraph [0027], one or more cameras (not shown) mounted around the truck 102 and coupled to the autonomy system 150. The cameras capture imagery of the roadway environment 100 surrounding the truck 102 within the cameras' field-of-view (e.g., perception radius 130) and generate image data for the imagery. The camera sends the image data generated to the perception module of the autonomy system 150).
Regarding claim 19, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 18).
However, Maharana does not specifically disclose wherein the object detection data comprises box cords of detected objects in the selected frames, confidence of each corresponding box cord, and object information of each detected object.
In additional, Schlacter discloses (Abstract, embodiments herein include an autonomy system of an automated vehicle performing operations for detecting and avoiding lazy or unpredictable vehicles on a roadway. The autonomy system generates bounding boxes for tracking traffic vehicles recognized in sensor data ...) wherein the object detection data (FIG. 1; paragraph [0020], the autonomy system 150 applies to image data generated by cameras of the truck 102 ...; paragraph [0028], the perception module of the autonomy system 150 may receive input sensor data from the various sensors, such as the one or more cameras ... the perception module of the autonomy system 150 may include, communicate with, or otherwise execute software for performing object tracking and/or object classification functions allowing the autonomy system 150 to perform object detection and classification operations) comprises box cords of detected objects in the selected frames (Paragraph [0023], using the image data, the autonomy system 150 may recognize, detect, or otherwise identify instances that the bounding box 142 or expanded box 144 of the traffic vehicle 140), confidence of each corresponding box cord (FIG. 3; paragraph [0055], the artificial intelligence model 310 can generate a respective prediction (e.g., classification, object location, object size/bounding box) for each cell extracted from the input image. As such, each cell can correspond to a respective prediction, presence, and location of an object within its respective area of the input image. The artificial intelligence model 310 may also generate one or more respective confidence values indicating a level of confidence that the predictions are correct. If an object represented in the image spans multiple cells, the cell with the highest prediction confidence can be utilized to detect the object), and object information of each detected object (Paragraph [0023], the autonomy system may, for example, detect instances when the traffic vehicle 140 veers into (or appears to veer into) the current travel lane of the truck 102).
It's noted that Schlacter describes “set of consequent frames”. But Schlacter describes the method for detecting object in the sequence of image frames. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana in view Träff in view of SOMMER incorporate the teachings of Schlacter, and applying the autonomy system of an automated vehicle performing operations for detecting and avoiding lazy or unpredictable vehicles on a roadway buffer taught by Schlacter to detect object in the sequence image. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana in view Träff in view of SOMMER according to the relied-upon teachings of Schlacter to obtain the invention as specified in claim.
Regarding claim 20, the combination of Maharana in view Träff in view of SOMMER discloses everything claimed as applied above (see claim 18).
However, Maharana does not specifically disclose wherein the method further comprises superimposing the object detection data onto a real-world view.
In additional, Schlacter discloses (Abstract, embodiments herein include an autonomy system of an automated vehicle performing operations for detecting and avoiding lazy or unpredictable vehicles on a roadway. The autonomy system generates bounding boxes for tracking traffic vehicles recognized in sensor data ...) wherein the method further comprises superimposing the object detection data (FIG. 1; paragraph [0020], the autonomy system 150 applies to image data generated by cameras of the truck 102 ... The autonomy system 150 places or overlays bounding boxes 142 around the objects detected in the image data. For instance, a bounding box 142 includes a rectangular box representation that the autonomy system 150 generates and applies (or “draws”) around an object in the image data, as identified by the computer vision engine. Each bounding box 142 represents the location and scale of potential or recognized objects within the imagery of the image data, such as traffic vehicles 140, pedestrians, bicycles, street signs, or any other relevant features in the environment 100 ...; paragraph [0028], the perception module of the autonomy system 150 may receive input sensor data from the various sensors, such as the one or more cameras ... the perception module of the autonomy system 150 may include, communicate with, or otherwise execute software for performing object tracking and/or object classification functions allowing the autonomy system 150 to perform object detection and classification operations) onto a real-world view (Paragraph [0027], one or more cameras (not shown) mounted around the truck 102 and coupled to the autonomy system 150. The cameras capture imagery of the roadway environment 100 surrounding the truck 102 within the cameras' field-of-view (e.g., perception radius 130) and generate image data for the imagery. The camera sends the image data generated to the perception module of the autonomy system 150).
It's noted that Schlacter describes “set of consequent frames”. But Schlacter describes the method for detecting object in the sequence of image frames. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the frame selection for streaming applications taught by Maharana in view Träff in view of SOMMER incorporate the teachings of Schlacter, and applying the autonomy system of an automated vehicle performing operations for detecting and avoiding lazy or unpredictable vehicles on a roadway buffer taught by Schlacter to detect object in the sequence image and overlays bounding boxes around the objects detected in the image data. Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify Maharana in view Träff in view of SOMMER according to the relied-upon teachings of Schlacter to obtain the invention as specified in claim.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Xilin Guo whose telephone number is (571)272-5786. The examiner can normally be reached Monday - Friday 9:00 AM-5:30 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XILIN GUO/Primary Examiner, Art Unit 2616
Prosecution Insights