DETAILED ACTION
This office action is in response to the application filed on December 31, 2024. Claims 1 – 20 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for priority based on U.S. provisional applications 63/619,789 filed on January 11, 2024.
Information Disclosure Statement
The information disclosure statement (IDS) was submitted March 18, 2025. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the Examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 - 6 and 11 - 16 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Boddeti et al., (US 2023/0048189 A1) referred to as Boddeti hereinafter.
Regarding Claim 1, Boddeti discloses a system for controlling video streaming (Fig. 1, Fig. 2, Fig. 5), the system comprising:
a video camera (Par. [0125], The computing device 900 may be include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems) configured to capture video (Par. [0022], video data that captures one or more of a scene change, a static scene, a low motion scene, or a high motion scene) and create a video stream (Par. [0046], stream video content across a network); and
at least one processor configured to (Par. [0070], a processor executing instructions stored in memory):
extract at least one video feature from the video stream (Par. [0049] the machine learning model (MLM(s)) 107 detects a static scene, scene change, or high motion scene (i.e. video feature) in the deployment video data, where a feature vector representing a static scene may be detected. Par. [0066], a vector representing a portion or attribute (e.g., a video frame or sequence of video frames) of video 108 content);
process the video stream according to at least one processing parameter to produce a processed video stream (Par. [0001] Various encoding parameters may correspond to encoding quality, video resolution (i.e. processing parameter), frame rate (i.e. processing parameter), audio and/or image bitrate, encoding mode, pixel aspect ratio, error correction, and more);
encode the processed video stream according to at least one encoding parameter to produce an encoded video stream (Fig. 1, Par. [0004], The video encoder may then use the value of the encoding parameter to encode subsequent video data for the streaming, Par. [0019] the encoding parameter(s) may correspond to a bitrate setting, such as a target bitrate for the encoded video, and/or an error correction or channel coding parameter, such as Forward Error Correction (FEC));
transmit the encoded video stream through a network (Par. [0001], content is being transmitted over digital networks using video streaming technologies Par. [0071] Step B502, transmit data to cause an encoder to use a first value of an encoding parameter to convert first video data into first encoded video data);
receive at least one network metric based on the encoded video stream transmitted through the network (Par. [0074]-[0075], Step B504, one or more embodiments (e.g., the server(s) 104) receive feedback associated with a receipt, using a network client (e.g., the content server 205 or the sender 105), of a stream comprising the first encoded video data (e.g., receiving a stream of the first encoded video data, where the feedback (i.e. network metric) is based at least on one or more of: a bitrate of the encoded video, a quantity of dropped frames of the encoded video, packet loss, stutter, network bandwidth, macroblock count of the encoded video, packet arrival time, video quality, network latency, and/or an encoded bitrate to target bitrate ratio (e.g., the delta value difference between corresponding values));
input the at least one video feature (Par. [0049], if the MLM(s) 107 detects a static scene, scene change, or high motion scene (i.e. input video feature) in the deployment video data, the MLM(s) 107 can predict the target encoding parameter value Y (i.e. predict update of parameter) based on this detection) and the at least one network metric to a machine learning model to predict updates to the at least one processing parameter and the at least one encoding parameter (Par. [0076] Step B506, one or more embodiments use (e.g., train) a machine learning model (MLM) to use data corresponding to the feedback (i.e. input network metric) to predict a second value of the encoding parameter. At training time or inference time post MLM deployment, such “data” (e.g., a particular quantity of dropped frames) can be used as input to predict that the next or future video data should be encoded at a particular bitrate value (the second value) (i.e. predict update of parameter)); and
process the video stream and encode the processed video stream according to the updates to the at least one processing parameter and the at least one encoding parameter (Par. [0078] Step B508, transmit data to cause the encoder to use the second value of the encoding parameter to convert second video data into second encoded video data).
Regarding Claim 2, Boddeti discloses Claim 1. Boddeti further discloses wherein the at least one processor executes the machine learning model as a reinforcement learning model that predicts the updates to the at least one processing parameter and the at least one encoding parameter (Par. [0004], The video encoder may then use the value of the encoding parameter to encode subsequent video data for the streaming. In some disclosed approaches, such predictions may be based on training the MLM(s) via reinforcement learning based on video encoded by the video encoder), receives a reward based on a performance metric computed from the updates (Par. [0004], A rewards metric(s) (e.g., for maximizing bitrate) (i.e. performance metric) may be used to train the MLM(s)), and updates prediction weights based on the reward (Par. [0032], The server(s) 104 updates the MLM(s) 107 weights per the computed rewards).
Regarding Claim 3, Boddeti discloses Claim 1. Boddeti further discloses wherein the performance metric for computing the reward comprises at least one of video freezing time, latency between a time of capturing the video to a time of displaying the video, or video quality (Par. [0020] The feedback may correspond to an encoded bitrate of the video and/or other information associated with streaming the video over the network. In one or more embodiments, feedback may correspond to one or more characteristics of the encoded video and/or one or more characteristics of the network that are associated with and/or caused by streaming the encoded video. Examples include packet loss, stutters, network bandwidth, frame drops, macroblock count (e.g., intra and/or inter), inter/intra macroblock ratio, packet arrival time, latency, video quality, etc.).
Regarding Claim 4, Boddeti discloses Claim 1. Boddeti further discloses wherein the at least one video feature extracted from the video stream comprises at least one of detail or motion in the video stream (Par. [0049] the machine learning model (MLM(s)) 107 detects a static scene, scene change, or high motion scene (i.e. video feature is motion) in the deployment video data, Par. [0066], a vector representing a portion or attribute (e.g., a video frame or sequence of video frames) of video 108 content).
Regarding Claim 5, Boddeti discloses Claim 1. Boddeti further discloses wherein the at least one network metric comprises at least one of network bandwidth, latency, packet loss, jitter and error rate Par. [0074]-[0075], Step B504, one or more embodiments (e.g., the server(s) 104) receive feedback associated with a receipt, using a network client (e.g., the content server 205 or the sender 105), of a stream comprising the first encoded video data (e.g., receiving a stream of the first encoded video data, where the feedback (i.e. network metric) is based at least on one or more of: a bitrate of the encoded video, a quantity of dropped frames of the encoded video, packet loss, stutter, network bandwidth, macroblock count of the encoded video, packet arrival time, video quality, network latency, and/or an encoded bitrate to target bitrate ratio (e.g., the delta value difference between corresponding values)).
Regarding Claim 6, Boddeti discloses Claim 1. Boddeti further discloses wherein the at least one processing parameter comprises at least one of video resolution, frame rate, or magnification (Par. [0001] Various encoding parameters may correspond to encoding quality, video resolution (i.e. processing parameter), frame rate (i.e. processing parameter), audio and/or image bitrate, encoding mode, pixel aspect ratio, error correction, and more); and wherein the at least one encoding parameter comprises video quantization or encoding rate (Fig. 1, Par. [0004], The video encoder may then use the value of the encoding parameter to encode subsequent video data for the streaming, Par. [0019] the encoding parameter(s) may correspond to a bitrate setting, such as a target bitrate (i.e. encoding rate) for the encoded video, and/or an error correction or channel coding parameter, such as Forward Error Correction (FEC)).
Method Claims 11 – 16 are drawn to the method of using the corresponding apparatus claimed in Claims 1 - 6. Therefore method Claims 11 – 16 correspond to apparatus Claims 1 – 6 and are rejected for the same reasons of anticipation as used above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 7 - 10 and 17 – 20 are rejected under 35 U.S.C. 103 as being unpatentable over Boddeti et al., (US 2023/0048189 A1) in view of Smolyanskiy et al., (US 2022/0138568 A1) referred to as Smolyanskiy hereinafter.
Regarding Claim 7, Boddeti discloses Claim 1. Boddeti further discloses wherein the video camera is a camera (Par. [0125], The computing device 900 may be include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems) that is configured to capture video (Par. [0022], video data that captures one or more of a scene change, a static scene, a low motion scene, or a high motion scene) and create the video stream from the video (Par. [0046], stream video content across a network); and wherein the at least one processor (Par. [0070], a processor executing instructions stored in memory) is further configured to transmit the video stream (Par. [0001], content is being transmitted over digital networks using video streaming technologies) to a wearable device that displays a viewport of the video (Par. [0090] The client device(s) 702 may include a wearable device, a virtual reality system (e.g., a headset, a computer, a game console, remote(s), controller(s), and/or other components), a streaming device (e.g., an NVIDIA SHIELD), and/or another type of device capable of supporting at least display of a game stream of content (i.e. display viewport of video) (e.g., game) sessions 726 and/or inputs to the content sessions 726 from an input device(s) 712).
Boddeti fails to explicitly teach the video camera is a 360° camera.
However, Smolyanskiy teaches the video camera is a 360° camera (Par. [0061], FIGS. 11A-11C, the sensor data 402 may include the data generated by stereo camera(s) 1168, wide-view camera(s) 1170 (e.g., fisheye cameras), infrared camera(s) 1172, surround camera(s) 1174 (e.g., 360 degree cameras), long-range and/or mid-range camera(s) 1198, and/or other sensor types).
References Boddeti and Smolyanskiy are considered to be analogous art because they relate to machine learning model (MLM) trained by reinforcement learning. Therefore, it would be obvious to one possessing ordinary skill in the art before the effective filing date of the claimed invention to specifying a 360° camera as taught by Smolyanskiy in the invention of Boddeti. This modification would allow a set of training data used to train the DNN to be generated using one or more real-world vehicle sensors indicative of movements of the one or more ego-vehicle in the environment over a period of time, where the set of training data is generated using any combination of various types of vehicle sensors (such as cameras, LiDAR sensors, radar sensors, etc.) from various different vehicles in numerous different real-world situations (See Smolyanskiy, Par. [0004]).
Regarding Claim 8, Boddeti in view of Smolyanskiy teaches claim 7. Boddeti further teaches wherein the wearable device is virtual reality (VR) goggles (Par. [0090] The client device(s) 702 may include a wearable device, , a virtual reality system (e.g., a headset (i.e. googles), a streaming device (e.g., an NVIDIA SHIELD), and/or another type of device capable of supporting at least display of a game stream of content).
Regarding Claim 9, Boddeti in view of Smolyanskiy teaches claim 7. Boddeti does not specifically teach a drone. Therefore, Boddeti fails to explicitly teach the video camera is mount to a drone for capturing the 360° video from a perspective of the drone.
However, Smolyanskiy further teaches wherein the video camera is mounted to a drone (Fig. 11A, Par. [0120] The autonomous vehicle 1100 may include a drone and/or another type of vehicle (e.g., that is unmanned)) for capturing the 360° video from a perspective of the drone (Par. [0061], FIGS. 11A-11C, the sensor data 402 may include the data generated by stereo camera(s) 1168, wide-view camera(s) 1170 (e.g., fisheye cameras), infrared camera(s) 1172, surround camera(s) 1174 (e.g., 360 degree cameras) (i.e. mounted to top of vehicle in Fig. 11A), long-range and/or mid-range camera(s) 1198, and/or other sensor types)).
References Boddeti and Smolyanskiy are considered to be analogous art because they relate to machine learning model (MLM) trained by reinforcement learning. Therefore, it would be obvious to one possessing ordinary skill in the art before the effective filing date of the claimed invention to specifying a camera mounted on a drone as taught by Smolyanskiy in the invention of Boddeti. This modification would allow a set of training data used to train the DNN to be generated using one or more real-world vehicle sensors indicative of movements of the one or more ego-vehicle in the environment over a period of time, where the set of training data is generated using any combination of various types of vehicle sensors (such as cameras, LiDAR sensors, radar sensors, etc.) from various different vehicles in numerous different real-world situations (See Smolyanskiy, Par. [0004]).
Regarding Claim 10, Boddeti in view of Smolyanskiy teaches claim 9. Boddeti further teaches wherein the processor is further configured to capture at least one parameter (Par. [0022], video data encoded by the video encoder in the training may be selected in order to account for categories of content that are more likely to result in a mismatch between target bitrate and encoded bitrate. Examples of categories include video data that captures one or more of a scene change, a static scene, a low motion scene, or a high motion scene (i.e. parameter)) and input the at least one parameter (Fig. 3, video data 303) to the machine learning model (Fig. 3, the machine learning model(s) 305) to predict the updates to the at least one processing parameter (Fig. 3, resolution values 307 (i.e. processing parameter))) and the at least one encoding parameter (Fig. 3, bitrate values 307 (i.e. encoding rate)).
Boddeti does not specifically teach a drone. Therefore, Boddeti fails to explicitly teach the processor is further configured to capture at least one drone parameter comprising at least one of velocity, position or altitude of the drone and input the at least one drone parameter to the machine learning model to predict updates.
However, Smolyanskiy further teaches the processor is further configured to capture at least one drone parameter comprising at least one of velocity, position or altitude of the drone (Par. [0068] The inputs 408 may include past location(s) 410 (i.e. position) (e.g., of actors in the environment, such as vehicles, pedestrians, bicyclists, robots, drones, watercraft, etc., depending on the implementation), state information 432 (e.g., velocity and/or acceleration data corresponding to the actors)) and input the at least one drone parameter to the machine learning model (Par. [0041] the prediction machine learning model MLM 104 may receive as input one or more locations and/or other attributes (e.g., velocity, acceleration, trajectory, etc. for one or more actors encoded in the simulation data 102)) to predict updates (Par. [0036] The training engine 112 may use the decoded data to evaluate the performance of the policy MLM 106 and/or the value function MLM 108 and update one or more parameters).
References Boddeti and Smolyanskiy are considered to be analogous art because they relate to machine learning model (MLM) trained by reinforcement learning. Therefore, it would be obvious to one possessing ordinary skill in the art before the effective filing date of the claimed invention to specifying a camera mounted on a drone as taught by Smolyanskiy in the invention of Boddeti. This modification would allow a set of training data used to train the DNN to be generated using one or more real-world vehicle sensors indicative of movements of the one or more ego-vehicle in the environment over a period of time, where the set of training data is generated using any combination of various types of vehicle sensors (such as cameras, LiDAR sensors, radar sensors, etc.) from various different vehicles in numerous different real-world situations (See Smolyanskiy, Par. [0004]).
Method Claims 17 – 20 are drawn to the method of using the corresponding apparatus claimed in Claims 7 - 10. Therefore method Claims 17 – 20 correspond to apparatus Claims 7 – 10 and are rejected for the same reasons of obviousness as used above.
Conclusion
The prior art references made of record are not relied upon but are considered pertinent to applicant's disclosure. Kranski et al. (US 2022/0314434 A1) teaches hybrid computing architectures using specialized processors to handle encoding or decoding of latent representations used to control dynamic mechanical systems.
Any inquiry concerning this communication should be directed to SUSAN E HODGES whose telephone number is (571)270-0498. The Examiner can normally be reached on Monday - Friday from 8:00 am (EST) to 4:00 pm (EST).
If attempts to reach the Examiner by telephone are unsuccessful, the Examiner's supervisor, Brian T. Pendleton, can be reached on (571) 272-7527. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://portal.uspto.gov/external/portal. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/Susan E. Hodges/Primary Examiner, Art Unit 2425