DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of claims 1-14 in the reply filed on 10/29/2025 is acknowledged.
Status of the Application
Claims 1-14 have been elected. Claims 15-20 have not been elected. Claims 1-14 are currently pending in this application.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 08/09/2024 was filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 2, 5, 9, 10, and 13 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1 and 9 discloses, “predicting, using a control network implemented on the edge device, optimal codec parameters based on dynamic network conditions and content of the video clip.” The specification states, “In various embodiments, in block 1002, raw video frames can be captured on an edge device, which also determines the maximum network bandwidth. This step sets the foundation for adaptive video compression by assessing both the video content and network capacity. Block 1004 involves predicting optimal codec parameters using a control network. The prediction can leverage dynamic network conditions and the content of the video clip, aiming to optimize compression without losing critical data for analysis. A control network predicts optimal codec parameters based on the video content and dynamic network conditions.” [0131]. One of ordinary skill in the art would not understand what the term “dynamic network conditions” is actually referencing in regards to the claim limitations.
Claims 1 and 9 disclose that the term, “optimal codec parameters” but there is a lack of criteria claimed for the “optimal” term. Throughout the specification, the term “optimal codec parameters” is disclosed without criteria for the codec parameters to be considered optimal.
Claims 1, 2, 9, and 10 use the terms, “optimal codec parameters”, “predicted codec parameters”, “the encoding parameters” interchangeably. One of ordinary skill in the art would not understand if they are all referencing the same parameters or different parameters.
Claims 1, 5, 9, and 13 use the term “a server-side vision model” and claims 1 and 9 use the term “a deep vision model”. The specification states, “feedback from the server-side deep learning-based vision model” [0027]. One of ordinary skill would not understand from the claim limitations if the models are the same or different because they are using different terms.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, 4, 5, 9, 10, 12, and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (Hereafter, “Zhang”) [“Understanding the Potential of Server-Driven Edge Video Analytics”] in view of NAMGOONG et al. (Hereafter, “Namgoong”) [WO 2024/031502 A1] in further view of Said et al. (Hereafter, “Said”) [“Differentiable bit-rate estimation for neural-based video codec enhancement”].
In regards to claim 1, Zhang discloses a system for optimizing video compression using end-to-end learning ([Abstract] Server-driven edge video analytics), comprising: one or more processor devices operatively coupled to a computer-readable storage medium, the processor devices being configured for: capturing, using an edge device ([Fig. 1] edge video sensor), raw video frames from a video clip ([Section 1] camera captures video frames) and determining maximum network bandwidth ([Section 1] the presented camera-to-server bandwidth constraints); predicting, using a control network implemented on the edge device, optimal codec parameters based on dynamic network conditions ([Section 1] the presented camera-to-server bandwidth constraints) and content of the video clip ([Abstract] low-quality-encoded stream sent to the server [Section 2] The quality of the video on which the server-side DNN extracts the feedback. This is measured either by quantization parameter (QP) in video codecs like H.264 [25] or by resolution. In our experiments, we will use QP.); encoding, ([Section 1] the camera first sends video frames with regions that do not contain detected objects in low quality to the server-side DNN) and propagating gradients from a server-side vision model to adjust the codec parameters ([Section 3.3] To compute the original saliency values, we run the backward propagation operation from the DNN output (sum of bounding box confidence scores) to the input uncompressed frame, which produces the per-pixel saliency (gradients with regard to the pixel RGB values). We use QP of 2 (a high quality) to encode these macroblocks and a varying low quality for the remaining macroblocks.); decoding, using a server, the video clip and analyzing the video clip with a deep vision model located on the server ([Section 1 and Fig. 1] server-side DNN receives the low-quality frames for input into the DNN [Section 3] In an ideal server-driven video analytics system, the server-side DNN extracts some feedback that indicates the minimum regions in each frame that must be encoded in high quality in order for the DNN to maximize accuracy); transmitting, using a feedback mechanism, analysis from the deep vision model back to the control network to facilitate end-to-end training of the system ([Section 1] based on the DNN output on these low-quality frames, the server sends the client a feedback that indicates which regions or frames are important (e.g., might contain new objects of interests)); and adjusting the encoding parameters based on the analysis from the deep vision model received from the feedback mechanism ([Section 1] upon receiving this feedback, the camera will resend the current frames or send future frames with only these important regions encoded in high quality).
Namgoong discloses a system for optimizing video compression using end-to-end learning ([0131] the neural networks (NNs) for encoders and decoders, via an end-to-end learning process), comprising: one or more processor devices operatively coupled to a computer-readable storage medium ([0182] FIG. 14 is a block diagram illustrating an example of a hardware implementation for a server 1400 employing a processing system 1414. In some examples, the server 1400 may be a device configured to communicate with one or more of the UEs or scheduled entities as discussed in any one or more of FIGs. 1 -13.), the processor devices being configured for: ([0152] At 1208, the second server 1204 generates a ground truth for the encoder and decoder training. For example, the second server 1204 may determine an expected decoder output based on channel state information that the second server 1204 receives from a set of UEs that are deployed by the UE vendor that operates the second server 1204. Also at 1208, the second server 1204 may select one quantization scheme from the set of quantization schemes to use for the encoder and decoder training.); encoding, using a differentiable surrogate model of a video codec ([0109] In some aspects, the machine learning process 600 involves training autoencoders with discrete latent variables where quantization is based on a shared embedding space e (e.g., codebook 610).), the video clip using the predicted codec parameters ([0154] At 1212, the second server 1204 may conduct a forward pass operation for its encoder NN by encoding a known data set. In addition, the second server 1204 may use the selected quantization scheme to quantize the output of encoder NN.) and propagating gradients from a server-side vision model to adjust the codec parameters ([0160] At 1224, the second server 1204 backward propagates gradients through the layers of the encoder NN. For example, the second server 1204 may apply the gradient received at 1222, and the gradients for the unquantized encoder output calculated based on the first quantization loss to calculate the gradients for the last layer of the encoder NN. This, in turn, may allow a gradient to be calculated for the second to last layer of the encoder NN. This process continues layer-by-layer until a gradient is calculated for the first layer of the encoder NN. The backward propagation is also applied to the codewords in the codebook based on the second quantization loss (e.g., as discussed above in conjunction with FIG. 6). Finally, the parameters of the encoder NN, the parameters of the decoder NN, and the codewords in the codebook are updated once, using all the gradients calculated from the backpropagation.); decoding, using a server, the video clip and analyzing the video clip with a deep vision model located on the server ([0156] At 1216, the first server 1202 may conduct a forward pass operation for its decoder NN by decoding the encoder output received from the second server 1204 at 1214.); transmitting, using a feedback mechanism, analysis from the deep vision model back to the control network to facilitate end-to-end training of the system ([0157] At 1218, the first server 1202 calculates a loss function based on the ground truth received at 1210 and the output of the last layer of the decoder NN. In some examples, the loss function is indicative of the error in a reconstructed signal (e.g., a reconstructed CSI) output by the decoder NN relative to the ground truth. In some examples, the loss function may be a mean square error function. Other forms of loss functions may be used in other examples. [0158] At 1220, the first server 1202 backward propagates gradients through the layers of the decoder NN. For example, the first server 1202 may calculate a first gradient based on the loss function for the last layer of the decoder NN. This, in turn, may allow a gradient to be calculated for the second to last layer of the decoder NN. This process continues layer-by-layer until a gradient is calculated for the first layer of the decoder NN. [0159] At 1222, the first server 1202 transmits the gradient for the first layer of the decoder NN to the second server 1204.); and adjusting the encoding parameters based on the analysis from the deep vision model received from the feedback mechanism ([0161] This completes one iteration of the encoder decoder learning whereby the parameters for all layers of the encoder NN and the decoder NN and the codewords in the codebook have been updated one time. [0164] At 1230, the second server 1204 updates the encoders of its associated UEs based on the trained encoder NN, the updated codebook, and the selected quantization scheme. For example, the second server 1204 may send a message to each UE indicating that the UE is to use a particular set of encoder parameters, a particular codebook, and a particular type of quantization for encoding operations when communicating with a network entity that is deployed by a network entity vendor that operates the first server 1202.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhang with the teachings of Namgoong in order to improve the performance of the system [See Namgoong].
Said discloses an end-to-end optimization, i.e., NN training that takes into account codec parameters and performance, wherein a differentiable codec proxy is used between neural pre-processing and neural post-processing that can accurately estimate performance factors and corresponding derivatives, as shown in Fig. 1(b), enabling NN gradient back propagation [See Said, Fig. 1 and Section I].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhang and Namgoong with the use of a differentiable codec proxy as taught by Said in order to enable neural network gradient back propagation for improving the accuracy of the performance factors and corresponding derivatives [See Said].
In regards to claim 2, the limitations of claim 1 have been addressed. Zhang discloses wherein the processor is further configured for dynamically assessing, using the control network, network bandwidth availability in real-time and adjusting the predicted codec parameters accordingly to optimize for both video quality and transmission efficiency ([Abstract] In this server-driven approach, an ideal feedback should (1) be derived from minimum information from the video sensor (2) incur minimum bandwidth usage to obtain (3) indicate the optimal video streaming/encoding scheme (e.g., the minimum frames/regions that require high encoding quality). [Section 1] maximize inference accuracy under the presented camera-to-server bandwidth constraints).
In regards to claim 4, the limitations of claim 1 have been addressed. Zhang discloses wherein the encoding further comprises application of a macroblock-wise quantization scheme ([Section 1] encoding macroblocks in low quality), ([Section 1] upon receiving this feedback, the camera will resend the current frames or send future frames with only these important regions encoded in high quality [Section 3.3] Given the original saliency per pixel, we select the 10% (16x16) macroblocks in the manner described in §3.2. We use QP of 2 (a high quality) to encode these macroblocks and a varying low quality for the remaining macroblocks. The encoding is done with H.264 [25]. Finally, we send the encoded video frame to the server-side DNN for inference.).
Namgoong discloses wherein the encoding further comprises application of a macroblock-wise quantization scheme, directed by the differentiable surrogate model ([0109] In some aspects, the machine learning process 600 involves training autoencoders with discrete latent variables where quantization is based on a shared embedding space e (e.g., codebook 610).), to selectively adjust compression levels across the video frame based on determined content complexity and importance ([0154] At 1212, the second server 1204 may conduct a forward pass operation for its encoder NN by encoding a known data set. In addition, the second server 1204 may use the selected quantization scheme to quantize the output of encoder NN. [0161] This completes one iteration of the encoder decoder learning whereby the parameters for all layers of the encoder NN and the decoder NN and the codewords in the codebook have been updated one time. [0164] At 1230, the second server 1204 updates the encoders of its associated UEs based on the trained encoder NN, the updated codebook, and the selected quantization scheme. For example, the second server 1204 may send a message to each UE indicating that the UE is to use a particular set of encoder parameters, a particular codebook, and a particular type of quantization for encoding operations when communicating with a network entity that is deployed by a network entity vendor that operates the first server 1202.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhang with the teachings of Namgoong in order to improve the performance of the system [See Namgoong].
Said discloses an end-to-end optimization, i.e., NN training that takes into account codec parameters and performance, wherein a differentiable codec proxy is used between neural pre-processing and neural post-processing that can accurately estimate performance factors and corresponding derivatives, as shown in Fig. 1(b), enabling NN gradient back propagation [See Said, Fig. 1 and Section I].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhang and Namgoong with the use of a differentiable codec proxy as taught by Said in order to enable neural network gradient back propagation for improving the accuracy of the performance factors and corresponding derivatives [See Said].
In regards to claim 5, the limitations of claim 1 have been addressed. Zhang fails to explicitly disclose wherein the feedback mechanism incorporates a loss function specifically designed to measure a discrepancy between original video frames and the decoded video frames as perceived by the server-side deep vision model, thereby guiding the adjustment of encoding parameters
Namgoong discloses wherein the feedback mechanism incorporates a loss function specifically designed to measure a discrepancy between original video frames and the decoded video frames as perceived by the server-side deep vision model, thereby guiding the adjustment of encoding parameters ([0157] At 1218, the first server 1202 calculates a loss function based on the ground truth received at 1210 and the output of the last layer of the decoder NN. In some examples, the loss function is indicative of the error in a reconstructed signal (e.g., a reconstructed CSI) output by the decoder NN relative to the ground truth. In some examples, the loss function may be a mean square error function. Other forms of loss functions may be used in other examples. [0158] At 1220, the first server 1202 backward propagates gradients through the layers of the decoder NN. For example, the first server 1202 may calculate a first gradient based on the loss function for the last layer of the decoder NN. This, in turn, may allow a gradient to be calculated for the second to last layer of the decoder NN. This process continues layer-by-layer until a gradient is calculated for the first layer of the decoder NN. [0159] At 1222, the first server 1202 transmits the gradient for the first layer of the decoder NN to the second server 1204. [0161] This completes one iteration of the encoder decoder learning whereby the parameters for all layers of the encoder NN and the decoder NN and the codewords in the codebook have been updated one time. [0164] At 1230, the second server 1204 updates the encoders of its associated UEs based on the trained encoder NN, the updated codebook, and the selected quantization scheme. For example, the second server 1204 may send a message to each UE indicating that the UE is to use a particular set of encoder parameters, a particular codebook, and a particular type of quantization for encoding operations when communicating with a network entity that is deployed by a network entity vendor that operates the first server 1202.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhang with the loss function and updating of encoding parameters of Namgoong in order to improve the performance of the system [See Namgoong].
Claim 9 lists all the same elements of claim 1, but in method form rather than system form. Therefore, the supporting rationale of the rejection to claim 1 applies equally as well to claim 9.
Claim 10 lists all the same elements of claim 2, but in method form rather than system form. Therefore, the supporting rationale of the rejection to claim 2 applies equally as well to claim 10.
Claim 12 lists all the same elements of claim 4, but in method form rather than system form. Therefore, the supporting rationale of the rejection to claim 4 applies equally as well to claim 12.
Claim 13 lists all the same elements of claim 5, but in method form rather than system form. Therefore, the supporting rationale of the rejection to claim 5 applies equally as well to claim 13.
Claim(s) 3, 6, 8, 11, and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Namgoong in further view of Said in even further view of ZHANG et al. (Hereafter, “Zhang907”) [US 2022/0385907 A1].
In regards to claim 3, the limitations of claim 1 have been addressed. Zhang fails to explicitly disclose wherein the differentiable surrogate model of the video codec further comprises a 3D convolutional neural network architecture for accurately modeling behavior of the codec and enabling propagation of gradients for the end-to-end training.
Zhang907 discloses wherein the differentiable surrogate model of the video codec further comprises a 3D convolutional neural network architecture ([0158] 3D convolution architectures) for accurately modeling behavior of the codec and enabling propagation of gradients for the end-to-end training ([0050] These multi-layered architectures may be trained one layer at a time and may be fine-tuned using back propagation.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhang with the use of 3D convolution architecture as taught by Zhang907 in order to improve compression performance [See Zhang907].
In regards to claim 6, the limitations of claim 3 have been addressed. Zhang fails to explicitly disclose wherein the 3D convolutional neural network architecture of the differentiable surrogate model includes layers configured for feature extraction, quantization parameter prediction, and compression artifact reduction.
Zhang907 discloses wherein the 3D convolutional neural network architecture of the differentiable surrogate model ([0158] 3D convolution architectures) includes layers configured for feature extraction ([0067] The DCN 200 may include a feature extraction section and a classification section. Upon receiving the image 226, a convolutional layer 232 may apply convolutional kernels (not shown) to the image 226 to generate a first set of feature maps 218.), quantization parameter prediction ([0070] The weights of the DCN 200 may then be adjusted so the output 222 of the DCN 200 is more closely aligned with the target output.), and compression artifact reduction ([0085] For certain compression methods (e.g., JPEG, BPG, among others), the distortion-based artifacts can take the form of blocking or other artifacts. In some cases, neural network based compression can be used and can result in high quality compression of image data and video data. In some cases, blurring and color shift are examples of artifacts.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhang with the use of 3D convolution architecture as taught by Zhang907 in order to improve compression performance [See Zhang907].
In regards to claim 8, the limitations of claim 6 have been addressed. Zhang fails to explicitly disclose wherein the differentiable surrogate model further comprises a multi-layer perceptron (MLP) for a final prediction of quantization parameters (QP), leveraging both spatial and temporal video features extracted by preceding 3D convolutional layers to optimize encoding for subsequent video frames.
Zhang907 discloses wherein the differentiable surrogate model further comprises a multi-layer perceptron (MLP) for a final prediction of quantization parameters (QP), leveraging both spatial and temporal video features extracted by preceding 3D convolutional layers to optimize encoding for subsequent video frames ([0049] In MLP neural networks, data may be fed into an input layer, and one or more hidden layers provide levels of abstraction to the data. Predictions may then be made on an output layer based on the abstracted data. MLPs may be particularly suitable for classification prediction problems where inputs are assigned a class or label. [0175] In some aspects, an MLP-based implicit neural model can be viewed as a convolution operation with 1×1 kernels. In some examples, the techniques described herein can generalize implicit models to convolutional architecture. [0182] In some aspects, implicit video representation can be implemented using a 3D MLP. For example, the MLP representation can easily be extended to video data by adding a third input that represents the frame number (or time component) t. In some examples, a SIREN architecture can be used with sine activations. [0183] In some cases, implicit video representation can be implemented using 3D convolutional networks. As previously noted, 3D MLP can be seen as a 1×1×1 convolution operation. Similar to the 2 dimensional case, the present technology can implement 3D MLPs into convolutional operations with 3 dimensional kernels. To keep number of parameters to a minimum, the present technology can use spatial kernels of size k×k×1, followed by frame-wise kernels of shape 1×1× k′.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhang with the use of 3D MLP as taught by Zhang907 in order to improve compression performance [See Zhang907].
Claim 11 lists all the same elements of claim 3, but in method form rather than system form. Therefore, the supporting rationale of the rejection to claim 3 applies equally as well to claim 11.
Claim 14 lists all the same elements of claim 8, but in method form rather than system form. Therefore, the supporting rationale of the rejection to claim 8 applies equally as well to claim 14.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Namgoong in further view of Said in even further view of Naphade et al. (Hereafter, “Naphade”) [US 2019/0294869 A1].
In regards to claim 7, the limitations of claim 1 have been addressed. Zhang discloses further comprising one or more cameras installed at traffic ([Section 3.3] traffic cameras) ([Section 1] video sensors send live video feeds to a server for compute-intensive DNN inference in real-time), the processor being configured to adjust video compression in real-time ([Section 1] video sensors send live video feeds to a server for compute-intensive DNN inference in real-time [Section 1] upon receiving this feedback, the camera will resend the current frames or send future frames with only these important regions encoded in high quality).
Naphade discloses further comprising one or more cameras installed at traffic intersections and pedestrian crossings and configured for capturing and transmitting real-time traffic footage, the processor being configured to adjust video compression in real-time responsive to traffic density and movement patterns detected in the real-time traffic footage ([0003] provide real-time insights pertaining to object motion (e.g., traffic flow), such as anomalies and various trajectory features (e.g., traffic flow, speed, etc.) [0017] Analyzing traffic at intersections and along roadways is vital to managing traffic flow, minimizing congestion and GHG emissions, maximizing the safety of pedestrians and vehicles, and responding to emergency situations. [0032] The LSTM network may be trained using an optimization algorithm—such as gradient descent—combined with backpropagation through time to compute the gradients needed during the optimization process in order to change each weight of the LSTM network in proportion to the derivative of the error (at the output layer of the LSTM network) with respect to the corresponding weight.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhang with optimization of the real-time traffic video as taught by Naphade in order to provide real-time insights pertaining to object motion (e.g., traffic flow), such as anomalies and various trajectory features (e.g., traffic flow, speed, etc.) [See Naphade].
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Kaitlin A Retallick whose telephone number is (571)270-3841. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chris Kelley can be reached at (571) 272-7331. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KAITLIN A RETALLICK/Primary Examiner, Art Unit 2482