DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Application
Claims 1-20 are currently pending in this application.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/10/2025 was filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
The information disclosure statement (IDS) submitted on 07/18/2025 was filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, 6, 14, 19, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ma et al. (Hereafter, “Ma”) [US 2025/0119560 A1] in view of Menon et al. (Hereafter, “Menon”) [US 2024/0121400 A1].
In regards to claim 1, Ma discloses a system for encoding video data ([0008] a device for processing video data), comprising: a memory device ([0008] a memory configured to store video data); and one or more processor devices coupled to the memory device ([0008] a processing system implemented in circuitry) and configured to: determine a temporal scaling factor ([0031] temporal downsampling ratio) ([0031] spatial downsampling ratio) ([0016] Neural networks may be designed to perform frame level spatial resampling which may include downsampling a frame spatially to get a lower spatial resolution at the encoder side and temporal resampling which may include dropping a certain number of frames (e.g., drop three out of every four frames) at the encoder side. [0031] The configuration parameters may include, for example, a quantization parameter, a spatial downsampling ratio, and/or a temporal downsampling ratio.); encode the temporally and spatially down-scaled video data as a neural network having a plurality of weight parameters ([0031] Video encoder 28 may further encode data representing updates to one or more of the neural networks. The updates may include a change to one or more of the weights of the neural network. [0059] In general, encoder 102 may signal to decoder 104 a first data structure that indicates an update to a second data structure. The second data structure may carry a neural network configuration associated with a compressed video (or image) bitstream. The compressed video bitstream may be for a machine learning task, e.g., video coding for machines (VCM), including object detection, object tracking, instance segmentation, and the like. [0060] The second data structure may include any or all of the following information: a type or purpose of the neural network, data defining a neural network structure, weights and biases of the neural network, and/or an identifier, such as a sequence number.); and generate a bit stream of encoded video data based on the plurality of weight parameters ([0032] Video encoder 28 may signal the data representing the neural networks and/or updates to the neural networks in high layer syntax (HLS) video data, such as in a sequence parameter set (SPS), a picture parameter set (PPS), or an adaptation parameter set (APS), or in a supplemental enhancement layer (SEI) message. [0033] Encapsulation unit 30 receives PES packets for elementary streams of a media presentation from audio encoder 26 and video encoder 28 and forms corresponding network abstraction layer (NAL) units from the PES packets. [0057] Per the techniques of this disclosure, encoder 102 may signal data representing the neural network(s) and/or configuration data in a media bitstream, in packet headers of packets of the media bitstream, or the like for reception by decoder 104.).
Menon discloses a system for encoding video data ([Abstract] encoding), comprising: a memory device ([0032] Computing device 201, which in some examples may be included in mobile device 201 and in other examples may be included in server 108, also may include a memory 202. Memory 202 may comprise a storage system configured to store a database 214 and an application 216.); and one or more processor devices coupled to the memory device ([0032] Application 216 (e.g., per) may include instructions which, when executed by a processor 204, cause computing device 201 to perform various steps and/or functions (e.g., implementing an SR-based ABR algorithm), as described herein.) and configured to: determine a temporal scaling factor based on a measure of temporal variability of the video data ([0005] for at least one segment of the plurality of segments of the video input, extract a spatial feature and a temporal feature; predict a bitrate-resolution pair based on the spatial feature and the temporal feature, using a discrete cosine transform (DCT)-based energy function); determine a spatial scaling factor based on a measure of spatial variability of the video data ([0005] for at least one segment of the plurality of segments of the video input, extract a spatial feature and a temporal feature; predict a bitrate-resolution pair based on the spatial feature and the temporal feature, using a discrete cosine transform (DCT)-based energy function); generate temporally and spatially ([0005] Per-title encode the at least one segment for the predicted bitrate-resolution pair. In some examples, the memory is further configured to store, and the one or more processors are further configured to implement, a machine learning module configured to perform the prediction of the bitrate-resolution pair. In some examples, the machine learning module comprises a neural network);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma with the teachings of Menon in order to improve the quality and performance of the system [See Menon].
In regards to claim 2, the limitations of claim 1 have been addressed. Ma fails to explicitly disclose wherein to determine the temporal and spatial scaling factors, the one or more processor devices are further configured to input the measures of the temporal and spatial variabilities of the video data to a first neural network comprising a first plurality of convolutional neural network layers to generate the temporal and spatial scaling factors.
Menon discloses wherein to determine the temporal and spatial scaling factors, the one or more processor devices are further configured to input the measures of the temporal and spatial variabilities of the video data to a first neural network comprising a first plurality of convolutional neural network layers to generate the temporal and spatial scaling factors ([0007] A method for predicting video encoding complexity may include: performing video complexity feature extraction on a video segment, wherein a plurality of low-complexity frame-based features are extracted; predicting video encoding complexity for the video segment using the plurality of low-complexity frame-based features, a predicted video encoding complexity comprising an encoding bitrate and an encoding time; and outputting a predicted encoding bitrate and a predicted encoding time. In some examples, predicting video encoding complexity comprises implementing a hybrid model using a convolutional neural network (CNN).).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma with the teachings of Menon in order to improve the quality and performance of the system [See Menon].
In regards to claim 6, the limitations of claim 1 have been addressed. Ma discloses wherein the one or more processor devices are further configured to transmit the temporal and spatial scaling factors and the bit stream of the encoded video data to a video decoder ([0017] The encoder may signal data to the decoder indicating an initial set of one or more neural networks, and then subsequently send the decoder updates representing different neural networks or different configurations or values for the neural networks. [0031] Video encoder 28 may further encode data representing updates to one or more of the neural networks. The updates may include a purpose or type for the neural network to be updated, an identifier for the neural network (e.g., an updated sequence number corresponding to the current update), a change to the structures of the neural network, a change to one or more of the weights of the neural network, a change to one or more of the bias values of the neural network, a new value for a configuration parameter, or a change to the configuration parameters (e.g., an additional configuration parameter or removal of a configuration parameter). The configuration parameters may include, for example, a quantization parameter, a spatial downsampling ratio, and/or a temporal downsampling ratio. [0057] Per the techniques of this disclosure, encoder 102 may signal data representing the neural network(s) and/or configuration data in a media bitstream, in packet headers of packets of the media bitstream, or the like for reception by decoder 104.).
In regards to claim 14, Ma discloses a system for decoding video data ([0043-0044] video decoder 48), comprising: a memory device ([0008] a memory configured to store video data); and one or more processor devices coupled to the memory device ([0008] a processing system implemented in circuitry) and configured to: receive a temporal scaling factor, a spatial scaling factor, and a plurality of weight parameters of down-scaled video data ([0046] According to the techniques of this disclosure, video decoder 48 may also receive updates to one or more of the neural networks. Such updates may indicate whether to use an alternative neural network for a particular task, to skip processing by a neural network for a particular task, to drop one or more inner droppable structures from a neural network, or other changes to the neural network, e.g., changes to bias values, weight values, configuration parameters, or the like. [0031] The configuration parameters may include, for example, a quantization parameter, a spatial downsampling ratio, and/or a temporal downsampling ratio.), factors ([0016] Neural networks may be designed to perform frame level spatial resampling which may include downsampling a frame spatially to get a lower spatial resolution at the encoder side and temporal resampling which may include dropping a certain number of frames (e.g., drop three out of every four frames) at the encoder side.); generate predicted video data corresponding to the down-scaled video data using the plurality of weight parameters ([0076] Decoder 104 may then update the determined neural network (256). Thus, after receiving new video data (258), decoder 104 may provide the video data to the neural networks (260), including the updated neural network(s), and receive processed (e.g., decoded and/or object detection data) data based on the video data (262).); and generate, using a post-processing neural network, decoded video data based on the predicted video data, the temporal scaling factor, and the spatial scaling factor ([0078] providing video data from the video bitstream to the updated neural network to cause the updated neural network to process the video data).
Menon discloses a system for decoding video data, comprising: a memory device ([0032] Computing device 201, which in some examples may be included in mobile device 201 and in other examples may be included in server 108, also may include a memory 202. Memory 202 may comprise a storage system configured to store a database 214 and an application 216.); and one or more processor devices coupled to the memory device ([0032] Application 216 (e.g., per) may include instructions which, when executed by a processor 204, cause computing device 201 to perform various steps and/or functions (e.g., implementing an SR-based ABR algorithm), as described herein.) and configured to: receive a temporal scaling factor, a spatial scaling factor, and a plurality of weight parameters of ,wherein the temporal scaling factor and the spatial scaling factor are based on a temporal variability and a spatial variability of an original version of the ([0005] extract a spatial feature and a temporal feature; predict a bitrate-resolution pair based on the spatial feature and the temporal feature, using a discrete cosine transform (DCT)-based energy function); generate predicted video data corresponding to the down-scaled video data using the plurality of weight parameters; and generate, using a post-processing neural network, decoded video data based on the predicted video data, the temporal scaling factor, and the spatial scaling factor ([0005] Per-title encode the at least one segment for the predicted bitrate-resolution pair. In some examples, the memory is further configured to store, and the one or more processors are further configured to implement, a machine learning module configured to perform the prediction of the bitrate-resolution pair. In some examples, the machine learning module comprises a neural network).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma with the teachings of Menon in order to improve the quality and performance of the system [See Menon].
Claims 19 and 20 are substantially the same as claims 1 and 14 and are thus rejected for reasons similar to those in rejecting claims 1 and 14.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ma in view of Menon in further view of New et al. (Hereafter, “New”) [US 2011/0026593 A1].
In regards to claim 3, the limitations of claim 2 have been addressed. Ma fails to explicitly disclose wherein to generate the temporally and spatially down-scaled video data, the one or more processor devices are further configured to: input the video data to a second neural network comprising a second plurality of convolutional neural network layers to generate a plurality of high-frequency components; add the plurality of high-frequency components to the video data to generate modified video data; and generate, using a space-time scaling filter, the temporally and spatially down- scaled video data by temporally and spatially down-scaling the modified video data based on the temporal and spatial scaling factors.
New discloses wherein to generate the temporally and spatially down-scaled video data, the one or more processor devices are further configured to: input the video data to a second neural network comprising a second plurality of convolutional neural network layers to generate a plurality of high-frequency components; add the plurality of high-frequency components to the video data to generate modified video data; and generate, using a space-time scaling filter, the temporally and spatially down- scaled video data by temporally and spatially down-scaling the modified video data based on the temporal and spatial scaling factors ([0041] Furthermore, the storing unit may be configured to replace, with the embedded data, a value indicated by one or more bits including at least an LSB (Least Significant Bit) in the data indicating the pixel value of the down-sampled image. [0042] Replacing LSBs with the embedded data in this way makes it possible to minimize errors in the pixel value of the down-sampled image. [0043] Furthermore, the storing unit may further include a coding unit configured to generate the embedded data by performing variable length coding on the high frequency components that are deleted by the deleting unit, and the restoring unit may be configured to restore the high frequency components from the embedded data by performing variable length decoding on the embedded data. [0044] Performing variable length coding on the high frequency components in this way makes it possible to reduce the data amount of the embedded data. As a result, it is possible to minimize errors resulting from replacement with the embedded data in the pixel values of the reference image (down-sampled image).).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of New in order to minimize errors in the reduction of the data amount of the embedded data [See New].
Claim(s) 4 and 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ma in view of Menon in further view of New in even further view of Galpin et al. (Hereafter, “Galpin”) [US 2023/0298219 A1].
In regards to claim 4, the limitations of claim 3 have been addressed. Ma fails to explicitly disclose wherein the one or more processor devices are configured to train the first neural network, the second neural network, and a post-processing neural network of a decoder to reduce a difference measure between the inputted video data to the first and second neural networks and a corresponding decoded video data.
Galpin discloses wherein the one or more processor devices are configured to train the first neural network, the second neural network, and a post-processing neural network of a decoder to reduce a difference measure between the inputted video data to the first and second neural networks and a corresponding decoded video data ([0033] DNNs are trained using several types of losses: “objective metric” and “subjective” metric. Loss based on an “objective” metric may be typically Mean Squared Error (MSE) or based on structural similarity (SSIM) for instance. The results may not be perceptually as good as the “subjective metric”, but the fidelity to the original signal (image) is higher. Loss based on “subjective” (or subjective by proxy) may be typically using Generative Adversarial Networks (GANs) during the training stage or advanced visual metric via a proxy Neural Network (NN). Depending on the loss used for training, the resulting parameters of the DNN model may be different. [0040] In a same way, a generic training set ensures that compression performance is consistent on a wide range of content, but a specific training set could reach better performances for specific applications. Additionally, auto-encoder solutions may be trained at given rate-points, i.e. the weights of the models are optimized for a specific range of bitrates of the transmitted bitstream. [0044] A training configuration is defined by a metric used in the loss function, and a training set of samples or batch which are input to the auto-encoder so that the auto-encoder learns its parameters. The other training configuration could differ from the first training configuration from the metric which could be an objective or perceptual/subjective quality metric and/or the training set which could be a generic training set or a training set with specific contents. The training configurations could also differ in the Lagrange parameters for updating or refining in a light way a DNN to adapt to different bitrate levels. [0072] The decoded picture can further go through post-decoding processing (385), for example, an inverse color transform (e.g. conversion from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping performing the inverse of the remapping process performed in the pre-encoding processing (201). The post-decoding processing can use metadata derived in the pre-encoding processing and signaled in the bitstream.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of Galpin in order to improve the performance of the system [See Galpin].
In regards to claim 5, the limitations of claim 3 have been addressed. Ma fails to explicitly disclose wherein the one or more processor devices are configured to not transmit trained weights corresponding to the first and the second plurality of convolutional neural network layers to a video decoder.
Galpin discloses wherein the one or more processor devices are configured to not transmit trained weights corresponding to the first and the second plurality of convolutional neural network layers to a video decoder ([0100] Also, in any one of the embodiments described here, only the decoder part could be retrained in the second training configuration or both the encoder part and the decoder part of the auto-encoder can be jointly retrained in the second training configuration.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of Galpin in order to improve the performance of the system [See Galpin].
Claim(s) 7, 8, 17, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ma in view of Menon in further view of “Learning Spatio-Temporal Downsampling for Effective Video Upscaling” by XIANG, X., et al. (Hereafter, “Xiang”).
In regards to claim 7, the limitations of claim 1 have been addressed. Ma fails to explicitly disclose wherein the temporal scaling factor has a value between zero and one and is proportional to the measure of temporal variability.
Xiang discloses wherein the temporal scaling factor has a value between zero and one and is proportional to the measure of temporal variability ([Page 3] Unliked these methods, we aim to freely resize the space-time volume with arbitrary scale ratios in this work. [Page 4] To ensure that the learned filters are low-pass, we add a softmax layer to regularize the weight values within the range [0, 1] and the sum to be 1. We then use striding to produce our desired downsampled frames in both space and time.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of Xiang in order to improve the spatio-temporal downsampling in the processing operations.
In regards to claim 8, the limitations of claim 1 have been addressed. Ma fails to explicitly disclose wherein the spatial scaling factor has a value between zero and one and is proportional to the measure of spatial variability.
Xiang discloses wherein the spatial scaling factor has a value between zero and one and is proportional to the measure of spatial variability ([Page 3] Unliked these methods, we aim to freely resize the space-time volume with arbitrary scale ratios in this work. [Page 4] To ensure that the learned filters are low-pass, we add a softmax layer to regularize the weight values within the range [0, 1] and the sum to be 1. We then use striding to produce our desired downsampled frames in both space and time.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of Xiang in order to improve the spatio-temporal downsampling in the processing operations.
In regards to claim 17, the limitations of claim 14 have been addressed. Ma fails to explicitly disclose wherein the temporal scaling factor has a value between zero and one and is proportional to the temporal variability.
Xiang discloses wherein the temporal scaling factor has a value between zero and one and is proportional to the temporal variability ([Page 3] Unliked these methods, we aim to freely resize the space-time volume with arbitrary scale ratios in this work. [Page 4] To ensure that the learned filters are low-pass, we add a softmax layer to regularize the weight values within the range [0, 1] and the sum to be 1. We then use striding to produce our desired downsampled frames in both space and time.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of Xiang in order to improve the spatio-temporal downsampling in the processing operations.
In regards to claim 18, the limitations of claim 14 have been addressed. Ma fails to explicitly disclose wherein the spatial scaling factor has a value between zero and one and is proportional to the spatial variability.
Xiang discloses wherein the spatial scaling factor has a value between zero and one and is proportional to the spatial variability ([Page 3] Unliked these methods, we aim to freely resize the space-time volume with arbitrary scale ratios in this work. [Page 4] To ensure that the learned filters are low-pass, we add a softmax layer to regularize the weight values within the range [0, 1] and the sum to be 1. We then use striding to produce our desired downsampled frames in both space and time.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of Xiang in order to improve the spatio-temporal downsampling in the processing operations.
Claim(s) 9, 11-13, 15, and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ma in view of Menon in further view of Galpin.
In regards to claim 9, the limitations of claim 1 have been addressed. Ma fails to explicitly disclose wherein to generate the bit stream of encoded video data, the one or more processor devices are further configured to: prune the plurality of weight parameters to generate a plurality of pruned weight parameters; quantize the plurality of pruned weight parameters to generate a plurality of quantized weight parameters; and entropy encode the plurality of quantized weight parameters to generate the bit stream of encoded video data.
Galpin discloses wherein to generate the bit stream of encoded video data, the one or more processor devices are further configured to: prune the plurality of weight parameters to generate a plurality of pruned weight parameters; quantize the plurality of pruned weight parameters to generate a plurality of quantized weight parameters; and entropy encode the plurality of quantized weight parameters to generate the bit stream of encoded video data ([0156] The weights update coding uses a fix, given entropy coder E and decoder E.sup.−1. These coder and decoder are fixed and known at the DNN-based decoder. As in the classical decoder, the weights are quantized. Other given coder/decoder can also be used to encode the update parameters, for example a given auto-encoder as in “Joint Autoregressive and hierarchical priors for learned image compression”, D. Minnen, J. Ballé, G. Toderici, NIPS 2018”, trained with a set of weights update. The weights update training set are for example given by domain adaptation or metric adaptation.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of Galpin in order to improve the performance of the system [See Galpin].
In regards to claim 11, the limitations of claim 1 have been addressed. Ma fails to explicitly disclose wherein the one or more processor devices are configured to train the neural network to reduce a loss function value comprising an output of a conditional general adversarial network.
Galpin discloses wherein the one or more processor devices are configured to train the neural network to reduce a loss function value comprising an output of a conditional general adversarial network ([0033] DNNs are trained using several types of losses: “objective metric” and “subjective” metric. Loss based on an “objective” metric may be typically Mean Squared Error (MSE) or based on structural similarity (SSIM) for instance. The results may not be perceptually as good as the “subjective metric”, but the fidelity to the original signal (image) is higher. Loss based on “subjective” (or subjective by proxy) may be typically using Generative Adversarial Networks (GANs) during the training stage or advanced visual metric via a proxy Neural Network (NN). Depending on the loss used for training, the resulting parameters of the DNN model may be different.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of Galpin in order to improve the performance of the system [See Galpin].
In regards to claim 12, the limitations of claim 1 have been addressed. Ma fails to explicitly disclose wherein the neural network comprises a multi-layer perceptron network and a plurality of convolutional neural network layers.
Galpin discloses wherein the neural network comprises a multi-layer perceptron network and a plurality of convolutional neural network layers ([0081] The encoder network 401 is usually composed of a set of convolutional layers with stride, allowing to reduce the spatial resolution of the input while increasing the depth, i.e. the number of channels of the input. Squeeze operations may also be used instead of strided convolutional layers (space-to-depth via reshaping and permutations). In the exemplary embodiment illustrated on FIG. 4A, three layers are shown but less or more layers could be used. [0151] FIG. 10 illustrates a diagram of an exemplary embodiment of an auto-encoder 1000 comprising a DNN-based encoder 1001 and a DNN-based decoder 1002 wherein the last layer 1003 is updated with a layer 1004 with weights w resulting in an updated layer 1005. In grey, the retrained/fine-tuned parts of the auto-encoder are shown. The training adaptation is shown in FIG. 10 for the layer update case, but the same principle can be applied to other variants of decoder modifications.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of Galpin in order to improve the performance of the system [See Galpin].
In regards to claim 13, the limitations of claim 1 have been addressed. Ma fails to explicitly disclose wherein the neural network is configured to: receive a frame index value; and output a predicted video frame corresponding to the temporally and spatially down-scaled video data and the frame index value.
Galpin discloses wherein the neural network is configured to: receive a frame index value; and output a predicted video frame corresponding to the temporally and spatially down-scaled video data and the frame index value ([0068] The encoder decodes an encoded block to provide a reference for further predictions. The quantized transform coefficients are de-quantized (240) and inverse transformed (250) to decode prediction residuals. Combining (255) the decoded prediction residuals and the predicted block, an image block is reconstructed. In-loop filters (265) are applied to the reconstructed picture to perform, for example, deblocking/SAO (Sample Adaptive Offset) filtering to reduce encoding artifacts. The filtered image is stored at a reference picture buffer (280).).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of Galpin in order to improve the performance of the system [See Galpin].
In regards to claim 15, the limitations of claim 14 have been addressed. Ma fails to explicitly disclose wherein to generate the decoded video data, the one or more processor devices are further configured to: generate, using a space-time scaling filter, up-scaled video data by temporally and spatially up-scaling the predicted video data based on the temporal and spatial scaling factors; and input the up-scaled video data to the post-processing neural network to generate the decoded video data.
Galpin discloses wherein to generate the decoded video data, the one or more processor devices are further configured to: generate, using a space-time scaling filter, up-scaled video data by temporally and spatially up-scaling the predicted video data based on the temporal and spatial scaling factors; and input the up-scaled video data to the post-processing neural network to generate the decoded video data ([0033] DNNs are trained using several types of losses: “objective metric” and “subjective” metric. Loss based on an “objective” metric may be typically Mean Squared Error (MSE) or based on structural similarity (SSIM) for instance. The results may not be perceptually as good as the “subjective metric”, but the fidelity to the original signal (image) is higher. Loss based on “subjective” (or subjective by proxy) may be typically using Generative Adversarial Networks (GANs) during the training stage or advanced visual metric via a proxy Neural Network (NN). Depending on the loss used for training, the resulting parameters of the DNN model may be different. [0040] In a same way, a generic training set ensures that compression performance is consistent on a wide range of content, but a specific training set could reach better performances for specific applications. Additionally, auto-encoder solutions may be trained at given rate-points, i.e. the weights of the models are optimized for a specific range of bitrates of the transmitted bitstream. [0044] A training configuration is defined by a metric used in the loss function, and a training set of samples or batch which are input to the auto-encoder so that the auto-encoder learns its parameters. The other training configuration could differ from the first training configuration from the metric which could be an objective or perceptual/subjective quality metric and/or the training set which could be a generic training set or a training set with specific contents. The training configurations could also differ in the Lagrange parameters for updating or refining in a light way a DNN to adapt to different bitrate levels. [0072] The decoded picture can further go through post-decoding processing (385), for example, an inverse color transform (e.g. conversion from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping performing the inverse of the remapping process performed in the pre-encoding processing (201). The post-decoding processing can use metadata derived in the pre-encoding processing and signaled in the bitstream.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of Galpin in order to improve the performance of the system [See Galpin].
In regards to claim 16, the limitations of claim 14 have been addressed. Ma fails to explicitly disclose wherein the one or more processor devices are configured to: train a pre-processing neural network of the transmitting device and the post- processing neural network to reduce a difference measure between the decoded video data and a corresponding video data input to the pre-processing neural network.
Galpin discloses wherein the one or more processor devices are configured to: train a pre-processing neural network of the transmitting device and the post- processing neural network to reduce a difference measure between the decoded video data and a corresponding video data input to the pre-processing neural network ([0033] DNNs are trained using several types of losses: “objective metric” and “subjective” metric. Loss based on an “objective” metric may be typically Mean Squared Error (MSE) or based on structural similarity (SSIM) for instance. The results may not be perceptually as good as the “subjective metric”, but the fidelity to the original signal (image) is higher. Loss based on “subjective” (or subjective by proxy) may be typically using Generative Adversarial Networks (GANs) during the training stage or advanced visual metric via a proxy Neural Network (NN). Depending on the loss used for training, the resulting parameters of the DNN model may be different. [0040] In a same way, a generic training set ensures that compression performance is consistent on a wide range of content, but a specific training set could reach better performances for specific applications. Additionally, auto-encoder solutions may be trained at given rate-points, i.e. the weights of the models are optimized for a specific range of bitrates of the transmitted bitstream. [0044] A training configuration is defined by a metric used in the loss function, and a training set of samples or batch which are input to the auto-encoder so that the auto-encoder learns its parameters. The other training configuration could differ from the first training configuration from the metric which could be an objective or perceptual/subjective quality metric and/or the training set which could be a generic training set or a training set with specific contents. The training configurations could also differ in the Lagrange parameters for updating or refining in a light way a DNN to adapt to different bitrate levels. [0072] The decoded picture can further go through post-decoding processing (385), for example, an inverse color transform (e.g. conversion from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping performing the inverse of the remapping process performed in the pre-encoding processing (201). The post-decoding processing can use metadata derived in the pre-encoding processing and signaled in the bitstream.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of Galpin in order to improve the performance of the system [See Galpin].
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ma in view of Menon in further view of IGNATOV et al. (Hereafter, “Ignatov”) [US 2022/0058491 A1].
In regards to claim 10, the limitations of claim 1 have been addressed. Ma fails to explicitly disclose wherein the one or more processor devices are configured to train the neural network to reduce a loss function value comprising an entropy penalization term.
Ignatov discloses wherein the one or more processor devices are configured to train the neural network to reduce a loss function value comprising an entropy penalization term ([0020] In particular, embodiments of the application propose three approaches for the enlargement of information capacity of a BNN, according to the principle of maximum entropy: [0021] Penalization for the loss of information entropy of a weight distribution in the BNN. [0037] In an implementation form of the first aspect, the device is configured to: determine an information loss based on the one or more determined information entropies, and append the information loss as the penalty term to the cost function.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Ma and Menon with the teachings of Ignatov in order to improve the accuracy of the neural network [See Ignatov].
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Kaitlin A Retallick whose telephone number is (571)270-3841. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chris Kelley can be reached at (571) 272-7331. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KAITLIN A RETALLICK/Primary Examiner, Art Unit 2482
Prosecution Insights