DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 2/25/26 has been entered.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-30 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 7, 13, 19 and 25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ratner et al. US Patent Pub. No.: 2021/0142176 A1, hereinafter, ‘Ratner’ in view of Hwang et al., hereinafter, ‘Hwang’ US Patent No.: 2019/01018618 and further in view of Dongbing et al. WO 2019/180414, hereinafter, ‘Dongbing’.
Consider Claim 1, 7 (corresponding system of claim 1), 13(Corresponding Method of Claim 13) Claim 19 (Corresponding NTCRM of Claim 1) and 25 (Corresponding Network training system of Claim 25), Ratner teaches One or more processors (e.g., see figure 4), comprising: circuitry (see also 0007) to: use one or more neural networks to perform a forward pass on each input frame of a sequence of input frames to generate a sequence of video frames select a subset of video frames from the sequence of video frames at a regular interval (i.e., input data to a machine learning model – claim 1 – “receiving, at the computer system, input data and a machine learning model to generate a prediction based on the input data”), the subset of video frames comprising two or more video frames and fewer video frames than the sequence of video frames (i.e., input data = video frames) (e.g., predictions based on input data – 0007. 0009, and 0017); and cause backpropagation to be performed perform backpropagation using the subset of video frames(i.e., this is met based on excluding or disregarding portions of “input data 102, such as a one or more images …” – 0017-0018 – see also excluding portions of input during backpropagation – 0006-0010).
However, Ratner does not specifically teach that the input data include video frames and to update weights of the one or more neural networks.
The Examiner respectfully submits that it is notoriously well known that a video is essentially a sequence of still images, called frames, played in rapid succession to create the illusion of motion. Each individual frame is a single, static image within the video. A digital image is fundamentally made up of tiny squares called pixels. A digital image is nothing more than data—numbers indicating variations of red, green, and blue at a particular location on a grid of pixels. Back Propagation is also known as "Backward Propagation of Errors" is a method used to train neural network. The goal of backpropagation is to reduce the difference between the model’s predicted output and the actual output by adjusting the weights and biases in the network. Ratner specifically teaches in claims 11 and 12 wherein the input data is an image and wherein the mask is a portion of the pixels of the image.
For the sake of clarity of record, in analogous art, Hwang teaches in 0065 “the neural network 300 can adjust the weights of the nodes using a training process called backpropagation. Backpropagation can include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update is performed for one training iteration. The process can be repeated for a certain number of iterations for each set of training images until the network 300 is trained well enough so that the weights of the layers are accurately tuned”. In 0018 “the computing device or apparatus may include a camera configured to capture video data (e.g., a video sequence) including video frames. In some cases, the computing device may include a camera device that may include a video codec. In some examples, a camera or other capture device that captures the video data is separate from the computing device, in which case the computing device receives the captured video data. The computing device may further include a network interface configured to communicate the video data.” (note: Hwang also specify a forward pass – see at least 0065-0066)
Therefore, it would have been obvious to a person of ordinary skill in the art before the effect filing date to simply substitute the input data as video frames to arrive at the predictable result to generate a sequence of video frames; and cause backpropagation to be performed perform backpropagation to update weights of the one or more neural networks using fewer than all video frames in the sequence generated using the one or more neural networks for the purpose of optimization as suggested by Ratner.
However, Ratner in view of Hwang does not specifically teach determine a spatial loss and a temporal loss for each video frame in the sequence of video frames; using spatial losses and temporal losses corresponding to the subset of video frames.
In analogous art, Dongbing teaches “loss functions 130 shown in Figure 1 are used to train the mapping-net 1 10 and tracking-net 120 via a backpropagation process as described herein. The loss functions include information about the geometric properties of stereo image pairs of the particular sequence of stereo image pairs that will be used during training. In this way the loss functions include geometric information that is specific to the sequence of images that will be used during training. For example, if the sequence of stereo images is generated by a particular stereo camera setup, the loss functions will include information related to the geometry of that setup. This means the loss functions can extract information about the physical environment from stereo training images. Aptly the loss functions may include spatial loss functions and temporal loss functions.”
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date to modify Ratner in view of Hwang with the teachings of Dongbing to try a spatial loss and a temporal loss for each video frame in the sequence of video frames; using spatial losses and temporal losses corresponding to the subset of video frames to arrive at the result of performing a forward pass on each input frame of a sequence of input frames to generate a sequence of video frames; determine a spatial loss and a temporal loss for each video frame in the sequence of video frames; select a subset of video frames from the sequence of video frames at a regular interval, the subset of video frames comprising two or more video frames and fewer video frames than the sequence of video frames; and cause backpropagation to be performed to update weights of the one or more neural networks using spatial losses and temporal losses corresponding to the subset of video frames for the purpose of improving the visual system.
Claim(s) 2-5, 8-11, 14-17, 20-23, and 26-29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ratner et al. US Patent Pub. No.: 2021/0142176 A1, hereinafter, ‘Ratner’ in view of Hwang et al., hereinafter, ‘Hwang’ US Patent No.: 2019/01018618 in view of Dongbing et al. WO 2019/180414, hereinafter, ‘Dongbing’ and further in view of SEN et al. US Patent Pub. No.: 2022/0128724 A1, hereinafter, ‘Sen’.
Consider Claims 2, 8, 14, 20, and 26, Ratner in view of Hwang teaches the claimed invention except wherein the one or more circuits are further to select one or more crop regions for the frames of the sequence of video frames to use to train the one or more neural networks.
In analogous art, Sen teaches wherein the one or more circuits are further to select one or more crop regions for the images of the sequence of images to use to train the one or more neural networks (i.e., this is met based on the augmented data which includes “cropping” - 0022).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date to try wherein the one or more circuits are further to select one or more crop regions for frames of the sequence of video frames use to train the one or more neural networks for the purpose of optimizing the neural network.
Consider Claims 28, Ratner in view of Hwang teaches the claimed invention except wherein the one or more circuits are further to inject one or more rendering artifacts into the synthetically-generated training data during training of the one or more neural networks.
In analogous art, Sen teaches wherein the one or more circuits are further to inject one or more rendering artifacts into the synthetically-generated training data during training of the one or more neural networks (i.e., this is met based on the augmented data - 0022).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date wherein the one or more circuits are further to inject one or more rendering artifacts into the synthetically-generated training data during training of the one or more neural networks for the purpose of optimizing the neural network.
Consider Claims 3, 9, 15, 21, and 27, Ratner in view of Hwang teaches the claimed invention except wherein the one or more circuits are further to determine pixel-level weightings for a spatial loss term and a temporal loss term, in a loss function to be used to train the one or more neural networks.
Sen teaches wherein the one or more circuits are further to determine pixel-level weightings for a spatial loss term and a temporal loss term, in a loss function to be used to train the one or more neural networks (i.e., this is met by “backpropagation -enabled process” – 0009 – it is understood that by definition backpropagation includes fine tuning weights of a neural network based on the loss).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date wherein the one or more circuits are further to determine pixel-level weightings for a spatial loss term and a temporal loss term, in a loss function to be used to train the one or more neural networks for the purpose of optimizing the neural network.
Consider Claims 4, 10, 16, and 22, Ratner in view of Hwang teaches the claimed invention except wherein the one or more circuits are further to determine the pixel-level weightings based at least in part upon one or more changes identified between one or more images of the sequence of images.
Sen teaches wherein the one or more circuits are further to determine the pixel-level weightings based at least in part upon one or more changes identified between one or more images of the sequence of images (e.g., the amount of loss or error) (i.e., this is met by “backpropagation -enabled process” – 0009 – it is understood that by definition backpropagation includes fine tuning weights of a neural network based on the loss -Backpropagation is just a way of propagating the total loss back into the neural network to know how much of the loss every node is responsible for, and subsequently updating the weights in a way that minimizes the loss by giving the nodes with higher error rates lower weights, and vice versa. the loss function calculates the difference between the network output and its expected output, after a training example has propagated through the network).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date wherein the one or more circuits are further to determine the pixel-level weightings based at least in part upon one or more changes identified between one or more images of the sequence of images for the purpose of optimizing the neural network.
Consider Claims 5, 11, 17, 23 and 29, Ratner in view of Hwang teaches the claimed invention except wherein the one or more circuits are further to apply lower loss weights to initial images in the sequence of images.
Sen teaches wherein the one or more circuits are further to apply lower loss weights to initial images in the sequence of images(i.e., as best understood by the Examiner this is dictated based on the amount of error in the initial set of images in the sequence)(this is met by “backpropagation -enabled process” – 0009 – it is understood that by definition backpropagation includes fine tuning weights of a neural network based on the loss -Backpropagation is just a way of propagating the total loss back into the neural network to know how much of the loss every node is responsible for, and subsequently updating the weights in a way that minimizes the loss by giving the nodes with higher error rates lower weights, and vice versa. the loss function calculates the difference between the network output and its expected output, after a training example has propagated through the network).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date wherein the one or more circuits are further to apply lower loss weights to initial images in the sequence of images for the purpose of optimizing the neural network.
Claim(s) 6, 12, 18, 24 and 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ratner et al. US Patent Pub. No.: 2021/0142176 A1, hereinafter, ‘Ratner’ in view of Hwang et al., hereinafter, ‘Hwang’ US Patent No.: 2019/01018618 in view of Dongbing et al. WO 2019/180414, hereinafter, ‘Dongbing’ and further in view of Blackmon et al., US Patent Pub. No.: 2017/0169602, hereinafter, ‘Blackmon’.
Consider Claims 6, 12, 18, 24 and 30, Ratner teaches the claimed invention except wherein the one or more circuits are further to train the one or more neural networks to perform real time super resolution image reconstruction with temporal smoothing for one or more input image sequences(i.e., video frames).
In analogous art, Blackmon teaches that it may be desired to render images at a high resolution with a high geometric LOD in real-time with a high frame rate, but the processing requirements for achieving this might be impractical on some devices (e.g. mobile devices such as smart phones, tablets or head mounted displays) which might have limited processing resources and power supplies. To address this, the idea of foveated rendering may be used, as described below … The region of interest may correspond to a foveal region of the image. Ray tracing naturally provides high detail and photo-realistic rendering, which human vision is particularly sensitive to in the foveal region; whereas rasterisation techniques are suited for providing temporal smoothing and anti-aliasing in a simple manner, and is therefore suited for use in the regions of the image that a user will see in the periphery of their vision. (e.g., see at least 0004 and 0008).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date to try to train the one or more neural networks to perform real time super resolution image reconstruction with temporal smoothing for one or more input image sequences (i.e., video frames) for the purpose of imaging improvement.
Claim(s) 31 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ratner et al. US Patent Pub. No.: 2021/0142176 A1, hereinafter, ‘Ratner’ in view of Hwang et al., hereinafter, ‘Hwang’ US Patent No.: 2019/01018618 in view of Dongbing et al. WO 2019/180414, hereinafter, ‘Dongbing’ in view of Caballero et al. US Patent No.: 10,701,394 B1, hereinafter, ‘Caballero’.
Consider Claim 31, Ratner as modified by Hwang teaches the claimed invention except wherein the sequence of input frames comprises a sequence of low-resolution video frames, and the circuitry is further to use the one or more neural networks to generate high-resolution video frames based, at least in part, on updated weights for the one or more neural networks from the backpropagation.
In analogous art, Caballero teaches wherein the sequence of input frames comprises a sequence of low-resolution video frames, and the circuitry is further to use the one or more neural networks to generate high-resolution video frames (e.g., see col. 8 lines 50-60 – low to high res ) based, at least in part, on updated weights for the one or more neural networks from the backpropagation (i.e., using a deep neural or convolutional Network).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date to modify Ratner as modified Hwang to try wherein the sequence of input frames comprises a sequence of low-resolution video frames, and the circuitry is further to use the one or more neural networks to generate high-resolution video frames based, at least in part, on updated weights for the one or more neural networks from the backpropagation for the purpose of enhancing image quality.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 11368758 B2 teaches a video on demand (VOD) service system is based on an artificial intelligence (AI) video learning platform. A VOD service system based on an AI video learning platform may perform video learning according to AI-based Super Resolution Convolutional Neural Networks (SRCNNs) to calculate a weight required for restoring a high image quality video from a high image quality VOD file, and then restore a low image quality VOD file to a high image quality VOD file using the calculated weight corresponding to the VOD file later on.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHARLES TERRELL SHEDRICK whose telephone number is (571)272-8621. The examiner can normally be reached 8A-5P.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew D Anderson can be reached at 571 272 4177. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHARLES T SHEDRICK/Primary Examiner, Art Unit 2646
Prosecution Insights