DETAILED ACTION
This action is in response to the application filed on March 12th, 2024. Claims 1-11 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see “Remarks”, filed April 3rd, 2026, with respect to the rejection of claims 1-11 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new grounds of rejection is made in view of VRBANČIČ.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3-5, and 8-11 are rejected under 35 U.S.C. 103 as being unpatentable over “A Decoupled Learning Scheme for Real-World Burst Denoising from Raw Images” (herein after referred to by its primary author, Liang) in view of “Transfer Learning With Adaptive Fine-Tuning” (herein after referred to by its primary author, VRBANČIČ)
In regards to claim 1, Liang teaches an information processing apparatus that trains a machine learning model for reducing noise in a moving image (Liang Section 3.1 “To denoise real-world burst image sequences of dynamic scenes, the CNN model should learn to simultaneously align frames and adapt to real-world noise from some training dataset.”), the information processing apparatus comprising: a training unit configured to perform a first training in which a first training dataset is applied to the machine learning model and a second training in which a second training dataset is applied to the machine learning model Liang Figure 1; Section 3.1 “To denoise real-world burst image sequences of dynamic scenes, the CNN model should learn to simultaneously align frames and adapt to real-world noise from some training dataset.” Examiner note: The CNN model in this reference is being trained to perform both tasks, aligning frames and removing noise. Furthermore, figure 1 shows that these trainings are separate.) wherein the machine learning model outputs an image as a processing result for a target frame for noise reduction from an input image consisting of a plurality of frames including the target frame (Liang Figure 1), and the first training is to reduce noise and the second training is to reduce degradation of image quality caused by variation between the plurality of frames (Liang Section 3.1 “To denoise real-world burst image sequences of dynamic scenes, the CNN model should learn to simultaneously align frames and adapt to real-world noise from some training dataset.” Examiner note: Frames are aligned in this reference due to some variation between the expected position in the frame, and the actual position in the frame. This is analogous to a variation between the plurality of frames).
Liang fails to teach one or more processors that execute one or more programs stored in a memory; and performing a first training in which a first training dataset is applied to the machine learning model and a second training in which a second training dataset is applied to the machine learning model after the first training has ended.
However, VRBANČIČ teaches one or more processors that execute one or more programs stored in a memory (VRBANČIČ Section V “The experiments were executed on a single Intel Core i7-6700K based PC, with 4 cores (8 threads) CPU running at 4 GHz, with 64 GB of RAM, and three Nvidia GeForce Titan
X Pascal GPUs each with 12 GB of dedicated GDDR5 memory, running the Linux Mint 19 operating system.”); and performing a first training in which a first training dataset is applied to the machine learning model and a second training in which a second training dataset is applied to the machine learning model after the first training has ended (VRBANČIČ Figure 1; Section III A “In machine learning terms, as presented in Fig. 1, transfer learning roughly translates to transferring the weights of the already trained model, specialized for a specific task, to the model solving a different, but related task [37].” Examiner note: This figure shows a method of obtaining a “pretrained” convolutional neural network, and applying the trained weights to another neural network, then performing a second training on that neural network. The weights transferred from the pretrained neural network, which was trained using a general image dataset, would be considered the first training with the first training dataset, as changing the weights of a neural network is considered training. The training with the target image dataset would be considered the second training with the second data set after the first training has ended).
VRBANČIČ is considered to be analogous to the claimed invention because they are both in the same field of training a machine learning model that undergoes multiple iterations of training. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Liang to include the teachings of VRBANČIČ, to provide the advantage of reduced time complexity (VRBANČIČ Section III A “Additionally, the process of training deep neural network architectures requires a lot of computational power and thus is a time consuming task. In such cases, with the utilization of the transfer learning techniques, one could benefit significantly in terms of time complexity as well as in terms of the large, required dataset.”)
In regards to claim 3, Liang in view of VRBANČIČ teaches the information processing apparatus according to claim 1, wherein the second training is to reduce image lag caused by movement between the plurality of frames (Liang Section 3.1 “The video dataset Dd contains rich dynamic scene motions…” Examiner note: Scene motions are analogous to a moving portion of the scene, which would cause image lag).
In regards to claim 4, Liang in view of VRBANČIČ teaches the information processing apparatus according to claim 3, wherein the one or more processors further function as a generating unit configured to generate, based on still images, the first training dataset and the second training dataset (Liang Section 3.1 “Considering the fact that there lacks a real-world burst image dataset of dynamic scenes with ground-truth clean images, we propose to use two types of datasets for training, which can be generated by using the publicly accessible data.”).
In regards to claim 5, Liang in view of VRBANČIČ teaches the information processing apparatus according to claim 3, wherein the one or more processors further function as a generating unit configured to generate the first training dataset and the second training dataset, and the generating unit generates the first training dataset and the second training dataset so that a maximum value of an amount of movement between frames that are used as the input image in the second training is greater than a maximum value of an amount of movement between frames that are used as the input image in the first training (Liang Section 3.1 “Dd can be easily built by using the many high quality video sequences [35], while Ds can be built by the existing frame averaging method [1]. These two datasets have complementary information. The video dataset Dd contains rich dynamic scene motions, but the noise is synthetic and not real. In contrast, the static burst dataset Ds does not contain scene motion, but can provide information of real noise statistics.”).
In regards to claim 8, Liang in view of VRBANČIČ teaches the information processing apparatus according to claim 1, wherein the machine learning model uses a neural network (Liang Section 3.1 “Given a sequence of N noisy raw images (e.g., in the Bayer color filter array (CFA) pattern [4]) captured by a handheld camera, denoted by I = {I1, I2, ..., IN }, our goal is to estimate a clean RGB image O from I, i.e., O = f(I; θ), where f(·; θ) denotes the denoising model (e.g., a CNN model in our work) parameterized by θ.” Examiner note: CNN stands for convolutional neural network).
In regards to claim 9, Liang in view of VRBANČIČ renders obvious the claim language as in the consideration of claim 1.
In regards to claim 10, Liang in view of VRBANČIČ renders obvious the claim language as in the consideration of claim 1.
In regards to claim 11, Liang in view of VRBANČIČ teaches a non-transitory computer-readable medium storing a program (VRBANČIČ Section V “The experiments were executed on a single Intel Core i7-6700K based PC, with 4 cores (8 threads) CPU running at 4 GHz, with 64 GB of RAM, and three Nvidia GeForce Titan X Pascal GPUs each with 12 GB of dedicated GDDR5 memory, running the Linux Mint 19 operating system.”) and renders obvious the remaining claim language as in the consideration of claim 1.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Liang in view of VRBANČIČ, and further in view of “Convolutional Neural Network With Automatic Learning Rate Scheduler for Fault Classification” (herein after referred to by its primary author, Wen)
In regards to claim 2, Liang in view of VRBANČIČ teaches the information processing apparatus according to claim 1, but fails to teach wherein an initial learning rate of the second training is lower than an initial learning rate of the first training.
However, Wen teaches wherein an initial learning rate of the second training is lower than an initial learning rate of the first training (Wen Figure 2; Algorithm 1 Examiner note: Algorithm 1 shows that for each training step of the training process, the training rate is initially set to η’, then, after performing the “Train DDPG” step, the learning rate is updated based on the predicted action. This shows that the learning rate will change (increase or decrease) for each iteration of training. In the case where the learning rate lowers, the initial learning rate of the second training would be lower than the initial learning rate of the first training).
Wen is considered to be analogous to the claimed invention because they are both solving the same problem of learning rate adjustment. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Liang in view of VRBANČIČ to include the teachings of Wen, to provide the advantage of automatic control of the learning rate, allowing for less time consumption (Wen Abstract “However, the traditional learning rate tuning methods either cost much time consumption or rely on the experts’ experiences, so it is a considerable barrier for the users. To overcome this drawback, this article proposes a CNN with automatic learning rate scheduler (AutoLR-CNN) for fault classification. First, the long short-term memory (LSTM) is used to extract the features of the past loss of CNN. Then, an agent based on deep deterministic policy gradient (DDPG) is trained to automatically control the learning rate for CNN online.”)
Claims 6-7 is rejected under 35 U.S.C. 103 as being unpatentable over Liang in view of VRBANČIČ, and further in view of “BrightFlow: Brightness-Change-Aware Unsupervised Learning of Optical Flow” (herein after referred to by its primary author, Marsal)
In regards to claim 6, Liang in view of VRBANČIČ teaches the information processing apparatus according to claim 1, but fails to teach wherein the second training is to reduce an effect caused by a change in brightness between the plurality of frames.
However, Marsal teaches wherein the second training is to reduce an effect caused by a change in brightness between the plurality of frames (Marsal Abstract “However, these frames could be subject to strong brightness changes due to the radiometric properties of scenes as well as their viewing conditions. In this paper, we present BrightFlow, a new method to train any optical flow estimation network in an unsupervised manner. It consists in training two networks that jointly estimate optical flow and brightness changes.”).
Marsal is considered to be analogous to the claimed invention because they are both in the same field of multiple training stages, with brightness variation adjustment. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Liang in view of VRBANČIČ to include the teachings of Marsal, to provide the advantage of handling brightness variation without supervision (Marsal Section 3.2 “The aforementioned losses enable to train an optical flow estimation model in an unsupervised way with specific solutions to deal with occluded pixels. However, the handling of brightness changes in the loss remains overlooked despite their impact on performances as shown in figure 2. Indeed, the soft census loss, while robust, is still sensitive to some brightness changes, misleading the photometric loss that would interpret them as errors in the optical flow estimation. It concerns mainly strong brightness changes, those that induce over/underexposure or very complex ones due to shadows for instance. To address this weakness, we propose BrightFlow, a new optical flow framework that handles brightness changes with no supervision (see figure 3).”)
In regards to claim 7, Liang in view of VRBANČIČ and Marsal teaches the information processing apparatus according to claim 6, wherein the one or more processors further function as a generating unit configured to generate the first training dataset and the second training dataset, and the generating unit generates the first training dataset and the second training dataset so that a brightness variation rate between frames that are used as the input image in the second training is greater than a brightness variation rate between frames that are used as the input image in the first training (Marsal Section 4.1 “We evaluate the performances of our method on standard datasets, namely Sintel [7], KITTI 2015 [30] and HD1K which exhibit strong brightness changes” Examiner note: This reference teaches that when performing brightness adjustment due to variation between frames, a dataset is need that exhibits strong brightness changes. This dataset would be compared to the dataset used for noise reduction in Liang, which is not said to have strong brightness changes.).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CALEB LOGAN ESQUINO whose telephone number is (703)756-1462. The examiner can normally be reached M-Fr 8:00AM-4:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CALEB L ESQUINO/ Examiner, Art Unit 2677
/ANDREW W BEE/ Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights