Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are cancelled, claims 21-43 are pending and are being examined.
Information Disclosure Statement
The information disclosure statements (IDSs) were submitted on 10/30th/2024, 06/02nd/2025, 12/08th/2025, 01/09th/2026, 03/17th/2026, 05/05th/2026, 06/30th/2026, 08/05th/2026, 08/18th/2026 . The submissions were in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Examiner's Note
The Examiner respectfully requests of the Applicant in preparing responses, to fully consider the entirety of the reference(s) as potentially teaching all or part of the claimed invention. It is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claim(s) above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim(s), typically other passages and figures will apply as well.
The Examiner further notes that when reading the preamble in the context of the entire claim, any preamble recitation is not limiting because the body of the claim describes a complete invention and the language recited solely in the preamble does not provide any distinct definition of any of the claimed invention’s limitations. Thus, the preamble of the claim(s) is not considered a limitation and is of no significance to claim construction. See Pitney Bowes, Inc. v. Hewlett-Packard Co., 182 F.3d 1298, 1305, 51 USPQ2d 1161, 1165 (Fed. Cir. 1999). See MPEP § 2111.02.
Response to Arguments
Applicant’s arguments, see REMARKS pages 7-13 filed 06/9th/2026,
regarding the 35 USC § 103 rejection of claims 21-41 have been considered and they are moot in light of the new rejection below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all
obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 21-24, 27-31, and 34-38 are rejected under 35 U.S.C. 102 as being unpatentable over CABALLERO (US10701394), in view of LOTTER (DEEP PREDICTIVE CODING NETWORKS FOR VIDEO PREDICTION AND UNSUPERVISED LEARNING).
Regarding claim 21, Caballero teaches modify the second frame based at least on motion of one or more pixels between the first frame and the second frame [Col. 29, Line 8-15] An efficient spatial transformer network can compensate for the motion of the frames fed to the SR network. The network attempts to find the best optical flow representation relating a new frame It+1 with a reference current frame It. The flow is pixel-wise dense, allowing to warp each pixel to a new spatial position, and the resulting pixel arrangement is interpolated back onto a regular grid using bilinear interpolation. The examiner notes that Caballero teaches compensating for the motion of the frames by attempting to find the best optical flow representation relating to a new frame).
However, Caballero is not relied upon to explicitly teach:
generate a second frame using one or more neural networks and a first frame as input to the one or more neural networks.
use the modified second frame as input to the one or more neural networks.
On the other hand, LOTTER teaches generate a second frame using one or more neural networks and a first frame as input to the one or more neural networks ([Page 15, first para.] While the models presented here were originally trained to predict one frame ahead, they can be made to predict multiple frames by treating predictions as actual input and recursively iterating. The examiner notes that Caballero and LOTTER are both directed to convolutional neural networks and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Caballero’s convolutional neural network to incorporate generate a second frame using one or more neural networks and a first frame as input to the one or more neural networks as taught by LOTTER [Page 15, first para] to predict multiple frames by treating predictions as actual input [Page 15, first para]).
Furthermore, LOTTER teaches use the modified second frame as input to the one or more neural networks ([Page 15, first para] Starting from the trained weights, the model was trained with a loss over 15 time steps, where the actual frame was inputted for the first 10 and then the model’s predictions were used as input to the network for the last 5. The examiner notes that LOTTER teaches using the modified frame as input to the neural network. The examiner further notes that Caballero and LOTTER are both directed to convolutional neural networks and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Caballero’s convolutional neural network to incorporate use the modified second frame as input to the one or more neural networks as taught by LOTTER [Page 15, first para] to predict multiple frames by treating predictions as actual input [Page 15, first para]).
Regarding claim 22, Caballero teaches the one or more processors of claim 21, wherein the one or more neural networks comprise an encoder/decoder neural network [Fig.12 and Fig. 14]
PNG
media_image1.png
680
1328
media_image1.png
Greyscale
PNG
media_image2.png
474
1340
media_image2.png
Greyscale
The examiner notes that Caballero teaches a neural network that comprises an encoder (Fig, 12) and a decoder (Fig. 14)).
Regarding claim 23 Caballero teaches wherein the one or more neural networks are to combine at least one filter kernel with the first frame and the modified second frame ([Col. 27, line 34-47] For example, in an alternative implementation, past frames can be used to output an estimation or interpolation of super-resolution corresponding to the last frame from the input block. In this implementation, a latency equivalent may not be introduced by avoiding or minimizing the use of a number of frames ahead of the super-resolution frame. The shape of weighting filters Wl is also extended by their temporal extent dl, and their tensor shape becomes dl x nl-1 x nl x kl x kl. One of the most straightforward approaches for a CNN to process videos is to match the temporal depth of the input layer to the number of frames d0=D0 . This will collapse all temporal information in the first layer and the remaining operations are identical to those in a single image SR network, meaning dl =1, l≥1. An illustration of early fusion is shown in FIG. 29A for D0=5, where the temporal dimension has been color coded and the output mapping to 2D space is omitted. The examiner notes that CABALLERO teaches applying a filter to a first frame and a second modified frame).
Regarding claim 24 Caballero teaches where in the one or more neural networks are to apply at least a first portion of at least one filter kernel to the modified second frame, wherein the modified second frame comprises reconstructed data, and to apply at least a second portion of the at least one filter kernel to the first frame ([Col. 8-9, Line 61-5] Implementations can include one or more of the following features. For example, the sub-pixel convolutional neural network can use a spatio-temporal model, the spatio-temporal model can include at least one input filter with a temporal depth that matches a number of selected from the plurality of low-resolution frames, and the at least one input filter can collapse temporal information in a first layer. The sub-pixel convolutional neural network can use a spatiotemporal model, the spatio-temporal model can include at least one layer, and a first layer of the at least one layer can merge frames in groups smaller than an input number of frames. The examiner notes that Caballero teaches applying a filter kernel during image enhancement to at least two input frames one of which is modified [Page 22, Fig. 20]).
Regarding claim 27, Caballero teaches wherein the first frame and the second frame are successive image frames and the motion comprises motion vectors or optical flow [Col. 29, Line 8-15] An efficient spatial transformer network can compensate for the motion of the frames fed to the SR network. The network attempts to find the best optical flow representation relating a new frame It+1 with a reference current frame It. The flow is pixel-wise dense, allowing to warp each pixel to a new spatial position, and the resulting pixel arrangement is interpolated back onto a regular grid using bilinear interpolation. The examiner notes that Caballero teaches consecutive frames with the motion represented by optical flow).
Claims 28-31, and 34 are rejected based upon the same rationale as the rejection of claims 21-24 and 27 since they are the system claims corresponding to the processor claims.
Claims 35-38 are rejected based upon the same rationale as the rejection of claims 21-24 since they are the method claims corresponding to the processor claims.
Claims 25, 32, and 39, are rejected under 35 U.S.C. 103 as being unpatentable over CABALLERO (US10701394), in view of LOTTER (DEEP PREDICTIVE CODING NETWORKS FOR VIDEO PREDICTION AND UNSUPERVISED LEARNING), in view of Giera (US20180311663A1).
Regarding claim 25 Caballero teaches the processor of claim 21. However, Caballero is not relied upon to explicitly teach the one or more neural networks comprise two or more filter kernels to be applied to different respective areas of at least one of the first frame or the modified second frame. On the other hand, Giera the one or more neural networks comprise two or more filter kernels to be applied to different respective areas of at least one of the first frame or the modified second frame ([0012] Each convolution layer convolves small regions of the image using a kernel (or multiple kernels). The examiner notes that Caballero and Giera are both directed to convolutional neural networks and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Caballero’s convolutional neural network to incorporate the one or more neural networks comprise two or more filter kernels to be applied to different respective areas of at least one of the first frame or the modified second frame as taught by Giera [0012] to generate activations to be used as the features that are input to the sub-classifier to assign a label to the input image [0012]).
Claim 32 is rejected based upon the same rationale as the rejection of claims 25 since it is the system claim corresponding to the processor claim.
Claim 39 is rejected based upon the same rationale as the rejection of claims 25 since it is the method claim corresponding to the processor claim.
Claims 26, 33, and 40, are rejected under 35 U.S.C. 103 as being unpatentable over CABALLERO (US10701394), in view of LOTTER (DEEP PREDICTIVE CODING NETWORKS FOR VIDEO PREDICTION AND UNSUPERVISED LEARNING), in view of BICHLER (US20210232897A1), further in view of Ali (US20120219236A1).
Regarding claim 26 Caballero teaches the processor of claim 24. However, Caballero is not relied upon to explicitly teach the circuitry is to further generate different filter kernels to be used at different respective locations of at least the first frame. On the other hand, Ali teaches the circuitry is to further generate different filter kernels to be used at different respective locations of at least the first frame ([0040] In other words, the transformation produces a variable kernel size for filtering different regions (i.e., pixels) of the image. The examiner interprets each different sized kernel to be a different kernel used at a different location of the image. The examiner further notes that Caballero and Ali are both directed to image processing and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Caballero’s image processing to incorporate the circuitry is to further generate different filter kernels to be used at different respective locations of at least the first frame as taught by Ali [0040] to determine a blur radius [0039]).
Claim 33 is rejected based upon the same rationale as the rejection of claims 26 since it is the system claim corresponding to the processor claim.
Claim 40 is rejected based upon the same rationale as the rejection of claims 26 since it is the method claim corresponding to the processor claim.
Claim 41 is rejected under 35 U.S.C. 103 as being unpatentable over CABALLERO (US10701394), in view of LOTTER (DEEP PREDICTIVE CODING NETWORKS FOR VIDEO PREDICTION AND UNSUPERVISED LEARNING), in view of Tschemezki (US20180336460A1).
Regarding claim 41 Caballero teaches the processor of claim 21. However, Caballero is not relied upon to explicitly teach the circuitry is further to provide an external recurrent neural network that is separate from a convolutional encoder-decoder network and to maintain temporal state information that provides temporal information for reconstructing one or more subsequent frames. On the other hand, Tschemezki teaches the circuitry is further to provide an external recurrent neural network that is separate from a convolutional encoder-decoder network and to maintain temporal state information that provides temporal information for reconstructing one or more subsequent frames ([0014] Neural networks, including CNNs (Convolutional Neural Networks) and LSTM (Long Short-Term Memory) networks, can be utilized for prediction. A CNN can incorporate spatially local properties of wildfires. A LSTM network can include the architecture of a CNN and can account for the temporal properties of wildfires and vegetation states. The examiner notes that Tschemezki teaches using a system consisting of a CNN for spatial features and an LSTM, which is a type of RNN [0002], for temporal properties. The examiner further notes that Caballero and Tschemezki are both directed machine learning and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Caballero’s machine learning system to incorporate the circuitry is further to provide an external recurrent neural network that is separate from a convolutional encoder-decoder network and to maintain temporal state information that provides temporal information for reconstructing one or more subsequent frames as taught by Tschemezki [0014] to predict the spatial and temporal properties of wildfires and vegetation states [0014]).
Claim 42-43 is rejected under 35 U.S.C. 103 as being unpatentable over CABALLERO (US10701394), in view of LOTTER (DEEP PREDICTIVE CODING NETWORKS FOR VIDEO PREDICTION AND UNSUPERVISED LEARNING), in view of TODERICI (Full Resolution Image Compression with Recurrent Neural Networks).
Regarding claim 42, Caballero teaches the processor of claim 21. However, Caballero is not relied upon to explicitly teach:
wherein generating the second frame comprises generating external state including a reconstructed first frame.
wherein using the modified second frame as input to the one or more neural networks comprises processing a subsequent frame, based on the modified second frame.
using the one or more neural networks to generate a reconstructed subsequent frame to the one or more neural networks comprises processing a subsequent frame, based on the modified second frame, using the one or more neural networks to generate a reconstructed subsequent frame.
On the other hand, TODERICI teaches wherein generating the second frame comprises generating external state including a reconstructed first frame ([Page 4, Sec. 2.2] In additive reconstruction, which is more widely used in traditional image coding, each iteration only tries to reconstruct the residual from the previous iterations. The final image reconstruction is then the sum of the outputs of all iterations (γ=1 in (1)). The examiner notes that TODERICI teaches cumulatively updating an external state after each iteration to reconstruct a frame. The examiner further notes that Caballero and TODERICI are both directed to neural networks and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Caballero’s machine learning system to incorporate wherein generating the second frame comprises generating external state including a reconstructed first frame as taught by TODERICI [Page 2, Sec. 2] to create a final image reconstruction from decoder outputs [Page 2, Sec. 2]).
Furthermore, LOTTER teaches wherein using the modified second frame as input to the one or more neural networks comprises processing a subsequent frame, based on the modified second frame ([Page 7, Sec. 3.2] Finally, although the PredNet shown here was trained to predict one frame ahead, it is also possible to predict multiple frames into the future, by feeding back predictions as the inputs and recursively iterating. The examiner notes that LOTTER teaches processing a frame based on a modified second frame. The examiner further notes that Caballero and LOTTER are both directed to convolutional neural networks and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Caballero’s machine learning system to incorporate wherein using the modified second frame as input to the one or more neural networks comprises processing a subsequent frame, based on the modified second frame as taught by LOTTER [Page 7, Sec. 3.2] to predict multiple frames into the future [Page 7, Sec. 3.2]).
Furthermore, LOTTER teaches using the one or more neural networks to generate a reconstructed subsequent frame to the one or more neural networks comprises processing a subsequent frame, based on the modified second frame, using the one or more neural networks to generate a reconstructed subsequent frame ([Page 2, Sec. 2] The representation layer, Rl, is a recurrent convolutional network that generates a prediction, A^l, of what the layer input, Al, will be on the next frame. The network takes the difference between Al and A^l and outputs an error representation, El, which is split into separate rectified positive and negative error populations. The error, El, is then passed forward through a convolutional layer to become the input to the next layer (Al+1). The recurrent prediction layer Rl receives a copy of the error signal El, along with top-down input from the representation layer of the next level of the network (Rl+1). The examiner notes that LOTTER teaches reconstructing a frame based on a modified frame. The examiner further notes that Caballero and LOTTER are both directed to convolutional neural networks and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Caballero’s machine learning system to incorporate using the one or more neural networks to generate a reconstructed subsequent frame to the one or more neural networks comprises processing a subsequent frame, based on the modified second frame, using the one or more neural networks to generate a reconstructed subsequent frames taught by LOTTER [Page 2, Sec. 2] to make local predictions of the input to the module [Page 2, Sec. 2]).
Regarding claim 43, Caballero teaches the processor of claim 21. However, Caballero is not relied upon to explicitly teach:
the modified second frame is provided as an external input to the one or more neural networks to generate a subsequent frame as output.
hidden state generated by individual layers of the one or more neural networks is not provided as input to the respective individual layers for generating the subsequent frame.
On the other hand, TODERICI teaches the modified second frame is provided as an external input to the one or more neural networks to generate a subsequent frame as output ([Page 4, Sec. 2.2] In additive reconstruction, which is more widely used in traditional image coding, each iteration only tries to reconstruct the residual from the previous iterations. The final image reconstruction is then the sum of the outputs of all iterations (γ=1 in (1)). The examiner notes that TODERICI teaches cumulatively updating an external state after each iteration to reconstruct a frame. The examiner further notes that Caballero and TODERICI are both directed to neural networks and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Caballero’s machine learning system to incorporate the modified second frame is provided as an external input to the one or more neural networks to generate a subsequent frame as output as taught by TODERICI [Page 2, Sec. 2] to create a final image reconstruction from decoder outputs [Page 2, Sec. 2]).
Furthermore, TODERICI teaches hidden state generated by individual layers of the one or more neural networks is not provided as input to the respective individual layers for generating the subsequent frame ([Page 4, Sec. 2.2] Despite trying to reconstruct the original image at each iteration, we only pass the previous iteration’s residual to the next iteration. The examiner notes that TODERICI teaches only propagating the residual of a layer and not the generated hidden state. The examiner further notes that Caballero and TODERICI are both directed to neural networks and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Caballero’s machine learning system to incorporate hidden state generated by individual layers of the one or more neural networks is not provided as input to the respective individual layers for generating the subsequent frame as taught by TODERICI [Page 4, Sec. 2.2] allow for a better reconstruction [Page 4, Sec. 2.2]).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
CRIVELLI (US20160111129A1)
“CARIVELLI teaches a method for editing a video sequence”
Smirnov (US20180315172A1)
“Smirnov teaches a method for performing noise reduction on an input image by first filtering the input image based on coarse noise models of pixels and then subsequently filtering the filtered input image based on finer noise models ”
Boulanger-Lewandowski (US 2015/0242180 Al)
“Boulanger-Lewandowski teaches a method for sound processing using RNNs”
Calle (US 2018/0358003 Al)
“Calle teaches a method for improving speech quality”
Navarrete (US 2019/0014320 Al)
“Navarrete teaches an image encoding/decoding method using convolutional neural networks”
Takagi (2012/0051426 Al)
“Takagi teaches a classification method to specify a frame to be subjected to sharp or blurred process”
Zhang (US 2017/0345140 Al)
“Zhang teaches a method for generating a simulated image from an input image”
Vogels (US 2018/0293713 Al)
“Vogels teaches a method for applying supervised machine learning using neural networks in denoising images rendered by Monte Carlo path tracing”
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAMCY ALGHAZZY whose telephone number is (571)272-8824. The examiner can normally be reached on M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, OMAR FERNANDEZ RIVAS can be reached on (571) 272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHAMCY ALGHAZZY/Examiner, Art Unit 2128
/OMAR F FERNANDEZ RIVAS/Supervisory Patent Examiner, Art Unit 2128
Prosecution Insights