DETAILED ACTION
Status of Claims
Claim(s) 1 is pending and are examined herein.
Claim(s) 1 is rejected under 35 U.S.C. §§ 112 and 103.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement IDS(s) submitted on June 27, 2023 is in compliance with the provisions of 37 CFR 1.97 and have been considered by the examiner.
Priority
Acknowledgment is made of the applicant’s claim for priority to foreign Application Serial No. TR2020/22517, filed December 30, 2020.
Claim Objections
Claim(s) 1 is objected due to the following issue. The claim recites in the preamble “A method for the model-independent solution of inverse problems with deep learning in image/video processing,” where the recitation of “the model-independent” should be “a model-independent” for proper antecedent basis in the claim.
Appropriate correction is required.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because of the following reasons: some reference characters/labels mentioned in the description of the specification (page 4, lines 1-12) are not define in the drawing (i.e., y : input image and
x
~
: estimated output image), the meaning of the dashed lines is not explained in the specification, and the drawing don’t show the recited claimed element “shock filter.” . they do not include the following reference sign(s) mentioned in the description: the asynchronous network architecture “203” found in the specification [0036] & [0037] is inconsistent with the drawing in Fig. 2 element “207”. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The specification is objected to because of the following informalities:
a. Page 4, lines 1-12, provides description of the drawing and recites the following:
“The elements and description of these elements for the better understanding of the invention are as follows.
y : input image
x
~
: estimated output image
K: General deep network architecture
D : Deep network performing the reconstruction
Q: Deep network estimating the distortion model
P : Deep network performing the regularization.
A : Distortion model
z : intermediate output image before regularization
u: Regularization parameters”
However, the disclosure only includes one figure that does not label and/or clearly reference all of these elements as described in these corresponding sections. It is noted that the drawing is objected to as described above.
Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION. —The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 1 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, for pre-AIA the applicant regards as the invention.
Regarding Claim 1, the claim recites limitations that renders the scope of the claimed invention indefinite for the following reasons:
The claim recites the limitation “i. obtaining a first estimate of an output image by taking an input image through a shock filter, and obtaining a first estimate of the output image by applying reconstruction to the input image based on an estimated model;” lines 4-6.
The claim step recites “a first estimate of an output image” and “a first estimate of the output image” to describe results produced by two distinct operations, a shock filter and an estimated model-based reconstruction, respectively. While both operations define the same element “a first estimate” of an image, the second operation reference the “output image” recited in the first operation (i.e., shock filter). It is unclear whether these two recitations define the same single element obtained jointly by both operations, two separate first estimate generated, or alternative ways of obtaining a single estimate of an image. The claim nor the specification specify whether these operations are combined, sequential, or alternative embodiments. As a result, a person ordinary skill in the art would not be able to determine with reasonable certainty what the claimed “first estimate” is or how it is obtained. For examination, the claim step is broadly interpreted as initializing the model estimate and shock filter for processing input image to generate an output image.
Additionally, the claim recites the limitation: “ii. obtaining a pre-regularization intermediate output image by putting the input image, model estimate and output image estimate to a deep network (D) which performs the reconstruction;” lines 7-9.
First, the claim step recites “model estimate” and “output image estimate” as inputs to deep network D without proper antecedent basis for either term. Step (i) recited “an estimated model” and “a first estimate of an output image,” respectively, which are not the same claim terms as “model estimate” and “output image estimate” recited in step (ii). Thus, it is unclear whether “the model estimate” and “output image estimate” recited in step (ii) refer back to the “estimated model” and “first estimate of an output image” of step (i), or whether they introduce new and distinct elements.
Second, step (ii) recites that deep network D “performs the reconstruction.” Step (i) recites “applying reconstruction to the input image based on an estimated model,” and step (ii) further recites that deep network D performs “the reconstruction” to generate the intermediate output image. The claim does not clarify whether the reconstruction of step (i) and the reconstruction of step (ii) performed by deep network (D) in step (ii) are the same operation or two separate reconstruction operations performed by different components.
Due to the lack of clarity and antecedent basis in the claim, a person ordinary skill in the art cannot determine with reasonable certainty the scope of step (ii) as claimed, rendering the claim indefinite under 35 U.S.C. § 112. For examination purposes, the reconstruction is broadly interpreted generating an intermediate output image using the model estimate and input image.
Furthermore, the claim recites: “iii. updating the output image estimate by putting the intermediate output image to a deep network (P) that performs the regularization;” lines 10-11.
Step (iii) recites deep network P “performs the regularization” without sufficient antecedent basis in the earlier steps for “a regularization” as a claim element. The term “pre-regularization” in step (ii) defines an expected regularization operation but does not expressly introduce “a regularization” with proper indefinite article. Since the claim step (iii) has no proper antecedent in the claim, a person ordinary skill in the art would not be able to determine with reasonable certainty the metes and bounds of the claimed process. For examination, the claim step is interpreted as performs a regularization.
Lastly, the claim recites: “iv. updating a distortion model estimate by putting the input image and output image estimate to a deep network (Q) which estimates the distortion model;” lines 12-13.
The claim uses four distinct model-related terms across its steps, specifically “estimated model” in step (i), “model estimate” in step (ii), “distortion model estimate” and “distortion model” in step (iv), without clarifying whether these terms refer to the same element or distinct elements. In particular, the claim does not establish whether the “estimated model” of step (i) is the same as the “distortion model” of step (iv), or whether “model estimate” of step (ii) and “distortion model estimate” of step (iv) are the same or different elements. This lack of clarity makes unclear to determine the inputs to deep network D, the outputs of network Q, and iterative process of the claimed method using these elements with different networks. Thus, the iterative process does not clearly define whether a single or multiple model exist making the scope of the claimed iterative method uncertain. a person ordinary skill in the art cannot determine the metes and bounds of the claimed method.
For at least the above reasons, claim 1 does not particularly point out and distinctly claim the invention and it therefore indefinite under 35 USC § 112(b).
In view of the above, the Examiner respectfully requests that Applicant thoroughly review the claim for compliance with the requirements set forth under 35 U.S.C. § 112.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al., (NPL: “A Model-driven Deep Neural Network for Single Image Rain Removal” (May 2020)) in view of Choi et al., (Pub. No.: US 20220122223 A1).
Regarding Claim 1,
Wang discloses the following:
A method for the model-independent solution of inverse problems with deep learning in image/video processing, the method comprising the steps of: (Wang, [Abstract] “in this paper, we propose a model-driven deep neural network for the task, with fully interpretable network structures. Specifically, based on the convolutional dictionary learning mechanism for representing rain, we pro pose a novel single image deraining model and utilize the proximal gradient descent technique to design an iterative algorithm only containing simple operators for solving the model. Such a simple implementation scheme facilitates us to unfold it into a new deep network architecture, called rain convolutional dictionary network (RCD Net), with almost every network module one-to-one corresponding to each operation involved in the algorithm. By end-to-end training the proposed RCDNet, all the rain kernels and proximal operators can be automatically extracted, faithfully characterizing the features of both rain and clean background layers, and thus naturally lead to its better deraining performance, especially in real scenarios.” [P. 1, Section: 1] “Figure 1. (a) Rain convolutional dictionary (RCD) model for rain layer. (b) The formulated optimization model and the corresponding iterative solution algorithm. (c) Visual illustration of the proposed RCDNet one-to-one corresponding to the algorithm (b).”) [Examiner’s Note: the iterative optimization model-driven Deep Neural Network algorithm for image/video processing reads on the model-independent solution for image/video processing.]
i. obtaining a first estimate of an output image by taking an input image ... and obtaining a first estimate of the output image ... based on an estimated model; (Wang, [pp. 3-4, Section: 3.1] “For a observed color rainy image denoted as O ∈ RH×W×3, where H and W are the height and width of the image, respectively, it can be rationally separated as: O=B+R, (1) where B and R represent the background and rain layers of the image, respectively. Then, the aim of most of DL-based deraining methods is to estimate the mapping function (expressed by a deep network) from O to B (or R).” [pp. 5-6, Section: 4.1] “Here, we set
M
(
0
)
as 0 and initialize
B
0
by a convolutional operator on
O
(
0
)
.” Further see Fig. 1) [Examiner’s Note: the initialization of via a convolutional operator applied to the input image reads on the first estimate. The initial rain model
M
(
0
)
estimate represents the starting estimate of the distortion model of the input image.]
ii. obtaining a pre-regularization intermediate output image by putting the input image, model estimate and output image estimate to a deep network (D) which performs the reconstruction; (Wang, [P. 5, Section: 4] “As shown in Fig. 2 (a), the proposed network consists of S stages, corresponding to S iterations of the algorithm for solving (4). Each stage achieves the sequential updates of M and B by M-net and B-net. As displayed in Fig. 2 (b), exactly corresponding to each iteration of the algorithm, in each stage of the network, M-net takes the observed rainy image
O
and the previous output
B
(
s
-
1
)
and
M
(
s
-
1
)
as inputs, and outputs updated
M
(
s
)
, and then B-net takes
O
and
M
(
s
)
as inputs, and outputs an updated
B
(
s
)
.” [P. 5, Section: 4.1] “We can procedures for the
s
t
h
stages of the RCDNet: ... B-net (see equation 12). ... Next, the B-net recovers the background B(s) estimated with current rain kernel and rain maps M(s), and fuses this estimated B(s) with the previously estimated B(s−1) by weighted parameters η2 and (1 − η2) to get the updated background B(s).” Further see Fig. 1.) [Examiner’s Note: the B-net receives the input image (O) and the M(s) current rain model estimate (distortion model) and the current estimated image B(s-1) to obtain an intermediate feature map
B
^
(
s
)
=
O
-
R
(
s
)
.]
iii. updating the output image estimate by putting the intermediate output image to a deep network (P) that performs the regularization; (Wang, [P. 5, Section: 4.1] “We can procedures for the
s
t
h
stages of the RCDNet: ... B-net:
B
s
=
p
r
o
x
N
e
t
θ
m
S
(
1
-
η
2
B
s
-
1
+
η
2
B
^
s
)
(see equation 12). Where
p
r
o
x
N
e
t
θ
m
(
S
)
(·) and
p
r
o
x
N
e
t
θ
b
(
S
)
(·)are two ResNets consisting of several Resblocks with the parameters
θ
m
(
S
)
and
θ
b
(
S
)
at the
s
t
h
stage, respectively. ... Next, the B-net recovers the background B(s) estimated with current rain kernel and rain maps M(s), and fuses this estimated B(s) with the previously estimated B(s−1) by weighted parameters η2 and (1 − η2) to get the updated background B(s).” [p. 4, Section: 3.2] “Instead of choosing a fixed regularizer in the model, the form of the proximal operator can be automatically learned from training data. More details will be presented in the next section. Updating B: Similarly, the quadratic approximation of the problem(4) with respect to B is: ...” Further see Fig. 1.) [Examiner’s Note: The proxNet network is functional describing the network P, where deep network (ResNet) that receives as inputs the intermediate estimate (previously estimated image) the weighted combination and outputs the updated (regularized background estimate B(s)). The proxNet is defined as learned regularizer encoding the prior of the background layer.]
iv. updating a distortion model estimate by putting the input image and output image estimate to a deep network (Q) which estimates the distortion model; (Wang, [P. 5, Section: 4] “As displayed in Fig. 2 (b), exactly corresponding to each iteration of the algorithm, in each stage of the network, M-net takes the observed rainy image
O
and the previous output
B
(
s
-
1
)
and
M
(
s
-
1
)
as inputs, and outputs updated
M
(
s
)
, and then B-net takes
O
and
M
(
s
)
as inputs, and outputs an updated
B
(
s
)
.” [P. 5, Section: 4.1] “We can procedures for the
s
t
h
stages of the RCDNet: ... M-net:
B
^
(
s
)
=
O
-
B
s
-
1
,
M
s
=
p
r
o
x
N
e
t
θ
m
S
(
M
s
-
1
+
E
(
s
)
)
)
(see equation 11). Where
p
r
o
x
N
e
t
θ
m
(
S
)
(·) and
p
r
o
x
N
e
t
θ
b
(
S
)
(·)are two ResNets consisting of several Resblocks with the parameters
θ
m
(
S
)
and
θ
b
(
S
)
at the
s
t
h
stage, respectively. ... Specifically,
B
^
(
s
)
is the rain layer estimated with the previous background
B
s
-
1
, and
R
~
(
s
)
is the rain layer achieved by the generative model (2) with the estimated
M
s
-
1
. Then the M-net calculates the residual information between the two rain layers obtained in this two ways, and extracts the residual information
E
(
s
)
of rain maps with the transposed convolution of rain kernels to update the rain map.” Further See Fig. 1 – M-net.) [Examiner’s Note: The M-net takes the input image
O
and the current output image estimate
B
s
-
1
with the previous model estimate
M
s
-
1
, and outputs the updated estimated rain model
M
s
. The
B
^
(
s
)
of the M-net operation computes the residual between the input and output estimate and used for the model update.] and
v. returning to step ii. and repeating all the steps with the updated output image and updated distortion model for a particular number of iterations. (Wang, [P. 5, Section: 4] “we build a network structure for single image rain removal task by unfolding each iterative steps of the aforementioned algorithm as the corresponding network module. We especially focus on making all network modules one-to-one corresponding to the algorithm implementation operators, for better interpretability. As shown in Fig. 2 (a), the proposed network consists of S stages, corresponding to S iterations of the algorithm for solving (4). Each stage achieves the sequential updates of M and B by M-net and B-net. As displayed in Fig. 2 (b), exactly corresponding to each iteration of the algorithm, in each stage of the network, M-net takes the observed rainy image
O
and the previous output
B
(
s
-
1
)
and
M
(
s
-
1
)
as inputs, and outputs updated
M
(
s
)
, and then B-net takes
O
and
M
(
s
)
as inputs, and outputs an updated
B
(
s
)
.” [P. 5, Section: 4.1] “All the parameters involved can be automatically learned from training data in an end-to-end manner ...” Figure 2. (a) The proposed network with S stages. The network takes a rainy image O as input and outputs the learned rain kernel C, rain map M, and clean background image B. (b) Illustration of the network architecture at the
s
t
h
stage. Each stage consists of M-net and B-net to accomplish the update of rain map M and background layer B, respectively. The images are better to be zoomed in on screen.)
While Wang describes the initialization of the model estimate of an input image and using convolutional operator and the learned rain kernels. Wang does not appear to explicitly teach:
i. obtaining a first estimate of an output image by taking an input image through a shock filter, and obtaining a first estimate of the output image by applying reconstruction to the input image based on an estimated model.
However, Wang in view of Choi teaches the limitation:
i. obtaining a first estimate of an output image by taking an input image through a shock filter, and obtaining a first estimate of the output image by applying reconstruction to the input image based on an estimated model; (Choi, [0060] “The kernel estimation algorithm 455 can estimate a blur kernel from a single image. Blur kernel estimation can also be performed offline for multiple multi-frame blended super resolution images and averaged together. The averaged blur kernel is used for the deconvolution step.” [0065] “As shown in FIG. 5A, the kernel estimation algorithm 455 estimates a blur kernel 320 using an extract luma channel algorithm 500, a bilateral filter algorithm 505, a shock filter algorithm 510, and first and second partial derivative algorithms 515 and 520. ... To restore the strong edges, the bilateral filter algorithm 505 and the shock filter algorithm 510 are applied sequentially to the luminance channel image 530. The bilateral filter algorithm 505 suppresses noise and small details in a bilateral-filtered image 535, and the shock filter algorithm 510 restores strong edges in a shock-filtered image 540.” [0072] “The deconvolution algorithm 605 produces a deconvolved image 625 using inputs including a low-resolution image 305 and a blur kernel 320 in a cost function. An estimation of the blur kernel 320 is described above with respect to FIGS. 5A and 5B.”) [Examiner’s Note: the kernel estimation using shock filter to process the input image and generate restored image, and deconvolution is applied to the input LR image using the estimated blur kernel to generate a deconvoluted image (i.e., reconstruction).]
Wang and Choi are from the same field of endeavor and their disclosure generally relates to (image processing using machine learning networks).
Accordingly, at the effective filing date, it would have been prima facie obvious to one ordinarily skill in the art to modify the combination of Wang and Choi to incorporate the techniques for kernel-aware super resolution as taught by Choi. One would have been motivated to make such a combination in order to overcome blurriness of an original image and adds denoising to get a clean and smooth image (Choi [0034]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
(Pub. No.: US 20220051371 A1) – “Manjit Hota” relates to “Method and imaging system for compensating lens distortion in an image.”
(Pub. No.: US 11798139 B2) – “Michael Slutsky” relates to “Noise-adaptive non-blind image deblurring.”
(Pub. No.: US 20180137606 A1) – “Zhang; Xinxin” relates to “Method And System For Image De-blurring.”
NPL: Gu, Jinjin, et al. "Blind super-resolution with iterative kernel correction."(2019).
NPL: Aggarwal, Hemant K., Merry P. Mani, and Mathews Jacob. "MoDL: Model-based deep learning architecture for inverse problems." (2019).
NPL: Huang, Yan, et al. "Unfolding the alternating optimization for blind super resolution." (2020).
NPL: Zhang, Kai "Deep unfolding network for image super-resolution." (2020).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SADIK ALSHAHARI whose telephone number is (703)756-4749. The examiner can normally be reached Monday - Friday, 9 a.m. 6 p.m. ET.
Examiner interviews are available via telephone, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached on (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.A.A./Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121