DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restriction
Applicant's election with traverse of Invention I in the reply filed on 04/26/2026 is acknowledged. The traversal is on the ground(s) that PCT unity principles are not the correct framework. This is found persuasive because the present application is in fact a bypass continuation filed under 35 U.S.C 111(a).
The requirement is withdrawn. However, U.S practice restriction requirements is attached below.
Restriction to one of the following inventions is required under 35 U.S.C. 121:
Invention I, claims 3–5, 11–12, and 15–19, drawn to a runtime video inpainting refinement / quality-control pipeline, characterized by generating an image initial mask template based on an initial blurred region in a first inpainting image, performing threshold-triggered morphological processing to obtain an image target mask template, performing threshold-triggered additional inpainting to obtain a second inpainting image, and determining a target inpainting image (including reserved-object contour complementation / consistency logic and/or fallback selection using the first inpainting image), classified in G06T 5/00.
Invention II, claims 6–10, drawn to a training the information propagation model used for inpainting, characterized by cyclic iterative training using training samples and parameter adjustment using a target loss function constructed based on a prediction inpainting image, an image prediction mask template, and an object prediction mask template (with multi-part loss / sub-loss definitions), classified in G06N 3/045.
The inventions are independent or distinct, each from the other because:
Invention I-II are related as subcombinations disclosed as usable together in a single combination. The subcombinations are distinct if they do not overlap in scope and are not obvious variants, and if it is shown that at least one subcombination is separately usable. In the instant case, subcombination I has separate utility such as a runtime artifact-detection and refinement framework driven by blurred-region-derived mask templates, blurred-pixel quantity thresholds, morphological processing, iterative inpainting refinement and/or reserved-object contour consistency logic with fallback output selection. Subcombination II has separate utility such as a specific training framework for the information propagation model, including cyclic iterative training with defined training samples and multi-part loss construction supervising prediction inpainting images together with image / object prediction mask templates. See MPEP § 806.05(d).
The examiner has required restriction between subcombinations usable together. Where applicant elects a subcombination and claims thereto are subsequently found allowable, any claim(s) depending from or otherwise requiring all the limitations of the allowable subcombination will be examined for patentability in accordance with 37 CFR 1.104. See MPEP § 821.04(a). Applicant is advised that if any claim presented in a divisional application is anticipated by, or includes all the limitations of, a claim that is allowable in the present application, such claim may be subject to provisional statutory and/or nonstatutory double patenting rejections over the claims of the instant application.
Restriction for examination purposes as indicated is proper because all the inventions listed in this action are independent or distinct for the reasons given above and there would be a serious search and/or examination burden if restriction were not required because one or more of the following reasons apply:
Inventions I–II require a different field of search (e.g. searching different main-groups/sub-groups or electronic resources, or employing different search strategies or search queries), consistent with their differing CPC classifications noted above (runtime artifact-detection and refinement framework versus training framework for the information propagation model).
Applicant is advised that the reply to this requirement to be complete must include (i) an election of an invention to be examined even though the requirement may be traversed (37 CFR 1.143) and (ii) identification of the claims encompassing the elected invention.
The election of an invention may be made with or without traverse. To reserve a right to petition, the election must be made with traverse. If the reply does not distinctly and specifically point out supposed errors in the restriction requirement, the election shall be treated as an election without traverse. Traversal must be presented at the time of election in order to be considered timely. Failure to timely traverse the requirement will result in the loss of right to petition under 37 CFR 1.144. If claims are added after the election, applicant must indicate which of these claims are readable upon the elected invention.
Should applicant traverse on the ground that the inventions are not patentably distinct, applicant should submit evidence or identify such evidence now of record showing the inventions to be obvious variants or clearly admit on the record that this is the case. In either instance, if the examiner finds one of the inventions unpatentable over the prior art, the evidence or admission may be used in a rejection under 35 U.S.C. 103 or pre-AIA 35 U.S.C. 103(a) of the other invention.
During a telephone conversation with Ms. Weng on 05/29/2026, an election was made without traverse to prosecute the Invention I, claims 3–5, 11–12, and 15–19. Affirmation of this election must be made by applicant in replying to this Office action. Claims 6-10 were withdrawn from further consideration by the examiner, 37 CFR 1.142(b), as being drawn to a non-elected invention.
Applicant is reminded that upon the cancelation of claims to a non-elected invention, the inventorship must be corrected in compliance with 37 CFR 1.48(a) if one or more of the currently named inventors is no longer an inventor of at least one claim remaining in the application. A request to correct inventorship under 37 CFR 1.48(a) must be accompanied by an application data sheet in accordance with 37 CFR 1.76 that identifies each inventor by his or her legal name and by the processing fee required under 37 CFR 1.17(i).
Claim Objections
Claims 1, 13 and 20 are objected to because of the following informalities:
i: The independent claims recite: "performing, when a first quantity of initial blurred pixels... morphological processing on an initial blurred region corresponding to the initial blurred pixel...".
ii: Similar to the above, the claims further recite: "...a second quantity of intermediate blurred pixels... inpainting processing on a pixel region corresponding to the intermediate blurred pixel...".
Appropriate correction is required.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4-5, 11-13 and 16-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 4-5 and 16-17 recite the limitations “contours of the second-type objects” in claims. There is insufficient antecedent basis for this limitation in the claim. Specifically, claims 4-5 and 16-17 recite “contours of the second-type objects” (plural forms), but the antecedent claims 3 and 15 introduces only “a second-type object” and “the second-type object” in the singular forms. There is no antecedent basis in the claim dependency chain for the plural “second-type objects” rendering the scope of this limitation indefinite. It is unclear whether the claims require one reserved object, multiple reserved objects, or a comparison between different contours of the same object in different mask templates.
Claims 11 and 18 depend from claims 1 and 13, respectively. Independent claims 1 and 13 require “determining a target inpainting image … based on the second inpainting image”. However, claims 11 and 18 further recite “using the first inpainting image as the target inpainting image … when the first quantity of initial blurred pixels … is less than the first threshold”. It is unclear whether the target inpainting image is required to be determined based on the second inpainting image, as recited in the independent claims, or instead is the first inpainting image, as recited in claims 11 and 18. Claims 12 and 19 depend from claims 1 and 13, respectively. Independent claims 1 and 13 require performing the second inpainting operation and determining a target inpainting image based on the second inpainting image. However, claims 12 and 19 further recite using the first inpainting image as the target inpainting image when the second quantity of intermediate blurred pixels is less than the second threshold. It is unclear whether the target inpainting image is required to be based on the second inpainting image, as recited in the independent claims, or instead is the first inpainting image, as recited in claims 12 and 19. Accordingly, the scope of claims 11–12 and 18–19 is unclear because the claims create ambiguity as to whether the target inpainting image is determined based on the second inpainting image or whether the first inpainting image is used directly as the target inpainting image when the recited threshold condition is not satisfied
Claims 13 and 20 recite the limitations “performing inpainting processing … in the candidate image…” in claims. There is insufficient antecedent basis for this limitation in the claim. The claims recite “performing mask processing … to obtain a target image after mask processing” but subsequently recite “performing inpainting processing … in the candidate image…”. The claims do not clearly establish whether “the candidate image” refers to the previously recited “target image after mask processing” or to a different image. Accordingly, the metes and bounds of the claimed subject matter are unclear.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1–2, 11–14 and 18–20 are rejected under 35 U.S.C. §103 as being unpatentable over Lao (Lao et al. “Flow-Guided Video Inpainting with Scene Templates.” ArXiv.org, 2021) in view of Schroers (Schroers et al., US 2021/0304387 A1, 2021), further in view of Kudelski (Kudelski et al, US 2022/0366544 A1, 2022).
Regarding claim 1, Lao teaches an image processing method, performed by a computer device, the method comprising:
performing mask processing on a first-type object comprised in an obtained target video frame image, to obtain a candidate image after mask processing;
( [Fig. 2], [Sec. 1, 3.1]: Lao teaches obtaining a video frame image
I
i
and a corresponding mask
M
i
for each frame, wherein the mask
M
i
indicates the region of the first-type object to be removed from the video frame image. Lao teaches performing mask processing on the first-type object by excluding the masked region
M
i
from the inpainting energy minimization (Eq. 1), such that the masked region is treated as the inpainting region. The resulting to-be-processed image, the video frame image with the masked region identified for inpainting, corresponds to the candidate image after mask processing. )
the first-type object being an image element for inpainting;
( [Fig. 2], [Fig. 7], [Fig. 9]: Lao teaches that the first-type object is a foreground object to be removed from the video, such as the human object depicted in Fig. 2 and Fig. 7 on the DAVIS dataset, or the vehicle and horse objects depicted in Fig. 9 on the Foreground Removal dataset. These foreground objects are the image elements for inpainting, i.e., the regions to be filled with background content. )
performing inpainting processing on the first-type object in the candidate image to obtain a first inpainting image,
( [Fig. 2], [Sec. 3-4]: Lao teaches performing inpainting processing on the first-type object in the candidate image by jointly inferring a scene template
f
and a set of warps
w
i
through the energy minimization of Eq. (1). Given the inferred scene template and warps, inpainting processing is performed by mapping the scene template into the masked region
M
i
of the candidate image via the inferred warp
w
i
, as described in §4. The inpainting result
P
t
computed using the L2-L1 optimization of Eq. (5) corresponds to the first inpainting image. )
and generating an image initial mask template based on an initial blurred region in the first inpainting image;
( [Sec. 4], [Sec. 5.2], [Fig. 8]: Lao teaches that the scene-template inpainting/interpolation may produce blur or artifacts, and further teaches that after the inpainting operation, some masked pixels may remain unfilled because they correspond to scene points that were never revealed in the entire video. Lao also teaches, in §5.2, estimating foreground/ missing mask regions by thresholding a residual between the scene template mapped to the image and the image itself, according to
R
t
=
∥
I
t
x
-
f
w
t
-
1
x
∥
2
2
>
α
, where
α
=
0.1
. Thus, Lao teaches residual-threshold generation/ updating of a mask region corresponding to unresolved/ defective pixels associated with the inpainting result. The residual-based estimated mask region
R
t
corresponds to the image initial mask template, and the unresolved/ defective/ unfilled region associated with the first inpainting result corresponds to the initial blurred region in the first inpainting image. )
performing, when a residual value associated with initial blurred pixels comprised in the image initial mask template reaches a first threshold,
( [Sec. 5.2], [Fig. 8], [Eq. (6)]: Lao teaches that masks may come from user annotation or segmentation algorithms and may be incomplete or erroneous. Lao further teaches estimating foreground masks by thresholding a residual between the scene template, computed from available noisy annotations, and the images, using
R
t
=
∣
I
t
x
-
f
w
t
-
1
x
∣
2
2
>
α
, where
α
=
0.1
. Lao states that the method infers missing annotations and corrects incorrect annotations; thereby performing threshold-based residual processing on a pixel region associated with an imperfect/ defective mask to obtain an updated mask, corresponding to the image target mask template. )
performing, when a residual value associated with at least one of intermediate blurred pixels comprised in the image target mask template reaches a second threshold, inpainting processing on a pixel region corresponding to the intermediate blurred pixel in the first inpainting image, to obtain a second inpainting image; and
( [Sec. 4], [Sec. 5.2], [Fig. 4, 8]: Lao teaches estimating missing or imperfect mask annotations by thresholding a residual between the scene template mapped to the image and the image itself, according to
R
t
=
∥
I
t
x
-
f
w
t
-
1
x
∥
2
2
>
α
, where
α
=
0.1
. Thus, Lao teaches determining a mask region based on whether a residual value associated with a pixel reaches a threshold. Lao further teaches that, after the scene-template/ flow-guided inpainting result is computed, some masked pixels may remain unfilled because they correspond to scene points that were never revealed in the video, and those remaining unfilled masked pixels are filled using DeepFill. Accordingly, Lao teaches performing subsequent inpainting processing on a pixel region corresponding to unresolved/ intermediate blurred pixels indicated by the mask to obtain a further inpainted image, corresponding to the second inpainting image. )
determining a target inpainting image corresponding to the candidate image based on the second inpainting image.
( [Abstract], [Sec. 3 & 4], [Figs. 2-4]: Lao teaches generating an inpainted video frame by mapping a scene template into the masked region of the target video frame image. Lao further teaches that, after the scene-template/ flow-guided inpainting operation is performed, some masked pixels may remain unfilled because the corresponding scene points were never revealed in the video, and those remaining unfilled masked pixels are filled using DeepFill. In the proposed mapping, the image obtained after applying DeepFill to the remaining unfilled masked pixels in the first inpainting image corresponds to the second inpainting image. The completed video frame after the remaining unfilled pixels are filled corresponds to the target inpainting image corresponding to the candidate image.)
Lao teaches the overall video inpainting pipeline, however does not expressly disclose threshold-based evaluation of problematic quantity of initial blurred pixels where Schroers teaches:
performing, when a first quantity of initial blurred pixels comprised in the image initial mask template reaches a first threshold,
( [Fig 4, Step 465], [0042-0048]: Schroers teaches detecting one or more anomaly candidates using the residual image, determining a residual value associated with each anomaly candidate, and identifying those anomaly candidates whose residual values meet or exceed a predetermined threshold residual value as anomalous pixels. Schroers further teaches that only anomaly candidates meeting or exceeding a brightness threshold are identified as anomalous. Schroers further discloses optionally performing temporal comparison by comparing locations of anomaly candidates in video frame 130b with corresponding locations in previous frame 130a or next frame 130c, and identifying anomaly candidates as actually anomalous based on the comparison. Schroers further discloses filtering anomaly candidates by clustering them based on location, using a clustering algorithm such as DBSCAN, and identifying as anomalous pixels those anomaly candidates that do not belong to a cluster or that occupy small or compact clusters, while disregarding clusters having a large size or non-centralized distribution. )
performing, when a second quantity of intermediate blurred pixels comprised in the image target mask template reaches a second threshold,
( [Fig 4, Step 466-467], [0043-0054]: Schroers teaches that several image features, such as edges and reflections, can produce high residual values after inpainting, and that one or more filtering criteria may be applied to identify true anomalies. After temporal filtering, clustering anomaly candidates, disregarding clusters based on large size or non-centralized distribution, and identifying anomaly candidates in small or compact clusters as anomalous pixels, Schroers further teaches second-stage predetermined criteria for high residual values and cluster geometry, including residual-value thresholds and cluster-size/ diameter/ ratio criteria. Schroers also teaches generating an error markup image indicating the location of anomalous pixels and optionally correcting the input image using an inpainted masked patch corresponding to the location of the anomalous pixel. )
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate Schroers' quantity-threshold gating technique into Lao's video inpainting pipeline to evaluate problematic pixels remaining after Lao’s first scene-template/ flow-guided inpainting result. Lao's residual-based mask estimation identifies defective pixel regions in the inpainting result but applies no gate to evaluate whether the quantity of such pixels warrants further processing, unconditionally triggering subsequent inpainting on every detected residual region regardless of its size or significance. Schroers recognizes this same inefficiency in the analogous context of automated pixel error detection, teaching that filtering criteria including a predetermined threshold residual value must be applied to confirm that a sufficient quantity of truly anomalous pixels exists before any correction action is taken, in order to avoid unnecessary processing on insignificant detections. The combination involves nothing more than applying Schroers' known quantity-threshold gating technique to Lao's known video inpainting pipeline to yield the predictable result of improved processing efficiency, consistent with the explicit rationale stated in the specification itself.
Lao [as modified by Schroers] teaches identifying an unresolved/ blurred/ problematic pixel region after the first inpainting result using residual-threshold and filtering analysis. However, Lao [as modified by Schroers] does not expressly disclose morphologically processing where Kudelski teaches:
performing morphological processing on an initial blurred region corresponding to the initial blurred pixel to obtain an image target mask template;
( [0040-0042]: Kudelski discloses a step of post-processing of the provided inpainting mask so as to fulfill machine learning model requirements, wherein the inpainting mask [corresponding to the initial blurred region identified from the initial blurred pixels] is subjected to mask post-processing operations prior to being input into the ML inpainting model. Kudelski further discloses that the inpainting mask post-processing step comprises morphological operations applied to the inpainting mask, including mask dilation, mask erosion, and/or mask refinement operations, to produce a refined/ processed mask, corresponding to the image target mask template obtained from morphological processing on the initial blurred region corresponding to the initial blurred pixel. )
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to further modify Lao [as modified by Schroers] by applying Kudelski’s morphological inpainting-mask post-processing to the unresolved/ blurred/ problematic region identified after the first inpainting result. Lao teaches that remaining unresolved masked pixels may require subsequent inpainting, Schroers teaches identifying problematic pixel regions using residual-threshold/ filtering criteria, and Kudelski teaches refining inpainting masks before machine-learning inpainting to provide smoother and more coherent mask regions. The modification would have been a predictable use of a known inpainting-mask refinement technique on a known inpainting mask region, with the expected benefit of improving mask coherence, reducing mask noise/ artifacts, and more reliably defining the pixel region selected for Lao’s subsequent inpainting operation.
Regarding claim 2, Lao [as modified by Schroers and Kudelski] teaches the method according to claim 1, wherein the performing inpainting processing on the first-type object in the candidate image to obtain a first inpainting image, and generating a corresponding image initial mask template based on an initial blurred region in the first inpainting image comprises:
inputting a video sequence comprising the candidate image into a trained information propagation model; and
( [Fig. 2], [Sec. 3-3.1]: Lao teaches inputting a video sequence comprising a plurality of video frame images
I
t
, together with their corresponding masks
M
t
, into a trained optimization-based scene propagation model. Specifically, Lao's energy minimization framework of Eq. (1) operates over the entire video sequence simultaneously, inferring a scene template
f
and a set of warps
w
t
from all frames collectively. The video sequence comprising the candidate image, i.e. the masked video frames, corresponds to the input video sequence, and the scene propagation model defined by Eq. (1)–(5) corresponds to the trained information propagation model because it propagates image information across frames using the inferred scene template and warps. )
performing, in the trained information propagation model, inpainting processing on the first-type object in the candidate image based on an image element in another video frame image in the video sequence to obtain the first inpainting image, and
( [Fig. 2], [Sec. 3–4], [Eq. (1), (5)]: Lao teaches performing inpainting processing on the masked region of the candidate image by propagating image elements/ pixels from other video frame images in the video sequence via the inferred warps
w
t
. Specifically, Lao's scene template
f
is constructed from all visible pixels across all frames, and the inpainting result
P
t
for each frame is obtained by mapping the scene template into the masked region using the inferred warp
w
t
-
1
, thereby filling the masked region of the candidate image based on image elements present in other video frames. The resulting inpainting result
P
t
corresponds to the first inpainting image. )
generating a corresponding image initial mask template based on the initial blurred region in the first inpainting image.
( [Sec. 4], [Sec. 5.2], [Fig. 8]: Lao teaches that the scene-template inpainting/interpolation may produce blur or artifacts in the inpainting result
P
t
, and further teaches estimating a residual-based mask region
R
t
by thresholding the residual between the scene template mapped to the image and the image itself, according to
R
t
=
∥
I
t
x
-
f
w
t
-
1
x
∥
2
2
>
α
, where
α
=
0.1
. This residual-based mask region
R
t
is generated based on the defective/ blurred regions remaining in the first inpainting result, and corresponds to the image initial mask template generated based on the initial blurred region in the first inpainting image. )
Regarding claim 11, Lao [as modified by Schroers and Kudelski] teaches the method according to claim 1, further comprising:
using the first inpainting image as the target inpainting image corresponding to the candidate image when the first quantity of the initial blurred pixels comprised in the image initial mask template is less than the first threshold.
( Lao: [Sec. 4], [Fig. 4]; Schroers: [Fig. 4, Step 465], [0042–0048]: Lao teaches that additional inpainting/ filling is performed for remaining unfilled masked pixels after the initial scene-template/ flow-guided inpainting result. Schroers teaches that, after computing the residual image and detecting anomaly candidates, only those anomaly candidates whose residual values meet or exceed the predetermined threshold are identified as anomalous and subjected to further processing. Schroers further teaches that, when the quantity of anomaly candidates does not meet the threshold criteria, for example, when all residual values fall below a threshold, the system skips to the next frame batch without performing any correction. In the proposed mapping, when the first quantity of initial blurred pixels comprised in the image initial mask template is less than the first threshold [when the defective pixel region is insufficiently significant to warrant further processing] the system skips the morphological processing and second inpainting pass, and the first inpainting image is used directly as the target inpainting image, consistent with Schroers' express teaching to skip further processing when the quantity threshold is not met. )
Regarding claim 12, Lao [as modified by Schroers and Kudelski] teaches the method according to claim 1, further comprising:
using the first inpainting image as the target inpainting image corresponding to the candidate image when the second quantity of the intermediate blurred pixels comprised in the image target mask template is less than the second threshold.
( [Fig. 4, Steps 466–468], [0043–0054]: Schroers teaches applying a second stage of filtering criteria, including residual-value thresholds and cluster-size, diameter, and height-to-width ratio criteria to the anomaly candidates remaining after the first threshold gate, and that clusters failing to meet these criteria are disregarded without triggering correction processing. Schroers further teaches that when the quantity of confirmed anomalous pixels after the second-stage filtering does not meet the threshold criteria, no correction of the input image is performed. In the proposed mapping, when the second quantity of intermediate blurred pixels comprised in the image target mask template, i.e. the morphologically processed mask produced by Kudelski's operations, is less than the second threshold, the system determines that the blurred region remaining after morphological processing is insufficiently significant to warrant the computationally expensive second inpainting pass, and accordingly uses the first inpainting image directly as the target inpainting image, consistent with Schroers' express teaching to skip correction when the second-stage quantity threshold is not met. )
Regarding claims 13–14 and 18–19, the rationale provided in the rejection of claims 1–2 and 11–12 is incorporated herein. In addition, Schroers [as modified by Chang and Kudelski] teaches a computer system or apparatus, including CPU/ processors/ memories/ RAM/ GPU/ PROM, to execute computer programs for image processing ( [Schroers: 0016]; [Kudelski: 0069] ). Accordingly, the method for image processing of claims 1–2 and 11–12 corresponds to the apparatus of claims 13–14 and 18–19, and performs the steps disclosed herein. Therefore, the claims are all rejected.
Regarding claim 20, the rationale provided in the rejection of claim 1 is incorporated herein. In addition, Schroers [as modified by Chang and Kudelski] teaches a computer system or apparatus, including CPU/ processors/ memories/ RAM/ GPU/ PROM, to execute computer programs for image processing ( Schroers: [0016]; Kudelski: [0069] ). Accordingly, the method for image processing of claim 1 corresponds to the non-transitory computer-readable storage medium of claim 20, and performs the steps disclosed herein. Therefore, the claims are all rejected.
Claims 3–5, and 15–17 are rejected under 35 U.S.C. §103 as being unpatentable over Lao [as modified by Schroers and Kudelski ] in view of Ke (Ke et al, Occlusion-Aware Video Object Inpainting, 2021).
Regarding claim 3, Lao [as modified by Schroers and Kudelski] teaches the method according to claim 2, wherein the method further comprises:
inputting an object initial mask template into the trained information propagation model, the object initial mask template being determined after identifying a second-type object comprised in the video frame image, and the second-type object being an image element that needs to be reserved; and
( [Fig. 8], [Sec. 3.1], [Sec. 5.2]: Lao teaches that masks
M
i
for each video frame image are provided as inputs to the scene template inference model, and that these masks may come from user annotation or object segmentation algorithms. Lao further teaches, in the context of incomplete annotations, estimating foreground masks by thresholdilding a residual
R
t
=
∥
I
t
x
-
f
w
t
-
1
x
∥
2
2
>
α
, where
α
=
0.1
, thereby identifying the foreground object region, i.e. the first-type object to be inpainted, and the complementary unmasked region constitutes the background scene that needs to be preserved. Lao further teaches, in the DAVIS dataset experiments, using pixel-wise per-frame annotations from a video object segmentation benchmark to identify annotated moving objects as the regions for inpainting, with the remaining unmasked region corresponding to the background scene that is to be preserved, i.e. the second-type object that needs to be reserved. In the proposed mapping, Lao's input mask
M
i
identifying the foreground object to be removed corresponds to the object initial mask template, which is determined after identifying the second-type object [the background/ reserved region], comprised in the video frame image. The object initial mask template, together with the video sequence, is inputted into Lao's scene template inference model, which corresponds to the trained information propagation model. )
performing, in the trained information propagation model, morphological processing on the second-type object in the object initial mask template to obtain an object target mask template.
( [Fig. 3], [0040-0042]: Kudelski teaches that an automatically proposed inpainting mask may be post-processed before being input into the machine-learning inpainting model, and that the mask post-processing may include smoothing/ coherence improvements and morphological operations including erosion, dilation, blurring, and thresholding. Kudelski further teaches that the processed mask is made smooth, coherent, and regular so as to be suitable for use by the inpainting model. In the proposed combination, the object initial mask template corresponding to the second-type object is morphologically post-processed using Kudelski’s mask-refinement operations, and the resulting refined/ coherent mask corresponds to the object target mask template. Lao’s scene-template/ warp-based video inpainting framework receives and uses the refined mask for inpainting, corresponding to processing in the trained information propagation model. )
Lao [as modified by Schroers and Kudelski] does not cleanly teach where Ke is directed to occlusion-aware video object inpainting and teaches recovering the complete shape/ appearance of an occluded object from its visible mask segmentation:
performing, in the trained information propagation model, object contour complementation processing on the second-type object in the object initial mask template to obtain an object target mask template.
( Ke, [Abstract]; [§1]; [§3]; [§3.1]; [Figs. 2-3]: Ke teaches VOIN, an occlusion-aware video object inpainting network, which recovers the complete shape and appearance of occluded objects in videos given visible mask segmentation of the object. The occluded object whose complete shape and appearance are recovered corresponds to the claimed second-type object, because the object is not removed but is preserved/ restored by completing its occluded portions. Ke further teaches that VOIN jointly performs video object shape completion and occluded texture generation, and that the shape completion module infers complete object shapes from visible mask regions and object semantics. Ke also teaches propagating temporally consistent object texture to the same moving object across frames, reinforcing that the object being processed is the reserved/restored second-type object. The visible mask segmentation/visible mask region of the occluded object corresponds to the object initial mask template. The recovery of the complete object shape/amodal segmentation mask corresponds to object contour complementation processing. The predicted complete object mask/amodal object shape corresponds to the object target mask template. )
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to further modify Lao [as modified by Schroers and Kudelski] by incorporating Ke’s known object shape/ contour completion technique into Lao’s video inpainting/ information-propagation framework. Lao [as modified by Schroers and Kudelski] teaches using video-frame information and masks for video inpainting, identifying problematic or reserved image regions using residual-threshold/ filtering analysis, and refining masks before subsequent inpainting. Ke addresses the same field of video object inpainting and teaches recovering the complete shape and appearance of an occluded object from visible mask segmentation. Ke further teaches that completing an object’s shape/ mask before appearance recovery improves restoration of the object’s complete shape and appearance. Therefore, applying Ke’s object shape/ contour completion to the object initial mask template in Lao’s mask-based video inpainting framework would have been a predictable use of a known video object mask-completion technique to obtain a more complete and accurate object target mask template, with the expected benefit of preserving/ restoring the second-type object and improving spatial/ temporal consistency of the final inpainting result.
Regarding claim 4, Lao [as modified by Schroers, Kudelski and Ke] teaches the method according to claim 3, wherein the determining a target inpainting image corresponding to the candidate image based on the second inpainting image comprises:
comparing the object initial mask template with the object target mask template to obtain a first comparison result, the first comparison result indicating whether contours of the second-type objects are consistent; and
( Ke, [Abstract], [§1], [§3], [§3.1], [§3.3], [Fig. 2], [Fig. 3], [Fig. 11]: Ke teaches recovering both the complete shape and appearance of occluded objects in videos given visible mask segmentation of the object. Ke further teaches that VOIN includes an object shape completion stage that computes amodal object shapes based on visible object content and recovers amodal segmentation masks for occluded video objects. The visible mask segmentation/ visible mask region corresponds to the object initial mask template, and the predicted complete/ amodal object mask corresponds to the object target mask template. Ke also teaches using visible masks and predicted complete masks for the same object, and detecting/ recovering the occluded region of the object. Accordingly, Ke teaches comparing the visible object mask/ initial contour with the predicted complete object mask/ target contour to determine whether the object contour indicated by the initial mask is consistent with the completed object contour, wherein a difference between the visible mask and the complete/ amodal mask indicates an occluded or missing contour region of the second-type object. )
processing the second inpainting image based on the first comparison result, to obtain the target inpainting image.
( Ke, [§3], [§3.2], [§3.3], [Fig. 3], [Fig. 4], [Fig. 11]: Ke teaches that, after object shape completion, VOIN performs object flow completion under guidance of the amodal object contour and then performs flow-guided video object inpainting. Ke teaches that, with the completed object and completed flow within its contour, motion trajectories are used to warp pertinent pixels to inpaint corrupted frames. Ke further teaches filling remaining pixels/ occluded regions using an occlusion-aware gated generator guided by amodal object masks and occlusion masks. Thus, Ke teaches processing an inpainting image based on the comparison/ difference between the visible object mask and completed object mask, wherein the comparison result identifies whether an occluded/ missing object contour region exists, to obtain a target inpainting image in which the second-type object is restored. In the proposed combination, this object-aware processing is applied to the second inpainting image obtained from Lao [as modified by Schroers and Kudelski] to generate the target inpainting image.. )
Regarding claim 5, Lao [as modified by Schroers, Kudelski and Ke] teaches the method according to claim 4, wherein the processing the second inpainting image based on the first comparison result, to obtain the target inpainting image comprises:
performing, if the first comparison result indicates that the contours of the second-type objects are inconsistent, inpainting processing on a pixel region corresponding to the second-type object in the second inpainting image to obtain a third inpainting image, and using the third inpainting image as the target inpainting image; and
( Ke, [§3], [§3.1], [§3.2], [§3.3], [Fig. 2], [Fig. 3], [Fig. 4], [Fig. 11]: Ke teaches that the object shape completion module recovers the complete/ amodal shape of an occluded video object from its visible mask region. Ke further teaches that the completed object contour guides object flow completion and that flow-guided video object inpainting is performed to recover the occluded object region. Ke explains that the fill-up regions are restricted to occluded regions of the target object, which may be provided or produced by the object shape completion module. Therefore, when comparison between the object initial mask template and object target mask template indicates that the visible contour and completed contour are inconsistent, i.e., an occluded or missing portion of the second-type object exists, Ke teaches performing object-aware inpainting on the corresponding pixel region of the second-type object to recover the missing/occluded object appearance. In the proposed combination, this inpainting is performed on the second inpainting image to obtain a third inpainting image, and the third inpainting image is used as the target inpainting image. )
using the second inpainting image as the target inpainting image if the first comparison result indicates that the contours of the second-type objects are consistent.
( Ke, [§3], [§3.1], [§3.3], [Fig. 2], [Fig. 3], [Fig. 11]: Ke teaches that object-aware inpainting is directed to occluded or missing object regions identified from the relationship between the visible object mask and the completed/amodal object mask. Ke further teaches that the fill-up regions are restricted to occluded regions of the target object. Thus, if the visible object contour and the completed object contour are consistent, meaning no occluded or missing object region is identified for further object restoration, no further object-region inpainting is required. In the proposed combination, it would have been obvious to use the already-obtained second inpainting image as the target inpainting image when the comparison result indicates contour consistency, thereby avoiding unnecessary additional inpainting when the second-type object is already complete. )
Regarding claims 15–17, the rationale provided in the rejection of claims 1–2 and 11–12 is incorporated herein. In addition, Lao [as modified by Schroers, Kudelski and Ke] teaches a computer system or apparatus, including CPU/ processors/ memories/ RAM/ GPU/ PROM, to execute computer programs for image processing ( [Schroers: 0016]; [Kudelski: 0069] ). Accordingly, the method for image processing of claims 3–5 corresponds to the apparatus of claims 15–17, and performs the steps disclosed herein. Therefore, the claims are all rejected.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEN KUDO whose telephone number is (571)272-4498. The examiner can normally be reached M-F 8am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
KEN KUDO
Examiner
Art Unit 2671
/KEN KUDO/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights