DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on September 27, 2024 and September 15, 2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-6, 8-13 and 15-19 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Seong, H., Oh, S.W., Price, B., Kim, E., Lee, JY. (2022). One-Trimap Video Matting. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13689. Springer, Cham. https://doi.org/10.1007/978-3-031-19818-2_25 (hereinafter Seong).
Regarding independent claim 1, Seong discloses A method (abstract, “we push this task toward a more practical setting and propose One-Trimap Video Matting network (OTVM) that performs video matting robustly using only one user-annotated trimap”) comprising:
receiving a video sequence (see Figure 1, “inputs” being multiple frames, and the labeled tripmap) and an input masked video frame for a first video frame of the video sequence (abstract, “we push this task toward a more practical setting and propose One-Trimap Video Matting network (OTVM) that performs video matting robustly using only one user-annotated trimap”);
generating alpha matte frames for the video sequence using the video sequence and the input masked video frame (Figure 1, the alpha prediction), wherein a first network generates masked video frames based on stored features of previous frames of the video sequence (page 7, “Therefore, we first separately train two modules without the connections between two. Specifically, we train the alpha matting module with GT trimaps”) and a second network generates the alpha matte frames based on the masked video frames and the stored features of the previous frames of the video sequence (Figure 1, the output of the previous frame is used as an input to the trimap propagation for predicting the new map and outputs); and
outputting an alpha matte video sequence representation of the video sequence which includes the generated alpha matte frames (Figure 1, the outputs in the last layer are read as the sequence).
Regarding dependent claim 2, the rejection of claim 1 is incorporated herein. Additionally, Seong in the combination further discloses wherein generating the alpha matte frames for the video sequence using the video sequence and the input masked video frame further comprises:
receiving, by the first network, the first video frame of the video sequence and the input masked video frame for the first video frame of the video sequence (Figure 1, input the frame and a trimap from a user to the alpha prediction and further the refinement);
generating, by the first network, a first masked video frame based on the first video frame of the video sequence and the input masked video frame for the first video frame of the video sequence (Figure 1 abstract, “A key of OTVM is the joint modeling of trimap propagation and alpha prediction. Starting from baseline trimap propagation and alpha prediction networks, our OTVM combines the two networks with an alpha-trimap refinement module to facilitate information flow”);
generating, by the second network, a first alpha matte frame based on the first masked video frame (abstract, “A key of OTVM is the joint modeling of trimap propagation and alpha prediction. Starting from baseline trimap propagation and alpha prediction networks, our OTVM combines the two networks with an alpha-trimap refinement module to facilitate information flow”); and
updating a frames features memory and a matting memory with at least features of the first alpha matte frame (the outputs from each process are fed to the next frame processing; page 5, “he generated trimap and predicted alpha matte via two light-weighted residual blocks. The refined trimap, alpha matte, and hidden features (dimension = 16) are fed to the trimap memory encoder so that they can be used for the next frames as a new memory.”).
Regarding dependent claim 3, the rejection of claim 2 is incorporated herein. Additionally, Seong in the combination further discloses wherein updating the frames features memory and the matting memory with at least the features of the first alpha matte frame further comprises:
passing the first video frame of the video sequence and the first alpha matte frame through an encoder to generate the features of the first alpha matte frame (Figure S1, the input of the RBG, trimap, and alpha are input into th eTrimap Encoder M; page 18, “We employed two independent ResNet50 [20] encoders to embed memory and query.”); and
storing the features of the first alpha matte frame in the matting memory (page 18, “We employed two independent ResNet50 [20] encoders to embed memory and query.” Page 9, “We used full-resolution inputs to achieve high-quality alpha matte results. For the memory management in the trimap propagation module, we generally follow STM [39] that stores the first and the previous frames to the memory by default, and additionally saves new memory periodically. We add the intermediate frames to the memory for every 10 frames. To avoid GPU memory overflow, we store only the last three intermediate frames and discard old frames.”).
Regarding dependent claim 4, the rejection of claim 2 is incorporated herein. Additionally, Seong further discloses further comprising:
updating the frames features memory with second features representing the first masked video frame (page 9, “we generally follow STM [39] that stores the first and the previous frames to the memory by default, and additionally saves new memory periodically”).
Regarding dependent claim 5, the rejection of claim 2 is incorporated herein. Seong further discloses further comprising:
consecutively processing each additional video frame of the video sequence by the first network and the second network to generate corresponding alpha matte frames (Figure 1, each frame is read as being processed consecutively; Figure 3, “Overall architecture of OTVM. Our trimap propagation network is inspired by space-time memory networks [39]. The network predicts a trimap based on the information from the previous frames and predictions that are embedded by the trimap memory encoder.”).
Regarding dependent claim 6, the rejection of claim 5 is incorporated herein. Additionally, Seong in the combination further discloses wherein consecutively processing each additional video frame comprises:
generating, by the first network, a next masked video frame for a next video frame of the video sequence using the stored features of the previous frames of the video sequence (Figure 1, the map and alpha are input into the processing for the next frame), wherein the next video frame is consecutive to a previous video frame, and wherein the stored features of the previous frames of the video sequence includes at least the first alpha matte frame representing the first video frame of the video sequence (Figure 1);
generating, by the second network, a next alpha matte frame representing the next video frame of the video sequence using the next masked video frame for the next video frame of the video sequence and the stored features of the previous frames of the video sequence (Figure 1, processing of the seond frame); and
updating the frames features memory and the matting memory with at least features of the next alpha matte frame (page 9, “we generally follow STM [39] that stores the first and the previous frames to the memory by default, and additionally saves new memory periodically”).
Regarding dependent claim 8, the rejection of claim 1 is incorporated herein. Additionally, Seong further discloses wherein the input masked video frame designates each pixel of the first video frame of the video sequence as a foreground pixel or a background pixel (abstract, “In this paper, we push this task toward a more practical setting and propose One-Trimap Video Matting network (OTVM) that performs video matting robustly using only one user-annotated trimap.” page 1, “Video matting is the task of predicting accurate alpha mattes from a video. This is an essential step in video editing applications requiring an accurate separation of the foreground and the background layers such as video composition. For each video frame I, it aims to divide the input color into three components: the foreground color, the background color, and the alpha matte.”).
Regarding independent claim 9, the rejection of claim 1 applies directly. Additionally, Seong further discloses A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations (abstract, “we push this task toward a more practical setting and propose One-Trimap Video Matting network (OTVM) that performs video matting robustly using only one user-annotated trimap;” the complexity of this network is read as needing to be implemented on a computer and programed to the desired parameters to be stored in memory and called upon for execution) comprising:
receiving a video sequence (see Figure 1, “inputs” being multiple frames, and the labeled tripmap) and an input masked video frame for a first video frame of the video sequence (abstract, “we push this task toward a more practical setting and propose One-Trimap Video Matting network (OTVM) that performs video matting robustly using only one user-annotated trimap”);
generating alpha matte frames for the video sequence using the video sequence and the input masked video frame (Figure 1, the alpha prediction), wherein a first network generates masked video frames based on stored features of previous frames of the video sequence (page 7, “Therefore, we first separately train two modules without the connections between two. Specifically, we train the alpha matting module with GT trimaps”) and a second network generates the alpha matte frames based on the masked video frames and the stored features of the previous frames of the video sequence (Figure 1, the output of the previous frame is used as an input to the trimap propagation for predicting the new map and outputs); and
outputting an alpha matte video sequence representation of the video sequence which includes the generated alpha matte frames (Figure 1, the outputs in the last layer are read as the sequence).
Regarding dependent claim 10, the rejection of claim 9 is incorporated herein. Additionally, Seong in the combination further discloses wherein the instructions to generate the alpha matte frames for the video sequence using the video sequence and the input masked video frame further comprise:
receiving, by the first network, the first video frame of the video sequence and the input masked video frame for the first video frame of the video sequence (Figure 1, input the frame and a trimap from a user to the alpha prediction and further the refinement);
generating, by the first network, a first masked video frame based on the first video frame of the video sequence and the input masked video frame for the first video frame of the video sequence (Figure 1 abstract, “A key of OTVM is the joint modeling of trimap propagation and alpha prediction. Starting from baseline trimap propagation and alpha prediction networks, our OTVM combines the two networks with an alpha-trimap refinement module to facilitate information flow”);
generating, by the second network, a first alpha matte frame based on the first masked video frame (abstract, “A key of OTVM is the joint modeling of trimap propagation and alpha prediction. Starting from baseline trimap propagation and alpha prediction networks, our OTVM combines the two networks with an alpha-trimap refinement module to facilitate information flow”); and
updating a frames features memory and a matting memory with at least features of the first alpha matte frame (the outputs from each process are fed to the next frame processing; page 5, “he generated trimap and predicted alpha matte via two light-weighted residual blocks. The refined trimap, alpha matte, and hidden features (dimension = 16) are fed to the trimap memory encoder so that they can be used for the next frames as a new memory.”).
Regarding dependent claim 11, the rejection of claim 10 is incorporated herein. Additionally, Seong further discloses wherein the instructions further comprise:
updating the frames features memory with second features representing the first masked video frame (page 9, “we generally follow STM [39] that stores the first and the previous frames to the memory by default, and additionally saves new memory periodically”).
Regarding dependent claim 12, the rejection of claim 10 is incorporated herein. Seong further discloses further wherein the instructions further comprise:
consecutively processing each additional video frame of the video sequence by the first network and the second network to generate corresponding alpha matte frames (Figure 1, each frame is read as being processed consecutively; Figure 3, “Overall architecture of OTVM. Our trimap propagation network is inspired by space-time memory networks [39]. The network predicts a trimap based on the information from the previous frames and predictions that are embedded by the trimap memory encoder.”).
Regarding dependent claim 13, the rejection of claim 12 is incorporated herein. Additionally, Seong in the combination further discloses wherein the instructions to consecutively process each additional video frame further comprise:
generating, by the first network, a next masked video frame for a next video frame of the video sequence using the stored features of the previous frames of the video sequence (Figure 1, the map and alpha are input into the processing for the next frame), wherein the next video frame is consecutive to a previous video frame, and wherein the stored features of the previous frames of the video sequence includes at least the first alpha matte frame representing the first video frame of the video sequence (Figure 1);
generating, by the second network, a next alpha matte frame representing the next video frame of the video sequence using the next masked video frame for the next video frame of the video sequence and the stored features of the previous frames of the video sequence (Figure 1, processing of the seond frame); and
updating the frames features memory and the matting memory with at least features of the next alpha matte frame (page 9, “we generally follow STM [39] that stores the first and the previous frames to the memory by default, and additionally saves new memory periodically”).
Regarding independent claim 15, the rejection of claim 1 applies directly. Additionally, Seong further discloses A system (abstract, “we push this task toward a more practical setting and propose One-Trimap Video Matting network (OTVM) that performs video matting robustly using only one user-annotated trimap”) comprising:
a memory component (abstract, “we push this task toward a more practical setting and propose One-Trimap Video Matting network (OTVM) that performs video matting robustly using only one user-annotated trimap;” the complexity of this network is read as needing to be implemented on a computer and programed to the desired parameters to be stored in memory and called upon for execution); and
a processing device coupled to the memory component (abstract, “we push this task toward a more practical setting and propose One-Trimap Video Matting network (OTVM) that performs video matting robustly using only one user-annotated trimap;” the complexity of this network is read as needing to be implemented on a computer), the processing device to perform operations comprising:
receiving a video sequence (see Figure 1, “inputs” being multiple frames, and the labeled tripmap) and an input masked video frame for a first video frame of the video sequence (abstract, “we push this task toward a more practical setting and propose One-Trimap Video Matting network (OTVM) that performs video matting robustly using only one user-annotated trimap”);
generating alpha matte frames for the video sequence using the video sequence and the input masked video frame (Figure 1, the alpha prediction), wherein a first network generates masked video frames based on stored features of previous frames of the video sequence (page 7, “Therefore, we first separately train two modules without the connections between two. Specifically, we train the alpha matting module with GT trimaps”) and a second network generates the alpha matte frames based on the masked video frames and the stored features of the previous frames of the video sequence (Figure 1, the output of the previous frame is used as an input to the trimap propagation for predicting the new map and outputs); and
outputting an alpha matte video sequence representation of the video sequence which includes the generated alpha matte frames (Figure 1, the outputs in the last layer are read as the sequence).
Regarding dependent claim 16, the rejection of claim 15 is incorporated herein. Additionally, Seong in the combination further discloses wherein the operations of generating the alpha matte frames for the video sequence using the video sequence and the input masked video frame further comprise:
receiving, by the first network, the first video frame of the video sequence and the input masked video frame for the first video frame of the video sequence (Figure 1, input the frame and a trimap from a user to the alpha prediction and further the refinement);
generating, by the first network, a first masked video frame based on the first video frame of the video sequence and the input masked video frame for the first video frame of the video sequence (Figure 1 abstract, “A key of OTVM is the joint modeling of trimap propagation and alpha prediction. Starting from baseline trimap propagation and alpha prediction networks, our OTVM combines the two networks with an alpha-trimap refinement module to facilitate information flow”);
generating, by the second network, a first alpha matte frame based on the first masked video frame (abstract, “A key of OTVM is the joint modeling of trimap propagation and alpha prediction. Starting from baseline trimap propagation and alpha prediction networks, our OTVM combines the two networks with an alpha-trimap refinement module to facilitate information flow”); and
updating a frames features memory and a matting memory with at least features of the first alpha matte frame (the outputs from each process are fed to the next frame processing; page 5, “he generated trimap and predicted alpha matte via two light-weighted residual blocks. The refined trimap, alpha matte, and hidden features (dimension = 16) are fed to the trimap memory encoder so that they can be used for the next frames as a new memory.”).
Regarding dependent claim 17, the rejection of claim 16 is incorporated herein. Additionally, Seong further discloses wherein the operations further comprise:
updating the frames features memory with second features representing the first masked video frame (page 9, “we generally follow STM [39] that stores the first and the previous frames to the memory by default, and additionally saves new memory periodically”).
Regarding dependent claim 18, the rejection of claim 16 is incorporated herein. Seong further discloses wherein the operations further comprise:
consecutively processing each additional video frame of the video sequence by the first network and the second network to generate corresponding alpha matte frames (Figure 1, each frame is read as being processed consecutively; Figure 3, “Overall architecture of OTVM. Our trimap propagation network is inspired by space-time memory networks [39]. The network predicts a trimap based on the information from the previous frames and predictions that are embedded by the trimap memory encoder.”).
Regarding dependent claim 19, the rejection of claim 18 is incorporated herein. Additionally, Seong in the combination further discloses wherein the operations of consecutively processing each additional video frame further comprise:
generating, by the first network, a next masked video frame for a next video frame of the video sequence using the stored features of the previous frames of the video sequence (Figure 1, the map and alpha are input into the processing for the next frame), wherein the next video frame is consecutive to a previous video frame, and wherein the stored features of the previous frames of the video sequence includes at least the first alpha matte frame representing the first video frame of the video sequence (Figure 1);
generating, by the second network, a next alpha matte frame representing the next video frame of the video sequence using the next masked video frame for the next video frame of the video sequence and the stored features of the previous frames of the video sequence (Figure 1, processing of the seond frame); and
updating the frames features memory and the matting memory with at least features of the next alpha matte frame (page 9, “we generally follow STM [39] that stores the first and the previous frames to the memory by default, and additionally saves new memory periodically”).
Allowable Subject Matter
Claims 7, 14 and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the closest prior arts of record teach methods of processing video data to generate alpha mattes.
However, none of them alone or in any combination teaches generating an initial alpha matte by inputting the video frame and first masked video frame through the second network and also generating the first alpha matte frame through the same process, where the features in the matting memory are combined with features of the first video frame and the first masked video frame in the decoder layer.
The closest prior art being Seong discloses, “we push this task toward a more practical setting and propose One-Trimap Video Matting network (OTVM) that performs video matting robustly using only one user-annotated trimap. A key of OTVM is the joint modeling of trimap propagation and alpha prediction. Starting from baseline trimap propagation and alpha prediction networks, our OTVM combines the two networks with an alpha-trimap refinement module to facilitate information flow (abstract).”
However, Seong fails to disclose generating an initial alpha matte by inputting the video frame and first masked video frame through the second network and also generating the first alpha matte frame through the same process, where the features in the matting memory are combined with features of the first video frame and the first masked video frame in the decoder layer.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
U.S. Publication No. 2023/0044969 to Yang et al. discloses, “ techniques of improving video matting (abstract)”
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Courtney J. Windsor whose telephone number is (571)272-3956. The examiner can normally be reached Monday - Friday 8:00 - 4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/COURTNEY JOAN NELSON/Primary Examiner, Art Unit 2661
Prosecution Insights