Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
The status of claim 1-20 is:
Claims 1-20 are pending.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 08/14/2024, 02/07/2025, and 01/12/2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 9-11, and 19-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Gubbi Lakshminarasimha et al. (U.S. Patent Publication No 2022/0366618 presented in the IDS received 08/14/2024, hereinafter “Gubbi”).
Regarding claim 1, Gubbi discloses a method of performing video enhancement, the method comprising:
identifying a first image from among a plurality of images included in a first video (Gubbi Fig. 3A 304: “Identify one or more smoky video frames”), wherein the first image comprises an anomaly (Gubbi Fig. 3A 304: “Identify one or more smoky video frames”, smokiness is the anomaly);
obtaining guide information corresponding to the anomaly from among a plurality of types of guide information (Gubbi Fig. 3A 306: “Generate a smoky feature map for each smoky video of the one or more smoky video frames, using smoke relevant features of the corresponding smoky video frame, wherein the smoky feature map for each smoky video frame comprises features of the one or more precise smoky regions”);
obtaining a second image to replace the first image based on the guide information (Gubbi Fig. 3A 312: “Identify a smoke-free reference video frame for each smoky video frame of the one or more smoky video frames, out of the one or more smoke-free video frames, wherein the smoke-free reference video frame for each smoky video frame is a smoke-free video frame out of the one or more smoke-free video frames, present just before the corresponding smoky video frame”); and
obtaining a second video in which the anomaly has been removed by using the second image (Gubbi Fig. 3B 314: “Generate a de-smoked video frame for each smoky video frame of the one or more smoky video frames, by compensating color information obtained from the corresponding smoke-free reference video frame, locally in the one or more precise smoke regions of the corresponding intermediate de-smoked video frame”; Gubbi Fig. 3B 316: “Stitch the de-smoked video frame for each smoky video frame of the one or more smoky video frames, to obtain a de-smoked video in the real-time”).
Regarding claim 11, it is rejected under the same analysis as claim 1 above along with Gubbi’s disclosures of a device comprising: at least one memory storing one or more instructions (Gubbi [0008]: “In another aspect, there is provided a system for localized smoke removal and color restoration of a real-time video, the system comprising: a memory storing instructions”); and at least one processor configured to execute the one or more instructions (Gubbi [0008]: “and one or more hardware processors coupled to the memory via the one or more I/O interfaces”).
Regarding claim 20, it is rejected under the same analysis as claim 1 above along with Gubbi’s disclosure of a non-transitory computer readable medium having instructions stored therein (Gubbi [0009]: “In yet another aspect, there is provided a computer program product comprising a non-transitory computer readable medium having a computer readable program embodied therein”).
Regarding claim 9, Gubbi discloses the method, wherein the identifying the first image comprises:
extracting feature information from each of the plurality of images (Gubbi [0045]: “In an embodiment, the smoke video frame identification model is an artificial intelligence (AI) based model and is obtained by training a convolutional neural network with encoder-decoder architecture (For example, U-Net), with a predefined number of smoky frames and smoke-free frames, followed by testing to check an accuracy of the model after the training”, encoder-decoder AIs extract feature information in the encoder part of the AI);
identifying a presence of the anomaly in an image of the plurality of images based on the feature information (Gubbi [0045]: “The convolutional neural network with encoder-decoder architecture captures context and enables precise segmentation (localization) of the smoke regions in the video frames”); and
identifying the image that includes the anomaly as the first image (Gubbi Fig. 2: the smoke video frame identification unit (202) identifies the video and sorts the frames with anomalies to 204).
Regarding claim 19, it is rejected under the same analysis as claim 9 above.
Regarding claim 10, Gubbi discloses the method, wherein the identifying the presence of the anomaly comprises performing a certain operation to detect the anomaly based on the feature information or using an anomaly detection model using the feature information as an input (Gubbi [0045]: “In an embodiment, the smoke video frame identification model is an artificial intelligence (AI) based model and is obtained by training a convolutional neural network with encoder-decoder architecture (For example, U-Net), with a predefined number of smoky frames and smoke-free frames, followed by testing to check an accuracy of the model after the training. The convolutional neural network with encoder-decoder architecture captures context and enables precise segmentation (localization) of the smoke regions in the video frames. The convolutional neural network with encoder-decoder architecture can be trained with lesser number of smoky video frames and smoke-free video frames. The smoke video frame identification model may be present in the smoke video frame identification unit 202 of FIG. 2. The unique frame index of each smoky video frame of the one or more smoky video frames, and each smoke-free video frame of the one or more smoke-free video frames, are stored in the repository 102b of the system 100 for subsequent use”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Gubbi in view of Obonai et al. (WO 2023095250 A1 using the translation provided herein, hereinafter “Obonai”).
Regarding claim 2, Gubbi discloses the method, wherein the obtaining guide information comprises:
generating, based on a type of the anomaly, first guide information from among a guide image, a guide mask, and a guide feature (Gubbi Fig. 3A 306: “Generate a smoky feature map for each smoky video of the one or more smoky video frames, using smoke relevant features of the corresponding smoky video frame, wherein the smoky feature map for each smoky video frame comprises features of the one or more precise smoky regions”).
Gubbi does not explicitly disclose the method, wherein the obtaining guide information comprises:
identifying whether the first guide information is valid; and
based on a result of identifying whether the first guide information is valid, identifying the first guide information as the guide information or identifying second guide information that is different from the first guide information as the guide information.
However, Obonai teaches the method, wherein the obtaining guide information comprises:
generating, based on a type of the anomaly, first guide information from among a guide image, a guide mask (Obonai Page 5: “The mask generation unit 202 reads parameters such as the size, shape, number, and slide amount of the masks stored in the auxiliary storage unit 207 and generates a mask 605 . The mask 605 is positioned differently for each applied image. In FIG. 6, one square mask is formed by changing the position in order for each image to be applied. As a result, n patterns of mask patterns 620-1 to 620-n are generated. Preferably, these patterns together can cover all locations in the image. The shape of the mask 605 is rectangular, which is suitable for sequentially masking the entire area, but the shape is not limited to this, and a specific shape can also be applied”), and a guide feature;
identifying whether the first guide information is valid (Obonai Page 5: “Next, the abnormality determination unit 205 compares the input image 610 with the reconstructed images 640-1 to 640-n reconstructed from the masked images 630-1 to 630-n. This comparison calculates the error in each region of the mask 605, and if there is an error equal to or greater than a certain threshold, it is determined that there is an "abnormality"”); and
based on a result of identifying whether the first guide information is valid, identifying the first guide information as the guide information (Obonai Page 5: “If the range (the number of pixels) is greater than or equal to a predetermined range, it can be determined as an abnormal portion. Note that the reference value for binarization can be a pixel value difference of a predetermined value or more (for example, a predetermined pixel value or more), and a suitable value can be used. FIG. 6 shows the binarized error images 650-1 to 650-n, and the error image 650-m shows a difference in the abnormal 602 portion”) or identifying second guide information that is different from the first guide information as the guide information (Obonai Page 5: “If all the errors are less than the threshold, it is determined that there is no abnormality”).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the multiple masks as taught by Obonai with the method of Gubbi because it would improve the method by allowing it to check the entire image with masks to determine which areas have abnormalities, and fix those abnormalities. This motivation for the combination of Gubbi and Obonai is supported by KSR exemplary rationale (D) Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results.
Regarding claim 12, it is rejected under the same analysis as claim 2 above.
Allowable Subject Matter
Claims 3-8 and 13-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AIDAN KEUP whose telephone number is (703)756-4578. The examiner can normally be reached Monday - Friday 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AIDAN KEUP/Examiner, Art Unit 2666 /Molly Wilburn/Primary Examiner, Art Unit 2666
Prosecution Insights