Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
In response to the amendments to independent claims 1, 10, and 15, the previously applied prior art rejections are withdrawn. However, upon further consideration, a new ground of rejection is made of independent claims 1, 10, and 15 under 35 U.S.C. § 103 as being unpatentable over Liu in view of Sheikh and in further view of He.
Claim 2 is noted as having been canceled by Applicant.
Response to Arguments
Applicant’s arguments, filed 21 January 2026 and taken in conjunction with the amendments to the claims, have been fully considered and are persuasive. As a result, the previously applied prior art rejections have been withdrawn. However, upon further consideration, a new ground of rejection is made of independent claims 1, 10, and 15 under 35 U.S.C. § 103 as being unpatentable over Liu in view of Sheikh and in further view of He.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-5, 7-11, and 13-19 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al. (“Self-Adaptively Learning to Demoiré from Focused and Defocused Image Pairs”, NeurIPS 2020, Vancouver, arXiv preprint arXiv:2011.02055, hereafter referred to as Liu) in view of Sheikh et al. (US PG Pub 20200186710, hereinafter “Sheikh”) and in further view of He et al. (“FHDe2Net: Full high definition demoireing network”. In European Conference on Computer Vision (pp. 713-729), hereafter referred to as He).
Regarding claim 1, Liu discloses a method comprising:
obtaining and identifying a first image and a second image (pg. 3, section “Joint Filtering”, para. 2, “moiré-free image only from a defocused and focused image pair”, wherein the image pair comprises the first and second images), wherein the first image comprises a Moiré pattern (pg. 5, section “Real Data”, “build a real dataset with 100 pairs, each with a focused moiré image and a defocused moiré-free image”, wherein the aforementioned first image is the focused moiré image); and
aligning the first image and the second image (pg. 6, “Image Alignment” subsection, wherein the images are aligned using a homography); and generating a corrected image by removing the Moiré pattern from the first image based on the second image and the alignment (Pgs. 7-8, “Comparison with State-of-the-Art”, and pgs. 12-13, Appendices B and C, wherein Appendix B is a representation of the algorithm whose output is the corrected/de-moiré-d image, and wherein Appendix C contains examples of the de-moiré-d images).
Although Liu describes the use of a phone with a triple image sensor setup including a higher-resolution close-range telephoto lens and alternate resolution wide and ultrawide lenses (pg. 6, Subsection “Image Capture”, to produce a wide variety of moiré patterns we use three types of smartphones…HUAWEI P30 PRO”), Liu does not explicitly disclose:
wherein the first image is obtained from a first sensor having a first resolution; or
wherein the second image is obtained from a second image sensor having a second resolution less than the first resolution.
However, Sheikh discloses:
wherein the first image is obtained from a first sensor having a first resolution (para. 0040-0042, wherein the first image is captured by the “center” image sensor having the higher acuity and resolution at the center of the field of view); and
wherein the second image is obtained from a second image sensor having a second resolution less than the first resolution (0040-0042, wherein the second image is captured one of either a higher pixel size, lower resolution image sensor having a better SNR or another lower-resolution camera enabling multi-frame moiré field of view).
Specifically, Sheikh discloses an imaging apparatus enabling positional control of different image sensors of an image array, each of which may be optimized to capture specific types of image, and the images which may be used for de-noising and de-moiré processing. The disclosure of Sheikh directed to multiple image sensors having different resolutions would be recognizable as an obvious improvement to the method and system of Liu by an ordinarily skilled artisan; specifically, the method and system of Sheikh would enable low-latency demoiréing of images using the method of Liu given simultaneous capture of focused (high-resolution) moiré image and defocused (low-resolution) moiré-free image pairs.
The combination of Liu in view of Sheikh fails to explicitly disclose wherein the generated corrected image has the first resolution upon removal of the Moiré pattern.
However, He discloses wherein the generated corrected image has the first resolution upon removal of the Moiré pattern (pgs. 4-5, section 2 and fig. 3, disclosing the rationale behind FHDe2 net, wherein the Moiré pattern is detected in high-definition images, and subsequently removed using downsampled and color-varying versions of the image and local and global refinement networks).
Specifically, He describes a method of neural network based high definition demoiréing using a custom network trained to perform both global and local reshaping along frequency sub-bands to remove moiré artifacts. Therefore, Liu and He both describe demoiréing methods within image frames with high frequency artifacts, utilizing a low-definition and high-definition image within the process (which would be capturable by the method of Sheikh). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the method and system of Liu and Sheikh within the environment of He, and utilized the video frame collection and real-time execution of He as the application of a known technique to a known device ready for improvement to yield the predictable result of a wider real-time application environment for the method of Liu modified by Sheikh, while enabling video demoiréing in addition to paired-image demoiréing.
Claims 10 and 15 are rejected, mutatis mutandis, for reasons similar to claim 1. Regarding claim 10, Liu further discloses a processor (pg. 6, Section 5, “experiments are conducted on a NVIDIA RTX 2080Ti GPU”); a memory storing one or more instructions and in electronic communication with the at least one processor (pg. 6, “Experiments” section, wherein the algorithm is implemented in PyTorch and executed by an NVIDIA RTX 2080Ti; and pg. 12 Appendix B, wherein the algorithm’s pseudocode as stored and executed by the computer necessitates the usage of a non-transitory computer readable medium); and wherein the at least one processor is further configured to execute the one or more instructions (pg. 6, “Experiments” section, wherein the algorithm is implemented in PyTorch and executed by an NVIDIA RTX 2080Ti; and pg. 12 Appendix B, wherein the algorithm’s pseudocode as stored and executed by the computer necessitates the usage of a non-transitory computer readable medium).Thus, it would have been obvious to the ordinarily skilled artisan to have combined the disclosures of Liu, Sheikh, and He according to claim 1.
Regarding claim 15, Liu further discloses a non-transitory computer readable medium storing code for image processing, the code comprising instructions executable by a processor (pg. 6, “Experiments” section, wherein the algorithm is implemented in PyTorch and executed by an NVIDIA RTX 2080Ti; and pg. 12 Appendix B, wherein the algorithm’s pseudocode as stored and executed by the computer necessitates the usage of a non-transitory computer readable medium).
Regarding claim 3, Liu in view of Sheikh and in further view of He describes all limitations of claim 1. Liu further describes cropping the second image based on a field of view of the first image to obtain a cropped second image, wherein the alignment is based on the cropped second image (pg. 6, “Image Capture” subsection, “As the focal length increases, the objects in the image will appear larger. We need to register each defocused and focused image pair. With the help of a white background, we first binarize the captured image to find its area and then crop it”; and pg. 5 fig. 2, wherein the defocused/non-moiré image is cropped to match a view of the focused/moiré image, and wherein the result is a cropped moiré-free image whose field of view matches the focused moiré image).
Regarding claim 4, Liu in view of Sheikh and in further view of He describes all limitations of claim 1. Liu further describes generating an alignment map between pixels of the first image and pixels of the second image, wherein the corrected image is generated based on the alignment map (pg. 6, “Image Capture” subsection, “We then align an image pair using the homography”; and pg. 5, figs. 2d and 2e, wherein fig. 2d shows the pixel-to-pixel correspondences for mapping the alignment, and fig. 2e shows the output of the alignment mapping process).
Regarding claim 5, Liu in view of Sheikh and in further view of He describes all limitations of claim 1. Liu further describes generating an aligned first image based on the first image; and generating an aligned second image based on the second image, wherein the corrected image is generated based on the aligned first image and the aligned second image (pg. 6, “Image Capture” subsection, “We then align an image pair using the homography”; and pg. 5, figs. 2d and 2e, wherein fig. 2d shows the pixel-to-pixel correspondences for mapping the alignment, and fig. 2e shows the output of the alignment mapping process, wherein both images are finally aligned as a result of their comparison with each other).
Regarding claim 7, Liu in view of Sheikh and in further view of He describes all limitations of claim 1. He further discloses detecting the Moiré pattern in the first image, wherein the corrected image is generated based on the detection of the Moiré pattern (pg. 4, fig. 3 description; and pgs. 6-10, sections 4.2-4.4, wherein the Moiré pattern is detected within the high-definition image due to both local and global sampling of image details, and the corrected high-definition image is generated as a result of global and local refinement applied to the first image). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have incorporated this particular disclosure of He within the method and system of Liu, Sheikh, and He according to the method of claim 1.
Regarding claim 8, Liu in view of Sheikh and in further view of He describes all limitations of claim 7. He further describes obtaining a plurality of frames of a video, wherein the plurality of frames includes the first image (pg. 5, Section 3 and figs. 3 and 4, “ground truth images are collected according to 18 categories of frequently observed contents on screens: wallpapers, sports video frames, film clips, documents, etc.”, any one of which may serve as the first image, and wherein the model architecture is specified to be able to perform real-time as an improvement over the state-of-the-art); and
selecting the first image from the plurality of video frames based on a detection frequency, wherein the Moiré pattern is detected based on the selection (pages 8-9, section 4.2 and figs. 6-7, wherein Frequency disentanglement and squeeze blocks are used to identify moiré and moiré-free image frames, and sub-bands of frequency are used to detect and classify moiré).
Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the disclosures of Liu, Sheikh, and He according to the method of claim 1.
Regarding claim 9, Liu in view of Sheikh and in further view of He describes all limitations of claim 8. Liu and Sheikh both describe the collection and usage of 2 images, while He further describes providing a test image to a correction neural network, wherein the corrected image is generated by the correction neural network (pg. 4, fig. 3, and pg. 6, Section 4, describing the feeding of the original image to the corrective network). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the neural network moiré correction method disclosed by He as the mechanism for the demoiréing method of Liu in view of Sheikh and in further view of He according to the rationale of claim 1.
Regarding claim 11, Liu in view of Sheikh and in further view of He describes all limitations of claim 10. Liu further discloses an alignment component configured to align the first image and the second image, wherein the corrected image is generated based on the alignment (pg. 6, “Image Capture” subsection, “We then align an image pair using the homography”; and pg. 5, figs. 2d and 2e, wherein fig. 2d shows the pixel-to-pixel correspondences for mapping the alignment, and fig. 2e shows the output of the alignment mapping process, wherein both images are finally aligned as a result of their comparison with each other).
Regarding claim 13, Liu in view of Sheikh and in further view of He describes all limitations of claim 10. Liu and Sheikh do not describe wherein the correction component comprises a correction neural network. However, He describes wherein the at least one processor is further configured to execute the one or more instructions to provide the first image and the second image to a correction neural network, wherein the corrected image is generated by the correction neural network (pg. 4, fig. 3, and pg. 6, Section 4, describing the feeding of the original image to the corrective network). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the neural network moiré correction method disclosed by He as the mechanism for the demoiréing method of Liu as modified by Sheikh according to the rationale of claim 1.
Regarding claim 14, Liu in view of Sheikh and in further view of He describes all limitations of claim 10. He further discloses wherein the at least one processor is further configured to execute the one or more instructions to detect the Moiré pattern in the first image, and wherein the corrected image is generated based on the detection of the Moiré pattern (pg. 4, fig. 3 description; and pgs. 6-10, sections 4.2-4.4, wherein the Moiré pattern is detected within the high-definition image due to both local and global sampling of image details, and the corrected high-definition image is generated as a result of global and local refinement applied to the first image). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have incorporated this particular disclosure of He within the method and system of Liu, Sheikh, and He according to the method of claim 1.
Regarding claim 16, Liu in view of Sheikh and in further view of He describes all limitations of claim 15. Sheikh further discloses wherein the code further comprises instructions executable by a processor to:
obtain the first image from a first sensor having the first resolution (para. 0040-0042, wherein the first image is captured by the “center” image sensor having the higher acuity and resolution at the center of the field of view); and
obtain the second image from a second image sensor having the second resolution less than the first resolution (0040-0042, wherein the second image is captured one of either a higher pixel size, lower resolution image sensor having a better SNR or another lower-resolution camera enabling multi-frame moiré field of view).
Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have incorporated this particular disclosure of Sheikh within the method and system of Liu, Sheikh, and He according to the method of claim 1.
Regarding claim 17, Liu in view of Sheikh and in further view of He discloses all limitations of claim 15. Liu further describes wherein the code further comprises instructions executable by the processor to crop the second image based on a field of view of the first image to obtain a cropped second image, wherein the alignment is based on the cropped second image (pg. 6, “Image Capture” subsection, “As the focal length increases, the objects in the image will appear larger. We need to register each defocused and focused image pair. With the help of a white background, we first binarize the captured image to find its area and then crop it”; and pg. 5 fig. 2, wherein the defocused/non-moiré image is cropped to match a view of the focused/moiré image, and wherein the result is a cropped moiré-free image whose field of view matches the focused moiré image).
Regarding claim 18, Liu in view of Sheikh and in further view of He discloses all limitations of claim 15. Liu further describes wherein the code further comprises instructions executable by the processor to generate an alignment map between pixels of the first image and pixels of the second image, wherein the corrected image is generated based on the alignment map (pg. 6, “Image Capture” subsection, “We then align an image pair using the homography”; and pg. 5, figs. 2d and 2e, wherein fig. 2d shows the pixel-to-pixel correspondences for mapping the alignment, and fig. 2e shows the output of the alignment mapping process).
Regarding claim 19, Liu in view of Sheikh and in further view of He discloses all limitations of claim 15. Liu further describes wherein the code further comprises instructions executable by the processor to generate an aligned first image based on the first image; and generate an aligned second image based on the second image, wherein the corrected image is generated based on the aligned first image and the aligned second image (pg. 6, “Image Capture” subsection, “We then align an image pair using the homography”; and pg. 5, figs. 2d and 2e, wherein fig. 2d shows the pixel-to-pixel correspondences for mapping the alignment, and fig. 2e shows the output of the alignment mapping process, wherein both images are finally aligned as a result of their comparison with each other).
Claims 6, 12, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Liu in view of Sheikh and He and in further view of Dai et al. (“Video Demoiréing with Relation-Based Temporal Consistency”. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 17622-17631), hereinafter “Dai”).
Regarding claims 6, 12, and 20, Liu in view of Sheikh and in further view of He describes all limitations of claims 1, 11, and 15, respectively. Liu describes image alignment, but the combination of Liu, Sheikh, and He does not describe the further limitations of claims 6, 12, and 20.
However, Dai describes wherein the code (implementing the method of claim 1) further comprises instructions executable by the processor to: identify a third image; and align the third image with the first image and the second image, wherein the corrected image is generated based on the third image (pgs. 17624-17625, Section 3.2, wherein the method takes in a third image input from the video stream as part of a three-image input series, performs image alignment of the three frames using downsampling and feature map alignment, and proceeds to correct/demoiré the image based on the alignment). Specifically, Dai describes a method and system for real-time video demoiréing which takes in a plurality of input frames and feeds them into a deep network comprising implicit frame alignment, feature aggregation, and convolution. Therefore, the combination of Liu, Sheikh, and He and Dai both describe frame demoiréing methods for image data. Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the three-frame input and alignment process of Dai within the method of Liu as modified by Sheikh and He as the application of a known technique to a known device to yield a predictable improvement; specifically, this modification would allow the method of Liu in view of Dai to demoiré video in real time rather than just still images.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROHAN TEJAS MUKUNDHAN whose telephone number is (571)272-2368. The examiner can normally be reached Monday - Friday 9AM - 6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 5712723838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROHAN TEJAS MUKUNDHAN/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights