Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claims 4-5 and 13-14 are rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. Claims 4 and 13, which depend on claims 1 and 10 respectively, recite “wherein the model architecture includes a U-net model”. This limitation, however, is recited in claims 1 and 10. Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 9-14, 18, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hu et al (US 2023/0094679 A1) in view of Kwatra et al (US 8,798,393 B2).
As to claim 1, Hu et al disclose a satellite image sequence region repairing method and apparatus that identifies cloud and shadow regions from multiple multispectral images such that pixel values of repair regions are replaced with pixel values of a non-cloud region of a reference image (see Title and Abstract). Hu et al disclose: A computer-implemented method for use in mitigating shadow segments from images, the method comprising: accessing, by a computing device, an original image of a geospatial location, the original image including a first shadow segment; generating, by the computing device, using a model architecture, a matte for the original image, based on the image and a shadow mask for the image (see Figure 1, a method for patching a cloud area of a satellite image sequence, and Figure 2, a flowchart of automatic cloud patching, and the respective descriptions; cloud areas and cloud shadow areas are identified from each of a plurality of multispectral images of a target area, reference images required for patching and a patching sequence are calculated, patching areas of the multispectral image are determined according to a cloud mask image (S11-S14 and S201-S203; see [0101], algorithm for cloud and cloud shadow detection). Hu et al further disclose their process of direct replacement cloud patching (S14-S16 and S204-S217), resulting in a multispectral image cloud patching product. The limitation: “…and storing the shadow-mitigated image in a memory” is met by S217 and [0152-0157]. As indicated above, the reference discloses an algorithm for cloud/shadow detection but fails to disclose: the model architecture including a U-Net model. However, this is not considered to be a patentable distinction. The examiner takes Official Notice that it was notoriously well-known in the art prior to the effective filing date of the invention to use a U-net model, a type of Convolutional Neural Network (CNN), for image segmentation. It would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the invention to modify Hu et al with a U-net model in order to ensure high accuracy in object boundaries. Further, Hu et al treat clouds and cloud shadows through patching/direct replacement, and therefore fail to disclose: (a) generating, by the computing device, a first histogram of tones of shadow pixels of said first shadow segment included in the original image; (b) generating, by the computing device, a second histogram of tones of an adjacent region of the original image, which is proximate to the first shadow segment, yet outside a boundary of the first shadow segment; (c) defining, by the computing device, a lookup table based on histogram matching between the first histogram and the second histogram; (d) relighting, by the computing device, the original image based on the lookup table; (e) based on the shadow mask and the matte, generating a shadow-mitigated image from the relighted image of the original image, whereby the first shadow segment in the original image is mitigated. Kwatra et al teach a method for removing illumination variation from images. The reference discloses that the images may be aerial images containing undesirable shadows caused by clouds or other objects (col. 1, lines 19-32). The reference uses a training module 103 to train an initialization module 101 to identify pixel clusters that represent non-illuminated (non-shadowed) portions of images (see col. 6, lines 3-26). A similarity module 105 determines the similarity of pixels in illuminated images to determine illumination values. Similarity module 105 identifies local pixel neighbors for each pixel, spatially located directly adjacent and forming a patch (see Figure 2, patches P(1753, 1352, 2233). It then compares shadowed and non-shadowed patches to determine similarity based on the difference of reflectance of, for example, pixel i of Patch P(i) and pixel j of Patch P(j) (see col. 6, line 27, to col. 7, line 40). As can be seen from Figure 2, Patches P(1352) and P(2233) are proximate to but outside a boundary of the shadow segment having Patch P(1753). The similarity module 105 determines a histogram of reflectance values of pixels in Patch P(i) and Patch P(j) and determines a weight for the similarity between patches (see. Col. 7, line 41, to col. 8, line 41). Optimization module 109 applies an energy function E to solve to solve for illumination values. The energy function smoothes the illumination values within a region of the image corresponding to Patch P(i), operating under the assumption that the illumination values of pixels within the path P(i) should be of equivalent or similar intensity (see col. 8, line 42, to col. 9, line 43). Kwatra et al is silent with respect to ‘defining… a lookup table based on histogram matching’, instead only indicating that an energy function solves for illumination values. However, this is not considered to be a patentable distinction. The examiner takes Official Notice that it was notoriously well-known in the art prior to the effective filing date of the invention to use a lookup table to map input values to precomputed output values, trading memory with processing time. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to implement Kwatra et al in this manner for the stated advantage. Turning again to Hu et al, cloud/cloud shadow patching requires multispectral data without cloud cover (Hu, [0102]). Accordingly, it would have been further obvious to one of ordinary skill in the art prior to the effective filing date of the invention to modify Hu et al with the method for removing illumination variations taught by Kwatra et al in order to mitigate cloud shadows without having to rely on reference images.
Claim 2 is met by that discussed above for claim 1.
As to claim 3, it is clear from the disclosure of both Hu et al and Kwatra et al, as discussed above in the rejection of claim 1, that each of perhaps plural shadow segments are patched or corrected for illumination variations.
Claim 4 is met by that discussed above for claim 1.
As to claim 5, Hu et al, as modified by the taking of Official Notice, as modified by Kwatra et al fails to disclose: The computer-implemented method of claim 4, wherein generating the matte includes generating the matte further based on a gain and bias from a ResNeXt regressor model of the model architecture. However, this is not considered to be a patentable distinction. The examiner takes Official Notice that it was notoriously well-known in the art prior to the effective filing date of the invention to include a ResNeXt network as the encoder backbone to a U-net, such that the final output layer is adapted for regression tasks. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to include a ResNeXt model in the U-net model of the combination for the key benefit of stronger feature extraction and pixel-level accuracy.
As to claim 9: The computer-implemented method of claim 1, wherein generating the first histogram includes counting pixel intensities for each of multiple tonal ranges is met as discussed above in the rejection of claim 1. As is evident from Figure 2 of Kwatra et al, patch P(1753) and either patch P(1352) or P(2233) constitute multiple tonal ranges.
Claims 10-14 and 18 are met by that discussed above for claim 1-5 and 9, respectively.
Claim 19 is met by that discussed above in the rejection of claim, except for the limitation of “a model architecture having a SP-net and an M-Net”. However, adapting a U-net architecture with SP-Net and M-Net is not considered to be a patentable distinction. The examiner takes Official Notice that it was notoriously well-known in the art prior to the effective filing date of the invention to adapt the U-net backbone with these added features to help solve the image decomposition tasks of predicting illumination parameters and shadow matte layers. It would have been obvious to one of ordinary skill in the art prior to the effective filing date of the invention to further modify the combination with these improvements for the stated advantages.
Allowable Subject Matter
Claim 6-8 and 15-17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: with respect to claims 6 and 15, Kwatra et al disclose a training model trained to identify pixel clusters using a training set of illuminated images, such that each image from the training set includes an indication (e.g. a label) of non-illuminated (non-shadowed) portions of the image (see col. 6, lines 3+). Accordingly, Kwatra et al teaches training with images having shadows. However, neither Hu et al nor Kwatra et al explicitly teach or suggest “a training set, which includes training shadow-free images, corresponding training shadow-added images, which correspond to the training shadow-free images, and corresponding training shadow masks for shadow segments(s) in the training shadow-added images”.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN W MILLER whose telephone number is 571-272-7353. The examiner can normally be reached Monday - Friday 7:30 AM - 4:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Colleen Fauz can be reached at 571-272-1667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN W MILLER/Supervisory Patent Examiner, Art Unit 2422
Prosecution Insights