DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This is in response to the applicant response filed on 10/08/2025. In the applicant’s response, claims 1-2, and 16-17 were amended. Accordingly, claims 1-20 are pending and being examined. Claims 1, 16, and 17 are independent form.
Claim Rejections - 35 USC § 102
3. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
4. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
5. Claims 1-5, 10, and 16 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Zhang et al (“Synthesis of High-Quality Visible Faces from Polarimetric Thermal Faces using Generative Adversarial Networks”, Int. J., of Computer Vision, 2019, hereinafter “Zhang”).
Regarding claim 1, Zhang discloses a method for cross-spectrum face recognition (the method of synthesizing High-Quality Visible Faces from Polarimetric Thermal Faces using Generative Adversarial Networks; see the title and fig.2), comprising:
receiving a visible image of a person (see the visible target (GT) faces in fig.3 (f)); receiving a cross-spectrum image of the person (see the thermal image faces in fig.3 (b)-fig.3(d));
training a neural network based on both the visible image and the cross-spectrum image (see the “GAN-based network” which is trained based on the thermal faces (i.e., the polarimetric images S0, S1, S2 shown in the training loss functions defined by Eqs.(3)-(6)) and the target face (i.e., the ground truth visible image Yt shown in the training loss functions defined by Eqs.(3)-(6)) in fig.2) by:
extracting a first representation from the visible image using a modified neural network architecture (see “the Discriminator” of the GAN-based network, which extracts the features from the target face shown by fig.2); extracting a second representation from the cross-spectrum image based on the modified neural network architecture (see “the Generator” of the GAN-based network, which is trained based the thermal faces and the target face and generates the estimated face shown by fig.2); converting the second representation to a modified representation using a domain adaptation sub-network, the domain adaptation sub-network comprising: a first residual block and a second residual block being immediately subsequent to the first residual block, the first residual block having the second representation as an input of the first residual block, the second residual block having the second representation and an output of the first residual block as an input of the second residual block, the modified representation being based on the second representation, the output of the first residual block (see “the Generator” shown in fig.2, and sec. 3.1, para.3, lines 11-24: “In addition, the output of each dense block is concatenated with the input of the corresponding dense block. Once we calculated features from all three streams, we concatenate together from all three branches along the depth (channel) dimension. Feature maps from each of the three streams are of size C × H × W. These feature maps are concatenated and are forwarded to the residual-fusion block, which consists of a res-block with 1×1 convolution layer. Then, the output of the residual-fusion block is regarded as the input for two different branches. To guarantee that the learned features contain geometric and textural facial information, a deep guidance sub-network (Xie and Tu 2015) is introduced at the end of the encoding part as one branch. The deep guided sub-network is part of the network that is branching out from the end of the encoder.” It should be noticed that: the output of the deep guidance sub-network [i.e., the 2nd] is the estimate face which is generated based on the output of the residual-fusion [i.e., the 1st] block and the output concatenated feature maps extracted from the thermal face images), and an output of the second residual block (see “Est” face “Generator” of fig.2; see fig.3(e)); and updating parameters of the neural network based on the modified representation (minimizing the loss defined by Eq(3); see sec.3.3).
Regarding claim 2, Zhang discloses the method of claim 1, wherein the visible image comprises a frontal face image of the person, and wherein the cross-spectrum image comprises an off-pose face image of the person, the off-pose face image being a frontal or non-frontal face image (see fig.3(a)-(d)).
Regarding claim 3, Zhang discloses the method of claim 1, further comprising: arranging a center of eyes, a nose, and corners of a mouth of the person to be disposed at a center of each of the visible image and the cross-spectrum image; and resizing the visible image and the cross-spectrum image to a predetermined size (see the second row of fig.6).
Regarding claim 4, Zhang discloses the method of claim 1, wherein the modified neural network architecture comprises a truncated trained neural network architecture (convolutional NN, see fig.5 and fig.5).
Regarding claim 5, Zhang discloses the method of claim 1, wherein the modified neural network architecture comprises a truncated visual geometry group with 16 layers (VGG16) (see VGG-16 in sec.3.3, para.1) or a truncated deep residual network with 50 layers (Resnet50).
Regarding claim 10, Zhang discloses the method of claim 1, wherein the first residual block comprises three 1 x1 convolutional layers, and wherein the second residual block comprises two 1 x 1 convolutional layers (sec. 3.1, para.3, lines 11-24).
Regarding claim 16, the scope of claim 16 is even broader than that of claim 1 and an inherent variation of claim 1, thus it is interpreted and rejected for the reasons set forth in the rejection of claim 1.
Claim Rejections - 35 USC § 103
6. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
7. Claims 6- 9 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang.
Regarding claims 6-9, although Zhang does not explicitly recite those features, they are obvious and common features existing in any VGG16 and/or Resnet50 which are widely used in the field of face recognition. It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was made to appreciate that those features are obvious and included in VGG16 and Resnet50. Therefore, the claim is unpatentable over Zhang.
8. Claims 17-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Mallat et al (“Cross-spectrum thermal to visible face recognition based on cascaded image synthesis”, IEEE, 2019, hereinafter “Mallat”).
Regarding claim 17, Zhang discloses the claimed invention expect for using off-pose images and converting the off-pose representation to a modified representation. In fact, Zhang is using front face images instead of off-pose face images for training and testing the neural networks. However, in the same field of endeavor, Mallat teaches that both of the front face images and the off-pose face image can be used to train neural networks for converting thermal images to visible images. See Abstract; and the 3rd row and the sixth row in fig.2. It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was made to incorporate the teachings of Mallat into the teachings of Zhang and train the NN of Zhang using off-pose face images taught by Mallat. Suggestion or motivation for doing so would have been to “generate visible-like colored images of high visual quality without requiring large amounts of training data” as thought by Mallat, see Abstract. Therefore, the claim is unpatentable over Zhang in view of Mallat.
Regarding claim 18, the combination of Zhang and Mallat discloses the method of claim 17, wherein the domain adaptation sub-network comprises a first residual block and a second residual block being immediately subsequent to the first residual block, the first residual block having the off-pose representation as an input of the first residual block, the second residual block having the off-pose representation and an output of the first residual block as an input of the second residual block, the modified representation being based on the off-pose representation, the output of the first residual block, and an output of the second residual block (Zhang , see “Generator” shown in fig.2, and sec. 3.1, para.3, lines 11-24: “In addition, the output of each dense block is concatenated with the input of the corresponding dense block. Once we calculated features from all three streams, we concatenate together from all three branches along the depth (channel) dimension. Feature maps from each of the three streams are of size C × H × W. These feature maps are concatenated and are forwarded to the residual-fusion block, which consists of a res-block with 1×1 convolution layer. Then, the output of the residual-fusion block is regarded as the input for two different branches. To guarantee that the learned features contain geometric and textural facial information, a deep guidance sub-network (Xie and Tu 2015) is introduced at the end of the encoding part as one branch. The deep guided sub-network is part of the network that is branching out from the end of the encoder.” It should be noticed that: the output of the deep guidance [i.e., 2nd] sub-network is the estimate face which is generated based on the output of the residual-fusion [i.e., 1st] block and the output concatenated feature maps extracted from the thermal face images).
Regarding claim 20, the combination of Zhang and Mallat discloses the method of claim 17, wherein the trained neural network comprises a truncated visual geometry group with 16 layers (VGG16) (Zhang, see “VGG-16” in sec.3.3, para.1) or a truncated deep residual network with 50 layers (Resnet50).
Allowable Subject Matter
9. The subject matter of claims 11-15, and 19, in combination with the base claim and intervening claims, were not found in the prior art of record. Claims 11-15, and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Response to Arguments
10. Applicant’s arguments, with respects to claim, filed on 10/08/2025, have been fully considered but they are not persuasive.
10-1 On page 9 of applicant’s response, regarding claim 1, applicant argues:
Zhang does not disclose or suggest various claimed features. For example, Zhang does not disclose or suggest at least "training a neural network based on both the visible image and the cross-spectrum image" as recited in Applicant's claim 1.
The examiner respectfully disagrees with the applicant’s arguments. As explained in the claim rejections, Zhang’s GAN-based network includes a generator, a discriminator sub-network, and a deep guided sub-network shown in fig.2, which is trained by using the thermal face images S0, S1, S2 and the corresponding ground truth (target) visible face image Yt based on the loss functions defined by Eqs.(3)-(6). Therefore, the argument is unpersuasive.
10-2 On page 9 of applicant’s response, regarding claim 1, applicant further argues:
Instead, Zhang's discriminator network receives the output of Zhang's generator network. Id at FIG. 2. However, Zhang's generator network takes only thermal images (SO-S2) as input and generates synthetic face images. Id These synthetic images are distinct from the actual images.
The examiner respectfully disagrees with the applicant’s arguments. It is because, the output of Zhang's generator network is not a “synthetic face image[s]” instead is an estimated visible face image (i.e., Est) which is similar the ground truth visible face image GT, as shown by fig.2. Therefore, the argument is unpersuasive.
10-3 On page 10 of applicant’s response, regarding claim 2, applicant further argues:
Applicant respectfully disagrees that these may be considered off-pose face images of the person. Applicant's disclosure states that, while off-pose face images may be frontal face images, frontal face images are considered "off-pose" if they have varying expressions or an object (such as glasses) on the person's face. See Applicant's disclosure at [0072] (numbering as published). Thus, not all frontal face images are off-pose. Zhang's frontal face images, all corresponding to the same expression and all having no object on the person's face, are not "off-pose" images. Claim 2 has been amended to clarify this difference.
The examiner respectfully disagrees with the applicant’s arguments. It is because the face images shown by fig.3(f) include face images having “varying expressions”, e.g., the top face and the bottom face in fig.3(f). Therefore, the argument is unpersuasive.
Conclusion
11. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
12. Any inquiry concerning this communication or earlier communications from the examiner should be directed to RUIPING LI whose telephone number is (571)270-3376. The examiner can normally be reached 8:30am--5:30pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, HENOK SHIFERAW can be reached on (571)272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit https://patentcenter.uspto.gov; https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center, and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RUIPING LI/Primary Examiner, Ph.D., Art Unit 2676