DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s amendment filed April 24th 2026 has been entered and made of record. Claims 1-19 are amended. Claims 1-20 are pending.
Applicant’s remarks in view of the newly presented amendments have been considered but are not found to be persuasive for at least the following reasons:
Applicant argues that the reference to Zhang does not disclose the newly amended claim limitations. Examiner disagrees. The newly added claim limitations are addressed below. The rejection in view of Zhang is maintained and accordingly made FINAL.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-11, 13-16 and 18-19 are rejected under 35 U.S.C. 102(a) as being anticipated by USPN 2021/0174072 to Zhang et al.
With regard to claim 1, Zhang discloses a processor (paragraphs [0017]-[0019], [0161]-[0162] Fig. 9, processor 1001 comprising:
one or more circuits (Fig. 7) to:
cause one or more first neural networks to generate one or more images comprising one or more features or attributes (paragraphs [0107]-[0108] and Fig. 7, generative model 30d, Generative model 30d includes models 30b and 30c which generate synthetic facial images by exaggerating facial expressions to create exaggerated images of facial micro-expressions), wherein the one or more features or attributes are identified according to a root cause determined for one or more performance metrics of one or more second neural networks that do not satisfy a threshold of performance (paragraphs [0107]-[0108], and Fig. 7, neural networks 30g and 30h perform facial recognition and discrimination on the generated augmented facial images. The neural networks 30g and 30h are used to train the generative model 30d by calculating error function 30k, which consists of loss values for the models and adjusting the weights of the models thereby training the generative models. With regard to a root cause being determined for the performance metric, Applicant’s specification only mentions a shortage of training images as a root cause for poor performance. For most facial recognition neural networks, if the system is not meeting performance metrics, the “root cause” is assumed to be a lack of training data. In a system that such as Zhang that is focused on generating synthetic images for the purpose of training the system, it follows that the intention is to generate more training images); and
update the second one or more neural networks based, at least in part, on the one or more images (paragraphs [0107]-[0108], and Fig. 7, The second neural networks 30g and 30h are also accordingly updated based on the error function calculator 30k. “The error function calculator 30k combines the three loss values into a model loss value, adjusts a weight of a parameter in the sample generative model 30d (the first generative sub-model 30b and the second generative sub-model 30c) according to the model loss value, adjusts a weight of a parameter in the sample discriminative model 30g according to the model loss value, and adjusts a weight of a parameter in the sample recognition model 30h according to the model loss value.” Weight values are adjusted in the second neural networks 30g and 30h based on the error function calculator and are thus updated based on the generated images.).
With regard to claim 2, Zhang discloses the processor of claim 1, wherein the one or more second neural networks are to perform facial recognition (Fig. 7, neural networks 30g and 30h recognize facial expressions).
With regard to claim 3, Zhang discloses the processor of claim 1, wherein the one or more first neural networks are to generate the one or more images based, at least part, on the one or more reference images (The neural networks in 30d generate augmented facial expression images 30e and 3f for example from reference image 30a. See paragraphs [0107]-[0109]).
With regard to claim 4, Zhang discloses the processor of claim 1, wherein the root cause indicates a shortage of images with one or more features of attributes used to train the second one or more neural networks (paragraphs [0107]-[0109], With regard to a root cause being determined for the performance metric, Applicant’s specification only mentions a shortage of training images as a root cause for poor performance. For most facial recognition neural networks, if the system is not meeting performance metrics, the “root cause” is assumed to be a lack of training data. In a system that such as Zhang that is focused on generating synthetic images for the purpose of training the system, it follows that the intention is to generate more training images. Zhang accordingly teaches that synthetic facial images are generated in an effort to train facial expression recognition and further teaches that different features are used to train the recognition as shown in Fig. 2. Different facial features and their attributes are used to generate training images when there is a lack or shortage of such training images).
With regard to claim 5, Zhang discloses the processor of claim 1, wherein the one or more features or attributes comprise one or more facial features or attributes (Fig. 2, image features and their respective attributes are used to modify facial expressions and generate synthetic training images for facial expression recognition training).
With regard to claim 6, Zhang discloses the processor of claim 1, wherein the one or more first neural networks comprise one or more Generative Adversarial Networks (GANs) (paragraph [0072], “Therefore, the image augmentation model may correspond to a sample generative model in an adversarial network, and the adversarial network includes the sample generative model and a sample discriminative model.”).
With regard to claim 7, the discussion of claim 1 applies.
With regard to claim 8, Zhang discloses the method of claim 7, wherein the one or more features or attributes comprise one or more facial features that cause the second neural network to misidentify the one or more facial features (paragraphs [0036]-[0043], and Fig. 2, 10b, 10c, 10d, 10e, 10f, 10g, 10h, 10k; Facial features are recognized and exaggerated in order to better recognize facial expressions. If the facial features and corresponding expressions are misidentified or poorly then the loss values or performance metrics discussed in paragraphs [0107]-[0109] will require the facial expression recognition and synthetic image generation to adjusts weights to continue training the neural networks).
With regard to claim 9, Zhang discloses the method of claim 7, further comprising: using the one or more second neural networks to perform facial recognition (paragraphs [0107]-[0109] and Fig. 7, neural networks 30g and 30h perform facial recognition);
calculating the one or more performance metrics of the one or more second neural networks (If the facial features and corresponding expressions are misidentified or poorly then the loss values or performance metrics discussed in paragraphs [0107]-[0109] will require the facial expression recognition and synthetic image generation to adjusts weights to continue training the neural networks);
identifying the root cause, based at least in part on the one or more performance metrics (paragraphs [0107]-[0109], When the loss values are large enough to continue to train and adjust the weights of the neural networks, this is interpreted as a performance metric being below a threshold of satisfaction. With regard to a root cause being determined for the performance metric, Applicant’s specification only mentions a shortage of training images as a root cause for poor performance. For most facial recognition neural networks, if the system is not meeting performance metrics, the “root cause” is assumed to be a lack of training data. In a system that such as Zhang that is focused on generating synthetic images for the purpose of training the system, it follows that the intention is to generate more training images).
With regard to claim 10, Zhang discloses the method of claim 7, wherein the root cause identifies a set of images from one or more training images used to train the one or more second neural networks that are below a quantity threshold (With regard to a root cause being determined for the performance metric, Applicant’s specification only mentions a shortage of training images as a root cause for poor performance. For most facial recognition neural networks, if the system is not meeting performance metrics, the “root cause” is assumed to be a lack of training data. In a system that such as Zhang that is focused on generating synthetic images for the purpose of training the system, it follows that the intention is to generate more training images. If the facial features and corresponding expressions are misidentified or poorly then the loss values or performance metrics discussed in paragraphs [0107]-[0109] will require the facial expression recognition and synthetic image generation to adjusts weights to continue training the neural networks).
With regard to claim 11, Zhang discloses the method of claim 7, further comprising:
causing the one or more second neural networks to perform pattern recognition (Fig. 2, model 20c is considered to recognize the pattern of different facial features and the exaggerated pattern of facial features to generate a facial expression matching the determine facial feature pattern. See also paragraphs [0036]-[0042]); and
wherein the one or more performance metrics of the one or more second neural networks indicate correctness in performing pattern recognition on a plurality of images (paragraphs [0036]-[0042], the performance or loss values of the facial expression recognition of 20 c is used to adjust the weights of the facial feature exaggeration components).
With regard to claim 13, the discussion of claim 1 applies.
With regard to claim 14, Zhang discloses the system of claim 13, wherein the one or more performance metrics indicate performance of the one or more second neural networks performing facial recognition on one or more other images (Fig. 7, The neural networks 30g and 30h are used to train the generative model 30d by calculating error function 30k, which consists of loss values for the models and adjusting the weights of the models thereby training the generative models. The loss values are considered to be the performance metrics used to train the models by adjusting the weights. See paragraphs [0107]-[0109]).
With regard to claim 15, Zhang discloses the system of claim 13, wherein the one or more processors are to further: select a reference image (Fig. 7, first image 30a is considered a reference image that is used to generate synthetic facial expression images 30e and 30f); and
Wherein the one or more first neural networks generate variations of the reference image to generate the one or more images (Fig. 7, first image 30a is considered a reference image that is used to generate synthetic facial expression images 30e and 30f. See also Fig. 2, neural network models 20a and 20b are used synthetically generate variations of the reference image 10a in order to exaggerate a facial expression for recognition).
With regard to claim 16, Zhang discloses the system of claim 13, wherein the one or more processors are to:
cause the one or more second neural networks to perform object classification (Fig. 2 neural network model 20c performs object classification by detecting and recognizing facial components or objects in order to detect and classify a facial expression); and
wherein the one or more performance metrics of the one or more second neural networks indicate correctness in performing object classification of one or more other images (Fig. 7, The neural networks 30g and 30h are used to train the generative model 30d by calculating error function 30k, which consists of loss values for the models and adjusting the weights of the models thereby training the generative models. The loss values are considered to be the performance metrics used to train the models by adjusting the weights. See paragraphs [0107]-[0109]. Loss values are considered an indicator of correctness for performing facial recognition).
With regard to claim 18, Zhang discloses the system of claim 13, wherein the one or more processors cause the one or more first neural networks to generate the one or more images with higher quality than one or more other images initially used to train the one or more second neural networks Fig. 7, The neural networks 30g and 30h are used to train the generative model 30d by calculating error function 30k, which consists of loss values for the models and adjusting the weights of the models thereby training the generative models. The loss values are considered to be the performance metrics used to train the models by adjusting the weights. See paragraphs [0107]-[0109]. Adjusting the weights of the models that generate the synthetic images is interpreted as causing the model to generate synthetic images of higher quality. The training is recursive and the quality of the synthetic images should increase over time with each iterative of weight adjustment).
With regard to claim 19, Zhang discloses the system of claim 13, wherein the one or more performance metrics comprises information indicating which types of features or attributes result in the one or more second neural networks misidentifying faces (paragraphs [0036]-[0043], and Fig. 2, 10b, 10c, 10d, 10e, 10f, 10g, 10h, 10k; Facial features are recognized and exaggerated in order to better recognize facial expressions. If the facial features and corresponding expressions are misidentified or poorly then the loss values or performance metrics discussed in paragraphs [0107]-[0109] will require the facial expression recognition and synthetic image generation to adjusts weights to continue training the neural networks).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 12 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of USPNs 2021/0174072 to Zhang et al. and 2023/0368502 to Ranganathan et al.
With regard to claims 12 and 17, Zhang discloses the method of claim 7, but does not disclose further comprising: determining a reference image associated with an identity;
filtering the one or more images generated by the one or more first neural networks by comparing the one or more images with the reference image; and
discarding an image from the one or more images that comprises one or more facial features that are different with the identity of the reference image.
Ranganathan discloses a system for training a facial image recognition by generating synthetic images in the form of altered facial images of a reference facial image of a specific person (See Fig. 1B). The system then attempts to recognize the image as the image of the person after the image has been synthetically altered (See Fig. 1C).
Facial images are then attempted to be recognized and based on the comparison the face is recognized as the known person (paragraphs [0017]-[0020]). If the image does not match the synthetic image then the person is not considered to be identified as the registered reference image person. Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the synthetic image generation taught by Ranganathan in combination with the facial feature recognition of Zhang in order to determine a robust facial verification system.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of USPNs 2021/0174072 to Zhang et al. and 2024/0326825 to Quintao Severgnini et al.
With regard to claim 20, Zhang discloses the system of claim 13, but does not disclose wherein the one or more processors are to the use one or more second neural networks to perform facial recognition in an autonomous vehicle.
Monitoring facial images and facial expressions of people inside vehicles is well known in the art. Quintao Severgnini discloses monitoring an occupant of the vehicle to recognize the face of the occupant as well as the emotional response to the occupant through facial expression recognition. Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the facial recognition taught by Zhang in the environment of monitoring the face of an autonomous vehicle as taught by Quintao Severgnini in order to observe and track the occupant’s emotional state.
FINAL REJECTION
Applicant’s amendment necessitated the grounds of rejection presented in the Office Action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WESLEY J TUCKER whose telephone number is (571)272-7427. The examiner can normally be reached 9AM-5PM Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JOHN VILLECCO can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WESLEY J TUCKER/Primary Examiner, Art Unit 2661
Prosecution Insights