DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on May 14, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is considered by examiner.
Claim Objections
Claims 2, 8, 9, 12-14 objected to because of the following informalities: Claims 2, 8, 9, 12-14 are each missing a comma after reciting the claim in which it depends in the preamble. For example, claim 2 recites “method of claim 1 further comprising:” and should be “method of claim 1, further comprising:”. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because the claim currently recites "A computer program product comprising program code instructions stored on at least one computer readable medium to execute the method steps according to claim 1" and the full scope of “computer readable medium” includes transitory signals.
In this case, while the specification exemplifies various forms of a medium, it does not disavow transitory signals (“A computer-usable or computer-readable medium may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.” ¶ [0071]). The specification states the medium can “propagate, or transport the program”, which is interpreted to include signals by one or ordinary skill in the art. The claim language does not clearly state that signals are excluded and the broadest reasonable interpretation is to therefore include transitory signals, which is not patent eligible subject matter.
The state-of-the-art at the time the invention was made included signals, carrier waves and other wireless communication modalities (e.g., RF, infrared, etc.) as media on which executable code was recorded and from which computers acquired such code. Thus, the full scope of the claim covers "signals" and their equivalents, which are non-statutory per se. (see In re Nuijten). The examiner suggests clarifying the claim to exclude such non-statutory signal embodiments, such as (but not limited to) reciting a " stored on at least one non-transitory computer readable medium", or equivalent, consistent with the corresponding original disclosure.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 6, 15 are rejected under 35 U.S.C. 103 as being unpatentable over Luo et al (US 2022/0222892) in view of Cole et al (Synthesizing Normalized Faces from Facial Identity Features).
Regarding Claim 1, Luo et al teach a method for controlling an output device based on a normalized image of a face generated from a captured non-normalized image of the face (method of generating a normalized 3-D model of a human face; Fig 1, 3, 4 and ¶ [0020], [0037], [0047]) comprising:
generating a random dataset of images representing synthetic non-normalized photographs of human faces and backgrounds (generating an image of face, generated from face photographs, using a GAN and the GAN images are manipulated, such as diffuse lighting conditions with different backgrounds, in creating a training dataset with multiple face images; Fig 1, 4 and ¶ [0020]-[0021], [0044]-[0045], [0049], [0052]);
generating an output dataset of synthetic normalized photographs (normalized 3-D model of human face are generated based on the synthesized image; Fig 1, 3 and ¶ [0020], [0044]) by: for each synthetic non-normalized photograph:
removing variations to generate a normalized image of each human face (the synthesized image are light normalized to generate 3-D facial images; Fig 1, 3 ¶ [0044], [0049]); and
removing the backgrounds surrounding each human face for each normalized image (a mask is applied to the normalized face dataset images to mask the background; ¶ [0049]);
generating a training dataset including the random dataset and the output dataset (training data 105 includes the normalized face dataset from the synthetic data and the refined face only image; Fig 1, 3 and ¶ [0020]-[0021], [0044]-[0049]);
Luo et al does not teach training a neural network to generate normalized photographs from non-normalized photographs using the training dataset; receiving a non-normalized facial image of an individual; applying the neural network to generate a normalized facial image from the non-normalized facial image; and controlling an output device based on the normalized image of the individual.
Cole et al is analogous art pertinent to the technological problem addressed in this application and teaches training a neural network to generate normalized photographs from non-normalized photographs using the training dataset (an input training image and ground truth (normalized) input image is used to train the CNN predicted features decoder to generate normalized (averaged) images; Fig 1, 2, 4, 6 and 3. Autoencoder Model – 3.2 Decoder, 6.2 Averaging to reduce lighting variation);
receiving a non-normalized facial image of an individual (a non-normalized input image is input to the Autoencoder; Fig 3 and 3. Autoencoder Model – 3.1 Encoder);
applying the neural network to generate a normalized facial image from the non-normalized facial image (the input image is processed by the Encoder and Decoder for texture and landmark locations to generate a final rendered (normalized) image; Fig 3 and 3.1 Encoder, 3.2 Decoder); and
controlling an output device based on the normalized image of the individual (the normalized image may be used to create a 3-D avatar; 7.3 3-D Model Fitting).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Luo et al with Cole et al including training a neural network to generate normalized photographs from non-normalized photographs using the training dataset; receiving a non-normalized facial image of an individual; applying the neural network to generate a normalized facial image from the non-normalized facial image; and controlling an output device based on the normalized image of the individual. By normalizing input photographs, features are identified that are consistent across changes in pose, lighting and expression for applications such as facial recognition, as recognition by Cole et al (¶ 2-4).
Regarding Claim 6, Luo et al in view of Cole et al teach the method of claim 1 (as described above), wherein the random dataset is generated according to a generative adversarial network (GAN) (Luo et al, generating an image of face, generated from face photographs, using a GAN; Fig 1, 4 and ¶ [0020]-[0021], [0044]-[0045], [0049]) and the applying comprises inverting the non-normalized facial image into the GAN space used for the random dataset (Luo et al, inversion is applied in the trained GAN to reconstruct geometry and texture of the face images; Fig 5 and ¶ [0040], [0045]).
Regarding Claim 15, Luo et al teach a computer program product comprising program code instructions stored on at least one computer readable medium (the CPU 131 may execute instructions stored on the memory 135 of the computing device; Fig 1 and ¶ [0023]-[0024], [0030]) to execute the method steps according to claim 1 (as described above), when said program code instructions are executed on a computer (the programs stored on the memory 135 are executed by the CPU 131 and/or GPU 132; Fig 1 and ¶ [0030]).
Claims 2, 4 are rejected under 35 U.S.C. 103 as being unpatentable over Luo et al (US 2022/0222892) in view of Cole et al (Synthesizing Normalized Faces from Facial Identity Features) and Raja et al (US 2020/0175290).
Regarding Claim 2, Luo et al in view of Cole et al teach the method of claim 1 (as described above).
Luo et al in view of Cole et al does not teach capturing a passport image of the individual; performing a comparison of the normalized image with the passport image; and, wherein the controlling comprises selectively controlling a gate to permit or deny passage of the individual through the gate according to the comparison.
Raja et al is analogous art pertinent to the technological problem addressed in this application and teaches capturing a passport image of the individual (an image is captured at a border control an automated passport gate; Fig 1, 6 and ¶ [0003]-[0005], [0045]); performing a comparison of the normalized image with the passport image (the morphed passport image is normalized and compared to determine if the face recognition image is a spoof; ¶ [0003]-[0005], [0045]); and, wherein the controlling comprises selectively controlling a gate to permit or deny passage of the individual through the gate according to the comparison (the face recognition system may be applied at a passport gate for border control; [0003]-[0005], [0045].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Luo et al in view of Cole et al with Raja et al including capturing a passport image of the individual; performing a comparison of the normalized image with the passport image; and, wherein the controlling comprises selectively controlling a gate to permit or deny passage of the individual through the gate according to the comparison. By using passport images of an individual and comparing the image to a morphed image of the passport image, facial recognition may be applied in a highly reliable and accurate manner to assist in border control processes, as recognized by Raja et al (¶ [0003]).
Regarding Claim 4, Luo et al in view of Cole et al teach the method of claim 1 (as described above).
Luo et al in view of Cole et al does not teach wherein the controlling comprises controlling a display device to generate the normalized facial image beside the non-normalized facial image of the individual.
Raja et al is analogous art pertinent to the technological problem addressed in this application and teaches wherein the controlling comprises controlling a display device to generate the normalized facial image beside the non-normalized facial image of the individual (image analysis is output to the user for further control of a passport gate; ¶ [0045], [0061]-[0062], [0070]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Luo et al in view of Cole et al with Raja et al including wherein the controlling comprises controlling a display device to generate the normalized facial image beside the non-normalized facial image of the individual. By using passport images of an individual and comparing the image to a morphed image of the passport image, facial recognition may be applied in a highly reliable and accurate manner to assist in border control processes, as recognized by Raja et al (¶ [0003]).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Luo et al (US 2022/0222892) in view of Cole et al (Synthesizing Normalized Faces from Facial Identity Features), Raja et al (US 2020/0175290) and Li et al (US 2022/0084204).
Regarding Claim 3, Luo et al in view of Cole et al and Raja et al teach the method of claim 2 (as described above).
Luo et al in view of Cole et al and Raja et al does not teach wherein the synthetic non-normalized photographs are generated from random noise and a pre-trained generative adversarial network (GAN) to thereby control the output device using a neural network without relying upon personal identifiable information (PII) in a training dataset.
Li et al is analogous art pertinent to the technological problem addressed in this application and teaches wherein the synthetic non-normalized photographs are generated from random noise and a pre-trained generative adversarial network (GAN) to thereby control the output device using a neural network without relying upon personal identifiable information (PII) in a training dataset (a pre-trained StyleGAN generator is used to generate a synthetic facial image from the input image including adding noise to the input image via feature maps in a per-block incorporation; Fig 3A and ¶ [0074]-[0075]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Luo et al in view of Cole et al and Raja et al with Li et al including wherein the synthetic non-normalized photographs are generated from random noise and a pre-trained generative adversarial network (GAN) to thereby control the output device using a neural network without relying upon personal identifiable information (PII) in a training dataset. By adding random noise to the image to generate the synthetic facial image, the generated image includes a stylistic and stochastic variation for fine-grained detailed at the pixel-level in the synthetic image, thereby improving control over the style properties of the generated images, as recognized by Li et al (¶ [0073]-[0074]).
Claims 5, 9, 10 are rejected under 35 U.S.C. 103 as being unpatentable over Luo et al (US 2022/0222892) in view of Cole et al (Synthesizing Normalized Faces from Facial Identity Features) and Shen et al (InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs).
Regarding Claim 5, Luo et al in view of Cole et al teach the method of claim 1 (as described above), wherein removing variations includes at least one of normalizing frontal facial illumination (Luo et al, the synthesized image are light normalized to generate 3-D facial images; Fig 1, 3 ¶ [0044], [0049]); generating a neutral pose (Luo et al, the textured 3-D face is generated with neutral geometry; Fig 6 and ¶ [0056], [0059]), generating a neutral expression (Luo et al, the textured 3-D face is generated with neutral expression; Fig 6 and ¶ [0056], [0059]), and creating a uniform light color background (Luo et al, the background lighting is normalized; Fig 6 and ¶ [0052]).
Luo et al in view of Cole et al does not teach removal of glasses as one of the variations in the normalization of frontal features.
Shen et al is analogous art pertinent to the technological problem addressed in this application and teaches removal of glasses as one of the variations in the normalization of frontal features (eyeglasses may be manipulated on/off with the InterFaceGAN; Fig 1, 10 and 4.3.3 Conditional Manipulation).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Luo et al in view of Cole et al with Shen et al including removal of glasses as one of the variations in the normalization of frontal features. By using a GAN model that may manipulate an image with eyeglasses, the face semantics are manipulated by the GAN individually and analyzed to demonstrate how the manipulation process is sensitive to different facial attributes while allowing for various manipulations without retraining the GAN, as recognized by Shen et al (1. Introduction ¶ 3-5).
Regarding Claim 9, Luo et al in view of Cole et al teach the method of claim 1 (as described above).
Luo et al in view of Cole et al does not teach balancing the demographic distribution of faces in the output dataset prior to the training.
Shen et al is analogous art pertinent to the technological problem addressed in this application and teaches balancing the demographic distribution of faces in the output dataset prior to the training (the training data included 10K synthetic pairs to account for demographic attributes, such as gender and features such as age and facial features; Figs 13-15 and 6.2. Training with Paired Synthetic Data Collected from InterFaceGAN).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Luo et al in view of Cole et al with Shen et al including balancing the demographic distribution of faces in the output dataset prior to the training. By using a GAN model that may manipulate the face semantics individually and analyzed to demonstrate how the manipulation process is sensitive to different facial attributes, the identity can be analyzed to determine how different facial attribute manipulations may influence the original image, as recognized by Shen et al (1. Introduction ¶ 3-5).
Regarding Claim 10, Luo et al in view of Cole et al and Shen et al teach the method of claim 9 (as described above), wherein the balancing is performed using an Interface (Interpreting Face) generative adversarial networks (InterfaceGAN) (Shen et al, the training data is generated using an InterFaceGAN; Figs 13-15 and 6.2. Training with Paired Synthetic Data Collected from InterFaceGAN).
Claims 11-14 are rejected under 35 U.S.C. 103 as being unpatentable over Raja et al (US 2020/0175290) in view of Luo et al (US 2022/0222892).
Regarding Claim 11, Raja et al teach a checkpoint apparatus (passport gate apparatus with electronic Machine Readable Travel Document (eMRTD) system; ¶ [0003]-[0005], [0045]) comprising:
a camera for receiving a non-normalized first image of a face an individual (a camera is used to capture a live captured face image ; Fig 1 and ¶ [0003]-[0005], [0045], [0072]);
an input device for receiving an identification document photograph of a second image of a face (the eMRTD system has access to a face reference image stored in the eMRTD passport system; ¶ [0003], [0072]);
a connection to a processor (the computer includes a processor and data storage (memory) with instructions; ¶ [0046]-[0047]) configured to generate a normalized image from the first image by removing a background and orienting the face of the first image directly towards a virtual camera (images are received by the processor and used for detecting manipulated images based on non-manipulated (normalized) images and generating a neutral pose; ¶ [0041], [0047]-[0048], [0072]), the normalized image being created from a neural network dataset that excludes the individual (the neural network is used for generating descriptor features of the image; ¶ [0041]-[0042]); and
an output device for selectively performing a function based upon whether the first image and the second image are the same individual (the SVM classifier uses the descriptor data to determine whether the presented face image belongs to the normal (passport specification image) 17 or is a morphed 18 image; Fig 1 and ¶ [0070]).
Raja et al does not teach to generate a normalized image from the first image by removing a background.
Luo et al is analogous art pertinent to the technological problem addressed in this application and teaches to generate a normalized image from the first image by removing a background (a mask is applied to the normalized face dataset images to mask the background; ¶ [0049]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the current application to combine the teachings of Raja et al with Luo et al including to generate a normalized image from the first image by removing a background. By masking the background from the original image, a normalized face pose image may easily be generated that preserves likeness of the input subject, as recognized by Luo et al (¶ [0052]).
Regarding Claim 12, Raja et al in view of Luo et al teach the apparatus of claim 11 (as discussed above) wherein the processor is configured to perform a determination whether the first image and the second image are of the same individual (Raja et al, analysis is performed to determine whether the captured face image is equivalent to the eMRTD passport specification image or a morphed image; ¶ [0003]-[0005], [0045], [0070], [0072]).
Regarding Claim 13, Raja et al in view of Luo et al teach the apparatus of claim 11 (as discussed above) wherein the function is the training of a neural network as to whether a subsequent first images and subsequent second images are of the same individual (Raja et al, the neural network trained to determine whether the captured face image is equivalent to the eMRTD passport specification image or a morphed (manipulated image; ¶ [0003]-[0005], [0045], [0048], [0070], [0074]).
Regarding Claim 14, Raja et al in view of Luo et al teach the apparatus of claim 11 (as discussed above) wherein the function is one of opening a gate, activating a luggage conveyor belt, printing a boarding pass, or printing a luggage tag if the first image and the second image are of the same individual (Raja et al, analysis is performed to determine whether the captured face image is equivalent to the eMRTD passport specification image at a passport gate (the border control gate will allow passage if images match); ¶ [0003]-[0005], [0045]).
Allowable Subject Matter
Claims 7-8 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding Claim 7, the following limitations in combination with the claims in which it depends are considered novel over the prior art, when the claim is considered as a whole:
Claim 7. The method of claim 1, wherein removing the backgrounds comprises:
calculating a symmetric loss for each image; calculating background statistics for each image;
preparing a positive dataset of a first subset of the images with: a symmetric loss in a first lower range; a background color in a first higher range; and a background diversity in a second lower range;
preparing a negative dataset of a second subset of the images with: the symmetric loss in a higher range greater than the first lower range; the background color in a third lower range than the second higher range; and the background diversity in a third range higher than the second lower range;
generating a trained linear classifier separating the positive dataset and the negative dataset for each image;
generating a separation plan from the trained linear classifier; and
learning a linear control function to generate each normalized image based on the foregoing.
Examiner note: “based on the foregoing” is interpreted to consider all claim limitations of claim 1 and 7 “to generate each normalized image.”
Claim 8 is dependent on claim 7 and therefore allowable for similar reasons.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Thomas et al (US 20250131769, application 18/920,610) teaches a system and method for generating distortion images, application from the same assignee and inventors, but is distinct from the current application as the ‘610 application claims relate to estimating an amount of distortion based on anchor and non-anchor landmarks, whereas the current application claims regard a face analysis of synthetic normalized and non-normalized photographs using a neural network. Both applications will be monitored throughout prosecution.
Zhang et al (Novel Counterfeit Feature Extraction Technique for Exposing Face-Swap Images Based on Deep Learning and Error Level Analysis) teach a deep learning and error level analysis system and method to identify face-swap images and includes processes to analysis the foreground from the background and perform a cropping process prior to analyzing the image to distinguish deep fake images from real images.
Ankile et al (Application of Facial Recognition using CNNs for Entry Access Control) teach a facial recognition CNN system and method used to classify faces and used to identify faces for identification of matching human faces from imposer images and accounts for the face and the background of the image, including lighting.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHLEEN M BROUGHTON whose telephone number is (571)270-7380. The examiner can normally be reached Monday-Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATHLEEN M BROUGHTON/Primary Examiner, Art Unit 2661