Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 4/15/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 11-14, 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Budhrani et al. (US 20200218885 A1) referred to as Budhrani hereinafter and further in view of Saito et al. (JP 2006133930 A) referred to as Saito hereinafter.
Regarding claim 1, Budhrani teaches A computer implemented method for processing images of the face of a subject, the method comprising: acquiring a first plurality of images of at least a portion of the face of a subject; (“retrieving a first plurality of facial landmarks of a provided facial image associated with a subject” Budhrani, abstract)
generating an initial facial mesh representative of said at least portion of the face of the subject by analysing multiple images of the first plurality of images to extract corresponding facial landmarks from each image of the subject and corresponding edges extending therebetween; (“the captured facial image is a three-dimensional facial model and the second plurality of facial landmarks includes two-dimensional facial landmarks and/or three-dimensional facial landmarks. For example, the captured facial image may include a geometrical model of at least a part of a head of the captured subject. The geometrical model may be represented by a three-dimensional mesh of geometric elements representing the surface of the head, such as the face.” Budhrani, para. [0026]) and (“The method 100 may proceed with item 112, wherein the differences between facial landmarks of the first plurality of facial landmarks and facial landmarks of the second plurality of facial landmarks may be calculated and used to extract features in item 114.” Budhrani, para. [0052])
acquiring a further plurality of images of at least a portion of the face of the subject after the issuance of each prompt; (“The further facial images may be used to further aid in evaluating the provided facial image and determining whether the provided facial image is a real facial image or an artificial facial image.” Budhrani, para. [0038])
generating a difference facial mesh for the subject after the issuance of each prompt by: extracting a plurality of edges characterising the distance between facial landmarks of the subject from an image selected from the further plurality of images of the face of the subject; (“at least some of the features in the feature vector may be based on distances between corresponding facial landmarks in the first set (or plurality) of facial landmarks and the second set (or plurality) of facial landmarks, thereby defining distance values or distance features. The distance features may be based on distances of corresponding facial landmarks in the two sets and/or distances of different facial landmarks (which may be determined based on semantic dependencies) in the two sets, which may result in up to Σ.sub.i=1.sup.n−1 i features for n landmarks.” Budhrani, para. [0054])
generating a further facial mesh from said plurality of edges and facial landmarks; determining change of at least some predetermined edges of the further facial mesh relative to corresponding edges of the initial facial mesh. (“a method for classifying facial images comprising retrieving a first plurality of facial landmarks of a provided facial image (claimed to be) associated with a subject, determining a corresponding second plurality of facial landmarks of a captured facial image of the subject, extracting at least one feature vector based on differences between at least some of the first plurality of facial landmarks and corresponding facial landmarks of the second plurality of facial landmarks” Budhrani, para. [0008])
However, Budhrani does not teach issuing a plurality of prompts to the subject indicative of predetermined emotional states;
Saito teaches issuing a plurality of prompts to the subject indicative of predetermined emotional states; (“The execution sequence setting unit 104 confirms whether the user identification information input by the user is registered in the user information storage unit 111 of the information storage unit 105, and if the registration is confirmed, a sequence such as a facial expression requested to the user To decide. For example, a sequence such as [laughter] .fwdarw. [anger] or [surprise] .fwdarw. [sadness].” Saito, p. 7, para. 2)
Budhrani and Saito are combinable because they are from the same field of endeavor, image processing.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Budhrani in light of Saito’s prompt of emotional state. One would have been motivated to do so because the amount of data accumulated and the amount of data processing required by the authentication processing device can be reduced. (Saito, p. 2)
Regarding claim 2, Saito teaches wherein said prompts comprise an emoticon indicative of different emotional states a subject may experience. (“As shown in FIG. 2, the user information storage unit 111 shown in FIG. 1 stores data associating registered user identifiers with registered user face image data. The registered user registers a plurality of predetermined facial expressions or facial images whose orientations are changed in advance. As shown in FIG. 2, each user has a registered facial expression pattern as shown in FIG. 2: facial expression pattern P01: [laughter], facial expression pattern P02: [anger], facial expression pattern P03: [surprise], ... facial expression pattern Pnn: [sadness]” Saito, p. 6, para. 2)
Budhrani and Saito are combinable because they are from the same field of endeavor, image processing.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Budhrani in light of Saito’s different emotional states. One would have been motivated to do so because the amount of data accumulated and the amount of data processing required by the authentication processing device can be reduced. (Saito, p. 2)
Regarding claim 3, Saito teaches wherein the emoticon is displayed to the subject on the same portable electronic device used for acquiring images of the subject. (“When the user ID input by the user matches the registered user ID stored in the user information storage unit 111 (step S103: Yes), the process proceeds to step S104, and the execution sequence setting unit 104 displays the user information storage unit 111.” Saito, p. 8, para. 6)
Budhrani and Saito are combinable because they are from the same field of endeavor, image processing.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Budhrani in light of Saito’s displaying the emoticon. One would have been motivated to do so because the amount of data accumulated and the amount of data processing required by the authentication processing device can be reduced. (Saito, p. 2)
Regarding claim 4, Saito teaches wherein the emoticon is indicative of an emotional state randomly selected from a group of predetermined emotional states. (“The authentication processing device 100 randomly selects a plurality of registered patterns from these registered patterns, determines a sequence in which the selected registered patterns are set in time series, and changes the facial expression to the user along the determined sequence. Request. For example, the expression pattern P01: [laughter] .fwdarw. P02: [anger], P03: [surprise] .fwdarw. Pnn: [sadness], Pmm: [unique registration pattern] .fwdarw. P01: [laughter].” Saito, p. 6, para. 3)
Budhrani and Saito are combinable because they are from the same field of endeavor, image processing.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Budhrani in light of Saito’s randomly selected emotional states. One would have been motivated to do so because the amount of data accumulated and the amount of data processing required by the authentication processing device can be reduced. (Saito, p. 2)
Regarding claim 5, Budhrani teaches determining a scaling factor by calculating the distance between certain landmarks in the initial facial mesh for a subject and the distance between the same landmarks of the same subject in the further facial mesh; (“Morphing may include identifying corresponding landmarks in both images 202a and 202b and blending both images 202a and 202b into each other based on a transformation of the individual facial landmarks, such as translation, rotation and scaling of the individual facial landmarks. The facial landmarks may be characteristic key points, which in case of facial images may correspond to facial landmarks, such as position and contour of the nose, location and contour of the eyes, a contour or shape of the mouth, and the like, in any combination.” Budhrani, para. [0058])
and applying said scaling factor in the comparison of other edges of the difference facial mesh with the initial facial mesh of that subject. (“The differentiator 314 may retrieve an axis of the respective facial image and may determine an orientation of the landmark vector with regard to the axis, such as an angle of the landmark vector with respect to the axis. The differentiator 314 may then compare corresponding landmark vectors derived from the two sets of facial landmarks and may extract a feature based on the differences of the orientations of the corresponding landmark vectors. For n facial landmarks in each set, this may result in n angular features.” Budhrani, para. [0063])
Regarding claim 6, Budhrani teaches wherein the image selected for the determination of the difference facial mesh is selected randomly or according to predetermined criteria from the further plurality of acquired images. (“Features based on angular values of differences between corresponding facial landmarks define a highly selective and reliable indicator enabling classification of provided facial images as artificial images or real images of the subject.” Budhrani, para. [0014])
Regarding claim 11, Budhrani teaches wherein each of said difference facial mesh is generated by: acquiring a first plurality of image frames of the face of a subject; (“the system further comprises one or more reader devices configured to read provided facial images and one or more scanner devices to scan faces of subjects to generate captured images of the subjects.” Budhrani, para. [0037])
generating an initial facial mesh representative of at least portion of the face of the subject of the face of the subject by analysing multiple images of the first plurality of images to extract corresponding facial landmarks from each image of the subject and corresponding edges extending therebetween; (“the captured facial image is a three-dimensional facial model and the second plurality of facial landmarks includes two-dimensional facial landmarks and/or three-dimensional facial landmarks. For example, the captured facial image may include a geometrical model of at least a part of a head of the captured subject. The geometrical model may be represented by a three-dimensional mesh of geometric elements representing the surface of the head, such as the face.” Budhrani, para. [0026]) and (“The method 100 may proceed with item 112, wherein the differences between facial landmarks of the first plurality of facial landmarks and facial landmarks of the second plurality of facial landmarks may be calculated and used to extract features in item 114.” Budhrani, para. [0052])
and acquiring a further plurality of images of the face of the subject subsequent to communicating to the subject said prompt; (“The further facial images may be used to further aid in evaluating the provided facial image and determining whether the provided facial image is a real facial image or an artificial facial image.” Budhrani, para. [0038])
generating a further facial mesh derived from a plurality of edges characterising the distance between facial landmarks of the subject extracted from an image randomly selected from the second plurality of images of the face of the subject; (“at least some of the features in the feature vector may be based on distances between corresponding facial landmarks in the first set (or plurality) of facial landmarks and the second set (or plurality) of facial landmarks, thereby defining distance values or distance features. The distance features may be based on distances of corresponding facial landmarks in the two sets and/or distances of different facial landmarks (which may be determined based on semantic dependencies) in the two sets, which may result in up to Σ.sub.i=1.sup.n−1 i features for n landmarks.” Budhrani, para. [0054])
determining the change of at least some predetermined edges of the further facial mesh relative to corresponding edges of the initial facial mesh; (“a method for classifying facial images comprising retrieving a first plurality of facial landmarks of a provided facial image (claimed to be) associated with a subject, determining a corresponding second plurality of facial landmarks of a captured facial image of the subject, extracting at least one feature vector based on differences between at least some of the first plurality of facial landmarks and corresponding facial landmarks of the second plurality of facial landmarks” Budhrani, para. [0008])
and evaluating by a model executing on one or more processors at a location remote from said first location; (“The provided facial image may be retrieved from the memory and may be further processed to determine the first plurality of facial landmarks or the first representation of facial landmarks. Additionally or as an alternative, at least some of the first plurality of facial landmarks may be previously generated during an application process for the electronic document and may be stored in the memory or on the electronic document.” Budhrani, para. [0010])
whether each of said received difference facial mesh corresponds to an expected emotional state for a subject following issuance of the prompt communicated to the subject;
PNG
media_image1.png
155
370
media_image1.png
Greyscale
Budhrani, para. [0068])
wherein said model is trained using a machine learning algorithm. (“The classifier 316 may be trained using a training set of facial images, which may include real facial images and artificial facial images.” Budhrani, para. [0065])
However, Budhrani does not teach A computer implemented method of verifying liveness of a subject depicted in a plurality of acquired images of the subject acquired at a first location comprising;
receiving over a network a plurality of difference facial mesh generated by a further processor at said first location following the issuance of a plurality of prompts indicative of an emotional state;
Saito teaches A computer implemented method of verifying liveness of a subject depicted in a plurality of acquired images of the subject acquired at a first location comprising; (“With reference to the block diagram of FIG. 1, the flow of processing when executing user authentication processing will be described. The user 10 as the person to be authenticated inputs information for identifying the user, such as a user ID and a password, to the input unit 107 when executing the authentication process.” Saito, p. 7, para. 2)
receiving over a network a plurality of difference facial mesh generated by a further processor at said first location following the issuance of a plurality of prompts indicative of an emotional state; (“The execution sequence setting unit 104 confirms whether the user identification information input by the user is registered in the user information storage unit 111 of the information storage unit 105, and if the registration is confirmed, a sequence such as a facial expression requested to the user To decide. For example, a sequence such as [laughter] .fwdarw. [anger] or [surprise] .fwdarw. [sadness].” Saito, p. 7, para. 2)
Budhrani and Saito are combinable because they are from the same field of endeavor, image processing.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Budhrani in light of Saito’s prompt of emotional state. One would have been motivated to do so because the amount of data accumulated and the amount of data processing required by the authentication processing device can be reduced. (Saito, p. 2)
Regarding claim 12, refer to the explanation of claim 2.
Regarding claim 13, refer to the explanation of claim 3.
Regarding claim 14, refer to the explanation of claim 4.
Regarding claim 16, refer to the explanation of claims 1 and 11.
Regarding claim 17, refer to the explanation of claim 13.
Regarding claim 18, refer to the explanation of claims 1 and 11.
Regarding claim 19, refer to the explanation of claim 13.
Regarding claim 20, Budhrani teaches wherein the step of evaluating by the machine learning algorithm is performed after transmission of each difference facial mesh over a network to by a processor of one or more remotely located servers. (“The provided facial image may be retrieved from the memory and may be further processed to determine the first plurality of facial landmarks or the first representation of facial landmarks. Additionally or as an alternative, at least some of the first plurality of facial landmarks may be previously generated during an application process for the electronic document and may be stored in the memory or on the electronic document.” Budhrani, para. [0010])
Claim(s) 7-10, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Budhrani and Saito as mentioned above and further in view of Ninuma et al. (US 20230029505 A1) referred to as Ninuma hereinafter.
Regarding claim 7, the combination of Budhrani and Saito does not teach wherein the machine learning algorithm is a classifier trained to identify all of the emotional states of the subject in the sets of images of the subject.
However, Ninuma teaches wherein the machine learning algorithm is a classifier trained to identify all of the emotional states of the subject in the sets of images of the subject. (“At block 380, the facial expression of the subject may be classified. For example, if the machine learning system identifies AU combination 6+12 (cheek raise and lip corner puller) representing happiness, the machine learning system may classify the subject's facial expression as a smile. Additionally or alternatively, the machine learning system may classify the user's level of emotion based on the AU combinations and the associated categories of intensity. For example, again using AU combination 6+12, if the combination is associated with maximum intensity E, the machine learning system may classify the subject's smile differently than if the AU combination is associated with minimum intensity A.” Ninuma, para. [0040]) and (“A machine learning system 430, trained using a dataset including at least one input image (such as described in block 310) and one or more synthesized images of subject 410A, identifies the AU combinations and categories of intensity 440 present in the at least one input image 420.” Ninuma, para. [0043])
Budhrani, Saito, and Ninuma are combinable because they are from the same field of endeavor, image processing.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Budhrani and Saito in light of Ninuma’s classifier trained to identify emotional states. One would have been motivated to do so because the efficiency and accuracy of facial recognition can be improved. (Ninuma, abstract)
Regarding claim 8, refer to the explanation of claim 7.
Regarding claim 9, refer to the explanation of claim 7.
Regarding claim 10, refer to the explanation of claim 7.
Regarding claim 15, refer to the explanation of claim 7.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 12596821 B2 Systems And Methods For Remotely Provisioning Facial Recognition Data To Heterogeneous Computing Platforms
US 20190332846 A1 METHOD, APPARATUS AND SYSTEM FOR 3D FACE TRACKING
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PARDIS SOHRABY whose telephone number is (571)270-0809. The examiner can normally be reached Monday - Friday 9 am till 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PARDIS SOHRABY/ Examiner, Art Unit 2664
/CHARLOTTE M BAKER/ Primary Examiner, Art Unit 2664
Prosecution Insights