DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 1 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
The Specification do not specifically disclose and define a “plurality of classification categories” which is claimed in claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-7, 9-14, 16-17 and 22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lemoff (US 2016/0086018 disclosed in Applicant’s IDS dated 06/09/2024), and further in view of Katz et al. (US 2020/0103980) hereinafter known as Katz, and Wolff et al. (US 2004/005086) hereinafter known as Wolff.
With regards to claim 1, Lemoff discloses a system (Abstract; an active-imaging system)(FIG. 2) for analyzing images for facial expression recognition ([0049]; “…a face recognition capability that has low confidence for a single captured image can be made high confidence through capturing may images of the same person at slightly different times…expressions,…etc.”)[0066]; “…frontal still images were collected with both neutral and talking expressions,…”),
comprising: at least one processor executing a code ([0035]; FIG. 2; The processor 150 can be a specially programmed general purpose computer for operating the user interface, … and running face recognition software.) for:
analyzing at least one short wave infrared (SWIR) image depicting a face of a person depicting a facial expression [0066], the at least one SWIR image captured by at least one SWIR sensor (FIG. 2; [0031][0033]; imager 122).
Lemoff teaches “Biometric technologies commonly used to identify people include: fingerprint, iris, DNA, and face recognition. Of these, the only one that potentially can be used for long-range standoff identification is face recognition. Other modalities that have been used to classify, …” [0005]. Lemoff also teaches that 6 SWIR-illuminated images are matched against a visible spectrum face database [0051].
Lemoff do not disclose classifying the face in the at least one SWIR image into a certain facial expression denoting a certain classification category selected from a plurality of predefined facial expressions denoting a plurality of classification categories.
In the same field of endeavor, Katz discloses systems, methods and non-transitory computer-readable media for triggering actions based on touch-free gesture detection. The system includes at least one processor that is configured to receive image information from an image sensor (Abstract). The image sensor can be a shortwave infrared (SWIR) image sensor [0041]. The reference teaches of machine learning components that can detect or predict features associated with expressions or emotions [0031][0032][0065][0081]. Katz further teaches that the processor utilizes machine learning algorithm to detect and classify gestures, activity or behavior relating to a face [0024][0048][0183].
In view of Katz, it would have been obvious to one of ordinary skill within the art before the effective filing date of the claimed invention to modify the processor of Lemoff with non-transitory computer-readable media that utilizes a machine learning algorithm capable of classifying facial expressions. The motivation is to classify facial attributes for identifying a person.
Katz teaches of using a processor configured to predict the user behavior, based on detecting and classifying the gesture in relation to a face of the user (claim 57).
Neither Lemoff nor Katz do not specifically disclose classifying the face into a certain facial expression denoting a certain classification category selected from a plurality of predefined facial expressions denoting a plurality of classification categories.
Wolff discloses an apparatus for thermal infrared imaging for face recognition (Abstract; [0002][0036]). The apparatus utilizes InGaAs FPAs that are sensitive to SWIR radiation [0023]. The reference teaches the use of neural networks or support vector machines (SVM) classifiers [0029] and a combination of two or more classifiers for identity verification, by combining voice and fingerprint, or voice and face biometrics ([0031]; The Examiner views the voice, fingerprint, and face biometric parameters as classification categories.). The classifiers provide a weighted combination of normalized scores from one or more of the reflective/thermal infrared imaging modalities. The applied weights can be varied to account for expression variation [0032]. Further. The reference teaches of a computer system and software used for the creation of a face representation template. The template is compared and matched with existing templates for face recognition applications. Further, the reference further teaches “This software performs the face detection process monitoring when a face is present in the scene of interest, and determines when to acquire imagery of unknown subjects, store imagery, create face representation templates, and perform matching comparisons for specified face recognition applications.” [0036]. Finally, the reference teaches the use of pattern classification utilizing feature representation [0028].
In view of Wolff, it would have been obvious to one of ordinary skill within the art before the effective filing date of the claimed invention to modify the processor of Lemoff with face recognition software that uses a plurality of classifiers (utilizing voice, fingerprint, and face biometric parameters) that are capable of classifying facial expression variations based on matching the said expressions to existing face representation template/s. The motivation is to match and classify facial expressions to a predefined template to determine the identification and/or emotional state of a person.
With regards to claim 2, modified Lemoff discloses the system of claim 1, wherein the plurality of predefined facial expressions include:
sad, anger, confusion, pain, surprise, fear. (Katz; [0065])
With regards to claim 3, modified Lemoff discloses the system of claim 1, further comprising:
extracting a plurality of facial features from the at least one SWIR image, wherein analyzing comprises analyzing the plurality of facial features. (Lemoff; [0062][0064])(Katz; [0032])[0035])
With regards to claim 4, modified Lemoff discloses the system of claim 1, wherein the at least one SWIR image comprises a single SWIR image, wherein analyzing comprises analyzing the single SWIR image (Lemoff; [0051]), and the classifying is for the single SWIR image (Wolff; [0031][0032][0033]; see the rejection of claim 1).
With regards to claim 5, modified Lemoff discloses the system of claim 1, further comprising at least one of:
(i) at least one filter that filters out electromagnetic radiation at wavelengths which are mostly non-absorbed by water vapor in air (Lemoff; [0038]; “ A narrow optical band-pass filter is placed in front of the FPA to pass only light near the 1550-nm illumination wavelength, rejecting all other ambient light.”), and
(ii) at least one short wave infrared (SWIR) illumination element that generates SWIR illumination for illumination of a face of a person (Lemoff; [0019]; “…an apparatus including an active-imaging system for capturing facial images illuminated with short-wave infrared light having a wavelength greater than 1400 nm and less than 1700 nm …”).
With regards to claim 6, modified Lemoff discloses the system of claim 5, wherein the at least one filter comprises a spectral narrow pass-band filter that:
(i) passes wavelengths about 1350-1450 nanometers (nm) and excludes wavelengths over about 1450 nm and below about 1350 nm (Lemoff; [0015][0038]; claim 5).
With regards to claim 7, modified Lemoff discloses the system of claim 5, further comprising controlling the SWIR illumination element for generating a target illumination pattern for illumination of the face (Lemoff; [0046[0047])(Wolff; [0028]), and capturing the at least one SWIR image while the target illumination pattern is being generated (Lemont; [0046]).
With regards to claim 9, modified Lemoff discloses the system of claim 5, wherein the at least one SWIR illumination element generates illumination at a band of wavelengths which are non-visible to a human eye (Lemoff; [0015]), and the at least one SWIR sensor senses at the band of wavelengths which are non-visible to the human eye (Lemoff; [0052][0059]).
With regards to claim 10, modified Lemoff discloses the system of claim 1, further comprising a visible light imaging sensor for capturing at least one visible light image (Katz; [0041][0044][0070]; light sensor)(Wolff; [0008]), and code (Lemoff; [0052]; face selection algorithm) for analyzing a combination of the at least one SWIR image and the at least one visible light image. (Lemoff; [0029][0061][0062])
With regards to claim 11, modified Lemoff discloses the system of claim 1, wherein the code further comprises instructions for analyzing the at least one SWIR image for distinguishing facial hair on the face from surrounding skin. (Lemoff; [0062]; beard mustache)
With regards to claim 12, modified Lemoff discloses the system of claim 1, wherein the code further comprises instructions for analyzing the at least one SWIR image for distinguishing long hair from a top of a head falling over the face from surrounding skin. (Katz; [0022])
With regards to claim 13, modified Lemoff discloses the system of claim 1, wherein the code further comprises instructions for analyzing the at least one SWIR image for distinguishing makeup on the face from surrounding skin. (Katz; [0022])
With regards to claim 14, modified Lemoff discloses the system of claim 1, wherein the at least one SWIR image comprises a plurality of SWIR frames of a SWIR video depicting an initial state of the face, changes from the initial state to the facial expression, and the face depicting the facial expression, wherein the code comprises instructions for analyzing the plurality of SWIR frames for detecting the changes from the initial state to the facial expression. (Lemoff; [0049][0051][0052])
With regards to claim 16, modified Lemoff discloses the system of claim 1, wherein the code further comprises instructions for analyzing the at least one SWIR image for distinguishing ears from neighboring hair. (Katz; [0098])
With regards to claim 17, modified Lemoff discloses the system of claim 1, wherein the code further comprises instructions for analyzing the at least one SWIR image for detecting reflections of SWIR illumination from at least one of: a tip of a nose. (Lemoff; [0052])(Katz; [0032])
With regards to claim 22, Katz discloses a method of training a machine-learning model [0022] for facial expression recognition [0032][0081], comprising:
creating a multi-record training dataset [0038], wherein a record comprises:
at least one SWIR image of a face of a sample individual depicting a facial expression captured by at least one SWIR sensor [0041][0048], and
a ground truth label indicating a certain facial expression depicted by the sample individual, the certain facial expression denoting a certain classification category selected from a plurality of predefined facial expressions denoting a plurality of classification categories [0032][0048][0065]; and
training a machine-learning model on the multi-record training dataset [0027][0030][0038][0040], for generating an outcome of a target facial expression classification category of a target individual selected from the plurality of classification categories in response to an input of at least one SWIR image of a target face of the target individual under SWIR illumination [0032][0048][0065].
Allowable Subject Matter
Claims 8, 15, and 18-21 and 23 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
With regards to claim 8, modified Lemoff do not disclose the system of claim 5, wherein the SWIR illumination element generates SWIR illumination at an intensity that does not triggers facial expressions indicating inconvenience, wherein a visible light illumination element generating visible light illumination at the intensity of the SWIR illumination element triggers facial expressions indicating inconvenience.
With regards to claim 15, modified Lemoff do not disclose the system of claim 1, wherein the code further comprises instructions for analyzing the at least one SWIR image for distinguishing regions of the face with high water content from neighboring regions of the face with low water content.
With regards to claim 18, modified Lemoff do not disclose the system of claim 1, wherein the code further comprises instructions for analyzing the at least one SWIR image for distinguishing teeth from surrounding lips.
With regards to claim 19, modified Lemoff do not disclose the system of claim 1, wherein the code further comprises instructions for analyzing the at least one SWIR image for detecting lines and/or creases on the face that are not visible or less visible on visible light images.
With regards to claim 20, modified Lemoff do not disclose the system of claim 1, wherein the code further comprises instructions for analyzing the at least one SWIR image for detecting facial lines below eyes and/or in proximity to a nose.
With regards to claim 21, modified Lemoff do not disclose the system of claim 1, wherein the code further comprises instructions for analyzing the at least one SWIR image for detecting regions of skin that are darker than surrounding neighboring skin, including at least one of: a mole, a birthmark, and a region of high pigmentation.
With regards to claim 23, modified Lemoff do not disclose a method of analyzing an image for facial expression recognition, comprising:
feeding at least one SWIR image of a target face of a target individual under SWIR illumination into a machine-learning model trained as in claim 20; and
obtaining a target facial expression classification category of the target individual selected from the plurality of classification categories as an outcome of the machine-learning model.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Grundmann et al. (US 2019/0228031)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUGH H MAUPIN whose telephone number is (571)270-1495. The examiner can normally be reached M-F 7:30 - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Uzma Alam can be reached at 571-272-3995. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HUGH MAUPIN/ Primary Examiner, Art Unit 2884