DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDSs) submitted on December 24, 2024 and August 22, 2025 are in compliance with the provisions of 37 CFR 1.97 and 1.98. Accordingly, the IDSs have been considered by the examiner and placed in the file.
Drawings
The drawings are objected to because Figs. 5 and 7 contain grammatical errors, namely, misspellings of the word “Pattern” as “Parttern”. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-12 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Independent claims 1, 11 and 12 recite specifying “the elements of the imaging condition suitable to be improved on the first face image”. Because these claims recite “a first state of the elements” and “a second state of the elements” it is unclear whether this limitation refers to specifying the elements of the imaging condition of the first state or the second state as suitable to be improved. For this reason, claims 1, 11 and 12 are rejected under 35 U.S.C. 112(b) as being indefinite.
Claims 2 – 10 are rejected under 35 U.S.C. 112(b) as being indefinite due to their dependencies from claim 1. Additionally, claims 2, 3, 7 and 9 refer to “elements of the imaging condition” without clearly indicating whether they are of the first state or the second state. For this additional reason, claims 2, 3, 7 and 9 are rejected as being indefinite.
The BRIs for limitations of claims 1, 11 and 12, as best as can be understood, are provided below.
Allowable Subject Matter
Claims 1-12 would be allowable if claims 1, 2, 3, 7, 9, 11 and 12 were amended to overcome the rejections under 35 U.S.C. 112(b).
The following is a statement of reasons for the indication of allowable subject matter.
Regarding independent claims 1, 11 and 12, none of the prior art teaches or suggests at least the combined limitations of:
inputting the second state to a machine learning model generated through training for each of action units (AUs) that represent movements of facial expression muscles, with states of the elements of the imaging condition for a face image as features and errors in estimated values with respect to ground truth values of intensities of the AUs as ground truth data, to estimate prediction errors for each of the AUs;
determining whether or not predetermined criteria are satisfied by all of the prediction errors for each of the AUs; and
specifying the elements of the imaging condition suitable to be improved on the first face image, based on a determination result as to whether or not the predetermined criteria are satisfied.
The BRI for “state of the elements of the imaging condition” is that it means one or more values corresponding to one or more conditions that exist at the time the image is acquired that affect some quality or characteristic of the acquired image. The BRI is based on paras. [0030], [0034] and [0041] of the present specification. Examples given of the image condition elements include distance of the subject from the camera when the image is captured, occlusion of the upper or lower parts of the subject’s face at the time of image acquisition, and the bright and darkness of the acquired image.
The BRI for “a machine learning model generated through training for each of action units (AUs) that represent movements of facial expression muscles, with states of the elements of the imaging condition for a face image as features and errors in estimated values with respect to ground truth values of intensities of the AUs as ground truth data” is based on para. [0055] and Fig. 4 of the present disclosure. Based on this portion of the present disclosure, the BRI is that the machine learning model is trained by using values corresponding to the imaging conditions as features and generating prediction errors in estimating intensity values of AUs of face images based on the corresponding ground truths for the intensity values of the AUs.
The BRI for “inputting the second state to a machine learning model … to estimate prediction errors for each of the AUs”, based on paras. [0058]-[0061] of the present specification, is that it means that imaging condition values of the second state are inputted to the trained machine learning model, which, in turn, generates an estimated prediction error for each of the AUs of the first image.
The BRI for “determining whether or not predetermined criteria are satisfied by all of the prediction errors for each of the AUs” and “specifying the elements of the imaging condition suitable to be improved on the first face image, based on a determination result as to whether or not the predetermined criteria are satisfied” is that a determination is made as to whether the estimated prediction errors satisfy some criteria and, based on that determination, imaging condition values that are suitable for improvement are specified for the first image.
None of the prior art teaches or suggests this combination of limitations according to the BRIs, namely, of using a machine learning model trained to receive, as input, image condition values of face images and to generate estimated prediction errors for AUs of the face images, to estimate prediction errors for each AU of a first image based on imaging conditions inputted to the model, and then determining whether the estimated prediction errors satisfy some predetermined criteria and specifying an image condition value(s) to be improved based on the determination.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. Publ. Appl. No. 2020/0058136 A1 to Morishita (hereinafter referred to as “Morishita”) discloses training a machine learning model comprising gaze estimators with images of eye regions of the face acquired under different imaging conditions to estimate the gaze of the subject (Paras. [0075] and [0079]). During operations, a first face image of a person is acquired and elements of the imaging condition are specified, such as “an installation angle of an imaging device. In addition, an imaging condition can include a parameter (a field angle or the like) of a lens of an imaging device, and an estimated range of a gaze at a time of capturing.” (Para. [0050]). The trained model estimates the gaze by integrating gaze estimates of gaze estimators based on the imaging conditions. (Paras. [0075]-[0079]). Morishita, taken alone or in combination with other prior art, does not teach or suggest the combination of limitations discussed above.
U.S. Publ. Appl. No. 2019/0102608 A1 to Wang et al. (hereinafter referred to as “Wang”) discloses a machine learning model that is trained to predict AUs of face images acquired under different imaging conditions, such as images acquired at different times. Given an imaging condition, the system can predict the image features of a physical entity, such as a human face. In some embodiments, after extracting image features (e.g., the RGB or grayscale values of one or more previously defined image blocks on the face, the attitude angle of the face, and/or the image size of the face) from a particular image taken under an imaging condition (e.g., the lighting condition provided by the display or flash, the position of the smartphone, etc.), the system can determine whether the extracted image features match the predicted image features corresponding to the same imaging condition. (Para. [0088]). Wang, taken alone or in combination with other prior art, does not teach or suggest the combination of limitations discussed above.
U.S. Publ. Appl. No. 2008/0279423 A1 to Zhang et al. (hereinafter referred to as “Zhang”) discloses a process for improving an illumination of a single image containing a face and taken under sub-optimal illumination conditions, comprising: computing an initial shape estimation of the face to generate a face image, wherein the face image is the portion of the image containing the face; classifying each pixel in the face image as one of: (a) a saturated pixel; (b) a shadow pixel; (c) a regular pixel; (d) an occluded pixel; weighting each pixel in the face image based on its classification; assigning each pixel to one of a plurality of regions in the face image based on the pixel classification; generating an albedo morphable model for each of the plurality of regions; and obtaining image parameters from the morphable models that are used to improve the illumination of the single image containing the face. (Claim 16). Zhang, taken alone or in combination with other prior art, does not teach or suggest the combination of limitations discussed above.
An article entitled “Facial Action Unit Detection Based on Teacher-Student Learning Framework for Partially Occluded Facial Images”, by Kawamura et al., published in 2021 in 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (hereinafter referred to as “Kawamura”) discloses
a teacher-student learning framework with two types of loss functions. We use teacher student learning to distill knowledge from the model of facial AU detection without occlusion. In the teacher-student learning framework, the teacher model is pre-trained on non-occluded facial images. The student model receives the occluded facial images and both the teacher model‘s output and ground truth data are utilized to drive the student model. Two types of loss functions are utilized to distill knowledge
from non-occluded facial images. The first type is distillation loss between the teacher model and the student model. This function constrains the student model to output the
same prediction values of AU occurrence as that of the teacher model. The second type is order regularization loss. The distillation loss does not consider the relation between
images such as the order of the output of prediction values.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL J SANTOS whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella can be reached at (571)272-2867. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL J. SANTOS/Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights