DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 09/27/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Status
Claim(s) 1, 5-7 and 9-10 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Grossinger et al (U.S. 20170243334 A1; Grossinger)
Claim(s) 2-3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger et al (U.S. 20170243334 A1; Grossinger), in view of Lei (U.S. 20210295015 A1).
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger et al (U.S. 20170243334 A1; Grossinger), in view of Morito et al (WO–2022091642 A1; Morito).
Examiner Noted: See the PDF of WO–2022091642 A1 provided by Examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 5-7 and 9-10 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Grossinger et al (U.S. 20170243334 A1; Grossinger).
Regarding claim 1, Grossinger discloses An information processing apparatus (Paragraph 10: “ a method and a system for reconstructing obstructed face portions for virtual reality environments”) comprising: at least one memory that is configured to store instructions; and at least one processor (Fig.5: a computer processor 520) that is configured to execute the instructions (Paragraph 34: “the processor is carried out via searching on networked databases.”) to:
acquire information about a person including at least an image of the person; (Paragraph 39: “In a first step, off-line 3D data (e.g., in the form of a 3D model) of the head of a person 610 is being obtained, possibly but not exclusively based on a plurality of 2D images of the person 612, 614, and 616. 2D images of the person 612, 614, and 616 may be captured in an off-line session prior to wearing the helmet … the off-line 3D data may be obtained using structured light technology, prior to wearing the virtual reality headgear.”)
detect a face area including a face of the person from the image; (Paragraph 40: “the off-line 3D data may be obtained using structured light technology, prior to wearing the virtual reality headgear. … depth map of the head or face of the user may be generated from which the off-line 3D data may be retrieved”)
estimate a hidden shield area in a case where at least a part of the face area is hidden; (Paragraph 31: “the sensing by the sensors may be in the form of image capturing and wherein said obtained data is an image of portions of the obstructed face.”; Paragraph 28: “a 3D data capturing device 510 configured to capture a 3D data (e.g., image) 512 of a scene containing at least one person wearing a face-obstructing object,”)
estimate an expression of the person on the basis of the information about the person; (Paragraph 41-42: “real-time 3D data (e.g., model) of the user 620 is obtained, while the person is wearing a face obstructing object 622 (e.g., the virtual reality headset) the real-time image is being constantly updated … Real-time parameters 644 may include, for example: position and orientation of the user's face, or expressions, but also how tired the person is, and what type of emotions are being experienced in real time. These real-time parameters 644 are all being used in order to estimate and reconstruct the face portions that are being obstructed by the virtual reality headset”)
generate an estimated expression image of an area corresponding to the shield area, in accordance with the estimated expression; (Paragraphs 41-43: “a 3D transformation may be applied in a reconstruction module 640 to off-line 3D data 610 or a portion of it 618 (which corresponds with the borders of the obstructed portions 634 which may be segmented out from 3D model 632). The 3D transformation may be based on real-time 3D data (model) 620 and more specifically, real-time parameters 644 from which the appearance of obstructed portions 634 may be estimated. … the product of the reconstruction module (or step) is a reconstructed real-time data (or model) 650 which includes a sub portion of area that is affected by non-obscured portions 652 … The appearance of this area may be estimated based on real-time parameters 644 relating to the non-obstructed portions. Another area is the area that is not affected by non-obstructed portions 654 (e.g., the eyes). The appearance of the eyes may be estimated based on another model, and using meta data 642 from sources external to real-time 2D data 622”) and
a generate a composite image on the basis of the image and the estimated expression image. (Paragraph 43: “ reconstructed real-time 3D data (or model) 650 may be merged into real-time 3D data (or model) possibly with the obstructed portions segmented out 632, to yield a reconstructed real-time 3D data (or model) of the head of the person, with the obstructed portions reconstructed. Model 660 may then be generated into a 3D image to be presented to the other person(s) participating in the virtual reality environment.”)
Regarding claim 5, Grossinger discloses the at least one processor is configured to execute the instructions to generate the estimated expression image on the basis of a previously registered image of the person in which at least the shield area is not shielded. (Paragraph 28: “A possible source for 2D images of users faces may be social networks such as add Facebook™ or LinkedIn™ which may store profile images of users. These 2D images may be used for generating 3D data of the head or the face of the user”; Paragraph 32; Paragraph 39: “off-line 3D data (e.g., in the form of a 3D model) of the head of a person 610 is being obtained, possibly but not exclusively based on a plurality of 2D images of the person 612, 614, and 616. 2D images of the person 612, 614, and 616 may be captured in an off-line session prior to wearing the helmet”
Regarding claim 6, Grossinger discloses the at least one processor is configured to execute the instructions to generate the estimated expression image generation unit generates the estimated expression image on the basis of the previously registered image of the person with an expression corresponding to the estimated expression estimated by the expression estimation unit. (Paragraph 27; Paragraph 32: “the sensors may be configured to sense facial gestures, and wherein the reconstructing of a face image is carried out by modeling sensed gestures to change the face relative to a base image of the obstructed face portions captured previously.”; Paragraph 53: “the 3D transformation may further include using data obtained from non-obstructed face portions of the person for estimating obstructed face portions that are affected by changes to the non-obstructed face portions. Specifically, the estimating may be carried out based on sensing facial expression from the real-time 3D data.”;
Regarding claim 7, Grossinger discloses wherein the at least one processor is configured to execute the instructions to: display the composite image instead of the image in a case the composite image is generated; and that superimpose and displays information indicating the generated image on the composite image. (Paragraph 29: “Computer processor 520 may be further configured to reconstruct 3D data of the (e.g., an image) of the obstructed face portions based on the obtained data. Computer processor 520 may be further configured to merge the reconstructed face image into respective location at the captured image of the scene. System 500 may further include a near eye display 570 configured to present the merged image to the specified user, wherein the merged image is placed into a computer-simulated environment adjustable based on the view point of the specified user.”; Paragraph 43; 45)
Regarding claim 9, Grossinger discloses An information processing method (Paragraph 10: “ a method and a system for reconstructing obstructed face portions for virtual reality environments”) comprising:
acquiring information about a person including at least an image of the person; (Paragraph 39: “In a first step, off-line 3D data (e.g., in the form of a 3D model) of the head of a person 610 is being obtained, possibly but not exclusively based on a plurality of 2D images of the person 612, 614, and 616. 2D images of the person 612, 614, and 616 may be captured in an off-line session prior to wearing the helmet … the off-line 3D data may be obtained using structured light technology, prior to wearing the virtual reality headgear.”)
detecting a face area including a face of the person from the image; (Paragraph 40: “the off-line 3D data may be obtained using structured light technology, prior to wearing the virtual reality headgear. … depth map of the head or face of the user may be generated from which the off-line 3D data may be retrieved”)
estimating a hidden shield area in a case where at least a part of the face area is hidden; (Paragraph 31: “the sensing by the sensors may be in the form of image capturing and wherein said obtained data is an image of portions of the obstructed face.”; Paragraph 28: “a 3D data capturing device 510 configured to capture a 3D data (e.g., image) 512 of a scene containing at least one person wearing a face-obstructing object,”)
estimating an expression of the person on the basis of the information about the person; (Paragraph 41-42: “real-time 3D data (e.g., model) of the user 620 is obtained, while the person is wearing a face obstructing object 622 (e.g., the virtual reality headset) the real-time image is being constantly updated … Real-time parameters 644 may include, for example: position and orientation of the user's face, or expressions, but also how tired the person is, and what type of emotions are being experienced in real time. These real-time parameters 644 are all being used in order to estimate and reconstruct the face portions that are being obstructed by the virtual reality headset”)
generating an estimated expression image of an area corresponding to the shield area, in accordance with the estimated expression; (Paragraphs 41-43: “a 3D transformation may be applied in a reconstruction module 640 to off-line 3D data 610 or a portion of it 618 (which corresponds with the borders of the obstructed portions 634 which may be segmented out from 3D model 632). The 3D transformation may be based on real-time 3D data (model) 620 and more specifically, real-time parameters 644 from which the appearance of obstructed portions 634 may be estimated. … the product of the reconstruction module (or step) is a reconstructed real-time data (or model) 650 which includes a sub portion of area that is affected by non-obscured portions 652 … The appearance of this area may be estimated based on real-time parameters 644 relating to the non-obstructed portions. Another area is the area that is not affected by non-obstructed portions 654 (e.g., the eyes). The appearance of the eyes may be estimated based on another model, and using meta data 642 from sources external to real-time 2D data 622”) and
generating a composite image on the basis of the image and the estimated expression image. (Paragraph 43: “ reconstructed real-time 3D data (or model) 650 may be merged into real-time 3D data (or model) possibly with the obstructed portions segmented out 632, to yield a reconstructed real-time 3D data (or model) of the head of the person, with the obstructed portions reconstructed. Model 660 may then be generated into a 3D image to be presented to the other person(s) participating in the virtual reality environment.”)
Regarding claim 10, Grossinger discloses A non-transitory recording medium on which a computer program (Paragraph 34: “the processor is carried out via searching on networked databases.”) that allows a computer to execute an information processing method is recorded, (Fig.5: a computer processor 520) the information processing method including:
acquiring information about a person including at least an image of the person; (Paragraph 39: “In a first step, off-line 3D data (e.g., in the form of a 3D model) of the head of a person 610 is being obtained, possibly but not exclusively based on a plurality of 2D images of the person 612, 614, and 616. 2D images of the person 612, 614, and 616 may be captured in an off-line session prior to wearing the helmet … the off-line 3D data may be obtained using structured light technology, prior to wearing the virtual reality headgear.”)
detecting a face area including a face of the person from the image; (Paragraph 40: “the off-line 3D data may be obtained using structured light technology, prior to wearing the virtual reality headgear. … depth map of the head or face of the user may be generated from which the off-line 3D data may be retrieved”)
estimating a hidden shield area in a case where at least a part of the face area is hidden; (Paragraph 31: “the sensing by the sensors may be in the form of image capturing and wherein said obtained data is an image of portions of the obstructed face.”; Paragraph 28: “a 3D data capturing device 510 configured to capture a 3D data (e.g., image) 512 of a scene containing at least one person wearing a face-obstructing object,”)
estimating an expression of the person on the basis of the information about the person; (Paragraph 41-42: “real-time 3D data (e.g., model) of the user 620 is obtained, while the person is wearing a face obstructing object 622 (e.g., the virtual reality headset) the real-time image is being constantly updated … Real-time parameters 644 may include, for example: position and orientation of the user's face, or expressions, but also how tired the person is, and what type of emotions are being experienced in real time. These real-time parameters 644 are all being used in order to estimate and reconstruct the face portions that are being obstructed by the virtual reality headset”)
generating an estimated expression image of an area corresponding to the shield area, in accordance with the estimated expression; (Paragraphs 41-43: “a 3D transformation may be applied in a reconstruction module 640 to off-line 3D data 610 or a portion of it 618 (which corresponds with the borders of the obstructed portions 634 which may be segmented out from 3D model 632). The 3D transformation may be based on real-time 3D data (model) 620 and more specifically, real-time parameters 644 from which the appearance of obstructed portions 634 may be estimated. … the product of the reconstruction module (or step) is a reconstructed real-time data (or model) 650 which includes a sub portion of area that is affected by non-obscured portions 652 … The appearance of this area may be estimated based on real-time parameters 644 relating to the non-obstructed portions. Another area is the area that is not affected by non-obstructed portions 654 (e.g., the eyes). The appearance of the eyes may be estimated based on another model, and using meta data 642 from sources external to real-time 2D data 622”) and
generating a composite image on the basis of the image and the estimated expression image. (Paragraph 43: “ reconstructed real-time 3D data (or model) 650 may be merged into real-time 3D data (or model) possibly with the obstructed portions segmented out 632, to yield a reconstructed real-time 3D data (or model) of the head of the person, with the obstructed portions reconstructed. Model 660 may then be generated into a 3D image to be presented to the other person(s) participating in the virtual reality environment.”)
.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2-3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger et al (U.S. 20170243334 A1; Grossinger), in view of Lei (U.S. 20210295015 A1).
Regarding claim 2, Grossinger discloses the shield area that is at least a hidden part of the face area, is a mask area hidden by a mask worn by the person.
However, Grossinger does not disclose the shield area least a hidden part of the face area, is a mask area hidden by a mask worn by the person.
Lei discloses the shield area least a hidden part of the face area, is a mask area hidden by a mask worn by the person. (Paragraph 42-43: “Step 202: acquiring a mask image, and combining, based on the coordinates of the key points, the mask image with the face image to generate a mask wearing face image containing a mask wearing face … the execution subject may also acquire the mask image … There is a mask wearing face in the mask wearing face image, that is, the face at the mask is occluded. In this way, at least one face in the face image is partially occluded, and the full face is no longer presented.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Grossinger by including face detection of an image that is taught by Lei, to make the invention that a method and apparatus for processing information; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving the detection accuracy such as a face wearing a mask, and a face not wearing a mask as well as enhancing the generating face image sample for training deep neural network.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claim 3, Grossinger, as modified by Lei, discloses all the claims invention. Grossinger further discloses the at least one processor is configured to execute the instructions to estimate the expression estimation unit estimates the expression of the person on the basis of an area around eyes of the person. (Paragraphs 42-43: “Real-time parameters 644 may include, for example: position and orientation of the user's face, or expressions, but also how tired the person is, and what type of emotions are being experienced in real time. … the appearance of this area may be estimated based on real-time parameters 644 relating to the non-obstructed portions. Another area is the area that is not affected by non-obstructed portions 654 (e.g., the eyes). The appearance of the eyes may be estimated based on another model, and using meta data 642 from sources external to real-time 2D data 622”; Paragraph 45: “. Examples for real time parameters may be direction of view of the participant, his or her physical condition such as how tired he or she is which may affect their eye positions and general appearance of their face and the like, their current mood e.g. happy/angry/sad etc., their nearby environment e.g. at home or outside on the beach in a hot weather.”)
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Grossinger et al (U.S. 20170243334 A1; Grossinger), in view of Morito et al (WO–2022091642 A1; Morito).
Regarding claim 4, Grossinger, discloses all the claims invention except wherein the at least one processor is configured to execute the instructions to: acquire learning information including sample information about a sample person with a predetermined expression and an expression label indicating the predetermined expression, estimate an expression of the sample person on the basis of the sample information, learn a method of estimating the expression of the person on the basis of the expression label and an estimation result of the expression of the sample person.
Morito discloses acquire learning information including sample information about a sample person with a predetermined expression and an expression label indicating the predetermined expression, (Paragraph 21 : “a predetermined medical cognitive function evaluation scale of each subject is used as a label (correct answer data). Specifically, the combination (data set) of the information about the facial expression score of each subject and the predetermined medical cognitive function evaluation scale of each subject is the teacher data (labeled data) in the machine learning of the learning model 410.”)
estimate an expression of the sample person on the basis of the sample information, (Paragraph 24-25: “a facial expression score is calculated for each of the plurality of facial images extracted from the plurality of images … Information 120 regarding the facial expression score regarding the person is generated based on the plurality of facial expression scores regarding the person”) and
learn a method of estimating the expression of the person on the basis of the expression label (Paragraph 28: “ in the learning model 410, the relationship between the information 120 regarding the facial expression score and the degree of cognitive function is learned. Specifically, in the learning model 410, the relationship between the information 120 regarding the facial expression score and the predetermined medical cognitive function evaluation scale is learned, and the trained model 420 is generated.”) and an estimation result of the expression of the sample person. (Paragraphs 31-32: “Then, a value (recognition degree score D) corresponding to a predetermined medical test value (MMSE) is output from the trained model 420. In other words, the cognitive function determination device 30 uses the trained model 420 to obtain a "recognition degree score" (a medical cognitive function evaluation scale by the MMSE) for the determination target person based on the information regarding the facial expression score of the determination target person.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Grossinger by including the processing of the learning stage in machine learning. that is taught by Morito, to make the invention that generates a learning model that learns the relationship between information pertaining to facial expression scores of each of a plurality of examinees and a prescribed medical cognitive function assessment scale indicating the cognitive function assessment results of each of the plurality of examinees; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving face recognition processing by using machine learning as well as reducing the error.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ono (U.S. 20170039411 A1), “ Image Capturing Apparatus and Image Processing Method”, teaches about an image capturing apparatus that can capture a visible light image and an infrared light image of the same object, the image capturing apparatus comprises a detection unit configured to detect a predetermined object in the visible light image, an extraction unit configured to extract feature information of a specific portion in the object detected in the infrared light image by the detection unit, and an estimation unit configured to estimate unique information of the predetermined object using the feature information extracted by the extraction unit.
Mori et al (U.S. 20060115157 A1), “Image Processing Device, Image Device, Image Processing Method”, teaches about an image including a face is input, a plurality of local features are detected from the input image, a region of a face in the image is specified using the plurality of detected local features, and an expression of the face is determined on the basis of differences between the detection results of the local features in the region of the face and detection results which are calculated in advance as references for respective local features in the region of the face.
Kaneda (U.S. 20110032378 A1), “a facial expression recognition apparatus, an image sensing apparatus, a facial expression recognition method, and a computer-readable storage medium”, teaches about a facial expression recognition apparatus detects a face image of a person from an input image, calculates a facial expression evaluation value corresponding to each facial expression from the detected face image, updates, based on the face image, the relationship between the calculated facial expression evaluation value and a threshold for determining a facial expression set for the facial expression evaluation value, and determines the facial expression of the face image based on the updated relationship between the facial expression evaluation value and the threshold for determining a facial expression.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Duy A Tran whose telephone number is (571)272-4887. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ONEAL R MISTRY can be reached at (313)-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUY TRAN/ Examiner, Art Unit 2674
/ONEAL R MISTRY/ Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights