Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 3-9, 11-17, and 19-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ishii (U.S. PGPUB 20200285834).
With respect to claim 1, Ishii discloses a computer-implemented method, comprising:
acquiring one or more images of a user (paragraph 81, The feature sensor 11 repeatedly captures a face image of an occupant, and the feature detecting unit 12 detects features (S1));
processing the one more images to determine three-dimensional positions of ears of the user based on a three-dimensional enrollment head geometry of the user (paragraph 65, FIG. 8 is a diagram illustrating a method for estimating three-dimensional coordinates of an undetected ear based on three-dimensional coordinates of the eyes, nose, and ear detected from face images, paragraph 82, The three-dimensional coordinates calculating unit 13 calculates three-dimensional coordinates of the features (S2)), wherein the three-dimensional enrollment head geometry is determined based on one or more enrollment images of the user (paragraph 51, As illustrated in FIG. 1, the feature sensor 11 is provided at a position that allows an image of the occupant's face to be captured, paragraph 59, The feature detecting unit 12 acquires face images of the occupant's face from the feature sensor 11, and detects features such as the eyes, nose, and ears from the face images…The feature detecting unit 12 transmits two-dimensional coordinates of the features to the three-dimensional coordinates calculating unit 13. Internal parameters and external parameters of a camera are calibrated. The three-dimensional coordinates calculating unit 13 uses these parameters and depth data acquired from the depth sensor to calculate coordinates of the features in three-dimensional space from the two-dimensional coordinates of the features); and
processing one or more audio signals to generate one or more processed audio signals based on the three-dimensional positions of the ears (paragraph 53, With the AV function, the facial feature detecting apparatus 10 can deliver optimized sound to the left and right ears of the occupant, paragraph 79, The calculation or estimation of the three-dimensional coordinates of both ears can also be suitably utilized for spatial sound, paragraph 85, Next, the ANC control unit 16 uses the three-dimensional coordinates of both of the occupant's ears to perform the ANC (S6)).
With respect to claim 3, Ishii discloses the computer-implemented method of claim 1, wherein the one or more enrollment images of the user are two-dimensional images captured by a camera (paragraph 59, The feature sensor 11 may be a stereo camera, or may be a monocular camera (which may be a stereo camera) plus a depth sensor).
With respect to claim 4, Ishii discloses the computer-implemented method of claim 1, wherein processing the one or more audio signals to generate the one or more processed audio signals is further based on a speaker configuration indicating one or more of locations or orientations of one or more speakers (paragraph 51, A feature sensor 11 that measures three-dimensional coordinates of features of the face, speakers 20a and 20b, paragraph 76, A factor updating unit 240 updates a noise control filter 230 so as to minimize an error signal that is the difference between noise detected by the microphones 18 and 19 disposed on the headrest 9 and pseudo noise generated by the speakers 20a and 20b disposed on the headrest 9, paragraph 79, For spatial sound, in order to transmit sound to both ears without delay, the distance from musical speakers to the positions of the ears are estimated, respectively. Then, the timing of outputting sound from the speakers is adjusted so that the sound is not delayed by the distances from the speakers to the ears).
With respect to claim 5, Ishii discloses the computer-implemented method of claim 1, wherein the one or more processed audio signals apply one or more audio effects to the one or more audio signals (paragraph 76, The microphones 18 and 19 observe noise reduction effects in the vicinity of the left and right ears, and the speakers 20a and 20b generate pseudo noise to eliminate noise in the vicinity of the left and right ears. A factor updating unit 240 updates a noise control filter 230 so as to minimize an error signal that is the difference between noise detected by the microphones 18 and 19 disposed on the headrest 9 and pseudo noise generated by the speakers 20a and 20b disposed on the headrest 9, paragraph 79, Then, the timing of outputting sound from the speakers is adjusted so that the sound is not delayed by the distances from the speakers to the ears. In addition, the sound output is adjusted such that the sound is output to the ears at approximately the same volume in accordance with the distances from speakers to the ears. In addition, the phase of sound is controlled such that the phase of sound reaches its peak when arriving at the ears).
With respect to claim 6, Ishii discloses the computer-implemented method of claim 1, wherein determining the three-dimensional positions of the ears of the user comprises: generating, based on the one or more images, two-dimensional landmark coordinates for two or more landmarks using a face detection model (paragraph 65, The right ear 51 and the left ear 52 are symmetrical with respect to the median line 223, the right eye 53 and the left eye 54 are bilaterally symmetrical with respect to the median line 223, and the nasal tip 55 is located on the median line 223); and
generating, using the enrollment head geometry, three-dimensional landmark coordinates based on the landmark depth estimates for the two-dimensional landmark coordinates, wherein the three-dimensional positions of the ears are based on the three-dimensional landmark coordinates (paragraph 73, Next, in the face direction as illustrated in FIG. 8, the right ear Y1 is unable to be detected from the face images. However, because the left ear Y2 has been detected, the feature position estimating unit 15 estimates three-dimensional coordinates of the right ear Y1, based on the fact that the right ear Y1 and the left ear Y2 are bilaterally symmetrical with respect to the median line 223).
With respect to claim 7, Ishii discloses the computer-implemented method of claim 6, further comprising: determining a head orientation vector based on the two-dimensional landmark coordinates for the two or more landmarks, and two or more corresponding three-dimensional landmarks in the three-dimensional enrollment head geometry (paragraph 94, In FIG. 13, a circle represents a face in which a triangle 301 connecting the right eye 53, the left eye 54, and the nasal tip 55 is formed. In the proposed system, the face direction can be accurately estimated by using three-dimensional data); and determining the landmark depth estimates based on the head orientation vector (paragraph 95, Namely, the face direction reverse rotation unit 22 reversely rotates three-dimensional coordinates of the right eye 53, the left eye 54, the nasal tip 55, and the left ear 52 by the estimated yaw angle. By reversely rotating the three-dimensional coordinates, the occupant's face faces the front. Therefore, the 3D model accumulation unit 23 stores the three-dimensional coordinates of the right eye 53, the left eye 54, the nasal tip 55, and the left ear 52 in the 3D model storage 24, as illustrated in the lower part of FIG. 14). By using the estimated yaw angle, this determines landmark depth estimates based on the head orientation vector.
With respect to claim 8, Ishii discloses the computer-implemented method of claim 6, wherein the three-dimensional positions of the ears of the user are determined based on one or more relationships in the enrollment head geometry, wherein the one or more relationships relate the three-dimensional landmark coordinates to the three-dimensional positions of the ears (paragraph 72, The median line 223 passes through the midpoint M and the nasal tip N, paragraph 73, However, because the left ear Y2 has been detected, the feature position estimating unit 15 estimates three-dimensional coordinates of the right ear Y1, based on the fact that the right ear Y1 and the left ear Y2 are bilaterally symmetrical with respect to the median line 223).
With respect to claim 9, Ishii discloses the computer-implemented method of claim 6, wherein the two or more landmarks include one or more of an eye center landmark, an eye outer point landmark, an eye inner point landmark, an eyebrow outer point landmark, and eyebrow inner point, a nose bridge landmark, a nose tip landmark, a nose base landmark, a nose root landmark, a glabella landmark, a mouth tip landmark, an upper lip midpoint landmark, a lower lip midpoint landmark, a chin landmark, or a jawline landmark (paragraphs 66-71, right eye, left eye, nasal tip, right ear, left ear).
With respect to claim 11, Ishii discloses the computer-implemented method of claim 1, further comprising: generating, using one or more speakers, a sound field that includes one or more audio effects based on the one or more processed audio signals (paragraph 76, A factor updating unit 240 updates a noise control filter 230 so as to minimize an error signal that is the difference between noise detected by the microphones 18 and 19 disposed on the headrest 9 and pseudo noise generated by the speakers 20a and 20b disposed on the headrest 9, paragraph 77, When the ANC is used in a three-dimensional sound field, a quiet zone can be created around the error sensors, but the size of the quiet zone is determined by the frequency of noise).
With respect to claim 12, Ishii discloses the computer-implemented method of claim 11, wherein the one or more speakers include one or more of headrest speakers, gaming chair speakers, or sound bar speakers (paragraph 52, As illustrated in FIG. 5, the speakers 20a and 20b and the microphones 18 and 19 are mounted on a headrest 9 to implement the headrest ANC).
With respect to claim 13, Ishii discloses the computer-implemented method of claim 11, wherein the audio effects include one or more of a spatial audio effect, noise cancellation, or crosstalk cancellation (paragraph 76, The ANC may be feedforward control, or may be hybrid control in which both feedback control and feedforward control are used. In addition, when a plurality of microphones and speakers are installed as in the present embodiment, control that removes a crosstalk component may be added, paragraph 79, The calculation or estimation of the three-dimensional coordinates of both ears can also be suitably utilized for spatial sound).
With respect to claim 14, Ishii discloses one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors (paragraph 58, The functions illustrated in FIG. 6 are functions or means implemented by causing the CPU to execute application software (or a program) loaded from the flash memory to the RAM and control various types of hardware), cause the one or more processors to perform the steps of:
acquiring two-dimensional image data of a user (paragraph 81, The feature sensor 11 repeatedly captures a face image of an occupant, and the feature detecting unit 12 detects features (S1));
determining three-dimensional ear positions of the user based on the two-dimensional image data and a user-specific three-dimensional head geometry (paragraph 65, FIG. 8 is a diagram illustrating a method for estimating three-dimensional coordinates of an undetected ear based on three-dimensional coordinates of the eyes, nose, and ear detected from face images, paragraph 82, The three-dimensional coordinates calculating unit 13 calculates three-dimensional coordinates of the features (S2)), wherein the user-specific three-dimensional head geometry is generated based on a subset of the two-dimensional image data (paragraph 51, As illustrated in FIG. 1, the feature sensor 11 is provided at a position that allows an image of the occupant's face to be captured, paragraph 59, The feature detecting unit 12 acquires face images of the occupant's face from the feature sensor 11, and detects features such as the eyes, nose, and ears from the face images…The feature detecting unit 12 transmits two-dimensional coordinates of the features to the three-dimensional coordinates calculating unit 13. Internal parameters and external parameters of a camera are calibrated. The three-dimensional coordinates calculating unit 13 uses these parameters and depth data acquired from the depth sensor to calculate coordinates of the features in three-dimensional space from the two-dimensional coordinates of the features); and
generating, using one or more speakers, a sound field that includes one or more audio effects based on the three-dimensional ear positions of the user (paragraph 53, With the AV function, the facial feature detecting apparatus 10 can deliver optimized sound to the left and right ears of the occupant, paragraph 77, When the ANC is used in a three-dimensional sound field, a quiet zone can be created around the error sensors, but the size of the quiet zone is determined by the frequency of noise, paragraph 79, The calculation or estimation of the three-dimensional coordinates of both ears can also be suitably utilized for spatial sound, paragraph 85, Next, the ANC control unit 16 uses the three-dimensional coordinates of both of the occupant's ears to perform the ANC (S6)).
With respect to claim 15, Ishii discloses the one or more non-transitory computer-readable media of claim 14, further comprising one or more speakers (paragraph 51, A feature sensor 11 that measures three-dimensional coordinates of features of the face, speakers 20a and 20b), and wherein the steps further comprise: generating, based on the two-dimensional image data, two-dimensional landmark coordinates for two or more landmarks using a face detection model (paragraph 65, The right ear 51 and the left ear 52 are symmetrical with respect to the median line 223, the right eye 53 and the left eye 54 are bilaterally symmetrical with respect to the median line 223, and the nasal tip 55 is located on the median line 223); and
generating, using the user-specific three-dimensional head geometry, three-dimensional landmark coordinates based on the and landmark depth estimates for the two-dimensional landmark coordinates, wherein the three-dimensional ear positions are based on the three-dimensional landmark coordinates (paragraph 73, Next, in the face direction as illustrated in FIG. 8, the right ear Y1 is unable to be detected from the face images. However, because the left ear Y2 has been detected, the feature position estimating unit 15 estimates three-dimensional coordinates of the right ear Y1, based on the fact that the right ear Y1 and the left ear Y2 are bilaterally symmetrical with respect to the median line 223).
With respect to claim 16, Ishii discloses the one or more non-transitory computer-readable media of claim 14, wherein the audio effects include one or more of a spatial audio effect, noise cancellation, or crosstalk cancellation (paragraph 76, The ANC may be feedforward control, or may be hybrid control in which both feedback control and feedforward control are used. In addition, when a plurality of microphones and speakers are installed as in the present embodiment, control that removes a crosstalk component may be added, paragraph 79, The calculation or estimation of the three-dimensional coordinates of both ears can also be suitably utilized for spatial sound).
With respect to claim 17, Ishii discloses the one or more non-transitory computer-readable media of claim 14, wherein the generating the one or more processed audio signals is further based on a speaker configuration of the one or more speakers (paragraph 51, A feature sensor 11 that measures three-dimensional coordinates of features of the face, speakers 20a and 20b, paragraph 76, A factor updating unit 240 updates a noise control filter 230 so as to minimize an error signal that is the difference between noise detected by the microphones 18 and 19 disposed on the headrest 9 and pseudo noise generated by the speakers 20a and 20b disposed on the headrest 9, paragraph 79, For spatial sound, in order to transmit sound to both ears without delay, the distance from musical speakers to the positions of the ears are estimated, respectively. Then, the timing of outputting sound from the speakers is adjusted so that the sound is not delayed by the distances from the speakers to the ears).
With respect to claim 19, Ishii discloses the one or more non-transitory computer-readable media of claim 14, wherein the three-dimensional ear positions are determined based on one or more ear relationships in the user-specific three-dimensional head geometry, wherein the one or more ear relationships relate the three-dimensional landmark coordinates to the three-dimensional positions of the ears (paragraph 72, The median line 223 passes through the midpoint M and the nasal tip N, paragraph 73, However, because the left ear Y2 has been detected, the feature position estimating unit 15 estimates three-dimensional coordinates of the right ear Y1, based on the fact that the right ear Y1 and the left ear Y2 are bilaterally symmetrical with respect to the median line 223).
With respect to claim 20, Ishii discloses a system (paragraph 51, FIG. 5 is a side view of a vehicle with a facial feature detecting apparatus) comprising: one or more speakers (paragraph 51, A feature sensor 11 that measures three-dimensional coordinates of features of the face, speakers 20a and 20b, and microphones 18 and 19 (speakers with microphones) are connected to the facial feature detecting apparatus 10); a camera that captures one or more images of a user (paragraph 59, The feature sensor 11 may be a stereo camera, or may be a monocular camera (which may be a stereo camera) plus a depth sensor); a memory storing instructions; and one or more processors, that when executing the instructions (paragraph 58, The facial feature detecting apparatus 10 functions as an information processing apparatus including a CPU, a RAM, a ROM, a flash memory, an I/O device, a communication device, and a battery. The functions illustrated in FIG. 6 are functions or means implemented by causing the CPU to execute application software (or a program) loaded from the flash memory to the RAM and control various types of hardware), are configured to perform the steps of claim 14; see rationale for rejection of claim 14.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ishii (U.S. PGPUB 20200285834) in view of Casado et al. (U.S. PGPUB 20200175292).
With respect to claim 2, Ishii discloses the computer-implemented method of claim 1. However, Ishii does not expressly disclose the one or more enrollment images are selected from the one or more images based on one or more of a face orientation or a face detection status.
Casado et al., who also deal with user enrollment, disclose a method wherein the one or more enrollment images are selected from the one or more images based on one or more of a face orientation or a face detection status (paragraph 142, enrollment engine 132 may analyze each of the plurality of images 960 to select a subset of images that adequately portray user 101 from multiple angles. For example, in some implementations, enrollment engine 132 may select one image that is a front view, another image that is a first side view, another image that is a second side view, and so forth).
Ishii and Casado et al. are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method wherein the one or more enrollment images are selected from the one or more images based on one or more of a face orientation or a face detection status, as taught by Casado et al., to the Ishii system, because this may identify the at least one image that is usable to create the biometric data (paragraph 143 of Casado et al.).
With respect to claim 18, Ishii as modified by Casado et al. disclose the one or more non-transitory computer-readable media of claim 14, wherein the subset of the two-dimensional image data is selected based on one or more of an indication of face orientation, or an indication of face detection (Casado et al.: paragraph 142, enrollment engine 132 may analyze each of the plurality of images 960 to select a subset of images that adequately portray user 101 from multiple angles. For example, in some implementations, enrollment engine 132 may select one image that is a front view, another image that is a first side view, another image that is a second side view, and so forth); see rationale for rejection of claim 2.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ishii (U.S. PGPUB 20200285834) in view of Milne et al. (U.S. PGPUB 20200107148).
With respect to claim 10, Ishii discloses the computer-implemented method of claim 1. However, Ishii does not expressly disclose processing the one or more audio signals includes: determining one or more head-related transfer functions (HRTFs) based on the three-dimensional positions of the ears; and modifying the one or more audio signals based on the HRTFs to generate the one or more processed audio signals.
Milne et al., who also deal with user enrollment, disclose a method wherein processing the one or more audio signals includes: determining one or more head-related transfer functions (HRTFs) based on the three-dimensional positions of the ears (paragraph 36, HRTF files may be generated by inputting, for example, to a machine learning module, or by accessing a database using as input, a series of photographs of a person's left and right ears taken from different positions and/or a three-dimensional (3D) image of the ears. Using the photographs, the machine learning module outputs a HRTF, preferably one for each ear); and modifying the one or more audio signals based on the HRTFs to generate the one or more processed audio signals (paragraph 45, Head orientation signals from the user's headphones or from another source (such as a camera imaging the user) may be received at block 706, and the corresponding FIR filter from the HRTF files selected for the sensed orientation. When a virtual venue has been selected, at block 708 it is concatenated with the user-personalized FIR filter selected at block 704 corresponding to the user's head orientation and then the concatenation is convoluted with the selected audio track and played).
Ishii and Milne et al. are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method wherein processing the one or more audio signals includes: determining one or more head-related transfer functions (HRTFs) based on the three-dimensional positions of the ears; and modifying the one or more audio signals based on the HRTFs to generate the one or more processed audio signals, as taught by Milne et al., to the Ishii system, because by convoluting an audio stream with a FIR filter, a modified audio stream is produced which is perceived by a listener to come not from, e.g., headphone speakers adjacent the ears of the listener but rather from relatively afar, as sound would come from an orchestra for example on a stage that the listener is in front of (paragraph 34 of Milne et al.).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. PGPUB 20230394886 to Carrigan et al. for a method of implementing a front profile, right ear, and left ear enrollment interface
U.S. PGPUB 20220322024 to Subramanian et al. for a method of implementing an enrollment process
U.S. PGPUB 20200372792 to Li et al. for a method of computing a head pose vector.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW GUS YANG whose telephone number is (571)272-5514. The examiner can normally be reached M-F 9 AM - 5:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW G YANG/Primary Examiner, Art Unit 2614
6/12/26
Prosecution Insights