DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 7-9 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application No. 20190043518 (Li) in view of U.S. Patent No. 2020/0312341 (Alves) and further in view of U.S. Patent Application No. 20200219506 (Bhowmik).
With respect to claim 1
Regarding receiving a first signal from one or more in-canal microphones positioned in an ear canal of a wearer, Li teaches a headset having an internal microphone positioned in or on an inside portion of an earpiece such that, during use, the microphone faces the user's ear or is positioned at least partially within the user's ear canal. Li ¶[0028].
Regarding the first signal being generated from speech of the wearer captured by the one or more in-canal microphones, Li teaches capturing the user's “own voice” with the internal microphone. The internal microphone receives acoustic components associated with the user's vocal utterance, including bone/flesh/air-conducted components. Li ¶¶[0027]-[0029].
Regarding the in-canal microphone being included in a first portion of a device worn by the wearer, with the first portion positioned at least partially in the ear canal, Li teaches an earpiece/earbud worn by the user and an internal microphone positioned at least partially within the ear canal. Li ¶¶[0023], [0028].
Regarding the first portion further including one or more sound-output devices configured to output sounds in the ear canal, Li teaches an earbud designed to be positioned in the user's ear canal and including a small loudspeaker. Li ¶[0023].
Li teaches an external microphone capturing the same wearer's speech but does not expressly disclose the claimed plurality of microphones in the second portion.
Alves teaches a wearable device having multiple external microphones 204 and 404. Alves ¶¶[0068]-[0073]. Each captures the wearer's input speech together with surrounding noise. Alves further teaches that signal 272 is representative of the input speech captured by both external microphones 204 and 404. Alves ¶[0072].
Accordingly, Alves teaches receiving the claimed multiple second signals from multiple microphones, the signals being generated from speech of the wearer.
It would have been obvious to modify Li's internal/external microphone headset to employ the multiple external microphones taught by Alves because both references address improved extraction of the wearer's speech in the presence of environmental noise. Li teaches processing headset microphone signals for improved own-voice extraction, while Alves teaches using multiple external microphone paths to improve capture and processing of wearer speech. Li ¶¶[0026]-[0029]; Alves ¶¶[0068]-[0073].
The modification represents the predictable use of additional known microphone inputs to improve the quality and reliability of the captured wearer speech.
Regarding processing the first signal to generate a first processed data set, Li processes the internal-microphone signal by filtering and otherwise processing the signal to obtain information concerning the wearer's own voice and to perform noise reduction. Li ¶¶[0029]-[0034].
Regarding processing the multiple second signals to generate a second processed data set, Alves processes the respective external-microphone signals using signal-processing operations including transform, echo cancellation and noise reduction and produces processed speech information representative of speech captured by both external microphones. Alves ¶¶[0069]-[0073].
Regarding providing the first processed data set and the second processed data set to one or more machine-learning or artificial-intelligence systems, Bhowmik teaches an ear-wearable local artificial-intelligence assistant receiving voice information, using speech-recognition circuitry to determine the content of a spoken query, and using the recognized query to determine a responsive action. Bhowmik ¶¶[0032]-[0035], [0058]-[0063].
It would have been obvious to have incorporated the teachings of Bhowmik (speech/AI processing) into both processed representations of the same utterance from Li/Alves in order to improve intelligibility of the user's speech, and use both available representations in the downstream speech-processing system as such would predictably improve the reliability of the speech information presented to the assistant.
Regarding receiving one or more responses from the one or more machine-learning or artificial-intelligence systems, Bhowmik teaches determining a response/responsive action to the recognized query and, where appropriate, receiving responsive information from a network resource. Bhowmik ¶¶[0055]-[0059], [0062]-[0063].
Regarding generating, based on the one or more responses, a third signal, Bhowmik expressly teaches that after determining the query content and responsive action, a speech-generation circuit generates an audio response. Bhowmik ¶[0063].
The audio-response signal generated by the speech-generation circuitry corresponds to the claimed third signal because the signal is generated based upon the assistant response and is thereafter used to produce audible output.
Regarding causing the one or more sound-output devices to output sounds in the ear canal based on the third signal, Bhowmik teaches playing the generated audio response through the ear-wearable device, Bhowmik ¶[0063], while Li teaches the ear-canal earbud containing the loudspeaker, Li ¶[0023].
Thus, the combination teaches:
in-canal wearer speech + externally captured wearer speech → separately processed speech data → AI processing → AI response → generated third/audio signal → ear-canal loudspeaker output.
It would have been obvious to have incorporated the teachings of Bhowmik into the processed speech signals of Li/Alves as the voice input to Bhowmik's AI assistant because Bhowmik expressly teaches an ear-wearable voice interface for obtaining assistant responses. The improved speech information produced by Li/Alves would predictably improve recognition of the wearer's request, after which Bhowmik's speech-generation circuitry generates the audible response.
With respect to claim 8,
Claim 8 recites substantially corresponding operations in the form of instructions stored on one or more non-transitory computer-readable media.
The limitations map as follows:
• first in-canal speech signal — Li ¶¶[0027]-[0029];
• in-canal microphone/first portion — Li ¶[0028];
• ear-canal sound-output device — Li ¶[0023];
• multiple external speech signals — Alves ¶¶[0068]-[0073];
• first processed data set — Li ¶¶[0029]-[0034];
• second processed data set — Alves ¶¶[0069]-[0073];
• providing both processed data sets to ML/AI processing — Li/Alves in view of Bhowmik ¶¶[0032]-[0035], [0058]-[0063];
• AI response — Bhowmik ¶¶[0055]-[0059], [0062]-[0063];
• third signal generated based on the response — Bhowmik ¶[0063];
• ear-canal output based on the third signal — Bhowmik ¶[0063]. Li ¶[0023]; and
• processor-executable stored instructions — Bhowmik ¶¶[0060]-[0066].
The motivations stated for claim 1 apply.
With respect to claims 2 and 9,
Regarding receiving multiple fourth signals from the multiple microphones, the multiple fourth signals generated from external sounds, Alves teaches external microphones 204 and 404 capturing surrounding/environmental noise in addition to the wearer's speech. Alves ¶¶[0039], [0068]-[0069].
Regarding generating, based on the multiple fourth signals, multiple noise-cancellation signals, Alves teaches feed-forward ANC processing of signals from the external microphones to generate noise/anti-noise information corresponding to the detected external noise. Alves ¶¶[0038]-[0042], [0068]-[0073].
Regarding causing the sound-output devices to output second sounds based on the multiple noise-cancellation signals, Alves teaches generating anti-noise for cancellation at the in-ear acoustic location. Alves ¶¶[0038]-[0042].
With respect to claims 7 and 14
Regarding the first device including one or more processors and wireless communication circuitry, Bhowmik teaches an ear-wearable device having processor, memory/storage and wireless communication circuitry. Bhowmik ¶¶[0041]-[0047], [0060].
Regarding the ML/AI systems including a first artificial-intelligence agent, Bhowmik teaches a local AI assistant capable of executing at the ear-wearable device itself. Bhowmik ¶¶[0032]-[0035], [0044]-[0047].
Regarding detecting that the first device is not coupled to a second device via the wireless communication circuitry, Bhowmik teaches monitoring wireless connectivity and determining when connection to a phone/network has been lost. Bhowmik ¶¶[0044]-[0047].
Regarding receiving a fourth signal from the one or more in-canal microphones, Li teaches receiving wearer speech at the internal/in-canal microphone. Li ¶¶[0028]-[0029].
Regarding processing the fourth signal to generate third data, Li teaches processing the internal-microphone signal to obtain processed own-voice information, while Bhowmik teaches processing voice input with speech-recognition circuitry to determine query content.
Regarding providing the third data to the first artificial-intelligence agent, Bhowmik provides the recognized query to the locally executed assistant.
Regarding receiving one or more second responses from the first artificial-intelligence agent, Bhowmik teaches determining a local responsive action.
Regarding generating, based on the one or more second responses, a fifth signal, Bhowmik's speech-generation circuitry generates the audio response based upon the responsive action. Bhowmik ¶[0063].
Regarding causing the one or more sound-output devices to output second sounds based on the fifth signal, Bhowmik plays the generated audio response on the ear-wearable device, while Li supplies the ear-canal loudspeaker.
With respect to claim 15,
The limitations are mapped as follows:
first portion positioned at least partially in the ear canal — Li ¶¶[0023], [0028];
one or more in-canal microphones — Li ¶¶[0028]-[0029];
in-canal microphones configured to capture sounds/vibrations in the ear canal — Li ¶¶[0027]-[0029];
one or more sound-output devices configured to output sound in the ear canal — Li ¶[0023];
second portion including multiple microphones configured to capture external sounds — Alves ¶¶[0068]-[0073];
one or more processors — Alves ¶[0038] and Bhowmik ¶[0060];
one or more memories storing instructions — Bhowmik ¶¶[0060]-[0066];
receiving first signal generated from wearer speech at in-canal microphone — Li ¶¶[0027]-[0029];
receiving multiple second signals generated from wearer speech — Alves ¶¶[0068]-[0073];
processing first signal to generate first processed data set — Li ¶¶[0029]-[0034];
processing multiple second signals to generate second processed data set — Alves ¶¶[0069]-[0073];
providing first and second processed data sets to ML/AI systems — Li/Alves in view of Bhowmik ¶¶[0032]-[0035], [0058]-[0063];
receiving AI response — Bhowmik ¶¶[0055]-[0059], [0062]-[0063];
generating third signal based on response — Bhowmik ¶[0063]; and
causing ear-canal output based upon third signal — Bhowmik ¶[0063] in view of Li ¶[0023].
The motivation stated for claim 1 applies equally to the corresponding structural implementation.
Claims 3 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application No. 20190043518 (Li) in view of U.S. Patent No. 2020/0312341 (Alves) and further in view of U.S. Patent Application No. 20200219506 (Bhowmik) and further in view of U.S. Patent No. 8,649,526 (Vernon).
Regarding detecting second sounds output by the one or more sound-output devices emanating from the ear canal, Vernon teaches speaker 120 generating sound into the user's ear and microphone 110 detecting audio generated by speaker 120 that has leaked or reflected from within the user's ear canal. Vernon, col. 7, ll. 1–10.
Vernon's earbud embodiment likewise teaches a speaker supplying audio into the ear canal and a microphone detecting audio generated by the earbud that has leaked out of the user's ear canal. Vernon, col. 7, l. 57–col. 8, l. 9.
Regarding generating, based on the second sounds, a noise-cancellation signal, Vernon uses the detected leaked speaker audio to determine a leakage transfer function used in generating the noise-cancelling signal. Vernon, col. 7, ll. 9–21; col. 8, ll. 10–17.
Vernon's claim 1 similarly requires generating the leakage transfer function based upon detected leaked portions of the generated sound and processing ambient noise using that function to generate the noise-cancellation signal. Vernon, col. 10, ll. 43–65.
Regarding causing the sound-output device to output third sounds based on the noise-cancellation signal, Vernon supplies the resulting noise-cancelling signal to speaker 120, which generates cancellation sound. Vernon, col. 7, ll. 17–31.
It would have been obvious to incorporated Vernon's leakage-responsive cancellation technique into the Li/Alves/Bhowik system and method because Vernon teaches that earbud leakage varies with fit and position and that detecting the actual leaked speaker sound permits the cancellation system to account for the acoustic leakage experienced by the particular wearer.
Such modification would predictably improve noise cancellation without changing the basic operation of the ear-worn system.
Claims 4 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application No. 20190043518 (Li) in view of U.S. Patent No. 2020/0312341 (Alves) and further in view of U.S. Patent Application No. 20200219506 (Bhowmik) and further in view of U.S. Patent No. 10,678,502 (Sheynblat).
Regarding providing the first processed data set to at least one speech-to-text model configured for in-canal speech, Sheynblat teaches an in-ear device acquiring wearer-generated audio and applying speech-recognition processing to that audio.
Sheynblat further teaches providing in-ear audio for speech/voice-recognition processing and obtaining recognized speech information, including textual information.
Additionally, Sheynblat teaches updating coefficients or weights of a speech-recognition technique based upon audio obtained from the in-ear device and subsequently applying the updated speech-recognition technique to additional audio.
Accordingly, Sheynblat teaches:
in-ear speech → speech-recognition/STT processing → recognition technique adapted using in-ear audio → subsequent recognition of in-ear audio.
It would have been obvious to have incorporated the teaching’s of Sheynblat into the system and method of Li/Alves/Bhowik in order to provide a speech-recognition technique to Li/Alves/Bhowik's processed in-canal speech.
Claims 5 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application No. 20190043518 (Li) in view of U.S. Patent No. 2020/0312341 (Alves) and U.S. Patent Application No. 20200219506 (Bhowmik) and U.S. Patent No. 10,678,502 (Sheynblat) and further in view of U.S. Patent No. 10,685,663 ( Karkkainen) and U.S. Patent Application No. 2019/0325861 (Singh).
Claims 5 and 12 additionally require, “modifying at least one foundation model using in-canal speech data to generate the at least one speech-to-text model configured for in-canal speech.”
The combination of Li, Alves, Bhowmik, and Sheynblat, as applied above, teaches the system and method of the respective parent claims, including obtaining and processing in-canal speech and speech recognition thereof. However, the combination of Li, Alves, Bhowmik, and Sheynblat does not expressly disclose modifying at least one foundation model using in-canal speech data to generate the at least one speech-to-text model configured for in-canal speech.
Karkkainen teaches training/modifying a machine-learning model using speech data obtained from an in-ear microphone. Specifically, Karkkainen teaches that, during training, inputs to the generative network include an in-ear microphone signal, that training may be performed using both the in-ear microphone and outside-the-ear microphone signals, and that the output of the training process is a trained neural network G (Karkkainen, col. 11, ll. 15–49; Fig. 11). Karkkainen further teaches accessing speech signals using multiple microphones including in-ear microphones, training generative model G, and outputting the trained generative model G (Karkkainen, col. 12, ll. 14–25; Fig. 13). Karkkainen additionally teaches receiving in-ear microphone speech 1115 from a speech database, determining a sample based on the in-ear microphone speech, and processing the in-ear microphone speech through a conditioned generator network during training (Karkkainen, col. 12, ll. 26–49; Fig. 14). Thus, Karkkainen teaches modifying/training a model using in-ear, i.e., in-canal, speech data.
Karkkainen, however, does not expressly disclose that the model trained using the in-ear speech data is a speech-to-text model configured for in-canal speech.
Singh teaches automatic speech recognition using domain adaptation to modify an acoustic/speech-recognition model for speech associated with a target acoustic domain. Singh explains that speech-recognition models trained using speech from one domain can perform poorly when used with speech from a different acoustic domain and teaches adapting such models using speech data from the target domain. Singh teaches training neural-network acoustic models using source-domain speech together with target-domain speech and teaches learning features from raw speech that account for acoustic variability/domain shift. Singh further teaches supervised domain adaptation in which a model trained on a relatively large source-domain speech data set is fine-tuned using speech from the target domain, thereby adapting the speech-recognition model to the target speech domain. See, e.g., Singh, ¶¶ [0005]–[0011], [0020]–[0030], and Fig. 3. Singh expressly describes automatic speech recognition and adaptation of acoustic models to target-domain speech.
It would have been obvious to one of ordinary skill in the art at the time of the invention to modify the speech-recognition system of Li/Alves/Bhowmik/Sheynblat in accordance with Karkkainen and Singh by using in-canal speech data as target-domain training data to modify/adapt the speech-recognition model for recognition of in-canal speech. Karkkainen teaches that in-ear microphone speech may be used in training a neural-network model, while Singh teaches adapting a speech-recognition model using speech associated with a target acoustic domain to account for acoustic variability between different speech domains. One of ordinary skill in the art would therefore have been motivated to apply Singh's domain-adaptation technique using Karkkainen's in-ear speech data as the target-domain speech data in order to reduce the acoustic mismatch between the speech data used to train the speech-recognition model and speech subsequently captured by an in-canal microphone, thereby improving speech-recognition performance for in-canal speech, with a reasonable expectation of success. Singh itself identifies acoustic mismatch/domain shift as a problem affecting speech-recognition performance and teaches domain adaptation as a solution to that problem.
Therefore, Li/Alves/Bhowmik/Sheynblat, and further in view of Karkkainen and Singh, teaches or suggests “modifying at least one foundation model using in-canal speech data to generate the at least one speech-to-text model configured for in-canal speech,” as recited in claims 5 and 12.
Claims 6 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application No. 20190043518 (Li) in view of U.S. Patent No. 2020/0312341 (Alves) and U.S. Patent Application No. 20200219506 (Bhowmik) and U.S. Patent No. 10,678,502 (Sheynblat) and further in view of U.S. Patent No. 8571860 ( Strope) .
The combination of Li/Alves/Bhowik/Sheynblat does not expressly disclose the additional limitations of claims 6 and 13 relating to [the claimed multiple speech-recognition/model limitation].
Strope teaches a speech-recognition system employing a plurality of speech-recognition systems (SRSs), wherein the SRSs may differ in their underlying acoustic models. Strope teaches that different acoustic models may target different conditions, including gender, accent, age range, background/foreground noise conditions, or a particular transmission channel, and may differ in architecture, size, and training procedure (Strope, col. 5, ll. 16–47; Fig. 2). Strope further teaches recognizing an incoming utterance using two or more SRSs in parallel, including executing four speech-recognition tasks in parallel in an attempt to recognize the same utterance (Strope, col. 5, ll. 48–57; Fig. 2).
With respect to the result/confidence aspect:
Strope further teaches that each SRS outputs a recognition result and a confidence measure, and that a recognition-result monitor tracks the recognition results and corresponding confidence values and may calculate combined confidence scores (Strope, col. 5, l. 65–col. 6, l. 19).
It would have been obvious to one of ordinary skill in the art at the time of the invention to modify or incorporate the teachings of Strope into the the speech-recognition system of Li/Alves/Bhowmik/Sheynblat in accordance with Strope to employ a plurality of speech-recognition systems/models for processing the speech signal. Strope teaches that different speech-recognition systems may employ different acoustic models directed to different speech conditions and may process the same utterance in parallel. One of ordinary skill in the art would have been motivated to employ Strope's multiple-model speech-recognition technique in the combined system to improve the reliability and accuracy of speech recognition by permitting the same speech input to be evaluated using speech-recognition models having different characteristics, with a reasonable expectation of success.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to William Deane whose telephone number is 571 - 272- 7484. The examiner can normally be reached on Monday - FRIDAY from 9:00 A.M. to 5:00 P.M. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Ahmad Matar, can be reached on 571-272-7488.
The official fax phone number for the organization where this application or proceeding is assigned is 571 -273-8300. However, unofficial faxes can be direct to the examiner's computer at 571 273 -7484.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://pair-direct.uspto.gov/
Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
30Sep2026
/WILLIAM J DEANE JR/ Primary Examiner, Art Unit 2693
