DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-20 are pending in this application.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 7-8, 11-12, 14-15, and 19-20 are rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over O’Neill et al., (US Pat. 9,432,768) in view of Rameau et al., (US Pub. 2022/0208194).
Regarding claim 1, O’Neill discloses a method comprising acts of:
recording speech signals from a user, using a first sensor and a second sensor of a wearable [silent speech] device (Figs. 1 and 7, Col. 6, lines 1-22, obtaining audio data by a microphone array of the wearable computer);
providing for a [silent speech] machine learning model for use with the wearable silent speech device (Figs. 1 and 7, Col. 6, lines 1-42, training and providing the localization model for use by the wearable computer; Col. 7, lines 36-55, the localization model may be developed by the machine learning);
determining whether the [silent speech] machine learning model is to be adjusted (Col. 18, lines 29-52, determining and performing the computationally intensive task when the recently collected data is uploaded); and
in response to determining the [silent speech] machine learning model is to be adjusted, adjusting the silent speech machine learning model based on at least the speech signals recorded using the first sensor and the second sensor (Figs. 1 and 7, Col. 18, lines 29-52, re-training of the localization model may occur when the recently collected data is uploaded).
O’Neill does not explicitly teach the bracketed limitation however Rameau does explicitly teach including the bracketed limitation:
a [silent speech] machine learning model (Rameau, [0019][0023]-[0026] recognizing silent speech based on sEMG signal by applying a predicative machine learning model).
Therefore, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to incorporate a wearable device and a method for training/re-training a machine learning model as taught by O’Neill with a method of recognizing of silent speech by detecting sEMG signals as taught by Rameau to provide better communication with patient on voice rest (Rameau, [Abstract]).
Regarding claim 2, O’Neil in view of Rameau discloses the method of claim 1, and O’Neil further discloses:
determining a subset of the recorded speech signals, wherein adjusting the silent speech machine learning model comprises (Col. 17, lines 26-67, detecting audio and determining speech and noise, such as street noise, footsteps, or any sound other than user’s speech and isolating user’s speech, which is emitted from a user’s mouth, from noise):
providing the subset of the recorded speech signals to the silent speech machine learning model (Fig. 6, Col. 17, lines 26-67, generating signal data at a microphone array of a wearable computer and motion data form motion sensors of the wearable computer);
conditioning the silent speech machine learning model based on the subset of the recorded speech signals (Fig. 6, Col. 17, lines 26-67, determining a direction of the user’s mouth using a localization model); and
processing, using the conditioned silent speech machine learning model, the recorded speech signals to generate a representation of one or more words spoken by the user (Fig. 6, Col. 17, lines 26-67, defining a beampattern having a beampattern direction in substantial alignment with the determined direction and obtaining a beamformed audio signal).
Regarding claim 3, O’Neil in view of Rameau discloses the method of claim 1, and O’Neil further discloses:
storing the recorded speech signals in non-volatile storage, the non-volatile storage storing historic speech signals recorded by the wearable silent speech device; and wherein adjusting the silent speech model comprises training the silent speech machine learning model based on the speech signals recorded using the first sensor and the second sensor and the historic speech signals (Col. 8, lines 10-19, “After the training process, the trained localization model 122 may be downloaded (e.g., pushed or pulled) over the network 128 to the wearable computer 104 where it may be stored in the data store 120 and used to perform localization for purposes of beamforming”).
Regarding claim 7, O’Neil in view of Rameau discloses the method of claim 1, and O’Neil further discloses:
determining whether the recorded speech signals are suitable for use in adjusting the silent speech machine learning model, and determining the silent speech machine learning model is to be adjusted in response to determining the recorded speech signals are suitable for use in adjusting the silent speech machine learning model (Col. 5, line 63- Col. 6, line 21, uploading periodically or by user command suitable data including audio data obtained by the microphone array and training or re-training the model and/or validating localization determinations using the remote computing resources).
Regarding claim 8, O’Neil in view of Rameau discloses the method of claim 7, and O’Neil further discloses:
wherein determining whether the recorded speech signals are suitable comprises determining, based on the speech signals, whether the user is speaking out loud and determining the recorded speech signals are suitable in response to determining the user is speaking out loud (Col. 5, line 63- Col. 6, line 21, Col. 11, lines 32-54, issuing voice commands which is emitted in the form of sound waves from the user’s mouth).
Regarding claim 11, O’Neil in view of Rameau discloses the method of claim 1, and O’Neil further discloses:
wherein determining whether the silent speech machine learning model is to be adjusted comprises: determining a performance metric of the silent speech machine learning model (Col. 13, lines 48-67, determining a plurality of confidence levels by the localization model); and
determining the silent speech machine learning model is to be adjusted in response to determining the performance metric is below a threshold level (Col. 14, line 56-Col. 15, line 30, adjusting the beam-width based on the confidence level output by the localization module in response to determining a direction of the user's mouth with a low confidence level).
Regarding claim 12, O’Neil in view of Rameau discloses the method of claim 1, and O’Neil further discloses:
wherein determining whether the silent speech machine learning model is to be adjusted comprises determining, based on a user input, whether the silent speech machine learning model is to be adjusted (Col. 13, lines 48-67, Col. 14, line 56-Col. 15, line 30, adjusting the beam-width based on the confidence level output by the localization module).
Regarding claim 14, O’Neil in view of Rameau discloses the method of claim 1, and O’Neil further discloses:
wherein determining whether the silent speech machine learning model is to be adjusted comprises determining a time since a last silent speech machine learning model adjustment, and in response to determining the time is above a threshold time, determining the silent speech machine learning model is to be adjusted (Col. 8, lines 10-19, re-training of the localization model may occur periodically, such as daily, weekly when data is collected and uploaded over the network to the remote computing resources).
Regarding claim 15, O’Neil in view of Rameau discloses the method of claim 1, and O’Neil further discloses:
analyzing the recorded speech signals; and selecting a subset of the recorded speech signals, wherein the adjusting is performed using the subset of the recorded speech signals (Col. 17, lines 26-67, detecting audio and determining speech and noise, such as street noise, footsteps, or any sound other than user’s speech and isolating user’s speech, which is emitted from a user’s mouth, from noise).
Regarding claim 19, O’Neill discloses a system for recognizing silent speech of a user, the system comprising:
a wearable [silent speech] device (Fig. 1, Col. 6, lines 1-22, a wearable computer);
at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor- executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method, the method comprising:
obtaining speech signals recorded from the user, using a first sensor and a second sensor of the wearable [silent speech] device (Figs. 1 and 7, Col. 6, lines 1-22, obtaining audio data by a microphone array of the wearable computer);
providing for a [silent speech] machine learning model for use with the wearable silent speech device (Figs. 1 and 7, Col. 6, lines 1-42, training and providing the localization model for use by the wearable computer; Col. 7, lines 36-55, the localization model may be developed by the machine learning);
determining whether the [silent speech] machine learning model is to be adjusted (Col. 18, lines 29-52, determining and performing the computationally intensive task when the recently collected data is uploaded); and
in response to determining the [silent speech] machine learning model is to be adjusted, adjusting the silent speech machine learning model based on at least the speech signals recorded using the first sensor and the second sensor (Figs. 1 and 7, Col. 18, lines 29-52, re-training of the localization model may occur when the recently collected data is uploaded).
O’Neill does not explicitly teach the bracketed limitation however Rameau does explicitly teach including the bracketed limitation:
a [silent speech] machine learning model (Rameau, [0019][0023]-[0026] recognizing silent speech based on sEMG signal by applying a predicative machine learning model).
Therefore, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to incorporate a wearable device and a method for training/re-training a machine learning model as taught by O’Neill with a method of recognizing of silent speech by detecting sEMG signals as taught by Rameau to provide better communication with patient on voice rest (Rameau, [Abstract]).
Regarding claim 20, Claim 20 is the corresponding medium claims to system claim19. Therefore, claim 20 is rejected using the same rationale as applied to claim 19 above.
Claims 4-5 and 16-18 are rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over O’Neill et al., (US Pat. 9,432,768) in view of Rameau et al., (US Pub. 2022/0208194) and further in view of Kapur et al., (US Pub. 2019/0074012).
Regarding claim 4, O’Neill in view of Rameau discloses the method of claim 3.
O’Neill in view of Rameau does not explicitly teach however Kapur does explicitly teach:
wherein training the silent speech machine learning model comprises performing a series of gradient steps based on a comparison of an output of the silent speech machine learning model to ground truth data (Kapur, [0125] [0176] “the first sentence of this definition specifies—by comparison to what would occur in ordinary speech”).
Therefore, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to incorporate a wearable device and a method for training/re-training a machine learning model as taught by O’Neill in view of Rameau with the system for silent speech interface using a method of comparing to ordinary speech as taught by Kapur to provide more accurate internal articulation recognition results (Kapur, [0052]).
Regarding claim 5, O’Neill in view of Rameau and further in view of Kapur discloses the method of claim 4.
O’Neill in view of Rameau does not explicitly teach however Kapur does explicitly teach: wherein the non-volatile storage stores the ground truth data associated with the historic speech signals (Kapur, [0105][0106] detecting in the signal and may be temporarily stored in memory in data buffer).
The previous motivation statement as in claim 4 is still applied.
Regarding claim 16, O’Neill in view of Rameau discloses the method of claim 3.
O’Neill in view of Rameau does not explicitly teach however Kapur does explicitly teach:
wherein adjusting the silent speech machine learning model comprises performing a gradient step of the silent speech machine learning model based on a comparison of an output of the silent speech machine learning model to ground truth data (Kapur, [0125] optimizing using a first order gradient descent and parameters are updated during training).
The previous motivation statement as in claim 4 is still applied.
Regarding claim 17, O’Neill in view of Rameau and further in view of Kapur discloses the method of claim 16.
O’Neill in view of Rameau does not explicitly teach however Kapur does explicitly teach: determining the ground truth data based on the recorded speech signals (Kapur, [0176]-[0186] determining speech based on the received ordinary speech).
The previous motivation statement as in claim 4 is still applied.
Regarding claim 18, O’Neill in view of Rameau and further in view of Kapur discloses the method of claim 17.
O’Neill in view of Rameau does not explicitly teach however Kapur does explicitly teach: wherein the ground truth data is determined using a second machine learning model, different from the silent speech machine learning model (Kapur, [0114][0115][0126][0147] signals were recorded for randomly chosen words from a specific vocabulary set. This data was used to train the recognition model).
The previous motivation statement as in claim 4 is still applied.
Claim 6 is rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over O’Neill et al., (US Pat. 9,432,768) in view of Rameau et al., (US Pub. 2022/0208194) and further in view of Manabe et al., (US Pub. 2005/0102134).
Regarding claim 6, O’Neill in view of Rameau discloses the method of claim 1.
O’Neill in view of Rameau does not explicitly teach however Manabe does explicitly teach: wherein the first sensor is an EMG sensor and the second sensor is a microphone (Manabe, Fig. 7, [0092] a device is provided with surface electrodes for EMG detection sensors 10 and a microphone 20).
Therefore, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to incorporate a wearable device and a method for training/re-training a machine learning model as taught by O’Neill in view of Rameau with the system for voice recognition by detecting EMG as taught by Manabe to improve recognition accuracy in voice recognition in a noisy environment and to prevent unnecessary communication (Manabe, [0022]).
Claim 9 is rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over O’Neill et al., (US Pat. 9,432,768) in view of Rameau et al., (US Pub. 2022/0208194) and further in view of Lloyd et al., (US Pub. 2011/0307253).
Regarding claim 9, O’Neill in view of Rameau discloses the method of claim 7.
O’Neill in view of Rameau does not explicitly teach however Lloyd does explicitly teach: wherein determining whether the recorded speech signals are suitable comprises determining, based on the speech signals, a level of background noise and determining the recorded speech signals are suitable in response to determining the level of background noise is below a threshold level (Lloyd, [0073] training a speech model using an audio signal with background audio below the defined threshold).
Therefore, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to incorporate a wearable device and a method for training/re-training a machine learning model as taught by O’Neill in view of Rameau with the method of determining that the background audio in the audio signal is below the defined threshold as taught by Lloyd to perform the noise compensation system which is improved by using an adapted user speech model that has been trained (Lloyd, [0032]).
Claims 10 and 13 are rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over O’Neill et al., (US Pat. 9,432,768) in view of Rameau et al., (US Pub. 2022/0208194) and further in view of Roach et al., (US Pub. 2021/0306751).
Regarding claim 10, O’Neill in view of Rameau discloses the method of claim 1, and O’Neill further discloses:
wherein determining whether the silent speech machine learning model is to be adjusted comprises determining whether the silent speech machine learning model requires user onboarding (Col. 5, line 63- Col. 6, line 21, uploading and training a model by user command; Col. 11 lines 32-54, issuing voice commands which is emitted in the form of sound waves from the user’s mouth).
O’Neill in view of Rameau does not explicitly teach however Roach does explicitly teach:
in response to determining the silent speech machine learning model requires user onboarding, prompting the user to speak one or more words or phrases, wherein the speech signals are recorded after the prompting (Roach, [0004][0005][0144] prompting a user to speak, e.g. ‘Begin Speaking Now’).
Therefore, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to incorporate a wearable device and a method for training/re-training a machine learning model as taught by O’Neill in view of Rameau with the system and method of using a device with voice communication as taught by Roach to improve the accuracy and power efficiency of a speech recognition system by filtering out audio signals that are not likely to be intended speech input (Roach, [0114]).
Regarding claim 13, O’Neill in view of Rameau discloses the method of claim 1.
O’Neill in view of Rameau does not explicitly teach however Roach does explicitly teach:
determining whether the wearable silent speech device is being powered on, and in response to determining the wearable silent speech device is being powered on, prompting the user to speak one or more words or phrases, wherein the speech signals are recorded after the prompting, and it is determined that the silent speech machine learning model is to be adjusted in response to determining the wearable silent speech device is being powered on (Roach, [0144] “ voice onset detection can be used to trigger subsequent events…the detection of an onset can serve as a trigger for a speech processing engine to activate”).
The previous motivation statement as in claim 10 is still applied.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Please see attached form PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SEONG-AH A. SHIN whose telephone number is (571)272-5933. The examiner can normally be reached 9 AM-3PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at 571-272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Seong-ah A. Shin
Primary Examiner
Art Unit 2659
/SEONG-AH A SHIN/Primary Examiner, Art Unit 2659