DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
1. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
2. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
3. Claims 1-7 & 9-20 are rejected under 35 U.S.C. 103 as being unpatentable over Davies et al. (US 20180315438 A1 hereinafter, Davies ‘438) in view of Haslam et al. (US 12,142,264 B2 hereinafter, Haslam ‘264).
Regarding claim 20; Davies ‘438 discloses a communication device configured to integrate deepfake voice simulations during a meeting (i.e. Presented herein are an apparatus, system and method to improve intelligibility of speech in an on-line voice communication environment, e.g., conference session, Voice over Internet Protocol (VoIP) call, etc. among a plurality of conference or communication participants. Paragraph 0011)
comprising:
a processor (Fig. 2, Processor 202);
and a tangible, non-transitory memory (Fig. 2, Memory 204) configured to communicate with the processor, the tangible, non-transitory memory having instructions stored thereon that, in response to execution by the processor, cause the processor to perform operations (i.e. The functions of the processor 202 may be implemented by logic encoded in one or more tangible (non-transitory) computer-readable storage media wherein the memory 204 stores data used for the operations and stores software or processor executable instructions that are executed to carry out the operations Paragraph 0015)
comprising:
identifying, using at least a detection mechanism (Fig. 4, Intelligibility Analysis Module 402), when speech intelligibility of a participant of a plurality of participants in the meeting is impacted by one or more environmental conditions (i.e. The intelligibility analysis module 402 is configured to analyze the intelligibility of received audio/voice data including detecting whether there is voice data loss/degradation in the received voice data, determine whether prediction probability of correctly compensating for a voice data loss/degradation is greater than a predetermined probability threshold, determine whether intelligibility of voice data is greater than an intelligibility threshold, etc. Paragraph 0052);
upon determining, using at least a simulated human auditory model, a speech intelligibility of the participant is below an intelligibility threshold (i.e. The second user device or the server detects voice data loss or poor intelligibility (degradation) in the first voice data and determines whether prediction probability of correctly compensating for the voice data loss or degradation is greater than a predetermined probability threshold. Paragraph 0010);
creating a local transcription of the speech by the participant in the meeting (i.e. The server 102 may also recognize the voice data, e.g., converting the voice data to phonemes or text to produce a speech transcription. Paragraph 0032);
and generating, using at least a digital voice model, a voice simulation of the speech using at least the local transcription and a deepfake synthesis of the participant to replicate vocal characteristics of the speech by the participant during the meeting (i.e. The compensated voice data are produced in a manner to mimic a speaker's voice by attempting to match original timbre (at minimal male/female voice) or re-using a predicted phoneme already spoken in a previous part of the conversation. The phoneme(s) may also be obtained from a stored recording of the user's voice. When audio loss/degradation between the server and the user device is detected, the user device determines whether it has received the phonemic transcription of the voice data from the server. If the transcription has been received, the user device does not have to predict the missing phoneme and can directly proceed to produce compensated voice data. Paragraphs 0067 & 0072)
Examiner reasonably believes that Davies ‘438 discloses one or more environmental conditions as expressed above. However, Examiner cites Haslam ‘264 to cure any deficiencies of Davies ‘438.
Haslam ‘264 discloses one or more environmental conditions (i.e. The present disclosure generally relates to noise reduction in open office environments, and more specifically to offering a mechanism by which a caller can lower their own speech volume while continuing to provide a high-quality conversation experience to a customer. Column 1, lines 18-22)
Davies ‘438 and Haslam ‘264 are combinable because they are from same field of endeavor of speech systems (Haslam ‘264 at “Technical Field”).
Before the effective filing date, it would have been obvious to a person of ordinary skill in the art to modify the speech system as taught by Davies ‘438 by adding one or more environmental conditions as taught by Haslam ‘264. The motivation for doing so would have been advantageous to reduce ambient noise in an office environment to eliminate obtrusive and distracting speech. Therefore, it would have been obvious to combine Davies ‘438 with Haslam ‘264 to obtain the invention as specified.
Regarding claim 1; Claim 1 contains substantially the same subject matter as claim 20. Therefore, claim 1 is rejected on the same grounds as claim 20.
Regarding claim 2; Davies ‘438 discloses wherein the one or more environmental conditions comprise at least one of ambient noise, background noise, jitter, echo, and bandwidth below a predetermined threshold that impact the audio intelligibility (i.e. if the prediction probability is not greater than the predetermined probability threshold, second compensating for the voice data loss or degradation by inserting noise to the first voice data to produce second compensated voice data; and outputting the first compensated voice data or the second compensated voice data. Paragraph 0078)
Regarding claim 3; Davies ‘438 discloses wherein the one or more environmental conditions comprises overlapping local and remote audio, and wherein the digital voice model integrates at least a deepfake synthesis to replace only a portion of the speech in generating the voice simulation depending at least partially on the evaluated noise types of the speech; or wherein the digital voice model integrates at least a deepfake synthesis to replace only a portion of the speech in generating the voice simulation depending at least partially on a severity of detected noise interference and whether the speech intelligibility and/or the audio intelligibility is below a concentration threshold (i.e. If the compensated voice data is determined not to be intelligible/clear, the server or client device may replace the unintelligible portion of the voice data with substitute voice data, which may be one of a comparable counterpart of historical voice data corresponding to the portion of the voice data, synthesized voice data based on the historical voice data, idealized voice data, or a combination thereof. Paragraphs 0011 & 0038-0039).
Regarding claim 4; Haslam ‘264 discloses wherein the digital voice model is or comprises a deep learning model (i.e. The recorded voice data will be stored as training data 330 and shared with a machine learning and/or artificial intelligence algorithm to train the synthesizer in a third stage 332. Column 9, lines 53-55)
Regarding claim 5; Davies ‘438 discloses eliminating, using at least the digital voice model, artifacts of the speech introduced by the one or more environmental conditions (i.e. If the compensated voice data is determined not to be intelligible/clear, the server or client device may replace the unintelligible portion of the voice data with substitute voice data, which may be one of a comparable counterpart of historical voice data corresponding to the portion of the voice data, synthesized voice data based on the historical voice data, idealized voice data, or a combination thereof. Paragraph 0011)
Regarding claim 6; Haslam ‘264 discloses fine tuning, using at least the digital voice model and stored formant characteristics of the participant, the voice simulation by evaluating whether an audio output of the voice simulation matches a tone and/or an inflection of the participant to synthesize a facsimile of the speech by the participant that mirrors a natural voice of the participant (i.e. Column 10, lines 5-10 discloses wherein a speech synthesizer 360 can then generate the synthesized speech in a fifth stage 380. In some embodiments, a prosodic analyzer 370 can also be implemented to generate a tone that matches the content and expression of the original sound signals. In some cases, the speech synthesizer 360 can specifically use portions of the previously recorded utterances from the training data. Column 6, lines 48-55 discloses wherein these words as well as information about tone in the speech signal 152 and other speech patterns typical for the first agent 110 which may have been previously collected by the system may be provided to a speech synthesizer 170. The speech synthesizer 170 receives this data and converts it back to artificially generated speech 180 that is clear, loud, and free of ambient noise.)
Regarding claim 7; Davies ‘438 discloses delivering the voice simulation to the plurality of the participants during the meeting (i.e. The audio blending/substitution module 408 is configured to receive the input audio/voice data, intelligibility analysis from the intelligibility analysis module 402, and the synthesized voice data from voice synthesis module 406, to produce compensated voice data by inserting noise or synthesized voice data into the input audio/voice data, and output the compensated voice data. Paragraph 0055).
Regarding claim 9; Davies ‘438 discloses converting, using at least a meeting server, the local transcription to a voice packet using phoneme generation logic and/or speech synthesis logic (i.e. If there is no match, at 518 the server synthesizes voice data based on the historical voice data for the voice data loss/degradation and produces second compensated voice data. If the predicted phoneme(s) or word(s) matches a phoneme or a word in the historical voice data, at 520 the server restores the voice data loss/degradation with the phoneme or word in the historical voice data and produces third compensated voice data. At 522, the server outputs one of the first, second, or third compensated voice data produced at 514, 518, or 520 to one or more receiving/destination user devices. Paragraph 0058)
Regarding claim 10; Davies ‘438 discloses wherein the identifying when speech intelligibility and/or the audio intelligibility of the participant comprises: upon determining, using at least a predictive model and severity of detected noise interference, whether the speech intelligibility and/or the audio intelligibility is below a concentration threshold, automatically switching to a conversational prosthetic mode that performs the local transcription in real-time to generate a textual representation of the speech (Fig. 4, Steps 402-404 i.e. The Intelligibility analysis module automatically decides whether to apply compensation. If the compensated voice data is determined not to be intelligible/clear, the server or client device may replace the unintelligible portion of the voice data with substitute voice data, which may be one of a comparable counterpart of historical voice data corresponding to the portion of the voice data, synthesized voice data based on the historical voice data, idealized voice data, or a combination thereof. Paragraphs 0011 & 0038-0039.).
Regarding claim 11; Davies ‘438 discloses replacing, using at least the digital voice model, audio sections of the speech comprising a lower quality with synthesized speech sections (i.e. The server or client device may replace the unintelligible portion of the voice data with substitute voice data, which may be one of a comparable counterpart of historical voice data corresponding to the portion of the voice data, synthesized voice data based on the historical voice data, idealized voice data, or a combination thereof. Paragraphs 0011 & 0062)
Regarding claim 12; Davies ‘438 discloses upon identifying one or more environmental conditions of a meeting participant of a plurality of meeting participants is causing speech of the meeting participant to be below an intelligibility threshold and/or an audio intelligibility threshold, implementing a voice-to-text model to create a local transcription of at least a portion of the speech (i.e. The server 102 may also recognize the voice data, e.g., converting the voice data to phonemes or text to produce a speech transcription. The server 102 may store the speech models and speech transcriptions as a part of historical voice data in the memory 204. Paragraph 0032)
and generating, using at least a digital voice model, a voice simulation of at least the portion of the speech of the meeting participant to replicate the local transcription with vocal characteristics of the portion of the speech by the participant during the meeting (i.e. The compensated voice data are produced in a manner to mimic a speaker's voice by attempting to match original timbre (at minimal male/female voice) or re-using a predicted phoneme already spoken in a previous part of the conversation. The phoneme(s) may also be obtained from a stored recording of the user's voice. When audio loss/degradation between the server and the user device is detected, the user device determines whether it has received the phonemic transcription of the voice data from the server. If the transcription has been received, the user device does not have to predict the missing phoneme and can directly proceed to produce compensated voice data. Paragraphs 0067 & 0072)
Examiner reasonably believes that Davies ‘438 discloses one or more environmental conditions as expressed above. However, Examiner cites Haslam ‘264 to cure any deficiencies of Davies ‘438.
Haslam ‘264 discloses one or more environmental conditions (i.e. The present disclosure generally relates to noise reduction in open office environments, and more specifically to offering a mechanism by which a caller can lower their own speech volume while continuing to provide a high-quality conversation experience to a customer. Column 1, lines 18-22)
Davies ‘438 and Haslam ‘264 are combinable because they are from same field of endeavor of speech systems (Haslam ‘264 at “Technical Field”).
Before the effective filing date, it would have been obvious to a person of ordinary skill in the art to modify the speech system as taught by Davies ‘438 by adding one or more environmental conditions as taught by Haslam ‘264. The motivation for doing so would have been advantageous to reduce ambient noise in an office environment to eliminate obtrusive and distracting speech. Therefore, it would have been obvious to combine Davies ‘438 with Haslam ‘264 to obtain the invention as specified.
Regarding claim 13; Davies ‘438 discloses continuously adjusting the voice simulation in real-time to react to one or more detected audio issues of the speech (i.e. A portion of the compensated voice data is replaced with substitute voice data to improve its intelligibility in real-time, e.g., a latency being less than a hundred milliseconds, or tens of milliseconds. Paragraph 0039)
Regarding claim 14; Davies ‘438 discloses wherein the voice-to-text model is implemented by: determining, using at least an acoustic model, a relationship between an audio signal and phonetic units of the speech (i.e. User 1 starts an audio communication session by speaking to a microphone (310) of the user device 106-1, which collects the user's speech as voice data. The processor 302 of the user device 106-1 then uses communication interface 306 to forward the voice data to a receiving/destination user device, for example, user device 106-2. The server 102, although it does not support a voice data loss/degradation compensation scheme, may recognize the voice data, e.g., converting the voice data to phonemes or text to produce a phonemic transcription and sends the speech transcription to a receiving/destination user device, such as user device 106-2. Paragraph 0041);
determining, using at least a language model, statistical patterns and a probability of a sequence of words of the speech (i.e. The user device 106-2 may also compare the voice pattern in the received voice data with prior stored voice patterns of users to identify the speaker of the voice data. In one embodiment, the user device 106-2 may train a learning process/machine of the second speech model establishing logic 324 to recognize voice patterns of the users and use the learning process/machine to identify the speaker of the received voice data based on the voice pattern. Paragraph 00044)
and converting, using at least a decoder, the local transcription of the speech to an original voice output (i.e. The server 102, although it does not support a voice data loss/degradation compensation scheme, may recognize the voice data, e.g., converting the voice data to phonemes or text to produce a phonemic transcription and sends the speech transcription to a receiving/destination user device, such as user device 106-2. Paragraph 0041)
Regarding claim 15; Davies ‘438 discloses wherein the voice-to-text model is configured to: receive an audio stream of the speech (Fig. 5, Step 502 i.e. The method 500 starts at 502 at which the server receives voice data. Paragraph 0056);
extract audio features of the audio stream (Fig. 5, Step 503 i.e. At 503, the server identifies the language spoken in the received voice data. Paragraph 0056)
analyze spectral features of the extracted audio features (Fig. 5, Step 504 i.e. At 504, the server determines whether there is voice data loss/degradation in the received voice data. Paragraph 0056)
assess a signal to noise ratio of the analyzed spectral features (Fig. 5, Step 514 i.e. At 514 the server compensates for the voice data loss/degradation by inserting noise to the received voice data to produce first compensated voice data. Paragraph 0058)
and classify, using a machine learning model of the voice-to-text model, attributes of the audio stream to determine determining that the speech is below the intelligibility threshold (i.e. If the prediction probability is not greater than the predetermined probability threshold, at 514 the server compensates for the voice data loss/degradation by inserting noise to the received voice data to produce first compensated voice data. If the prediction probability is greater than the predetermined probability threshold. Paragraph 0058)
Regarding claim 16; Davies ‘438 discloses accessing, using at least the digital voice model, a database of pre-stored formant characteristics of the participant to support generation of the voice simulation (i.e. For example, the user device 106-2 may maintain a database storing idealized voice data (phonemes and/or words). The user device 106-2 may compare the compensated voice data with the idealized voice data to rate the compensated voice data. Paragraph 0050)
and evaluating, using at least the pre-stored formant characteristics of the participant and the voice simulation, whether an audio output of the voice simulation matches a tone and/or an inflection of the participant to synthesize a facsimile of at least the portion of the speech of the participant (i.e. To use the historical voice data in order to compensate for the voice data loss/degradation, the server 102 determines whether a predicted phoneme or word for the voice data loss/degradation in the received voice data matches a phoneme or a word in the historical voice data. Paragraph 0035)
Regarding claim 17; Davies ‘438 discloses wherein the digital voice model comprises a deep neural network model trained to generate a high-quality audio facsimile of at least the portion of the speech using at least the local transcription and pre-stored formant characteristics of the participant (i.e. A server or client device may determine whether the compensated voice data is intelligible/clear. If the compensated voice data is determined not to be intelligible/clear, the server or client device may replace the unintelligible portion of the voice data with substitute voice data, which may be one of a comparable counterpart of historical voice data corresponding to the portion of the voice data, synthesized voice data based on the historical voice data, idealized voice data, or a combination thereof. Paragraphs 0011 & 0062).
Regarding claim 18; Davies ‘438 discloses wherein the generating the voice simulation using a locally stored voice model of the participant integrated back into the meeting to replacing any distorted or overlapping audio of the speech (i.e. The server or client device may replace the unintelligible portion of the voice data with substitute voice data, which may be one of a comparable counterpart of historical voice data corresponding to the portion of the voice data, synthesized voice data based on the historical voice data, idealized voice data, or a combination thereof. Paragraphs 0011 & 0062).
Regarding claim 19; Davies ‘438 discloses preventing voice of the speech below the intelligibility threshold from flowing over a cloud-based voice over internet protocol (VoIP) calling system (i.e. system and method to improve intelligibility of speech in an on-line voice communication environment, e.g., conference session, Voice over Internet Protocol (VoIP) call, etc. among a plurality of conference or communication participants. A server or client device may detect voice data loss or degradation upon receiving voice data of audio and determine a proper method to compensate for the voice data loss or degradation in order to output compensated voice data to be presented to a receiving user. Paragraph 0011)
and notifying, using at least a meeting server, the other participants that the participant is experiencing one or more environmental conditions below the intelligibility threshold resulting in the speech being converted to text by the voice-to-text model of the meeting server (i.e. Before outputting the compensated voice data, a server or client device may determine whether the compensated voice data is intelligible/clear. If the compensated voice data is determined not to be intelligible/clear, the server or client device may replace the unintelligible portion of the voice data with substitute voice data, which may be one of a comparable counterpart of historical voice data corresponding to the portion of the voice data, synthesized voice data based on the historical voice data, idealized voice data, or a combination thereof. Paragraph 0011)
Allowable Subject Matter
Claim 8 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Examiners Statement of Reasons for Allowance
The cited reference (Davies ‘438) teaches wherein a method comprises: obtaining, at an apparatus, first voice data from a first user device associated with a first speaker participant in a communication session; detecting voice data loss or degradation in the first voice data; determining whether prediction probability of correctly compensating for the voice data loss or degradation is greater than a predetermined probability threshold; if the prediction probability is greater than the predetermined probability threshold, first compensating for the voice data loss or degradation using historical voice data received by the apparatus prior to receiving of the first voice data, the first compensating producing first compensated voice data; if the prediction probability is not greater than the predetermined probability threshold, second compensating for the voice data loss or degradation by inserting noise to the first voice data to produce second compensated voice data; and outputting the first compensated voice data or the second compensated voice data.
The cited reference (Haslam ‘264) teaches a method and system of reducing noise associated with telephony-based activities occurring in shared workspaces is provided. An end-user may lower their own voice to a whisper or other less audible or intelligible utterances and submit such low-quality audio signals to an automated speech recognition system via a microphone. The words identified by the automated speech recognition system are provided to a speech synthesizer, and a synthesized audio signal is created artificially that carries the content of the original human-produced utterances. The synthesized audio signal is significantly more audible and intelligible than the original audio signal. The method allows customer support agents to speak at barely audible levels yet be heard clearly by their customers.
The cited references fail to disclose pausing, using at least a meeting server, a video data feed of the speech; and shifting, using at least the meeting server, the video data feed to a lower bandwidth codec configured to maintain voice intelligibility. As a result, and for these reasons, Examiner indicates Claims 1-7 & 9-20 as allowable subject matter.
Relevant Prior Art References Not Relied Upon
1. Sargsyan et al. (US 11,100,941 B2) - Example speech enhancement and noise suppression systems and methods are described. In one implementation, a method receives an audio file comprising a combination of voice data and noise data, and divides the audio file into multiple frames. The method performs a discrete Fourier transform on each frame of a first subset of the multiple frames to provide a plurality of frequency-domain outputs, which are input to a neural network. A ratio mask is obtained as an output from the neural network and clean voice coefficients are computed using the ratio mask. The method outputs an audio file having enhanced speech based on the computed clean voice coefficients.
2. Popovic et al. (US 7,212,628 B2) - A method of suppressing an echo signal in a communication path such as acoustic coupling between a speaker and a microphone in a telephone device handset includes the steps of monitoring a transmitted signal supplied to the communication path to determine an attribute thereof; generating a leaky mask based on the determined attribute and an attribute of a received echo signal; and partially cancelling the received echo signal using the leaky mask..
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARCUS T. RILEY, ESQ. whose telephone number is (571)270-1581. The examiner can normally be reached 9-5 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached at 571-272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MARCUS T. RILEY, ESQ.
Primary Examiner
Art Unit 2654
/MARCUS T RILEY/Primary Examiner, Art Unit 2654
