DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
A typical human being would not be able to determine a set of prosodic properties to be utilized in generating synthesized speech audio data. Therefore, the 101 rejection has been vacated in light of claim amendments.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-2, 8-11, and 17-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-19 of U.S. Patent No. 12190859 in view of Liu et al. (USPG 2020/0265829, hereinafter Liu), and further in view of Nasu et al. (USPG 2014/0180668, hereinafter Nasu).
Since Liu and Nasu are analogous in the art because they are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to use the known technique of converting speech of a telephone conversation into text and then translate and covert the text back to speech for the receiving user. One of ordinary skill in the art would have recognized that the results of the combination were predictable since the use of that known technique provides the rationale to arrive at a conclusion of obviousness. See KSR International Co. v. Teleflex Inc., 82 USPQ2d 1385 (U.S. 2007).
Claims 3-4 and 12-13 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-19 of U.S. Patent No. 12190859 in view of Liu, further in view of Nasu, and further in view of Dao et al. (USPG 2009/0012793, hereinafter Dao).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to use the known technique of displaying text to allow the user to select text to convert to speech. One of ordinary skill in the art would have recognized that the results of the combination were predictable since the use of that known technique provides the rationale to arrive at a conclusion of obviousness. See KSR International Co. v. Teleflex Inc., 82 USPQ2d 1385 (U.S. 2007).
Claims 7 and 16 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-19 of U.S. Patent No. 12190859 in view of Liu, further in view of Nasu, further in view of Dao, and further in view of Xu et al. (USPG 2010/0218141, hereinafter Xu).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to use the known technique of rendering the best candidate more prominently that the rest of the candidates. One of ordinary skill in the art would have recognized that the results of the combination were predictable since the use of that known technique provides the rationale to arrive at a conclusion of obviousness. See KSR International Co. v. Teleflex Inc., 82 USPQ2d 1385 (U.S. 2007).
Claims of application Claims of the patent
1. A method implemented by one or more processors, the method comprising:
receiving, via a client device of a user, audio data that captures a spoken utterance of an additional user, the spoken utterance being provided by the additional user as part of a conversation between the user and the additional user;
generating, based on processing the audio data that captures the spoken utterance of the additional user, one or more suggestions, each of the one or more suggestions including a corresponding textual segment that is responsive to the spoken utterance; determining a set of prosodic properties, from among a plurality of disparate sets of prosodic properties, to be utilized in generating synthesized speech audio data that includes synthesized speech corresponding to a given suggestion of the one or more suggestions; generating, using the set of prosodic properties, the synthesized speech audio data that includes synthesized speech corresponding to the given suggestion of the one or more suggestions; and causing the synthesized speech audio data to be rendered as part of the conversation between the user and the additional user.
2. The method of claim 1, wherein the set of prosodic properties is automatically determined based on one or more of: a relationship between the user and the additional user, a location of the client device of the user, features of the given suggestion that is included in the given suggestion, features of the spoken utterance provided by the additional user.
3. The method of claim 1, further comprising: prior to the synthesized speech audio data that includes the synthesized speech corresponding to the given suggestion of the one or more suggestions: causing the one or more suggestions to be visually rendered at the client device via a display of the client device.
4. The method of claim 3, further comprising: receiving, via the client device of the user, a selection of the given suggestion via the display of the client device, wherein generating the synthesized speech audio data that includes the synthesized speech corresponding to the given suggestion is in response to receiving the selection of the given suggestion.
5 The method of claim 3, wherein generating the synthesized speech audio data that includes the synthesized speech corresponding to the given suggestion is in response to determining that no selection of any one of the one or more suggestions is received within a threshold duration of time relative to the one or more suggestions being visually rendered at the client device.
6. The method of claim 5, further comprising: determining a ranking of the one or more suggestions, wherein the given suggestion is a highest ranking suggestion of the one or more suggestions.
7. The method of claim 3, further comprising: prior to causing the one or more suggestions to be visually rendered at the client device: determining a ranking of the one or more suggestions, wherein causing the one or more suggestions to be visually rendered at the client device comprises: causing a highest ranking suggestion, of the one or more suggestions, to be visually rendered more prominently than other suggestions, of the one or more suggestions.
8. The method of claim 1, wherein the additional user is located remotely from the user, and wherein the conversation between the user and the additional user is a telephonic conversation or video conversation.
9. The method of claim 1, wherein the additional user is located in a same physical environment of the user, and wherein the conversation between the user and the additional user is an in-person conversation.
All other claims are similar to the above claims.
1. A method implemented by one or more processors, the method comprising:
detecting, via one or more microphones of a client device of a given user, spoken input of the given user; determining, based on processing the spoken input of the given user, a textual segment for conveying in a conversation in which the given user is a participant; identifying an additional participant in the conversation, the additional participant being in addition to the given user, and the additional participant being physically located in an environment with the given user; determining at least one attribute of a relationship between the given user and the additional participant; determining, based on the at least one attribute of the relationship between the given user and the additional participant, a given set of one or more prosodic properties, wherein the given set of the one or more prosodic properties is a first set of the one or more prosodic properties in response to determining the at least one attribute of the relationship between the given user and the additional participant is a first attribute, and wherein the given set of the one or more prosodic properties is a second set of the one or more prosodic properties in response to determining the at least one attribute of the relationship between the given user and the additional participant is a second attribute; generating synthesized speech audio data that includes synthesized speech that incorporates the textual segment and that is synthesized with the given set of the one or more prosodic properties, wherein generating the synthesized speech audio data comprises synthesizing the synthesized speech with the given set of the one or more prosodic properties responsive to determining the given set of the one or more prosodic properties based on the attribute of the relationship between the given user and the additional participant; and causing the synthesized speech to be rendered via one or more speakers of the client device and/or an additional client device, wherein the rendered synthesized speech is audibly perceptible to the additional participant.
2. The method of claim 1, wherein determining, based on processing the spoken input of the given user, the textual segment, comprises: processing the spoken input using a speech recognition model to generate the textual segment.
3. The method of claim 2, wherein the speech recognition model is an on-device speech recognition model and/or is trained for recognizing speech of speech impaired users.
4. The method of claim 1, further comprising, subsequent to causing the synthesized speech to be rendered: detecting, via one or more of the microphones of the client device, an additional participant spoken input, of the additional participant; processing the additional participant spoken input using a speech recognition model to generate an additional participant textual segment that is a recognition of the additional participant spoken input; determining that an additional textual segment is a candidate response to the additional participant textual segment; and determining to display a graphical element that conveys the additional textual segment responsive to determining that the additional textual segment is the candidate response to the additional participant textual segment.
5. The method of claim 4, wherein identifying the additional participant in the conversation comprises: performing speaker identification using the additional participant spoken input; and identifying the additional participant based on the speaker identification.
6. The method of claim 5, wherein performing the speaker identification comprises: generating, at the client device, a spoken input embedding based on processing the additional participant spoken input using a speaker identification model; and comparing, at the client device, the spoken input embedding to a pre-stored embedding for the additional participant, the pre-stored embedding being previously stored locally at the client device responsive to authorization by the additional participant.
7. The method of claim 4, wherein determining that the additional textual segment is the candidate response to the additional participant textual segment is further based on the at least one attribute of the relationship between the given user and the additional participant.
8. The method of claim 7, wherein determining that the additional textual segment is the candidate response to the additional participant textual segment is further based on the at least one attribute of the relationship between the given user and the additional participant comprises: generating a superset of initial candidate responses based on the additional participant textual segment, the superset including the additional textual segment; and selecting, from the superset of initial candidate responses, the additional textual segment as the candidate response based on the at least one of the attributes of the relationship between the given user and the additional participant.
9. The method of claim 4, further comprising: determining at least one classification of a location of the client device; wherein determining that the additional textual segment is the candidate response to the additional participant textual segment is further based on the at least one classification of the location.
10. The method of claim 4, further comprising: in response to receiving a user selection, from the given user, of the graphical element that conveys the additional textual segment: generating additional synthesized speech audio data that includes additional synthesized speech that incorporates the additional textual segment and that is synthesized with the one or more prosodic properties, wherein generating the additional synthesized speech audio data comprises synthesizing the additional synthesized speech with the one or more prosodic properties; and causing the additional synthesized speech to be rendered via one or more of the speakers of the client device and/or the additional client device, wherein the rendered additional synthesized speech is audibly perceptible to the additional participant.
11. The method of claim 1, further comprising: identifying a further additional participant in the conversation, the further additional participant being in addition to the given user and being in addition to the additional participant; determining the given set of the one or more prosodic properties based on both: (a) the attribute of the relationship between the given user and the additional participant, and (b) one or more additional attributes of an additional relationship between the given user and the further additional participant.
12. The method of claim 1, further comprising: identifying a further additional participant in the conversation, the further additional participant being in addition to the given user and being in addition to the additional participant; determining the given set of the one or more prosodic properties based on the attribute of the relationship between the given user and the additional participant, in lieu of one or more additional attributes of an additional relationship between the given user and the further additional participant, responsive to: determining that the relationship between the given user and the additional participant is more formal than the additional relationship between the given user and the further additional participant.
13. The method of claim 1, further comprising: determining at least one classification of a location of the client device; wherein determining the prosodic properties is further based on the classification of the location of the client device.
14. The method of claim 1, wherein the one or more prosodic properties comprise one or more of intonation, tone, stress, and rhythm.
15. The method of claim 1, wherein identifying the additional participant in the conversation comprises: identifying the additional participant based on one or more images captured by a camera of the client device.
16. The method of claim 15, wherein the camera comprises an adjustable viewport, and wherein the one or more images are captured subsequent to adjustment, of the adjustable viewport, responsive to other user interface input at the client device of the given user.
17. The method of claim 15, wherein identifying the additional participant based on the images captured by the camera of the client device comprises: rendering an image, of the one or more images, on a display of the client device, wherein the image captures multiple humans, including the additional participant; receiving other user interface input at the client device of the given user that designates an area, of the image, that captures the additional participant and excludes all other of the multiple humans; and identifying the additional participant based on processing of the area of the image.
All other claims are similar to the above claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 10-11, 19-20, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al. (USPG 2020/0265829, hereinafter Liu) in view of Nasu et al. (USPG 2014/0180668, hereinafter Nasu).
Regarding claims 1, 10, and 19, Liu discloses a method, system, and non-transitory CRM, implemented by one or more processors, the method comprising:
at least one processor (figure 6); and memory storing instructions that, when executed (figure 6), cause the at least one processor to be operable to:
generating, based on processing the audio data that captures the spoken utterance of the additional user, one or more suggestions, each of the one or more suggestions including a corresponding textual segment that is responsive to the spoken utterance (figure 5a, step 505; receiving a voiced content as a result of a user input; and figure 5b, the voiced content is parsed to obtained a text; also see paragraph 101, “the natural language processor 127 may extract the linguistic features of the voiced content 123 from the audio using Hidden Markov Models, mathematical modeling or other techniques described in this disclosure. Moreover, in some embodiments of step 515, the natural language processor may further use POS tagging to identify and understand the characteristics of each word being analyzed and converted into text by the natural language processor 127”);
determining a set of prosodic properties (paragraphs 86-87, “Vocoder parameters and features can be extracted from the voice samples 153. The vocoder may take the desired features and parameters from the voice samples 153 and perform complex transforms to generate an audio waveform that estimates the desired parameters of speech, including the phase, rhythm, stress, intonation, etc.”), from among a plurality of disparate sets of prosodic properties, to be utilized in generating synthesized speech audio data that includes synthesized speech corresponding to a given suggestion of the one or more suggestions (paragraphs 102-103, “The voice samples 153 may be used as training data and/or examples for conditioning the neural networks to generate the waveforms of the modified version of the voiced content 123 having particular characteristics, parameters and sounds akin to the sounds of the human speech presented in the voice samples 153”; and/or process in figure 5a, particularly steps 523-527);
generating, using the set of prosodic properties, the synthesized speech audio data that includes synthesized speech corresponding to the given suggestion of the one or more suggestions (paragraphs 102-103 and/or process in figure 5a, particularly steps 523-527); and
causing the synthesized speech audio data to be rendered as part of the conversation between the user and the additional user (paragraphs 102-103 and/or process in figure 5a, particularly steps 523-527).
Liu fails to explicitly disclose, however, Nasu teaches receiving, via a client device of a user, audio data that captures a spoken utterance of an additional user, the spoken utterance being provided by the additional user as part of a telephone call between the user and the additional user and via an additional client device of the additional user (referring abstract section, a telephone conversation between two users of two independent devices are received and converted to text; the converted text is then translated and synthesized back to speech for the receiving user).
Since Liu and Nasu are analogous in the art because they are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to use the known technique of converting speech of a telephone conversation into text and then translate and covert the text back to speech for the receiving user. One of ordinary skill in the art would have recognized that the results of the combination were predictable since the use of that known technique provides the rationale to arrive at a conclusion of obviousness. See KSR International Co. v. Teleflex Inc., 82 USPQ2d 1385 (U.S. 2007).
Regarding claims 2, 11, and 20, Liu further discloses wherein the set of prosodic properties is automatically determined based on one or more of: a relationship between the user and the additional user, a location of the client device of the user, features of the given suggestion that is included in the given suggestion, features of the spoken utterance provided by the additional user (paragraph 99, “ the recommendation engine 109 may not only consider past user habits, interests and experiences such as those stored by the user context database 110, but also the voices that may be more recently heard by the user and stored in the historical bank of voices maintained by the user history database 112, the permissibly available voices that the content provider may have the rights to use and/or imitate and the emotional state or sentiment of the user at the present time that the voiced content 123 is scheduled to be delivered”, taking the user motional state or sentiment into account when determining prosodic properties).
Regarding claim 24, Liu further discloses the method of claim 1, wherein generating, using the set of prosodic properties, the synthesized speech audio data that includes the synthesized speech corresponding to the given suggestion of the one or more suggestions comprises: processing, using a speech synthesis model, the set of prosodic properties and the given suggestion to generate the synthesized speech audio data (paragraphs 86-87, synthesizing speech according to assigned prosodic information).
Claims 3-4, 12-13, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Liu in view of Chen, and further in view of Dao et al. (USPG 2009/0012793, hereinafter Dao).
Regarding claims 3-4, 12-13, and 21-22, Liu fails to explicitly disclose, however, Dao teaches prior to the synthesized speech audio data that includes the synthesized speech corresponding to the given suggestion of the one or more suggestions: causing the one or more suggestions to be visually rendered at the client device via a display of the client device (paragraphs 12 and 31; also see figure 1, displaying text for the user to select to convert to speech); and further comprising: receiving, via the client device of the user, a selection of the given suggestion via the display of the client device, wherein generating the synthesized speech audio data that includes the synthesized speech corresponding to the given suggestion is in response to receiving the selection of the given suggestion (paragraphs 12 and 31; also see figure 1, displaying text for the user to select to convert to speech).
Since Liu and Dao are analogous in the art because they are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to use the known technique of displaying text to allow the user to select text to convert to speech. One of ordinary skill in the art would have recognized that the results of the combination were predictable since the use of that known technique provides the rationale to arrive at a conclusion of obviousness. See KSR International Co. v. Teleflex Inc., 82 USPQ2d 1385 (U.S. 2007).
Claims 7 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Liu in view of Chen, further in view of Dao, and further in view of Xu et al. (USPG 2010/0218141, hereinafter Xu).
Regarding claims 7 and 16, the modified Liu fails to explicitly disclose, however, Xu teaches prior to causing the one or more suggestions to be visually rendered at the client device: determining a ranking of the one or more suggestions, wherein causing the one or more suggestions to be visually rendered at the client device comprises: causing a highest ranking suggestion, of the one or more suggestions, to be visually rendered more prominently than other suggestions, of the one or more suggestions (figure 4, “Larry” is visually rendered more prominently that other suggestions; also see paragraphs 32-33, “primary candidate” would have a higher score than alternative candidates).
Since the modified Liu and Xu are analogous in the art because they are from the same field of endeavor, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to use the known technique of rendering the best candidate more prominently that the rest of the candidates. One of ordinary skill in the art would have recognized that the results of the combination were predictable since the use of that known technique provides the rationale to arrive at a conclusion of obviousness. See KSR International Co. v. Teleflex Inc., 82 USPQ2d 1385 (U.S. 2007).
Potential Allowable Subject Matter
Claims 5-6, 14-15, and 23 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and resolving the double patenting issues.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Nagasaka (USPG 2022/0165293) teaches converting speech to text for processing before converting the text to speech that is considered pertinent to the claimed invention.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUYEN X VO whose telephone number is (571)272-7631. The examiner can normally be reached M-F, 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh Mehta can be reached at 571-272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HUYEN X VO/Primary Examiner, Art Unit 2656
