DETAILED ACTION
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/22/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4-6, 10-13 and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Maxwell et al. ( US 20220139417 A1), hereinafter referenced as Maxwell, in view of Engelke et al. (US 20180270350 A1), hereinafter referenced as Engelke.
Regarding Claim 1, Maxwell teaches a method comprising:
after storing the audio message, generating, by an automated generation system that includes one or more first machine learning models, video that includes sign language content corresponding to the audio message ( Maxwell: Para.[0038], calls or portion of the calls may be saved in AI database/server 250. Para.[0047], Fig. 5, AI servers 112 ( first machine learning model) generates video by using an avatar simulator which is configured to generate the video data of an avatar that performs simulated sign language of the voice of the hearing-capable user, when displayed by the video communication device 102. Para.[0052],[0053], Fig. 8 illustrates artificial intelligence server 112 which is configured to perform the speech-to-text translation using voice recognition software to perform the real-time transcription ( or translation into sign language via simulated avatar) to generate the return far-end information to the video communication device 102);
storing the video ( Maxwell: Para.[0038], Fig. 2, video files can be stored in AI server 250);
and after storing the video, training one or more second machine learning models of an automated recognition system configured to translate sign language into language data using the video and language data from the audio message ( Maxwell: Para.[0036]-[0039], Fig. 3, the training station 301 may be configured to update databases of the AI engine 110 which include translation database 250 ( second machine learning algorithm). The training station 301 receive the video data that includes the sign language content as well as the translated output as synthesized audio to review the accuracy of the automatic translation for training. This review process may occur after the call has ended with the trainer reviewing stored video and a corresponding transcript of the translation from the call. The review process may also occur during the call such that the training station receives real-time video and the corresponding transcript of the translation during a live call for the trainer to review. The translated output files may be saved as a text file with a textual transcript and/or an audio file with the synthesized audio translation for the trainer to review).
Maxwell while teaching the method of claim 1, fails to explicitly teach the claimed, in response to a communication session not being established between a first communication device and a second communication device, obtaining an audio message from the first communication device; storing the audio message;
However, Engelke does teach the claimed, in response to a communication session not being established between a first communication device and a second communication device, obtaining an audio message from the first communication device ( Engelke: Para.[0099],[0101], Fig. 1 illustrates a communication system 10 including an AU's ( assisted user (hearing impaired), second communication device) communication device 12, an HU's ( hearing user, first communication device) telephone or other type communication device 14, linked via any network connection capable of facilitating a voice call between the AU and the HU. Para.[0022], AU device can obtain recorded voice messages from HU device ( without establishing a session or live call) to provide captioning of voice messages);
storing the audio message ( Engelke: Para.[0022],[0140], Fig. 6, HU’s voice messages are stored in a rolling buffer);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Engelke’s teaching of automated voice-to-text captioning for hearing impaired users , into the system and method of performing automated translation services during a real-time communication session between a video communication device associated with a hearing-impaired user and a far-end communication device associated with a hearing-capable user through an artificial intelligence (AI) translation engine, taught by Maxwell, because, this would improve the remote communication using a telephone for a hearing impaired person.( Engelke, Para.[0005]-[0010]).
Claim 12 is system claim comprising: one or more computer readable mediums including instructions; one or more computing systems coupled to the one or more computer readable mediums and configured to execute the instructions to cause or direct the system to perform operations, the operations comprising (Maxwell: Para.[0061],[0062], Fig. 11, The processor 1120 may coordinate the communication between the various devices as well as execute instructions stored in computer-readable media of the memory device 1130): performing the steps in method claim 1 above and as such, claim 12 is similar in scope and content to claim 1 and therefore, claim 12 is rejected under similar rationale as presented against claim 1 above.
Regarding Claim 2, Maxwell in view of Engelke teach the method of claim 1. Engelke further teaches, further comprising before training the automated recognition system, obtaining consent from at least one of a first user associated with the first communication device and a second user associated with the second communication device ( Engelke: Para.[0361], real voice recordings of AU ( assisted user, hearing impaired) - HU( hearing user) calls may only be used for training purposes after authorization is sought and received).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Engelke’s teaching of automated voice-to-text captioning for hearing impaired users , into the system and method of performing automated translation services during a real-time communication session between a video communication device associated with a hearing-impaired user and a far-end communication device associated with a hearing-capable user through an artificial intelligence (AI) translation engine, taught by Maxwell, because, this would improve the remote communication using a telephone for a hearing impaired person.( Engelke, Para.[0005]-[0010]).
Claim 13 is system claim performing the steps in method claim 2 above and as such, claim 13, is similar in scope and content to claim 2 and therefore, claim 13 is rejected under similar rationale as presented against claim 2 above.
Regarding Claim 4, Maxwell in view of Engelke teach the method of claim 1. Engelke further teaches, further comprising before obtaining the audio message, directing second audio to the first communication device ( Engelke: Para.[0197],[0198], when there is a substantial delay in obtaining audio message and transcription, HU may utter "Are you there?" or "Did you hear me?", which are "line check words" (LCWs). Automated voice engine can present the line check words immediately via the AU's device during a call regardless of which words have been transcribed and presented to an AU. The AU, seeing line check words or a phrase can verbally respond ( second audio towards HU) that the captioning service is lagging but catching up so that the parties can avoid or at least minimize confusion).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Engelke’s teaching of automated voice-to-text captioning for hearing impaired users , into the system and method of performing automated translation services during a real-time communication session between a video communication device associated with a hearing-impaired user and a far-end communication device associated with a hearing-capable user through an artificial intelligence (AI) translation engine, taught by Maxwell, because, this would improve the remote communication using a telephone for a hearing impaired person.( Engelke, Para.[0005]-[0010]).
Claim 15 is system claim performing the steps in method claim 4 above and as such, claim 15, is similar in scope and content to claim 4 and therefore, claim 15 is rejected under similar rationale as presented against claim 4 above.
Regarding Claim 5, Maxwell in view of Engelke teach the method of claim 4. Engelke further teaches, wherein second audio is generated via an automated system to interact with a user of the first communication device ( Engelke: Para.[0197],[0198], when there is a substantial delay in obtaining audio message and transcription, HU may utter "Are you there?" or "Did you hear me?", which are "line check words" (LCWs). a system processor may automatically respond to any line check words by broadcasting a voice message ( second message) to the HU indicating that transcription is lagging and will catch up shortly.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Engelke’s teaching of automated voice-to-text captioning for hearing impaired users , into the system and method of performing automated translation services during a real-time communication session between a video communication device associated with a hearing-impaired user and a far-end communication device associated with a hearing-capable user through an artificial intelligence (AI) translation engine, taught by Maxwell, because, this would improve the remote communication using a telephone for a hearing impaired person.( Engelke, Para.[0005]-[0010]).
Claim 16 is system claim performing the steps in method claim 5 above and as such, claim 16, is similar in scope and content to claim 5 and therefore, claim 16 is rejected under similar rationale as presented against claim 5 above.
Regarding Claim 6, Maxwell in view of Engelke teach the method of claim 1. Engelke further teaches, wherein the video is generated in response to a request from a user associated with the second communication device to view the video ( Engelke: Para.[0551], AU ( assisted user, second device) user may, at the beginning of the communication issue a query/request to HU ( hearing user, firs device) to “turn on camera” to generate video).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Engelke’s teaching of automated voice-to-text captioning for hearing impaired users , into the system and method of performing automated translation services during a real-time communication session between a video communication device associated with a hearing-impaired user and a far-end communication device associated with a hearing-capable user through an artificial intelligence (AI) translation engine, taught by Maxwell, because, this would improve the remote communication using a telephone for a hearing impaired person.( Engelke, Para.[0005]-[0010]).
Claim 17 is system claim performing the steps in method claim 6 above and as such, claim 17, is similar in scope and content to claim 6 and therefore, claim 17 is rejected under similar rationale as presented against claim 6 above.
Regarding Claim 10, Maxwell in view of Engelke teach the method of claim 1. Maxwell further teaches, at least one non-transitory computer-readable media configured to store one or more instructions that, in response to being executed by a system, cause or direct the system to perform the method of claim 1 ( Maxwell: Para.[0062], Fig.11, The processor 1120 may coordinate the communication between the various devices as well as execute instructions stored in computer-readable media of the memory device 1130. The memory device 1130 may include volatile and non-volatile memory storage for the video communication device 102).
Regarding Claim 11, Maxwell teaches a method comprising:
after storing the audio message, generating, by an automated generation system that includes one or more first machine learning models, video that includes sign language content corresponding to the audio message ( Maxwell: Para.[0038], calls or portion of the calls may be saved in AI database/server 250. Para.[0047], Fig. 5, AI servers 112 ( first machine learning model) generates video by using an avatar simulator which is configured to generate the video data of an avatar that performs simulated sign language of the voice of the hearing-capable user, when displayed by the video communication device 102. Para.[0052],[0053], Fig. 8 illustrates artificial intelligence server 112 which is configured to perform the speech-to-text translation using voice recognition software to perform the real-time transcription ( or translation into sign language via simulated avatar) to generate the return far-end information to the video communication device 102);
storing the video ( Maxwell: Para.[0038], Fig. 2, video files can be stored in AI server 250);
Maxwell while teaching the method of claim 1, fails to explicitly teach the claimed, in response to a communication session not being established between a first communication device and a second communication device, obtaining an audio message from the first communication device; storing the audio message;
However, Engelke does teach the claimed, in response to a communication session not being established between a first communication device and a second communication device, obtaining an audio message from the first communication device ( Engelke: Para.[0099],[0101], Fig. 1 illustrates a communication system 10 including an AU's ( assisted user, hearing impaired) communication device 12, an HU's ( hearing user) telephone or other type communication device 14, linked via any network connection capable of facilitating a voice call between the AU and the HU. Para.[0022], AU device can obtain recorded voice messages from HU device ( without establishing a session or live call) to provide captioning of voice messages);
storing the audio message ( Engelke: Para.[0022],[0140], Fig. 6, HU’s voice messages are stored in a rolling buffer);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Engelke’s teaching of automated voice-to-text captioning for hearing impaired users , into the system and method of performing automated translation services during a real-time communication session between a video communication device associated with a hearing-impaired user and a far-end communication device associated with a hearing-capable user through an artificial intelligence (AI) translation engine, taught by Maxwell, because, this would improve the remote communication using a telephone for a hearing impaired person.( Engelke, Para.[0005]-[0010]).
Claims 3, 7, 14 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Maxwell et al. ( US 20220139417 A1), hereinafter referenced as Maxwell, in view of Engelke et al. (US 20180270350 A1), hereinafter referenced as Engelke, further in view of Liu et al. (US 20240404429 A1), hereinafter referenced as Liu.
Regarding Claim 3, Maxwell in view of Engelke teach the method of claim 1. Maxwell in view of Engelke fail to explicitly teach the claimed, further comprising after storing the video, training one or more of the first machine learning models of the automated generation system using the video and the language data.
However, Liu does teach the claimed, further comprising after storing the video, training one or more of the first machine learning models of the automated generation system using the video and the language data ( Liu: Para.[0039],[0047], Figs. 2, 3, an AI-based sign language avatar interpreter determination process 300 uses an AI-based sign language avatar interpreter program 200 which can train the machine learning models and natural language processing algorithms using language data and training data. Training data can comprise video streams and audio streams that are uploaded to the program 200. For each video stream, the program 200 can analyze the video stream and convert the sign language gestures to text. The program 200 can analyze the text and can generate the corresponding type of sign language).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Liu’s teaching of artificial intelligence virtual sign language Avatar interpreter, into the system and method, taught by Maxwell in view of Engelke, because, this would improve the virtual communication experience of people over the internet where by detecting if there are members in a virtual audience that require sign language and the type of sign language they understand, translate the sign language used by a presenter to the type of sign language understood by one or more members, and generate the appropriate sign language movements on a digital avatar.( Liu, Para.[0010],[0011]).
Claim 14 is system claim performing the steps in method claim 3 above and as such, claim 14, is similar in scope and content to claim 3 and therefore, claim 14 is rejected under similar rationale as presented against claim 3 above.
Regarding Claim 7, Maxwell in view of Engelke teach the method of claim 1. Maxwell in view of Engelke fail to explicitly teach the claimed, further comprising transcribing the audio message using automated speech recognition to generate text corresponding to the sign language content, wherein the text is used to train the one or more second machine learning models.
However, Liu does teach the claimed, further comprising transcribing the audio message using automated speech recognition to generate text corresponding to the sign language content, wherein the text is used to train the one or more second machine learning models ( Liu: Para.[0039],[0047], Figs. 2, 3, an AI-based sign language avatar interpreter determination process 300 uses an AI-based sign language avatar interpreter program 200 which can train the machine learning models and natural language processing algorithms using language data and training data. Language data may comprise sign language samples and voice samples that are uploaded to the program 200. Training data can comprise video streams and audio streams that are uploaded to the program 200. For each audio stream, the program 200 can analyze the audio stream and can convert the speech into text and analyze the text and can generate the corresponding type of sign language).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Liu’s teaching of artificial intelligence virtual sign language Avatar interpreter, into the system and method, taught by Maxwell in view of Engelke, because, this would improve the virtual communication experience of people over the internet where by detecting if there are members in a virtual audience that require sign language and the type of sign language they understand, translate the sign language used by a presenter to the type of sign language understood by one or more members, and generate the appropriate sign language movements on a digital avatar.( Liu, Para.[0010],[0011]).
Claim 18 is system claim performing the steps in method claim 7 above and as such, claim 18, is similar in scope and content to claim 7 and therefore, claim 18 is rejected under similar rationale as presented against claim 7 above.
Claims 8, 9, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Maxwell et al. ( US 20220139417 A1), hereinafter referenced as Maxwell, in view of Engelke et al. (US 20180270350 A1), hereinafter referenced as Engelke, further in view of Medalion et al. (US 11978456 B2 ), hereinafter referenced as Medalion.
Regarding Claim 8, Maxwell in view of Engelke teach the method of claim 1. Maxwell in view of Engelke fail to explicitly teach the claimed, further comprising: storing a plurality of audio messages and corresponding videos that include the audio message and the video, each of the audio messages generated from a different communication session not being established; determining which of the plurality of audio messages and corresponding videos is usable for training; and training the one or more second machine learning models of the automated recognition system using the audio messages and corresponding videos determined to be usable for training.
However, Medalion does teach the claimed, further comprising: storing a plurality of audio messages and corresponding videos that include the audio message and the video, each of the audio messages generated from a different communication session not being established ( Medalion: Column 6, lines 15-42, 52-64, Fig. 1, a computer, such as a server ( or group of servers) 101, receives and records conversations conducted via a network 102, among pairs or groups of participants using respective computers 103-1, 103-2, 103-3, 103-4. Server 101 capture and collect recordings of web conferences ( recordings of different conferences, not live conferences).The data stream includes both an audio stream, containing speech uttered by the participants, conference metadata, video stream, containing visual recordings of the participants and/or visual information generated regarding the participants. Server 101 receives and stores a corpus of recorded conversations in memory );
determining which of the plurality of audio messages and corresponding videos is usable for training ( Medalion: Column 11, lines 26-36, the visual information, customer relationship management (CRM) data and historical facial information of known speakers ( usable information), may be used to generate a training set. The training set may be trained with an video samples of individuals speaking or not speaking) ;
and training the one or more second machine learning models of the automated recognition system using the audio messages and corresponding videos determined to be usable for training ( Medalion: Column 11, lines 49-51, the training set may be used to train the machine learning algorithm).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Medalion’s teaching of using visual information in a video stream of a recording streaming teleconference among a plurality of participants to diarize speech, into the system and method, taught by Maxwell in view of Engelke, because, this would improve the diarization techniques during a recorded streaming teleconference by utilizing machine learning techniques tied to transcription data of the conversation, video data and training sets with common utterances tagged with speaker information. ( Medalion, Column 5, lines 53-67, column 6, lines 1-13).
Claim 19 is system claim performing the steps in method claim 8 above and as such, claim 19, is similar in scope and content to claim 8 and therefore, claim 19 is rejected under similar rationale as presented against claim 8 above.
Regarding Claim 9, Maxwell in view of Engelke, further in view of Medalion teach the method of claim 8. Engelke further teaches, wherein a first audio message [and corresponding first video] is determined to be usable for training based on obtaining consent from one or more users associated with the first audio message ( Engelke: Para.[0361], real voice recordings of AU ( assisted user, hearing impaired) - HU( hearing user) calls may only be used for training purposes after authorization is sought and received. Para.[0645], images or video may be provided to an HU to give a visual representation of the AU so that one can get a sense from non-verbal queues of effectiveness of AU communications. Prior to presenting AU images or video to others, device may seek AU authorization ).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Engelke’s teaching of automated voice-to-text captioning for hearing impaired users , into the system and method, taught by Maxwell in view of Madelion, because, this would improve the remote communication using a telephone for a hearing impaired person.( Engelke, Para.[0005]-[0010]).
Claim 20 is system claim performing the steps in method claim 9 above and as such, claim 20, is similar in scope and content to claim 9 and therefore, claim 20 is rejected under similar rationale as presented against claim 9 above.
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant's disclosure.
Peralta et al. (US 12437673 B2) teaches a system and method for bidirectional automatic sign language translation and visualization. of the system includes at least two communication-capable devices for receiving and processing information from the system's input and/or output and showing the output of the system. At least two individuals are communicating with each other, both using different modes of communication such as sign language and spoken language. The individuals are able to utilize two separate computing devices, with the system installed or disposed, to translate the information the individuals are signing.
Daniali et al. (US 20230215295 A1) teaches systems, methods, and computer-readable media for real-time manipulation and animation of 3D rigged virtual models to generate sign language translation. Source video and audio data associated with content is provided to a neural network to determine choreographic actions that may be used to modify and animate the articulation control points of a 3D model within a 3D space. The animated 3D virtual model may be presented in relation to the source content to provide sign language translation of the source content.
Kelly et al. (US 12518653 B2) teaches a real time sign language recognition method that allows Deaf and Hard of Hearing individuals to sign into any apparatus with a camera to extract target information (such as a translation in a target language). Embodiments of the disclosure relate to artificial intelligence (AI), machine learning (ML) and more particularly machine translation and processing of signed languages as an assistive technology for Deaf and Hard of Hearing (D/HH) Individuals.
Kumar et al. (US 20240320449 A1 ) teaches an example methodology implementing the disclosed techniques which includes, by a computing device, receiving a first video stream captured by a camera, analyzing images within the first video stream to recognize a first regional sign language, and determining a caption that portrays a meaning conveyed by the recognized first regional sign language. The method also includes, by the computing device, translating the first caption to a neutral language to generate a neutral language caption, translating the neutral language caption to generate a regional language caption, the regional language associated with an intended recipient of the first video stream, and generating a second video stream composed of a second regional sign language images representing the regional language caption. The method may also include sending the second video stream to the intended recipient of the first video stream. The first video stream may be received during a sign language conversation session.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NADIRA SULTANA whose telephone number is (571)272-4048. The examiner can normally be reached M-F,7:30 am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras D. Shah can be reached on (571) 270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NADIRA SULTANA/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
07/21/2026
Prosecution Insights