ILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This action is responsive to the RCE filed on 4/14/26.
Claim(s) 45-47, 68-70 & 78-80 is/are presented for examination.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 45-47, 68, 78 & 80 is/are rejected under 35 U.S.C. 103 as being unpatentable over Liao, U.S. Patent/Pub. No. US 2021/0065582 A1 in view of Mahajan, U.S. Patent/Pub. No. US 11044287 B1, and further in view of Reece, US 2021/0264900 A1.
As to claim 45, Liao teaches a method comprising:
receiving audio data onto the remote device, wherein the audio data captures sounds spoken by the remote party during the time interval, wherein the remote device converts the audio data into words of text, and wherein the remote device captures prosodic information describing the sounds spoken by the remote party during the time interval (Liao, page 2, paragraph 19-20 & 23; page 9, paragraph 80; i.e., [0020] This may be achieved by utilizing a speech recognition algorithm that converts spoken words to text in real-time, determining the number of syllables in the words spoken for a given time period and calculating the speaking rate based on the number of syllables, instead of the number of words. Furthermore, phonetic features of the audio signal such as pitch, intensity or energy (e.g., formant) may be taken into account to determine the speaking rate; [0080] receiving a transcript for the audio data, the transcript including a plurality of words spoken during the speech rehearsal session);
receiving onto a local device the words of text (Liao, page 1, paragraph 4; i.e., [0004] receiving a transcript for the audio data, the transcript including a plurality of words spoken during the speech rehearsal session);
receiving onto the local device the prosodic information corresponding to the sounds spoken by the remote party during the time interval that were converted into the words of text (Liao, page 2, paragraph 19-20 & 23; page 5, paragraph 43-44 & 46-47; page 9, paragraph 80; i.e., [0020] This may be achieved by utilizing a speech recognition algorithm that converts spoken words to text in real-time, determining the number of syllables in the words spoken for a given time period. Furthermore, phonetic features of the audio signal such as pitch, intensity or energy (e.g., formant) may be taken into account to determine the speaking rate; [0044] In response to the request, transcribed text corresponding to the audio data may be received, at 525. In one implementation, the information relating to the transcribed text may include metadata such as when the text is received and the duration of the speech results);
storing the words of text and the prosodic information in association with one another (Liao, page 2, paragraph 19-20 & 23; page 5, paragraph 43-44 & 46-47; page 9, paragraph 80; i.e., [0020] This may be achieved by utilizing a speech recognition algorithm that converts spoken words to text in real-time, determining the number of syllables in the words spoken for a given time period and calculating the speaking rate based on the number of syllables, instead of the number of words. Furthermore, phonetic features of the audio signal such as pitch, intensity or energy (e.g., formant) may be taken into account to determine the speaking rate).
But Liao failed to teach the claim limitation wherein receiving onto a remote device digital video data captured during a time interval at a location of a remote party, wherein the digital video data depicts the remote party and the video data includes video timestamps; the prosodic information includes prosodic timestamps; and generating an indicator at a segment of the video data based at least on the stored prosodic information at a corresponding video timestamp based on the prosodic timestamps.
However, Mahajan teaches the limitation wherein receiving onto a remote device digital video data captured during a time interval at a location of a remote party, wherein the digital video data depicts the remote party (Mahajan, col 1, lines 53-60; i.e., facilitates the collaboration of different users through an online platform that connects the computing devices of online meeting participants. Online meeting content that is transmitted during an online meeting may include audio, video and text data that is encoded into data packets that are transmitted between online participant computing).
It would have been obvious to one of ordinary skill in the art before the effective date of the claimed invention to modify Liao to substitute transmission periodically from Mahajan for time period from Liao to fastest transmissions of the data to each of the different client with the best reliability (Mahajan, col 15, lines 50-55).
However, Reece teaches the limitation wherein and the video data includes video timestamps (Reece, page 5, paragraph 62-64; page 9, paragraph 108; i.e., [0064] generate a transcription of the conversation, based on the acoustic and video recordings. This text data may include timestamps to align the transcript with the acoustic and video data); the prosodic information includes prosodic timestamps (Reece, page 5, paragraph 62-64; page 9, paragraph 108; i.e., [0108] Conversation analysis indicators (e.g., 706, 708, 710) may be stored with timestamps, for correlation with the source utterance and acoustic/video data of the conversation. In other words, conversation analysis indicators may be stored in a series, based on a series (i.e., sequence) of utterances and/or concatenated speaker data. The stored conversation analysis indicators may be graphed, visualized, and analyzed in aggregate by conversation analytics system); and generating an indicator at a segment of the video data based at least on the stored prosodic information at a corresponding video timestamp based on the prosodic timestamps (Reece, page 5, paragraph 62-64; page 9, paragraph 108; i.e., [0064] generate a transcription of the conversation, based on the acoustic and video recordings. This text data may include timestamps to align the transcript with the acoustic and video data; [0108] Conversation analysis indicators (e.g., 706, 708, 710) may be stored with timestamps, for correlation with the source utterance and acoustic/video data of the conversation. In other words, conversation analysis indicators may be stored in a series, based on a series (i.e., sequence) of utterances and/or concatenated speaker data. The stored conversation analysis indicators may be graphed, visualized, and analyzed in aggregate by conversation analytics system).
It would have been obvious to one of ordinary skill in the art before the effective date of the claimed invention to modify Liao to substitute conversation score from Reece for rate of speech from Liao to produce data indicative of a goal ( e.g., minimizing a loss function) (Reece, page 2, paragraph 36).
As to claim 46, Liao-Mahajan-Reece teaches the method as recited in claim 45, wherein the prosodic information includes information selected from the group consisting of: lengths of the sounds spoken by the remote party, a time of occurrence of a pitch point of a word spoken by the remote party, and an amplitude of a sound spoken by the remote party (Liao, page 2, paragraph 19-20 & 23; page 9, paragraph 80; i.e., [0020] This may be achieved by utilizing a speech recognition algorithm that converts spoken words to text in real-time, determining the number of syllables in the words spoken for a given time period and calculating the speaking rate based on the number of syllables. Furthermore, phonetic features of the audio signal such as pitch, intensity or energy (e.g., formant) may be taken into account to determine the speaking rate).
As to claim 47, Liao-Mahajan-Reece teaches the method as recited in claim 45, wherein displaying on a graphical user interface of the local device the words of text and the prosodic information associated with the words of text (Liao, page 2, paragraph 19-20 & 23; page 6, paragraph 50; i.e., [0020] This may be achieved by utilizing a speech recognition algorithm that converts spoken words to text in real-time, determining the number of syllables in the words spoken for a given time period and calculating the speaking rate based on the number of syllables, instead of the number of words. Furthermore, phonetic features of the audio signal such as pitch, intensity or energy (e.g., formant) may be taken into account to determine the speaking rate).
As to claim 68, Liao-Mahajan-Reece teaches the method as recited in claim 45, wherein the words of text and the prosodic information are not stored together with the video data that was captured during the time interval (Liao, page 2, paragraph 19-20 & 23; page 3, paragraph 27; i.e., [0020] This may be achieved by utilizing a speech recognition algorithm that converts spoken words to text in real-time, determining the number of syllables in the words spoken for a given time period and calculating the speaking rate based on the number of syllables, instead of the number of words. [0023] In addition, in some implementations, a user device can be configured to transmit data captured locally during use of relevant application(s) to the cloud or the local ML program).
As to claim 78, Liao teaches a method comprising:
wherein the data is captured during a time interval, wherein the data (Liao, page 2, paragraph 23; page 6, paragraph 52; i.e., [0023] In addition, in some implementations, a user device can be configured to transmit data captured locally during use of relevant application(s) to the cloud or the local ML program; [0052]
time stamp in every audio frame of the audio data. This may involve providing the values of pitch, and the first, second and third formants);
determining a value of a physiological parameter of the remote party using the data, wherein the value of the physiological parameter is determined remotely at the location of the remote party, and wherein the value of the physiological parameter is timestamped (Liao, page 2, paragraph 23; page 6, paragraph 52; i.e., [0023] to transmit data captured locally during use of relevant application(s) to the cloud or the local ML program; [0052] recurrent convolutional neural network may be developed that examines every time stamp in every audio frame of the audio data. This may involve providing the values of pitch, and the first, second and third formants);
receiving onto the remote device audio data that captures sounds spoken by the remote party during the time interval, wherein the remote device generates prosodic information from the audio data (Liao, page 1, paragraph 4; page 2, paragraph 19-20 & 23; page 9, paragraph 80; i.e., [0020] Furthermore, phonetic features of the audio signal such as pitch, intensity or energy (e.g., formant) may be taken into account to determine the speaking rate; [0023] to transmit data captured locally during use of relevant application(s) to the cloud or the local ML program; [0080] receiving a transcript for the audio data, the transcript including a plurality of words spoken during the speech rehearsal session);
receiving onto a local device the prosodic information corresponding to the sounds spoken by the remote party during the time interval (Liao, page 1, paragraph 4; i.e., [0004] receiving a transcript for the audio data, the transcript including a plurality of words spoken during the speech rehearsal session);
converting the prosodic information into words of text at the local device, wherein the words of text correspond to the sounds spoken by the remote party during the time interval (Liao, page 2, paragraph 19-20 & 23; page 5, paragraph 43-44 & 46-47; page 9, paragraph 80; i.e., [0020] This may be achieved by utilizing a speech recognition algorithm that converts spoken words to text in real-time, determining the number of syllables in the words spoken for a given time period and calculating the speaking rate based on the number of syllables. Furthermore, phonetic features of the audio signal such as pitch, intensity or energy (e.g., formant) may be taken into account to determine the speaking rate; [0044] In response to the request, transcribed text corresponding to the audio data may be received, at 525. The transcribed text may be provided to the speech rehearsal assistance service in real-time as the user is speaking);
receiving the value of the physiological parameter onto the local device (Liao, page 1, paragraph 4; i.e., [0004] receiving a transcript for the audio data, the transcript including a plurality of words spoken during the speech rehearsal session); and
storing the words of text and the value of the physiological parameter such that the words of text are associated with the value of the physiological parameter (Liao, page 2, paragraph 19-20 & 23; page 5, paragraph 43-44 & 46-47; page 9, paragraph 80; i.e., [0020] This may be achieved by utilizing a speech recognition algorithm that converts spoken words to text in real-time, determining the number of syllables in the words spoken for a given time period and calculating the speaking rate based on the number of syllables. Furthermore, phonetic features of the audio signal such as pitch, intensity or energy (e.g., formant) may be taken into account to determine the speaking rate).
But Liao failed to teach the claim limitation wherein capturing video data on a remote device at a location of a remote party, and wherein the video data depicts the remote party; determining a value using the video data includes timestamps; generating an indicator at a segment of the video data based at least on the stored value of the physiological parameter associated with the stored words of text converted from the prosodic information at a corresponding video timestamp based on the timestamped value of the physiological parameter.
However, Mahajan teaches the limitation wherein capturing video data on a remote device at a location of a remote party, and wherein the video data depicts the remote party; determining a value using the video data (Mahajan, col 1, lines 53-60; i.e., facilitates the collaboration of different users through an online platform that connects the computing devices of online meeting participants. Online meeting content that is transmitted during an online meeting may include audio, video and text data that is encoded into data packets that are transmitted between online participant computing).
It would have been obvious to one of ordinary skill in the art before the effective date of the claimed invention to modify Liao to substitute transmission periodically from Mahajan for time period from Liao to fastest transmissions of the data to each of the different client with the best reliability (Mahajan, col 15, lines 50-55).
However, Reece teaches the limitation wherein capturing video data on a remote device at a location of a remote party, and wherein the video data includes timestamps (Reece, figure 10 & 16; page 5, paragraph 62-64; page 7, paragraph 85; page 8, paragraph 98; page 9, paragraph 108; page 14, paragraph 157; i.e., [0064] generate a transcription of the conversation, based on the acoustic and video recordings. This text data may include timestamps to align the transcript with the acoustic and video data); determining a value using the video data (Reece, figure 10 & 16; page 5, paragraph 62-64; page 7, paragraph 85; page 8, paragraph 98; page 9, paragraph 108; page 14, paragraph 157; i.e., [0108] Conversation analysis indicators (e.g., 706, 708, 710) may be stored with timestamps, for correlation with the source utterance and acoustic/video data of the conversation. In other words, conversation analysis indicators may be stored in a series, based on a series (i.e., sequence) of utterances and/or concatenated speaker data. The stored conversation analysis indicators may be graphed, visualized, and analyzed in aggregate by conversation analytics system); generating an indicator at a segment of the video data based at least on the stored value of the physiological parameter associated with the stored words of text converted from the prosodic information at a corresponding video timestamp based on the timestamped value of the physiological parameter (Reece, figure 10 & 16; page 5, paragraph 62-64; page 7, paragraph 85; page 8, paragraph 98; page 9, paragraph 108; page 14, paragraph 157; i.e., [0064] generate a transcription of the conversation, based on the acoustic and video recordings. This text data may include timestamps to align the transcript with the acoustic and video data; [0108] Conversation analysis indicators (e.g., 706, 708, 710) may be stored with timestamps, for correlation with the source utterance and acoustic/video data of the conversation. In other words, conversation analysis indicators may be stored in a series, based on a series (i.e., sequence) of utterances and/or concatenated speaker data. The stored conversation analysis indicators may be graphed, visualized, and analyzed in aggregate by conversation analytics system; [0157] Emotional indicators 1508 include high level emotion data consistent with each of the data modalities. Conversation synthesis ML system 1506 can analyze features from the data modalities to generate more emotional indicators 1508. In the illustrated implementation, emotional indicators 1508 are dependent on gaze and facial expression data from video data modality 1602).
It would have been obvious to one of ordinary skill in the art before the effective date of the claimed invention to modify Liao to substitute conversation score from Reece for rate of speech from Liao to produce data indicative of a goal ( e.g., minimizing a loss function) (Reece, page 2, paragraph 36).
As to claim 80, Liao-Mahajan-Reece teaches the method as recited in claim 78, wherein marking the transcript at a location during which the value of the physiological parameter surpasses a threshold for the physiological parameter (Liao, page 9, paragraph 80-84; i.e., [0080] receiving a transcript for the audio data, the
transcript including a plurality of words spoken during the speech rehearsal session;
[0083] determining if the speaking rate is within a threshold range; and [0084] enabling display of a notification on a display device in real time, if the speaking rate falls outside the threshold range).
Claim(s) 69-70 & 79 is/are rejected under 35 U.S.C. 103 as being unpatentable over Liao, U.S. Patent/Pub. No. US 2021/0065582 A1 in view of Mahajan, U.S. Patent/Pub. No. US 11044287 B1, and Reece, US 2021/0264900 A1, and further in view of Margon, US 2012/0146796 A1.
As to claim 69, Liao-Mahajan-Reece teaches the method as recited in claim 45. But Liao-Mahajan-Reece failed to teach the claim limitation wherein the audio data captures background noise at the location of the remote party, further comprising: determining a value of the background noise based on the audio data; receiving the value of the background noise onto the local device; and determining a magnitude of a remote party combination feature locally at the local device, wherein the remote party combination feature is determined based on the prosodic information combined with the value of the background noise.
However, Margon teaches the limitation wherein the audio data captures background noise at the location of the remote party, further comprising:
determining a value of the background noise based on the audio data (Margon, page 3, paragraph 43; i.e., [0043] The more random the noise background is the more effective the present invention is at looking for non-random changes in the received signal. The present invention measures both the rate of change and the magnitude of the change); receiving the value of the background noise onto the local device (Margon, page 3, paragraph 43; i.e., [0043] The more random the noise background. The present invention measures both the rate of change and the magnitude of the change); and determining a magnitude of a remote party combination feature locally at the local device, wherein the remote party combination feature is determined based on the prosodic information combined with the value of the background noise (Margon, page 3, paragraph 43; page 5, paragraph 67; i.e., [0043] The more random the noise background is the more effective the present invention is at looking for non-random changes in the received signal. The present invention measures both the rate of change and the magnitude of the change; [0067] The techniques, systems, and methods described herein to measure heart rhythm and respiration).
It would have been obvious to one of ordinary skill in the art before the effective date of the claimed invention to modify Liao-Mahajan-Reece to substitute modified signals from Margon for audio signal from Liao-Mahajan-Reece to of recording the monitored physiological function upon detection of the movement (Margon, page 1, paragraph 10).
As to claim 70, Liao-Mahajan-Reece teaches the method as recited in claim 45. But Liao-Mahajan-Reece failed to teach the claim limitation wherein the words of text and the prosodic information are displayed to a health professional, and wherein the remote party is a patient.
However, Margon teaches the limitation wherein the words of text and the prosodic information are displayed to a health professional, and wherein the remote party is a patient (Margon, page 2, paragraph 39; i.e., [0039] measuring heart rate and detecting heart defects, and respiratory rate and the modified signal reflected back from the heart moving within the patient).
It would have been obvious to one of ordinary skill in the art before the effective date of the claimed invention to modify Liao-Mahajan-Reece to substitute modified signals from Margon for audio signal from Liao-Mahajan-Reece to of recording the monitored physiological function upon detection of the movement (Margon, page 1, paragraph 10).
As to claim 79, Liao-Mahajan-Reece teaches the method as recited in claim 78. But Liao-Mahajan-Reece failed to teach the claim limitation wherein the physiological parameter is selected from the group consisting of: an instantaneous heart rate of the remote party at a first time instant within the time interval, an average heart rate of the remote party over the time interval, a heart rate variability of heart beats of the remote party during the time interval, an average breathing rate of the remote party over the time interval, and an average pupil dilation amount of the remote party over the time interval.
However, Margon teaches the limitation wherein the physiological parameter is selected from the group consisting of: an instantaneous heart rate of the remote party at a first time instant within the time interval, an average heart rate of the remote party over the time interval, a heart rate variability of heart beats of the remote party during the time interval, an average breathing rate of the remote party over the time interval, and an average pupil dilation amount of the remote party over the time interval (Margon, page 3, paragraph 43; i.e., [0043] The more random the noise background. The present invention measures both the rate of change and the magnitude of the change); receiving the value of the background noise onto the local device (Margon, page 3, paragraph 43; i.e., [0043] The present invention measures both the rate of change and the magnitude of the change); and determining a magnitude of a remote party combination feature locally at the local device, wherein the remote party combination feature is determined based on the prosodic information combined with the value of the background noise (Margon, page 3, paragraph 43; page 5, paragraph 67; i.e., [0043] The more random the noise background. The present invention measures both the rate of change and the magnitude of the change; [0067] The techniques, systems, and methods described herein to measure heart rhythm and respiration).
It would have been obvious to one of ordinary skill in the art before the effective date of the claimed invention to modify Liao-Mahajan-Reece to substitute modified signals from Margon for audio signal from Liao-Mahajan-Reece to of recording the monitored physiological function upon detection of the movement (Margon, page 1, paragraph 10).
Response to Arguments
Applicant’s arguments with respect to claim(s) 45-47, 68-70, 78-80 has/have been considered but are moot in view of the new ground(s) of rejection. Applicant’s arguments include the failure of previously applied art to expressly disclose “generating an indicator at a segment of the video data based at least on the stored prosodic information at a corresponding video timestamp based on the prosodic timestamps” (see Applicant’s response, 4/14/26, page 8-9). It is evident from the detailed mappings found in the above rejection(s) that Reece disclosed this functionality (see Reece, page 5, paragraph 62-64; page 9, paragraph 108). Further, it is clear from the numerous teachings (previously and currently cited) that the provision for “generating an indicator at a segment of the video data based at least on the stored prosodic information at a corresponding video timestamp based on the prosodic timestamps” was widely implemented in the networking art. Thus, Applicant’s arguments drawn toward distinction of the claimed invention and the prior art teachings on this point are not considered persuasive.
Response to Arguments
Applicant’s argument(s) filed 4/14/26 have been fully considered but they are not persuasive.
Argument 1
Appellant argues on page 1 of the Argument (claim(s) 45 & 78),
“No combination of Liao, Mahajan, Reece, and/or Margon can render amended claim 45 or amended claim 78 unpatentable because, even considered together, no combination of Liao, Mahajan, Reece, and/or Margon discloses or suggests "generating an indicator at a segment of the video data" based on the stored prosodic information or the stored value of physiological parameter”.
Examiner’s response to Argument 1:
As describe in Reece, the invention disclosed the method of (Reece, figure 10 & 16; page 5, paragraph 62-64; page 7, paragraph 85; page 8, paragraph 98; page 9, paragraph 108; page 14, paragraph 157; i.e., [0064] generate a transcription of the conversation, based on the acoustic and video recordings. This text data may include timestamps to align the transcript with the acoustic and video data; [0108] Conversation analysis indicators (e.g., 706, 708, 710) may be stored with timestamps, for correlation with the source utterance and acoustic/video data of the conversation. In other words, conversation analysis indicators may be stored in a series, based on a series (i.e., sequence) of utterances and/or concatenated speaker data. The stored conversation analysis indicators may be graphed, visualized, and analyzed in aggregate by conversation analytics system; [0157] Emotional indicators 1508 include high level emotion data consistent with each of the data modalities. Conversation synthesis ML system 1506 can analyze features from the data modalities to generate more emotional indicators 1508. In the illustrated implementation, emotional indicators 1508 are dependent on gaze and facial expression data from video data modality 1602), conversation analysis indicators (e.g., 706, 708, 710) may be stored with timestamps, for correlation with the source utterance and acoustic/video data of the conversation. In other words, conversation analysis indicators may be stored in a series, based on a series (i.e., sequence) of utterances and/or concatenated speaker data. The stored conversation analysis indicators may be graphed, visualized, and analyzed in aggregate by conversation analytics system). Therefore, a reasonable interpretation of Reece has been made, and the Examiner believes the “generating an indicator at a segment of the video data based at least on the stored value of the physiological parameter associated with the stored words of text converted from the prosodic information at a corresponding video timestamp based on the timestamped value of the physiological parameter” of Reece are within the scope of such interpretation.
Listing of Relevant Arts
Jawahar, U.S. Patent/Pub. No. US 20220215830 A1 discloses speech of a live session and correspond timestamps.
Muyal, U.S. Patent/Pub. No. US 20180308524 A1 discloses time stamp of the video with the corresponding speech.
Contact Information
The present application is being examined under the pre-AIA first to invent provisions.
THUONG NGUYEN whose telephone number is (571)272-3864. The examiner can normally be reached on Monday-Friday 9:00-6:00.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Noel Beharry can be reached on 571-270-5630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THUONG NGUYEN/Primary Examiner, Art Unit 2416
Prosecution Insights