DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending and have been examined. Claims 1, 19, and 20 are independent.
This Application was published as U.S. 2025/0372098 A1.
Apparent priority: June 4, 2024.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on March 31, 2025 and January 20, 2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Drawings
The drawings are objected to because reference characters appearing in the drawings are not mentioned in the description, and a reference character mentioned in the description does not appear in the drawings.
Reference characters 356M, 356N, and 356O in FIG. 8, 928 in FIG. 9, and 1212 in FIG. 12 are not mentioned in the description. Conversely, speech profile 150D is mentioned in paragraph [0211] but is not shown in the drawings. The references to probe buffers 140A and 140B in paragraph [0233] are addressed in the Specification objection below.
37 CFR 1.84(p)(5) provides:
Reference characters not mentioned in the description shall not appear in the drawings. Reference characters mentioned in the description must appear in the drawings.
Appropriate correction is required to make the reference characters in the drawings and description consistent, without introducing new matter. Identify in the description the elements represented by the drawing reference characters, and correct the description or drawings as appropriate for speech profile 150D.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as "amended." If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either "Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
a. In paragraph [0085], “during the a non-conversation mode 174” should apparently read “during the non-conversation mode 174.”
b. In paragraph [0111], “during the the non-conversation mode 174” should apparently read “during the non-conversation mode 174.”
c. In paragraph [0233], “the probe buffer 140A and the probe buffer 140B” should apparently read “the probe buffer 340A and the probe buffer 340B.”
d. In paragraphs [0264] and [0266], “in response to determining receiving the user input 1122” should apparently read “in response to receiving the user input 1122.”
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 (Statutory Category). Claim 1 recites a device, a machine, and claims 2-18 depend from that device claim. Claim 19 recites a method, a process. Claim 20 recites a non-transitory computer-readable medium storing executable instructions, a manufacture. Each claim therefore falls within a statutory category of invention.
Step 2A, Prong One (Recitation of a Judicial Exception). Independent claims 1, 19, and 20 recite the following:
1. A device comprising: a memory configured to store enrolled speech profiles; and one or more processors configured to: obtain multiple audio embeddings representing speech that is identified as associated with a single talker in an audio stream; determine a first speech profile based on the multiple audio embeddings; determine a similarity metric based on a comparison of the first speech profile to a second speech profile of the enrolled speech profiles; and based on the similarity metric, determine whether to combine the first speech profile and the second speech profile.
19. A method comprising: obtaining multiple audio embeddings representing speech that is identified as associated with a single talker in an audio stream; determining a first speech profile based on the multiple audio embeddings; determining a similarity metric based on a comparison of the first speech profile to a second speech profile of enrolled speech profiles; and based on the similarity metric, determining whether to combine the first speech profile and the second speech profile.
20. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: obtain multiple audio embeddings representing speech that is identified as associated with a single talker in an audio stream; determine a first speech profile based on the multiple audio embeddings; determine a similarity metric based on a comparison of the first speech profile to a second speech profile of enrolled speech profiles; and based on the similarity metric, determine whether to combine the first speech profile and the second speech profile.
The limitations “determine a similarity metric based on a comparison of the first speech profile to a second speech profile of the enrolled speech profiles” in claim 1 and the corresponding determining limitations in claims 19 and 20 recite a mathematical comparison of speech-profile data. The specification describes the metric as a distance between centroid vectors or between nearest audio embeddings, including cosine or Euclidean distance (published paragraphs [0113]-[0116]). Determining the measure of similarity between the numerical representations is a mathematical calculation even though a particular formula is not recited. The subsequent determination of whether to combine the profiles is an evaluation based on that measure. For the evaluation, a person provided the profile data and the comparison result can assess how alike the profiles are and decide whether their records should be grouped. This evaluation is a mental process. The computer obtains the single-talker embeddings, determines a profile from them, compares that profile with a stored profile, and makes the grouping decision. The rejection does not depend on treating the extraction of audio embeddings or operation of a neural network as practically performable in the human mind.
Step 2A, Prong Two (Integration into a Practical Application). The additional elements are the device, memory storing enrolled speech profiles, and processors of claim 1; obtaining the multiple audio embeddings representing speech identified with a single talker and determining the first profile in each independent claim; and the non-transitory medium and execution of its instructions by processors in claim 20. Obtaining the identified embeddings supplies the data to the comparison. Determining the profile organizes that data for the comparison. These steps are recited by their results, without a particular embedding-generation or profile-building technique. The specification permits the functions to be implemented with a CPU and DSPs, memory containing instructions, and ordinary computing devices (published paragraphs [0293]-[0294] and [0297]). These components provide the computing environment and storage for the comparison and decision.
Considered as a whole, each independent claim ends with deciding whether to combine two profiles; it does not require performing a particular profile-merging operation, using a specified metric calculation, or using the result to control a speech-separation or recognition process. The similarity metric is claimed at the level of the desired comparison result. The alternative centroid and nearest-embedding distances described in the specification are not limitations of these independent claims. Although the disclosure also describes improved speech recognition using filtered audio, these claims do not recite that filtering arrangement. The additional elements, individually and in combination, confine the mathematical comparison and grouping decision to speech-profile data processed with computing components. They do not integrate the exception into a practical application that improves the operation of a computer or another technology. The claims are directed to the abstract idea.
Step 2B (Inventive Concept). The additional elements do not amount to significantly more than the exception. The specification describes the claimed functions as implementable in hardware, processor-executed software, or combinations of the two, and identifies RAM, flash memory, ROM, registers, and hard disks as known storage media (published paragraphs [0335]-[0336]). Together with the CPU, DSP, and instruction-memory disclosure in paragraphs [0293]-[0294], this supports the finding that the recited execution and storage components perform their ordinary receiving, processing, and storing functions. The independent claims specify no unconventional configuration of those components. In the ordered combination, the components obtain data for the comparison, hold the profiles, and execute the comparison-based decision. This amounts to instructions to apply the exception with ordinary computing and data-storage functions, rather than an inventive concept. The conclusion rests on this analysis of the additional elements and ordered combination, not merely on the absence of a named metric or on the prior-art rejections below.
The dependent claims have been considered and do not cure the deficiencies of the independent claims. Each dependent claim is addressed separately below.
Claim 2 recites sets of embeddings, determining respective clusters, and determining the metric from a distance between the clusters. This supplies a mathematical form for the comparison, and is expressly considered rather than treated as omitting every metric limitation. Clustering numerical vectors and measuring their distance further limits the mathematical concept; the claim adds no particular technical implementation of the cluster-distance calculation. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 3 recites assigning a profile identifier and enrolling a profile when it is not to be combined and is not already enrolled. This is conditional classification and storage of the result of the comparison. The identifier records the classification; it does not change the operation of the memory or processors. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 4 recites adding the first profile’s embedding set to the second profile and discarding the first profile when combination is selected. These steps carry out the grouping decision by consolidating and deleting data records. No particular memory organization or technique for performing that consolidation is specified. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 5 recites generating an audio embedding, processing it with a talker detection neural network to obtain probability values, and applying a probability threshold to identify a talker. The neural network is an additional computer-implemented tool; its operation is not characterized as a human mental act. The claim defines its input and desired probability outputs without a particular network architecture or training technique. The probabilities and threshold comparison further specify numerical classification of the input data. The specification describes neural networks and their layers as machine-learning implementations at published paragraphs [0047]-[0054]. The network supplies classified data to the same profile-comparison process. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 6 further recites identifying a single talker when the other probability values are below the threshold. This narrows the numerical classification rule of claim 5 to the case in which only the first talker qualifies. It does not add a different implementation of the neural network or the subsequent profile comparison. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 7 recites enrollment buffers associated with talkers, generating an embedding, adding an unmatched embedding to the first talker’s buffer, and obtaining the multiple embeddings from that buffer. The buffers organize and temporarily store the input data used to form the profile. The claim specifies the data to store and the condition for storing it, without a particular buffer-management mechanism or memory arrangement beyond those functions. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 8 recites matching an embedding to an enrolled but immature profile and comparing that profile with another profile when its embedding count is below a maturity threshold. Counting items, evaluating maturity by a numerical threshold, and conditionally performing another comparison further limit the mathematical evaluation. The incorporated buffering and enrollment operations of claim 7 remain data collection and storage for that evaluation. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 9 recites endpoint data, generation and matching of an embedding, attribution to a profile identifier, and recording a segment as in progress when no segment for that identifier is in progress. The added rule records whether a matching speaker’s turn is open. Matching, labeling, and conditionally updating a turn record further organize the analyzed speech data. The claim does not use the endpoint record to perform a particular signal-processing operation, such as separating or filtering audio. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 10 recites recording a segment start time from a timestamp of the audio portion. This stores the time associated with the classified data. A timestamp further describes the record; it does not provide a technical implementation of the comparison or a new timing mechanism. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 11 recites ending an in-progress segment when an audio portion is not attributed to that segment’s speaker. This adds the complementary condition for closing the record opened under claim 9. The rule tracks the outcome of speaker attribution and updates the record accordingly. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 12 recites recording the segment end time from an audio-portion timestamp. This adds another time value to the turn record, with the same data-recording character as the start time. The full dependency on claims 11, 9, and 1 adds no specified technical use of the resulting interval. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 13 recites generating transcript text and a label containing the speaker’s name and the segment’s start and end times. This presents the identified content, speaker attribution, and timing information to a reader. The claim specifies the content of the transcript and label without a particular speech-recognition technique or change to the operation of the display or processor. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 14 recites displaying the transcript and combining two profiles when user input changes a label to an existing profile name. This receives a human correction and consolidates the records assigned to the same person. Display and input perform their ordinary presentation and reception functions; the profile combination implements the corrected grouping. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 15 recites evaluating whether at least two speech profiles are represented and transitioning from non-conversation mode to conversation mode. The mode transition records the outcome of a count-based classification. The claim does not recite the selective activation or deactivation of speech-processing components described in published paragraphs [0270]-[0274], or a particular change in processor power consumption. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 16 recites the reverse transition when fewer than two profiles are represented. This adds a complementary numerical condition to the mode classification of claim 15, without specifying a different technical operation of the computing components. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 17 recites performing the profile-combination decision during conversation mode. This selects when the abstract comparison and decision are performed based on the conversation classification. It does not add an implementation of the metric or a further technical use of the result. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim 18 recites determining during non-conversation mode whether to split a profile into multiple profiles. This adds the converse grouping decision and a condition for performing it. It does not require a particular split algorithm or recite the component deactivation described in the specification. Considered with the limitations inherited through its dependency chain, this addition neither integrates the exception into a practical application nor supplies significantly more than the exception.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless -(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention; or(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 19, and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Khoury (U.S. 2021/0326421 A1), hereinafter Khoury.
Regarding claim 1, Khoury discloses:
1. A device comprising: (Khoury, para. [0111]: “The analytics server 102 may be any computing device comprising one or more processors and software, and capable of performing the various processes and tasks described herein.”)
a memory configured to store enrolled speech profiles; and (Khoury, para. [0014]: “a speaker database comprising non-transitory machine-readable storage media configured to store data records containing speaker profiles”)
(Khoury, para. [0253]: “In step 612, the server updates a stored voiceprint and an existing enrolled (or known) speaker profile in the database, in response to the server determining that the embedding for the particular known speaker satisfies the similarity threshold (in step 606).” The enrolled (or known) speaker profiles stored in the speaker database read on the enrolled speech profiles.)
one or more processors configured to: (Khoury, para. [0258]: “The method 700a described below is performed by a server of an analytics system (e.g., analytics server) executing machine-readable software code, though any number of processors of any number of computing devices may perform the various operations of the method 700a.”; see also para. [0111].)
obtain multiple audio embeddings representing speech that is identified as associated with a single talker in an audio stream; (Khoury, para. [0259]: “A machine-learning architecture, comprising various machine-learning models (e.g., an embedding extractor model), extracts embeddings for inbound speakers based upon the features extracted from the audio signal.”)
(Khoury, para. [0261]: “Cluster 721, Cluster 723, Cluster 725, Cluster 727, Cluster 729 (collectively referred to as “clusters 720”) represent the embedding clusters extracted from the audio signal.”)
(Khoury, para. [0137]: “The clusters represent a collection of utterances similar to a particular speaker (e.g., a speaker cluster) where a voiceprint represents the centroid for the speaker cluster.” The embeddings of one embedding cluster of clusters 720, extracted from the audio signal and grouped as the utterances of a particular speaker, read on the multiple audio embeddings representing speech that is identified as associated with a single talker in an audio stream.)
determine a first speech profile based on the multiple audio embeddings; (Khoury, para. [0140]: “The analytics server 102 may create a voiceprint associated with a speaker based on one or more embeddings.”)
(Khoury, para. [0262]: “The server obtains one or more new label identifiers and voiceprints from a newly created speaker profile.” The newly created speaker profile of a cluster of clusters 720, holding the voiceprint created from that cluster’s embeddings, reads on the first speech profile.)
determine a similarity metric based on a comparison of the first speech profile to a second speech profile of the enrolled speech profiles; and (Khoury, para. [0264]: “The server computes each particular voiceprint-pair similarity score by evaluating, for example, the relative distance between the two voiceprints of the pair, the cosine similarity of the two voiceprints, or the PLDA of the two voiceprints.”)
(Khoury, para. [0265]: “Each cell in the similarity score matrix 738 represents a similarity score comparing a known cluster of known clusters 730 and a cluster of clusters 720.” The voiceprint-pair similarity score reads on the similarity metric. The speaker profile of a known cluster of known clusters 730, retrieved from the database (para. [0263]), reads on the second speech profile of the enrolled speech profiles.)
based on the similarity metric, determine whether to combine the first speech profile and the second speech profile. (Khoury, para. [0266]: “Optionally, the server may determine whether each particular maximum voiceprint-pair similarity score satisfies a preconfigured re-label threshold (sometimes called a migration threshold).”)
(Khoury, para. [0268]: “In addition to re-labeling the label identifiers, the server may determine to merge the voiceprints by averaging each of the voiceprints or applying the machine-learning architecture to the voiceprints to algorithmically combine the voiceprints.”)
(Khoury, para. [0270]: “The server created a new speaker profile for Cluster 729 because Cluster 729 was not similar enough to any of the putative speaker profiles.” Merging the two voiceprints of the pair into one reads on combining the first speech profile and the second speech profile. The threshold test on the voiceprint-pair similarity score, on which the pair is merged or the new profile is kept as a separate profile, reads on determining, based on the similarity metric, whether to combine the two profiles; see also para. [0170].)
Regarding claim 2, Khoury discloses:
2. The device of claim 1, wherein the first speech profile includes a first set of audio embeddings including at least the multiple audio embeddings, (Khoury, para. [0145]: “The speaker profile includes a voiceprint, a speaker cluster (e.g., embeddings associated with the voiceprint), a speaker identifier, and metadata.” The speaker cluster of embeddings held in the new speaker profile reads on the first set of audio embeddings.)
wherein the second speech profile includes a second set of audio embeddings, and wherein the one or more processors are configured to: (Khoury, para. [0168]: “The analytics server 102 may query a database (e.g., content database 112 or analytics database 104) and retrieve a set of speaker profiles (including voiceprints, speaker clusters containing embeddings, subscriber identifiers, or speaker identifiers).”)
determine a first cluster based on the first set of audio embeddings; (Khoury, para. [0261]: “Cluster 721, Cluster 723, Cluster 725, Cluster 727, Cluster 729 (collectively referred to as “clusters 720”) represent the embedding clusters extracted from the audio signal.”)
determine a second cluster based on the second set of audio embeddings; and (Khoury, para. [0263]: “The server receives the clusters Known 731, Known 733, Known 735, and Known 737 (collectively referred to as “known clusters 730”) in response to querying a database for the subscriber identifier associated with the clusters 720 extracted from the audio signal.”)
determine the similarity metric based on a distance between the first cluster and the second cluster. (Khoury, para. [0264]: “The server computes each particular voiceprint-pair similarity score by evaluating, for example, the relative distance between the two voiceprints of the pair, the cosine similarity of the two voiceprints, or the PLDA of the two voiceprints.”)
(Khoury, para. [0137]: “The clusters represent a collection of utterances similar to a particular speaker (e.g., a speaker cluster) where a voiceprint represents the centroid for the speaker cluster.” The relative distance between the two voiceprints of the pair, each voiceprint being the centroid of its speaker cluster, reads on the distance between the first cluster and the second cluster.)
Regarding claim 3, Khoury discloses:
3. The device of claim 1, wherein the one or more processors are configured to, based on determining that the first speech profile and the second speech profile are not to be combined and that the first speech profile is not one of the enrolled speech profiles, designate the first speech profile as associated with a first profile identifier (Khoury, para. [0270]: “The server created a new speaker profile for Cluster 729 because Cluster 729 was not similar enough to any of the putative speaker profiles. The server generated a new label identifier Known 740 for cluster 729 to represent the server creating a new speaker profile.” Cluster 729 is one of the clusters 720 extracted from the audio signal and is not one of the known clusters 730. The new label identifier reads on the first profile identifier.)
and add the first speech profile to the enrolled speech profiles. (Khoury, para. [0251]: “The server generates the new speaker profile in the analytics database or another memory location configured to store temporary or guest speaker profiles.”)
Regarding claim 19, Khoury discloses the corresponding subject matter:
Claim 19 is a method claim with profile-obtaining, profile-determining, similarity-comparing, and combination-decision limitations corresponding to those of claim 1, and is rejected under the same rationale. The full claim is reproduced below, and each of its limitations is mapped to Khoury.
19. A method comprising: (Khoury, para. [0258]: “FIG. 7A shows operational steps of a method 700a for correcting label identifiers (e.g., speaker identifiers, subscriber identifiers) of one or more voiceprints according to current and/or historical information.”)
obtaining multiple audio embeddings representing speech that is identified as associated with a single talker in an audio stream; (Khoury, para. [0259]: “A machine-learning architecture, comprising various machine-learning models (e.g., an embedding extractor model), extracts embeddings for inbound speakers based upon the features extracted from the audio signal.”)
(Khoury, para. [0261]: “Cluster 721, Cluster 723, Cluster 725, Cluster 727, Cluster 729 (collectively referred to as “clusters 720”) represent the embedding clusters extracted from the audio signal.”)
(Khoury, para. [0137]: “The clusters represent a collection of utterances similar to a particular speaker (e.g., a speaker cluster) where a voiceprint represents the centroid for the speaker cluster.” The embeddings of one embedding cluster of clusters 720, extracted from the audio signal and grouped as the utterances of a particular speaker, read on the multiple audio embeddings representing speech that is identified as associated with a single talker in an audio stream.)
determining a first speech profile based on the multiple audio embeddings; (Khoury, para. [0140]: “The analytics server 102 may create a voiceprint associated with a speaker based on one or more embeddings.”)
(Khoury, para. [0262]: “The server obtains one or more new label identifiers and voiceprints from a newly created speaker profile.” The newly created speaker profile of a cluster of clusters 720, holding the voiceprint created from that cluster’s embeddings, reads on the first speech profile.)
determining a similarity metric based on a comparison of the first speech profile to a second speech profile of enrolled speech profiles; (Khoury, para. [0264]: “The server computes each particular voiceprint-pair similarity score by evaluating, for example, the relative distance between the two voiceprints of the pair, the cosine similarity of the two voiceprints, or the PLDA of the two voiceprints.”)
(Khoury, para. [0265]: “Each cell in the similarity score matrix 738 represents a similarity score comparing a known cluster of known clusters 730 and a cluster of clusters 720.” The voiceprint-pair similarity score reads on the similarity metric. The speaker profile of a known cluster of known clusters 730, retrieved from the database (para. [0263]), reads on the second speech profile of the enrolled speech profiles.)
and based on the similarity metric, determining whether to combine the first speech profile and the second speech profile. (Khoury, para. [0266]: “Optionally, the server may determine whether each particular maximum voiceprint-pair similarity score satisfies a preconfigured re-label threshold (sometimes called a migration threshold).”)
(Khoury, para. [0268]: “In addition to re-labeling the label identifiers, the server may determine to merge the voiceprints by averaging each of the voiceprints or applying the machine-learning architecture to the voiceprints to algorithmically combine the voiceprints.”)
(Khoury, para. [0270]: “The server created a new speaker profile for Cluster 729 because Cluster 729 was not similar enough to any of the putative speaker profiles.” Merging the two voiceprints of the pair into one reads on combining the first speech profile and the second speech profile. The threshold test on the voiceprint-pair similarity score, on which the pair is merged or the new profile is kept as a separate profile, reads on determining, based on the similarity metric, whether to combine the two profiles; see also para. [0170].)
Regarding claim 20, Khoury discloses the corresponding subject matter:
Claim 20 is a non-transitory computer-readable-medium claim with profile-obtaining, profile-determining, similarity-comparing, and combination-decision limitations corresponding to those of claim 1, and is rejected under the same rationale. The full claim is reproduced below, and each of its limitations is mapped to Khoury.
20. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: (Khoury, para. [0424]: “When implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium.”)
obtain multiple audio embeddings representing speech that is identified as associated with a single talker in an audio stream; (Khoury, para. [0259]: “A machine-learning architecture, comprising various machine-learning models (e.g., an embedding extractor model), extracts embeddings for inbound speakers based upon the features extracted from the audio signal.”)
(Khoury, para. [0261]: “Cluster 721, Cluster 723, Cluster 725, Cluster 727, Cluster 729 (collectively referred to as “clusters 720”) represent the embedding clusters extracted from the audio signal.”)
(Khoury, para. [0137]: “The clusters represent a collection of utterances similar to a particular speaker (e.g., a speaker cluster) where a voiceprint represents the centroid for the speaker cluster.” The embeddings of one embedding cluster of clusters 720, extracted from the audio signal and grouped as the utterances of a particular speaker, read on the multiple audio embeddings representing speech that is identified as associated with a single talker in an audio stream.)
determine a first speech profile based on the multiple audio embeddings; (Khoury, para. [0140]: “The analytics server 102 may create a voiceprint associated with a speaker based on one or more embeddings.”)
(Khoury, para. [0262]: “The server obtains one or more new label identifiers and voiceprints from a newly created speaker profile.” The newly created speaker profile of a cluster of clusters 720, holding the voiceprint created from that cluster’s embeddings, reads on the first speech profile.)
determine a similarity metric based on a comparison of the first speech profile to a second speech profile of enrolled speech profiles; (Khoury, para. [0264]: “The server computes each particular voiceprint-pair similarity score by evaluating, for example, the relative distance between the two voiceprints of the pair, the cosine similarity of the two voiceprints, or the PLDA of the two voiceprints.”)
(Khoury, para. [0265]: “Each cell in the similarity score matrix 738 represents a similarity score comparing a known cluster of known clusters 730 and a cluster of clusters 720.” The voiceprint-pair similarity score reads on the similarity metric. The speaker profile of a known cluster of known clusters 730, retrieved from the database (para. [0263]), reads on the second speech profile of the enrolled speech profiles.)
and based on the similarity metric, determine whether to combine the first speech profile and the second speech profile. (Khoury, para. [0266]: “Optionally, the server may determine whether each particular maximum voiceprint-pair similarity score satisfies a preconfigured re-label threshold (sometimes called a migration threshold).”)
(Khoury, para. [0268]: “In addition to re-labeling the label identifiers, the server may determine to merge the voiceprints by averaging each of the voiceprints or applying the machine-learning architecture to the voiceprints to algorithmically combine the voiceprints.”)
(Khoury, para. [0270]: “The server created a new speaker profile for Cluster 729 because Cluster 729 was not similar enough to any of the putative speaker profiles.” Merging the two voiceprints of the pair into one reads on combining the first speech profile and the second speech profile. The threshold test on the voiceprint-pair similarity score, on which the pair is merged or the new profile is kept as a separate profile, reads on determining, based on the similarity metric, whether to combine the two profiles; see also para. [0170].)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4-8 and 15-17 are rejected under 35 U.S.C. 103 as being unpatentable over Khoury (U.S. 2021/0326421 A1) in view of Park (U.S. 2022/0180859 A1), hereinafter Park.
Regarding claim 4, Khoury discloses the device of claim 1, as set forth above, and further discloses:
4. The device of claim 1, wherein the one or more processors are configured to, in response to determining that the first speech profile and the second speech profile are to be combined, add a first set of audio embeddings of the first speech profile to the second speech profile (Khoury, para. [0170]: “If the analytics server 102 determines that the voiceprints are close (e.g., the relative distance satisfies a threshold), the voiceprints and associated label identifiers may be merged.”)
and discard the first speech profile. (Khoury, para. [0134]: “If the similarity of the clusters satisfies one or more thresholds, the analytics server 102 may merge the two voiceprints into a single voiceprint (e.g., taking the average of the voiceprints).”)
(Khoury, para. [0170]: “The analytics server 102 may migrate the label identifiers associated with the old labeled set to the new anonymous set such that the speaker identifier and/or subscriber identifier of the new anonymous set are replaced with the speaker identifier and/or subscriber identifier of the old labeled set.” A single voiceprint remaining under the known profile’s identifier, with the new profile’s own identifier replaced, reads on discarding the first speech profile.)
Khoury does not teach the embedding-set aspects of the combining recited in the claim. Park teaches updating an enrolled profile from a set of embeddings collected for the same talker (Park, para. [0077]: “The profile manager 126 updates the user speech profile 150 (that matched the audio feature data sets stored in the probe buffer 240) based on the audio feature data sets stored in the probe buffer 240.”).
Khoury and Park are analogous art concerning passive enrollment and maintenance of speaker profiles built from embeddings. Khoury’s speaker profile holds, with its voiceprint, the speaker cluster of embeddings associated with the voiceprint (para. [0145]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Khoury so that, when two profiles are merged, the set of embeddings of the newly created profile is added to the known speaker profile and the known profile’s voiceprint is updated from the enlarged set, in the manner of Park’s update of an enrolled profile from a collected set of embeddings, instead of only averaging the two voiceprints. The adding of the first profile’s set of audio embeddings to the second speech profile is this modification. One of ordinary skill would have been motivated to make this modification in order that the surviving profile is built from more of the speaker’s utterances and is therefore more mature. Khoury teaches that “a speaker voiceprint enrolled with 50 utterances is much more mature than a voiceprint enrolled with only 1 utterance” (para. [0052]).
Regarding claim 5, Khoury discloses the device of claim 1, as set forth above, and further discloses:
5. The device of claim 1, wherein the one or more processors are configured to: generate an audio embedding of an audio portion of the audio stream; (Khoury, para. [0259]: “A machine-learning architecture, comprising various machine-learning models (e.g., an embedding extractor model), extracts embeddings for inbound speakers based upon the features extracted from the audio signal.”)
Khoury does not teach the talker detection neural network aspects of the claim. Park teaches:
process, using a talker detection neural network, the audio embedding to generate one or more probability values, each probability value indicating an estimate of a probability that the audio portion represents speech of a corresponding talker of a set of talkers; and (Park, para. [0052]: “In a particular aspect, the audio feature data set 252 includes an audio feature vector, such as an embedding vector.”)
(Park, para. [0059]: “For example, analyzing the audio feature data sets 252 includes applying a speaker segmentation neural network (or another machine-learning-based system) to the audio feature data sets 252.”)
(Park, para. [0054]: “each segmentation score 254 indicates a probability that the audio feature data set input to the segmentor 124 represents speech of a corresponding talker” The speaker segmentation neural network, which outputs for each talker a probability that the input represents that talker’s speech, reads on the talker detection neural network, and the segmentation scores 254 read on the probability values.)
based on determining that a first probability value of the one or more probability values is greater than or equal to a probability threshold, determine that the audio portion is identified as associated with a first talker. (Park, para. [0060]: “For example, the data set segmentation result 256 output by the segmentor 124 indicates that speech of a talker 292 is detected in response to determining that a segmentation score 254 for the talker 292 satisfies (e.g., is greater than) the segmentation threshold 257.” A probability value that is greater than the threshold meets the recited greater-than-or-equal-to condition.)
Khoury and Park are analogous art concerning passive enrollment of speaker profiles from embeddings of an audio signal that includes speech of several speakers. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Khoury to process each extracted embedding with Park’s speaker segmentation neural network and threshold test, so that the talker of each audio portion is identified before the embeddings are grouped by speaker. One of ordinary skill would have been motivated to make this modification in order to keep speech of more than one talker out of a speaker’s profile. Khoury determines the beginning and end of an utterance “using speaker detection or other conventional speaker segmentation solutions” (para. [0119]), and Park teaches that “audio portions corresponding to multiple talkers are skipped or disregarded for generating or updating user speech profiles 150” (para. [0081]).
Regarding claim 6, the device of claim 5 is taught by Khoury in view of Park, as set forth above. Khoury does not teach the single-talker determination recited in the claim. Park teaches:
6. The device of claim 5, wherein the one or more processors are configured to, based on determining that each of the remaining probability values is less than the probability threshold, determine that the audio portion is identified as associated with the single talker, and that the single talker includes the first talker. (Park, para. [0066]: “In a particular aspect, the segmentor 124, in response to determining that the segmentation score 254A satisfies the segmentation threshold 257 and each of the segmentation score 254B and the segmentation score 254C fails to satisfy the segmentation threshold 257, generates a data set segmentation result 256 indicating that the audio portion 151 corresponds to speech of the talker 292A and does not correspond to speech of either the talker 292B or the talker 292C.”)
The rationale for this modification is similar to the rationale provided for claim 5.
Regarding claim 7, Khoury discloses the device of claim 1, as set forth above. Khoury does not teach the enrollment-buffer aspects of the claim. Park teaches:
7. The device of claim 1, wherein the memory is configured to store enrollment buffers associated with a set of talkers, and wherein the one or more processors are configured to: (Park, para. [0068]: “For example, the memory 232 includes an enroll buffer 234 and a probe buffer 240 designated for each of the talkers 292.”)
generate an audio embedding of an audio portion of the audio stream; and (Park, para. [0052]: “For example, the feature extractor 222 is configured to extract audio features of an audio portion 151 of the audio stream 141 to generate an audio feature data set (AFDS) 252. In a particular aspect, the audio feature data set 252 includes an audio feature vector, such as an embedding vector.”)
based at least in part on determining that the audio portion is identified as associated with a first talker and that the audio embedding does not match any of the enrolled speech profiles, add the audio embedding to a first enrollment buffer associated with the first talker, (Park, para. [0070]: “If the audio feature data set 252 does not match an existing user speech profile, the profile manager 126 is configured to, during an enrollment stage, add the audio feature data set 252 to an enroll buffer 234 and, in response to determining that the audio feature data sets 252 stored in the enroll buffer 234 satisfy the enrollment threshold 264, generate a user speech profile 150 based on the audio feature data sets 252 stored in the enrollment buffer 234.”)
(Park, para. [0078]: “During the enrollment stage, the profile manager 126, in response to determining that the segmentation result 236 indicates that the audio feature data set 252 represents speech of a talker 292 (e.g., the talker 292A), adds the audio feature data set 252 to an enroll buffer 234 (e.g., the enroll buffer 234A) corresponding to the talker 292.”)
wherein the multiple audio embeddings are obtained from the first enrollment buffer. (Park, para. [0080]: “The profile manager 126 thus generates the user speech profile 150C based audio feature data sets of audio portions corresponding to the same talker 292 (e.g., the talker 292A) that are stored in the enroll buffer 234 (e.g., the enroll buffer 234A) designated for the talker 292 (e.g., the talker 292A).” The audio feature data sets stored in the enroll buffer 234A, from which the new profile is generated, read on the multiple audio embeddings obtained from the first enrollment buffer.)
Khoury and Park are analogous art concerning passive enrollment of speaker profiles from embeddings. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Khoury to hold the embeddings that match no stored speaker profile in Park’s per-talker enroll buffers and to create the new speaker profile of claim 1 from the embeddings held in the buffer. One of ordinary skill would have been motivated to make this modification in order to improve the accuracy of the newly created profile. Park teaches that “Using multiple audio feature data sets to generate the user speech profile 150C improves accuracy of the user speech profile 150A in representing the speech of the talker 292A” (para. [0080]).
Regarding claim 8, the device of claim 7 is taught by Khoury in view of Park, as set forth above. Claim 8 recites “the first speech profile that is included in the enrolled speech profiles” and that “the multiple audio embeddings are obtained from the first speech profile.” The claim is interpreted as directed to the case in which the first speech profile has already been added to the enrolled speech profiles and has not yet reached the maturity threshold, consistent with published paragraphs [0167]-[0168] of the specification. Khoury further discloses:
8. The device of claim 7, wherein the one or more processors are configured to, based at least in part on determining that the audio embedding matches the first speech profile that is included in the enrolled speech profiles (Khoury, para. [0253]: “In step 612, the server updates a stored voiceprint and an existing enrolled (or known) speaker profile in the database, in response to the server determining that the embedding for the particular known speaker satisfies the similarity threshold (in step 606).”)
and that a count of a first set of audio embeddings of the first speech profile is less than a maturity threshold, (Khoury, para. [0153]: “For example, the server compares a maturity factor (e.g., number of utterances) against a preconfigured maturity threshold corresponding to the maturity factor (e.g., threshold number of utterances).”)
(Khoury, para. [0154]: “If the analytics server 102 determines that the voiceprint is not mature (e.g., does not satisfy a maturity threshold), then the analytics server 102 employs condition-dependent adaptive thresholding to determine high similarity thresholds and low similarity thresholds for particular voiceprints.”)
compare the first speech profile to the second speech profile to determine the similarity metric, (Khoury, para. [0264]: “The server computes each particular voiceprint-pair similarity score by evaluating, for example, the relative distance between the two voiceprints of the pair, the cosine similarity of the two voiceprints, or the PLDA of the two voiceprints.”)
(Khoury, para. [0267]: “In some implementations, the voiceprints may be associated with various re-label thresholds that depend upon a level of maturity for a given voiceprint in the voiceprint-pair.”)
wherein the multiple audio embeddings are obtained from the first speech profile. (Khoury, para. [0168]: “The analytics server 102 may query a database (e.g., content database 112 or analytics database 104) and retrieve a set of speaker profiles (including voiceprints, speaker clusters containing embeddings, subscriber identifiers, or speaker identifiers).” The embeddings of the speaker cluster contained in the retrieved speaker profile read on the multiple audio embeddings obtained from the first speech profile.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have performed Khoury’s voiceprint-pair comparison for an enrolled speaker profile that an incoming embedding has matched and whose number of utterances is below the maturity threshold, and not for a mature profile. One of ordinary skill would have been motivated to make this modification in order to save processing by directing the comparison at the profiles that are not yet mature. Khoury teaches that the server “may determine not to correct label identifiers if the voiceprints retrieved by the analytics server 102 are associated with a high confidence value” and that a voiceprint created from active enrollment embeddings “may be considered pure and mature” (para. [0167]).
Regarding claim 15, Khoury discloses the device of claim 1, as set forth above. The recited non-conversation mode and conversation mode are interpreted as operating states of the device selected according to whether speech of at least two talkers is present, and speech “associated with” a speech profile is interpreted as speech of a talker whom the device identifies by that talker’s speech profile. Khoury does not teach the conversation-mode aspects of the claim. Park teaches:
15. The device of claim 1, wherein the one or more processors are configured to: determine, in a non-conversation mode, whether a first portion of the audio stream corresponds to speech associated with at least two speech profiles; and (Park, para. [0088]: “The talker detector 278 is configured to determine, in the power mode 282, whether the audio stream 141 corresponds to speech of at least two distinct talkers.”)
(Park, para. [0045]: “For example, the output may include the profile ID 155 of the user speech profile 150 that matches the talker-homogenous audio segment 111.” The power mode 282 reads on the non-conversation mode. Each talker being reported as the user speech profile that the talker’s speech matches, speech of at least two distinct talkers reads on speech associated with at least two speech profiles.)
based on determining that the first portion of the audio stream corresponds to speech associated with at least two speech profiles, transition to a conversation mode. (Park, para. [0088]: “The one or more processors 220 are configured to, in response to determining that an output of the talker detector 278 indicates that the audio stream 141 corresponds to speech of at least two distinct talkers, transition from the power mode 282 to the power mode 284 and activate the segmentor 124.” The power mode 284 reads on the conversation mode.)
Khoury and Park are analogous art concerning identification of the speakers of a multi-speaker audio signal by their stored profiles. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Khoury’s device to operate in Park’s two modes, determining in the first mode whether the audio stream corresponds to speech of at least two talkers and transitioning to the second mode when it does. One of ordinary skill would have been motivated to make this modification in order to reduce resource consumption while a single talker is speaking. Park teaches that “Staying in (or transitioning to) the power mode 282 when the segmentor 124 does not have to be used reduces overall resource consumption.” (para. [0088]). In the combination each talker is identified by a stored profile, because Khoury likewise identifies each speaker of a multi-speaker audio sample by “the closest match between the particular speaker embedding and a particular voiceprint” (para. [0051]).
Regarding claim 16, the device of claim 15 is taught by Khoury in view of Park, as set forth above. Khoury does not teach the return to the non-conversation mode recited in the claim. Park teaches:
16. The device of claim 15, wherein the one or more processors are configured to: determine, in the conversation mode, whether a second portion of the audio stream corresponds to speech associated with fewer than two speech profiles; and (Park, para. [0142]: “The talker detector 278, in the power mode 284, determines whether multiple talkers are detected, at 910.” Multiple talkers not being detected is fewer than two talkers being detected and, each talker being identified by the matching speech profile as set forth for claim 15, reads on speech associated with fewer than two speech profiles.)
based on determining that the second portion of the audio stream corresponds to speech associated with fewer than two speech profiles, transition to the non-conversation mode. (Park, para. [0142]: “Alternatively, the talker detector 278, in response to determining that multiple talkers are not detected within a threshold count of audio feature data sets, transitions the one or more processors 220 from the power mode 284 to the power mode 282.”)
The rationale for this modification is similar to the rationale provided for claim 15.
Regarding claim 17, the device of claim 15 is taught by Khoury in view of Park, as set forth above. Khoury further discloses:
17. The device of claim 15, wherein the one or more processors are configured to, during the conversation mode, determine whether to combine the first speech profile and the second speech profile. (Khoury, para. [0266]: “Optionally, the server may determine whether each particular maximum voiceprint-pair similarity score satisfies a preconfigured re-label threshold (sometimes called a migration threshold).”)
Khoury does not teach that the determination is made during the conversation mode. Park teaches:
17. The device of claim 15, wherein the one or more processors are configured to, during the conversation mode, determine whether to combine the first speech profile and the second speech profile. (Park, para. [0090]: “In a particular aspect, the one or more processors 220 are configured to, in response to determining that the segmentation result 236 indicates that the audio stream 141 corresponds to speech of at least two distinct talkers, transitions from the power mode 282 to the power mode 284 and activates the profile manager 126, the one or more audio analysis applications 180, or a combination thereof. For example, the profile manager 126, in the power mode 284, performs the comparison of the audio feature data set 252 to the plurality of user speech profiles 150.”)
Khoury and Park are analogous art concerning comparison of speaker embeddings with stored speaker profiles. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Khoury so that the profile comparison of claim 1, including the determination whether to combine the two profiles, is performed while the device is in Park’s power mode 284, which is entered when speech of at least two talkers is detected. One of ordinary skill would have been motivated to make this modification in order to conserve energy by activating the profile comparison only when it is needed. Park teaches that the processors “conserve energy by operating in the power mode 282 (as compared to the power mode 284) and transition to the power mode 284 as needed to activate components that do not operate in the power mode 282” (para. [0087]).
Claims 9-13 are rejected under 35 U.S.C. 103 as being unpatentable over Khoury (U.S. 2021/0326421 A1) in view of Fu (U.S. 10,978,073 B1), hereinafter Fu.
Regarding claim 9, the recited end-point detection data and its in-progress indication are interpreted in light of published paragraphs [0090], [0091], [0095] and [0097] of the specification. The end-point detection data maps the profile identifier of each enrolled speech profile to an audio segment start time and an audio segment end time. No audio segment of a profile identifier is in-progress when no valid start time is recorded for it. Updating the data to indicate that an audio segment is in-progress is recording the start time of the segment, and updating the data to indicate that the audio segment has ended is recording its end time.
Khoury discloses the device of claim 1, as set forth above, and further discloses:
generate an audio embedding of an audio portion of the audio stream, the multiple audio embeddings including the audio embedding; (Khoury, para. [0259]: “A machine-learning architecture, comprising various machine-learning models (e.g., an embedding extractor model), extracts embeddings for inbound speakers based upon the features extracted from the audio signal.”)
based on determining that the audio embedding matches the first speech profile associated with a first profile identifier, designate the audio portion as representing speech corresponding to the first profile identifier; and (Khoury, para. [0250]: “If the server determines that the similarity score for the particular speaker embedding satisfies the high similarity threshold, then the server determines that the speaker is likely associated with a known registered user.”)
(Khoury, para. [0137]: “The analytics server 102 identifiers speaker clusters using speaker identifiers.” The speaker identifier of the matched speaker profile reads on the first profile identifier.)
Khoury does not teach the end-point detection aspects of the claim. Fu teaches:
9. The device of claim 1, wherein the memory is configured to store end-point detection data indicating time periods of audio segments associated with the enrolled speech profiles, (Fu, col. 9, ll. 44-48: “In some embodiments, each segment of the audio-form conversation and the synchronized text generated at process 1306 is associated with one or more timestamps, each timestamp corresponding to the start time, and/or the end time.”)
(Fu, col. 10, ll. 37-42: “In some embodiments, at the process 1308, the speaker label is assigned to each segment of text synchronized to one corresponding segment of audio-form conversation, by matching a voiceprint of the corresponding segment of audio-form conversation to a reference voiceprint corresponding to a speaker label.” The start and end timestamps kept for each segment, each segment being labeled with the speaker whose reference voiceprint it matches, read on the end-point detection data indicating time periods of audio segments associated with the enrolled speech profiles.)
wherein each of the audio segments includes one or more audio portions of the audio stream, and wherein the one or more processors are configured to: (Fu, col. 9, ll. 40-44: “In certain embodiments, the conversation (e.g., audio-form conversation and/or the synchronized text) is segmented at process 1304 into different segments when a speaker change occurs and/or a natural pause occurs.”)
(Fu, col. 9, ll. 56-57: “In some examples, each segment is spoken by a single speaker.”)
based on determining that the audio portion represents speech corresponding to the first profile identifier and that the end-point detection data indicates that no audio segment of the first profile identifier is in-progress, update the end-point detection data to indicate that an audio segment of the first profile identifier is in-progress. (Fu, col. 9, ll. 40-44: “In certain embodiments, the conversation (e.g., audio-form conversation and/or the synchronized text) is segmented at process 1304 into different segments when a speaker change occurs and/or a natural pause occurs.”)
(Fu, col. 9, ll. 44-48: “In some embodiments, each segment of the audio-form conversation and the synchronized text generated at process 1306 is associated with one or more timestamps, each timestamp corresponding to the start time, and/or the end time.” Under the interpretation stated above, beginning a new segment at a speaker change, when the speaker of the audio has no segment already begun, and associating the new segment with the timestamp corresponding to its start time reads on updating the end-point detection data to indicate that an audio segment of the first profile identifier is in-progress.)
Khoury and Fu are analogous art concerning identification of the speakers of a multi-speaker audio signal by matching voiceprints. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Khoury to keep, for the speakers it identifies by their stored speaker profiles, Fu’s record of single-speaker segments, each segment begun at a speaker change and associated with a start time and an end time. One of ordinary skill would have been motivated to make this modification in order to increase the comprehensiveness and accuracy of the information extractable from the conversation. Fu states that it is highly desirable “to increase the value of conversations, such as human-to-human conversations, at least by increasing the comprehensiveness and accuracy of information extractable from the conversations” (col. 1, ll. 31-36).
Regarding claim 10, the device of claim 9 is taught by Khoury in view of Fu, as set forth above. Khoury does not teach the start-time aspects of the claim. Fu teaches:
10. The device of claim 9, wherein the one or more processors are configured to update the end-point detection data to indicate a start time of the audio segment of the first profile identifier, (Fu, col. 9, ll. 48-52: “In certain embodiments, each segment of the audio-form conversation and the synchronized text generated at process 1306 is associated with a segment timestamp, the segment timestamp indicating the start time, the segment duration, and/or the end time.”)
and wherein the start time is based on a time stamp associated with the audio portion. (Fu, col. 9, ll. 44-48: “In some embodiments, each segment of the audio-form conversation and the synchronized text generated at process 1306 is associated with one or more timestamps, each timestamp corresponding to the start time, and/or the end time.” The timestamp corresponding to the start time of the segment’s audio reads on the time stamp associated with the audio portion.)
The rationale for this modification is similar to the rationale provided for claim 9.
Regarding claim 11, the device of claim 9 is taught by Khoury in view of Fu, as set forth above. Khoury does not teach the segment-ended update recited in the claim. Fu teaches:
11. The device of claim 9, wherein the one or more processors are configured to, based on determining that the audio portion is designated as not representing speech corresponding to a second profile identifier of the second speech profile, and that the end-point detection data indicates that an audio segment of the second profile identifier is in-progress, update the end-point detection data to indicate that the audio segment of the second profile identifier has ended. (Fu, col. 9, ll. 40-44: “In certain embodiments, the conversation (e.g., audio-form conversation and/or the synchronized text) is segmented at process 1304 into different segments when a speaker change occurs and/or a natural pause occurs.”)
(Fu, col. 9, ll. 44-48: “In some embodiments, each segment of the audio-form conversation and the synchronized text generated at process 1306 is associated with one or more timestamps, each timestamp corresponding to the start time, and/or the end time.” Under the interpretation stated for claim 9, a speaker change is the arrival of audio that is not the speech of the speaker of the segment then in progress. Ending that segment at the speaker change and associating it with the timestamp corresponding to its end time reads on updating the end-point detection data to indicate that the audio segment of the second profile identifier has ended.)
The rationale for this modification is similar to the rationale provided for claim 9.
Regarding claim 12, the device of claim 11 is taught by Khoury in view of Fu, as set forth above. Khoury does not teach the end-time aspects of the claim. Fu teaches:
12. The device of claim 11, wherein the one or more processors are configured to update the end-point detection data to indicate an end time of the audio segment of the second profile identifier, (Fu, col. 9, ll. 48-52: “In certain embodiments, each segment of the audio-form conversation and the synchronized text generated at process 1306 is associated with a segment timestamp, the segment timestamp indicating the start time, the segment duration, and/or the end time.”)
and wherein the end time is based on a time stamp associated with the audio portion. (Fu, col. 9, ll. 44-48: “In some embodiments, each segment of the audio-form conversation and the synchronized text generated at process 1306 is associated with one or more timestamps, each timestamp corresponding to the start time, and/or the end time.” The timestamp corresponding to the end time of the segment’s audio reads on the time stamp associated with the audio portion.)
The rationale for this modification is similar to the rationale provided for claim 9.
Regarding claim 13, the device of claim 12 is taught by Khoury in view of Fu, as set forth above. Khoury does not teach the transcript aspects of the claim. Fu teaches:
13. The device of claim 12, wherein the one or more processors are configured to generate a transcript including: text corresponding to the audio segment of the second profile identifier, (Fu, col. 9, ll. 1-3: “At the process 1304, a conversation (e.g., an audio-form conversation received at process 1302) is automatically transcribed into synchronized text.”)
(Fu, col. 10, ll. 7-10: “In some examples, each segment of audio-form conversation corresponds to one segment of synchronized text, and the segment of synchronized text is synchronized with that segment of audio-form conversation.”)
and a label associated with the text, wherein the label is based on the end-point detection data and indicates a second profile name of the second speech profile, a start time of the audio segment of the second profile identifier, and the end time of the audio segment of the second profile identifier. (Fu, col. 13, ll. 24-27: “each segment is accompanied with a speaker label (e.g., name and/or picture)”)
(Fu, col. 13, ll. 28-31: “also includes a segment timestamp indicating the start of the segment and/or the end of the segment” The speaker label and the segment timestamp shown with each segment of text read on the label, and the speaker name in the speaker label reads on the second profile name; see also col. 10, ll. 26-28, and FIG. 8.)
The rationale for this modification is similar to the rationale provided for claim 9.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Khoury (U.S. 2021/0326421 A1) in view of Fu (U.S. 10,978,073 B1), and further in view of Baughman (U.S. 2018/0286409 A1), hereinafter Baughman.
Regarding claim 14, the device of claim 13 is taught by Khoury in view of Fu, as set forth above. Khoury does not teach the display and user-input aspects of the claim. Fu teaches:
14. The device of claim 13, wherein the one or more processors are configured to: provide the transcript to a display device; and (Fu, col. 5, ll. 41-46: “For example, the processor 140 is configured to generate a transformed conversation and send the transformed conversation to the presenter 150 (e.g., a display) of the system 100.”)
in response to receiving a user input indicating that the label is to be updated to indicate an input profile name associated with a third speech profile of the enrolled speech profiles, combine the second speech profile and the third speech profile. (Fu, col. 11, ll. 35-37: “In yet another example, speaker labels automatically assigned (e.g., at process 1308) are editable (e.g., by a user and/or human).”)
(Fu, col. 10, ll. 56-60: “In certain examples, the process 1300 automatically identifies a new speaker voice print, but the user has not provided the name and/or the picture of the speaker yet; hence the speaker is determined to be, for example, an unknown speaker.”)
Khoury and Fu do not teach combining the second and third speech profiles recited in the claim. Baughman teaches:
14. The device of claim 13, wherein the one or more processors are configured to: provide the transcript to a display device; and in response to receiving a user input indicating that the label is to be updated to indicate an input profile name associated with a third speech profile of the enrolled speech profiles, combine the second speech profile and the third speech profile. (Baughman, para. [0022]: “The cluster action Merge directs the speaker diarization engine 120 to merge a cluster with a previous cluster because the previous cluster and the cluster shares the same speaker, based on similar/identical cluster epicenters of the respective clusters.”)
(Baughman, para. [0022]: “By use of the human reinforcement learning, the speaker diarization engine 120 reviews and adjusts speaker labels according to inputs/feedbacks from human listeners.”)
Khoury, Fu and Baughman are analogous art concerning the labeling of the speakers of a conversation from their stored speaker data. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Khoury and Fu so that, when the user edits the speaker label of the second speech profile to the name of a third, already enrolled speech profile, the second and third speech profiles are merged, as Baughman merges a cluster with a previous cluster that shares the same speaker and adjusts speaker labels on feedback from human listeners, and as Khoury merges the voiceprints and label identifiers of two profiles found to belong to the same speaker (Khoury, para. [0170]). In the combination each speech profile holds a speaker cluster of embeddings (Khoury, para. [0145]), so Baughman’s merge of a cluster into the previous cluster of the same speaker, applied to the clusters held in the second and third speech profiles, is the combining of those two profiles. The user’s entry of the enrolled name identifies the two profiles as the same speaker; using that identification as the occasion for the merge is part of this modification. One of ordinary skill would have been motivated to make this modification in order to improve confidence in the speaker labels. Baughman merges human reinforcement with the cluster evidences “to improve confidence of speaker labels as identified during the diarization” (para. [0022]).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Khoury (U.S. 2021/0326421 A1) in view of Park (U.S. 2022/0180859 A1), and further in view of Kompe (U.S. 2005/0187770 A1), hereinafter Kompe.
Regarding claim 18, the device of claim 15 is taught by Khoury in view of Park, as set forth above. The recited split is read in light of published paragraphs [0123], [0126]-[0127] and [0271] of the specification, as determining whether audio embeddings of the first speech profile are to be moved out of it into a new, separate speech profile. Khoury does not teach the non-conversation-mode aspect of the claim. Park teaches:
18. The device of claim 15, wherein the one or more processors are configured to, during the non-conversation mode, determine whether to split the first speech profile into multiple speech profiles. (Park, para. [0089]: “For example, in response to an output of the talker detector 278 indicating that a single talker is detected, the one or more processors 220 remain or transition to the power mode 282. In the power mode 282, the profile manager 126 generates or updates a user speech profile 150 of the single talker based on the audio feature data sets 252.”)
Khoury and Park do not teach the split determination recited in the claim. Kompe teaches:
18. The device of claim 15, wherein the one or more processors are configured to, during the non-conversation mode, determine whether to split the first speech profile into multiple speech profiles. (Kompe, para. [0031]: “According to a further preferred embodiment of the inventive method upon addition of at least one further utterance or speech input of a known speaker it is checked on whether or not the associated speaker model and/or speaker cluster can be split up into distinct sub-models and/or sub-clusters, respectively.”)
(Kompe, para. [0033]: “Preferably, the splitting process is carried out if a distance between said sub-clusters and/or sub-models exceeds a predefined threshold, in which case the distinct sub-models and/or said sub-clusters are generated and used for discriminating and describing distinct speakers.”)
(Kompe, para. [0029]: “According to a further idea of the present invention a generated speaker model and/or a generated speaker cluster is in each case stored together with the input speech associated therewith or with sequences thereof.”)
(Kompe, para. [0077]: “A split is assumed if said distance as a combination of acoustic distance and differences in user profiles exceeds a predefined threshold. In this case, two distinct speaker models are generated and two different user profiles are associated with them:” The known speaker’s stored speaker model, held with the speech it was built from, reads on the first speech profile, and the two distinct speaker models generated on a split read on the multiple speech profiles.)
Khoury, Park and Kompe are analogous art concerning the maintenance of stored speaker profiles built from a speaker’s utterances. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Khoury and Park so that, when the profile manager updates the profile of the single detected talker in Park’s power mode 282, it checks, as Kompe checks a known speaker’s model upon each added utterance, whether that profile can be split into distinct profiles, and splits it when the distance between its sub-clusters exceeds the threshold. Performing the check at that profile update, and so in the power mode 282, is part of this modification. One of ordinary skill would have been motivated to make this modification in order to separate the speech of different speakers held in one profile. Kompe teaches that “any given speaker model might erroneously comprise speech from more than one speakers” (para. [0074]) and that on a split “acoustical similar speakers which have very different user profiles ten be separated” (para. [0079]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure:
a. Togami (U.S. 2024/0096346 A1): derives talker embedding vectors, applies personalized noise suppression for individual talkers, and merges single-talker transcriptions into a multi-talker transcription.
b. Park, T. J., et al., “A Review of Speaker Diarization: Recent Advances with Deep Learning,” arXiv:2101.09624v1, January 24, 2021: surveys speaker diarization, speaker embeddings and clustering, and integration with speech recognition.
c. Swaminathan (U.S. 2023/0298561 A1): uses a combined representation of multiple sound sources to selectively retain or remove their sounds from an input audio signal.
d. Tritschler (U.S. 6,424,946 B1): transcribes an audio source while concurrently segmenting it into speaker turns and identifying the speaker of each segment against an enrolled speaker database, records the start time and speaker label of each segment in a speaker turn database, and prompts the user for the name of an unknown speaker.
e. Fu (U.S. 11,423,911 B1), a patent other than the Fu reference applied above: segments live conversation elements, assigns a segment speaker to each segment, transcribes the segments, and transmits the result to subscribers.
f. Nagai (JP 2006-208482 A): identifies the conference attendant who is currently speaking by comparing voiceprint data extracted from the speech with stored voiceprint data of each attendant, stores the results with the elapsed time from the start of the conference, and displays the utterance state of each attendant over time.
g. Rohatgi (U.S. 2024/0112681 A1): generates unlabeled voiceprints from clusters of voice signals, adds newly captured voice signals to the matching cluster, and provides an editor with which a user may force a merge of different clusters or delete a cluster.
The International Search Report and Written Opinion for PCT/US2025/031561, mailed October 7, 2025 and submitted with the January 20, 2026 IDS, are also of record. The rejections above are based on the identified reference disclosures and the stated examination analysis.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YUVAL H. LEVENTAL whose telephone number is (571) 270-3130. The examiner can normally be reached Monday-Friday, 8:00 AM - 5:00 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, PIERRE-LOUIS DESIR, can be reached at (571) 272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YUVAL HAIM LEVENTAL/Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659
