DETAILED ACTION
1. This action is responsive to remarks filed 6/9/26.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
3. Claims 1-3, 5-6, 8, 12-14, 16-18, 20 have been amended.
Response to Arguments
4. Applicant’s arguments filed have been fully considered but are moot based on the new grounds of rejection responsive to the amendments (where Applicant argues that “Horton fails to disclose all of the elements of the amended claims”).
Claim Rejections - 35 USC § 102
5. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
6. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
7. Claims 1-8, 10, 12-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Horton et al (2020/0411013).
Regarding claim 1 Horton et al (2020/0411013) teaches A method (0065: system and method; 0133: processing engine for implementing the system and method; figure 5), comprising:
obtaining, by a computing system and from one or more sources that provide media files over one or more networks, a plurality of media files that each includes speech of one or more speakers (0133/fig 5: processing engine for implementing;
65; 69: persons participating; [0125] Further embodiments will now be disclosed that provide enrollment and authentication of individuals both inside and outside secure facilities using audio data in any format, for example, live or pre-recorded video, audio, or any other type of call or recording, originating from any source, for example, land line or wireless live calls, pre-recorded calls, video or audio messages, and video or audio recordings of any type; 136;
140 audio files with speakers);
processing, by the computing system, the plurality of media files to generate indexed data, wherein the indexed data includes, for each media file of the plurality of media files, a corresponding embedding for each speaker of the one or more speakers and one or more transcripts of speech that each corresponds to one or more languages detected in the speech in the media file, and wherein the embeddings within the indexed data are organized independently of known speaker identity (131 transcribe and index audio data for keyword searching; 140: the system may also provide robust transcription and indexing of audio files; files may be subject to segmentation analysis, voice print generation, speaker identification and assignment, and location in the biometric voice print database;
20: biometric voice print (BVP); 38-39 speaker identification to biometric voice prints;
41: transcribing the audio files; [0043] Transcribing the audio files to a text format may include using an automatic language detection process to distinguish languages spoken by the speakers.; [0129] separate persons participating in the call, referred to as speakers, in the audio data; [0130] create individual biometric voice print databases;
139: custom indices of the biometric voiceprint database 508 related to speaker identities, key words, or any other fields or records of the biometric voice print database;
141 language detection; 142; 143; 146 keyword indexing);
{where Horton teaches
[0039] The method may also include assigning speaker identification records to biometric voice prints with authenticated speakers, and storing the speaker identification records and the biometric voice prints in the biometric voice print database.
138: The processing engine 500 may then transform the model into a matrix that can be stored as a voice print in the biometric voice print database 508 in the form of, for example, a binary file. If the identity of the speaker is known, the resulting voice print can then be assigned to the known speaker via a speaker identification record in the biometric voice print database 508. If the identity of speaker is not known, the resulting voice print may then be assigned to a default speaker, for example, Jaimie Doe, via a record in the biometric voice print database 508, and may be reserved for future identification. As mentioned above, the biometric voice print database 508 may be located in a memory of the processing engine 500.
139: custom indices of the biometric voiceprint database 508 related to speaker identities
Where it appears Applicant is attempting to recite that the embeddings (independent of speaker identity) are embeddings initially created and stored before speaker identification used in subsequent steps, but appears to be taught by Horton, which teaches storing mathematical representations of speaker voices, and then performing identification}
receiving, by the computing system, an indication of at least one of a selection of a particular speaker from the one or more speakers or a selection of a particular keyword from a plurality of keywords ([0093] h) matching 34 the speaker to a BVP by identifying all existing voice samples matched by the incoming voice sample.; 148 – receiving words and phrases;
149 a user may upload an audio file which a speaker presents a threat to an emergency operator. The user may then request that the system identify the speaker from the pool of voice prints in the biometric voice print database 508 assigned to some or all speakers that meet certain criteria);
generating, by the computing system, one or more correlations based on the indexed data, wherein the one or more correlations include at least one of a first association among the particular speaker and the one or more speakers that corresponds to embeddings within the indexed data to isolate unknown speakers from within corresponding one or more transcripts or a second association among the particular keyword and keywords detected in transcripts as spoken by the one or more speakers that corresponds to the embeddings within the indexed data to isolate the unknown speakers from within the corresponding one or more transcripts (20: The detection system can be used to verify the identity of persons both inside and outside the facility and to detect known persons-of-interest (POI) in subsequent calls 93; 95: An occasion where this is useful is if the inmate is on a watch list and the agency wants to know who is leaving him a voice mail in real time; 146-148: keywords; when the established criteria is met; recognizing phrases…discussions.
149 a user may upload an audio file which a speaker presents a threat to an emergency operator. The user may then request that the system identify the speaker from the pool of voice prints in the biometric voice print database 508 assigned to some or all speakers that meet certain criteria); and
outputting, by the computing system, based on the one or more correlations, an indication regarding the one or more correlations
(0020: The detection system can be used to verify the identity of persons both inside and outside the facility and to detect known persons-of-interest (POI) in subsequent calls. ;The secure facility handling the call will be notified in real time if a caller is a known person inside the facility, a person-of-interest, a previous caller, or a new caller
[0039] The method may also include assigning speaker identification records to biometric voice prints with authenticated speakers, and storing the speaker identification records and the biometric voice prints in the biometric voice print database.
142: The automatic language detection process provides the ability to offer notification, alerting and routing options based on the spoken languages, such as real time notification when a speaker utters certain words or phrases in a particular language
148 notifications and alerts).
Regarding claim 2 Horton teaches The method of claim 1, wherein processing the plurality of media files to generate indexed data includes clustering excerpts from the plurality of media files (120: The original call assigned to the calling number is then reassigned to the newly identified speaker, forming a call cluster. Additional calls from this or other numbers identified as including this speaker may be added to the cluster with the goal of grouping the audio files by speaker).
Regarding claim 3 Horton teaches The method of claim 1, wherein processing the plurality of media files to generate indexed data includes:
extracting the corresponding embedding for each speaker of the one or more speakers from overlapping windows from each of the plurality of media files (20: BVP; 38-39 speaker ID; 74: BVPs for the caller as well as the called party;
fig 4; 96: window; overlap); and
applying clustering to the corresponding embedding for each speaker of the one or more speakers (120 cluster…grouping the audio files by speaker, using the speaker embeddings/ID).
Regarding claim 4 Horton teaches The method of claim 1, further comprising:
generating master data by matching keywords included in a watchlist to the transcripts (131; 139; 146-148 keyword indexing
[0148] Some exemplary topics and phrases may include “Sexual Misconduct”, “Threat Potential”, and “Drug Related.” The lexicons may be modified by the user to create custom and potentially shared templates from which notifications and alerts may be automatically generated when the established criteria is met. Some exemplary applications may include recognizing phrases such as “Call XXX-XXX0XXXX” which may indicate that a prohibited 3 way call may be attempted, and recognizing discussions about the weather that may indicate planning for unrest, disruptions, riots or escapes.).
Regarding claim 5 Horton teaches The method of claim 1,
wherein outputting the indication includes generating the indication based on determining the second association indicates that at least one media file of the plurality of media files includes a speaker speaking the particular keyword
(131; 142: The automatic language detection process provides the ability to offer notification, alerting and routing options based on the spoken languages, such as real time notification when a speaker utters certain words or phrases in a particular language; [0148] Some exemplary topics and phrases may include “Sexual Misconduct”, “Threat Potential”, and “Drug Related.” The lexicons may be modified by the user to create custom and potentially shared templates from which notifications and alerts may be automatically generated when the established criteria is met. Some exemplary applications may include recognizing phrases such as “Call XXX-XXX0XXXX” which may indicate that a prohibited 3 way call may be attempted, and recognizing discussions about the weather that may indicate planning for unrest, disruptions, riots or escapes).
Regarding claim 6 Horton teaches The method of claim 1, wherein the indexed data includes, for each speaker, at least one of:
a gender identifier, or an identifier for the one or more languages detected in the speech in the media file (141-142: language detection).
Regarding claim 7 Horton teaches The method of claim 6, further comprising:
determining, by the computing system, a subset of the one or more speakers, where each speaker of the subset of the one or more speakers is associated with the particular speaker (120: grouping the audio files by speaker; 79-84), and
wherein determining the subset includes:
determining that the speakers of the subset of one or more speakers and the particular speaker speak in a same media file (74: need for the facility is the ability to identify the individuals receiving outgoing calls from within the facility who could possibly be involved in a criminal activity in collusion with the inmates. The current disclosure describes an automated process to create BVPs for the caller as well as the called party in calls originating either inside or outside a facility, and to create BVP's from any source of audio data originating from inside or outside the facility.; 95: An occasion where this is useful is if the inmate is on a watch list and the agency wants to know who is leaving him a voice mail in real time.; 98; 140).
Regarding claim 8 Horton teaches The method of claim 1, where outputting the indication includes generating the indication based on:
identifying, by the computing system, that the first association indicates one or more media files that include speech by the particular speaker (149: request the system return all calls that originate from the secure facility in which the person of interest is a speaker.).
Regarding claim 10 Horton teaches The method of claim 1,
wherein the indexed data includes, for a media file of the plurality of media files, respective identifiers for multiple speakers of the one or more speakers that speak in the media file (140: indexing of audio files…with speakers…speaker identification; 38-39: speaker ID;
[0094] The system may be able to identify one or both speakers during a call, which permits the system to operate in real time and to continue to confirm speaker identification throughout the duration of the call.), and
wherein generating the one or more correlations based on the indexed data comprises identifying an association among the multiple speakers based on the identifiers for the multiple speakers that speak in the media file ([0094] The system may be able to identify one or both speakers during a call, which permits the system to operate in real time and to continue to confirm speaker identification throughout the duration of the call.; 140; 149).
Regarding claim 12 Horton teaches The method of claim 1, wherein processing the plurality of media files to generate the indexed data includes:
processing the plurality of media files to generate, for each media file, respective embeddings for the one or more speakers having speech in the media file (79-84: processing selected segments to create a BVP for the target person); and
matching one or more speaker embeddings included in a watchlist to one or more embeddings from the respective embeddings for the one or more speakers having speech in the media file (20-21; 74; 94-95; 149 – identifying person of interest/those on watchlist).
Regarding claim 13 Horton teaches The method of claim 1, wherein, for each media file of the plurality of media files, one or more keywords from the plurality of keywords are present in the one or more transcripts of speech that each corresponds to the one or more languages detected in the speech in the media file (145-148 transcription; keywords).
Regarding claim 14 Horton teaches The method of claim 1, wherein the indication is a first indication, and wherein the method further comprises:
receiving, by the computing system, an enrollment media file including speech of at least one speaker (65;125; 140 – receiving media files; 79-84: creating a BVP);
processing, by the computing system, the enrollment media file, wherein processing the enrollment media file includes:
extracting an embedding for the at least one speaker from the enrollment media file (77; 79-84 creating a BVP; 140), and
matching the embedding for the at least one speaker to a cluster of one or more clusters, wherein each cluster of the plurality of clusters corresponds to a respective speaker of a plurality of speakers (120 – clustering by speaker based on BVP/ID); and
outputting, by the computing system, based on matching the embedding for the at least one speaker to the cluster, a second indication that includes an indication of a match between the at least one speaker and a speaker of the plurality of speakers
(20; 74; [0094] The system may be able to identify one or both speakers during a call, which permits the system to operate in real time and to continue to confirm speaker identification throughout the duration of the call; 120;
[0149] The disclosed embodiments advantageously facilitates user requests for biometric analysis of an audio file where a speaker is believed to be a current inmate, former inmate, an associate of a current or former inmate, a person promoting criminal activity, or generally a person of interest. For example, a user may upload an audio file which a speaker presents a threat to an emergency operator. The user may then request that the system identify the speaker from the pool of voice prints in the biometric voice print database 508 assigned to some or all speakers that meet certain criteria, such as inmates released over a specified time frame. As another example, the user may provide an audio file including speech from a person of interest and request the system return all calls that originate from the secure facility in which the person of interest is a speaker.).
Regarding claim 15 Horton teaches The method of claim 1, wherein the plurality of media files include media files with audio events, and wherein generating the correlations includes generating correlations that include associations among the audio events (125; 140 – obtaining audio and identifying data/relationships/associations of the content).
Regarding claim 16 Horton teaches A computing system, comprising:
memory; and
one or more programmable processors in communication with the memory and configured to:
obtain, from one or more sources that provide media files over one or more networks, a plurality of media files that each includes speech of one or more speakers;
process the plurality of media files to generate indexed data, wherein the indexed data includes, for each media file of the plurality of media files, a corresponding embedding for each speaker of the one or more speakers identified in the media file and a corresponding transcript and one or more transcripts of speech that each corresponds to one or more languages detected for each language identified in the speech in the media file, and wherein the embeddings within the indexed data are organized independently of known speaker identity;
receive an indication of at least one of a selection of a particular speaker from the one or more speakers or a selection of a particular keyword from a plurality of keywords;
generate one or more correlations based on the indexed data, wherein the one or more correlations include at least one of a first association among the particular speaker and the one or more speakers that corresponds to embeddings within the indexed data to isolate unknown speakers from within corresponding one or more transcripts or a second association among the particular keyword and keywords detected in transcripts as spoken by the one or more speakers that corresponds to the embeddings within the indexed data to isolate the unknown speakers from within the corresponding one or more transcripts; and
output, based on the one or more correlations, an indication regarding the one or more correlations.
Claim recites limitations similar to claim 1 and is rejected for similar rationale and reasoning
Claim 17 recites limitations similar to claim 2 and is rejected for similar rationale and reasoning
Claim 18 recites limitations similar to claim 3 and is rejected for similar rationale and reasoning
Claim 19 recites limitations similar to claim 4 and is rejected for similar rationale and reasoning
Regarding claim 20 Horton teaches Non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to:
obtain, from one or more sources that provide media files over one or more networks, a plurality of media files that each includes speech of one or more speakers;
process the plurality of media files to generate indexed data, wherein the indexed data includes, for each media file of the plurality of media files, a corresponding embedding for each speaker of the one or more speakers identified in the media file and a corresponding transcript and one or more transcripts of speech that each corresponds to one or more languages detected for each language identified in the speech in the media file, and wherein the embeddings within the indexed data are organized independently of known speaker identity;
receive an indication of at least one of a selection of a particular speaker from the one or more speakers or a selection of a particular keyword from a plurality of keywords;
generate one or more correlations based on the indexed data, wherein the one or more correlations include at least one of a first association among the particular speaker and the one or more speakers that corresponds to embeddings within the indexed data to isolate unknown speakers from within corresponding one or more transcripts or a second association among the particular keyword and keywords detected in transcripts as spoken by the one or more speakers that corresponds to the embeddings within the indexed data to isolate the unknown speakers from within the corresponding one or more transcripts; and
output, based on the one or more correlations, an indication regarding the one or more correlations.
Claim recites limitations similar to claim 1 and is rejected for similar rationale and reasoning
Claim Rejections - 35 USC § 103
8. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
9. Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Horton et al (2020/0411013) in view of Mathew (2020/0076845).
Regarding claim 9 Horton does not specifically teach where Mathew (2020/0076845) teaches The method of claim 1, wherein the one or more sources comprise a Clearnet site and a darknet site (22 deep web, dark web, surface web).
It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Mathew for an improved system allowing for additional media to be obtained by crawling in spaces throughout the web.
Horton already teaches accessing media for transcription, identification, and indexing, and one could look to Mathew to provide access to additional sources for an improved system while presenting a reasonable expectation of success, and with the incorporation further improving on the detection and prevention of potential threats.
10. Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Horton et al (2020/0411013) in view of Karas et al (2003/0088397).
Regarding claim 11 Horton teaches The method of claim 1, further comprising: outputting, by the computing system, a graphical user interface (GUI), wherein the GUI includes {one or more visual representations of the} one or more correlations (fig 5; para: 21; 134 user interface; 136; 141; 142; 148 notifications and alerts; 149);
But does not specifically teach where Karas teaches one or more visual representations of the one or more correlations (41; 51 display; to allow a user to navigate and utilize the indexed information stream data; 60; 61; 72)
It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Karas for an improved system to allow user to better observe the correlations in helping to detect and eliminate potential threats.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAUN A ROBERTS whose telephone number is (571)270-7541. The examiner can normally be reached Monday-Friday 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached on 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov.
For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHAUN ROBERTS/Primary Examiner, Art Unit 2655
Prosecution Insights