DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application is being examined under the pre-AIA first to invent provisions.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/11/2024 and 04/23/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
103 Rejection Arguments
Applicant asserts:
Applicant argues, on page 6-7, that the prior art does not teach the amended claims.
Examiner response:
Examiner respectfully disagrees. Applicant’s arguments with respect to claim(s) 1 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Updated 103 rejection is shown below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 and 27 are rejected under 35 U.S.C. 103 as being unpatentable over Zeyu et al; US 20210256978 A1 filed on Feb 13, 2020 (hereinafter “Zeyu”) in view of Haozhe Chen et al; “SPEECH PATTERN BASED BLACK-BOX MODEL WATERMARKING FOR AUTOMATIC SPEECH RECOGNITION” published on May 2, 2022 (hereinafter “Chen”) in further view of Christopher et al; “An Introduction to Information Retrieval” available online May 2022 (hereinafter “Christopher”) in further view of Shaun Jaikarran Bharrat et al; US 20220086175 A1 field on Aug 28, 2021 (hereinafter “Bharrat”)
Regarding Claim 1, Zeyu teaches A method comprising: generating, by an entity, a watermarked training dataset for training [automated speech recognition (ASR) models,] [wherein the watermarked training dataset is proprietary to the entity] (Zeyu Paragraph 0028; "Generally, an audio watermark detector may be trained using any suitable training dataset selected or generated based on the application of interest. For example, to detect doctored single-person speech, a training dataset can be formed using a collection of audio clips of a single person speaking at a time. Generally, the audio clips may be embedded with a particular watermark using a particular embedding technique." Examiner notes that training dataset embedded with watermark (ie watermarked training dataset) is generated/formed for training by an entity (person speaking))
wherein generating the watermarked training dataset includes: obtaining a training dataset that includes unwatermarked speech audio; (Zeyu Paragraph 0028; "a training dataset can be formed using a collection of audio clips of a single person speaking at a time… to train an audio watermark detector, an audio clip may be randomly selected from the collection, and the selected clip may be embedded with a watermark based on a first metric (e.g., 50% of the time)." Examiner notes that obtained/formed training dataset includes unwatermarked audio clips of a single person speaking (ie speech audio); to train the audio watermark detector, the clips are selected and embedded with a watermark suggesting they are unwatermarked prior)
And adding an acoustic watermark feature to the portion of the unwatermarked speech audio to obtain acoustically watermarked speech audio; (Examiner refers to previous mapping to show that an acoustic watermark feature (watermark) is added/embedded to the portion (50% of the time) of the unwatermarked speech audio (audio clip) to obtain acoustically watermarked speech audio)
Adding the acoustically watermarked speech audio to the training dataset to obtain the watermarked training dataset (Zeyu Paragraph 0028; "an audio watermark detector may be trained using any suitable training dataset selected or generated based on the application of interest… a training dataset can be formed using a collection of audio clips of a single person speaking at a time… to train an audio watermark detector, an audio clip may be randomly selected from the collection, and the selected clip may be embedded with a watermark based on a first metric (e.g., 50% of the time)" Examiner notes that acoustically watermarked speech audio (audio clip selected from collection to be embedded watermark) is added/selected for the training data set (suitable training dataset) to obtain the watermarked training dataset)
Zeyu does not teach training dataset for training automated speech recognition (ASR) models,
identifying a portion of the unwatermarked speech audio that corresponds to a target output [token]
and training an ASR model for speech recognition using the watermarked training dataset,
wherein inputting the unwatermarked speech audio into machine learning (ML) models, including the trained ASR model, that have been trained based on the watermarked training dataset elicits a different text output from the trained ML models than inputting the acoustically watermarked speech audio into the trained ML models.
However, Chen does teach training dataset for training automated speech recognition (ASR) models, (Chen Page 3 Paragraph 4; “Then they randomly selects audios from the training set D… to form a subset Dw … then divide Dw into n groups ... After that, generate n speech clips of model owners … and a secret key ... For each input audio x in Dw, trigger audios x can be generated according to Eq. 2.1.1. After that, let T be the target label of each trigger audio (audios added the triggers) to form a trigger set Tw … Mix the trigger set Tw with the corresponding clean audio set Dw to form a fine-tuning set. Fine-tune the model and embed a watermark.” Examiner notes that training dataset (fine-tuning set) is used for training ASR models)
identifying a portion of the unwatermarked speech audio that corresponds to a target output [token] (Chen Page 2 Paragraph 4; “If we only add the trigger to a few frames of audio to generate the trigger audio for CTC-based ASR models, because of the conditional independence, the left frames which without triggers are required to be recognized as the ground-truth label (when the audio frame occurred in the clean audios)” Examiner notes that a portion of the unwatermarked speech audio (audio frame in the clean audios) that corresponds to a target output (ground truth-label) is identified/recognized)
and training an ASR model for speech recognition using the watermarked training dataset, (Chen Page 2 Paragraph 2; “We propose a black-box ASR model watermarking framework by fine-tuning the model on the trigger audios” Chen Page 3 Paragraph 4; “Then they randomly selects audios from the training set D… to form a subset Dw … then divide Dw into n groups ... After that, generate n speech clips of model owners … and a secret key ... For each input audio x in Dw, trigger audios x can be generated according to Eq. 2.1.1. After that, let T be the target label of each trigger audio (audios added the triggers) to form a trigger set Tw … Mix the trigger set Tw with the corresponding clean audio set Dw to form a fine-tuning set. Fine-tune the model and embed a watermark.” Examiner notes that an ASR model is trained/fine-tuned for speech recognition using the watermarked training dataset (fine-tuning set))
wherein inputting the unwatermarked speech audio into machine learning (ML) models, including the trained ASR model, that have been trained based on the watermarked training dataset elicits a different text output from the trained ML models than inputting the acoustically watermarked speech audio into the trained ML models.
(Chen Page 2 Paragraph 4; “If we only add the trigger to a few frames of audio to generate the trigger audio for CTC-based ASR models, because of the conditional independence, the left frames which without triggers are required to be recognized as the ground-truth label (when the audio frame occurred in the clean audios) and be recognized as the trigger label (when the audio frame occurred in the trigger audio)at the same time, which will cause a significant drop in accuracy of the watermarked model.” Examiner notes that wherein inputting the unwatermarked speech audio (audio frame occurred in the clean audios) into the machine learning models including trained ASR model (CTC-based ASR models and watermarked model) elicits a different text output from the trained ML models (ground-truth label vs trigger label) than inputting the acoustically watermarked speech audio (audio frame occurred in the trigger audio) into the trained ML model)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu and Chen. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Chen teaches a black-box model watermarking framework for protecting the IP of ASR models. One of ordinary skill would have motivation to combine Zeyu and Chen to implement aspects of Chen for a robust watermarking scheme with little impact on accuracy “Experiments on the state-of-the-art open-source ASR system DeepSpeech demonstrate the feasibility of the proposed water marking scheme, which is robust against five kinds of attacks and has little impact on accuracy.” (Chen Abstract).
Zeyu in view of Chen does not teach [identifying a portion of the unwatermarked speech audio that corresponds to] a target output token
However, Christopher does teach [identifying a portion of the unwatermarked speech audio that corresponds to] a target output token
(Christopher Page 22 Paragraph 3; "Given a character sequence and a defined document unit, tokenization is the task of chopping it up into pieces, called tokens , perhaps at the same time throwing away certain characters, such as punctuation. Here is an example of tokenization: Input: Friends, Romans, Countrymen, lend me your ears; Output: [Friends], [Romans], [Countrymen], [lend] [me] [your] [ears];" Examiner notes target output is represented in tokens)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu, Chen, and Christopher. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Chen teaches a black-box model watermarking framework for protecting the IP of ASR models. Christopher teaches tokenization. One of ordinary skill would have motivation to combine Zeyu, Chen, and Christopher to use tokenization to add meaningful boundaries to the writing system “Three reasons why this approach is appealing are that an individual Chinese character is more like a syllable than a letter and usually has some semantic content, that most words are short (the commonest length is 2 characters), and that, given the lack of standardization of word breaking in the writing system, it is not always clear where word boundaries should be placed anyway. Even in English, some cases of where to put word boundaries are just orthographic conventions” (Christopher Page 26 Paragraph 1).
Zeyu in view of Chen in further view of Christopher does not teach A method comprising: generating, by an entity, a [watermarked] training dataset [for training automated speech recognition (ASR) models,] wherein the watermarked training dataset is proprietary to the entity
However, Bharrat does teach A method comprising: generating, by an entity, a [watermarked] training dataset [for training automated speech recognition (ASR) models,] wherein the watermarked training dataset is proprietary to the entity (Bharrat Paragraph 0101; “The actual customer data, e.g., call data which is featurized and used in training the synthetic data generator neural network, may, and typically does, include proprietary information such as calling party numbers which are extracted from call data records, session detail records and/or service request detail records.” Examiner notes that a proprietary training data set (actual customer data) is generated/extracted wherein the training data set is proprietary to the entity (customer))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu, Chen, Christopher, and Bharrat. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Chen teaches a black-box model watermarking framework for protecting the IP of ASR models. Christopher teaches tokenization. Bharrat teaches Methods and apparatus for implementing and operating malicious transaction detection systems. One of ordinary skill would have motivation to combine Zeyu, Chen, Christopher, and Bharrat to train ML models with proprietary data to help mitigate, reduce and/or eliminate the harm that can be caused “The present invention also relates to methods, systems, and apparatus for taking actions to mitigate, reduce and/or eliminate the harm that can be caused by communications sessions, e.g., calls, identified as potentially fraudulent, suspicious, and/or malicious.” (Christopher Page 26 Paragraph 1).
Regarding Claim 27, Zeyu teaches A system comprising: a processor; and a memory storing programming instructions for execution by the processor, the programming instructions, upon execution by the processor, causing the system to perform the following operations: (Zeyu Paragraph 0053; “ various functions can be carried out by a processor executing instructions stored in memory. The methods can also be embodied as computer-usable instructions stored on computer storage media.”)
Claim 27 is a system claim of method claim 1 and is accordingly rejected using substantially similar rationale as to that which is set for with respect to claim 1.
Claim(s) 21, 23-26, 28, and 30-33 are rejected under 35 U.S.C. 103 as being unpatentable over Zeyu et al; US 20210256978 A1 filed on Feb 13, 2020 (hereinafter “Zeyu”) in view of Haozhe Chen et al; “SPEECH PATTERN BASED BLACK-BOX MODEL WATERMARKING FOR AUTOMATIC SPEECH RECOGNITION” published on May 2, 2022 (hereinafter “Chen”) in further view of Christopher et al; “An Introduction to Information Retrieval” available online May 2022 (hereinafter “Christopher”) in further view of Shaun Jaikarran Bharrat et al; US 20220086175 A1 field on Aug 28, 2021 (hereinafter “Bharrat”) in further view of Mark William Gerrard; US 20200329327 A1 filed on May 24, 2017 (hereinafter “Gerrard”)
Regarding claim 21, Zeyu does not teach The method of claim 1, wherein the acoustic watermark feature is a frequency tone at or below 200 hertz.
However, Gerrard does teach The method of claim 1, wherein the acoustic watermark feature is a frequency tone at or below 200 hertz. (Gerrard Paragraph 0044; “the system uses a signature audio signal (watermark) to stamp the pre-processed audio file. Comparing an original file to an output audio file can be used to detect if the file has been pre-processed through the presence of the watermark. The watermark can be embodied by putting a signal in the out-of-band portion of the signal with respect to speaker playback. For example by encoding a signature audio signal (e.g., a beep or other sound pattern) in the very low bass region (e.g., below 200 Hz) of the input audio file.” Examiner notes that the acoustic watermark feature (signature audio signal) is a frequency tone at or below 200 hertz (below 200 hz))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu, Chen, Christopher, and Gerard. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Chen teaches a black-box model watermarking framework for protecting the IP of ASR models. Christopher teaches tokenization. Gerard teaches inputting a watermark in an out of band portion of the audio signal. One of ordinary skill would have motivation to combine Zeyu, Chen, Christopher, and Gerard to encode a watermark in an out of band portion of the watermark to make it imperceptible to listeners and maximizing robustness against removal.
Regarding claim 23, Zeyu does not teach The method of claim 1, wherein the acoustic watermark feature is a non-speech signal component.
However, Gerrard does teach The method of claim 1, wherein the acoustic watermark feature is a non-speech signal component. (Gerrard Paragraph 0044; “the system uses a signature audio signal (watermark) to stamp the pre-processed audio file. Comparing an original file to an output audio file can be used to detect if the file has been pre-processed through the presence of the watermark. The watermark can be embodied by putting a signal in the out-of-band portion of the signal with respect to speaker playback. For example by encoding a signature audio signal (e.g., a beep or other sound pattern) in the very low bass region (e.g., below 200 Hz) of the input audio file.” Examiner notes that the acoustic watermark feature is a non-speech signal component (a beep or other sound pattern))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu, Chen, Christopher, and Gerard. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Chen teaches a black-box model watermarking framework for protecting the IP of ASR models. Christopher teaches tokenization. Gerard teaches inputting a watermark in an out of band portion of the audio signal. One of ordinary skill would have motivation to combine Zeyu, Chen, Christopher, and Gerard to encode a watermark in an out of band portion of the watermark to make it imperceptible to listeners and maximizing robustness against removal.
Regarding claim 24, Zeyu does not teach The method of claim 1, wherein the acoustic watermark feature is a noise component.
However, Gerrard does teach The method of claim 1, wherein the acoustic watermark feature is a noise component. (Gerrard Paragraph 0044; “the system uses a signature audio signal (watermark) to stamp the pre-processed audio file. Comparing an original file to an output audio file can be used to detect if the file has been pre-processed through the presence of the watermark. The watermark can be embodied by putting a signal in the out-of-band portion of the signal with respect to speaker playback. For example by encoding a signature audio signal (e.g., a beep or other sound pattern) in the very low bass region (e.g., below 200 Hz) of the input audio file.” Examiner notes that the acoustic watermark feature is a noise component (a beep or other sound pattern))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu, Chen, Christopher, and Gerard. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Chen teaches a black-box model watermarking framework for protecting the IP of ASR models. Christopher teaches tokenization. Gerard teaches inputting a watermark in an out of band portion of the audio signal. One of ordinary skill would have motivation to combine Zeyu, Chen, Christopher, and Gerard to encode a watermark in an out of band portion of the watermark to make it imperceptible to listeners and maximizing robustness against removal.
Regarding claim 25, Zeyu does not teach The method of claim 1, wherein adding the acoustic watermark feature includes adding a noise component.
However, Gerrard does teach The method of claim 1, wherein adding the acoustic watermark feature includes adding a noise component. (Gerrard Paragraph 0044; “the system uses a signature audio signal (watermark) to stamp the pre-processed audio file. Comparing an original file to an output audio file can be used to detect if the file has been pre-processed through the presence of the watermark. The watermark can be embodied by putting a signal in the out-of-band portion of the signal with respect to speaker playback. For example by encoding a signature audio signal (e.g., a beep or other sound pattern) in the very low bass region (e.g., below 200 Hz) of the input audio file.” Examiner notes that adding the acoustic watermark feature (signature audio signal) includes adding/encoding a noise component (beep or other sound pattern))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu, Chen, Christopher, and Gerard. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Chen teaches a black-box model watermarking framework for protecting the IP of ASR models. Christopher teaches tokenization. Gerard teaches inputting a watermark in an out of band portion of the audio signal. One of ordinary skill would have motivation to combine Zeyu, Chen, Christopher, and Gerard to encode a watermark in an out of band portion of the watermark to make it imperceptible to listeners and maximizing robustness against removal.
Regarding claim 26, Zeyu does not teach The method of claim 25, wherein adding the noise component removes an existing noise component from the portion of the unwatermarked speech audio.
However, Gerrard does teach The method of claim 25, wherein adding the noise component removes an existing noise component from the portion of the unwatermarked speech audio. (Gerrard Paragraph 0044; “the system uses a signature audio signal (watermark) to stamp the pre-processed audio file. Comparing an original file to an output audio file can be used to detect if the file has been pre-processed through the presence of the watermark. The watermark can be embodied by putting a signal in the out-of-band portion of the signal with respect to speaker playback. For example by encoding a signature audio signal (e.g., a beep or other sound pattern) in the very low bass region (e.g., below 200 Hz) of the input audio file.” Examiner notes that adding the noise component (beep or other sound pattern) removes an existing noise component (putting a signal in the out-of-band portion is removing/modifying existing noise component to encode/put in the watermark))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu, Chen, Christopher, and Gerard. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Chen teaches a black-box model watermarking framework for protecting the IP of ASR models. Christopher teaches tokenization. Gerard teaches inputting a watermark in an out of band portion of the audio signal. One of ordinary skill would have motivation to combine Zeyu, Chen, Christopher, and Gerard to encode a watermark in an out of band portion of the watermark to make it imperceptible to listeners and maximizing robustness against removal.
Regarding claim 28, claim 28 has similar limitations as of claim 21, except it is a system claim (Zeyu Paragraph 0053), therefore it is rejected under the same rationale as claim 21.
Regarding claim 30, claim 30 has similar limitations as of claim 23, except it is a system claim (Zeyu Paragraph 0053), therefore it is rejected under the same rationale as claim 23.
Regarding claim 31, claim 31 has similar limitations as of claim 24, except it is a system claim (Zeyu Paragraph 0053), therefore it is rejected under the same rationale as claim 24.
Regarding claim 32, claim 32 has similar limitations as of claim 25, except it is a system claim (Zeyu Paragraph 0053), therefore it is rejected under the same rationale as claim 25.
Regarding claim 33, claim 33 has similar limitations as of claim 26, except it is a system claim (Zeyu Paragraph 0053), therefore it is rejected under the same rationale as claim 26.
Claim(s) 34 is rejected under 35 U.S.C. 103 as being unpatentable over Zeyu et al; US 20210256978 A1 filed on Feb 13, 2020 (hereinafter “Zeyu”) in view of Dillon Niederhut; “How to tell if someone trained a model on your data” available online Jul 11, 2022 (hereinafter “Niederhut”) in further view of Haozhe Chen et al; “SPEECH PATTERN BASED BLACK-BOX MODEL WATERMARKING FOR AUTOMATIC SPEECH RECOGNITION” published on May 2, 2022 (hereinafter “Chen”) in further view of Shaun Jaikarran Bharrat et al; US 20220086175 A1 field on Aug 28, 2021 (hereinafter “Bharrat”)
Regarding Claim 34, Zeyu teaches A method comprising: receiving, from an entity, a watermarked training dataset for training automated speech recognition (ASR) models, [wherein the watermarked training dataset is proprietary to the entity, and] (Zeyu Paragraph 0028; "Generally, an audio watermark detector may be trained using any suitable training dataset selected or generated based on the application of interest. For example, to detect doctored single-person speech, a training dataset can be formed using a collection of audio clips of a single person speaking at a time. Generally, the audio clips may be embedded with a particular watermark using a particular embedding technique." Examiner notes that training dataset embedded with watermark (ie watermarked training dataset) is generated/formed for training from an entity (person speaking))
[detecting whether a third-party automated speech recognition (ASR) model has been trained using] watermarked training dataset that includes acoustically watermarked speech audio and unwatermarked speech audio, wherein the acoustically watermarked speech audio is a version of the unwatermarked speech audio that includes an acoustic watermark feature, (Zeyu Paragraph 0028; "a training dataset can be formed using a collection of audio clips of a single person speaking at a time… to train an audio watermark detector, an audio clip may be randomly selected from the collection, and the selected clip may be embedded with a watermark based on a first metric (e.g., 50% of the time)." Examiner notes that training dataset (ie watermarked training dataset) includes acoustically watermarked speech audio (selected clip that is embedded with a watermark) and an unwatermarked speech audio (collection of audio clips of a single person speaking), wherein the acoustically watermarked speech audio is a version of the unwatermarked speech audio that includes an acoustic watermark feature (the audio clip/unwatermarked speech audio is chosen to encoded into a version that is acoustically watermarked))
Zeyu does not teach detecting whether a third-party automated speech recognition (ASR) model has been trained using a watermarked training dataset
And wherein detecting whether the third-party ASR model has been trained using the watermarked training dataset comprises: inputting the unwatermarked speech audio into the third-party ASR model to elicit a first text output;
inputting the acoustically watermarked speech audio into the third-party ASR model to elicit a second text output;
and determining that the third-party [ASR] model has been trained using the watermarked training dataset based on comparing the second text output of the third-party [ASR] model with the first text output of the third-party [ASR] model.
wherein inputting the unwatermarked speech audio into machine learning (ML) models that have been trained based on the watermarked training dataset elicits a different text output from the trained ML models than inputting the acoustically watermarked speech audio into the trained ML models.
However, Niederhut does teach detecting whether a third-party [automated speech recognition (ASR)] model has been trained using a watermarked training dataset
(Niederhut Paragraph 18; “If the classification confidence is higher on the watermarked version, this is a signal to you that the model was trained on your poisoned data!” Examiner notes that a third-party model is detected to be using a watermarked training dataset (poisoned data))
And wherein detecting whether the third-party [ASR] model has been trained using the watermarked training dataset comprises: inputting the unwatermarked [speech audio] into the third-party [ASR] model to elicit a first text output; (Niederhut Paragraph 18; “After an image recognition model has been trained on these watermarked images, you can give it two copies of a new image -- one watermarked, and one not. If the classification confidence is higher on the watermarked version, this is a signal to you that the model was trained on your poisoned data!” Examiner notes that an unwatermarked image is input into the third-party model (image recognition model) to elicit a first text output (classification confidence))
inputting the [acoustically] watermarked [speech audio] into the third-party [ASR] model to elicit a second text output; (Examiner refers to previous mapping to show that the watermarked image is input into the third-party model to elicit a second text output (classification confidence))
and determining that the third-party [ASR] model has been trained using the watermarked training dataset based on comparing the second text output of the third-party [ASR] model with the first text output of the third-party [ASR] model. (Niederhut Paragraph 18; “After an image recognition model has been trained on these watermarked images, you can give it two copies of a new image -- one watermarked, and one not. If the classification confidence is higher on the watermarked version, this is a signal to you that the model was trained on your poisoned data!” Examiner notes that determining that the third-party model (image recognition model) has been trained using the watermarked training dataset (watermarked/poisoned data) based on comparing the second text output of the third-party model with the first text output of the third-party model (classification confidence of both inputs is compared to determine if model was trained on poisoned data))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu and Niederhut. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Niederhut teaches how to tell if someone trained a model on your data. One of ordinary skill would have motivation to combine Zeyu and Niederhut to easily identify whether your data has been used to train a model without attackers being aware of the poisoned data “They find that with just 1% of training data watermarked, they can tell with high confidence that their watermarked data was used to train the model.” (Niederhut Paragraph 18).
Zeyu in view of Niederhut does not teach ASR model
Speech audio data
wherein inputting the unwatermarked speech audio into machine learning (ML) models, including the trained ASR model, that have been trained based on the watermarked training dataset elicits a different text output from the trained ML models than inputting the acoustically watermarked speech audio into the trained ML models.
However, Chen does teach ASR model (Chen Page 2 Paragraph 2; “We propose a black-box ASR model watermarking framework by fine-tuning the model on the trigger audios, which generated by spreading speech clips over the clean audio and replacing the corresponding labels with the steganography texts.”)
Speech audio data (Examiner refers to previous mapping to show that speech audio data is speech clips)
wherein inputting the unwatermarked speech audio into machine learning (ML) models, including the trained ASR model, that have been trained based on the watermarked training dataset elicits a different text output from the trained ML models than inputting the acoustically watermarked speech audio into the trained ML models.
(Chen Page 2 Paragraph 4; “If we only add the trigger to a few frames of audio to generate the trigger audio for CTC-based ASR models, because of the conditional independence, the left frames which without triggers are required to be recognized as the ground-truth label (when the audio frame occurred in the clean audios) and be recognized as the trigger label (when the audio frame occurred in the trigger audio)at the same time, which will cause a significant drop in accuracy of the watermarked model.” Examiner notes that wherein inputting the unwatermarked speech audio (audio frame occurred in the clean audios) into the machine learning models including trained ASR model (CTC-based ASR models and watermarked model) elicits a different text output from the trained ML models (ground-truth label vs trigger label) than inputting the acoustically watermarked speech audio (audio frame occurred in the trigger audio) into the trained ML model)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu, Niederhut, and Chen. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Niederhut teaches how to tell if someone trained a model on your data. Chen teaches a black-box model watermarking framework for protecting the IP of ASR models. One of ordinary skill would have motivation to combine Zeyu, Niederhut, and Chen to implement aspects of Chen for a robust watermarking scheme with little impact on accuracy “Experiments on the state-of-the-art open-source ASR system DeepSpeech demonstrate the feasibility of the proposed water marking scheme, which is robust against five kinds of attacks and has little impact on accuracy.” (Chen Abstract).
Zeyu in view of Neiderhut in further view of Chen does not teach A method comprising: generating, by an entity, a [watermarked] training dataset [for training automated speech recognition (ASR) models,] wherein the watermarked training dataset is proprietary to the entity
However, Bharrat does teach A method comprising: generating, by an entity, a [watermarked] training dataset [for training automated speech recognition (ASR) models,] wherein the watermarked training dataset is proprietary to the entity (Bharrat Paragraph 0101; “The actual customer data, e.g., call data which is featurized and used in training the synthetic data generator neural network, may, and typically does, include proprietary information such as calling party numbers which are extracted from call data records, session detail records and/or service request detail records.” Examiner notes that a proprietary training data set (actual customer data) is generated/extracted wherein the training data set is proprietary to the entity (customer))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu, Niederhut, Chen, and Bharrat. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Niederhut teaches how to tell if someone trained a model on your data. Chen teaches a black-box model watermarking framework for protecting the IP of ASR models. Bharrat teaches Methods and apparatus for implementing and operating malicious transaction detection systems. One of ordinary skill would have motivation to combine Zeyu, Niederhut, Chen, and Bharrat to train ML models with proprietary data to help mitigate, reduce and/or eliminate the harm that can be caused “The present invention also relates to methods, systems, and apparatus for taking actions to mitigate, reduce and/or eliminate the harm that can be caused by communications sessions, e.g., calls, identified as potentially fraudulent, suspicious, and/or malicious.” (Christopher Page 26 Paragraph 1).
Claim(s) 35, and 37-38 are rejected under 35 U.S.C. 103 as being unpatentable over Zeyu et al; US 20210256978 A1 filed on Feb 13, 2020 (hereinafter “Zeyu”) in view of Dillon Niederhut; “How to tell if someone trained a model on your data” available online Jul 11, 2022 (hereinafter “Niederhut”) in further view of Haozhe Chen et al; “SPEECH PATTERN BASED BLACK-BOX MODEL WATERMARKING FOR AUTOMATIC SPEECH RECOGNITION” published on May 2, 2022 (hereinafter “Chen”) in further view of Shaun Jaikarran Bharrat et al; US 20220086175 A1 field on Aug 28, 2021 (hereinafter “Bharrat”) in further view of Mark William Gerrard; US 20200329327 A1 filed on May 24, 2017 (hereinafter “Gerrard”)
Regarding claim 35, Zeyu does not teach The method of claim 34, wherein the acoustic watermark feature is a frequency tone at or below 200 hertz.
However, Gerrard does The method of claim 34, wherein the acoustic watermark feature is a frequency tone at or below 200 hertz. (Gerrard Paragraph 0044; “the system uses a signature audio signal (watermark) to stamp the pre-processed audio file. Comparing an original file to an output audio file can be used to detect if the file has been pre-processed through the presence of the watermark. The watermark can be embodied by putting a signal in the out-of-band portion of the signal with respect to speaker playback. For example by encoding a signature audio signal (e.g., a beep or other sound pattern) in the very low bass region (e.g., below 200 Hz) of the input audio file.” Examiner notes that the acoustic watermark feature (signature audio signal) is a frequency tone at or below 200 hertz (below 200 hz))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu, Niederhut, Chen, and Gerrard. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Niederhut teaches how to tell if someone trained a model on your data. Chen teaches a black-box model watermarking framework for protecting the IP of ASR models. Gerard teaches inputting a watermark in an out of band portion of the audio signal. One of ordinary skill would have motivation to combine Zeyu, Niederhut, Chen, and Gerrard to encode a watermark in an out of band portion of the watermark to make it imperceptible to listeners and maximizing robustness against removal.
Regarding claim 37, Zeyu does not teach The method of claim 34, wherein the acoustic watermark feature is a non-speech signal component.
However, Gerrard does teach The method of claim 34, wherein the acoustic watermark feature is a non-speech signal component. (Gerrard Paragraph 0044; “the system uses a signature audio signal (watermark) to stamp the pre-processed audio file. Comparing an original file to an output audio file can be used to detect if the file has been pre-processed through the presence of the watermark. The watermark can be embodied by putting a signal in the out-of-band portion of the signal with respect to speaker playback. For example by encoding a signature audio signal (e.g., a beep or other sound pattern) in the very low bass region (e.g., below 200 Hz) of the input audio file.” Examiner notes that the acoustic watermark feature is a non-speech signal component (a beep or other sound pattern))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu, Niederhut, Chen, and Gerrard. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Niederhut teaches how to tell if someone trained a model on your data. Chen teaches a black-box model watermarking framework for protecting the IP of ASR models. Gerard teaches inputting a watermark in an out of band portion of the audio signal. One of ordinary skill would have motivation to combine Zeyu, Niederhut, Chen, and Gerrard to encode a watermark in an out of band portion of the watermark to make it imperceptible to listeners and maximizing robustness against removal.
Regarding claim 38, Zeyu does not teach The method of claim 37, wherein the acoustic watermark feature that is a noise component.
However, Gerrard does teach The method of claim 37, wherein the acoustic watermark feature that is a noise component. (Gerrard Paragraph 0044; “the system uses a signature audio signal (watermark) to stamp the pre-processed audio file. Comparing an original file to an output audio file can be used to detect if the file has been pre-processed through the presence of the watermark. The watermark can be embodied by putting a signal in the out-of-band portion of the signal with respect to speaker playback. For example by encoding a signature audio signal (e.g., a beep or other sound pattern) in the very low bass region (e.g., below 200 Hz) of the input audio file.” Examiner notes that the acoustic watermark feature is a noise component (a beep or other sound pattern))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Zeyu, Niederhut, Chen, and Gerrard. Zeyu teaches a method for secure audio watermarking and audio authenticity verification. Niederhut teaches how to tell if someone trained a model on your data. Chen teaches a black-box model watermarking framework for protecting the IP of ASR models. Gerard teaches inputting a watermark in an out of band portion of the audio signal. One of ordinary skill would have motivation to combine Zeyu, Niederhut, Chen, and Gerrard to encode a watermark in an out of band portion of the watermark to make it imperceptible to listeners and maximizing robustness against removal.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL DUC TRAN whose telephone number is (571)272-6870. The examiner can normally be reached Mon-Fri 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.D.T./Examiner, Art Unit 2147
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights