Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on June 02, 2025 and October 03, 2025 are being considered by the examiner.
Claim Objections
The following claims are objected to because of the following informalities:
Claim 2 states “aspatial” which should be corrected to “a spatial”.
Claim 8, 19, 20 state “the external audio AI system” which should be corrected to “the external AI audio system”.
Claim 12 states “the external audio AI” which should be corrected to “the external AI audio system”.
Claim 13 states “the AI audio system”, where it should describe “the external AI audio system”.
Additionally, claim 13 states “aspatial” which should be corrected to “a spatial”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-11 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "the external AI audio system" in lines 6 and 9. There is insufficient antecedent basis for this limitation in the claim. Regarding Claim 1 describes, “an artificial intelligence, AI, audio system” in line 1, and later describes “the external AI audio system” in lines 6 and 9. “The external AI audio system” lacks proper antecedent basis, as it isn’t clear as to whether this is the initial AI audio system or a second external AI audio system.
Claim 2 recites the limitation “the AI audio system”, where it should describe “the external AI audio system” so there isn’t ambiguity regarding which of the two possible AI systems from claim 1 is being referenced.
Claim 3 recites “wherein generating for enhancing an external artificial intelligence, AI, audio system comprises…” This preamble doesn’t clearly link to a limitation in claim 1. Claim 1 describes a method for enhancing an AI audio system, where that method includes a step of generating at least one an alternative audio test case, but neither of those sections match up with the preamble of claim 3.
Claim 9 recites ”the system”. There is insufficient antecedent basis for this limitation in the claim. The independent claim identifies both the claimed method for enhancing an AI audio system and an external AI audio system. It is unclear which system controls the parameter.
Claims 4-8, 10, and 11 are rejected due to inheriting the indefiniteness issue from claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim [ 1, 2, 3, 12, 13, 14, 20] are rejected under 35 U.S.C. 103 as being unpatentable over
Rajan (Rajan, Sai Sathiesh, Sakshi Udeshi, and Sudipta Chattopadhyay. "Aequevox: Automated fairness testing of speech recognition systems." International Conference on Fundamental Approaches to Software Engineering. Cham: Springer International Publishing, 2022., hereinafter Rajan) in view of Wang (Wang, Yuancheng, et al. "Audit: Audio editing by following instructions with latent diffusion models." Advances in Neural Information Processing Systems 36 (2023): 71340-71357., hereinafter Wang).
Regarding claim 1,
Rajan teaches
A computer implemented method for enhancing an artificial intelligence, AI, audio system, the method comprising:
[Page 6, Example "AequeVox uses two ASR systems and obtains the transcript of this speech.'];
[Page 10, 4. Datasets and Experimental Setup "We evaluate AequeVox on three commercial ASR systems from Google Cloud Platform (GCP), IBM Cloud, and Microsoft Azure.
We use the standard models for GCP and Azure, and the Broadband Model for IBM. In all three cases, the audio samples were identically encoded as .wav files using Linear 16 encoding." where AI audio system is mapped to Google Cloud Platform (GCP), IBM Cloud, and Microsoft Azure. ]
[Page 3, Introduction "This information can be further used to improve the ASR system."]
receiving at least one audio test case, the at least one test case comprising
[Page 8, System Overview "AequeVox achieves this by taking a dataset of speech which contains data from two or more different groups (e.g. male and female speakers, Native English and Non-native English speakers) and modifies these speech snippets through a set of transformations (MT)." where dataset of speech are the test cases]
generating at least one an alternative audio test case, wherein generating
[Page 8, System Overview " ..and modifies these speech snippets through a set of transformations (MT)." where dataset of speech are the test cases and these are modified]
[Page 8, System Overview "These are then divided into base group transformed speech(GRTB) and the transformed speech for other groups (GRTk j k 2 f1; 2; ng)."]
[Page 7, Metamorphic Transformations of Sound "AequeVox uses the addition of noise (Figure 2 (b)), amplitude modification(Figure 2 (c)), frequency modification (Figure 2 (d)), amplitude clipping
(Figure 2 (e)), frame drops (Figure 2 (f)), low-pass filters (Figure 2 (g)),and high-pass filters (Figure 2 (h)) as metamorphic speech transformations." where modification of the speech is done. ]
challenging the external AI audio system with the at least one audio test case, and with the at least one alternative audio test case to determine a set of results; and
[Page 6, Example : "AequeVox uses two ASR systems and obtains the transcript of this speech."]
[Page 6, Example : "AequeVox then simulates a noisy environment by adding noise to the speech and obtains the transcript of this transformed speech"];
[Page 8, System Overview "..between the outputs of the two ASR systems is captured by dB and dTB for the original and transformed speech respectively"];
[Page 10, 4. Datasets and Experimental Setup "ASR Systems under Test: We evaluate AequeVox on three commercial ASR systems from Google Cloud Platform (GCP), IBM Cloud, and Microsoft Azure."];
comparing the results, using a distance function, to determine if the respective modification causes a problem in the external AI audio system.
[Page 6, Example : "AequeVox then employs differential testing to find the word-level levenshtein distance [31] between these two sets of transcripts"]
[Page 8, System Overview "..,we compute the difference between the word-level levenshtein distance for the original and transformed speech."]
[Page 8, System Overview "We compare these metrics and if DB exceeds Dk by some threshold , we classify this as an error for the base group (GRB) and more specifically a violation of fairness (see Figure 3)."]
However, Rajan does not teach
audio test case comprising a set of sound components and generating comprises using a generative AI model
But Wang teaches
set of sound components
[Page 4, Section 3.2 Generating Triplet Training Data for Each Editing Task "Adding We randomly select two audio clips A and B from the text-audio datasets, then combine A and B to get a new audio clip C."];
[Page 5, Section 3.2 Generating Triplet Training Data for Each Editing Task "Dropping We use C as the input audio and A as the output audio"].
generating comprises using a generative AI model
[Page 2, I Introduction "As shown in Figure 1, AUDIT is a latent diffusion model which takes the audio to be edited and the editing instruction as conditions and generates the edited audio as output."]
[Page 4, Latent Diffusion Model "Our text-guided audio editing model can be seen as a conditional latent diffusion model that takes the latent representation zin of the input (to be edited) mel-spectrogram
xin and text embeddings as conditions."]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Rajan with Wang to generate the modified audio test cases, in order to provide more realistic and diverse audio modifications while preserving portions not intended to be changed, thereby improving Rajan's ability to identify errors and weakness in the tested ASR system.
Regarding claim 2, rejection of claim 1 is incorporated.
Rajan teaches
The method of claim 1, wherein the AI audio system is one of a list, the list comprising: a speech-to-text system; a voice recognition system; a music recognition system; a text-to-speech system; and aspatial audio processing system.
[Page 6, Example : "AequeVox uses two ASR systems and obtains the transcript of this speech."]
[Page 6, Example : "AequeVox then simulates a noisy environment by adding noise to the speech and obtains the transcript of this transformed speech."];
[Page 10, 4. Datasets and Experimental Setup "ASR Systems under Test: We evaluate AequeVox on three commercial ASR systems from Google Cloud Platform (GCP), IBM Cloud, and Microsoft Azure."];
Regarding claim 3, rejection of claim 1 is incorporated.
Rajan teaches
The method of claim 1, wherein generating for enhancing an external artificial intelligence, AI,
audio system comprises generating a further one of the alternative audio test cases, the further one of the alternative audio test cases comprising a second modification.
[Page 8, System Overview : In our implementation we use eight distinct metamorphic transformations as (see Figure 2)."]
[Page 8, System Overview : "AequeVox achieves this by taking a dataset of speech which contains data from two or more different groups (e.g. male and female speakers, Native English and Non-native
English speakers) and modifies these speech snippets through a set of transformations (MT)."];
[Page 7, Metamorphic Transformations of Sound "AequeVox uses the addition of noise (Figure 2 (b)), amplitude modification (Figure 2 (c)), frequency modification (Figure 2 (d)), amplitude clipping
(Figure 2 (e)), frame drops (Figure 2 (f)), low-pass filters (Figure 2 (g)), and high-pass filters (Figure 2 (h)) as metamorphic speech transformations. We choose these transformations because they are the most common distortions for sound in various environments [2]." where multiple transformations maps to generating alternative audio test with an second modification ].
[Page 6, Algorithm 1 AequeVox Fairness Testing : "for do"
PNG
media_image1.png
91
592
media_image1.png
Greyscale
"where "for do" implies repeats the transformation]
[Page 3, "As a byproduct of our AequeVox framework, we highlight words that contribute to errors by comparing the word counts from the original speech. This information can be further used to improve the ASR system."]
Regarding claim 12,
Rajan teaches
A system for enhancing an external artificial intelligence, AI, audio system, the system comprising: a memory; at least one processor in communication with memory; and program instructions executable by one or more processor via the memory to perform a method comprising:
[Abstract “Automatic Speech Recognition (ASR) systems have become ubiquitous. They can be found in a variety of form factors and are increasingly important in our daily lives. As such, ensuring that these systems are equitable to different subgroups of the population is crucial. In this paper, we introduce, AequeVox, an automated testing framework for evaluating the fairness of ASR systems. AequeVox simulates different environments to assess the effectiveness of ASR systems for different
populations. In addition, we investigate whether the chosen simulations are comprehensible to humans. We further propose a fault localization technique capable of identifying words that are not robust to these varying environments. Both components of AequeVox are able to operate in the absence of ground truth data.”]
[Page 6, Example "AequeVox uses two ASR systems and obtains the transcript of this speech.'];
[Page 10, 4. Datasets and Experimental Setup "We evaluate AequeVox on three commercial ASR systems from Google Cloud Platform (GCP), IBM Cloud, and Microsoft Azure.
We use the standard models for GCP and Azure, and the Broadband Model for IBM. In all three cases, the audio samples were identically encoded as .wav files using Linear 16 encoding." where AI audio system is mapped to Google Cloud Platform (GCP), IBM Cloud, and Microsoft Azure. ]
[Page 3, Introduction "This information can be further used to improve the ASR system."]
receiving at least one audio test case, the at least one test case comprising
[Page 8, System Overview "AequeVox achieves this by taking a dataset of speech which contains data from two or more different groups (e.g. male and female speakers, Native English and Non-native English speakers) and modifies these speech snippets through a set of transformations (MT)." where dataset of speech are the test cases]
generating at least one an alternative audio test case, wherein generating
[Page 8, System Overview " ..and modifies these speech snippets through a set of transformations (MT)." where dataset of speech are the test cases and these are modified]
[Page 8, System Overview "These are then divided into base group transformed speech(GRTB) and the transformed speech for other groups (GRTk j k 2 f1; 2; ng)."]
[Page 7, Metamorphic Transformations of Sound "AequeVox uses the addition of noise (Figure 2 (b)), amplitude modification(Figure 2 (c)), frequency modification (Figure 2 (d)), amplitude clipping
(Figure 2 (e)), frame drops (Figure 2 (f)), low-pass filters (Figure 2 (g)),and high-pass filters (Figure 2 (h)) as metamorphic speech transformations." where modification of the speech is done. ]
challenging the external AI audio system with the at least one audio test case, and with the at least one alternative audio test case to determine a set of results; and
[Page 6, Example : "AequeVox uses two ASR systems and obtains the transcript of this speech."]
[Page 6, Example : "AequeVox then simulates a noisy environment by adding noise to the speech and obtains the transcript of this transformed speech"];
[Page 8, System Overview "..between the outputs of the two ASR systems is captured by dB and dTB for the original and transformed speech respectively"];
[Page 10, 4. Datasets and Experimental Setup "ASR Systems under Test: We evaluate AequeVox on three commercial ASR systems from Google Cloud Platform (GCP), IBM Cloud, and Microsoft Azure."];
comparing the results, using a distance function, to determine if the respective modification causes a problem in the external AI audio system.
[Page 6, Example : "AequeVox then employs differential testing to find the word-level levenshtein distance [31] between these two sets of transcripts"]
[Page 8, System Overview "..,we compute the difference between the word-level levenshtein distance for the original and transformed speech."]
[Page 8, System Overview "We compare these metrics and if DB exceeds Dk by some threshold , we classify this as an error for the base group (GRB) and more specifically a violation of fairness (see Figure 3)."]
However, Rajan does not teach
audio test case comprising a set of sound components and generating comprises using a generative AI model
But Wang teaches
set of sound components
[Page 4, Section 3.2 Generating Triplet Training Data for Each Editing Task "Adding We randomly select two audio clips A and B from the text-audio datasets, then combine A and B to get a new audio clip C."];
[Page 5, Section 3.2 Generating Triplet Training Data for Each Editing Task "Dropping We use C as the input audio and A as the output audio"].
generating comprises using a generative AI model
[Page 2, I Introduction "As shown in Figure 1, AUDIT is a latent diffusion model which takes the audio to be edited and the editing instruction as conditions and generates the edited audio as output."]
[Page 4, Latent Diffusion Model "Our text-guided audio editing model can be seen as a conditional latent diffusion model that takes the latent representation zin of the input (to be edited) mel-spectrogram
xin and text embeddings as conditions."]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Rajan with Wang to generate the modified audio test cases, in order to provide more realistic and diverse audio modifications while preserving portions not intended to be changed, thereby improving Rajan's ability to identify errors and weakness in the tested ASR system.
Regarding claim 13, rejection of claim 12 is incorporated.
Rajan in view of Wang does teach the system of claim 12, wherein the AI audio system is one of a list, the list comprising: a speech-to-text system; a voice recognition system; a music recognition system; a text-to-speech system; and aspatial audio processing system.
Claim 13 is rejected for the same reason as claim 2.
Regarding claim 14, rejection of claim 12 is incorporated.
Rajan in view of Wang does teach The system of claim 12, wherein generating at least one
alternative audio test case comprises generating a further one of the alternative audio test cases, the further one of the alternative audio test cases comprising a second modification.
Claim 14 is rejected for the same reason as claim 3.
Regarding claim 20,
Rajan teaches
A computer program product for enhancing an external artificial intelligence, AI, audio system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method comprising:
[Page 18, 7 Related Work “Unlike AequeVox, recent works on fairness testing have focused on credit
rating [18,52,5,61,45,47,46,44], computer vision [13,7] or NLP systems [36,48].”];
[Page 6, Example "AequeVox uses two ASR systems and obtains the transcript of this speech.'];
[Page 10, 4. Datasets and Experimental Setup "We evaluate AequeVox on three commercial ASR systems from Google Cloud Platform (GCP), IBM Cloud, and Microsoft Azure.]
We use the standard models for GCP and Azure, and the Broadband Model for IBM. In all three cases, the audio samples were identically encoded as .wav files using Linear 16 encoding." where AI audio system is mapped to Google Cloud Platform (GCP), IBM Cloud, and Microsoft Azure. ]
[Page 3, Introduction "This information can be further used to improve the ASR system."]
receiving at least one audio test case, the at least one test case comprising
[Page 8, System Overview "AequeVox achieves this by taking a dataset of speech which contains data from two or more different groups (e.g. male and female speakers, Native English and Non-native English speakers) and modifies these speech snippets through a set of transformations (MT)." where dataset of speech are the test cases]
generating at least one an alternative audio test case, wherein generating
[Page 8, System Overview " ..and modifies these speech snippets through a set of transformations (MT)." where dataset of speech are the test cases and these are modified]
[Page 8, System Overview "These are then divided into base group transformed speech(GRTB) and the transformed speech for other groups (GRTk j k 2 f1; 2; ng)."]
[Page 7, Metamorphic Transformations of Sound "AequeVox uses the addition of noise (Figure 2 (b)), amplitude modification(Figure 2 (c)), frequency modification (Figure 2 (d)), amplitude clipping
(Figure 2 (e)), frame drops (Figure 2 (f)), low-pass filters (Figure 2 (g)),and high-pass filters (Figure 2 (h)) as metamorphic speech transformations." where modification of the speech is done. ]
challenging the external AI audio system with the at least one audio test case, and with the at least one alternative audio test case to determine a set of results; and
[Page 6, Example : "AequeVox uses two ASR systems and obtains the transcript of this speech."]
[Page 6, Example : "AequeVox then simulates a noisy environment by adding noise to the speech and obtains the transcript of this transformed speech"];
[Page 8, System Overview "..between the outputs of the two ASR systems is captured by dB and dTB for the original and transformed speech respectively"];
[Page 10, 4. Datasets and Experimental Setup "ASR Systems under Test: We evaluate AequeVox on three commercial ASR systems from Google Cloud Platform (GCP), IBM Cloud, and Microsoft Azure."];
comparing the results, using a distance function, to determine if the respective modification causes a problem in the external AI audio system.
[Page 6, Example : "AequeVox then employs differential testing to find the word-level levenshtein distance [31] between these two sets of transcripts"]
[Page 8, System Overview "..,we compute the difference between the word-level levenshtein distance for the original and transformed speech."]
[Page 8, System Overview "We compare these metrics and if DB exceeds Dk by some threshold , we classify this as an error for the base group (GRB) and more specifically a violation of fairness (see Figure 3)."]
However, Rajan does not teach
audio test case comprising a set of sound components and generating comprises using a generative AI model
But Wang teaches
set of sound components
[Page 4, Section 3.2 Generating Triplet Training Data for Each Editing Task "Adding We randomly select two audio clips A and B from the text-audio datasets, then combine A and B to get a new audio clip C."];
[Page 5, Section 3.2 Generating Triplet Training Data for Each Editing Task "Dropping We use C as the input audio and A as the output audio"].
generating comprises using a generative AI model
[Page 2, I Introduction "As shown in Figure 1, AUDIT is a latent diffusion model which takes the audio to be edited and the editing instruction as conditions and generates the edited audio as output."]
[Page 4, Latent Diffusion Model "Our text-guided audio editing model can be seen as a conditional latent diffusion model that takes the latent representation zin of the input (to be edited) mel-spectrogram
xin and text embeddings as conditions."]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Rajan with Wang to generate the modified audio test cases, in order to provide more realistic and diverse audio modifications while preserving portions not intended to be changed, thereby improving Rajan's ability to identify errors and weakness in the tested ASR system.
Claim [4, 5, 6, 7, 9, 11, 15, 16, 17, 18] are rejected under 35 U.S.C. 103 as being unpatentable over Rajan (Rajan, Sai Sathiesh, Sakshi Udeshi, and Sudipta Chattopadhyay. "Aequevox: Automated fairness testing of speech recognition systems." International Conference on Fundamental Approaches to Software Engineering. Cham: Springer International Publishing, 2022., hereinafter Rajan) in view of Wang (Wang, Yuancheng, et al. "Audit: Audio editing by following instructions with latent diffusion models." Advances in Neural Information Processing Systems 36 (2023): 71340-71357., hereinafter Wang) and in further view of Du (Du, Xiaoning, et al. "Deepcruiser: Automated guided testing for stateful deep learning systems." arXiv preprint arXiv:1812.05339 (2018).)
Regarding claim 4, rejection of claim 1 in incorporated.
Rajan teaches the method of claim 1, wherein generating the at least one alternative audio test
case comprises
[Page 8, System Overview : In our implementation we use eight distinct metamorphic transformations as (see Figure 2)."]
[Page 8, System Overview : "AequeVox achieves this by taking a dataset of speech which contains data from two or more different groups (e.g. male and female speakers, Native English and Non-native
English speakers) and modifies these speech snippets through a set of transformations (MT)."];
[Page 7, Metamorphic Transformations of Sound "AequeVox uses the addition of noise (Figure 2 (b)), amplitude modification (Figure 2 (c)), frequency modification (Figure 2 (d)), amplitude clipping
(Figure 2 (e)), frame drops (Figure 2 (f)), low-pass filters (Figure 2 (g)), and high-pass filters (Figure 2 (h)) as metamorphic speech transformations. We
choose these transformations because they are the most common distortions for sound in various environments [2]." where multiple transformations maps to generating alternative audio test with an second modification ].
[Page 6, Algorithm 1 AequeVox Fairness Testing : "for do"
PNG
media_image1.png
91
592
media_image1.png
Greyscale
"where "for do" implies repeats the transformation]
However, Rajan in view of Wang does not teach generating a set of iterative alternative audio test cases
But Du teaches generating a set of iterative alternative audio test cases
[Page 3, Introduction "In this paper, we focus on the ASR domain4, and incorporate 8 metamorphic transformations to generate new audio test inputs."]
[Page 7 Section 3 The Overview of Coverage-guided Testing Framework DeepCruiser "With a chosen coverage criterion, the testing process starts with a set of initial seeds (some audios under practical ASR testing
scenarios), namely the seed pool. Then in each iteration, we select an audio based on heuristics and generate a mutant from it by applying certain transformations. If the
generated mutant triggers some defects, e.g., the transcription shows a large difference from the expectation, it is labeled as a failing test. Otherwise, we check whether the
mutant improves the test coverage with respect to the chosen criteria and include it into the seed pool if so.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Rajan in view of Wang with Du in which generated audio mutants are returned to a seed pool and selected for further transformation, in order to systematically generate a diverse set of progressively modified audio test cases and thereby improve the detection of errors and weakness in the test ASR system.
Regarding claim 5, rejection of claim 4 in incorporated.
Rajan teaches in response to receiving the at least one audio test case,
and wherein generating the
[Page 8, System Overview : "AequeVox achieves this by taking a dataset of speech which contains data from two or more different groups (e.g. male and female speakers, Native English and Non-native
English speakers) and modifies these speech snippets through a set of transformations (MT)."];
[Page 7, Metamorphic Transformations of Sound "AequeVox uses the addition of noise (Figure 2 (b)), amplitude modification (Figure 2 (c)), frequency modification (Figure 2 (d)), amplitude clipping
(Figure 2 (e)), frame drops (Figure 2 (f)), low-pass filters (Figure 2 (g)), and high-pass filters (Figure 2 (h)) as metamorphic speech transformations. We choose these transformations because they are the most common distortions for sound in various environments [2]." where multiple transformations maps to generating alternative audio test with an second modification ].
[Page 6, Algorithm 1 AequeVox Fairness Testing : "for do"
PNG
media_image1.png
91
592
media_image1.png
Greyscale
"where "for do" implies repeats the transformation]
Rajan in view of Wang does not teach
creating a trajectory
or each increment on the trajectory, determining a set of operations mnemonics to apply;
generating a set of iterative alternative audio test cases
But Du teaches
creating a trajectory
[Page 14, 6.2 Coverage-Guided Testing "The inputs of DeepCruiser include initial seeds I and the RNN-based deep learning system R."]
[Page 13, 6.1 Metamorphic Transformations of Audio Signals "…when an audio input a0 is processed with a sequence of transformations (e.g.,a0 ,! a1; : : : ; ,! ai), there is a higher chance that the audio cannot be well recognized by human due to accumulated distortions."]
or each increment on the trajectory, determining a set of operations mnemonics to apply;
[Page 12, 6.1 Metamorphic Transformations of Audio Signals "Inspired by these practical scenarios, we derive a set of transformation operators to mimic the environment interference.
Overall, they can be classified into four categories:
– Volume-related transformations (VRT): ChangeVolume, LowPassFilter, HighPassFilter.
– Speed-related transformations (SRT): PitchShift, ChangeSpeed.
– Clearness-related transformations (CRT): AddWhiteNoise, ExtractHarmonic
– Unaffected transformations (UAT): DRC, Trim."]
[Page 14, 6.2 Coverage-guided Testing "Based on the already selected transformations of a, DeepCruiser randomly picks one transformation t (c.f. Section 6.1). If no further transformation is allowed to be picked (i.e., t = null) under the metamorphic relation constraints, DeepCruiser will select next input from the queue. For a picked transformation t, a random parameter is picked. DeepCruiser will generate a new audio a0 with transformation t and perform the transformation with the deep learning system R. If the prediction result is inconsistent with the original seed, a0 will be added into the failed tests. For audio in ASR, we decide a failed test by checking whether the Word/Character Error Rate (W/CER) exceeds a certain threshold. If the mutated test is not a failed test and increases the overall coverage, DeepCruiser adds it into the test queue and updates the coverage of tests in the test queue."
generating a set of iterative alternative audio test cases
[Page 3, Introduction "In this paper, we focus on the ASR domain4, and incorporate 8 metamorphic transformations to generate new audio test inputs."]
[Page 7 Section 3 The Overview of Coverage-guided Testing Framework DeepCruiser "With a chosen coverage criterion, the testing process starts with a set of initial seeds (some audios under practical ASR testing scenarios), namely the seed pool. Then in each iteration, we select an audio based on heuristics and generate a mutant from it by applying certain transformations. If the generated mutant triggers some defects, e.g., the transcription shows a large difference from the expectation, it is labeled as a failing test. Otherwise, we check whether the mutant improves the test coverage with respect to the chosen criteria and include it into the seed pool if so.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Rajan in view of Wang with Du in order to systematically generate diverse, successfully modified audio test cases and thereby improve the identification of errors and weakness in the test ASR system.
Regarding claim 6, rejection of claim 3 in incorporated.
Rajan teaches
The method of claim 3,
[Page 9, Example: The transformation is represented by T. Here, T 2 MT is the transformation and 2 paramT is the parameter of the transformation, which controls the
severity of the transformation."];
[Page 10, 4 Datasets and Experimental Setup "In each of the following transformations, we vary a parameter, . We call this the transformation parameter."];
[Page 10, 4 Datasets and Experimental Setup "Amplitude Scaling (Amp): For amplitude scaling, we scale the audio sequence by a constant by multiplying each individual audio sample by ."]
[Page 10, 4 Datasets and Experimental Setup "Drop/Frame: For Drop, AequeVox divides the audio into 20ms chunks. % of these chunks are then randomly discarded (amplitude set to zero) from the audio."]
[Page 10, 4 Datasets and Experimental Setup "Noise Addition (Noise): represents signal to noise (SNR) ratio [27] of the transformed audio signal. A lower means higher noise in the transformed audio."]
[Page 13, RQ2: What are the effects of transformations on comprehensibility? "Survey participants were presented with the original audio file along with a set of transformed
speech files in order of increasing intensity."]
Rajan in view of Wang does not teach wherein the second modification comprises; a first modification; applied to the corresponding at least one alternative audio test case.
Du teaches the second modification comprises; a first modification; applied to the corresponding at least one alternative audio test case.
[Page 13 , 6.1 Metamorphic Transformations of Audio Signals "However, when an audio input a0 is processed with a sequence of transformations (e.g., a0 ,! a1; : : : ; ,! ai), there is a higher chance that the audio cannot be well recognized by human due to accumulated distortions."]
[Page 14, 6.1 Metamorphic Transformations of Audio Signals "(2) an audio, generated by mutating from a seed after a sequence of transformations T, is limited to be mutated by a transformation
t 2 S [ UAT such that (ftg [ S) \ T = ;, where S 2 fV R; SRT;CRTg"]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Rajan in view of Wang with Du so that later alternative audio test cases include accumulated or progressively stronger feature changes. This would permit the tester to evaluate the AI audio system across increasing levels of input complexity and identify degree of modification at which the system begins producing erroneous results. This combination would be predictable because both Rajan in view of Wang and Du test speech recognition systems by applying audio transformations and evaluating the resulting outputs.
Regarding claim 7, rejection of claim 6 in incorporated.
Rajan teaches the method of claim 6, wherein the complexity level between the first modification and the second modification follows a complexity function.
[Page 9, Example: The transformation is represented by T. Here, T 2 MT is the transformation and 2 paramT is the parameter of the transformation, which controls the
severity of the transformation."];
[Page 10, 4 Datasets and Experimental Setup "In each of the following transformations, we vary a parameter, . We call this the transformation parameter."];
[Page 10, 4 Datasets and Experimental Setup "Amplitude Scaling (Amp): For amplitude scaling, we scale the audio sequence by a constant by multiplying each individual audio sample by ."]
[Page 10, 4 Datasets and Experimental Setup "Drop/Frame: For Drop, AequeVox divides the audio into 20ms chunks. % of these chunks are then randomly discarded (amplitude set to zero) from the audio."]
[Page 10, 4 Datasets and Experimental Setup "Noise Addition (Noise): represents signal to noise (SNR) ratio [27] of the transformed audio signal. A lower means higher noise in the transformed audio."]
Regarding claim 9, rejection of claim 1 in incorporated.
Rajan in view of Wang does not teach the method of claim 1, wherein a parameter defines which type of modifications the system is allowed to perform.
But Du teaches the method of claim 1, wherein a parameter defines which type of modifications the system is allowed to perform.
[Page 12 6.1 Metamorphic Transformations of Audio Signals "..we derive a set of transformation operators to mimic the environment interference. Overall, they can be classified into four categories:
– Volume-related transformations (VRT): ChangeVolume, LowPassFilter, HighPassFilter.
– Speed-related transformations (SRT): PitchShift, ChangeSpeed.
– Clearness-related transformations (CRT): AddWhiteNoise, ExtractHarmonic
– Unaffected transformations (UAT): DRC, Trim."]
[Page 14, 6.1 Metamorphic Transformations of Audio Signals "(1) we carefully set the parameters to ensure that a single step transformation (on volume, speed or clearness) does not violate the metamorphic
relations (i.e., human hearing are not affected); and (2) an audio, generated by mutating from a seed after a sequence of transformations T, is limited to be mutated by a transformation
t 2 S [ UAT such that (ftg [ S) \ T = ;, where S 2 fV R; SRT;CRTg. Intuitively, the constraints make sure that a mutant is generated by altering the volume,
speed or clearness of a seed input at most once. If this constraints is unsatisfied, the audio will not be transformed further."]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Rajan in view of Wang with Du by using a parameter to define the permitted types of audio modifications. This would constrain the generative testing process to valid or desired transformations and enable controlled selection among modifications operations, thereby producing relevant alternative audio test cases while avoiding unsuitable modifications.
Regarding claim 11, rejection of claim 4 in incorporated.
Rajan teaches the method of claim 4, further comprising analyzing the results to determine a
frequency and/or clustering of the results.
[Page 9, Example "As seen in Algorithm 2, AequeVox builds a word count dictionary for each word in WC and WCT for the original speech and for each 2 paramT respectively. For each word, AequeVox finds the difference in the number of appearances for a word in WC and in WCT for 2 paramT . To compute the difference, we locate the minimum number of appearances across all the transformation
parameters 2 paramT (i.e. min_count in Algorithm 2)."
[Page 10, Example "If the difference exceeds some user-defined threshold !, then AequeVox classifies the respective words as non robust w.r.t the group GR and transformation T"]
Regarding claim 15, rejection of claim 12 in incorporated.
Rajan in view of Wang in view of Du teaches the system of claim 12, wherein generating at least
one alternative audio test case comprises generating a set of iterative alternative audio test cases, comprising a series of modifications to the at least one audio test case.
Claim 15 is rejected for the same reason as claim 4.
Regarding claim 16, rejection of claim 15 in incorporated.
Rajan in view of Wang in view of Du teaches the system of claim 15, further comprising:
responsive to receiving the at least one audio test case, creating a trajectory;
for each increment on the trajectory, determining a set of operations mnemonics to apply;
and wherein generating the set of iterative alternative audio test case comprises applying one of the operations mnemonics to the at least one audio test case to provide a corresponding modification.
Claim 16 is rejected for the same reason as claim 5.
Regarding claim 17, rejection of claim 14 in incorporated.
Rajan in view of Wang in view of Du teaches the system of claim 14, wherein the second modification comprises a greater complexity level than the first modification, the complexity level comprising a level of feature change applied to the corresponding at least one alternative audio test case.
Claim 17 is rejected for the same reason as claim 6.
Regarding claim 18, rejection of claim 17 in incorporated.
Rajan in view of Wang in view of Du teaches The system of claim 17, wherein the complexity level between the first modification and the second modification follows a complexity function.
Claim 18 is rejected for the same reason as claim 7.
Claim [8, 19] are rejected under 35 U.S.C. 103 as being unpatentable over Rajan (Rajan, Sai Sathiesh, Sakshi Udeshi, and Sudipta Chattopadhyay. "Aequevox: Automated fairness testing of speech recognition systems." International Conference on Fundamental Approaches to Software Engineering. Cham: Springer International Publishing, 2022., hereinafter Rajan) in view of Wang (Wang, Yuancheng, et al. "Audit: Audio editing by following instructions with latent diffusion models." Advances in Neural Information Processing Systems 36 (2023): 71340-71357., hereinafter Wang) and in further view of Konopka (US 6941264 B2, hereinafter Konopka).
Regarding claim 8, rejection of claim 1 in incorporated.
Rajan teaches
The method of claim 1, further comprising:
in response to determining a problem in the external audio AI system,
[Page 10, 4. Datasets and Experimental Setup "ASR Systems under Test: We evaluate AequeVox on three commercial ASR systems from Google Cloud Platform (GCP), IBM Cloud, and Microsoft Azure." where Google Cloud Platform maps to external audio AI system];
[Page 6, Example "AequeVox uses two ASR systems and obtains the transcript of this speech. AequeVox then employs differential testing to find the word-level levenshtein distance [31] between these two sets of transcripts." where differential testing is done to determine a problem ];
[Page 6, Example "AequeVox then simulates a noisy environment by adding noise to the speech and obtains the transcript of this transformed speech" where transformed speech is the alternative audio test case]
However, Rajan in view of Wang does not teach sending information about the at least one
But Konopka teaches sending information about the at least one
[Column 7, lines 1-6 "Such utterances, and their correction information, are stored; and as mentioned above, when traffic on the communications link between the user's site and the remote central site is light, these utterances and their correction data, together with the criteria data that identify the user's class are transferred, or uploaded to that centra
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Rajan in view of Wang with Konopka by transmitting an alternative audio test case identified as causing a recognition problem, together with related corrected information, to the external speech recognition system. This would permit the external system to use the problematic audio to correct its speech models, thereby improving recognition accuracy.
Regarding claim 19 rejection of claim 12 in incorporated.
Rajan in view of Wang in view of Konopka teaches
The system of claim 12, further comprising:
in response to determining a problem in the external audio AI system, sending information about the at least one alternative audio test case to the external audio AI system.
Claim 19 is rejected for the same reasons as claim 8.
Claim [10] are rejected under 35 U.S.C. 103 as being unpatentable over Rajan (Rajan, Sai Sathiesh, Sakshi Udeshi, and Sudipta Chattopadhyay. "Aequevox: Automated fairness testing of speech recognition systems." International Conference on Fundamental Approaches to Software Engineering. Cham: Springer International Publishing, 2022., hereinafter Rajan) in view of Wang (Wang, Yuancheng, et al. "Audit: Audio editing by following instructions with latent diffusion models." Advances in Neural Information Processing Systems 36 (2023): 71340-71357., hereinafter Wang) and in further view of Margarit (US 20140006471, hereinafter Margarit).
Regarding claim 10 rejection of claim 1 in incorporated.
Rajan in view of Wang does not teach distance function is a user defined distance function.
But Margarit teaches distance function is a user defined distance function.
[0021 "The network 10A may employ or modulate weighting matrixes (see FIGS. 5A to 5B) to reduce a difference between the expected result or label and a result or label predicted by the network, instance, or model 10A, 50. An error or distance E may be determined by a user defined distance function in an embodiment"]
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Rajan in view of Wang with Margarit by making the distance function user-defined, thereby allowing the tester to select a distance measure appropriate for the particular AI audio system and desired error detection criteria.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHEZA ABDUL AZIZ whose telephone number is (571)272-9610. The examiner can normally be reached Monday-Friday 7:30am-5pm Alternate Fridays off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHEZA ABDUL AZIZ/Examiner, Art Unit 2657
/DANIEL C WASHBURN/Supervisory Patent Examiner, Art Unit 2657
Prosecution Insights