DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-18 are pending in this application.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Should applicant desire to obtain the benefit of foreign priority under 35 U.S.C. 119(a)-(d) prior to declaration of an interference, a certified English translation of the foreign application must be submitted in reply to this action. 37 CFR 41.154(b) and 41.202(e). Failure to provide a certified translation may result in no benefit being accorded for the non-English application.
CLAIM INTERPRETATION
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“a feature extraction module configured to extract features of an audio clip to be detected to determine,… a feature fusion module, configured to fuse the Mel spectrum-based fish school feeding depth speech spectrum feature vector…and a feeding intensity type determination module…”, in claim 6;
“a triangular frequency filter bank arrangement unit, configured to arrange a plurality of triangular filters in a frequency range of a fish school feeding sound signal to form a triangular frequency filter bank..;
a fast Fourier transform processing unit, configured to perform fast Fourier transform on a sound signal in the audio clip to be detected using the triangular…
an energy spectrum determination unit, configured to determine an energy spectrum according to the filtered sound signal;
a signal energy determination unit, configured to determine signal energy in each Mel filter according to the energy spectrum;
a Mel spectrogram determination unit, configured to determine a Mel spectrogram of the fish school feeding sound signal according to the signal energy;
a Mel spectrum-based fish school feeding depth speech spectrum feature vector extraction unit,…” in claim 7.
“a spectral parameter generating unit…
a constant-Q transform spectrogram generation unit,..
a CQT-based fish school feeding depth speech spectrum feature vector extraction unit…”, in claim 8”.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6-8 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim limitations “a feature extraction module configured to extract features of an audio clip to be detected to determine,… a feature fusion module, configured to fuse the Mel spectrum-based fish school feeding depth speech spectrum feature vector…and a feeding intensity type determination module…”, in claim 6;
“a triangular frequency filter bank arrangement unit, configured to arrange a plurality of triangular filters in a frequency range of a fish school feeding sound signal to form a triangular frequency filter bank..;
a fast Fourier transform processing unit, configured to perform fast Fourier transform on a sound signal in the audio clip to be detected using the triangular…
an energy spectrum determination unit, configured to determine an energy spectrum according to the filtered sound signal;
a signal energy determination unit, configured to determine signal energy in each Mel filter according to the energy spectrum;
a Mel spectrogram determination unit, configured to determine a Mel spectrogram of the fish school feeding sound signal according to the signal energy;
a Mel spectrum-based fish school feeding depth speech spectrum feature vector extraction unit,…” in claim 7.
“a spectral parameter generating unit…
a constant-Q transform spectrogram generation unit,..
a CQT-based fish school feeding depth speech spectrum feature vector extraction unit…”, in claim 8”; invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. The specification is devoid of adequate structure to perform the claimed functions. In particular, the feature extraction module configured to extract features of an audio to determined MEL spectrum based speech spectrum vector, CQT based spectrum feature vector, and STFT based spectrum feature vector and further comprising the units described in claims 7 and 8 is simply depicted in an abstract manner in Fig. 2. None of the units recited in claim 7 and 8 are exemplified in any drawing, thus, the structure or explanation of their structure is not defined or exemplified in the disclosure. The disclosure recites in [0120] “For the convenience of description, the previous apparatus is described by dividing the functions into various units. Certainly, when the present disclosure is implemented, the functions of each unit can be implemented in the same or multiple pieces of software and/or hardware. Those skilled in the art should understand that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may use a form of an entire hardware embodiment, an entire software embodiment or an embodiment combining software and hardware”. Therefore, as would be recognized by those of ordinary skill in the art, these units perform functions that can be performed in any number of ways in hardware, software or a combination of the two as stated by the disclosure. The specification does not provide sufficient details such that one of ordinary skill in the art would understand which structure or structures perform(s) the claimed function. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
There is no disclosure of any particular structure, either explicitly or inherently, to perform each of the functions above.
Therefore, the claims are indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 14-18 are rejected under 35 USC 101 because the claimed invention is directed to non-statutory subject matter.
Claims 14-18, recite “A computer readable storage medium…”.
The broadest reasonable interpretation of a computer readable storage medium typically covers forms of non-transitory tangible media and transitory propagating signals per se in view of the ordinary and customary meaning of computer usable medium (see MPEP 2111.01), wherein the transitory propagating signals are non-statutory subject matter.
The disclosure recites [0127] “…The computer readable medium includes a persistent and a non-persistent, a removable and a non-removable medium, which can implement information storage by using any method or technology …” [0128] recites “ According to the definition of the present disclosure, the computer readable medium does not include a transitory medium (transitory media), such as a modulated data signal and a modulated carrier””.
The claims recite the term “a computer readable storage medium” which is different than “The computer readable medium” as recited in the disclosure.
The claims can be amended to recite “A non-transitory computer readable storage medium” to correct the deficiency.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wanchao Li et al, hereinafter referred as “Wanchao” (“Feature fusion strategy and improved GhostNet for accurate recognition of fish feeding behavior”, this reference inventor is “by another” and the foreign priority has not been perfected since not translation has been provided).
As per claim 1, Wanchao teaches a method for classifying feeding intensity of fish school (see abstract “In this paper, a novel fish feeding intensity detection method based on the fusion of multiple features (Mel spectrogram, STFT, and CQT feature map) and the LC-GhostNet lightweight network was proposed), comprising:
extracting features of an audio clip to be detected (see page 5 col 2 “First, the fish feeding acoustic features were extracted and visualized to obtain Mel spectrograms, STFT, and CQT feature maps. ) to determine a Mel spectrum-based fish school feeding depth speech spectrum feature vector (see page 5 col 2 “First, the fish feeding acoustic features were extracted and visualized to obtain Mel spectrograms, STFT, and CQT feature maps. ), a Constant-Q Transform (CQT)-based fish school feeding depth speech spectrum feature vector (see page 5 col 2 “First, the fish feeding acoustic features were extracted and visualized to obtain Mel spectrograms, STFT, and CQT feature maps.), and a Short-Time Fourier Transform (STFT)-based fish school feeding depth speech spectrum feature vector (see page 5 col 2 “First, the fish feeding acoustic features were extracted and visualized to obtain Mel spectrograms, STFT, and CQT feature maps”; also, see page 14 Col 1 3.5 Fish feeding intensity detection and recognition system… Data pre-processing module, including data segmentation and extraction of feeding sound features (mainly Mel, CQT, and STFT features );
fusing the Mel spectrum-based fish school feeding depth speech spectrum feature vector, the CQT-based fish school feeding depth speech spectrum feature vector and the STFT-based fish school feeding depth speech spectrum feature vector to generate a fused feature spectrogram (see page 3 Col 2 “Proposed approach
The fusion strategy for fish feeding intensity recognition was
described in this section. Firstly, the acoustic features of the sound
dataset were obtained. Next, the 2D acoustic features (Mel spectrogram,
STFT, and CQT) were converted to 3D TFR, and…” ); and
inputting the fused feature spectrogram into a deep convolutional neural network model constructed by historical audio clips corresponding to different types of feeding intensities to determine a feeding intensity type corresponding to the audio clip to be detected (see ), wherein the feeding intensity type comprises “strong”, “medium”, “weak” and “none” (see page 3 Col 2 “2.3… Finally, we have completed the task of fish feeding intensity identification using the LC-GhostNet, and different CNN models and evaluation metrics were illustrated. The details of each part were shown below”; also, see Figs. 2-3).
As per claim 2, Wanchao teaches the method according to claim 1, Wanchao further teaches wherein extracting the Mel spectrum-based fish school feeding depth speech spectrum feature vector comprises:
arranging a plurality of triangular filters in a frequency range of a fish school feeding sound signal to form a triangular frequency filter bank (see page 4 Col 1 equation 2),
wherein the triangular frequency filter bank comprises a plurality of band-pass filters, the band-pass filters are Mel filters, a transfer function of each band-pass filter is:
PNG
media_image1.png
200
400
media_image1.png
Greyscale
wherein Hm(k) is the band-pass filter, m is a serial number of a Mel filter, M is a number of the Mel filters, f(m) is a center frequency of a m-th Mel filter, f(m+1) is a center frequency of a (m+1)-th Mel filter, and f(m−1) is a center frequency of a (m−1)-th Mel filter (see page 4 Col 1 equation 2 );
performing fast Fourier transform on a sound signal in the audio clip to be detected using the triangular frequency filter bank, and converting the sound signal from a time domain to a frequency domain, so as to generate a filtered sound signal (see page 4 col 1 “…When the Mel filter bank was designed, a FFT is performed on the windowed signal y(n) to convert the time domain signal to the frequency domain. As shown in Eqn. (4), k represents the k-th spectral line in the frequency domain…”, see equation 3);
determining an energy spectrum according to the filtered sound signal (see page 4 Col 1 equation 5);
determining signal energy in each Mel filter according to the energy spectrum (see page 4 Col 1 “The energy spectrum E(i, k) is obtained by squaring X(i, k) after the FFT, and the expression is calculated as follows. E(i, k) = [X(i, k)… Afterwards, the obtained energy spectrum is passed through M Mel
filter banks to obtain the energy of the signal in the each filter S(i, m).” );
determining a Mel spectrogram of the fish school feeding sound signal according to the signal energy (see page 4 Col 2 “The above method and steps can be used to obtain the M × N order matrix containing the signal energy magnitude information, and the Mel spectrogram of the fish feeding signal can be obtained by coloring according to the relationship between energy magnitude and color shade one-to-one mapping.);
and extracting the Mel spectrum-based fish school feeding depth speech spectrum feature vector in the audio clip to be detected according to the Mel spectrogram (see page 4 Col 2).
As per claim 3, Wanchao teaches the method according to claim 1, Wanchao further teaches wherein extracting the CQT-based fish school feeding depth speech spectrum feature vector comprises:
performing constant-Q transform on a sound signal in the audio clip to be detected to generate spectral parameters after constant-Q transform (see page 4 Col 2 2.3.1.3 “CQT…”);
generating a constant-Q transform spectrogram according to the spectral parameters (see page 4 Col 2 2.3.1.3 “CQT…” and page 5 Col 2 “Finally, the CQT is applied to the speech signal x(m), and the frequency component of the k-th octave of the transformed N-th frame is determined by Eqn. (11). Where, wNk (m) denotes the window function and Xcqt(k) denotes the spectral parameter after CQT…” and see equation 11; also, see Figs 2-3);
and extracting a CQT-based fish school feeding depth speech spectrum feature vector in the audio clip to be detected according to the constant-Q transform spectrogram (also, see Figs 2-3; also, see page 3 Col 2).
As per claim 4, Wanchao teaches the method according to claim 1, Wanchao further teaches wherein extracting the STFT-based fish school feeding depth speech spectrum feature vector comprises:
adding a short-time window function moving along time axis to a sound signal in the audio clip to be detected, and intercepting a non-stationary signal at each moment by the short-time window function (see page 4 Col 2 2.3.1.2 “The principle of Short-time Fourier Transform (STFT) is that the signal multiplied by the window function and then the Fourier transform is performed, and by shifting the window function on the time axis, a group of local spectra of the signal is obtained by analyzing the signal segment by segment (Nisar et al., 2016). As shown in Eqn. (7), s[n] represents the audio signal with window length L, and w[t] represents
the short-time window function. In this work, the sampling rate was set to 22050, L to 2048, the hop length to 512, and the desired number of Chroma bins to output was 12”),
wherein a signal within a short-time window is a stationary signal (see page 4 Col 2 2.3.1.2 );
performing Fourier transform on the non-stationary signal to generate a time-frequency spectrum of each moment (see page 4 Col 2 2.3.1.2 );
and extracting the STFT-based fish school feeding depth speech spectrum feature vector in the audio clip to be detected according to the time-frequency spectrum (see page 4 Col 2 2.3.1.2; also, see Figs. 2-3; see page 5 Col 2; see page 8 Col 2).
As per claim 5, Wanchao teaches the method according to claim 1, Wanchao further teaches wherein constructing the deep convolutional neural network model comprises:
acquiring historical video clips and historical sound signals of the fish school before, during and after feeding, respectively (see page 8 Col 2 “…Next, a training set, validation set and test set of 8:1:1 were created by randomly selecting audio segments. 5447 audio segments were used for training, 684 segments for validation; 681 segments were used for testing…”; see page 10 Col 1-2; see page 14 Col 1 );
dividing different types of feeding intensities according to the historical video clips, and synchronously clipping the historical sound signals to determine the historical audio clips corresponding to different types of feeding intensities (see page 8 Col 2; see page 10 Col 1-2; see page 14 Col 1 “…Specifically, we firstly constructed a dataset of fish feeding processes and divided the dataset into 4 different feeding states (strong, medium, weak, and none), followed by extracting Mel, STFT, and CQT features of feeding sounds using the Librosa library, then fused these features using feature image stitching; finally, the fused features were input to the LC-GhostNet for further feature extraction and classification..” ); and
and constructing the deep convolutional neural network model according to the historical audio clips (see page 8 Col 2; see page 10 Col 1-2; see page 14 Col 1 ).
As to claim 6, this claim is the system claim corresponding to the method claim 1 and is rejected for the same reasons mutatis mutandis (Wanchao teaches a system see Figs. 1, 2-3, 14; see page 2 Col 2 and see page 14 Col 1 3.5).
As to claim 7, this claim is the system claim corresponding to the method claim 2 and is rejected for the same reasons mutatis mutandis.
As to claim 8, this claim is the system claim corresponding to the method claim and is rejected for the same reasons mutatis mutandis.
As to claim 9, this claim is the electronic device apparatus claim corresponding to the method claim 1 and is rejected for the same reasons mutatis mutandis (Wanchao teaches a system and apparatus see Figs. 1, 2-3, 14; see page 14 Col 1 3.5).
As to claim 10, this claim is the electronic device apparatus claim corresponding to the method claim 2 and is rejected for the same reasons mutatis mutandis.
As to claim 11, this claim is the electronic device apparatus claim corresponding to the method claim 3 and is rejected for the same reasons mutatis mutandis.
As to claim 12, this claim is the electronic device apparatus claim corresponding to the method claim 4 and is rejected for the same reasons mutatis mutandis.
As to claim 13, this claim is the electronic device apparatus claim corresponding to the method claim 5 and is rejected for the same reasons mutatis mutandis.
As to claim 14, this claim is the computer readable medium claim corresponding to the method claim 1 and is rejected for the same reasons mutatis mutandis (Wanchao teaches a computer system and apparatus which suggests a memory and see Figs. 1, 2-3, 14; see page 14 Col 1 3.5).
As to claim 15, this claim is the computer readable medium claim corresponding to the method claim 2 and is rejected for the same reasons mutatis mutandis.
As to claim 16, this claim is the computer readable medium claim corresponding to the method claim 3 and is rejected for the same reasons mutatis mutandis.
As to claim 17, this claim is the computer readable medium claim corresponding to the method claim 4 and is rejected for the same reasons mutatis mutandis.
As to claim 18, this claim is the computer readable medium claim corresponding to the method claim 5 and is rejected for the same reasons mutatis mutandis.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-18 are rejected under 35 U.S.C. 103 as being unpatentable over Li et al (CN 115578678) in view of Chan et al (“Using different combination strategies to combine features for audio scene classification based on Convolutional Neural Networks”) and Cui et al (“FISH FEEDING INTENSITY ASSESSMENT IN AQUACULTURE: A NEW AUDIO DATASET AFFIA3K AND A DEEP LEARNING ALGORITHM).
As per claim 1, Li teaches a method for classifying feeding intensity of fish school (see abstract “The invention relates to a fish feeding intensity classification method and system”; and see [0051), comprising:
extracting features of an audio clip to be detected (see [0054] “S101, Obtain the audio data of the fish at the current moment”) to determine a Mel spectrum-based fish school feeding depth speech spectrum feature vector (see [0055] “S102, Extract the Mel frequency cepstral coefficients from the audio data at the current
moment”; also, see [0077 “(c) Obtain the Mel spectrum by passing the spectrum obtained by FFT through the Mel filter bank…”),
(see [0048] “Figure 7 is a Mel frequency spectrogram (strong feeding sound) of the Mel filter bank in an embodiment of the present invention”; also, see [0076]-[0077] “(c) Obtain the Mel spectrum by passing the spectrum obtained by FFT through the Mel filter bank…”; also, see [0079]); and
inputting the fused feature spectrogram into a deep convolutional neural network model constructed by historical audio clips corresponding to different types of feeding intensities to determine a feeding intensity type corresponding to the audio clip to be detected (see [0062] –[0065] “the training process of the fish feeding intensity
classification model is as follows: Video and audio data of fish at different feeding stages were acquired. The feeding stages included: before feeding, during feeding, and after feeding. In a recirculating aquaculture system (RAS), a Hikvision color camera (model: DS-2SC3Q140MY-TE) and an omnidirectional
LST-DH01 digital hydrophone were used to acquire video and audio data of fish before feeding, during feeding, and after feeding, as shown in Figure 3…”,…before feeding suggests historical audio data ; also, see [0086] “The neural network that employs the attention mechanism is based on the mobile_V3_Small network structure. It replaces part of the Squeeze-and-Excitation block (SENet) with the Convolution block Attention Module (CBAM) attention mechanism. CBAM integrates channel attention and spatial attention mechanisms, which can take both into account and achieve better results ), wherein the feeding intensity type comprises “strong”, “medium”, “weak” (see [0066] “…Figure 4. From left to right in Figure 4, the categories are "strong", "medium", and "weak". For details, please refer to Table 1…. By reviewing the videos, the feeding intensity of the fish was divided into three types:
"strong", "medium", and "weak". Then, based on the video's classification intensity and time period, the synchronized audio data was subjected to the same cropping process to obtain audio segments of three feeding intensity types, as shown in Figure 5, from left to right: "strong", "medium", and "weak").
Li does not explicitly teach:
extracting features of an audio clip to be detected (see [0054] “S101, Obtain the audio data of the fish at the current moment”) to determine a Constant-Q Transform (CQT)-based fish school feeding depth speech spectrum feature vector, and a Short-Time Fourier Transform (STFT)-based fish school feeding depth speech spectrum feature vector; fusing the Mel spectrum-based fish school feeding depth speech spectrum feature vector, the CQT-based fish school feeding depth speech spectrum feature vector and the STFT-based fish school feeding depth speech spectrum feature vector to generate a fused feature spectrogram, and inputting the fused feature spectrogram into a deep convolutional neural network model constructed by historical audio clips corresponding to determine different types of classification intensities; and
wherein the feeding intensity type comprises on category such as “none”.
Chan teaches a system and a CNN-based fusion of multiple features multi-strategy sound scene classification method comprising extracting features of an audio clip to be detected to determine a Constant-Q Transform (CQT)-based speech/audio spectrum feature vector, and a Short-Time Fourier Transform (STFT)-based speech/audio spectrum feature vector, (see page 2 section 2, 2.1, 2.32, 2.3, and 2.4), fusing the Mel spectrum-based speech/audio spectrum feature vector, the CQT-based speech/audio spectrum feature vector and the STFT-based speech/audio spectrum feature vector to generate a fused feature spectrogram, and inputting the fused feature spectrogram into a deep convolutional neural network model to determine different types of classification intensities (see Figs. 1 and 2; also, see pages 2-3 section 2, 2.2, 2.3 “2.2. The first combination strategy In the first strategy, each audio recording is first split into frames, and for each frame, the Mel spectrogram, STFT feature and CQT feature is extracted respectively, and then two or three of them are stacked together to form the multi-channel input of CNN. In Figure 1, we take the three-channel input for illustration. 2.3. The second combination strategy In the second combination strategy, each audio recording is first split into frames, and then for each frame, the Mel spectrogram, STFT feature and CQT feature is extracted respectively, and then two or three of them are concatenated together to form a new longer representation for each frame, each audio recording as that described in formula (1) is taken as the input of CNN. In formula (1), we show the concatenation of three kinds of features for illustration, M, S, and C indicates Mel spectrogram, STFT feature, and CQT feature respectively.”; also, see page 3 2.4 “In the third combination strategy, each audio recording is split into frames, and for each frame, the Mel spectrogram, STFT feature and CQT feature is extracted respectively, two or three of them are then used for combination. For the features which are used for combination, each kind of feature is taken as the input of a one-channel CNN to generate new feature, the new features are then concatenated together, this new concatenated feature is then taken as the input of the CNN model for classification. In Figure 2, we show the combination of all three kinds of features to illustrate the third combination strategy”, concatenated suggests fusion).
Therefore, it would have been obvious to one of ordinary skilled in the art before effective filing date of the claimed invention to which said subject matter pertains to have modified Li’s invention to include CNN-based fusion of multiple features multi-strategy sound scene classification method comprising extracting features of an audio clip to be detected to determine a Constant-Q Transform (CQT)-based speech/audio spectrum feature vector, and a Short-Time Fourier Transform (STFT)-based speech/audio spectrum feature vector, fusing the Mel spectrum-based speech/audio spectrum feature vector, the CQT-based speech/audio spectrum feature vector and the STFT-based speech/audio spectrum feature vector to generate a fused feature spectrogram, and inputting the fused feature spectrogram into a deep convolutional neural network model to determine different types of classification intensities as taught by Chan in order to improve a sound classification by using the combination of Mel, CQT and STFT methods (see page 5 conclusion 4. Conclusion
In this paper, we propose three combination strategies to combine different kinds of features for classification, and we do experiments to test which combination strategy would get better results, to test if combining more features would get better results. Also we do experiments to compare our proposed classification system with some other classification systems proposed in the references. The
conclusion is that the second combination strategy, that is, concatenating different kinds of features directly for classification, has achieved the best results; on the whole, combining more features would get better results. Our proposed classification system performs much better than the DCASE2017
baseline system; although its performance is worse than two other systems on the development dataset, it performs much better on the evaluation dataset, which means that the proposed system has much better generalization ability”).
Li-Chan does not explicitly teach wherein the feeding intensity type comprises on category such as “none”.
However, Cui et al teaches a system and method comprising determining a feeding intensity using a CNN model and MEL Spectrogram, wherein the feeding intensity type comprises on category such as “none” (see Table 1 In page 1 feeding intensity “none”; see page 4 Col 2 “…We can see that the classification precision for the “Strong” and “None” feeding intensity is higher, as compared with the classification precision for “Weak” and “Medium” feeding intensity”…”; see page 5 Conclusion ).
Therefore, it would have been obvious to one of ordinary skilled in the art before effective filing date of the claimed invention to which said subject matter pertains to have modified Li-Chan’s combination as taught about to include determining a feeding intensity using a CNN model and Mel spectrogram, wherein the feeding intensity type comprises on category such as “none” as taught by Cui in order to include the intensity category of “none” to determine when fish do not respond to food (see table 1 in page 1) an to evaluate the intensity change of fish appetite during feeding procedure, which potentially improves the farming efficiency and saves the feed cost in industrial aquaculture (see page 1 Col 1; assessing the fish feeding intensity avoids providing food to fish when they are not hungry which causes pollution in the water).
As per claim 2, Li-Chan-Cui teaches the method according to claim 1, Li further teaches wherein extracting the Mel spectrum-based fish school feeding depth speech spectrum feature vector (see 0074-0079) comprises:
arranging a plurality of triangular filters in a frequency range of a fish school feeding sound signal to form a triangular frequency filter bank (see page 5 claim 1 “A triangular frequency filter bank is used to perform spectral conversion on the Fourier transform audio data; also, see [0077] ),
wherein the triangular frequency filter bank comprises a plurality of band-pass filters, the band-pass filters are Mel filters, a transfer function of each band-pass filter is (see [0077]):
PNG
media_image1.png
200
400
media_image1.png
Greyscale
wherein Hm(k) is the band-pass filter, m is a serial number of a Mel filter, M is a number of the Mel filters, f(m) is a center frequency of a m-th Mel filter, f(m+1) is a center frequency of a (m+1)-th Mel filter, and f(m−1) is a center frequency of a (m−1)-th Mel filter (see equation 1 in page 91
PNG
media_image2.png
200
400
media_image2.png
Greyscale
; also, see [0079] in the machine translation);
performing fast Fourier transform on a sound signal in the audio clip to be detected using the triangular frequency filter bank, and converting the sound signal from a time domain to a frequency domain, so as to generate a filtered sound signal (see [0076] “(b) For each short-time analysis window, the signal is transformed from the time domain to the frequency domain by Fast Fourier Transform (FFT) to obtain the corresponding linear spectrum”);
determining an energy spectrum according to the filtered sound signal (see [0084]);
determining signal energy in each Mel filter according to the energy spectrum (see [0084] “(e) By performing a discrete cosine transform (DCT) on S(m) to obtain the cepstral frequency domain, the MFCC can be obtained.);
determining a Mel spectrogram of the fish school feeding sound signal according to the signal energy (see [0084] “(e) By performing a discrete cosine transform (DCT) on S(m) to obtain the cepstral frequency domain, the MFCC can be obtained.”); and
extracting the Mel spectrum-based fish school feeding depth speech spectrum feature vector in the audio clip to be detected according to the Mel spectrogram (see [0077] “(c) Obtain the Mel spectrum by passing the spectrum obtained by FFT through the Mel filter bank”).
Cui also teaches a Mel Spectrogram which is a very well-known spectrogram method to obtain audio/sound spectrograms (see abstract).
As per claim 3, Li-Chan-Cui teaches the method according to claim 1, Chan further teaches extracting the CQT-based speech/audio spectrum feature vector comprises: performing constant-Q transform on a sound signal in the audio clip to be detected to generate spectral parameters after constant-Q transform (see Fig. 1-2 a CQT transform consists in obtaining spectral parameters; also, see page 2 par. 2 “…We choose three different kinds of features which are very often used in the field of ASC for combination, that is, the Mel spectrogram, Short-Time Fourier Transform (STFT) feature and Constant Q-Transform (CQT) feature…” and last par. “2.2. The first combination strategy In the first strategy, each audio recording is first split into frames, and for each frame, the Mel spectrogram, STFT feature and CQT feature is extracted respectively, and then two or three of them are stacked together to form the multi-channel input of CNN. In Figure 1, we take the three-channel input for illustration”; also, see page 3 par. 2 “…CQT feature is extracted respectively, two or three of them are then used for combination…”), generating a constant-Q transform spectrogram according to the spectral parameters (see Fig. 1 and 2), and extracting a CQT-based speech spectrum feature vector in the audio clip to be detected according to the constant-Q transform spectrogram (see Fig. 1-2 a CQT transform consists in obtaining spectral parameters; also, see page 2 par. 2 “…We choose three different kinds of features which are very often used in the field of ASC for combination, that is, the Mel spectrogram, Short-Time Fourier Transform (STFT) feature and Constant Q-Transform (CQT) feature…” and last par. “2.2. The first combination strategy In the first strategy, each audio recording is first split into frames, and for each frame, the Mel spectrogram, STFT feature and CQT feature is extracted respectively, and then two or three of them are stacked together to form the multi-channel input of CNN. In Figure 1, we take the three-channel input for illustration”; also, see page 3 par. 2 “…CQT feature is extracted respectively, two or three of them are then used for combination…”).
Therefore, it would have been obvious to one of ordinary skilled in the art before effective filing date of the claimed invention to which said subject matter pertains to have modified Li-Chan-Cui invention to include extracting the CQT-based speech/audio spectrum feature vector comprises: performing constant-Q transform on a sound signal in the audio clip to be detected to generate spectral parameters after constant-Q transform, generating a constant-Q transform spectrogram according to the spectral parameters, and extracting a CQT-based speech spectrum feature vector in the audio clip to be detected according to the constant-Q transform spectrogram as taught by Chan in order to apply the three different methods of MEL, CQT and STFT to classify sounds and apply the CQT method to fish feeding depth speech/audio in combination with the MEL and STFT method in order to improve a sound classification by using the combination of Mel, CQT and STFT methods (see page 5 conclusion 4. Conclusion
In this paper, we propose three combination strategies to combine different kinds of features for classification, and we do experiments to test which combination strategy would get better results, to test if combining more features would get better results. Also we do experiments to compare our proposed classification system with some other classification systems proposed in the references. The
conclusion is that the second combination strategy, that is, concatenating different kinds of features directly for classification, has achieved the best results; on the whole, combining more features would get better results. Our proposed classification system performs much better than the DCASE2017
baseline system; although its performance is worse than two other systems on the development dataset, it performs much better on the evaluation dataset, which means that the proposed system has much better generalization ability”).
As per claim 4, Li-Chan-Cui teaches the method according to claim 1, Chan further teaches wherein extracting the STFT-based speech spectrum feature vector comprises (see Fig. 1 and 2): adding a short-time window function moving along time axis to a sound signal in the audio clip to be detected, and intercepting a non-stationary signal at each moment by the short-time window function (see page 4 par. 2 “…for STFT, the window length and hop length are set to be 298 samples and 256 samples respectively; for CQT, the hop length is set to be 256 samples…”), wherein a signal within a short-time window is a stationary signal (see (see page 4 par. 2 “…for STFT, the window length and hop length are set to be 298 samples and 256 samples respectively; for CQT, the hop length is set to be 256 samples…”), performing Fourier transform on the non-stationary signal to generate a time-frequency spectrum of each moment (see Fig. 1-2 CQ feature extraction; also, see page 3), and extracting the STFT-based speech spectrum feature vector in the audio clip to be detected according to the time-frequency spectrum (see Fig. 1-2 CQ feature extraction; also, see page 3).
Therefore, it would have been obvious to one of ordinary skilled in the art before effective filing date of the claimed invention to which said subject matter pertains to have modified Li-Chan-Cui invention to include extracting the STFT-based speech spectrum feature vector comprises: adding a short-time window function moving along time axis to a sound signal in the audio clip to be detected, and intercepting a non-stationary signal at each moment by the short-time window function, wherein a signal within a short-time window is a stationary signal, performing Fourier transform on the non-stationary signal to generate a time-frequency spectrum of each moment, and extracting the STFT-based speech spectrum feature vector in the audio clip to be detected according to the time-frequency spectrum as taught by Chan in order to apply the three different methods of MEL, CQT and STFT to classify sounds and apply the STFT method to fish feeding depth speech/audio in combination with the MEL and CQT method in order to improve a sound classification by using the combination of Mel, CQT and STFT methods (see page 5 conclusion 4. Conclusion In this paper, we propose three combination strategies to combine different kinds of features for classification, and we do experiments to test which combination strategy would get better results, to test if combining more features would get better results. Also we do experiments to compare our proposed classification system with some other classification systems proposed in the references. The
conclusion is that the second combination strategy, that is, concatenating different kinds of features directly for classification, has achieved the best results; on the whole, combining more features would get better results. Our proposed classification system performs much better than the DCASE2017
baseline system; although its performance is worse than two other systems on the development dataset, it performs much better on the evaluation dataset, which means that the proposed system has much better generalization ability”).
As per claim 5, Li-Chan-Cui teaches the method according to claim 1,
Li further teaches wherein constructing the deep convolutional neural network model comprises: acquiring historical video clips and historical sound signals of the fish school before, during and after feeding, respectively (see [0062] –[0065] “the training process of the fish feeding intensity classification model is as follows: Video and audio data of fish at different feeding stages were acquired. The feeding stages included: before feeding, during feeding, and after feeding. In a recirculating aquaculture system (RAS), a Hikvision color camera (model: DS-2SC3Q140MY-TE) and an omnidirectional LST-DH01 digital hydrophone were used to acquire video and audio data of fish before feeding, during feeding, and after feeding, as shown in Figure 3…”,…before feeding suggests historical audio data ; also, see [0086] “The neural network that employs the attention mechanism is based on the mobile_V3_Small network structure. It replaces part of the Squeeze-and-Excitation block (SENet) with the Convolution block Attention Module (CBAM) attention mechanism. CBAM integrates channel attention and spatial attention mechanisms, which can take both into account and achieve better results), dividing different types of feeding intensities according to the historical video clips, and synchronously clipping the historical sound signals to determine the historical audio clips corresponding to different types of feeding intensities (see [0066] “…Figure 4. From left to right in Figure 4, the categories are "strong", "medium", and "weak". For details, please refer to Table 1…. By reviewing the videos, the feeding intensity of the fish was divided into three types: “strong", "medium", and "weak". Then, based on the video's classification intensity and time period, the synchronized audio data was subjected to the same cropping process to obtain audio segments of three feeding intensity types, as shown in Figure 5, from left to right: "strong", "medium", and "weak"), and constructing the deep convolutional neural network model according to the historical audio clips (see [0062-0065]).
As to claim 6, this claim is the system claim corresponding to the method claim 1 and is rejected for the same reasons mutatis mutandis (Li further teaches a system see Abstract, see claim 7 in page 9, see [0001], [0004], [0030-0037] and [0038]).
As to claim 7, this claim is the system claim corresponding to the method claim 2 and is rejected for the same reasons mutatis mutandis.
As to claim 8, this claim is the system claim corresponding to the method claim and is rejected for the same reasons mutatis mutandis.
As to claim 9, this claim is the electronic device apparatus claim corresponding to the method claim 1 and is rejected for the same reasons mutatis mutandis ((Li further teaches a system and apparatus see Abstract, see claim 7 in page 9, see [0001], [0004], [0030-0037] and [0038] “A fish feeding intensity classification system includes: at least one processor, at least one memory, and computer program instructions stored in the memory, wherein the computer program instructions are executed by the processor to implement the fish feeding intensity classification method”).
As to claim 10, this claim is the electronic device apparatus claim corresponding to the method claim 2 and is rejected for the same reasons mutatis mutandis.
As to claim 11, this claim is the electronic device apparatus claim corresponding to the method claim 3 and is rejected for the same reasons mutatis mutandis.
As to claim 12, this claim is the electronic device apparatus claim corresponding to the method claim 4 and is rejected for the same reasons mutatis mutandis.
As to claim 13, this claim is the electronic device apparatus claim corresponding to the method claim 5 and is rejected for the same reasons mutatis mutandis.
As to claim 14, this claim is the computer readable medium claim corresponding to the method claim 1 and is rejected for the same reasons mutatis mutandis (Li further teaches a system see Abstract, see claim 7 in page 9, see [0001], [0004], [0030-0037] and [0038] “A fish feeding intensity classification system includes: at least one processor, at least one memory, and computer program instructions stored in the memory, wherein the computer program instructions are executed by the processor to implement the fish feeding intensity classification method).
As to claim 15, this claim is the computer readable medium claim corresponding to the method claim 2 and is rejected for the same reasons mutatis mutandis.
As to claim 16, this claim is the computer readable medium claim corresponding to the method claim 3 and is rejected for the same reasons mutatis mutandis.
As to claim 17, this claim is the computer readable medium claim corresponding to the method claim 4 and is rejected for the same reasons mutatis mutandis.
As to claim 18, this claim is the computer readable medium claim corresponding to the method claim 5 and is rejected for the same reasons mutatis mutandis.
Conclusion
The prior art made of record and not relied upon, as cited in PTO form 892, is considered pertinent to applicant's disclosure.
Jing Chi et al, (L-GhostNet: Extract Better Quality Features), teaches a method of a convolutional neural network L-Ghost net widely used for image recognition, object detection and feature extraction of data, which can be adapted for audio processing. This model has been improved by introducing a CA method instead of SE (see page 2363 “This section focuses on the L-GhostNet network structure proposed in the paper. Based on GhostNet, L-GhostNet first uses an improved LG instead of the redundant Ghost module in GhostNet, then introduces an improved CA instead of SE, which significantly reduces the parameters and computation while ensuring recognition accuracy and network flexibility.). It is relevant to the invention as described in the disclosure in Figs. 6A and 6B.
Wikipedia Constant Q transform teaches the formulas of the well-known methodologies of frequency transformation, wherein the equations presented in the disclosure eqn 9-11 are disclosed in this document. The document suggest that these equation were widely known and used.
Muhammad Huzaifah (“Comparison of Time-Frequency Representations for
Environmental Sound Classification using Convolutional Neural Networks”) teaches that combining signal processing methods such as CQT, MEL spectrum, and STFT and CWT, impacts the accuracy of classification of sounds/speech greater than using a single process method (see Abstract and see page 1 “This letter builds upon previous comparative studies by focusing on the specifics of a CNN model as opposed to more
traditional classifiers. We investigate four common approaches to obtain the time-frequency representation, namely the short time Fourier transform (STFT) with both linear and Mel scales, the constant-Q transform (CQT) and the continuous Wavelet
transform (CWT), while addressing additional considerations like window size. The impact of the different approaches is evaluated in comparison to baseline MFCC features on two publicly available environmental sound datasets (ESC-50, UrbanSound8K) through the classification performance of several CNN variants”; also, see page 3 Col 2 and table I).
Xing (US 20180276540) teaches the combination of MEL spectrogram, CQT, and STFT (see 0049), using a convolutional NN to classify the features extracted from MEL spectrogram, CQT, and STFT (see 0028-0030).
Bataev et al (US 20240135920) teaches a system and method for classifying audio/speech comprising a CQT, Mel spectrum, and STFT audio representation modules (see [0031-0032]), and including a CNN for to process spectrograms to generate a transcription (see 0035-0036 and 0038-0040).
Examiner respectfully requests, in response to this Office action, support be shown for language added to any original claims on amendment and any new claims. That is, indicate support for newly added claim language by specifically pointing to page(s) and line number(s) in the specification and/or drawing figure(s). This will assist Examiner in prosecuting the application.
When responding to this Office Action, Applicant is advised to clearly point out the patentable novelty which he or she thinks the claims present, in view of the state of the art disclosed by the references cited or the objections made. Applicant must also show how the amendments avoid or differentiate from such references or objections. See 37 CFR 1.111 (c).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OLVIN LOPEZ ALVAREZ whose telephone number is (571) 270-7686 and fax (571) 270-8686. The examiner can normally be reached Monday thru Friday from 9:00 A.M. to 6:00 P.M.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Robert Fennema, can be reached at (571) 272-2748. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/O. L./
Examiner, Art Unit 2117
/ROBERT E FENNEMA/Supervisory Patent Examiner, Art Unit 2117
Prosecution Insights