DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 6 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Sarzen (US 2021/0298272 A1) in view of Berckmans (US 2022/0007618 A1).
Regarding claim 6, Sarzen discloses a system for monitoring abnormal state of swines based on edge computing (Sarzen discloses a farm animal operation monitoring system that uses edge computing to detect condition states of interest – including respiratory illness and stress – in collections of farm animals, explicitly including pigs (Sarzen [0013], [0038], [0041]).), comprising:
a data acquisition and processing module (Sarzen teaches a sound collection device 115 and local processing device 105 (Sarzen [0041], [0044])), an edge computing gateway (Sarzen teaches a edge computing device (Sarzen [0041], [0096], [0135])), a cloud server (Sarzen teaches a cloud computing environment (Sarzen [0041]-[0042], [0130], [0143])) and a client (Sarzen teaches notification/dashboard on user equipment (client) (Sarzen [0045],[0067], [0140]));
the data acquisition and processing module is used for acquiring an audio signal and processing the audio signal (Sarzen teaches processing audio data based on sound acquired from a sound collection device, with processing performed on the local device (Sarzen [0041], [0078], [0140]));
the edge computing gateway is used for analyzing an abnormal sound according to processed audio signal (Sarzen teaches the edge computing device performs the sound processing/analysis (Sarzen [0041])), and transmitting analysis results to the cloud server (Sarzen teaches use of a cloud computing environment for storing/using the monitoring outputs (and associated dashboard usage), which supports transmitting results to a cloud server (Sarzen [0041]-[0042], [0130])), wherein the analysis results comprise a classification result (Sarzen discloses that the edge computing device applies a machine learning library to the processed audio stream to classify detected sound types of interest — including pig coughs and stress vocalizations — associated with farm animal condition states, thereby generating a classification result (Sarzen [0038], [0096], [0140]).); and
the cloud server is used for constructing an early warning model of swine abnormal state, obtaining early warning information based on the analysis results and the early warning model of the swine abnormal state, and transmitting the early warning information to the client (Sarzen teaches cloud computing environment supporting dashboard/reporting and storage/aggregation of monitoring outputs (Sarzen [0041]-[0042], [0130]). And sending notifications viewable via a dashboard on user equipment (client) (Sarzen [0045], [0140])).
However, Sarzen does not expressly disclose "a positioning result". Specifically, Sarzen teaches that its edge computing device classifies sound types of interest from the acquired audio stream, but does not disclose that the edge computing gateway also generates a positioning result – i.e., a spatial localization of the source of the detected abnormal sound within the facility – as part of the analysis results transmitted to the cloud server. Sarzen's microphone disclosure at [0044] acknowledges that multiple wireless microphones at various locations can be used to generate audio from different vantage points within a farm animal operation, but Sarzen does not disclose using those microphones to perform sound source localization or to generate a positioning result identifying the spatial origin of a detected abnormal sound.
In an analogous art, Berckmans teaches localizing sounds of interest using two or more microphones, and explicitly defines localization as detecting the sound source coordinates in a three-dimensional area of interest (i.e., a positioning result). See Berckmans [0078], [0082], [0087], [0092], [0104].
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Sarzen’s edge-based farm animal sound monitoring architecture to additionally generate and transmit a positioning (localization) result as taught by Berckmans, because Sarzen already provides sound-based detection/classification and cloud/user notification for animal health/welfare monitoring, and Berckmans teaches determining sound source coordinates from multi-microphone audio. Adding Berckmans’ localization to Sarzen’s system would have predictably provided actionable information about where the detected abnormal sound originated (e.g., which pen/area), enabling faster targeted intervention and improving the utility of the alerts generated and delivered to the client. This is a predictable use of known localization techniques in a known livestock audio monitoring system, with each component performing its known function (Sarzen [0041], [0044], [0130], [0140]; Berckmans [0001], [0087]).
Regarding claim 10, Sarzen in view of Berckmans discloses the system for monitoring the abnormal state of the swines based on the edge computing according to claim 6, wherein the early warning information obtained by the early warning model of the swine abnormal state comprises several early warning states (Berckmans explicitly discloses that the monitoring system assigns several tiered warning states based on detected livestock sounds – green indicating normal status, yellow indicating an intermediate state requiring increased vigilance, and red indicating a potential disease outbreak based on a high amount of detected coughing, sneezing, snicking, and/or screaming (Berckmans [0095]). It would have been obvious to incorporate these tiered warning states into Sarzen's early warning information output, as doing so provides the user with a structured, severity-differentiated alert that more clearly communicates the urgency of detected abnormal conditions than the binary presence/absence notification of Sarzen alone. Such tiered warning systems are a well-recognized and routinely implemented feature of monitoring systems, and their incorporation into Sarzen's system yields only predictable results.), frequencies of abnormal sounds (Berckmans discloses at [0095] that warning states are assigned based on the amount of detected abnormal sounds – expressly establishing that the system tracks the frequency/quantity of abnormal sound events as the operative basis for determining warning severity. It would have been obvious to one of ordinary skill in the art that a system which assigns tiered warning states based on the amount of detected abnormal sounds would include the underlying frequency information in the early warning information transmitted to the client, as such information is both inherently generated by the system and directly useful to the farmer or veterinarian in assessing the severity of the detected condition. Sarzen at [0069] further confirms that tracking frequency of occurrence of detected sound types over time is a recognized practice in farm animal sound monitoring systems. The inclusion of frequency data in the transmitted early warning information is therefore an obvious and predictable implementation detail for a person of ordinary skill in the art.) and position information of the abnormal sounds (Berckmans explicitly discloses generating position information of abnormal sounds through multi-microphone sound source localization, producing directional coordinates of the origin of detected abnormal livestock sounds and identifying the location of unhealthy animals within the facility (Berckmans [0082], [0087], [0092], [0104]). Including this position information in the early warning information transmitted to the client is the natural and obvious extension of generating it at the edge level – the purpose of localization, as Berckmans expressly discloses at [0082], is precisely to inform a user of the location of healthy and unhealthy animals, which requires that information to be communicated to the client. This yields the predictable result of a more actionable early warning notification.).
Claim(s) 7-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sarzen (US 2021/0298272 A1) in view of Berckmans (US 2022/0007618 A1) as applied to claim 6 above, and further in view of Adavanne et al., "Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks," IEEE Journal of Selected Topics in Signal Processing, Vol. 13, No. 1, March 2019, pp. 34–48 (hereinafter "Adavanne").
Regarding claim 7, Sarzen in view of Berckmans discloses the system for monitoring the abnormal state of the swines based on the edge computing according to claim 6, wherein the data acquisition and processing module comprises a microphone array, a digital signal processing module, a power amplification module, a power supply module and a communication module, wherein the power amplification module is used for performing amplification processing on the audio data, wherein the power supply module is used for supplying power to the microphone array, the digital signal processing module, the power amplification module and the communication module (Berckmans discloses a data acquisition device comprising a circular microphone array with six microphones (202-1 through 202-6) for collecting livestock audio, and a processing module that performs digital signal processing operations on the acquired audio signals including beamforming and feature extraction (Berckmans [0091], [0099], [0104]). Berckmans further discloses a gateway providing communication connectivity between the local device and external networks constituting the claimed communication module (Berckmans [0101]). A power amplification module and a power supply module are generic and necessary hardware components of any electronic device incorporating microphones and signal processing circuitry, and their inclusion in the data acquisition and processing module would have been obvious to one of ordinary skill in the art as a matter of routine engineering design.);
wherein the microphone array comprises a plurality of directional microphones, a number of the microphones is not less than four, and the microphones are used for collecting audio data, wherein the audio data comprises a multi-channel synchronous audio signal (Berckmans discloses a circular array of six microphones that captures audio using directional beamforming techniques, satisfying the plurality of directional microphones requirement of not less than four (Berckmans [0046], [0091], [0092]).).
However, Sarzen in view of Berckmans does not expressly disclose (1) collected multi- channel audio signal is encoded and converted into a first-order ambisonics (FOA) signal; and (2) the digital signal processing module adopts a field programmable gate array (FPGA) module, used for extracting feature parameters after the amplification processing on the audio data and transmitting the feature parameters to the edge computing gateway through the communication module.
Regarding the FPGA limitation, the examiner takes official notice that the use of an FPGA as the digital signal processing module in an embedded audio acquisition system for extracting feature parameters from acquired audio signals and transmitting those parameters to a downstream computing device was well-established and widely practiced in the electronic engineering arts at the time of the invention. FPGAs were well known to offer low latency, parallel processing capability, and real-time configurability that make them a routine and predictable choice for embedded audio feature extraction tasks of the type recited in the claim – specifically, processing amplified multi-channel audio signals and transmitting extracted feature parameters to an edge computing gateway via a communication module. The selection of an FPGA for this function over other known DSP hardware implementations such as dedicated DSP chips or general-purpose processors is a routine design choice that yields only predictable results.
Regarding the FOA encoding limitation, Adavanne explicitly discloses that multi-channel audio captured from a microphone array is encoded and converted into first-order ambisonics (FOA) format comprising four channels, and that this FOA format is used as the input to a joint sound event detection and localization system. See Section I.A; Section III.A; Section III.A.3.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Adavanne with Sarzen (as modified by Berckmans). All three references are directed to the same technical field of acoustic sound event detection and localization using multi-channel audio. A person of ordinary skill in the art implementing the edge computing swine monitoring system of Sarzen and Berckmans would have recognized that the system requires a data acquisition module capable of supplying multi-channel audio input suitable for the detection and localization tasks performed at the edge. Adavanne expressly teaches that FOA-format encoding of multi-channel microphone audio is a suitable and effective input format for joint sound event detection and localization – the precise functions performed by the edge computing gateway of Sarzen and Berckmans. The incorporation of Adavanne's FOA encoding into the data acquisition module is therefore a combination of known elements using known methods to achieve predictable results.
Regarding claim 8, Sarzen in view of Berckmans discloses the system for monitoring the abnormal state of the swines based on the edge computing according to claim 6, but does not expressly disclose wherein the edge computing gateway comprises an abnormal sound detection model and an abnormal sound positioning model, both of the abnormal sound detection model and the abnormal sound positioning model comprise a depth feature extraction layer, a plurality of BiLSTM or BiGRU layers, a full connection layer and an activation layer.
In an analogous art, Adavanne discloses a convolutional recurrent neural network (SELDnet) for joint sound event detection (SED) and direction-of-arrival (DOA) estimation comprising: multiple layers of 2D convolutional neural network (CNN) performing deep spatial and spectral feature extraction (Section II.B; Section IV.1 – three CNN layers with 64 nodes each) mapping to the claimed depth feature extraction layer; two bidirectional GRU (BiGRU) layers with 128 nodes each (Section II.B; Section IV.1) mapping to the claimed plurality of BiLSTM or BiGRU layers; parallel fully connected output branches for the SED and DOA tasks respectively, each comprising FC layers (Section II.B; Section IV.1 – one FC layer with 128 nodes) mapping to the claimed full connection layer; and activation functions throughout including ReLU, sigmoid, and tanh (Section II.B) mapping to the claimed activation layer. Adavanne further discloses that the SED output functions as a confidence measure that gates the DOA output, such that DOA estimates are chosen only when the SED output exceeds a threshold, establishing that both the detection and positioning models are operationally connected within a single integrated architecture (Section II.B; Section IV).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to apply Adavanne's SELDnet architecture – CNN feature extraction layers, BiGRU layers, fully connected layers, and activation functions – to the abnormal sound detection model and abnormal sound positioning model of the swine edge computing monitoring system taught by Sarzen and Berckmans. Both Adavanne and the system of Sarzen/Berckmans address the identical technical problem of detecting and localizing acoustic sound events from multi-channel audio. A person of ordinary skill in the art implementing a swine acoustic monitoring system would have been directly motivated to apply this established, state-of-the-art architecture to the detection and positioning models of the edge computing gateway, with a reasonable expectation of improved detection and localization accuracy. The combination requires only routine adaptation of a known architecture to a known application domain and yields only predictable results.
Regarding claim 9, Sarzen in view of Berckmans and Adavanne disclose a system for monitoring the abnormal state of the swines based on the edge computing according to claim 8, wherein the depth feature extraction layer comprises a plurality of convolution layers, batch normalization layers, activation function ReLu, pooling layers and Dropout (Adavanne explicitly discloses each of these components as elements of the CNN-based depth feature extraction layers of SELDnet. Specifically, Adavanne discloses multiple layers of 2D CNN – with the optimum configuration comprising three CNN layers with 64 nodes each – constituting the claimed plurality of convolution layers (Section II.B; Section IV.1). Adavanne discloses that after each CNN layer, output activations are normalized using batch normalization, constituting the claimed batch normalization layers (Section II.B). Adavanne discloses that each CNN layer uses a rectified linear unit (ReLU) activation, directly teaching the claimed activation function ReLU (Section II.B). Adavanne discloses that dimensionality is reduced using max-pooling along the frequency axis after each CNN layer, with a pooling configuration of (8, 8, 2) across the three layers, constituting the claimed pooling layers (Section II.B; Section IV.1). With respect to Dropout, Adavanne explicitly discloses dropout as a tuned regularization parameter within the same CNN architecture, evaluating dropout rates in the range of [0, 0.1, 0.2, 0.3, 0.4, 0.5] during hyperparameter optimization (Section III.D.1; Section IV.1). While Adavanne reports that the optimal configuration for its evaluated datasets used no dropout, Adavanne's explicit evaluation of dropout within this architecture establishes that dropout was a known and routinely considered component of CNN-based feature extraction layers for audio processing. It would have been obvious to one of ordinary skill in the art to include dropout as a regularization component in the depth feature extraction layer of the claimed system, as dropout was a well-established technique for reducing overfitting in convolutional neural networks, and its inclusion yields only the predictable result of improved model generalization.).
Allowable Subject Matter
Claims 1 – 2 and 4 allowed.
Reasons for Allowance
The following is an examiner’s statement of reasons for allowance: Claim 1 is allowed because the prior art of record does not teach or suggest, alone or in combination, a method for monitoring abnormal state of swines based on edge computing: performing standard normalization processing on a feature parameter set comprising amplitude spectrogram, decibel amplitude spectrogram and phase spectrogram; constructing a training set by sending pseudorandom sequences by point sound source in different pigsty scenes, performing correlation operation on the pseudorandom sequences at a receiving end and a sending end to obtain impulse response in a corresponding scene, collecting environmental background noises, and superimposing the background noises with different energies following convolution with different impulse responses to obtain a noisy signal set comprising multiple signal-to-noise ratios; and a mask-based inference workflow wherein the classification result is used as a mask to determine whether the abnormal sound is in an active state before inputting the decibel amplitude spectrogram and phase spectrogram into the abnormal sound positioning model, with ensemble learning performed on multi-channel outputs of both models.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RAJSHEED O BLACK-CHILDRESS whose telephone number is (571)270-7838. The examiner can normally be reached M to F, 10am to 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Quan-Zhen Wang can be reached at (571) 272-3114. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RAJSHEED O BLACK-CHILDRESS/Examiner, Art Unit 2685