DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in responsive to communication(s): original application filed on 03/29/2023, said application claims a priority filing date of 04/29/22. Claims 1-37 are pending. Claims 1, 14, and 25 are independent.
Claim Objections
Claims 8-9, 11-12, 14-16, 18, 21-24, 32-33, and 35-36 are objected to because of the following informalities:
in Claims 8, 24, and 32, lines 2-3, "... include at least one of fault mode diagnosis, fault localization, and operating condition at the time of fault detection" appears to be "... include at least one of fault mode diagnosis, fault localization, and operating condition at a time of fault detection";
in Claims 9 and 33, lines 1-2, "… wherein fault mode diagnosis includes … and fault localization includes …" appears to be "… wherein the fault mode diagnosis includes … and the fault localization includes …";
in Claims 11 and 35, line 2, "… a shortcut pathway is employed in the residual structure …" appears to be "… a shortcut pathway is employed in the RLU structure …" according to their respective based Claims 10 and 34;
in Claims 11 and 35, line 6, "… where x and y are the input and output of the residual structure …" appears to be "… where x and y are the input and the output of the RLU structure …";
in Claims 12 and 36, line 1, "… wherein the fault domain data …" appears to be "… wherein the bearing fault domain data …" according to Claim 1;
in Claim 14, line 9, "… integrating the grayscale images with the information maps to build …" appears to be "… integrating the 2D grayscale images with the information maps to build …";
in Claim 14, line 12, "… training the DR-CNN with …" appears to be "… training the multi-task DR-CNN with …";
in Claim 15, lines 1-3, "… wherein the segments are based on the sampling rate of the monitoring sensors and the rotating speed of monitored rotating components" appears to be "… wherein the segments are based on a sampling rate of the monitoring sensors and a rotating speed of the one or more monitored rotating components";
in Claim 16, lines 2-3, "… or the training epoch reaches a pre-determined threshold" appears to be "… or a training epoch reaches a pre-determined threshold";
in Claim18, line 1, "… wherein the DR-CNN includes …" appears to be "… wherein the multi-task DR-CNN includes …";
in Claim 21, lines 1-2, "… wherein the deep residual convolutional neural network (DR-CNN) includes …" appears to be "… wherein the multi-task deep residual convolutional neural network (DR-CNN) includes …";
in Claim 22, lines 5-6, "… the method further comprises including current operating conditions … to input to the DR-CNN" appears to be "… the method further comprises current operating conditions … to input to the multi-task DR-CNN";
in Claim 23, line 1, "A method of using the trained DR-CNN of claim 14 … providing input data sets of monitored rotating component sensor data to the trained DR-CNN, and operating the DR-CNN for outputting … based on such monitored sensor data" appears to be "A method of using the trained multi-task DR-CNN of claim 14 … providing input data sets of monitored rotating component sensor data to the trained multi-task DR-CNN, and operating the trained multi-task DR-CNN for outputting … based on the monitored rotating component sensor data ".
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-37 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "... fusing the pre-processed monitoring data with bearing fault domain data to build fused information maps; processing the fused information maps through a machine-learned enhanced discriminate feature learning based deep residual convolutional neural network (DR-CNN) model trained to diagnosis faults from fused information maps ..." in lines 6-10, which rendering the claim indefinite because it is unclear whether the third instance of "fused information maps" is the same or different to the first two instances of "fused information maps". Clarification is required.
Claims 2-13 are rejected for fully incorporating the deficiency of their respective base claims.
Claim 5 recites the limitation "... wherein the DR-CNN model is trained with dynamic training procedure using the fused information maps" in lines 1-2, which rendering the claim indefinite because "... fusing the pre-processed monitoring data with bearing fault domain data to build fused information maps; processing the fused information maps through a machine-learned enhanced discriminate feature learning based deep residual convolutional neural network (DR-CNN) model trained to diagnosis faults from fused information maps ..." is also recited in its based claim and it is unclear which instance of "fused information maps" (first or third instance?) in the based claim is referred by "the fused information maps" used to train the DNN-model recited here. Clarification is required.
Claim 11 recites the limitation "... a shortcut pathway is employed in the residual structure to connect the input and the output of the stacked layers directly ..." in lines 2-3 (see also Claim objections to Claim 11), which rendering the claim indefinite because "... as an output of the DR-CNN model ..." and "... wherein the deep residual convolutional neural network (DR-CNN) model includes a residual learning unit (RLU) structure having two convolutional layers, two Batch Normalization (BN) layers, and one ReLU activation layer" are also recited in its based Claims 1 and 10 respectively, and (1) there is insufficient antecedent basis for the limitation "the input"; (2) it is unclear whether "the stacked layers" is referred to "a residual learning unit (RLU) structure having two convolutional layers, two Batch Normalization (BN) layers, and one ReLU activation layer" recited in its based Claim 10; and (3) it is unclear whether "the output of the stacked layers" is the same or different to "an output of the DR-CNN model" recited in its based Claim 1 (NOTE: according to FIG. 5, an input/output of the RLU structure may be different to "an input/output of DR-CNN model"). Clarification is required.
Claim 11 recites the limitation "the integrated RLU" in line 4. There is insufficient antecedent basis for this limitation in the claim. Clarification is required.
Claim 11 recites the limitation "the residual function" in lines 6-7. There is insufficient antecedent basis for this limitation in the claim. Clarification is required.
Claim 11 recites the limitation "the residual mapping" in line 7. There is insufficient antecedent basis for this limitation in the claim. Clarification is required.
Claim 12 is rejected for fully incorporating the deficiency of their respective base claims.
Claim 13 recites the limitation "... current operating conditions to become part of the fused information maps to input to the DR-CNN model" in lines 1-2, which rendering the claim indefinite because "... fusing the pre-processed monitoring data with bearing fault domain data to build fused information maps; processing the fused information maps through a machine-learned enhanced discriminate feature learning based deep residual convolutional neural network (DR-CNN) model trained to diagnosis faults from fused information maps ..." is also recited in its based claim and it is unclear which instance of "fused information maps" (first or third instance?) in the based claim is referred by "the fused information maps" (including "current operating conditions") input to the DR-CNN model recited here. Clarification is required.
Claim 14 recites the limitation "... partitioning into segments 1-dimensional (1-D) monitoring data samples from monitoring sensors associated with one or more monitored rotating components; converting the monitoring data segments from different sensors into 2-dimensional (2-D) grayscale images ..." in lines 3-6, which rendering the claim indefinite because (1) it is unclear whether "monitoring sensors" and "different sensors" are the same or different; and (2) if "monitoring sensors" and "different sensors" are different, it is unclear how can "the monitoring data segments" from "different sensors" are referred to "segments" partitioned from "1D monitoring data samples" obtained from "monitoring sensors". Clarification is required.
Claim 14 recites the limitation "... training an enhanced discriminate feature learning based multi-task CNN for rotating component fault diagnosis … which are used as input to a multi-task deep residual convolutional neural network (DR-CNN) … training the DR-CNN with the fused information images by using a dynamic training strategy to learn fault diagnosis" in lines 1-13, which rendering the claim indefinite because (1) it is unclear "an enhanced discriminate feature learning based multi-task CNN" and "a multi-task deep residual convolutional neural network (DR-CNN)" are the same or different; and (2) it is unclear "rotating component fault diagnosis" and "fault diagnosis" are the same or different. Clarification is required.
Claims 15-24 are rejected for fully incorporating the deficiency of their respective base claims.
Claim 23 recites the limitation "... conduct rotating component fault diagnosis, comprising … for outputting fault diagnosis based on …" in lines 1-4, which rendering the claim indefinite because "... training an enhanced discriminate feature learning based multi-task CNN for rotating component fault diagnosis … training the DR-CNN with the fused information images … to learn fault diagnosis" is also recited in its based claim and it is unclear whether these instances of "rotating component fault diagnosis" and "fault diagnosis" are the same or different.
Claim 24 is rejected for fully incorporating the deficiency of their respective base claims.
Claim 24 recites the limitation "... wherein the rotating component fault diagnosis ..." in line 1, which rendering the claim indefinite because "... training an enhanced discriminate feature learning based multi-task CNN for rotating component fault diagnosis ..." and "... to conduct rotating component fault diagnosis ..." are also recited in its based Claims 1 and 23 respectively, and it is unclear which instance of "rotating component fault diagnosis" recited in its based claims is referred by "the rotating component fault diagnosis" recited here. Clarification is required.
Claim 25 recites the limitation "... fusing the pre-processed monitoring data with bearing fault domain data to build fused information maps; and a machine-learned enhanced discriminate feature learning based deep residual convolutional neural network (DR-CNN) model trained to diagnose rotating component faults from fused information maps, for receiving and processing the fused information maps ..." in lines , which rendering the claim indefinite because (1) i.
Claims 26-37 are rejected for fully incorporating the deficiency of their respective base claims.
Claim 29 recites the limitation "... wherein the DR-CNN model is trained with dynamic training procedure using the fused information maps" in lines 1-2, which rendering the claim indefinite because ".
Claim 35 recites the limitation "... a shortcut pathway is employed in the residual structure to connect the input and the output of the stacked layers directly ..." in lines 2-3 (see also Claim objections to Claim 35), which rendering the claim indefinite because "... a machine-learned enhanced discriminate feature learning based deep residual convolutional neural network (DR-CNN) model … for receiving and processing the fused information maps … for outputting one or more sets of fault information ..." and "... wherein the deep residual convolutional neural network (DR-CNN) model includes a residual learning unit (RLU) structure having two convolutional layers, two Batch Normalization (BN) layers, and one ReLU activation layer" are also recited in its based Claims 1 and 10 respectively, and (1) there is insufficient antecedent basis for the limitation "the input"; (2) it is unclear whether "the stacked layers" is referred to "a residual learning unit (RLU) structure having two convolutional layers, two Batch Normalization (BN) layers, and one ReLU activation layer" recited in its based Claim 10; and (3) it is unclear whether "the output of the stacked layers" is the same or different to "an output of the DR-CNN model" recited in its based Claim 25 (NOTE: according to FIG. 5, an input/output of the RLU structure may be different to "an input/output of DR-CNN model"). Clarification is required.
Claim 35 recites the limitation "the integrated RLU" in line 4. There is insufficient antecedent basis for this limitation in the claim. Clarification is required.
Claim 35 recites the limitation "the residual function" in line. There is insufficient antecedent basis for this limitation in the claim. Clarification is required.
Claim 35 recites the limitation "the residual mapping" in line . There is insufficient antecedent basis for this limitation in the claim. Clarification is required.
Claim 36 is rejected for fully incorporating the deficiency of their respective base claims.
Claim 37 recites the limitation "... current operating conditions as part of the fused information maps to be input to the DR-CNN model" in lines , which rendering the claim indefinite because ".
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 6-7, 10-12, 25-27, 30-31, and 34-36 are rejected under 35 U.S.C. 103 as being unpatentable over Su et al. (CN 114239384 A, pub. date: 03/25/2022), hereinafter Su in view of Liu et al. ("Attention Module Magnetic Flux Leakage Linked Deep Residual Network for Pipeline In-Line Inspection", Sensors 2022, 22, 2230, March 14, 2022, pp. 1-24), hereinafter Liu.
Independent Claims 1 and 25
Su discloses a method [system (Claim 25)] for rotating component fault diagnosis (Su, Abstract and ¶¶ [0004]-[0005] with FIG. 1: provide a rolling bearing fault diagnosis method based on a nonlinear metric prototype network, which comprises the steps of constructing a cascade attention prototype nonlinear metric network, performing classification training on the constructed network, performing data processing on data with diagnosis, inputting the data into the trained cascade attention prototype nonlinear metric network; the cascade attention prototype nonlinear metric network comprises a sample set division module, a prototype calculation module, a cascade attention mechanism learning module and a nonlinear metric strategy classification training module), comprising: [one or more processors (Su, Abstract and ¶¶ [0004]-[0005] with FIG. 1: inherited in a computer system user for constructing a cascade attention prototype nonlinear metric network, performing classification training on the constructed network, and performing bearing fault diagnosis using the trained cascade attention prototype nonlinear metric network) programmed for (Claim 25)]
receiving raw monitoring data from at least one monitoring sensor associated with one or more operating bearings [rotating components (Claim 25)] (Su, ¶ [0003]: the rolling bearing fault diagnosis method based on deep learning is rapidly developed in the past few years, and fault diagnosis and identification are carried out on vibration signals by utilizing strong feature dimension reduction and mode identification capability of a neural network; ¶¶ [0035]-[0039] with FIGS. 7-11: a schematic diagram of a vibration signal of a rolling bearing collected in a state (a)-(e) by the MFS experimental apparatus; ¶ [0080] with FIGS. 7-11: a comparison test is carried out by utilizing a vibration signal of a Machine Fault Simulator (MFS); the experiment simulates 5 health states of the rolling bearing, collects the belt end bearing Y-axis vibration signal of the simulator under the 44Hz conversion frequency, and the sampling frequency is 10240 Hz; each set of health status data was repeatedly collected 6 times; the original vibration waveforms for the five different conditions are shown in FIGS. 7-11);
pre-processing the raw monitoring data to generate pre-processed monitoring data (Su, ¶¶ [0051] and [0053]: processing vibration data of the rolling bearing to be diagnosed and identified; before dividing a sample set, normalizing original vibration signal samples; ¶ [0080]: after obtaining the vibration signals of five different states, data preprocessing is required for the vibration signals; and first, the vibration data with a length of 102400 is divided into 25 samples, each sample containing 4096 data points, so that the number of samples per class is 25 × 6 = 150);
fusing the pre-processed monitoring data with bearing fault domain data to build fused information maps (Su, ¶¶ [0006]-[0008], [0010]-[0014], [0050], and [0053]-[0060] with FIG. 2: dividing the sample set into a support set (i.e., for constructing fault domain data) and a query set (i.e., query samples from vibration signal) by using a sample set dividing module; inputting the divided data sets into a prototype calculation module to obtain feature maps corresponding to samples in the data sets using a feature extractor for embedding sample xi in the sample set L into feature space, and calculating class prototypes through the feature maps of the support sets (i.e., constructing fault domain data); for type c faults, prototype Pc is generated by using support set S; splicing/joining the feature map of the query set samples (i.e., from vibration signal collected) with the prototypes of all categories one by one (i.e., fault domain data constructed from the support set) based on the calculated prototypes, and extracting the long-distance correlation of the spliced/joined samples by adopting a cascade attention mechanism learning module);
processing the fused information maps through a machine-learned enhanced discriminate feature learning based deep [a machine-learned enhanced discriminate feature learning based deep (Claim 25)] (Su, ¶ [0009]: inputting the long-distance correlation extracted by the cascade attention mechanism learning module into a nonlinear metric strategy classification training module for classification training; ¶¶ [0049]-[0051] with FIG.1: 1st part of FIG. 1 is nonlinear metric prototype network training; training a nonlinear metric prototype network including (a) based on the limited label sample set, the limited label sample set is divided into a training set and a test set, wherein the training set is further divided into a support set and a query set, the samples are mapped to an embedding space through a prototype network, and various types of prototypes are calculated based on the support set; (b) splicing/joining the query samples and the class prototypes in the embedding space one by one, and sending the query samples and the class prototypes into a cascade attention module to extract non-local information; and (c) finally, the similarity between the sample and the prototype is better measured through a nonlinear metric module so as to improve the fault diagnosis performance; initializing all parameters of the nonlinear metric prototype network based on the steps, and feeding training samples through a gradient descent algorithm to train parameters of a network optimization model; ¶ [0059]: since the prototype network measures the similarity between a sample and a class prototype in a linear manner, the linear measurement is intended to directly calculate the distance between features by predefining a fixed metric (e.g. Euclidean distance), which requires that a feature extractor can extract obvious discriminant features as prototype representations, whereas a mechanical vibration signal is difficult to extract fault features with high recognizability under the condition of few labeled samples; secondly, the fixed linear metric cannot learn the non-linear relationship between complex signals, and the diagnostic performance thereof will be greatly reduced; aiming at the defects of prototype network linear measurement, a learnable nonlinear classifier is used for replacing a prototype network fixed linear metric method, class prototypes and query sample features are spliced/joined, nonlinear measurement is learned through a nonlinear neural network, and similarity scoring is carried out on each batch of spliced samples to complete sample category identification; ¶¶ [0015]-[0028] and [0060]-[0075] with FIGS. 2 and 6: the cascade attention mechanism learning module comprises a channel attention submodule and a space attention submodule; the cascade attention mechanism learning module performs convolution on the input spliced sample and extracts a feature F; splicing/joining a query sample and the C type prototype feature, and inputting the spliced/joined sample 1 (xi) into the convolution block in the cascade attention module, performing preliminary feature extraction to the spliced feature to obtain the feature map F [Symbol font/0xCE] RH×W×C, where H×W×C represents the height, width, and number of channels of the feature map, respectively; the convolution block adopted in the process of carrying out convolution on the input spliced sample by the cascade attention mechanism learning module comprises a convolution layer, a pooling layer, a BN layer and an activation function; in the cascade attention module, the feature map F [Symbol font/0xCE] RH×W×C is controlled flow into the channel attention and spatial attention modules, respectively (i.e., respectively inputting the feature F into a channel attention submodule and a space attention submodule), wherein the channel attention submodule self-adaptively adjusts feature values among channels, establishes a channel dependency relationship and obtains a channel attention feature Fc'; the space attention submodule focuses on the position information of the target sample in the input feature mapping and ignores the unimportant target features to obtain a space attention feature Fs'; finally, performing information fusion on the channel attention feature Fc' and spatial attention feature Fs', and then accumulating the fused feature information and the input feature F to obtain the long-distance correlation of the spliced sample; in the channel attention submodule, a global average pooling operation is firstly adopted to compress the feature F in a space dimension, and the space information of feature mapping is aggregated to generate a channel attention feature map Uc [Symbol font/0xCE] Rl×l×C, then pass through two convolution blocks to extract the nonlinear relationship between each channel, the channel dimensions of the two convolution blocks are first subjected to dimensionality reduction processing and then to dimensionality ascending processing, and then an activation function is used for obtaining a channel attention weight S; the internal network structure of the channel attention submodule is shown in FIG. 6, wherein CPBA represents the corresponding convolutional layer, pooling layer, BN layer and activation function, and CBA represents the convolutional layer, BN layer and activation function (i.e., the channel attention submodule comprises a global average pooling layer (G), a first convolution block (CBA), and a second convolution block (CBA), wherein each convolution block is composed of a convolution layer (C), a BN layer (B) and an activation function (A); the feature F is input into the global average pooling layer, the first convolution block, and the second convolution block which are cascaded to obtain a channel information structure S through extraction), and then multiplying the input feature F by a channel information structure S matrix, and fusing/adding the generated result with the feature information F to obtain the channel attention weighted feature Fc'; in the space attention module, firstly, a convolution layer is adopted to extract information from the feature F, and the output feature of the convolution layer are subjected to channel fusion to obtain a space attention feature map Us [Symbol font/0xCE] RHWl×l; the activation function is then used to obtain the spatial attention weight S'; the network structure of the spatial attention submodule is shown in FIG. 6, where CB represents the corresponding convolutional layer and BN layer (i.e., the spatial attention submodule comprises a third convolution block and a global average pooling layer, wherein the third convolution block is composed of a convolution layer and a BN layer; the feature F is input into the cascaded third convolution block and the global average pooling layer to extract a spatial information structure S'); then, the input feature F is multiplied by the space information structure S' matrix, and the generated result is fused/added with the feature information F to obtain the space attention weighted feature Fs [Symbol font/0xCE] RH×W×C; inputting the features extracted by the attention module into a nonlinear measurement module to realize effective few-shot learning (FSL) for bearing fault diagnosis, and conveying the spliced sample to the nonlinear metric module hr(C) through a series of continuous mapping of network layers, the module finally outputs a number C of scaler values Vj,r between 0 and 1 through softmax, wherein Vj,r represents similarity between query samples xjQ and a certain type of prototype PC; i.e. probability of query samples values belonging to the class; The difference between linear metric method (FIG.4) and nonlinear metric method (FIG. 5) based on the prototype network are shown in FIGS. 4-5; in order to improve the accuracy of the classifier, a network model is trained by minimizing the classification loss of class prototypes corresponding to the query samples and the support set; the mean square error is used as a loss function; calculating the mean square error LMSE using the similarity probability value Vj,r output through the procedures described above, and labels for the query samples yjQ with labels yrP belongs to the class prototype; finally, the network model is trained by minimizing the above equation (i.e., LMSE); after the class prototype is spliced with the feature map of the query sample, when directly inputting the spliced samples into a nonlinear measurement network, the long-distance correlation of the spliced samples with double-increased feature dimension cannot be captured because of the influence caused by the size of a receptive field; therefore, a cascade attention mechanism is used to extract the long-distance correlation of the spliced samples, so as to better extract the nonlinear relation between the sample and the prototype through the nonlinear measurement module; extract the feature maps through the prototype calculation module, calculate the prototype for the support set feature, splice/join the query sample feature and various prototypes one by one in the cascade attention module, then extract the long-distance correlation of the spliced sample through the cascade attention mechanism, and finally input the feature extracted by the cascade attention module into the nonlinear measurement module; thereby realizing the accurate and effective bearing fault diagnosis under the condition of small sample; ¶ [0094]: provide an improved FSL method of a rolling bearing fault diagnosis model aiming at an application scene of the shortage of fault labeled data, which is called as a cascade attention and nonlinear metric improved prototype network (CANM-ProNet); first, the prototype calculation module extracts feature maps of the support set and the query set, and calculates a prototype using the feature maps of the support set; the query feature map is then concatenated with each prototype and a cascade attention module is introduced to extract non-local information of the concatenated features; finally, a non-linear metric module is presented for better measuring the similarity between the samples and the prototype to improve fault diagnosis performance; numerous experiments have shown that this method is more efficient than other methods with fewer samples of faults); and providing, as an output of the D [, for outputting one or more sets of fault information regarding the one or more operating rotating components (Claim 25)] (Su, ¶¶ [0049]-[0051] with FIG.1: 2nd part of FIG. 1 is fault diagnosis and identification; fault diagnosis of rolling bearing under small sample includes (a) processing vibration data of the rolling bearing to be diagnosed and identified; (b) inputting identification data to be diagnosed into the trained nonlinear metric prototype network in the process 1; and (c) outputting a fault diagnosis result by the trained nonlinear metric prototype network; ¶¶ [0076]-[0079]: the identification process of fault diagnosis/identification comprises the following steps: S21, processing identification data to be diagnosed; s22, inputting identification data to be diagnosed into a nonlinear metric prototype network, and outputting a fault diagnosis result by the network; the nonlinear metric prototype network is a small sample supervised learning model and mainly comprises a prototype calculation module, a cascade attention module, and a nonlinear metric module).
Su fails to explicitly disclose a deep residual convolutional neural network (DR-CNN) model to diagnosis faults.
Liu teaches a system and a method for identification of defects/faults using convolutional neural network (Liu, 1st paragraph of Section 2.1.1 in Pages 3-4 and Abstract in Page 1), wherein a deep residual convolutional neural network (DR-CNN) model to diagnosis faults (Liu, Abstract in Page 1: a pipeline MFL inspection signal identification method based on improved deep residual convolutional neural network and attention module is proposed; an improved deep residual network based on the VGG16 convolution neural network is constructed to automatically learn the features from the MFL image signals and perform the identification of pipeline features and defects; 4th paragraph – 6th paragraph of Section 1 in Pages 2-3: deep learning methods have become a hotspot in the research of signal identification and fault diagnosis in recent years; to further improve the identification accuracy for the complex pipeline MFL signals, a pipeline MFL inspection feature identification model based on an improved deep residual convolutional neural network is proposed; the proposed method not only automatically learns the features from the MFL inspection images and performs the classification and identification of pipeline features and defects such as welds, tees, flanges, and corrosion, but also solves the problems of the great influence of noise, compound features, and other factors on the feature identification results in the process of in-line inspection; the proposed method effectively improves the classification of pipeline features, and provides an effective method for pipeline features identification; an MFL in-line inspection method based on attention module and convolution residual modules is proposed, which effectively improve accuracy and efficiency; aiming at the influence of the complex operating environment, high noises, composite defects to MFL in-line inspection of oil and gas pipelines, attention module composed of channel attention and spatial attention are designed to fully extract MFL image feature information; to solve the problem of gradient dispersion caused by the increase of the number of network layers, an improved residual convolutional neural network is constructed to reduce the error of the deep network as well as the amount of calculation parameters, and effectively improve the training efficiency; Section 2.1.1 with FIG. 1 in Pages 3-5: the improved CNN with strong generalization, feature extraction and identification can effectively identify the fault types of rolling bearings; Section 2.2.3 with FIGS. 7-8 in Pages 9-11: to improve the deep convolution network and propose a new mechanism to enhance the signal feature extraction ability of the network, the attention model was derived from a human visual attention model; while processing data, the visual system quickly focuses on the target areas that need to be focused by scanning the global scene, and allocates limited computing resources to these key parts; this mechanism can greatly reduce the amount of data to be processed, ignore unimportant information, and provide more manageable and relevant information for higher-level perceptual reasoning and complex visual processing; it is one of the core technologies in deep learning worthy of attention and in-depth understanding; the convolutional block attention module (CBAM) is an attention module combining spatial with channel information; CBAM adopts max-pooling and average-pooling to generate weights through the channel and spatial dimensions; adding the attention mechanism module can further extract the interested small defects target area from the background, thus the network can better learn small target defects; at the same time, the interference of the background to the target is suppressed, thus improving the learning ability of the network to the detailed features of small targets and enhance the ability of feature learning; the method introduces the attention mechanism and designs the spatial attention module (Spatial_AM) and the channel attention module (Channel_AM); in Channel_AM, the property of maximum pooling is used to capture the inter class information between MFL image pixels, and the average pooling is used to capture the intra class information between pixels; these two information as weights are applied to the original feature map as attention to assist feature extraction; the module is connected between the feature map extraction module and the feature map decoding module; at the same time, the Spatial_AM is also designed; by using the spatial attention mechanism composed of global pooling, convolution, and activation function, the semantic information extraction is further refined, and the information is multiplied with the original feature map as a weight.; Channel attention module: when extracting features in the channel dimension, average pooling and max pooling are considered simultaneously; Figure 7 shows the design scheme of the channel attention module; the module consists of convolution, batch regularization, and an activation function, which can extract mixed information by integrating channel and semantic information; then, the average pooling module, convolution and activation function ReLU are adopted to process the features, where ADD is the addition operation and MUL is the multiplication; operation, to obtain the function Xcavg; at the same time, the max pooling module, convolution and activation function ReLU are used in parallel with the average pooling module for another feature extraction to obtain the function XcMax; the designed attention feature map has features of both average pooling and max pooling; the attention feature is multiplied with the input feature map and superimposed with the input feature to as the weight to influence the input feature map; finally, a structure similar to the jump connection was used to reduce the negative impact of the attention module on the input feature map, and the Sigmoid activation function was used to output the final feature map; the designed module can not only efficiently guide the acquisition of intraclass information through the average pooling operation, but also extracts more edge information through the max pooling operation, which can efficiently improve the acquisition of feature information; spatial attention module: by using the spatial attention module, useful information in the input image can be focused on; Figure 8 shows the designed spatial attention module, which focuses on the spatial or semantic feature information in the feature map; by use of global average pooling, the length and width of the feature map are compressed into one, leaving only the spatial information; then, the convolutional layer is used to learn the association between spatial information and classification information (semantic information), and batch regularization and activation function Sigmoid are used to transform this association into nonlinear change; to avoid excessive loss of feature information caused by pooling, it was multiplied by the feature map without average pooling; the result of multiplication is input to the next module as the weight to influence the input feature map, so as to complete the task of refining semantic information; Section 2.3 with FIGS. 9-11 and Table 1 in Pages 11-14: a new deep neural network model is effectively constructed by constructing a residual network model and designing an attention model to enhance the feature learning ability of MFL in-line inspection signals to improve the identification accuracy; to solve the problems of gradient dispersion or explosion and network degradation caused by network deep stacking, Kaiming He et al. proposed a new network structure, namely, residual network (ResNet), which constructs a new deep network by introducing a residual block; the essence of the ResNet design is to ensure that the internal structure of the model has the ability of identity mapping so that the deep network has the same performance as the shallow network; through identity mapping, there is no degradation due to continued stacking in the process of stacking the network; the block structure of the Identity Residual module (Identity_RES) is shown in Figure 9; the output H(x) in model is: H(x) = F(x) + x, where F(x) is the residual mapping after learning, H(x) is the low-level mapping of the partial fitting, and x is the input vector; usually, F(x) is expressed as F(x, { Wn }) to highlight the relationship between the input weights and update weights; therefore, the n-th residual unit can be expressed as in Equations (7)-(8), where xn+1 and xn represent the output and input of the n-th residual unit, respectively; h(xn) represents unit mapping, and fReLU is the ReLU activation function; as can be seen from Equation (9), the features for learning from layer n to layer N are shown in Equations (9)-(10), where Equation (10) represents the actual updated gradient of the loss when passing through the n-th layer; the first part in the formula represents the preserved gradient of directly transmitting the original features through the identity channel; the second part is the residual gradient related to the weight parameters of the residual network; if the output size of the previous layer and the input size of the current layer do not match each other, it is necessary to add a convolutional layer to match the output of the previous layer, that is, the Convolutional Residual module (Conv_RES); the structure of the convolutional residual module is shown as in Figure 10; an MFL signal identification method based on improved residual convolutional neural network is proposed, in which a new network is constructed by improving the residual network and introducing an attention module; the feature identification model proposed in this method has the following advantages: (a) a convolution network is adopted to extract features from the original data, which reduces the difficulty of feature extraction and enhances the universal applicability of pipeline defects and pipeline features; (b) the attention layer is added to obtain the weighted feature map under the joint action of channel attention and spatial attention to further extract the image feature information and reduce the noises impact on the feature identification results during the inspection in the pipeline; and (c) two different residual modules (identity and convolution) are introduced to deepen the depth of the deep network but to effectively reduce the computing number of parameters, decrease the errors of the deep network, save the training time, and improve the training effect; the improved residual convolutional neural network model based on VGG16 proposed in this method is illustrated in Figure 11; the input of the entire network is a pseudo-color image of the MFL in-line inspection; after data preprocessing, the image size was set to 112 × 112 × 3; after normalization, the RGB value is converted to the range of (0, 1); the normalized image is input to the first convolutional layer which has sixteen 3 × 3 convolution kernels with a stride of 1, then is input into Channel_AM and Spatial_AM after passing through the batch standardization layer and activation layer; it then enters three consecutive identical residual modules, each of which includes 16 convolution kernels with a size of 3 × 3 and a stride of 1, and has an output of 112 × 112 × 16; next, the feature map is fed into the convolutional residual module, and on the one hand, the feature map A is obtained through ReLU activation function → convolution → batch standardization → ReLU activation function → convolution → batch standardization, on the other hand, feature map B is obtained after convolution → batch standardization; more features can be extracted from feature maps A and B through the superposition of the merging layer, and the size of the output feature map was 56 × 56 × 32; then, the feature map passes through two identical residual modules, one convolutional residual module, and two identical residual modules, and an output image of 28 × 28 × 64 is obtained; the obtained feature map was fed into the global mean pooling layer to reduce the number of parameters and overfitting; finally, it was connected to the full connection layer using Softmax for classification; the specific parameters for the feature identification and classification of the actual network structure are listed in Table 1).
Su and Liu are analogous art because they are from the same field of endeavor, a system and a method for identification of defects/faults using convolutional neural network. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Liu to Su. Motivation for doing so would reduce the error of the deep network as well as the amount of calculation parameters, and effectively improve the training efficiency .
Claims 2 and 26
Su in view of Liu discloses all the elements as stated in Claims 1 and 25 respectively and further discloses wherein the DR-CNN model includes two different attention modules employed to enhance learning ability of fault related discriminate features (Su, ¶¶ [0015]-[0026] and [0060]-[0067] with FIGS. 2 and 6: the cascade attention mechanism learning module comprises a channel attention submodule and a space attention submodule; the cascade attention mechanism learning module performs convolution on the input spliced sample and extracts a feature F; respectively inputting the feature F into a channel attention submodule and a space attention submodule, wherein the channel attention submodule adaptively adjusts feature values among channels, establishes a channel dependency relationship and obtains a channel attention feature Fc'; the space attention submodule focuses on the position information of the target sample in the input feature mapping to obtain a space attention feature Fs'; performing information fusion on attention feature of channel Fc' and spatial attention feature Fs', and then accumulating the fused feature information and the input feature F to obtain the long-distance correlation of the spliced sample; the channel attention submodule comprises a global average pooling layer, a first convolution block, and a second convolution block, wherein each convolution block is composed of a convolution layer, a BN layer and an activation function; the feature F is input into the global average pooling layer, the first convolution block, and the second convolution block which are cascaded to obtain a channel information structure body S through extraction, and the matrix product of the feature F unified by the channel information structure body S and the feature F are added to be used as the output of the channel attention submodule; the spatial attention submodule comprises a third rolling block and a global average pooling layer, wherein the third rolling block is composed of a rolling layer and a BN layer; the feature F is input into the cascaded third rolling block and the global average pooling layer to extract a spatial information structure S'; the value obtained by multiplying the spatial information structure S' by the input feature F is added with the feature F to obtain a spatial attention feature Fs') (Liu, Section 2.2.3 with FIGS. 7-8 in Pages 9-11: to improve the deep convolution network and propose a new mechanism to enhance the signal feature extraction ability of the network, the attention model was derived from a human visual attention model; while processing data, the visual system quickly focuses on the target areas that need to be focused by scanning the global scene, and allocates limited computing resources to these key parts; this mechanism can greatly reduce the amount of data to be processed, ignore unimportant information, and provide more manageable and relevant information for higher-level perceptual reasoning and complex visual processing; it is one of the core technologies in deep learning worthy of attention and in-depth understanding; the convolutional block attention module (CBAM) is an attention module combining spatial with channel information; CBAM adopts max-pooling and average-pooling to generate weights through the channel and spatial dimensions; adding the attention mechanism module can further extract the interested small defects target area from the background, thus the network can better learn small target defects; at the same time, the interference of the background to the target is suppressed, thus improving the learning ability of the network to the detailed features of small targets and enhance the ability of feature learning; the method introduces the attention mechanism and designs the spatial attention module (Spatial_AM) and the channel attention module (Channel_AM); in Channel_AM, the property of maximum pooling is used to capture the inter class information between MFL image pixels, and the average pooling is used to capture the intra class information between pixels; these two information as weights are applied to the original feature map as attention to assist feature extraction; the module is connected between the feature map extraction module and the feature map decoding module; at the same time, the Spatial_AM is also designed; by using the spatial attention mechanism composed of global pooling, convolution, and activation function, the semantic information extraction is further refined, and the information is multiplied with the original feature map as a weight.; Channel attention module: when extracting features in the channel dimension, average pooling and max pooling are considered simultaneously; Figure 7 shows the design scheme of the channel attention module; the module consists of convolution, batch regularization, and an activation function, which can extract mixed information by integrating channel and semantic information; then, the average pooling module, convolution and activation function ReLU are adopted to process the features, where ADD is the addition operation and MUL is the multiplication; operation, to obtain the function Xcavg; at the same time, the max pooling module, convolution and activation function ReLU are used in parallel with the average pooling module for another feature extraction to obtain the function XcMax; the designed attention feature map has features of both average pooling and max pooling; the attention feature is multiplied with the input feature map and superimposed with the input feature to as the weight to influence the input feature map; finally, a structure similar to the jump connection was used to reduce the negative impact of the attention module on the input feature map, and the Sigmoid activation function was used to output the final feature map; the designed module can not only efficiently guide the acquisition of intraclass information through the average pooling operation, but also extracts more edge information through the max pooling operation, which can efficiently improve the acquisition of feature information; spatial attention module: by using the spatial attention module, useful information in the input image can be focused on; Figure 8 shows the designed spatial attention module, which focuses on the spatial or semantic feature information in the feature map; by use of global average pooling, the length and width of the feature map are compressed into one, leaving only the spatial information; then, the convolutional layer is used to learn the association between spatial information and classification information (semantic information), and batch regularization and activation function Sigmoid are used to transform this association into nonlinear change; to avoid excessive loss of feature information caused by pooling, it was multiplied by the feature map without average pooling; the result of multiplication is input to the next module as the weight to influence the input feature map, so as to complete the task of refining semantic information).
Claims 3 and 27
Su in view of Liu discloses all the elements as stated in Claims 2 and 26 respectively and further discloses wherein the two different attention models comprise a Channel Attention Module (CAM) and a Non-local Attention Module (NLAM) (Su, ¶¶ [0050] and [0094]: splicing the query samples and the class prototypes in the embedding space one by one, and sending the query samples and the class prototypes into a cascade attention module to extract non-local information ; the query feature map is then concatenated with each prototype and a cascade attention module is introduced to extract non-local information of the concatenated features; ¶¶ [0015]-[0026] and [0060]-[0067] with FIGS. 2 and 6: the cascade attention mechanism learning module comprises a channel attention submodule and a space attention submodule (i.e., non-local information attention submodule); the cascade attention mechanism learning module performs convolution on the input spliced sample and extracts a feature F; respectively inputting the feature F into a channel attention submodule and a space attention submodule, wherein the channel attention submodule adaptively adjusts feature values among channels, establishes a channel dependency relationship and obtains a channel attention feature Fc'; the space attention submodule focuses on the position information of the target sample in the input feature mapping to obtain a space attention feature Fs'; performing information fusion on attention feature of channel Fc' and spatial attention feature Fs', and then accumulating the fused feature information and the input feature F to obtain the long-distance correlation of the spliced sample; the channel attention submodule comprises a global average pooling layer, a first convolution block, and a second convolution block, wherein each convolution block is composed of a convolution layer, a BN layer and an activation function; the feature F is input into the global average pooling layer, the first convolution block, and the second convolution block which are cascaded to obtain a channel information structure body S through extraction, and the matrix product of the feature F unified by the channel information structure body S and the feature F are added to be used as the output of the channel attention submodule; the spatial attention submodule comprises a third rolling block and a global average pooling layer, wherein the third rolling block is composed of a rolling layer and a BN layer; the feature F is input into the cascaded third rolling block and the global average pooling layer to extract a spatial information structure S'; the value obtained by multiplying the spatial information structure S' by the input feature F is added with the feature F to obtain a spatial attention feature Fs') (Liu, Section 2.2.3 with FIGS. 7-8 in Pages 9-11: to improve the deep convolution network and propose a new mechanism to enhance the signal feature extraction ability of the network, the attention model was derived from a human visual attention model; while processing data, the visual system quickly focuses on the target areas that need to be focused by scanning the global scene, and allocates limited computing resources to these key parts; this mechanism can greatly reduce the amount of data to be processed, ignore unimportant information, and provide more manageable and relevant information for higher-level perceptual reasoning and complex visual processing; it is one of the core technologies in deep learning worthy of attention and in-depth understanding; the convolutional block attention module (CBAM) is an attention module combining spatial with channel information; CBAM adopts max-pooling and average-pooling to generate weights through the channel and spatial dimensions; adding the attention mechanism module can further extract the interested small defects target area from the background, thus the network can better learn small target defects; at the same time, the interference of the background to the target is suppressed, thus improving the learning ability of the network to the detailed features of small targets and enhance the ability of feature learning; the method introduces the attention mechanism and designs the spatial attention module (Spatial_AM) and the channel attention module (Channel_AM); in Channel_AM, the property of maximum pooling is used to capture the inter class information between MFL image pixels, and the average pooling is used to capture the intra class information between pixels; these two information as weights are applied to the original feature map as attention to assist feature extraction; the module is connected between the feature map extraction module and the feature map decoding module; at the same time, the Spatial_AM is also designed; by using the spatial attention mechanism composed of global pooling, convolution, and activation function, the semantic information extraction is further refined, and the information is multiplied with the original feature map as a weight.; Channel attention module: when extracting features in the channel dimension, average pooling and max pooling are considered simultaneously; Figure 7 shows the design scheme of the channel attention module; the module consists of convolution, batch regularization, and an activation function, which can extract mixed information by integrating channel and semantic information; then, the average pooling module, convolution and activation function ReLU are adopted to process the features, where ADD is the addition operation and MUL is the multiplication; operation, to obtain the function Xcavg; at the same time, the max pooling module, convolution and activation function ReLU are used in parallel with the average pooling module for another feature extraction to obtain the function XcMax; the designed attention feature map has features of both average pooling and max pooling; the attention feature is multiplied with the input feature map and superimposed with the input feature to as the weight to influence the input feature map; finally, a structure similar to the jump connection was used to reduce the negative impact of the attention module on the input feature map, and the Sigmoid activation function was used to output the final feature map; the designed module can not only efficiently guide the acquisition of intraclass information through the average pooling operation, but also extracts more edge information through the max pooling operation, which can efficiently improve the acquisition of feature information; spatial attention module: by using the spatial attention module, useful information in the input image can be focused on; Figure 8 shows the designed spatial attention module, which focuses on the spatial or semantic feature information in the feature map; by use of global average pooling, the length and width of the feature map are compressed into one, leaving only the spatial information; then, the convolutional layer is used to learn the association between spatial information and classification information (semantic information), and batch regularization and activation function Sigmoid are used to transform this association into nonlinear change; to avoid excessive loss of feature information caused by pooling, it was multiplied by the feature map without average pooling; the result of multiplication is input to the next module as the weight to influence the input feature map, so as to complete the task of refining semantic information).
Claims 6 and 30
Su in view of Liu discloses all the elements as stated in Claims 1 and 25 respectively and further discloses wherein the at least one monitoring sensor comprises a plurality of sensors associated with a plurality of bearings, said sensors including at least one of vibration sensors, triaxial vibration sensors, and Acoustic Emission (AE) sensors (Su, ¶ [0003]: the rolling bearing fault diagnosis method based on deep learning is rapidly developed in the past few years, and fault diagnosis and identification are carried out on vibration signals by utilizing strong feature dimension reduction and mode identification capability of a neural network; ¶¶ [0035]-[0039] with FIGS. 7-11: a schematic diagram of a vibration signal of a rolling bearing collected in a state (a)-(e) by the MFS experimental apparatus (i.e., vibration signals are collected by vibration sensor in the MFS experimental apparatus); ¶ [0080] with FIGS. 7-11: a comparison test is carried out by utilizing a vibration signal of a Machine Fault Simulator (MFS); the experiment simulates 5 health states of the rolling bearing, collects the belt end bearing Y-axis vibration signal of the simulator under the 44Hz conversion frequency, and the sampling frequency is 10240 Hz; each set of health status data was repeatedly collected 6 times; The original vibration waveforms for the five different conditions are shown in FIGS. 7-11).
Claims 7 and 31
Su in view of Liu discloses all the elements as stated in Claims 6 and 30 respectively and further discloses wherein the plurality of sensors is placed in different directions (Liu, Section 2.2.1 with FIG. 3 in Pages 6-7: different numbers of Hall sensors covering the circumference of the pipeline; i.e., sensors is placed in different directions around the circumference of the pipeline) and have a pre-determined sampling rate (Su, ¶ [0080] with FIGS. 7-11: a comparison test is carried out by utilizing a vibration signal of a Machine Fault Simulator (MFS); the experiment simulates 5 health states of the rolling bearing, collects the belt end bearing Y-axis vibration signal of the simulator under the 44Hz conversion frequency, and the sampling frequency is 10240 Hz; each set of health status data was repeatedly collected 6 times; the original vibration waveforms for the five different conditions are shown in FIGS. 7-11).
Claims 10 and 34
Su in view of Liu discloses all the elements as stated in Claims 1 and 25 respectively and further discloses wherein the deep residual convolutional neural network (DR-CNN) model includes a residual learning unit (RLU) structure having two convolutional layers, two Batch Normalization (BN) layers, and one ReLU activation layer (Liu, Section 2.3 with FIGS. 9-11 and Table 1 in Pages 11-14: a new deep neural network model is effectively constructed by constructing a residual network model and designing an attention model to enhance the feature learning ability of MFL in-line inspection signals to improve the identification accuracy; to solve the problems of gradient dispersion or explosion and network degradation caused by network deep stacking, Kaiming He et al. proposed a new network structure, namely, residual network (ResNet), which constructs a new deep network by introducing a residual block; the essence of the ResNet design is to ensure that the internal structure of the model has the ability of identity mapping so that the deep network has the same performance as the shallow network; through identity mapping, there is no degradation due to continued stacking in the process of stacking the network; the block structure of the Identity Residual module (Identity_RES) is shown in Figure 9 (i.e., the Identity Residual module having two convolutional layers, two Batch Normalization (BN) layers, and one ReLU activation layer); the output H(x) in model is: H(x) = F(x) + x, where F(x) is the residual mapping after learning, H(x) is the low-level mapping of the partial fitting, and x is the input vector; usually, F(x) is expressed as F(x, { Wn }) to highlight the relationship between the input weights and update weights; therefore, the n-th residual unit can be expressed as in Equations (7)-(8), where xn+1 and xn represent the output and input of the n-th residual unit, respectively; h(xn) represents unit mapping, and fReLU is the ReLU activation function; as can be seen from Equation (9), the features for learning from layer n to layer N are shown in Equations (9)-(10), where Equation (10) represents the actual updated gradient of the loss when passing through the n-th layer; the first part in the formula represents the preserved gradient of directly transmitting the original features through the identity channel; the second part is the residual gradient related to the weight parameters of the residual network; if the output size of the previous layer and the input size of the current layer do not match each other, it is necessary to add a convolutional layer to match the output of the previous layer, that is, the Convolutional Residual module (Conv_RES); the structure of the Convolutional Residual module is shown as in Figure 10; an MFL signal identification method based on improved residual convolutional neural network is proposed, in which a new network is constructed by improving the residual network and introducing an attention module; the feature identification model proposed in this method has the following advantages: (a) a convolution network is adopted to extract features from the original data, which reduces the difficulty of feature extraction and enhances the universal applicability of pipeline defects and pipeline features; (b) the attention layer is added to obtain the weighted feature map under the joint action of channel attention and spatial attention to further extract the image feature information and reduce the noises impact on the feature identification results during the inspection in the pipeline; and (c) two different residual modules (identity and convolution) are introduced to deepen the depth of the deep network but to effectively reduce the computing number of parameters, decrease the errors of the deep network, save the training time, and improve the training effect; the improved residual convolutional neural network model based on VGG16 proposed in this method is illustrated in Figure 11; the input of the entire network is a pseudo-color image of the MFL in-line inspection; after data preprocessing, the image size was set to 112 × 112 × 3; after normalization, the RGB value is converted to the range of (0, 1); the normalized image is input to the first convolutional layer which has sixteen 3 × 3 convolution kernels with a stride of 1, then is input into Channel_AM and Spatial_AM after passing through the batch standardization layer and activation layer; it then enters three consecutive identical residual modules, each of which includes 16 convolution kernels with a size of 3 × 3 and a stride of 1, and has an output of 112 × 112 × 16; next, the feature map is fed into the convolutional residual module, and on the one hand, the feature map A is obtained through ReLU activation function → convolution → batch standardization → ReLU activation function → convolution → batch standardization, on the other hand, feature map B is obtained after convolution → batch standardization; more features can be extracted from feature maps A and B through the superposition of the merging layer, and the size of the output feature map was 56 × 56 × 32; then, the feature map passes through two identical residual modules, one convolutional residual module, and two identical residual modules, and an output image of 28 × 28 × 64 is obtained; the obtained feature map was fed into the global mean pooling layer to reduce the number of parameters and overfitting; finally, it was connected to the full connection layer using Softmax for classification; the specific parameters for the feature identification and classification of the actual network structure are listed in Table 1).
Claims 11 and 35
Su in view of Liu discloses all the elements as stated in Claims 10 and 34 respectively and further discloses wherein: a shortcut pathway is employed in the residual structure to connect the input and the output of the stacked layers directly; and the integrated RLU is defined as: y = F(xi, Wi) + x, where x and y are the input and output of the residual structure, respectively, F is the residual function, which represents the residual mapping to be learned, F(x, Wi) + x is operated by the shortcut pathway connection and element-wise addition (Liu, Section 2.3 with FIGS. 9-11 and Table 1 in Pages 11-14: a new deep neural network model is effectively constructed by constructing a residual network model and designing an attention model to enhance the feature learning ability of MFL in-line inspection signals to improve the identification accuracy; to solve the problems of gradient dispersion or explosion and network degradation caused by network deep stacking, Kaiming He et al. proposed a new network structure, namely, residual network (ResNet), which constructs a new deep network by introducing a residual block; the essence of the ResNet design is to ensure that the internal structure of the model has the ability of identity mapping so that the deep network has the same performance as the shallow network; through identity mapping, there is no degradation due to continued stacking in the process of stacking the network; the block structure of the Identity Residual module (Identity_RES) is shown in Figure 9 (i.e., the Identity Residual module having two convolutional layers, two Batch Normalization (BN) layers, and one ReLU activation layer); the output H(x) in model is: H(x) = F(x) + x, where F(x) is the residual mapping after learning, H(x) is the low-level mapping of the partial fitting, and x is the input vector; usually, F(x) is expressed as F(x, { Wn }) to highlight the relationship between the input weights and update weights; therefore, the n-th residual unit can be expressed as in Equations (7)-(8), where xn+1 and xn represent the output and input of the n-th residual unit, respectively; h(xn) represents unit mapping, and fReLU is the ReLU activation function; as can be seen from Equation (9), the features for learning from layer n to layer N are shown in Equations (9)-(10), where Equation (10) represents the actual updated gradient of the loss when passing through the n-th layer; the first part in the formula represents the preserved gradient of directly transmitting the original features through the identity channel; the second part is the residual gradient related to the weight parameters of the residual network; if the output size of the previous layer and the input size of the current layer do not match each other, it is necessary to add a convolutional layer to match the output of the previous layer, that is, the Convolutional Residual module (Conv_RES); the structure of the Convolutional Residual module is shown as in Figure 10; an MFL signal identification method based on improved residual convolutional neural network is proposed, in which a new network is constructed by improving the residual network and introducing an attention module; the feature identification model proposed in this method has the following advantages: (a) a convolution network is adopted to extract features from the original data, which reduces the difficulty of feature extraction and enhances the universal applicability of pipeline defects and pipeline features; (b) the attention layer is added to obtain the weighted feature map under the joint action of channel attention and spatial attention to further extract the image feature information and reduce the noises impact on the feature identification results during the inspection in the pipeline; and (c) two different residual modules (identity and convolution) are introduced to deepen the depth of the deep network but to effectively reduce the computing number of parameters, decrease the errors of the deep network, save the training time, and improve the training effect; the improved residual convolutional neural network model based on VGG16 proposed in this method is illustrated in Figure 11; the input of the entire network is a pseudo-color image of the MFL in-line inspection; after data preprocessing, the image size was set to 112 × 112 × 3; after normalization, the RGB value is converted to the range of (0, 1); the normalized image is input to the first convolutional layer which has sixteen 3 × 3 convolution kernels with a stride of 1, then is input into Channel_AM and Spatial_AM after passing through the batch standardization layer and activation layer; it then enters three consecutive identical residual modules, each of which includes 16 convolution kernels with a size of 3 × 3 and a stride of 1, and has an output of 112 × 112 × 16; next, the feature map is fed into the convolutional residual module, and on the one hand, the feature map A is obtained through ReLU activation function → convolution → batch standardization → ReLU activation function → convolution → batch standardization, on the other hand, feature map B is obtained after convolution → batch standardization; more features can be extracted from feature maps A and B through the superposition of the merging layer, and the size of the output feature map was 56 × 56 × 32; then, the feature map passes through two identical residual modules, one convolutional residual module, and two identical residual modules, and an output image of 28 × 28 × 64 is obtained; the obtained feature map was fed into the global mean pooling layer to reduce the number of parameters and overfitting; finally, it was connected to the full connection layer using Softmax for classification; the specific parameters for the feature identification and classification of the actual network structure are listed in Table 1).
Claims 12 and 36
Su in view of Liu discloses all the elements as stated in Claims 11 and 35 respectively and further discloses wherein the fault domain data includes at least one of fault mechanisms, expert empirical knowledge, rotating mechanisms, fault characteristics, and common fault patterns that can be extracted from historical monitoring data (Su, ¶¶ [0006]-[0008], [0010]-[0014], [0050], and [0053]-[0060] with FIG. 2: dividing the sample set into a support set (i.e., for constructing fault domain data) and a query set (i.e., query samples from vibration signal) by using a sample set dividing module; inputting the divided data sets into a prototype calculation module to obtain feature maps corresponding to samples in the data sets using a feature extractor for embedding sample xi in the sample set L into feature space, and calculating class prototypes through the feature maps of the support sets (i.e., constructing fault domain data); for type c faults, prototype Pc is generated by using support set S; splicing/joining the feature map of the query set samples (i.e., from vibration signal collected) with the prototypes of all categories one by one (i.e., fault domain data constructed from the support set) based on the calculated prototypes, and extracting the long-distance correlation of the spliced/joined samples by adopting a cascade attention mechanism learning module; ¶ [0082] with Table 1: health status of rolling bearings: State: N represents normal bearing; State IF represents Inner-Race Fault; State: OF represents Outer-Race Fault; State: BF represents Rolling-Ball Fault; State: MF represents Composite-Bearing Fault).
Claims 4-5, 8-9, 13, 28-29, 32-33, and 37 are rejected under 35 U.S.C. 103 as being unpatentable over Su in view of Liu as applied to Claims 2, 4, 1, 26, 28, and 25 respectively above, and further in view of Guo et al. ("Multitask Convolutional Neural Network With Information Fusion for Bearing Fault Diagnosis and Localization", IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 67, NO. 9, September 25, 2019), hereinafter Guo.
Claims 4 and 28
Su in view of Liu discloses all the elements as stated in Claims 2 and 26 respectively and except failing to explicitly disclose wherein the DR-CNN model includes two classifiers employed for multi-task diagnosis.
Guo teaches a system and a method related to fault diagnosis using Convolutional Neural Network (Guo, Abstract), wherein the DR-CNN model includes two classifiers employed for multi-task diagnosis (Guo, Abstract in Page 8005: propose a rolling element bearing fault diagnosis and localization approach based on multitask convolutional neural network (CNN) with information fusion; domain knowledge, operating conditions, and vibration signals are fused into a three-dimensional input that can be processed well by CNN; a multitask CNN with dynamic training rates is constructed to simultaneously accomplish two tasks, fault diagnosis and localization; Section 1 in Pages 8005-8006: most deep learning-based fault diagnostic methods are totally data driven, ignore the domain knowledge that has been developed and used for fault diagnosis in the last decades; the operating conditions of the system, including rotating speed and load, have significant influence on vibration signals; to make full use of data-driven methods, it is desirable to integrate system and operating information; domain knowledge, such as fault mechanism and characteristics, must be integrated with data to make sure the deep learning methods converge based on fault characteristics; most rotating machinery have more than one bearing; if a fault happens in one bearing, it is important to distinguish the fault type and locate the faulty bearing; this will not only reduce the labor cost and downtime, but also optimize the logistics; vibration-based condition monitoring systems often use multiple sensors in which each sensor monitors one component; sensor fusion is, therefore, critical to increase the diagnosis accuracy; most existing sensor fusion methods are at the data level and only classify fault types, in which the differences in sensor locations and the location of fault bearing are not considered; no attempt was made to integrate the useful heterogeneous information, such as operating information, into deep learning input, or to differentiate the location of fault bearing; proposes a bearing fault diagnosis and localization approach based on multitask CNN and information fusion, which combines fault characteristic frequencies and operating conditions with signals of multiple sensors; signals of multiple sensors are decomposed into continuous wavelet coefficient matrices (CWCMs) using continuous wavelet transform; then, an information map is built for each bearing based on its structure and operating conditions; the CWCMs and information maps are fused into a three-dimensional (3-D) matrix, which is served as input to multitask CNN for feature extraction; two classifiers are constructed in multitask CNN for fault diagnosis and fault localization, respectively; an information map of domain knowledge is built for each bearing as part of the deep learning method input; it helps to locate fault features in CWCMs and guides CNN to converge in a quick and accurate way; the information map includes bearing operating conditions, rotating speed, and loading profile; with these additional multiple-dimensional (m-D) information, quicker convergence and higher classification accuracy can be achieved; signals of multiple sensors are used as the input of the deep leaning algorithm; it helps to achieve high accuracy in fault diagnosis and locate the fault bearing, which provides convenience for maintenance and replacement of bearings; a multitask CNN structure is proposed with a dynamic training method for both fault diagnosis and fault localization; the dynamic training method is able to balance the convergence rate of two classification tasks such that fault diagnosis and localization can be achieved simultaneously by one deep learning model and one training process; Section II with FIG. 1 in Pages 8006-8007: based on m-D information fusion and multitask CNN, a novel deep learning approach for simultaneous fault diagnosis and localization of bearing is proposed; the proposed approach aims to address the following challenges: a) the use of domain knowledge in deep learning-based fault diagnosis; b) integrating varying operating conditions information in the input of deep learning; 3) use a single deep learning network for both fault diagnosis and localization; Fig. 1 illustrates the procedure of the proposed method, which consists of construction of CWCMs and information maps, m-D information fusion, multitask CNN training, and fault diagnosis and localization; the main steps of the proposed method are described as follows: Step 1: mount accelerometers on bearings under monitoring; collect vibration signals from multiple sensors for different bearings; record operating conditions including rotating speed and load for different bearings; Step 2: resample vibration signals with a virtual sampling frequency (VSF) according to rotating speed; decompose signals into CWCMs using continuous wavelet transform; build information maps by integrating domain knowledge and operating conditions of the bearings; then, CWCMs and information maps are combined to achieve an m-D information matrix as the input of multitask CNN; Step 3: construct a multitask CNN with two classifiers; set parameters of convolutional layers, fully connected layers, and classifiers according to the input size and number of categories; Step 4: prepare training and test data using m-D information matrices; set up two labels (one is for fault diagnosis and the other one is for fault location) for each sample and divide the samples into training data and test data; adopt the dynamic learning-rate method to train the multitask CNN until the training termination condition is met; test the trained system using test data; Step 5: put the trained system into use; process online data using the same VSF and integrate the operating condition into m-D information matrices as real-time input; use the trained multitask CNN for fault diagnosis and localization; Section II.A with FIG. 2 in Pages 8007-8008: Fig. 2 shows the CWCMs of vibration signals of bearing in different fault conditions (e.g., (a) Normal. (b) Ball fault. (c) Inner race fault. (d) Outer race fault) in Case Western Reserve University (CWRU) bearing data; Section II.B with FIG. 3 in Pages 8008-8009: the domain knowledge in fault diagnosis includes fault mechanisms, fault characteristics, diagnosis rules, and feature extraction algorithms designed based on mechanical principles, modeling, or data analysis; deep learning algorithms also have the ability to deal with complex heterogeneous data; it is expected that the domain knowledge, transformed to the proper format and integrated in deep learning input, will greatly increase the accuracy of diagnosis and localization; Fig. 3 shows the information maps built from the vibration signals of drive-end bearing and fan-end bearing in CWRU bearing data; Section II.C in Page 8009 with FIG. 3 in Page 8010: Fig. 4 shows the detailed integration process; in this figure, n accelerometers are used to detect faults of n bearings; through the construction and integration of CWCMs and information maps from multiple resources, m-D information that include vibration signals, domain knowledge of FCFs, and operating conditions are fused into the CNN input; Section IV.A with FIG. 5 in Page 8010: for simultaneous fault diagnosis and localization, it involves two classification tasks that need two classifiers for diagnosis and localization separately; a multitask CNN model that has two classifiers trained together is proposed to achieve the two classification tasks simultaneously; Fig. 5 shows the information flow of the proposed multitask CNN; the front convolutional layers (FCL) are similar to that in conventional CNN; different from existing works, two fully connected layers (F1, F2 ) with two softmax classifiers (C1, C2) are used after the last convolutional layer; the two classifiers (C1, C2) are for fault diagnosis and fault localization, respectively; the two fully connected layers and classifiers are trained simultaneously in multitask CNN; Section IV.B in Pages 8010-8011; using m-D information as input, the multitask CNN is trained for fault diagnosis and fault localization; because the training of fault type needs to extract fault indicators or features, it is usually much more complicated than the training of fault localization that only needs to identify which sensing signal has the bigger vibration energy in certain CWCM areas; therefore, to achieve quick and accurate convergence, fault type is trained first in each training epoch, which needs a larger learning rate and more learning time; however, it is difficult to determine the ratio between the learning rates for fault type and fault location; in this article, an algorithm is proposed to generate different dynamic learning rates for the two tasks to balance the training of the two classifiers; an algorithm is proposed to generate different dynamic learning rates for the two tasks to balance the training of the two classifiers; the dynamic learning rates are calculated by the training state of the last epoch and change in every epoch; the faster training classifier will change to a smaller learning rate; then, the two classifiers can achieve simultaneous convergence).
Su in view of Liu, and Guo are analogous art because they are from the same field of endeavor, a system and a method related to fault diagnosis using Convolutional Neural Network. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Guo to Su in view of Liu. Motivation for doing so would not only reduce the labor cost and downtime, but also optimize the logistics by distinguishing the fault type and locating the faulty bearing simultaneously (Guo, the 5th paragraph of Section I in Page 8006).
Claims 5 and 29
Su in view of Liu and Guo discloses all the elements as stated in Claims 4 and 28 respectively and further discloses wherein the DR-CNN model is trained with dynamic training procedure using the fused information maps (Guo, Abstract in Page 8005: propose a rolling element bearing fault diagnosis and localization approach based on multitask convolutional neural network (CNN) with information fusion; domain knowledge, operating conditions, and vibration signals are fused into a three-dimensional input that can be processed well by CNN; a multitask CNN with dynamic training rates is constructed to simultaneously accomplish two tasks, fault diagnosis and localization; Section 1 in Pages 8005-8006: most deep learning-based fault diagnostic methods are totally data driven, ignore the domain knowledge that has been developed and used for fault diagnosis in the last decades; the operating conditions of the system, including rotating speed and load, have significant influence on vibration signals; to make full use of data-driven methods, it is desirable to integrate system and operating information; domain knowledge, such as fault mechanism and characteristics, must be integrated with data to make sure the deep learning methods converge based on fault characteristics; most rotating machinery have more than one bearing; if a fault happens in one bearing, it is important to distinguish the fault type and locate the faulty bearing; this will not only reduce the labor cost and downtime, but also optimize the logistics; vibration-based condition monitoring systems often use multiple sensors in which each sensor monitors one component; sensor fusion is, therefore, critical to increase the diagnosis accuracy; most existing sensor fusion methods are at the data level and only classify fault types, in which the differences in sensor locations and the location of fault bearing are not considered; no attempt was made to integrate the useful heterogeneous information, such as operating information, into deep learning input, or to differentiate the location of fault bearing; proposes a bearing fault diagnosis and localization approach based on multitask CNN and information fusion, which combines fault characteristic frequencies and operating conditions with signals of multiple sensors; signals of multiple sensors are decomposed into continuous wavelet coefficient matrices (CWCMs) using continuous wavelet transform; then, an information map is built for each bearing based on its structure and operating conditions; the CWCMs and information maps are fused into a three-dimensional (3-D) matrix, which is served as input to multitask CNN for feature extraction; two classifiers are constructed in multitask CNN for fault diagnosis and fault localization, respectively; an information map of domain knowledge is built for each bearing as part of the deep learning method input; it helps to locate fault features in CWCMs and guides CNN to converge in a quick and accurate way.; the information map includes bearing operating conditions, rotating speed, and loading profile; with these additional multiple-dimensional (m-D) information, quicker convergence and higher classification accuracy can be achieved; signals of multiple sensors are used as the input of the deep leaning algorithm; it helps to achieve high accuracy in fault diagnosis and locate the fault bearing, which provides convenience for maintenance and replacement of bearings; a multitask CNN structure is proposed with a dynamic training method for both fault diagnosis and fault localization; the dynamic training method is able to balance the convergence rate of two classification tasks such that fault diagnosis and localization can be achieved simultaneously by one deep learning model and one training process; Section II with FIG. 1 in Pages 8006-8007: based on m-D information fusion and multitask CNN, a novel deep learning approach for simultaneous fault diagnosis and localization of bearing is proposed; the proposed approach aims to address the following challenges: a) the use of domain knowledge in deep learning-based fault diagnosis; b) integrating varying operating conditions information in the input of deep learning; 3) use a single deep learning network for both fault diagnosis and localization; Fig. 1 illustrates the procedure of the proposed method, which consists of construction of CWCMs and information maps, m-D information fusion, multitask CNN training, and fault diagnosis and localization; the main steps of the proposed method are described as follows: Step 1: mount accelerometers on bearings under monitoring; collect vibration signals from multiple sensors for different bearings; record operating conditions including rotating speed and load for different bearings; Step 2: resample vibration signals with a virtual sampling frequency (VSF) according to rotating speed; decompose signals into CWCMs using continuous wavelet transform; build information maps by integrating domain knowledge and operating conditions of the bearings; then, CWCMs and information maps are combined to achieve an m-D information matrix as the input of multitask CNN; Step 3: construct a multitask CNN with two classifiers; set parameters of convolutional layers, fully connected layers, and classifiers according to the input size and number of categories; Step 4: prepare training and test data using m-D information matrices; set up two labels (one is for fault diagnosis and the other one is for fault location) for each sample and divide the samples into training data and test data; adopt the dynamic learning-rate method to train the multitask CNN until the training termination condition is met; test the trained system using test data; Step 5: put the trained system into use; process online data using the same VSF and integrate the operating condition into m-D information matrices as real-time input; use the trained multitask CNN for fault diagnosis and localization; Section II.A with FIG. 2 in Pages 8007-8008: Fig. 2 shows the CWCMs of vibration signals of bearing in different fault conditions (e.g., (a) Normal. (b) Ball fault. (c) Inner race fault. (d) Outer race fault) in Case Western Reserve University (CWRU) bearing data; Section II.B with FIG. 3 in Pages 8008-8009: the domain knowledge in fault diagnosis includes fault mechanisms, fault characteristics, diagnosis rules, and feature extraction algorithms designed based on mechanical principles, modeling, or data analysis; deep learning algorithms also have the ability to deal with complex heterogeneous data; it is expected that the domain knowledge, transformed to the proper format and integrated in deep learning input, will greatly increase the accuracy of diagnosis and localization; Fig. 3 shows the information maps built from the vibration signals of drive-end bearing and fan-end bearing in CWRU bearing data; Section II.C in Page 8009 with FIG. 3 in Page 8010: Fig. 4 shows the detailed integration process; in this figure, n accelerometers are used to detect faults of n bearings; through the construction and integration of CWCMs and information maps from multiple resources, m-D information that include vibration signals, domain knowledge of FCFs, and operating conditions are fused into the CNN input; Section IV.A with FIG. 5 in Page 8010: for simultaneous fault diagnosis and localization, it involves two classification tasks that need two classifiers for diagnosis and localization separately; a multitask CNN model that has two classifiers trained together is proposed to achieve the two classification tasks simultaneously; Fig. 5 shows the information flow of the proposed multitask CNN; the front convolutional layers (FCL) are similar to that in conventional CNN; different from existing works, two fully connected layers (F1, F2 ) with two softmax classifiers (C1, C2) are used after the last convolutional layer; the two classifiers (C1, C2) are for fault diagnosis and fault localization, respectively; the two fully connected layers and classifiers are trained simultaneously in multitask CNN; Section IV.B in Pages 8010-8011; using m-D information as input, the multitask CNN is trained for fault diagnosis and fault localization; because the training of fault type needs to extract fault indicators or features, it is usually much more complicated than the training of fault localization that only needs to identify which sensing signal has the bigger vibration energy in certain CWCM areas; therefore, to achieve quick and accurate convergence, fault type is trained first in each training epoch, which needs a larger learning rate and more learning time; however, it is difficult to determine the ratio between the learning rates for fault type and fault location; in this article, an algorithm is proposed to generate different dynamic learning rates for the two tasks to balance the training of the two classifiers; an algorithm is proposed to generate different dynamic learning rates for the two tasks to balance the training of the two classifiers; the dynamic learning rates are calculated by the training state of the last epoch and change in every epoch; the faster training classifier will change to a smaller learning rate; then, the two classifiers can achieve simultaneous convergence).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Guo to Su in view of Liu. Motivation for doing so would balance the convergence rate of two classification tasks such that the two classification tasks can be achieved simultaneously by one deep learning model and one training process (Guo, the 7th paragraph of Section I in Page 8006; and Section IV.B in Pages 8010-8011).
Claims 8 and 32
Su in view of Liu and Guo discloses all the elements as stated in Claims 4 and 28 respectively and further discloses wherein the one or more sets of fault information include at least one of fault mode diagnosis, fault localization, and operating condition at the time of fault detection (Su, ¶ [0082] with Table 1: health status of rolling bearings: State: N represents normal bearing; State IF represents Inner-Race Fault; State: OF represents Outer-Race Fault; State: BF represents Rolling-Ball Fault; State: MF represents Composite-Bearing Fault) (Guo, Section II.A with FIG. 2 in Pages 8007-8008: Fig. 2 shows the CWCMs of vibration signals of bearing in different fault conditions (e.g., (a) Normal. (b) Ball fault. (c) Inner race fault. (d) Outer race fault) in Case Western Reserve University (CWRU) bearing data; Section II.B with FIG. 3 in Pages 8008-8009: Fig. 3 shows the information maps built from the vibration signals of drive-end bearing and fan-end bearing in CWRU bearing data).
Claims 9 and 33
Su in view of Liu and Guo discloses all the elements as stated in Claims 9 and 32 respectively and further discloses wherein fault mode diagnosis includes inner-race fault, outer-race fault, ball fault, and normal, and fault localization includes drive end bearing, middle bearing, non-drive end bearing, and normal (Su, ¶ [0080]: collects the belt end bearing Y-axis vibration signal; ¶ [0082] with Table 1: health status of rolling bearings: State: N represents normal bearing; State IF represents Inner-Race Fault; State: OF represents Outer-Race Fault; State: BF represents Rolling-Ball Fault; State: MF represents Composite-Bearing Fault) (Guo, Section II.A with FIG. 2 in Pages 8007-8008: Fig. 2 shows the CWCMs of vibration signals of bearing in different fault conditions (e.g., (a) Normal. (b) Ball fault. (c) Inner race fault. (d) Outer race fault) in Case Western Reserve University (CWRU) bearing data; Section II.B with FIG. 3 in Pages 8008-8009: Fig. 3 shows the information maps built from the vibration signals of drive-end bearing and fan-end bearing in CWRU bearing data).
Claims 13 and 37
Su in view of Liu discloses all the elements as stated in Claims 1 and 24 respectively and further discloses current operating data/signal to become part of the fused information maps to input to the DR-CNN model (Su, ¶¶ [0049]-[0051] with FIG.1: 2nd part of FIG. 1 is fault diagnosis and identification; fault diagnosis of rolling bearing under small sample includes (a) processing vibration data of the rolling bearing to be diagnosed and identified; (b) inputting identification data to be diagnosed into the trained nonlinear metric prototype network in the process 1; and (c) outputting a fault diagnosis result by the trained nonlinear metric prototype network; ¶¶ [0076]-[0079]: the identification process of fault diagnosis/identification comprises the following steps: S21, processing identification data to be diagnosed; s22, inputting identification data to be diagnosed into a nonlinear metric prototype network, and outputting a fault diagnosis result by the network; the nonlinear metric prototype network is a small sample supervised learning model and mainly comprises a prototype calculation module, a cascade attention module, and a nonlinear metric module).
Su in view of Liu fails to explicitly disclose current operating conditions to become part of the fused information maps to input to the DR-CNN model.
Guo teaches a system and a method related to fault diagnosis using Convolutional Neural Network (Guo, Abstract), wherein current operating conditions to become part of the fused information maps to input to the DR-CNN model (Guo, Abstract in Page 8005: propose a rolling element bearing fault diagnosis and localization approach based on multitask convolutional neural network (CNN) with information fusion; domain knowledge, operating conditions, and vibration signals are fused into a three-dimensional input that can be processed well by CNN; a multitask CNN with dynamic training rates is constructed to simultaneously accomplish two tasks, fault diagnosis and localization; Section 1 in Pages 8005-8006: most deep learning-based fault diagnostic methods are totally data driven, ignore the domain knowledge that has been developed and used for fault diagnosis in the last decades; the operating conditions of the system, including rotating speed and load, have significant influence on vibration signals; to make full use of data-driven methods, it is desirable to integrate system and operating information; domain knowledge, such as fault mechanism and characteristics, must be integrated with data to make sure the deep learning methods converge based on fault characteristics; most rotating machinery have more than one bearing; if a fault happens in one bearing, it is important to distinguish the fault type and locate the faulty bearing; this will not only reduce the labor cost and downtime, but also optimize the logistics; vibration-based condition monitoring systems often use multiple sensors in which each sensor monitors one component; sensor fusion is, therefore, critical to increase the diagnosis accuracy; most existing sensor fusion methods are at the data level and only classify fault types, in which the differences in sensor locations and the location of fault bearing are not considered; no attempt was made to integrate the useful heterogeneous information, such as operating information, into deep learning input, or to differentiate the location of fault bearing; proposes a bearing fault diagnosis and localization approach based on multitask CNN and information fusion, which combines fault characteristic frequencies and operating conditions with signals of multiple sensors; signals of multiple sensors are decomposed into continuous wavelet coefficient matrices (CWCMs) using continuous wavelet transform; then, an information map is built for each bearing based on its structure and operating conditions; the CWCMs and information maps are fused into a three-dimensional (3-D) matrix, which is served as input to multitask CNN for feature extraction; two classifiers are constructed in multitask CNN for fault diagnosis and fault localization, respectively; an information map of domain knowledge is built for each bearing as part of the deep learning method input; it helps to locate fault features in CWCMs and guides CNN to converge in a quick and accurate way; the information map includes bearing operating conditions, rotating speed, and loading profile; with these additional multiple-dimensional (m-D) information, quicker convergence and higher classification accuracy can be achieved; signals of multiple sensors are used as the input of the deep leaning algorithm; it helps to achieve high accuracy in fault diagnosis and locate the fault bearing, which provides convenience for maintenance and replacement of bearings; a multitask CNN structure is proposed with a dynamic training method for both fault diagnosis and fault localization; the dynamic training method is able to balance the convergence rate of two classification tasks such that fault diagnosis and localization can be achieved simultaneously by one deep learning model and one training process; Section II with FIG. 1 in Pages 8006-8007: based on m-D information fusion and multitask CNN, a novel deep learning approach for simultaneous fault diagnosis and localization of bearing is proposed; the proposed approach aims to address the following challenges: a) the use of domain knowledge in deep learning-based fault diagnosis; b) integrating varying operating conditions information in the input of deep learning; 3) use a single deep learning network for both fault diagnosis and localization; Fig. 1 illustrates the procedure of the proposed method, which consists of construction of CWCMs and information maps, m-D information fusion, multitask CNN training, and fault diagnosis and localization; the main steps of the proposed method are described as follows: Step 1: mount accelerometers on bearings under monitoring; collect vibration signals from multiple sensors for different bearings; record operating conditions including rotating speed and load for different bearings; Step 2: resample vibration signals with a virtual sampling frequency (VSF) according to rotating speed; decompose signals into CWCMs using continuous wavelet transform; build information maps by integrating domain knowledge and operating conditions of the bearings; then, CWCMs and information maps are combined to achieve an m-D information matrix as the input of multitask CNN; Step 3: construct a multitask CNN with two classifiers; set parameters of convolutional layers, fully connected layers, and classifiers according to the input size and number of categories; Step 4: prepare training and test data using m-D information matrices; set up two labels (one is for fault diagnosis and the other one is for fault location) for each sample and divide the samples into training data and test data; adopt the dynamic learning-rate method to train the multitask CNN until the training termination condition is met; test the trained system using test data; Step 5: put the trained system into use; process online data using the same VSF and integrate the operating condition into m-D information matrices as real-time input; use the trained multitask CNN for fault diagnosis and localization; Section II.A with FIG. 2 in Pages 8007-8008: Fig. 2 shows the CWCMs of vibration signals of bearing in different fault conditions (e.g., (a) Normal. (b) Ball fault. (c) Inner race fault. (d) Outer race fault) in Case Western Reserve University (CWRU) bearing data; Section II.B with FIG. 3 in Pages 8008-8009: the domain knowledge in fault diagnosis includes fault mechanisms, fault characteristics, diagnosis rules, and feature extraction algorithms designed based on mechanical principles, modeling, or data analysis; deep learning algorithms also have the ability to deal with complex heterogeneous data; it is expected that the domain knowledge, transformed to the proper format and integrated in deep learning input, will greatly increase the accuracy of diagnosis and localization; Fig. 3 shows the information maps built from the vibration signals of drive-end bearing and fan-end bearing in CWRU bearing data; Section II.C in Page 8009 with FIG. 3 in Page 8010: Fig. 4 shows the detailed integration process; in this figure, n accelerometers are used to detect faults of n bearings; through the construction and integration of CWCMs and information maps from multiple resources, m-D information that include vibration signals, domain knowledge of FCFs, and operating conditions are fused into the CNN input; Section IV.A with FIG. 5 in Page 8010: for simultaneous fault diagnosis and localization, it involves two classification tasks that need two classifiers for diagnosis and localization separately; a multitask CNN model that has two classifiers trained together is proposed to achieve the two classification tasks simultaneously; Fig. 5 shows the information flow of the proposed multitask CNN; the front convolutional layers (FCL) are similar to that in conventional CNN; different from existing works, two fully connected layers (F1, F2 ) with two softmax classifiers (C1, C2) are used after the last convolutional layer; the two classifiers (C1, C2) are for fault diagnosis and fault localization, respectively; the two fully connected layers and classifiers are trained simultaneously in multitask CNN; Section IV.B in Pages 8010-8011; using m-D information as input, the multitask CNN is trained for fault diagnosis and fault localization; because the training of fault type needs to extract fault indicators or features, it is usually much more complicated than the training of fault localization that only needs to identify which sensing signal has the bigger vibration energy in certain CWCM areas; therefore, to achieve quick and accurate convergence, fault type is trained first in each training epoch, which needs a larger learning rate and more learning time; however, it is difficult to determine the ratio between the learning rates for fault type and fault location; in this article, an algorithm is proposed to generate different dynamic learning rates for the two tasks to balance the training of the two classifiers; an algorithm is proposed to generate different dynamic learning rates for the two tasks to balance the training of the two classifiers; the dynamic learning rates are calculated by the training state of the last epoch and change in every epoch. The faster training classifier will change to a smaller learning rate; then, the two classifiers can achieve simultaneous convergence).
Su in view of Liu, and Guo are analogous art because they are from the same field of endeavor, a system and a method related to fault diagnosis using Convolutional Neural Network. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Guo to Su in view of Liu. Motivation for doing so would (1) a deep learning model trained using the data of a certain machinery can be extended to diagnose another machinery with similar structure or component and (2) achieve quicker convergence and higher classification accuracy (Guo, the 4th and 7th paragraphs of Section I in Page 8006).
Claims 14-24 are rejected under 35 U.S.C. 103 as being unpatentable over Su in view of Guo, Liu, and WANG et al. ("Fault Diagnosis of Bearings Based on Multi-Sensor Information Fusion and 2D Convolutional Neural Network", IEEE Access, VOLUME 9, 2021, Feb. 3, 2021, pp. 23717-23725), hereinafter WANG.
Independent Claim 14
Su discloses a method of training an enhanced discriminate feature learning based (Su, Abstract and ¶¶ [0004]-[0005] with FIG. 1: provide a rolling bearing fault diagnosis method based on a nonlinear metric prototype network, which comprises the steps of constructing a cascade attention prototype nonlinear metric network, performing classification training on the constructed network, performing data processing on data with diagnosis, inputting the data into the trained cascade attention prototype nonlinear metric network; the cascade attention prototype nonlinear metric network comprises a sample set division module, a prototype calculation module, a cascade attention mechanism learning module and a nonlinear metric strategy classification training module; ¶ [0093]: the similarity of the spliced sample can be better judged through nonlinear measurement, and meanwhile, the long-distance correlation of the spliced sample is calculated by using the cascade attention, so as to further obtain the more distinguishing features, so that the method has higher identification accuracy for each class), comprising:
partitioning into segments 1-dimensional (1-D) monitoring data samples from monitoring sensors associated with one or more monitored rotating components; (Su, ¶ [0003]: the rolling bearing fault diagnosis method based on deep learning is rapidly developed in the past few years, and fault diagnosis and identification are carried out on vibration signals by utilizing strong feature dimension reduction and mode identification capability of a neural network; ¶¶ [0006], [0050]-[0051], and [0053] with FIG. 2: dividing the sample set into a support set (i.e., for constructing fault domain data) and a query set (i.e., query samples from vibration signal) by using a sample set dividing module; FIG. 2 also shown a plurality of segments in a query set (queryj); processing vibration data of the rolling bearing to be diagnosed and identified; before dividing a sample set, normalizing original vibration signal samples; ¶¶ [0035]-[0039] with FIGS. 7-11: a schematic diagram of a vibration signal of a rolling bearing collected in a state (a)-(e) by the MFS experimental apparatus; ¶ [0080] with FIGS. 7-11: a comparison test is carried out by utilizing a vibration signal of a Machine Fault Simulator (MFS); the experiment simulates 5 health states of the rolling bearing, collects the belt end bearing Y-axis vibration signal of the simulator under the 44Hz conversion frequency, and the sampling frequency is 10240 Hz; each set of health status data was repeatedly collected 6 times; after obtaining the vibration signals of five different states, data preprocessing is required for the vibration signals; and first, the vibration data with a length of 102400 is divided into 25 samples, each sample containing 4096 data points, so that the number of samples per class is 25 × 6 = 150; the original vibration waveforms for the five different conditions are shown in FIGS. 7-11);
transforming a set of domain knowledge about operations and faults into information maps; integrating the grayscale images with the information maps to build fused information images, which are used as input to a (Su, ¶¶ [0006]-[0008], [0010]-[0014], [0050], and [0053]-[0060] with FIG. 2: dividing the sample set into a support set (i.e., for constructing fault domain knowledge data) and a query set (i.e., query samples from vibration signal) by using a sample set dividing module; inputting the divided data sets into a prototype calculation module to obtain feature maps corresponding to samples in the data sets using a feature extractor for embedding sample xi in the sample set L into feature space, and calculating class prototypes through the feature maps of the support sets (i.e., constructing fault domain knowledge data from the feature maps of the support sets); for type c faults, prototype Pc is generated by using support set S; splicing/joining the feature map of the query set samples (i.e., from vibration signal collected) with the prototypes of all categories one by one (i.e., fault domain knowledge data constructed from the support set) based on the calculated prototypes, and extracting the long-distance correlation of the spliced/joined samples by adopting a cascade attention mechanism learning module; ¶ [0009]: inputting the long-distance correlation extracted by the cascade attention mechanism learning module into a nonlinear metric strategy classification training module for classification training; ¶¶ [0049]-[0051] with FIG.1: 1st part of FIG. 1 is nonlinear metric prototype network training; training a nonlinear metric prototype network including (a) based on the limited label sample set, the limited label sample set is divided into a training set and a test set, wherein the training set is further divided into a support set and a query set, the samples are mapped to an embedding space through a prototype network, and various types of prototypes are calculated based on the support set; (b) splicing/joining the query samples and the class prototypes in the embedding space one by one, and sending the query samples and the class prototypes into a cascade attention module to extract non-local information; and (c) finally, the similarity between the sample and the prototype is better measured through a nonlinear metric module so as to improve the fault diagnosis performance; initializing all parameters of the nonlinear metric prototype network based on the steps, and feeding training samples through a gradient descent algorithm to train parameters of a network optimization model; ¶ [0059]: since the prototype network measures the similarity between a sample and a class prototype in a linear manner, the linear measurement is intended to directly calculate the distance between features by predefining a fixed metric (e.g. Euclidean distance), which requires that a feature extractor can extract obvious discriminant features as prototype representations, whereas a mechanical vibration signal is difficult to extract fault features with high recognizability under the condition of few labeled samples; secondly, the fixed linear metric cannot learn the non-linear relationship between complex signals, and the diagnostic performance thereof will be greatly reduced; aiming at the defects of prototype network linear measurement, a learnable nonlinear classifier is used for replacing a prototype network fixed linear metric method, class prototypes and query sample features are spliced/joined, nonlinear measurement is learned through a nonlinear neural network, and similarity scoring is carried out on each batch of spliced samples to complete sample category identification; ¶¶ [0015]-[0028] and [0060]-[0075] with FIGS. 2 and 6: the cascade attention mechanism learning module comprises a channel attention submodule and a space attention submodule; the cascade attention mechanism learning module performs convolution on the input spliced sample and extracts a feature F; splicing/joining a query sample and the C type prototype feature, and inputting the spliced/joined sample 1 (xi) into the convolution block in the cascade attention module, performing preliminary feature extraction to the spliced feature to obtain the feature map F [Symbol font/0xCE] RH×W×C, where H×W×C represents the height, width, and number of channels of the feature map, respectively; the convolution block adopted in the process of carrying out convolution on the input spliced sample by the cascade attention mechanism learning module comprises a convolution layer, a pooling layer, a BN layer and an activation function; in the cascade attention module, the feature map F [Symbol font/0xCE] RH×W×C is controlled flow into the channel attention and spatial attention modules, respectively (i.e., respectively inputting the feature F into a channel attention submodule and a space attention submodule), wherein the channel attention submodule self-adaptively adjusts feature values among channels, establishes a channel dependency relationship and obtains a channel attention feature Fc'; the space attention submodule focuses on the position information of the target sample in the input feature mapping and ignores the unimportant target features to obtain a space attention feature Fs'; finally, performing information fusion on the channel attention feature Fc' and spatial attention feature Fs', and then accumulating the fused feature information and the input feature F to obtain the long-distance correlation of the spliced sample; in the channel attention submodule, a global average pooling operation is firstly adopted to compress the feature F in a space dimension, and the space information of feature mapping is aggregated to generate a channel attention feature map Uc [Symbol font/0xCE] Rl×l×C, then pass through two convolution blocks to extract the nonlinear relationship between each channel, the channel dimensions of the two convolution blocks are first subjected to dimensionality reduction processing and then to dimensionality ascending processing, and then an activation function is used for obtaining a channel attention weight S; the internal network structure of the channel attention submodule is shown in FIG. 6, wherein CPBA represents the corresponding convolutional layer, pooling layer, BN layer and activation function, and CBA represents the convolutional layer, BN layer and activation function (i.e., the channel attention submodule comprises a global average pooling layer (G), a first convolution block (CBA), and a second convolution block (CBA), wherein each convolution block is composed of a convolution layer (C), a BN layer (B) and an activation function (A); the feature F is input into the global average pooling layer, the first convolution block, and the second convolution block which are cascaded to obtain a channel information structure S through extraction), and then multiplying the input feature F by a channel information structure S matrix, and fusing/adding the generated result with the feature information F to obtain the channel attention weighted feature Fc'; in the space attention module, firstly, a convolution layer is adopted to extract information from the feature F, and the output feature of the convolution layer are subjected to channel fusion to obtain a space attention feature map Us [Symbol font/0xCE] RHWl×l; the activation function is then used to obtain the spatial attention weight S'; the network structure of the spatial attention submodule is shown in FIG. 6, where CB represents the corresponding convolutional layer and BN layer (i.e., the spatial attention submodule comprises a third convolution block and a global average pooling layer, wherein the third convolution block is composed of a convolution layer and a BN layer; the feature F is input into the cascaded third convolution block and the global average pooling layer to extract a spatial information structure S'); then, the input feature F is multiplied by the space information structure S' matrix, and the generated result is fused/added with the feature information F to obtain the space attention weighted feature Fs [Symbol font/0xCE] RH×W×C; inputting the features extracted by the attention module into a nonlinear measurement module to realize effective few-shot learning (FSL) for bearing fault diagnosis, and conveying the spliced sample to the nonlinear metric module hr(C) through a series of continuous mapping of network layers, the module finally outputs a number C of scaler values Vj,r between 0 and 1 through softmax, wherein Vj,r represents similarity between query samples xjQ and a certain type of prototype PC; i.e. probability of query samples values belonging to the class; The difference between linear metric method (FIG.4) and nonlinear metric method (FIG. 5) based on the prototype network are shown in FIGS. 4-5; in order to improve the accuracy of the classifier, a network model is trained by minimizing the classification loss of class prototypes corresponding to the query samples and the support set; the mean square error is used as a loss function; calculating the mean square error LMSE using the similarity probability value Vj,r output through the procedures described above, and labels for the query samples yjQ with labels yrP belongs to the class prototype; finally, the network model is trained by minimizing the above equation (i.e., LMSE); after the class prototype is spliced with the feature map of the query sample, when directly inputting the spliced samples into a nonlinear measurement network, the long-distance correlation of the spliced samples with double-increased feature dimension cannot be captured because of the influence caused by the size of a receptive field; therefore, a cascade attention mechanism is used to extract the long-distance correlation of the spliced samples, so as to better extract the nonlinear relation between the sample and the prototype through the nonlinear measurement module; extract the feature maps through the prototype calculation module, calculate the prototype for the support set feature, splice/join the query sample feature and various prototypes one by one in the cascade attention module, then extract the long-distance correlation of the spliced sample through the cascade attention mechanism, and finally input the feature extracted by the cascade attention module into the nonlinear measurement module; thereby realizing the accurate and effective bearing fault diagnosis under the condition of small sample; ¶ [0094]: provide an improved FSL method of a rolling bearing fault diagnosis model aiming at an application scene of the shortage of fault labeled data, which is called as a cascade attention and nonlinear metric improved prototype network (CANM-ProNet); first, the prototype calculation module extracts feature maps of the support set and the query set, and calculates a prototype using the feature maps of the support set; the query feature map is then concatenated with each prototype and a cascade attention module is introduced to extract non-local information of the concatenated features; finally, a non-linear metric module is presented for better measuring the similarity between the samples and the prototype to improve fault diagnosis performance; numerous experiments have shown that this method is more efficient than other methods with fewer samples of faults; ¶¶ [0049]-[0051] with FIG.1: 2nd part of FIG. 1 is fault diagnosis and identification; fault diagnosis of rolling bearing under small sample includes (a) processing vibration data of the rolling bearing to be diagnosed and identified; (b) inputting identification data to be diagnosed into the trained nonlinear metric prototype network in the process 1; and (c) outputting a fault diagnosis result by the trained nonlinear metric prototype network; ¶¶ [0076]-[0079]: the identification process of fault diagnosis/identification comprises the following steps: S21, processing identification data to be diagnosed; s22, inputting identification data to be diagnosed into a nonlinear metric prototype network, and outputting a fault diagnosis result by the network; the nonlinear metric prototype network is a small sample supervised learning model and mainly comprises a prototype calculation module, a cascade attention module, and a nonlinear metric module).
Su fails to explicitly disclose (1) training a feature learning based multi-task CNN; (2) converting the monitoring data segments from different sensors into 2-dimensional (2-D) grayscale images; (3) a multi-task deep residual convolutional neural network (DR-CNN) having multiple classifiers for multi-task diagnosis; and (4) training the DR-CNN with the fused information images by using a dynamic training strategy to learn fault diagnosis.
Guo teaches a system and a method related to fault diagnosis using Convolutional Neural Network (Guo, Abstract), wherein training a feature learning based multi-task CNN (Guo, Abstract in Page 8005: propose a rolling element bearing fault diagnosis and localization approach based on multitask convolutional neural network (CNN) with information fusion; a multitask CNN with dynamic training rates is constructed to simultaneously accomplish two tasks, fault diagnosis and localization; Section 1 in Pages 8005-8006: proposes a bearing fault diagnosis and localization approach based on multitask CNN and information fusion, which combines fault characteristic frequencies and operating conditions with signals of multiple sensors; two classifiers are constructed in multitask CNN for fault diagnosis and fault localization, respectively; a multitask CNN structure is proposed with a dynamic training method for both fault diagnosis and fault localization; Section II with FIG. 1 in Pages 8006-8007: use a single deep learning network for both fault diagnosis and localization; Fig. 1 illustrates the procedure of the proposed method, which consists of construction of CWCMs and information maps, m-D information fusion, multitask CNN training, and fault diagnosis and localization; Section II with FIG. 1 in Pages 8006-8007: adopt the dynamic learning-rate method to train the multitask CNN until the training termination condition is met; Section IV.A with FIG. 5 in Page 8010: a multitask CNN model that has two classifiers trained together is proposed to achieve the two classification tasks simultaneously; the two fully connected layers and classifiers are trained simultaneously in multitask CNN; Section IV.B in Pages 8010-8011; using m-D information as input, the multitask CNN is trained for fault diagnosis and fault localization);
transforming a set of domain knowledge about operations and faults into information maps; integrating the grayscale images with the information maps to build fused information images (Guo, Abstract in Page 8005: domain knowledge, operating conditions, and vibration signals are fused into a three-dimensional input that can be processed well by CNN; Section 1 in Pages 8005-8006: most deep learning-based fault diagnostic methods are totally data driven, ignore the domain knowledge that has been developed and used for fault diagnosis in the last decades; the operating conditions of the system, including rotating speed and load, have significant influence on vibration signals; to make full use of data-driven methods, it is desirable to integrate system and operating information; domain knowledge, such as fault mechanism and characteristics, must be integrated with data to make sure the deep learning methods converge based on fault characteristics; an information map of domain knowledge is built for each bearing as part of the deep learning method input; it helps to locate fault features in CWCMs and guides CNN to converge in a quick and accurate way; the information map includes bearing operating conditions, rotating speed, and loading profile; with these additional multiple-dimensional (m-D) information, quicker convergence and higher classification accuracy can be achieved; signals of multiple sensors are used as the input of the deep leaning algorithm; it helps to achieve high accuracy in fault diagnosis and locate the fault bearing, which provides convenience for maintenance and replacement of bearings; Section II.B with FIG. 3 in Pages 8008-8009: the domain knowledge in fault diagnosis includes fault mechanisms, fault characteristics, diagnosis rules, and feature extraction algorithms designed based on mechanical principles, modeling, or data analysis; deep learning algorithms also have the ability to deal with complex heterogeneous data; it is expected that the domain knowledge, transformed to the proper format and integrated in deep learning input, will greatly increase the accuracy of diagnosis and localization; for fault diagnosis of rolling element bearings, one of the most important domain knowledge is the fault characteristics frequency (FCF); according to the fault mechanism, when a bearing fault happens, the fault characteristics frequency components will appear in the vibration signals; the FCFs are calculated as shown in Eqn. (4), where fIR, fOR, and fBA are the FCF of inner race fault, outer race fault, and ball fault, respectively, Z is the number of rolling elements, d is the rolling element diameter, D is the pitch diameter, and α is the contact angle; an information map of FCF is built to make sure that the deep learning model converges in a way related to the FCF), which are used as input to a multi-task deep convolutional neural network (D-CNN) having multiple classifiers for multi-task diagnosis (Guo, Section 1 in Pages 8005-8006: most rotating machinery have more than one bearing; if a fault happens in one bearing, it is important to distinguish the fault type and locate the faulty bearing; this will not only reduce the labor cost and downtime, but also optimize the logistics; vibration-based condition monitoring systems often use multiple sensors in which each sensor monitors one component; sensor fusion is, therefore, critical to increase the diagnosis accuracy; most existing sensor fusion methods are at the data level and only classify fault types, in which the differences in sensor locations and the location of fault bearing are not considered; no attempt was made to integrate the useful heterogeneous information, such as operating information, into deep learning input, or to differentiate the location of fault bearing; proposes a bearing fault diagnosis and localization approach based on multitask CNN and information fusion, which combines fault characteristic frequencies and operating conditions with signals of multiple sensors; signals of multiple sensors are decomposed into continuous wavelet coefficient matrices (CWCMs) using continuous wavelet transform; then, an information map is built for each bearing based on its structure and operating conditions; the CWCMs and information maps are fused into a three-dimensional (3-D) matrix, which is served as input to multitask CNN for feature extraction; two classifiers are constructed in multitask CNN for fault diagnosis and fault localization, respectively; a multitask CNN structure is proposed with a dynamic training method for both fault diagnosis and fault localization; Section II with FIG. 1 in Pages 8006-8007: based on m-D information fusion and multitask CNN, a novel deep learning approach for simultaneous fault diagnosis and localization of bearing is proposed; the proposed approach aims to address the following challenges: a) the use of domain knowledge in deep learning-based fault diagnosis; b) integrating varying operating conditions information in the input of deep learning; 3) use a single deep learning network for both fault diagnosis and localization; Fig. 1 illustrates the procedure of the proposed method, which consists of construction of CWCMs and information maps, m-D information fusion, multitask CNN training, and fault diagnosis and localization; the main steps of the proposed method are described as follows: Step 1: mount accelerometers on bearings under monitoring; collect vibration signals from multiple sensors for different bearings; record operating conditions including rotating speed and load for different bearings; Step 2: resample vibration signals with a virtual sampling frequency (VSF) according to rotating speed; decompose signals into CWCMs using continuous wavelet transform; build information maps by integrating domain knowledge and operating conditions of the bearings; then, CWCMs and information maps are combined to achieve an m-D information matrix as the input of multitask CNN; Step 3: construct a multitask CNN with two classifiers; set parameters of convolutional layers, fully connected layers, and classifiers according to the input size and number of categories; Step 4: prepare training and test data using m-D information matrices; set up two labels (one is for fault diagnosis and the other one is for fault location) for each sample and divide the samples into training data and test data; adopt the dynamic learning-rate method to train the multitask CNN until the training termination condition is met; test the trained system using test data; Step 5: put the trained system into use; process online data using the same VSF and integrate the operating condition into m-D information matrices as real-time input; use the trained multitask CNN for fault diagnosis and localization; Section II.C in Page 8009 with FIG. 3 in Page 8010: Fig. 4 shows the detailed integration process; in this figure, n accelerometers are used to detect faults of n bearings; through the construction and integration of CWCMs and information maps from multiple resources, m-D information that include vibration signals, domain knowledge of FCFs, and operating conditions are fused into the CNN input; Section IV.A with FIG. 5 in Page 8010: for simultaneous fault diagnosis and localization, it involves two classification tasks that need two classifiers for diagnosis and localization separately; a multitask CNN model that has two classifiers trained together is proposed to achieve the two classification tasks simultaneously; Fig. 5 shows the information flow of the proposed multitask CNN; the front convolutional layers (FCL) are similar to that in conventional CNN; different from existing works, two fully connected layers (F1, F2 ) with two softmax classifiers (C1, C2) are used after the last convolutional layer; the two classifiers (C1, C2) are for fault diagnosis and fault localization, respectively; the two fully connected layers and classifiers are trained simultaneously in multitask CNN; Section IV.B in Pages 8010-8011; using m-D information as input, the multitask CNN is trained for fault diagnosis and fault localization);
training the D-CNN with the fused information images by using a dynamic training strategy to learn fault diagnosis (Guo, Abstract in Page 8005: a multitask CNN with dynamic training rates is constructed to simultaneously accomplish two tasks, fault diagnosis and localization; Section 1 in Pages 8005-8006: a multitask CNN structure is proposed with a dynamic training method for both fault diagnosis and fault localization; the dynamic training method is able to balance the convergence rate of two classification tasks such that fault diagnosis and localization can be achieved simultaneously by one deep learning model and one training process; Section II with FIG. 1 in Pages 8006-8007: adopt the dynamic learning-rate method to train the multitask CNN until the training termination condition is met; Section IV.B in Pages 8010-8011; because the training of fault type needs to extract fault indicators or features, it is usually much more complicated than the training of fault localization that only needs to identify which sensing signal has the bigger vibration energy in certain CWCM areas; therefore, to achieve quick and accurate convergence, fault type is trained first in each training epoch, which needs a larger learning rate and more learning time; however, it is difficult to determine the ratio between the learning rates for fault type and fault location; in this article, an algorithm is proposed to generate different dynamic learning rates for the two tasks to balance the training of the two classifiers; an algorithm is proposed to generate different dynamic learning rates for the two tasks to balance the training of the two classifiers; the dynamic learning rates are calculated by the training state of the last epoch and change in every epoch; the faster training classifier will change to a smaller learning rate; then, the two classifiers can achieve simultaneous convergence)
Su and Guo are analogous art because they are from the same field of endeavor, a system and a method related to fault diagnosis using Convolutional Neural Network. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Guo to Su. Motivation for doing so would (1) help to locate fault features in and guides CNN to converge in a quick and accurate way; (2) not only reduce the labor cost and downtime, but also optimize the logistics.
Su in view of Guo fails to explicitly disclose (1) converting the monitoring data segments from different sensors into 2-dimensional (2-D) grayscale images; and (2) a deep residual convolutional neural network (DR-CNN) for diagnosis.
Liu teaches a system and a method for identification of defects/faults using convolutional neural network (Liu, 1st paragraph of Section 2.1.1 in Pages 3-4 and Abstract in Page 1), wherein a deep residual convolutional neural network (DR-CNN) for diagnosis (Liu, Abstract in Page 1: a pipeline MFL inspection signal identification method based on improved deep residual convolutional neural network and attention module is proposed; an improved deep residual network based on the VGG16 convolution neural network is constructed to automatically learn the features from the MFL image signals and perform the identification of pipeline features and defects; 4th paragraph – 6th paragraph of Section 1 in Pages 2-3: deep learning methods have become a hotspot in the research of signal identification and fault diagnosis in recent years; to further improve the identification accuracy for the complex pipeline MFL signals, a pipeline MFL inspection feature identification model based on an improved deep residual convolutional neural network is proposed; the proposed method not only automatically learns the features from the MFL inspection images and performs the classification and identification of pipeline features and defects such as welds, tees, flanges, and corrosion, but also solves the problems of the great influence of noise, compound features, and other factors on the feature identification results in the process of in-line inspection; the proposed method effectively improves the classification of pipeline features, and provides an effective method for pipeline features identification; an MFL in-line inspection method based on attention module and convolution residual modules is proposed, which effectively improve accuracy and efficiency; aiming at the influence of the complex operating environment, high noises, composite defects to MFL in-line inspection of oil and gas pipelines, attention module composed of channel attention and spatial attention are designed to fully extract MFL image feature information; to solve the problem of gradient dispersion caused by the increase of the number of network layers, an improved residual convolutional neural network is constructed to reduce the error of the deep network as well as the amount of calculation parameters, and effectively improve the training efficiency; Section 2.1.1 with FIG. 1 in Pages 3-5: the improved CNN with strong generalization, feature extraction and identification can effectively identify the fault types of rolling bearings; Section 2.2.3 with FIGS. 7-8 in Pages 9-11: to improve the deep convolution network and propose a new mechanism to enhance the signal feature extraction ability of the network, the attention model was derived from a human visual attention model; while processing data, the visual system quickly focuses on the target areas that need to be focused by scanning the global scene, and allocates limited computing resources to these key parts; this mechanism can greatly reduce the amount of data to be processed, ignore unimportant information, and provide more manageable and relevant information for higher-level perceptual reasoning and complex visual processing; it is one of the core technologies in deep learning worthy of attention and in-depth understanding; the convolutional block attention module (CBAM) is an attention module combining spatial with channel information; CBAM adopts max-pooling and average-pooling to generate weights through the channel and spatial dimensions; adding the attention mechanism module can further extract the interested small defects target area from the background, thus the network can better learn small target defects; at the same time, the interference of the background to the target is suppressed, thus improving the learning ability of the network to the detailed features of small targets and enhance the ability of feature learning; the method introduces the attention mechanism and designs the spatial attention module (Spatial_AM) and the channel attention module (Channel_AM); in Channel_AM, the property of maximum pooling is used to capture the inter class information between MFL image pixels, and the average pooling is used to capture the intra class information between pixels; these two information as weights are applied to the original feature map as attention to assist feature extraction; the module is connected between the feature map extraction module and the feature map decoding module; at the same time, the Spatial_AM is also designed; by using the spatial attention mechanism composed of global pooling, convolution, and activation function, the semantic information extraction is further refined, and the information is multiplied with the original feature map as a weight.; Channel attention module: when extracting features in the channel dimension, average pooling and max pooling are considered simultaneously; Figure 7 shows the design scheme of the channel attention module; the module consists of convolution, batch regularization, and an activation function, which can extract mixed information by integrating channel and semantic information; then, the average pooling module, convolution and activation function ReLU are adopted to process the features, where ADD is the addition operation and MUL is the multiplication; operation, to obtain the function Xcavg; at the same time, the max pooling module, convolution and activation function ReLU are used in parallel with the average pooling module for another feature extraction to obtain the function XcMax; the designed attention feature map has features of both average pooling and max pooling; the attention feature is multiplied with the input feature map and superimposed with the input feature to as the weight to influence the input feature map; finally, a structure similar to the jump connection was used to reduce the negative impact of the attention module on the input feature map, and the Sigmoid activation function was used to output the final feature map; the designed module can not only efficiently guide the acquisition of intraclass information through the average pooling operation, but also extracts more edge information through the max pooling operation, which can efficiently improve the acquisition of feature information; spatial attention module: by using the spatial attention module, useful information in the input image can be focused on; Figure 8 shows the designed spatial attention module, which focuses on the spatial or semantic feature information in the feature map; by use of global average pooling, the length and width of the feature map are compressed into one, leaving only the spatial information; then, the convolutional layer is used to learn the association between spatial information and classification information (semantic information), and batch regularization and activation function Sigmoid are used to transform this association into nonlinear change; to avoid excessive loss of feature information caused by pooling, it was multiplied by the feature map without average pooling; the result of multiplication is input to the next module as the weight to influence the input feature map, so as to complete the task of refining semantic information; Section 2.3 with FIGS. 9-11 and Table 1 in Pages 11-14: a new deep neural network model is effectively constructed by constructing a residual network model and designing an attention model to enhance the feature learning ability of MFL in-line inspection signals to improve the identification accuracy; to solve the problems of gradient dispersion or explosion and network degradation caused by network deep stacking, Kaiming He et al. proposed a new network structure, namely, residual network (ResNet), which constructs a new deep network by introducing a residual block; the essence of the ResNet design is to ensure that the internal structure of the model has the ability of identity mapping so that the deep network has the same performance as the shallow network; through identity mapping, there is no degradation due to continued stacking in the process of stacking the network; the block structure of the Identity Residual module (Identity_RES) is shown in Figure 9; the output H(x) in model is: H(x) = F(x) + x, where F(x) is the residual mapping after learning, H(x) is the low-level mapping of the partial fitting, and x is the input vector; usually, F(x) is expressed as F(x, { Wn }) to highlight the relationship between the input weights and update weights; therefore, the n-th residual unit can be expressed as in Equations (7)-(8), where xn+1 and xn represent the output and input of the n-th residual unit, respectively; h(xn) represents unit mapping, and fReLU is the ReLU activation function; as can be seen from Equation (9), the features for learning from layer n to layer N are shown in Equations (9)-(10), where Equation (10) represents the actual updated gradient of the loss when passing through the n-th layer; the first part in the formula represents the preserved gradient of directly transmitting the original features through the identity channel; the second part is the residual gradient related to the weight parameters of the residual network; if the output size of the previous layer and the input size of the current layer do not match each other, it is necessary to add a convolutional layer to match the output of the previous layer, that is, the Convolutional Residual module (Conv_RES); the structure of the convolutional residual module is shown as in Figure 10; an MFL signal identification method based on improved residual convolutional neural network is proposed, in which a new network is constructed by improving the residual network and introducing an attention module; the feature identification model proposed in this method has the following advantages: (a) a convolution network is adopted to extract features from the original data, which reduces the difficulty of feature extraction and enhances the universal applicability of pipeline defects and pipeline features; (b) the attention layer is added to obtain the weighted feature map under the joint action of channel attention and spatial attention to further extract the image feature information and reduce the noises impact on the feature identification results during the inspection in the pipeline; and (c) two different residual modules (identity and convolution) are introduced to deepen the depth of the deep network but to effectively reduce the computing number of parameters, decrease the errors of the deep network, save the training time, and improve the training effect; the improved residual convolutional neural network model based on VGG16 proposed in this method is illustrated in Figure 11; the input of the entire network is a pseudo-color image of the MFL in-line inspection; after data preprocessing, the image size was set to 112 × 112 × 3; after normalization, the RGB value is converted to the range of (0, 1); the normalized image is input to the first convolutional layer which has sixteen 3 × 3 convolution kernels with a stride of 1, then is input into Channel_AM and Spatial_AM after passing through the batch standardization layer and activation layer; it then enters three consecutive identical residual modules, each of which includes 16 convolution kernels with a size of 3 × 3 and a stride of 1, and has an output of 112 × 112 × 16; next, the feature map is fed into the convolutional residual module, and on the one hand, the feature map A is obtained through ReLU activation function → convolution → batch standardization → ReLU activation function → convolution → batch standardization, on the other hand, feature map B is obtained after convolution → batch standardization; more features can be extracted from feature maps A and B through the superposition of the merging layer, and the size of the output feature map was 56 × 56 × 32; then, the feature map passes through two identical residual modules, one convolutional residual module, and two identical residual modules, and an output image of 28 × 28 × 64 is obtained; the obtained feature map was fed into the global mean pooling layer to reduce the number of parameters and overfitting; finally, it was connected to the full connection layer using Softmax for classification; the specific parameters for the feature identification and classification of the actual network structure are listed in Table 1).
Su in view of Guo, and Liu are analogous art because they are from the same field of endeavor, a system and a method for identification of defects/faults using convolutional neural network. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Liu to Su in view of Guo. Motivation for doing so would reduce the error of the deep network as well as the amount of calculation parameters, and effectively improve the training efficiency .
Su in view of Guo and Liu fails to explicitly disclose converting the monitoring data segments from different sensors into 2-dimensional (2-D) grayscale images.
WANG teaches a system and a method relating to fault diagnosis using CNN (WANG, ABSTRACT), wherein converting the monitoring data segments from different sensors into 2-dimensional (2-D) grayscale images. (WANG, ABSTRACT in Page 23717: based on the characteristics of mechanical vibration signal propagation in space, a new multi-sensor information fusion method is proposed to implement fault classification; construct the time-domain vibration signals of multiple sensors from different position into a rectangular two-dimensional matrix, and then uses an improved 2D CNN to realize signal classification; SECTION I in Pages 23717- 23718: considering the characteristics of the vibration signal of the mechanical system itself, combined with the characteristics of the neural network calculation, propose a method of obtaining the vibration signal of the motor by using multi-channel sensors; the data of the multi-channel sensors is strongly correlated with the row and column in the proposed method; secondly, combined with the characteristics of signal input, the CNN model is improved; the convolution kernel with different lengths and columns is innovatively used, and the CNN model is simplified; thirdly, the above proposed method is verified on the public data set, and compared with the existing method and the previous method, the superiority and accuracy of the proposed method are verified; SECTION II.A in Page 23718: the original data of the vibration signal is segmented and stacked as two-dimensional image or the time-frequency image obtained by frequency domain analysis; the images are used as the input of the CNN; SECTION II.B in Page 23718-23719: analyze the vibration signal along the space when the mechanical failure occurs through the qualitative analysis method to reflect the position and extent of the mechanical failure.; the vibration signals captured by multiple sensors can be fused and provide rich information to diagnose faults of rotating machinery; using vibration signals captured by multiple sensors to diagnose faults, it can obtain more comprehensive fault information, avoid the shortcomings of limited information captured by single sensor, and acquire complementary information of sensors at different positions to achieve better diagnosis results; according to the fusion operation level, it can be divided into three kinds of information fusion, named as data level fusion, feature level fusion, and decision level fusion; directly transfer the spatial characteristics of the vibration signal to the CNN, because the CNN does not need a clear intermediate signal feature, and it can obtain some weight matrix by training with labeled data; SECTION III with FIG. 1 in Pages 23719-23720: the signal-to-image conversion method is presented to handle the raw signals from different sensors; use a deep learning model to automatically extract features, and propose a simple and effective method to convert the raw signals into CNN input; as shown in FIGURE 1, in the proposed signal-to-image conversion method, vibration signals from three different positions of the motor are input in time series, and there are three data points in each sampling time, and the three data points represent the spatial correlation of the motor vibrations, in time dimension, the each of three sensor sequence reflects the change in vibration over time; take time as the horizontal axis and space as the vertical axis; take 90 points in time and 3 points in space; when selecting data on the time dimension, a sliding window mechanism is adopted; after selecting 90 points, each data selection window moves forward 45 time points; this constitutes an image in which both the row and the column of the motor vibration signal are correlated; when the image is input to the neural network, normalization of (-1, 1) is made; after the raw signals have been converted to images, the CNN model can be trained to classify these images; the size of the convolution kernel can be adjusted according to the specific conditions of the dataset; the convolution kernel is used to extract the local region features; after the previous analysis, the image transformed by the vibration signals from different sensors have strong correlation on the rows and columns; according to the characteristics of the input signal, the data along the time axis has a large amount of data, therefore, we considered to use a convolution kernel of different length and width to extract the features of the signal; specifically, the shape of the convolution kernel can be adjusted according to the characteristics of the data set; kernels in the convolutional layer can better suppress high frequency noise compared with small kernels; the proposed CNN with long kernels can acquire good representations of the input signals and improve the performance of the network; in this way, the model is simplified, making it a reality to deploy a fault diagnosis CNN model at the edge).
Su in view of Guo and Liu, and WANG are analogous art because they are from the same field of endeavor, a system and a method relating to fault diagnosis using CNN. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to apply the teaching of Liu to Su in view of Guo. Motivation for doing so would (1) prevent complex data pre-processing; and (2) .
Claim 15
Su in view of Guo, Liu, and WANG discloses all the elements as stated in Claim 14 and further discloses wherein the segments are based on the sampling rate of the monitoring sensors (Su, ¶¶ [0006], [0050]-[0051], and [0053] with FIG. 2: dividing the sample set into a support set (i.e., for constructing fault domain data) and a query set (i.e., query samples from vibration signal) by using a sample set dividing module; FIG. 2 also shown a plurality of segments in a query set (queryj); processing vibration data of the rolling bearing to be diagnosed and identified; before dividing a sample set, normalizing original vibration signal samples; ¶ [0080]: collect the belt end bearing Y-axis vibration signal of the simulator under the 44Hz conversion frequency, and the sampling frequency is 10240 Hz; specifically, each set of health status data was repeatedly collected 6 times; after obtaining the vibration signals of five different states, data preprocessing is required for the vibration signals, and first, the vibration data with a length of 102400 is divided into 25 samples, each sample containing 4096 data points, so that the number of samples per class is 25 × 6 = 150) and the rotating speed of monitored rotating components (Guo, Step 2 of Section II in Page 8007: resample vibration signals with a virtual sampling frequency (VSF) according to rotating speed; Section III.A in Pages 8007-8008: when the sampling frequency of the vibration signal is constant, under the same rotating speed, the same scale corresponds to the same MRF and the information map built for a bearing will always be the same size; however, when rotating speed varies, with this constant sampling frequency, CWCMs will have different sizes; to address this problem, this article adopts a data preprocessing method in which vibration signals are resampled with a VSF f by a polynomial interpolation function to make sure the processed signals have the same MRF).
Claim 16
Su in view of Guo, Liu, and WANG discloses all the elements as stated in Claim 14 and further discloses wherein the training is terminated once DR-CNN performance satisfies pre-defined requirements or the training epoch reaches a pre-determined threshold (Liu, Section 3.3 with FIG. 14 in Pages 16-17: network training was conducted for 80 cycles, totaling 5600 iterations for training; the curve of the model loss value changing with the number of iterations in the training process and the prediction accuracy curve of the training and testing changing with the number of iterations are shown in Figure 14; the experimental results showed that the loss value of the model decreased rapidly and the accuracy increased rapidly in the first 2000 training sessions; at approximately the 3000th training session, the model began to converge, and finally when the training reached 3500 iterations, the loss value of the model tended to be stable) (Guo, Step 4 of Section II in Page 8007: adopt the dynamic learning-rate method to train the multitask CNN until the training termination condition is met; Section V.C.(1) in Page 8013: the training is terminated when the objective errors of the two tasks become less than 0.0005).
Claim 17
Su in view of Guo, Liu, and WANG discloses all the elements as stated in Claim 16 and further discloses wherein the training is terminated once the training epoch reaches at least 200 iterations (Liu, Section 3.3 with FIG. 14 in Pages 16-17: network training was conducted for 80 cycles, totaling 5600 iterations for training; the curve of the model loss value changing with the number of iterations in the training process and the prediction accuracy curve of the training and testing changing with the number of iterations are shown in Figure 14; the experimental results showed that the loss value of the model decreased rapidly and the accuracy increased rapidly in the first 2000 training sessions; at approximately the 3000th training session, the model began to converge, and finally when the training reached 3500 iterations, the loss value of the model tended to be stable).
Claim 18
Su in view of Guo, Liu, and WANG discloses all the elements as stated in Claim 14 and further discloses wherein the DR-CNN includes two different attention modules employed to enhance learning ability of fault related discriminate features (Liu, Section 2.2.3 with FIGS. 7-8 in Pages 9-11: to improve the deep convolution network and propose a new mechanism to enhance the signal feature extraction ability of the network, the attention model was derived from a human visual attention model; while processing data, the visual system quickly focuses on the target areas that need to be focused by scanning the global scene, and allocates limited computing resources to these key parts; this mechanism can greatly reduce the amount of data to be processed, ignore unimportant information, and provide more manageable and relevant information for higher-level perceptual reasoning and complex visual processing; it is one of the core technologies in deep learning worthy of attention and in-depth understanding; the convolutional block attention module (CBAM) is an attention module combining spatial with channel information; CBAM adopts max-pooling and average-pooling to generate weights through the channel and spatial dimensions; adding the attention mechanism module can further extract the interested small defects target area from the background, thus the network can better learn small target defects; at the same time, the interference of the background to the target is suppressed, thus improving the learning ability of the network to the detailed features of small targets and enhance the ability of feature learning; the method introduces the attention mechanism and designs the spatial attention module (Spatial_AM) and the channel attention module (Channel_AM); in Channel_AM, the property of maximum pooling is used to capture the inter class information between MFL image pixels, and the average pooling is used to capture the intra class information between pixels; these two information as weights are applied to the original feature map as attention to assist feature extraction; the module is connected between the feature map extraction module and the feature map decoding module; at the same time, the Spatial_AM is also designed; by using the spatial attention mechanism composed of global pooling, convolution, and activation function, the semantic information extraction is further refined, and the information is multiplied with the original feature map as a weight.; Channel attention module: when extracting features in the channel dimension, average pooling and max pooling are considered simultaneously; Figure 7 shows the design scheme of the channel attention module; the module consists of convolution, batch regularization, and an activation function, which can extract mixed information by integrating channel and semantic information; then, the average pooling module, convolution and activation function ReLU are adopted to process the features, where ADD is the addition operation and MUL is the multiplication; operation, to obtain the function Xcavg; at the same time, the max pooling module, convolution and activation function ReLU are used in parallel with the average pooling module for another feature extraction to obtain the function XcMax; the designed attention feature map has features of both average pooling and max pooling; the attention feature is multiplied with the input feature map and superimposed with the input feature to as the weight to influence the input feature map; finally, a structure similar to the jump connection was used to reduce the negative impact of the attention module on the input feature map, and the Sigmoid activation function was used to output the final feature map; the designed module can not only efficiently guide the acquisition of intraclass information through the average pooling operation, but also extracts more edge information through the max pooling operation, which can efficiently improve the acquisition of feature information; spatial attention module: by using the spatial attention module, useful information in the input image can be focused on; Figure 8 shows the designed spatial attention module, which focuses on the spatial or semantic feature information in the feature map; by use of global average pooling, the length and width of the feature map are compressed into one, leaving only the spatial information; then, the convolutional layer is used to learn the association between spatial information and classification information (semantic information), and batch regularization and activation function Sigmoid are used to transform this association into nonlinear change; to avoid excessive loss of feature information caused by pooling, it was multiplied by the feature map without average pooling; the result of multiplication is input to the next module as the weight to influence the input feature map, so as to complete the task of refining semantic information).
Claim 19
Su in view of Guo, Liu, and WANG discloses all the elements as stated in Claim 18 and further discloses wherein the two different attention models comprise a Channel Attention Module (CAM) and a Non-local Attention Module (NLAM) (Liu, Section 2.2.3 with FIGS. 7-8 in Pages 9-11: to improve the deep convolution network and propose a new mechanism to enhance the signal feature extraction ability of the network, the attention model was derived from a human visual attention model; while processing data, the visual system quickly focuses on the target areas that need to be focused by scanning the global scene, and allocates limited computing resources to these key parts; this mechanism can greatly reduce the amount of data to be processed, ignore unimportant information, and provide more manageable and relevant information for higher-level perceptual reasoning and complex visual processing; it is one of the core technologies in deep learning worthy of attention and in-depth understanding; the convolutional block attention module (CBAM) is an attention module combining spatial with channel information; CBAM adopts max-pooling and average-pooling to generate weights through the channel and spatial dimensions; adding the attention mechanism module can further extract the interested small defects target area from the background, thus the network can better learn small target defects; at the same time, the interference of the background to the target is suppressed, thus improving the learning ability of the network to the detailed features of small targets and enhance the ability of feature learning; the method introduces the attention mechanism and designs the spatial attention module (Spatial_AM) (i.e., non-local attention module) and the channel attention module (Channel_AM); in Channel_AM, the property of maximum pooling is used to capture the inter class information between MFL image pixels, and the average pooling is used to capture the intra class information between pixels; these two information as weights are applied to the original feature map as attention to assist feature extraction; the module is connected between the feature map extraction module and the feature map decoding module; at the same time, the Spatial_AM is also designed; by using the spatial attention mechanism composed of global pooling, convolution, and activation function, the semantic information extraction is further refined, and the information is multiplied with the original feature map as a weight.; Channel attention module: when extracting features in the channel dimension, average pooling and max pooling are considered simultaneously; Figure 7 shows the design scheme of the channel attention module; the module consists of convolution, batch regularization, and an activation function, which can extract mixed information by integrating channel and semantic information; then, the average pooling module, convolution and activation function ReLU are adopted to process the features, where ADD is the addition operation and MUL is the multiplication; operation, to obtain the function Xcavg; at the same time, the max pooling module, convolution and activation function ReLU are used in parallel with the average pooling module for another feature extraction to obtain the function XcMax; the designed attention feature map has features of both average pooling and max pooling; the attention feature is multiplied with the input feature map and superimposed with the input feature to as the weight to influence the input feature map; finally, a structure similar to the jump connection was used to reduce the negative impact of the attention module on the input feature map, and the Sigmoid activation function was used to output the final feature map; the designed module can not only efficiently guide the acquisition of intraclass information through the average pooling operation, but also extracts more edge information through the max pooling operation, which can efficiently improve the acquisition of feature information; spatial attention module: by using the spatial attention module, useful information in the input image can be focused on; Figure 8 shows the designed spatial attention module, which focuses on the spatial or semantic feature information in the feature map; by use of global average pooling, the length and width of the feature map are compressed into one, leaving only the spatial information; then, the convolutional layer is used to learn the association between spatial information and classification information (semantic information), and batch regularization and activation function Sigmoid are used to transform this association into nonlinear change; to avoid excessive loss of feature information caused by pooling, it was multiplied by the feature map without average pooling; the result of multiplication is input to the next module as the weight to influence the input feature map, so as to complete the task of refining semantic information).
Claim 20
Su in view of Guo, Liu, and WANG discloses all the elements as stated in Claim 14 and further discloses wherein the monitoring sensors comprise a plurality of sensors associated with a plurality of rotating components, said sensors including at least one of vibration sensors, triaxial vibration sensors, and Acoustic Emission (AE) sensors (Su, ¶ [0003]: the rolling bearing fault diagnosis method based on deep learning is rapidly developed in the past few years, and fault diagnosis and identification are carried out on vibration signals by utilizing strong feature dimension reduction and mode identification capability of a neural network; ¶¶ [0035]-[0039] with FIGS. 7-11: a schematic diagram of a vibration signal of a rolling bearing collected in a state (a)-(e) by the MFS experimental apparatus (i.e., vibration signals are collected by vibration sensor in the MFS experimental apparatus); ¶ [0080] with FIGS. 7-11: a comparison test is carried out by utilizing a vibration signal of a Machine Fault Simulator (MFS); the experiment simulates 5 health states of the rolling bearing, collects the belt end bearing Y-axis vibration signal of the simulator under the 44Hz conversion frequency, and the sampling frequency is 10240 Hz; each set of health status data was repeatedly collected 6 times; The original vibration waveforms for the five different conditions are shown in FIGS. 7-11) (WANG, SECTION II.B in Pages 23718-23719: in certain rotating machinery with a long rotor, a number of vibration sensors are intentionally arranged at certain intervals to monitor the running state of the system) (Liu, Section 2.2.1 with FIG. 3 in Pages 6-7: different numbers of Hall sensors covering the circumference of the pipeline; i.e., sensors is placed in different directions around the circumference of the pipeline).
Claim 21
Su in view of Guo, Liu, and WANG discloses all the elements as stated in Claim 14 and further discloses wherein the deep residual convolutional neural network (DR-CNN) includes a residual learning unit (RLU) structure having two convolutional layers, two Batch Normalization (BN) layers, and one ReLU activation layer (Liu, Section 2.3 with FIGS. 9-11 and Table 1 in Pages 11-14: a new deep neural network model is effectively constructed by constructing a residual network model and designing an attention model to enhance the feature learning ability of MFL in-line inspection signals to improve the identification accuracy; to solve the problems of gradient dispersion or explosion and network degradation caused by network deep stacking, Kaiming He et al. proposed a new network structure, namely, residual network (ResNet), which constructs a new deep network by introducing a residual block; the essence of the ResNet design is to ensure that the internal structure of the model has the ability of identity mapping so that the deep network has the same performance as the shallow network; through identity mapping, there is no degradation due to continued stacking in the process of stacking the network; the block structure of the Identity Residual module (Identity_RES) is shown in Figure 9 (i.e., the Identity Residual module having two convolutional layers, two Batch Normalization (BN) layers, and one ReLU activation layer); the output H(x) in model is: H(x) = F(x) + x, where F(x) is the residual mapping after learning, H(x) is the low-level mapping of the partial fitting, and x is the input vector; usually, F(x) is expressed as F(x, { Wn }) to highlight the relationship between the input weights and update weights; therefore, the n-th residual unit can be expressed as in Equations (7)-(8), where xn+1 and xn represent the output and input of the n-th residual unit, respectively; h(xn) represents unit mapping, and fReLU is the ReLU activation function; as can be seen from Equation (9), the features for learning from layer n to layer N are shown in Equations (9)-(10), where Equation (10) represents the actual updated gradient of the loss when passing through the n-th layer; the first part in the formula represents the preserved gradient of directly transmitting the original features through the identity channel; the second part is the residual gradient related to the weight parameters of the residual network; if the output size of the previous layer and the input size of the current layer do not match each other, it is necessary to add a convolutional layer to match the output of the previous layer, that is, the Convolutional Residual module (Conv_RES); the structure of the Convolutional Residual module is shown as in Figure 10; an MFL signal identification method based on improved residual convolutional neural network is proposed, in which a new network is constructed by improving the residual network and introducing an attention module; the feature identification model proposed in this method has the following advantages: (a) a convolution network is adopted to extract features from the original data, which reduces the difficulty of feature extraction and enhances the universal applicability of pipeline defects and pipeline features; (b) the attention layer is added to obtain the weighted feature map under the joint action of channel attention and spatial attention to further extract the image feature information and reduce the noises impact on the feature identification results during the inspection in the pipeline; and (c) two different residual modules (identity and convolution) are introduced to deepen the depth of the deep network but to effectively reduce the computing number of parameters, decrease the errors of the deep network, save the training time, and improve the training effect; the improved residual convolutional neural network model based on VGG16 proposed in this method is illustrated in Figure 11; the input of the entire network is a pseudo-color image of the MFL in-line inspection; after data preprocessing, the image size was set to 112 × 112 × 3; after normalization, the RGB value is converted to the range of (0, 1); the normalized image is input to the first convolutional layer which has sixteen 3 × 3 convolution kernels with a stride of 1, then is input into Channel_AM and Spatial_AM after passing through the batch standardization layer and activation layer; it then enters three consecutive identical residual modules, each of which includes 16 convolution kernels with a size of 3 × 3 and a stride of 1, and has an output of 112 × 112 × 16; next, the feature map is fed into the convolutional residual module, and on the one hand, the feature map A is obtained through ReLU activation function → convolution → batch standardization → ReLU activation function → convolution → batch standardization, on the other hand, feature map B is obtained after convolution → batch standardization; more features can be extracted from feature maps A and B through the superposition of the merging layer, and the size of the output feature map was 56 × 56 × 32; then, the feature map passes through two identical residual modules, one convolutional residual module, and two identical residual modules, and an output image of 28 × 28 × 64 is obtained; the obtained feature map was fed into the global mean pooling layer to reduce the number of parameters and overfitting; finally, it was connected to the full connection layer using Softmax for classification; the specific parameters for the feature identification and classification of the actual network structure are listed in Table 1).
Claim 22
Su in view of Guo, Liu, and WANG discloses all the elements as stated in Claim 14 and further discloses wherein: the set of domain knowledge includes at least one of fault mechanisms, expert empirical knowledge, rotating mechanisms, fault characteristics, and common fault patterns that can be extracted from historical monitoring data; and the method further comprises including current operating conditions as part of the fused information images to input to the DR-CNN (Guo, Abstract in Page 8005: domain knowledge, operating conditions, and vibration signals are fused into a three-dimensional input that can be processed well by CNN; Section 1 in Pages 8005-8006: most deep learning-based fault diagnostic methods are totally data driven, ignore the domain knowledge that has been developed and used for fault diagnosis in the last decades; the operating conditions of the system, including rotating speed and load, have significant influence on vibration signals; to make full use of data-driven methods, it is desirable to integrate system and operating information; domain knowledge, such as fault mechanism and characteristics, must be integrated with data to make sure the deep learning methods converge based on fault characteristics; an information map of domain knowledge is built for each bearing as part of the deep learning method input; it helps to locate fault features in CWCMs and guides CNN to converge in a quick and accurate way; the information map includes bearing operating conditions, rotating speed, and loading profile; with these additional multiple-dimensional (m-D) information, quicker convergence and higher classification accuracy can be achieved; signals of multiple sensors are used as the input of the deep leaning algorithm; it helps to achieve high accuracy in fault diagnosis and locate the fault bearing, which provides convenience for maintenance and replacement of bearings; proposes a bearing fault diagnosis and localization approach based on multitask CNN and information fusion, which combines fault characteristic frequencies and operating conditions with signals of multiple sensors; signals of multiple sensors are decomposed into continuous wavelet coefficient matrices (CWCMs) using continuous wavelet transform; then, an information map is built for each bearing based on its structure and operating conditions; the CWCMs and information maps are fused into a three-dimensional (3-D) matrix, which is served as input to multitask CNN for feature extraction; two classifiers are constructed in multitask CNN for fault diagnosis and fault localization, respectively; a multitask CNN structure is proposed with a dynamic training method for both fault diagnosis and fault localization; Section II with FIG. 1 in Pages 8006-8007: based on m-D information fusion and multitask CNN, a novel deep learning approach for simultaneous fault diagnosis and localization of bearing is proposed; the proposed approach aims to address the following challenges: a) the use of domain knowledge in deep learning-based fault diagnosis; b) integrating varying operating conditions information in the input of deep learning; 3) use a single deep learning network for both fault diagnosis and localization; Fig. 1 illustrates the procedure of the proposed method, which consists of construction of CWCMs and information maps, m-D information fusion, multitask CNN training, and fault diagnosis and localization; Step 1: mount accelerometers on bearings under monitoring; collect vibration signals from multiple sensors for different bearings; record operating conditions including rotating speed and load for different bearings; Step 2: resample vibration signals with a virtual sampling frequency (VSF) according to rotating speed; decompose signals into CWCMs using continuous wavelet transform; build information maps by integrating domain knowledge and operating conditions of the bearings; then, CWCMs and information maps are combined to achieve an m-D information matrix as the input of multitask CNN; Step 3: construct a multitask CNN with two classifiers; set parameters of convolutional layers, fully connected layers, and classifiers according to the input size and number of categories; Step 4: prepare training and test data using m-D information matrices; set up two labels (one is for fault diagnosis and the other one is for fault location) for each sample and divide the samples into training data and test data; adopt the dynamic learning-rate method to train the multitask CNN until the training termination condition is met; test the trained system using test data; Step 5: put the trained system into use; process online data using the same VSF and integrate the operating condition into m-D information matrices as real-time input; use the trained multitask CNN for fault diagnosis and localization; Section II.B with FIG. 3 in Pages 8008-8009: the domain knowledge in fault diagnosis includes fault mechanisms, fault characteristics, diagnosis rules, and feature extraction algorithms designed based on mechanical principles, modeling, or data analysis; deep learning algorithms also have the ability to deal with complex heterogeneous data; it is expected that the domain knowledge, transformed to the proper format and integrated in deep learning input, will greatly increase the accuracy of diagnosis and localization; for fault diagnosis of rolling element bearings, one of the most important domain knowledge is the fault characteristics frequency (FCF); according to the fault mechanism, when a bearing fault happens, the fault characteristics frequency components will appear in the vibration signals; the FCFs are calculated as shown in Eqn. (4), where fIR, fOR, and fBA are the FCF of inner race fault, outer race fault, and ball fault, respectively, Z is the number of rolling elements, d is the rolling element diameter, D is the pitch diameter, and α is the contact angle; an information map of FCF is built to make sure that the deep learning model converges in a way related to the FCF; Section II.C in Page 8009 with FIG. 3 in Page 8010: Fig. 4 shows the detailed integration process; in this figure, n accelerometers are used to detect faults of n bearings; through the construction and integration of CWCMs and information maps from multiple resources, m-D information that include vibration signals, domain knowledge of FCFs, and operating conditions are fused into the CNN input).
Claim 23
Su in view of Guo, Liu, and WANG discloses all the elements as stated in Claim 14 and further discloses providing input data sets of monitored rotating component sensor data to the trained DR-CNN, and operating the DR-CNN for outputting fault diagnosis based on such monitored sensor data (Liu, Abstract in Page 1: a pipeline MFL inspection signal identification method based on improved deep residual convolutional neural network and attention module is proposed; an improved deep residual network based on the VGG16 convolution neural network is constructed to automatically learn the features from the MFL image signals and perform the identification of pipeline features and defects; 4th paragraph – 6th paragraph of Section 1 in Pages 2-3: deep learning methods have become a hotspot in the research of signal identification and fault diagnosis in recent years; to further improve the identification accuracy for the complex pipeline MFL signals, a pipeline MFL inspection feature identification model based on an improved deep residual convolutional neural network is proposed; the proposed method not only automatically learns the features from the MFL inspection images and performs the classification and identification of pipeline features and defects such as welds, tees, flanges, and corrosion, but also solves the problems of the great influence of noise, compound features, and other factors on the feature identification results in the process of in-line inspection; the proposed method effectively improves the classification of pipeline features, and provides an effective method for pipeline features identification; an MFL in-line inspection method based on attention module and convolution residual modules is proposed, which effectively improve accuracy and efficiency; aiming at the influence of the complex operating environment, high noises, composite defects to MFL in-line inspection of oil and gas pipelines, attention module composed of channel attention and spatial attention are designed to fully extract MFL image feature information; to solve the problem of gradient dispersion caused by the increase of the number of network layers, an improved residual convolutional neural network is constructed to reduce the error of the deep network as well as the amount of calculation parameters, and effectively improve the training efficiency; Section 2.3 with FIGS. 9-11 and Table 1 in Pages 11-14: a new deep neural network model is effectively constructed by constructing a residual network model and designing an attention model to enhance the feature learning ability of MFL in-line inspection signals to improve the identification accuracy; to solve the problems of gradient dispersion or explosion and network degradation caused by network deep stacking, Kaiming He et al. proposed a new network structure, namely, residual network (ResNet), which constructs a new deep network by introducing a residual block; the essence of the ResNet design is to ensure that the internal structure of the model has the ability of identity mapping so that the deep network has the same performance as the shallow network; through identity mapping, there is no degradation due to continued stacking in the process of stacking the network; the block structure of the Identity Residual module (Identity_RES) is shown in Figure 9; the output H(x) in model is: H(x) = F(x) + x, where F(x) is the residual mapping after learning, H(x) is the low-level mapping of the partial fitting, and x is the input vector; usually, F(x) is expressed as F(x, { Wn }) to highlight the relationship between the input weights and update weights; therefore, the n-th residual unit can be expressed as in Equations (7)-(8), where xn+1 and xn represent the output and input of the n-th residual unit, respectively; h(xn) represents unit mapping, and fReLU is the ReLU activation function; as can be seen from Equation (9), the features for learning from layer n to layer N are shown in Equations (9)-(10), where Equation (10) represents the actual updated gradient of the loss when passing through the n-th layer; the first part in the formula represents the preserved gradient of directly transmitting the original features through the identity channel; the second part is the residual gradient related to the weight parameters of the residual network; if the output size of the previous layer and the input size of the current layer do not match each other, it is necessary to add a convolutional layer to match the output of the previous layer, that is, the Convolutional Residual module (Conv_RES); the structure of the convolutional residual module is shown as in Figure 10; an MFL signal identification method based on improved residual convolutional neural network is proposed, in which a new network is constructed by improving the residual network and introducing an attention module; the feature identification model proposed in this method has the following advantages: (a) a convolution network is adopted to extract features from the original data, which reduces the difficulty of feature extraction and enhances the universal applicability of pipeline defects and pipeline features; (b) the attention layer is added to obtain the weighted feature map under the joint action of channel attention and spatial attention to further extract the image feature information and reduce the noises impact on the feature identification results during the inspection in the pipeline; and (c) two different residual modules (identity and convolution) are introduced to deepen the depth of the deep network but to effectively reduce the computing number of parameters, decrease the errors of the deep network, save the training time, and improve the training effect; the improved residual convolutional neural network model based on VGG16 proposed in this method is illustrated in Figure 11; the input of the entire network is a pseudo-color image of the MFL in-line inspection; after data preprocessing, the image size was set to 112 × 112 × 3; after normalization, the RGB value is converted to the range of (0, 1); the normalized image is input to the first convolutional layer which has sixteen 3 × 3 convolution kernels with a stride of 1, then is input into Channel_AM and Spatial_AM after passing through the batch standardization layer and activation layer; it then enters three consecutive identical residual modules, each of which includes 16 convolution kernels with a size of 3 × 3 and a stride of 1, and has an output of 112 × 112 × 16; next, the feature map is fed into the convolutional residual module, and on the one hand, the feature map A is obtained through ReLU activation function → convolution → batch standardization → ReLU activation function → convolution → batch standardization, on the other hand, feature map B is obtained after convolution → batch standardization; more features can be extracted from feature maps A and B through the superposition of the merging layer, and the size of the output feature map was 56 × 56 × 32; then, the feature map passes through two identical residual modules, one convolutional residual module, and two identical residual modules, and an output image of 28 × 28 × 64 is obtained; the obtained feature map was fed into the global mean pooling layer to reduce the number of parameters and overfitting; finally, it was connected to the full connection layer using Softmax for classification; the specific parameters for the feature identification and classification of the actual network structure are listed in Table 1) (Guo, Step 5 of Section II in Page 8007: put the trained system into use. Process online data using the same VSF and integrate the operating condition into m-D information matrices as real-time input; use the trained multitask CNN for fault diagnosis and localization).
Claim 24
Su in view of Guo, Liu, and WANG discloses all the elements as stated in Claim 14 and further discloses wherein the rotating component fault diagnosis include at least one of fault mode diagnosis, fault localization, and operating condition at the time of fault detection (Guo, Abstract in Page 8005: propose a rolling element bearing fault diagnosis and localization approach based on multitask convolutional neural network (CNN) with information fusion; a multitask CNN with dynamic training rates is constructed to simultaneously accomplish two tasks, fault diagnosis and localization; Section 1 in Pages 8005-8006: proposes a bearing fault diagnosis and localization approach based on multitask CNN and information fusion, which combines fault characteristic frequencies and operating conditions with signals of multiple sensors; two classifiers are constructed in multitask CNN for fault diagnosis and fault localization, respectively; a multitask CNN structure is proposed with a dynamic training method for both fault diagnosis and fault localization; FIG. 2 in Page 8008: CWCMs of vibration signals of bearing in different fault conditions. (a) Normal. (b) Ball fault. (c) Inner race fault. (d) Outer race fault; FIG. 3 in Page 8009: information maps of drive-end and fan-end bearings in CWRU data. (a) Drive-end bearing. (b) Fan-end bearing).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Liang et al. ("Rolling Bearing Fault Diagnosis Based on One-Dimensional Dilated Convolution Network With Residual Connection", IEEE Access, VOLUME 9,, Feb. 16, 2021, pp. 31078-31091) discloses in Abstract of Page 31078 that (1) in order to diagnose the faults of rolling bearing under different noisy environments and different load domains, a new method named one-dimensional dilated convolution network with residual connection is proposed; (2)the proposed method uses the one-dimensional time-domain signals of rolling bearing as input; (3) Zigzag dilated convolution is introduced into convolution neural network, which can effectively improve the receptive field of the convolutional layer; (4) a multi-level residual connection structure with different weight coefficients is constructed, so that the lower layer features of convolution neural network can be transferred to the upper layer, which improves the feature learning ability; and (5) moreover, in order to enhance the useful features and weaken the useless features, we add the attention module Squeeze-and-Excitation (SE) block after each sub-residual structure. Liang further discloses in Section I of Pages 31078-31080 that (1) Li et al. applied the attention mechanism to help the CNN locate fault information, which had a high fault diagnosis accuracy rate under limited data samples; (2) Xu et al. proposed a feature fusion process with attention mechanism and combined this process with deep learning model, finally achieved good generalization performance in the actual bearing variable load environment; (3) a zigzag dilated residual connection block with optimal dilation rate is proposed; (4) the signals of rolling bearing have strong time-varying characteristics; (5) in order to extract the characteristic information, construct a zigzag dilated residual connection block, which can not only expand the receiving field of the convolutional layer, but also avoid the grid effect; (6) then analyze the influence of different dilation rate combinations and determine the optimal dilation rate combination; (7) finally, apply this residual block in the proposed method and obtain excellent experimental results; (8) a residual connection structure with optimal weight coefficients is proposed; (9) in order to enhance the fault diagnosis effect of the proposed method, construct the global residual and sub-residual weight coefficient structure in the network, so that the upper and lower convolutional layers of the network transfer appropriate feature information, and finally analyze the influence of different weight coefficients; (10) a neural network fault diagnosis model with attention mechanism is proposed. In order to improve the feature information recognition ability of the proposed method, add an attention mechanism module (SE block) after each sub-residual structure and use this module to learn the output of useful information features by each sub-residual block and suppress useless Information characteristics; (11) the influences of different dilation rates and different residual connection weight coefficients for the fault diagnosis effect are analyzed through experiments, and experiments in different noise environments and different load domains are carried out; and (12) the results show that the proposed method has higher fault diagnosis accuracy than other methods.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HWEI-MIN LU whose telephone number is (313)446-4913. The examiner can normally be reached Mon - Fri: 9:00 AM - 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela D. Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HWEI-MIN LU/Primary Examiner, Art Unit 2142