DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-2, 4-6 and 8-20 are now pending.
Claim Rejections - 35 USC § 112
Previous 112(b) rejections have been withdrawn in view of amendments made by the applicant.
Response to Arguments
Applicant's amendments filed 03/23/2026 rejections have been fully considered and entered.
Applicant argues that independent claims 1 and 15 are now incorporated with the features of dependent claim 3 and 8, basically stating the neural networks are not any generic networks but a CNN and recurrent neural networks, and steps utilizing neural networks involving extracting features and assessing health based on extracted features cannot be practically performed as a mental process.
In reply, examiner asserts that applicant’s amendments and corresponding arguments have considered persuasive, therefore 101 rejections have been withdrawn.
Applicant further argues that independent claims have now been amended with alternative features of claim 8 which weren’t positively recited before, therefore are not taught by any of the cited reference
In reply, examiner asserts just because alternative features which weren’t deemed to be rejected before does not mean that the references do not teach them, for instance, De Filippo teaches that and wherein the quantification of the one or more anomalous regions (as discussed in paragraphs 107-108) includes a proximity between multiple anomalous regions (anomalous edges are found after processing all pixel on each edge and performing elimination (see fig. 25). As a result, the actual anomalous regions enclosed by these edges are highlighted by a bounding box as clearly shown in fig. 26 showing proximity as highlighted edges between analogous regions, paragraph 117).
Claim Rejections - 35 USC § 101
Previous 101 rejections to claims have been withdrawn in view of amendments made by the applicant.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-6, 8-10, and 12-20 are rejected under 35 U.S.C. 103 as being unpatentable over Aydemir Gurkan et al., “Image-Based Prognostics Using Deep Learning Approach", IEEE Transactions on Industrial Informatics, Vol. 16, No. 9, September 2020 in view of De Filippo et al., WO 2021/224893.
Regarding claim 1, Gurkan discloses a method for monitoring the health of a component of a machine (note that “data used in this article is collected from rotational element thrust bearing” (i.e., this rotational element thrust bearing is a component of the industrial system/machine, page 5962, col.1, paragraph 4, where estimating time to failure (TTF) of industrial system utilizing LSTM networks for condition monitoring and health management with health status, abstract, page 5956, col. 2, lines 1-7), comprising:
obtaining non-destructive structural integrity testing data of a component of a machine (abstract, page 5956, col. 2, paragraphs 1-2, dataset of infrared image streams is collected from machinery and degradation image streams are utilized by using infrared thermography for estimating TTF (i.e., non-destructive structural integrity testing data of machine));
utilizing a first machine learning algorithm (“all of the convolutional, LSTM and fully connected layer parameters are trained using BPTT algorithm”, page 5959, col. 2, 5th paragraph), which comprises a feature extraction neural network (page 5959, fig. 3, 1st convolutional layer and also “methodology consists of two convolutional layers preceding a single LSTM layer to extract compact information” abstract, page 5958, B. ‘CNNs’) convolutional neural network (abstract, page 5957, 1st paragraph, architecture with convolution layers as shown in fig. 1, page 5957, 3rd paragraph, page 5958, B. ‘CNNs’), to extract and quantify one or more features of the component from the non-destructive structural integrity testing data (page 5957 (Overview of Methodology, 3rd paragraph), “two convolutional layers with two pooling layers extract the spatial information from the individual images in the degradation streams” with vectorization of features (just above LSTM layer) as shown in fig. 3);
utilizing a second machine learning algorithm (“all of the convolutional, LSTM and fully connected layer parameters are trained using BPTT algorithm”, page 5959, col. 2, 5th paragraph), which comprises health assessment neural network (LSTM Layer as shown in fig. 3, page 5959 and LSTM layer is after convolution layers and which estimates TTF value from the extracted features, abstract, also note page 5959, fig. 3 and C. ‘LSTM Networks’, LSTM networks are subclass of recurrent neural networks (RNNs)) which is a recurrent neural network (abstract, page 5957, 1st paragraph, architecture with LSTM layer with neural network, page 5959, fig. 3 and C. ‘LSTM Networks’, LSTM networks are subclass of recurrent neural networks (RNNs)), to perform a health assessment of the component based on the extracted features (page 5959 and as shown in fig.3, the linear regression such as TTF(t), where TTF is the health management with condition monitoring and health status as discussed above in page 5956, col. 2, lines 1-7, is projected by the LSTM layer based on the features extracted/outputted from the convolution layers, also described in abstract)
wherein the feature extraction neural network and the health assessment neural network are different from each other (convolution layer (first type) is different from LSTM layer (second type) as clearly shown in fig.3, abstract);
and providing a notification of the health assessment (abstract, page 5956, col. 2, lines 1-7, providing an effective tool for predictive maintenance practitioners toward digitization of maintenance processes (i.e., providing notification to practitioner) based on deep learning for estimating time-to-failure (TTF) of an industrial system using its degradation image where TTF estimation/prediction is given as output);
wherein the non-destructive structural integrity testing data comprises one or more images of the component (abstract, page 5956, col. 2, paragraphs 1-2, dataset of infrared image streams).
Gurkan fails to further disclose extracted features comprise one or more anomalous regions of the one or more images; and wherein the quantification of the one or more anomalous regions includes a proximity between multiple anomalous regions.
However, De Filippo teaches extracted features comprise one or more anomalous regions of the one or more images (detecting of anomalous regions in images (i.e., defects) such as “visual images or videos in the sensed data may be fed into the visual sub-module to locate the defects”, paragraph 69 and to predict the remaining life/health of a component such as “upon the detections of respective defects, data in relation to the defects may be fed to an optional prediction step… At prediction step, the remaining lifetime of the defects and/or structures is estimated by using the prediction trained model. In some examples, the prediction model may be combined with at least one neural network”, paragraph 70); and wherein the quantification of the one or more anomalous regions (as discussed in paragraphs 107-108) includes a proximity between multiple anomalous regions (anomalous edges are found after processing all pixel on each edge and performing elimination (see fig. 25). As a result, the actual anomalous regions enclosed by these edges are highlighted by a bounding box as clearly shown in fig. 26 showing proximity as highlighted edges between analogous regions, paragraph 117).
Gurkan and De Filippo are combinable because they both are in the same field of endeavor dealing with predictive analysis and health assessment of a component.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to combine the teachings of Gurkan with the teachings of De Filippo for the benefit of providing efficient and effective understanding on the remaining life and risk related to existing buildings and components as taught by De Filippo at paragraph 4.
Regarding claim 2, Gurkan further discloses wherein the second machine learning algorithm further bases the health assessment of the component on a service history of the component (page 5957, col.1, 2nd paragraph, last step (1), “combining deep learning approaches to separate learning of spatial and temporal content of the image streams to improve prediction accuracy in the context of prognostics”, i.e., separate learning of spatial and temporal content of the image streams is interpreted as service history of component).
Regarding claim 4, Gurkan further discloses wherein the health assessment comprises an estimate of a remaining life of the component (abstract, page 5956, 1st paragraph, lines 3-6, in estimating TTF consists of achieving “a profitable predictive maintenance, a smart machine requires efficient predictive analytics to estimate the remaining life of its components in order to take self-corrective actions”).
Regarding claim 5, Gurkan further discloses wherein the health assessment comprises an estimated amount of time the component can be used in the machine before a next maintenance event should be performed for the component (page 5956, last paragraph – page 5957, 1st paragraph, time series data is captured for estimating Time-To-Failure (TTF) to accurately predict remaining useful life (RUL) of components of rotatory machinery providing for instance an estimation of how much (amount of time) useful life is left of turbofan engines before time to failure and thus requiring maintenance event as part of predictive maintenance).
Regarding claim 6, Gurkan further discloses wherein the health assessment comprises an indication of whether it is acceptable to continue using the component in the machine (page 5956, last paragraph, last line – page 5957, 1st paragraph, accurately predicting remaining useful life (RUL) of components of rotatory machinery providing for instance an estimation of how much useful life is left of turbofan engines indicating how much longer is it acceptable to continue using this component is common practice in TTF estimation as part of predictive maintenance described in abstract).
Regarding claim 8, Gurkan fails to further disclose wherein the quantification of the one or more anomalous regions also includes at least one of a depth of the anomalous region, a width of the anomalous region, a length of the anomalous region, an area of the anomalous region, a location of the anomalous region, a severity of the one or more anomalous regions, and a shape of the anomalous region.
However, De Filippo teaches wherein the quantification of the one or more anomalous regions includes at least one of a depth of the anomalous region, a width of the anomalous region (De Filippo, paragraph 108, defects assessment in the form of crack width), a length of the anomalous region, an area of the anomalous region (De Filippo, paragraph 108, defects assessment in the form of defect area), a location of the anomalous region (De Filippo, paragraph 108, bounding box), a severity of the one or more anomalous regions (De Filippo, “severity of each defect”, paragraph 108), and a shape of the anomalous region (De Filippo, object segmentation to assess the severity of detected defects, paragraph 107).
Gurkan and De Filippo are combinable because they both are in the same field of endeavor dealing with predictive analysis and health assessment of a component.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to combine the teachings of Gurkan with the teachings of De Filippo for the benefit of providing efficient and effective understanding on the remaining life and risk related to existing buildings and components as taught by De Filippo at paragraph 4.
Regarding claim 9, Gurkan further discloses wherein the one or more images are x-ray images (page 5956, col. 2, 1st paragraph, X-ray imaging).
Regarding claim 10, Gurkan further discloses wherein the one or more images are computed tomography (CT) images (page 5963, col. 2, [6], “X-ray tomographic imaging”).
Regarding claim 12, Combination of Gurkan with De Filippo further teaches wherein the one or more images are ultrasonic c-scan images (De Filippo, deep learning algorithm may allow operators to train the new defect types via ultrasonic sensors, which may perform a scan which is a process in which the beam is generated from an emitter of a scanner which is reflected back from the target to be received by a receiver in an instrument so that an accurate location of a point of reflection may be calculated, paragraphs 49, 52).
Gurkan and De Filippo are combinable because they both are in the same field of endeavor dealing with predictive analysis and health assessment of a component.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to combine the teachings of Gurkan with the teachings of De Filippo for the benefit of providing efficient and effective understanding on the remaining life and risk related to existing buildings and components as taught by De Filippo at paragraph 4.
Regarding claim 13, Combination of Gurkan with De Filippo further teaches wherein the non-destructive structural integrity testing data (Gurkan, abstract, page 5956, col. 2, paragraphs 1-2, dataset of infrared image streams) comprises vibration testing data from vibration testing of the component (Gurkan, vibration sensor, page 5956, col. 1, 1st paragraph and De Filippo, paragraphs 49, 89, 6, vibration analysis from vibration testing of object).
Gurkan and De Filippo are combinable because they both are in the same field of endeavor dealing with predictive analysis and health assessment of a component.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to combine the teachings of Gurkan with the teachings of De Filippo for the benefit of providing efficient and effective understanding on the remaining life and risk related to existing buildings and components as taught by De Filippo at paragraph 4.
Regarding claim 17, Gurkan further discloses wherein the non-destructive structural integrity testing data comprises one or more images of the component (abstract, page 5956, col. 2, paragraphs 1-2, dataset of infrared image streams); and the one or more images include at least one of the following: x-ray images (page 5956, col. 2, 1st paragraph, X-ray imaging), computed tomography (CT) images (page 5963, col. 2, [6], “X-ray tomographic imaging”), fluorescent penetrant inspection (FPI) images.
Gurkan fails to further disclose extracted features comprise one or more anomalous regions of the one or more images.
However, De Filippo teaches extracted features comprise one or more anomalous regions of one or more images (detecting of anomalous regions in images (i.e., defects) such as “visual images or videos in the sensed data may be fed into the visual sub-module to locate the defects”, paragraph 69 and to predict the remaining life/health of a component such as “upon the detections of respective defects, data in relation to the defects may be fed to an optional prediction step… At prediction step, the remaining lifetime of the defects and/or structures is estimated by using the prediction trained model. In some examples, the prediction model may be combined with at least one neural network”, paragraph 70); and the one or more images include at least one of the following: ultrasonic c-scan images (deep learning algorithm may allow operators to train the new defect types via ultrasonic sensors, which may perform a scan which is a process in which the beam is generated from an emitter of a scanner which is reflected back from the target to be received by a receiver in an instrument so that an accurate location of a point of reflection may be calculated, paragraphs 49, 52).
Gurkan and De Filippo are combinable because they both are in the same field of endeavor dealing with predictive analysis and health assessment of a component.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to combine the teachings of Gurkan with the teachings of De Filippo for the benefit of providing efficient and effective understanding on the remaining life and risk related to existing buildings and components as taught by De Filippo at paragraph 4.
Regarding claim 18, is an apparatus claim version of claim 13 reciting similar features and thus is rejected on the same rationale as presented for claim 13.
Regarding claim 14, Gurkan further discloses wherein the component is a first type of component (page 5962, 4th paragraph, “data used in this article is collected from rotational element thrust bearing” (i.e., this rotational element thrust bearing is the first type of component) and the method comprises:
training the second machine learning algorithm (“all of the convolutional, LSTM and fully connected layer parameters are trained using BPTT algorithm”, page 5959, col. 2, 5th paragraph) based on predefined component life acceptance parameters for the first type of component (page 5959, col.2, equation (9), TTFi is the predefined component life acceptance parameters);
comparing a plurality of the quantified extracted features for a plurality of the components of the first type with actual end of life data for the plurality of components (page 5959, col.2, equation (9) defines comparison between TTFi (i.e., predefined component life acceptance parameters) and TTF^i (i.e., prediction based on plurality of the quantified extracted features);
and dynamically adjusting the predefined component life acceptance parameters based on the comparing (page 5959, col. 2, paragraphs 2-5, the BPTT algorithm is a back-propagation algorithm and as commonly known in the art that back-propagation algorithm adjusts weights of trained neural network based on a function such as comparison function as described in equation (9), thus, dynamically adjusting the predefined component life acceptance parameters (output of equation 9) based on the comparing)
Regarding claim 15, Gurkan discloses a computing device (estimating time to failure (TTF) of industrial system, abstract, where system could be a smart machine, page 5956, col. 1, paragraph 1 such as NVIDIA board, page 5962, col.1, paragraph 3), comprising: processing circuitry operatively connected to memory (Long-Short Term Memory (LSTM) network, abstract), the processing circuitry configured to: utilize a first machine learning algorithm (“all of the convolutional, LSTM and fully connected layer parameters are trained using BPTT algorithm”, page 5959, col. 2, 5th paragraph), Rest of the claim recites similar features as claim 1 and thus is rejected on the same rationale.
Regarding claim 16, Gurkan further discloses wherein the health assessment comprises at least one of: an indication of whether it is acceptable to continue using the component in the machine (page 5956, last paragraph, last line – page 5957, 1st paragraph, accurately predicting remaining useful life (RUL) of components of rotatory machinery providing for instance an estimation of how much useful life is left of turbofan engines indicating how much longer is it acceptable to continue using this component is common practice in TTF estimation as part of predictive maintenance described in abstract); an estimate of a remaining life of the component (abstract, page 5956, 1st paragraph, lines 3-6, in estimating TTF consists of achieving “a profitable predictive maintenance, a smart machine requires efficient predictive analytics to estimate the remaining life of its components in order to take self-corrective actions”); and an estimated amount of time the component can be used in the machine before a next maintenance event should be performed for the component (page 5956, last paragraph – page 5957, 1st paragraph, time series data is captured for estimating Time-To-Failure (TTF) to accurately predict remaining useful life (RUL) of components of rotatory machinery providing for instance an estimation of how much (amount of time) useful life is left of turbofan engines before time to failure and thus requiring maintenance event as part of predictive maintenance).
Regarding claim 19, is an apparatus claim version of claim 2 reciting similar features and thus is rejected on the same rationale as presented for claim 2.
Regarding claim 20, is an apparatus claim version of claim 14 reciting similar features and thus is rejected on the same rationale as presented for claim 14.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Aydemir Gurkan et al., “Image-Based Prognostics Using Deep Learning Approach", IEEE Transactions on Industrial Informatics, Vol. 16, No. 9, September 2020 in view of De Filippo et al., WO 2021/224893 as applied in claim 1 above and further in view of Anonymous, “Nondestructive testing”, Wikipedia, November 19, 2023, https://en.wikipedia.org/wiki/Nondestructive_testing.
Regarding claim 11, Combination of Gurkan with De Filippo fails to further teach wherein one or more images are fluorescent penetrant inspection (FPI) images.
However, Anonymous teaches wherein one or more images are fluorescent penetrant inspection (FPI) images (nondestructive liquid penetrant testing of images involving fluorescent dyes, page 2, paragraph 2).
Gurkan and De Filippo are combinable with Anonymous (Wikipedia) because they all are in the same field of endeavor dealing with predictive and non-destructive analysis of a component.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to combine the teachings of Gurkan and De Filippo with the teachings of Anonymous (Wikipedia) for the benefit of providing effective NDT methods to avoid a sudden failure of a component that would cause significant hazard or economic loss as taught by Anonymous (Wikipedia) at page 2 (applications).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
QIN et al., US 2023/0141864
Welf et al., US 2021/0073962
Lim et al., US 2018/0322366
Somanath et al., US 2020/0173885
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAWANDEEP DHINGRA whose telephone number is (571) 270-1231. The examiner can normally be reached 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abderrahim Merouan can be reached at (571) 270-5254. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAWAN DHINGRA/Examiner, Art Unit 2683
Prosecution Insights