DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114 was filed in this application after a decision by the Patent Trial and Appeal Board, but before the filing of a Notice of Appeal to the Court of Appeals for the Federal Circuit or the commencement of a civil action. Since this application is eligible for continued examination under 37 CFR 1.114 and the fee set forth in 37 CFR 1.17(e) has been timely paid, the appeal has been withdrawn pursuant to 37 CFR 1.114 and prosecution in this application has been reopened pursuant to 37 CFR 1.114.
Applicant’s submission filed on 05/17/2026 has been entered.
Response to Arguments
Applicant's arguments filed on 05/17/2026 have been fully considered but are moot in view of the new ground(s) rejection in view of Mian et al (U.S. 20100100275 A1; Mian).
Claim Status
Claims 17-34 is/are interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim(s) 17-36 is/are rejected under 35 U.S.C. 103 as being unpatentable over Junhua Sun et al (“Automatic multi-fault recognition in TFDS based on convolutional neural network” ; Sun), in view of Mian et al (U.S. 20100100275 A1; Mian).
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “a deep learning engine”, “an objector inspector”, “image compressing unit”, “ pre-analysis inspector”, “ post-analysis inspector”, “sensing device”, “trigger logic”, “a data acquirer”, “an image compression unit”, “a deep learning trainer”, “an inspection component”, “ pre-analysis inspection”, “ post-analysis inspection” in claims 17-34.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification discloses in Paragraph 32: “ The computer system 30 is shown including a processing component 32 (e.g., one or more processors), … the processing component 32 executes program code, such as the inspection program 40, which is at least partially fixed in storage component 34” as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 17-36 is/are rejected under 35 U.S.C. 103 as being unpatentable over Junhua Sun et al (“Automatic multi-fault recognition in TFDS based on convolutional neural network” ; Sun), in view of Mian et al (U.S. 20100100275 A1; Mian).
Regarding claims 17, Sun discloses an environment for inspecting an object of an identifiable apparatus, (1.Introduction: “First, it automatically captures images of some vital parts of freight trains. Then, the images are transmitted to the data server in monitor room for analysis by indoor inspectors to find out the faults of relevant parts. By observing the TFDS images, the indoor inspectors will inform the outdoor inspectors to confirm and handle the problem if the fault exists”) the environment comprising:
an inspection system (1. Introduction: “The whole system is showed in Fig. 1. AFRS is a two-stage system”) including:
a deep learning engine configured to implement a deep learning model (The CNN-based detection model) in order to analyze apparatus image data (Fig.3: TFDS images) and identify object image data, (Fig.3: shaft bolt (SB), side frame key (SFK), and end bolt (EB).) wherein the object image data corresponds to a region of interest for the object (3.2.1. Coarse Detection: “With a trained CNN-based detection network, the predicted bounding-boxes of the three critical parts are detected, including 10–20 bounding-boxes for each category per image.”) of the identifiable apparatus in the apparatus image data; (3.2: Target region Detection: “The CNN-based detection model detects the critical parts in TFDS images including shaft bolt (SB), side frame key (SFK), and end bolt (EB). The critical parts are showed in Fig. 3 … Accurate detection of EB is of great importance to ensure a higher accuracy.”) and
an object inspector configured to provide an inspection outcome for the object of the identifiable apparatus based on the object image data, (3.3. Fault determination: “in the second stage of AFRS, we establish another CNN model to determine multiple faults. The architecture is showed in Fig. 6 … we just take this CNN model for all fault determination by separate training, which means that we just respectively train the network with corresponding dataset aiming at a specific region.”)
wherein the inspection outcome for the object is determined by comparing the object image data with a set of reference equipment (Each train image) related to the object. (Fig.7 and 4.1. Dataset description: “We collect the images (1400 pixel×1024 pixel) from TFDS and randomly select images to build the train set and test set. Each train image is labelled according to the format of the PASCAL VOC dataset [30]. In the process of target region detection, the train set owns 2321 images. In the process of fault determination, 2000 SB regions including 640 fault regions and 1500 SFK regions including 540 fault regions are cropped from the TFDS images for training.”)
However, Sun does not disclose wherein the inspection outcome for the object is determined by directly comparing the object image data with image data corresponding to at least one of a set of reference equipment images related to the object.
Mian discloses an object inspector configured to provide an inspection outcome for the object of the identifiable apparatus based on the object image data, (Paragraph 59: “advanced analysis component 42C can determine whether any particular conditions or faults of interest are indicated by the image data … the image processing includes performing edge detection and segmentation upon an image (e.g., using thresholding processes), assembling/recognizing individual segments as part of one or more features (feature extraction), and assembling the features into "blobs" or objects, which can be compared against known likely objects using, for example, templating, expert system recognition, and/or the like.”) wherein the inspection outcome for the object is determined by directly comparing the object image data with image data corresponding to at least one of a set of reference equipment images related to the object. (Paragraph 64: “advanced analysis component 42C can analyze an individual image of a rail wheel 8 for anomalies. … After detection and definition of edges and features by analysis component 42B, advanced analysis component 42C can compare the identified features of the wheel 8 and related assemblies to a typical profile.”; Paragraph 75: “advanced analysis component 42C can determine which locations, if any, on the rail wheel may be hot, and compare these locations with the structure of the wheel and related components to determine the actual condition that may be present.”; Paragraph 84: “system 10 detects one or more defects by comparing the temperature of a component currently being analyzed with components of the same type that are adjacent (e.g., on the same vehicle) or have been recently analyzed (e.g., on a recently imaged vehicle). When a sufficient difference is noted between one component and other comparable components, system 10 can identify the component as including a defect.”, it shows that “ related component” and/or “components of the same type” and/or “other comparable components” is interpreted as “at least one of a set of reference equipment images related to the object”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Sun by including analysis subsystem that is taught by Mian, to make the invention that analysis of various components of a vehicle using thermal image data; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving improve accurately determine the presence or absence of the various flaws or faults as well as reducing false positives/negatives.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claims 18, Sun, as modified by Mian, discloses all the claims invention. Sun further discloses the object inspector comprises at least two sub-components, the at least two sub-components including: a pre-analysis inspector configure to receive the apparatus image data; and a post-analysis inspector configure to determine the inspection outcome for the object of the identifiable apparatus. (Fig.7 and 4.1. Dataset description: “We collect the images (1400 pixel×1024 pixel) from TFDS and randomly select images to build the train set and test set. In the process of target region detection, the train set owns 2321 images. Each train image is labelled according to the format of the PASCAL VOC dataset [30]. In the process of fault determination, 2000 SB regions including 640 fault regions and 1500 SFK regions including 540 fault regions are cropped from the TFDS images for training.”)
Regarding claims 19, Sun, as modified by Mian, discloses all the claims invention. Sun further discloses the object inspector further receives identification information for the identifiable apparatus, (Fig. 7. The four typical fault conditions of SBs and SFKs are presented in Fig. 7, including bolt missing of SB (Fig. 7(c)), bolt looseness of SB (Fig. 7(d)), missing of SB (Fig. 7(e)), and missing of SFK (Fig. 7(f)).”) and wherein the object inspector obtains the set of reference equipment images using the identification information for the identifiable apparatus. (4.1. Dataset description: “In the process of fault determination, 2000 SB regions including 640 fault regions and 1500 SFK regions including 540 fault regions are cropped from the TFDS images for training.”
Regarding claims 20, Sun, as modified by Mian, discloses all the claims invention. Sun further discloses the object inspector further obtains representation data using the identification information for the identifiable apparatus, wherein the representation data includes information relating to the object for a type of the identifiable apparatus, and wherein the deep learning engine uses the representation data to identify the object image data. (Fig.7 and 4.1. Dataset description: “We collect the images (1400 pixel×1024 pixel) from TFDS and randomly select images to build the train set and test set. In the process of target region detection, the train set owns 2321 images. Each train image is labelled according to the format of the PASCAL VOC dataset [30]. In the process of fault determination, 2000 SB regions including 640 fault regions and 1500 SFK regions including 540 fault regions are cropped from the TFDS images for training.”)
Regarding claims 21, Sun, as modified by Mian, discloses the deep learning engine returns a confidence level associated with the object image data, (Sun: 3.2.2. “Accurate Localization: “ We consider the score as the confidence score for corresponding bounding-box and a suitable threshold for the confidence score is set to select out the most likely regions”) and wherein the object inspector requests human assistance (Mian: Figs. 2-3; Paragraph 46: “evaluation program 40 can manage a set of interfaces (e.g., graphical user interface(s), application program interface, and/or the like) that enable human and/or system users 11 to interact with evaluation program 40. Further, evaluation program 40 can manage (e.g., store, retrieve, create, manipulate, organize, present, etc.) the data, such as vehicle data 50, using any solution.”; Paragraph 58: “decision making component 42D can provide the appropriate action(s), if necessary, for processing by one or more user systems 11A-11C.”) in response to the confidence level being below a predetermined threshold. (Mian: Paragraphs 54-56: “Analysis component 42B can perform any combination of one or more image analysis operations on the image vehicle data 50 including, but not limited to, thresholding, edge detection, region definition and segmentation, and/or the like. … Advanced analysis component 42C can provide the results of the determination of the existence or non-existence of the set of conditions on rail vehicle 4 for processing by decision making component 42D. Decision making component 42D can determine what action(s) are to be performed in response to the set of conditions present on the rail vehicle.”)
Regarding claims 22, Sun, as modified by Mian, discloses the object inspector receives second object image data (Sun: Fig.3: Side frame key – bounding box 2 or End bolt (EB) – bounding box 3) in response to the human assistance request, (Mian: Figs. 2-3; Paragraph 46: “evaluation program 40 can manage a set of interfaces (e.g., graphical user interface(s), application program interface, and/or the like) that enable human and/or system users 11 to interact with evaluation program 40. Further, evaluation program 40 can manage (e.g., store, retrieve, create, manipulate, organize, present, etc.) the data, such as vehicle data 50, using any solution.”; Paragraph 58: “decision making component 42D can provide the appropriate action(s), if necessary, for processing by one or more user systems 11A-11C.”) wherein the second object image data is processed by the object inspector to determine the inspection outcome. Sun: Fig.7 and 4.1. Dataset description: “We collect the images (1400 pixel×1024 pixel) from TFDS and randomly select images to build the train set and test set. In the process of target region detection, the train set owns 2321 images. Each train image is labelled according to the format of the PASCAL VOC dataset [30]. In the process of fault determination, 2000 SB regions including 640 fault regions and 1500 SFK regions including 540 fault regions are cropped from the TFDS images for training.”)
Regarding claims 23, Sun, as modified by Mian discloses all the claims invention. Sun further discloses the object inspector stores the second object image data as a reference equipment image related to the identifiable apparatus. (Fig.7 and 4.1. Dataset description: “In the process of fault determination, 2000 SB regions including 640 fault regions and 1500 SFK regions including 540 fault regions are cropped from the TFDS images for training … The four typical fault conditions of SBs and SFKs are presented in Fig. 7, including bolt missing of SB (Fig. 7(c)), bolt looseness of SB (Fig. 7(d)), missing of SB (Fig. 7(e)), and missing of SFK (Fig. 7(f)).”)
Regarding claims 24, Sun, as modified by Mian discloses all the claims invention. Sun further discloses the object inspector provides the second object image data, data indicating the inspection outcome for the object, and the apparatus image data, for inclusion in a training database for the deep learning engine. (Fig.7 and 4.1. Dataset description: “We collect the images (1400 pixel×1024 pixel) from TFDS and randomly select images to build the train set and test set. In the process of target region detection, the train set owns 2321 images. Each train image is labelled according to the format of the PASCAL VOC dataset [30In the process of fault determination, 2000 SB regions including 640 fault regions and 1500 SFK regions including 540 fault regions are cropped from the TFDS images for training.”)
Regarding claims 25, Sun, as modified by Mian discloses all the claims invention. Mian further discloses further comprising an acquisition system, (Fig.3: acquisition subsystem 12, data acquisition component 42A) the acquisition system including: a plurality of sensing devices for acquiring data regarding the identifiable apparatus; (Paragraph 53: “data acquisition component 42A is shown receiving data from … one or more non-image data capture devices 25, such as an RFID device, an acoustic sensing system, wheel sensors, and/or the like.”) triggering logic configured to process the data regarding the identifiable apparatus to determine when to start and stop acquiring image data of the identifiable apparatus; and a set of cameras configured to acquire the image data of the identifiable apparatus (Paragraph 53: “data acquisition component 42A is shown receiving data from an infrared imaging device 22, a second imaging device 24 (e.g., a visible imaging device)”) in response to a signal received from the triggering logic. (Paragraph 53: “data acquisition component 42A can receive a signal from a vehicle sensing system of an approaching set of rail vehicles 4, and trigger the devices of acquisition subsystem 12 to initialize. Similarly, data acquisition component 42A can determine when no additional rail vehicles 4 are approaching and trigger the devices of acquisition subsystem 12 to shut down/sleep.”)
Regarding claims 26, Sun, as modified by Mian discloses all the claims invention. Mian further discloses the acquisition system further includes at least one illuminator configured for operation in conjunction with at least one of the set of cameras. (Figs. 1 and 3 ; Paragraph 41: “the second imaging device 24 comprises a visible light imaging device having a higher resolution than infrared imaging device 22.”)
Regarding claims 27, Sun, as modified by Mian discloses all the claims invention. Mian further discloses the at least one illuminator is collocated with at least one of the set of cameras. (Figs. 1 and 3: an infrared imaging device 22 and a second imaging device 24. ; Paragraph 41: “the second imaging device 24 comprises a visible light imaging device having a higher resolution than infrared imaging device 22.”)
Regarding claims 28, Sun, as modified by Mian discloses all the claims invention. Mian further discloses the inspection system further includes a data acquirer configured to receive data from the acquisition system and forwarding the apparatus image data for processing by the object inspector, (Fig.3 and Paragraph 54: “Data acquisition component 42A can provide the pre-processed vehicle data 50 on the rail vehicle 4 for processing by an analysis component 42B. Analysis component 42B can perform any combination of one or more image analysis operations on the image vehicle data 50 including, but not limited to, thresholding, edge detection, region definition and segmentation, and/or the like.”) and wherein the environment further comprises a high speed data connection between the set of cameras and the data acquisition component. (Paragraph 67: “data acquisition component 42A can process multiple near-identical infrared images, e.g., those captured by a high speed infrared imaging device 22, to combine the infrared image data to eliminate noise, frame artifacts, blurs from insects, dirt, or the like.”)
Regarding claims 29, Sun, as modified by Mian discloses all the claims invention. Mian further discloses the inspection system includes an image compression unit comprising at least one central processing unit and at least one graphics processing unit, (Fig.3: Analysis component 42B) wherein the object inspector provides apparatus image data for compression by the image compression unit (Paragraph 60: “FIG. 4 shows an illustrative set of examples of infrared image processing, which illustrate the processing that can be performed by analysis component 42B and/or advanced analysis component 42C of FIG. 3, according to an embodiment. Images 52A, 54A are vertically-compressed visible light images of two passing rail wheels 8 (FIG. 1B).”) and transmits the compressed apparatus image data for storage in a training database for the deep learning engine. (Paragraph 55: “Analysis component 42B, can provide the processed vehicle data 50 and/or raw vehicle data 50 for processing by advanced analysis component 42C. Advanced analysis component 42C can evaluate the results of the analysis performed by analysis component 42B using any solution. For example, advanced analysis component 42C can perform rule-based analysis (e.g., if region A temp>region B temp+X degrees, then . . . ), Bayesian or neural network processing,”)
Regarding claims 30, Sun, as modified by Mian discloses all the claims invention. Sun further discloses comprising a training system including: a deep learning training trainer configured to train the deep learning engine; (3.2. Target region detection: “ In the first stage of AFRS, we propose a coarse-to-fine scheme to detect the target regions. First, a CNN-based detection model is established and trained for coarse detection”) and a training database including image data for training the deep learning engine, wherein the inspection component transmits image data acquired during an inspection for storage in the training database, (Fig.7 and 4.1. Dataset description: “We collect the images (1400 pixel×1024 pixel) from TFDS and randomly select images to build the train set and test set. In the process of target region detection, the train set owns 2321 images. Each train image is labelled according to the format of the PASCAL VOC dataset [30].”) and wherein the deep learning training component periodically retrains a deep learning model using the training database and deploys an updated deep learning model for use in the inspection component. (3.3. Fault Determination: “As shown in Fig. 1, the process of AFRS is first detection and then determination of each fault, … we just take this CNN model for all fault determination by separate training, which means that we just respectively train the network with corresponding dataset aiming at a specific region.”)
Regarding claims 31, Sun discloses an environment for inspecting an object of a rail vehicle, (1.Introduction: “First, it automatically captures images of some vital parts of freight trains. Then, the images are transmitted to the data server in monitor room for analysis by indoor inspectors to find out the faults of relevant parts. By observing the TFDS images, the indoor inspectors will inform the outdoor inspectors to confirm and handle the problem if the fault exists”) the environment comprising:
an inspection system (1. Introduction: “The whole system is showed in Fig. 1. AFRS is a two-stage system”) including:
a deep learning engine configured to implement a deep learning model (The CNN-based detection model) in order to analyze rail vehicle image data (Fig.3: TFDS images) and identify object image data, (Fig.3: shaft bolt (SB), side frame key (SFK), and end bolt (EB).) wherein the object image data corresponds to a region of interest for the object of the rail vehicle in the rail vehicle image data; (3.2.1. Coarse Detection: “With a trained CNN-based detection network, the predicted bounding-boxes of the three critical parts are detected, including 10–20 bounding-boxes for each category per image.”) of the apparatus in the apparatus image data; (3.2: Target region Detection: “The CNN-based detection model detects the critical parts in TFDS images including shaft bolt (SB), side frame key (SFK), and end bolt (EB). The critical parts are showed in Fig. 3 … Accurate detection of EB is of great importance to ensure a higher accuracy.”) and
an inspection component for providing an inspection outcome for the object of the rail vehicle based on the object image data, (3.3. Fault determination: “in the second stage of AFRS, we establish another CNN model to determine multiple faults. The architecture is showed in Fig. 6 … we just take this CNN model for all fault determination by separate training, which means that we just respectively train the network with corresponding dataset aiming at a specific region.”)
wherein the inspection outcome for the object is determined by comparing the object image data with a set of reference equipment images (Each train image) related to the object. Fig.7 and 4.1. Dataset description: “We collect the images (1400 pixel×1024 pixel) from TFDS and randomly select images to build the train set and test set. Each train image is labelled according to the format of the PASCAL VOC dataset [30]. In the process of target region detection, the train set owns 2321 images. In the process of fault determination, 2000 SB regions including 640 fault regions and 1500 SFK regions including 540 fault regions are cropped from the TFDS images for training.”)
However, Sun does not disclose wherein the inspection outcome for the object is determined by directly comparing the object image data with image data corresponding to at least one of a set of reference equipment images related to the object.
Mian discloses an inspection component for providing an inspection outcome for the object of the rail vehicle based on the object image data, (Paragraph 59: “advanced analysis component 42C can determine whether any particular conditions or faults of interest are indicated by the image data … the image processing includes performing edge detection and segmentation upon an image (e.g., using thresholding processes), assembling/recognizing individual segments as part of one or more features (feature extraction), and assembling the features into "blobs" or objects, which can be compared against known likely objects using, for example, templating, expert system recognition, and/or the like.”) wherein the inspection outcome for the object is determined by directly comparing the object image data with image data corresponding to at least one of a set of reference equipment images related to the object. (Paragraph 64: “advanced analysis component 42C can analyze an individual image of a rail wheel 8 for anomalies. … After detection and definition of edges and features by analysis component 42B, advanced analysis component 42C can compare the identified features of the wheel 8 and related assemblies to a typical profile.”; Paragraph 75: “advanced analysis component 42C can determine which locations, if any, on the rail wheel may be hot, and compare these locations with the structure of the wheel and related components to determine the actual condition that may be present.”; Paragraph 84: “system 10 detects one or more defects by comparing the temperature of a component currently being analyzed with components of the same type that are adjacent (e.g., on the same vehicle) or have been recently analyzed (e.g., on a recently imaged vehicle). When a sufficient difference is noted between one component and other comparable components, system 10 can identify the component as including a defect.”, it shows that “ related component” and/or “components of the same type” and/or “other comparable components” is interpreted as “at least one of a set of reference equipment images related to the object”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Sun by including analysis subsystem that is taught by Mian, to make the invention that analysis of various components of a vehicle using thermal image data; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving improve accurately determine the presence or absence of the various flaws or faults as well as reducing false positives/negatives.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention
Regarding claims 32, Sun, as modified by Mian, discloses all the claims invention. Sun further discloses the inspection component comprises at least two sub-components, the at least two sub-components including: a pre-analysis inspection component for receiving the rail vehicle image data; and a post-analysis inspection component for determining the inspection outcome for the object of the rail vehicle. (Fig.7 and 4.1. Dataset description: “We collect the images (1400 pixel×1024 pixel) from TFDS and randomly select images to build the train set and test set. In the process of target region detection, the train set owns 2321 images. Each train image is labelled according to the format of the PASCAL VOC dataset [30]. In the process of fault determination, 2000 SB regions including 640 fault regions and 1500 SFK regions including 540 fault regions are cropped from the TFDS images for training.”)
Regarding claims 33, Sun, as modified by Mian, discloses the deep learning engine returns a confidence level associated with the object image data, (Sun: 3.2.2. “Accurate Localization: “ We consider the score as the confidence score for corresponding bounding-box and a suitable threshold for the confidence score is set to select out the most likely regions”) and wherein the inspection component requests human assistance (Mian: Figs. 2-3; Paragraph 46: “evaluation program 40 can manage a set of interfaces (e.g., graphical user interface(s), application program interface, and/or the like) that enable human and/or system users 11 to interact with evaluation program 40. Further, evaluation program 40 can manage (e.g., store, retrieve, create, manipulate, organize, present, etc.) the data, such as vehicle data 50, using any solution.”; Paragraph 58: “decision making component 42D can provide the appropriate action(s), if necessary, for processing by one or more user systems 11A-11C.”) in response to the confidence level being below a predetermined threshold. (Mian: Paragraphs 54-56: “Analysis component 42B can perform any combination of one or more image analysis operations on the image vehicle data 50 including, but not limited to, thresholding, edge detection, region definition and segmentation, and/or the like. … Advanced analysis component 42C can provide the results of the determination of the existence or non-existence of the set of conditions on rail vehicle 4 for processing by decision making component 42D. Decision making component 42D can determine what action(s) are to be performed in response to the set of conditions present on the rail vehicle.”)
Regarding claims 34, Sun, as modified by Mian discloses all the claims invention. Sun further discloses the inspection component provides the second object image data, (Fig.3: Side frame key – bounding box 2 or End bolt (EB) – bounding box 3) data indicating the inspection outcome for the object, and the rail vehicle image data, for inclusion in a training database for the deep learning engine. (Fig.7 and 4.1. Dataset description: “We collect the images (1400 pixel×1024 pixel) from TFDS and randomly select images to build the train set and test set. In the process of target region detection, the train set owns 2321 images. Each train image is labelled according to the format of the PASCAL VOC dataset [30In the process of fault determination, 2000 SB regions including 640 fault regions and 1500 SFK regions including 540 fault regions are cropped from the TFDS images for training.”)
Regarding claims 35, Sun discloses a method of inspecting an object of an identifiable apparatus, (1.Introduction: “First, it automatically captures images of some vital parts of freight trains. Then, the images are transmitted to the data server in monitor room for analysis by indoor inspectors to find out the faults of relevant parts. By observing the TFDS images, the indoor inspectors will inform the outdoor inspectors to confirm and handle the problem if the fault exists”) the method comprising:
analyzing, using a deep learning model (The CNN-based detection model) implemented on a deep learning engine, apparatus image data (Fig.3: TFDS images) to identify object image data, (Fig.3: shaft bolt (SB), side frame key (SFK), and end bolt (EB).) wherein the object image data corresponds to a region of interest for the object (3.2.1. Coarse Detection: “With a trained CNN-based detection network, the predicted bounding-boxes of the three critical parts are detected, including 10–20 bounding-boxes for each category per image.”) of the identifiable apparatus in the apparatus image data; (3.2: Target region Detection: “The CNN-based detection model detects the critical parts in TFDS images including shaft bolt (SB), side frame key (SFK), and end bolt (EB). The critical parts are showed in Fig. 3 … Accurate detection of EB is of great importance to ensure a higher accuracy.”)
obtaining a set of reference equipment images related to the object using identification information for the identifiable apparatus; (Fig.7 and 4.1. Dataset description: “We collect the images (1400 pixel×1024 pixel) from TFDS and randomly select images to build the train set and test set.”) and
determining an inspection outcome for the object by comparing the object image data with the set of reference equipment images (Each train image) related to the object, (Fig.7 and 4.1. Dataset description: “We collect the images (1400 pixel×1024 pixel) from TFDS and randomly select images to build the train set and test set. In the process of target region detection, the train set owns 2321 images. Each train image is labelled according to the format of the PASCAL VOC dataset [30]. In the process of fault determination, 2000 SB regions including 640 fault regions and 1500 SFK regions including 540 fault regions are cropped from the TFDS images for training.”)
wherein the inspection outcome is determined in response to the deep learning model indicating a confidence level associated with the object image data that is high a predetermined threshold. (3.2.2. Accurate Localization: “ We consider the score as the confidence score for corresponding bounding-box and a suitable threshold for the confidence score is set to select out the most likely regions”)
However, Sun does not disclose determining an inspection outcome for the object by directly comparing the object image data with image data corresponding to at least one of the set of reference equipment images related to the object, wherein the inspection outcome is determined with human assistance in response to the deep learning model indicating a confidence level associated with the object image data that is below a predetermined threshold.
Mian discloses determining an inspection outcome for the object by directly comparing the object image data with image data corresponding to at least one of the set of reference equipment images related to the object, (Paragraph 59: “advanced analysis component 42C can determine whether any particular conditions or faults of interest are indicated by the image data … the image processing includes performing edge detection and segmentation upon an image (e.g., using thresholding processes), assembling/recognizing individual segments as part of one or more features (feature extraction), and assembling the features into "blobs" or objects, which can be compared against known likely objects using, for example, templating, expert system recognition, and/or the like.”; Paragraph 64: “advanced analysis component 42C can analyze an individual image of a rail wheel 8 for anomalies. … After detection and definition of edges and features by analysis component 42B, advanced analysis component 42C can compare the identified features of the wheel 8 and related assemblies to a typical profile.”; Paragraph 75: “advanced analysis component 42C can determine which locations, if any, on the rail wheel may be hot, and compare these locations with the structure of the wheel and related components to determine the actual condition that may be present.”; Paragraph 84: “system 10 detects one or more defects by comparing the temperature of a component currently being analyzed with components of the same type that are adjacent (e.g., on the same vehicle) or have been recently analyzed (e.g., on a recently imaged vehicle). When a sufficient difference is noted between one component and other comparable components, system 10 can identify the component as including a defect.”, it shows that “ related component” and/or “components of the same type” and/or “other comparable components” is interpreted as “at least one of a set of reference equipment images related to the object” ) wherein the inspection outcome is determined with human assistance (Figs. 2-3; Paragraph 46: “evaluation program 40 can manage a set of interfaces (e.g., graphical user interface(s), application program interface, and/or the like) that enable human and/or system users 11 to interact with evaluation program 40. Further, evaluation program 40 can manage (e.g., store, retrieve, create, manipulate, organize, present, etc.) the data, such as vehicle data 50, using any solution.”; Paragraph 58: “decision making component 42D can provide the appropriate action(s), if necessary, for processing by one or more user systems 11A-11C.”) in response to the deep learning model indicating a confidence level associated with the object image data that is below a predetermined threshold. (Paragraphs 54-56: “Analysis component 42B can perform any combination of one or more image analysis operations on the image vehicle data 50 including, but not limited to, thresholding, edge detection, region definition and segmentation, and/or the like. … Advanced analysis component 42C can provide the results of the determination of the existence or non-existence of the set of conditions on rail vehicle 4 for processing by decision making component 42D. Decision making component 42D can determine what action(s) are to be performed in response to the set of conditions present on the rail vehicle.”)
Therefore, it would been obvious to one having ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Sun by including analysis subsystem that is taught by Mian, to make the invention that analysis of various components of a vehicle using thermal image data; thus, one of ordinary skilled in the art would have been motivated to combine the references since this will improving improve accurately determine the presence or absence of the various flaws or faults as well as reducing false positives/negatives.
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filling date of the claimed invention.
Regarding claims 36, Sun, as modified by Mian discloses all the claims invention. Sun further discloses the identifiable apparatus is a rail vehicle. (1. Introduction: First, it automatically captures images of some vital parts of freight trains. Then, the images are transmitted to the data server in monitor room for analysis by indoor inspectors to find out the faults of relevant parts.”)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Singh (U.S. 20170066459 A1), “Rail Track Asset Survey System”, teaches about e surveying of rail track assets, such as objects or equipment in the vicinity of a railroad. It also teaches about a railroad track asset surveying system comprising an image capture sensor, a location determining system, and an image processor. The image capture sensor is mounted to a railroad vehicle. The location determining system holds images captured by the image capture sensor. The image processor includes an asset classifier and an asset status analyser. The asset classifier detects an asset in one or more captured images and classifies the detected asset by assigning an asset type to the detected asset from a predetermined list of asset types according to one or more features in the captured image.
Distante et al (U.S. 20090196486 A1), “Automatic Method And System For Visual Inspection Of Railway Infrastructure”, teaches about a visual inspection system and method for the maintenance of infrastructures, in particular railway infrastructures. It is a system able to operate in real time, wholly automatically, for the automatic detection of the presence/absence of characterizing members of the infrastructure itself, for example the coupling locks fastening the rails to the sleepers.
Kobayashi (U.S. 20180089818 A1), “Image Inspection Device, Image Inspection Method and Image Inspection Program”, teaches about an image inspection device, an image inspection method, and an image inspection program capable of performing inspection processing at high speed by using learning type image recognition technology.
Ren et al (U.S. 20180268257 A1), “Surface Defect Detection”, teaches about neural network detection of surface defects on aircraft engine components. It also teaches about The method includes: providing (i) a pre-trained deep learning network and (ii) a learning machine network; providing a set of pixelated training images of aircraft engine components exhibiting examples of different classes of surface defect; training the trainable weights of the learning machine network on the set of training images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Duy A Tran whose telephone number is (571)272-4887. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ONEAL R MISTRY can be reached at (313)-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUY TRAN/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights