DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because new grounds of rejection are made in view of Rajagopalan (US 20180117718A1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-6 and 8-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feng (US 20240134007A1) in view of Rajagopalan (US 20180117718A1), further in view of Hoshikawa (US 20220256750A1).
As per Claim 1, Feng teaches a computer-implemented method comprising: obtaining one or more RGB images and one or more thermal images of a part (inspection system having a visual image sensor and a thermal image sensor, align an RGB image (received from the image sensor) with a thermal image (received from the thermal image sensor), [0188], for example, the inspection system may be used to scan a structure (e.g., a roof) for the presence of moisture, [0045]); processing the one or more RGB images and the one or more thermal images to generate a fused image data set (combine RGB and thermal information, thermal and RGB image data fusion, [0262]); providing the fused image data set as input to a first machine learning (ML) model that is trained to detect one or more anomaly types of parts; determining, using the first ML model, one or more anomalies of the part (machine learning processor trained to detect thermal anomalies in the aligned RGB image and thermal image data, the thermal anomalies may include water penetration/infiltration, air leakage, improper insulation, and thermal bridging, and the machine learning processor may be trained to classify thermal anomalies as one of these types, [0189], [0045]). Feng teaches that the first ML model determines the presence of moisture in the roof (utilizes artificial intelligence to analyze the images to identify an anomaly of interest, the anomaly of interest may be, for example, moisture, for example, the inspection system may be used to scan a structure (e.g., a roof) for the presence of moisture, [0045]). Feng teaches that identifying moisture damage to the roof is essential for maintaining the structural integrity of buildings (roof moisture scans are essential for maintaining the structural integrity of buildings by identifying moisture damage to the roof, [0004]). Thus, it would have been obvious to one of ordinary skill in the art that it provides the determined one or more anomalies of the part for maintenance of the part, in order to maintain the structural integrity [0045, 0004]. Feng teaches detecting pipes [0055].
However, Feng does not expressly teach that the part is a pipe part connecting two pipes of a pipeline system. However, Rajagopalan teaches obtaining one or more color images or one or more thermal images of a pipe part connecting two pipes of a pipeline system (obtain a thermal image of the weld joint/region, [0534], images provided by the inspection camera may be color images, [0535], weld joint 1026 connecting the pipe segments 1022 (1022a and 1022b) of the pipeline 1024, the pipe segments 1022 (1022a and 1022b) may be referred to as pipes, [0215]); providing the image data set as input to a processor; determining, using the processor, one or more anomalies of the pipe part (one or more processors 5140 may be configured to analyze the image(s) captured by the inspection camera to detect any defects present in the weld joint, [0533]); and providing the determined one or more anomalies of the pipe part for maintenance of the pipe part (if the inspection detector detects positional misalignment/pipe alignment error, instructions are sent to the workers for placement and thickness of the shims needed to correct positional misalignment/pipe alignment error, the workers, remove the clamp, place the shims, and replace the clamp, [0845]). Thus, this teaching of the pipe part from Rajagopalan can be implemented into the part of Feng so that the part is a pipe part connecting two pipes of a pipeline system; and thus it obtains one or more RGB images and one or more thermal images of a pipe part connecting two pipes of a pipeline system; provides the fused image data set as input to a first machine learning (ML) model that is trained to detect one or more anomaly types of pipe parts; determining, using the first ML model, one or more anomalies of the pipe part; and providing the determined one or more anomalies of the pipe part for maintenance of the pipe part.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Feng so that the part is a pipe part connecting two pipes of a pipeline system because Rajagopalan suggests that it is well-known in the art to inspect the pipe part [0004].
However, Feng and Rajagopalan do not teach that the pipe part is a pipe flange. However, Hoshikawa teaches obtaining one or more images of a flange; determining one or more anomalies of the flange (acquire an image by imaging a flange portion, determining whether the flange portion is defective based on the image, [0008]). Thus, this teaching from Hoshikawa can be implemented into the pipe part of the combination of Feng and Rajagopalan so that the pipe part is a pipe flange; and thus it obtains one or more RGB images and one or more thermal images of a pipe flange connecting two pipes of a pipeline system; provides the fused image data set as input to a first machine learning (ML) model that is trained to detect one or more anomaly types of pipe flanges; determines, using the first ML model, one or more anomalies of the pipe flange; and provides the determined one or more anomalies of the pipe flange for maintenance of the pipe flange.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Feng and Rajagopalan so that the pipe part is a pipe flange because Hoshikawa suggests that it is well-known in the art to inspect a flange [0002-0003].
As per Claim 2, Feng teaches wherein processing the one or more RGB images and the one or more thermal images to generate the fused image data set comprises aligning one of the one or more RGB images with one of the one or more thermal images to a common coordinate system (thermal and RGB images could be aligned, [0290], these images are fairly well aligned, create custom datasets from captured thermal and RGB imagery, [0291]).
9. As per Claim 3, Feng teaches wherein determining the one or more anomalies of the part comprises determining one or more backbone models (image classification) in the first ML model based on the one or more anomaly types of the part (machine learning processor trained to detect thermal anomalies in the aligned RGB image and thermal image, the thermal anomalies may include water penetration/infiltration, air leakage, improper insulation, and thermal bridging, and the machine learning processor may be trained to classify thermal anomalies as one of the types, [0189], [0045]).
However, Feng does not teach that the part is a pipe flange. However, the combination of Rajagopalan and Hoshikawa teaches the part is a pipe flange, as discussed in the rejection for Claim 1. Thus, this teaching from Rajagopalan and Hoshikawa can be implemented into the part of Feng so that the part is a pipe flange, and thus determining the one or more anomalies of the pipe flange comprises determining one or more backbone models in the first ML model based on the one or more anomaly types of the pipe flange. This would be obvious for the reasons given in the rejection for Claim 1.
10. As per Claim 4, Feng teaches wherein the one or more backbone models comprise at least one backbone model for object detection, image classification, instance segmentation, or regression [0189].
11. As per Claim 5, Feng does not teach wherein the one or more anomaly types of the pipe flange comprise at least one of missing parts in the pipe flange, misaligned faces of the pipe flange, or external corrosion of the pipe flange. However, Rajagopalan teaches wherein the one or more anomaly types of the pipe part comprise misaligned faces of the pipe part (if the inspection detector detects positional misalignment/pipe alignment error, instructions are sent to the workers for placement and thickness of the shims needed to correct positional misalignment/pipe alignment error, the workers, remove the clamp, place the shims, and replace the clamp, [0845]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Feng so that the one or more anomaly types of the pipe part comprise misaligned faces of the pipe part because Rajagopalan suggests that this is needed in order for the pipes to function properly [0845].
However, Feng and Rajagopalan do not teach that the pipe part is a pipe flange. However, the combination of Rajagopalan and Hosikawa teaches that the pipe part is a pipe flange, as discussed in the rejection for Claim 1. Thus, the combination of Rajagoapalan and Hosikawa teaches wherein the one or more anomaly types of the pipe flange comprise at least one of missing parts in the pipe flange, misaligned faces of the pipe flange, or external corrosion of the pipe flange. This would be obvious for the reasons given in the rejection for Claim 1.
12. As per Claim 6, Feng teaches the machine learning processor is trained to detect thermal anomalies in the aligned RGB image and thermal image [0189]. Feng teaches increasing the diversity of data in a dataset of training machine learning models [0079]. Thus, it would have been obvious to one of ordinary skill in the art that before determining the one or more anomalies of the part using the first ML model, training the first ML model using a plurality of RGB images and a plurality of thermal images of a plurality of parts, which is the dataset of training the first ML model so that it is trained to detect thermal anomalies in the aligned RGB image and thermal image [0189, 0079, 0045].
However, Feng does not teach that the part is a pipe flange. However, the combination of Rajagopalan and Hoshikawa teaches the part is a pipe flange, as discussed in the rejection for Claim 1. Thus, this teaching from Rajagopalan and Hoshikawa can be implemented into the part of Feng so that the part is a pipe flange, and thus before determining the one or more anomalies of the pipe flange using the first ML model, training the first ML model using a plurality of RGB images and a plurality of thermal images of a plurality of pipe flanges. This would be obvious for the reasons given in the rejection for Claim 1.
13. As per Claim 8, Feng teaches wherein an output of the first ML model comprises at least one of one or more classification labels for the one or more anomaly types of the part [0189, 0045]. Thus, teaches wherein an output of the first ML model comprises at least one of one or more classification labels for the one or more anomaly types of the part, one or more regression values for estimating a flange size, a bolt length, or a flange misalignment inclination, one or more object detection bounding boxes, or one or more segmentation masks for areas with anomalies [0189, 0045].
However, Feng does not teach that the part is the pipe flange. However, the combination of Rajagopalan and Hoshikawa teaches the part is a pipe flange, as discussed in the rejection for Claim 1. Thus, this teaching from Rajagopalan and Hoshikawa can be implemented into the part of Feng so that the part is a pipe flange, and thus an output of the first ML model comprises at least one of one or more classification labels for the one or more anomaly types of the pipe flange. This would be obvious for the reasons given in the rejection for Claim 1.
14. As per Claim 9, the combination of Feng, Rajagopalan, and Hoshikawa teaches the maintenance of the pipe flange, as discussed in the rejection for Claim 1.
However, Feng does not teach wherein the maintenance of the pipe flange comprises at least one of replacing the pipe flange, installing missing parts of the pipe flange, adjusting an alignment between faces of the pipe flange, or repairing the pipe flange to prevent leakage from the pipe flange. However, Rajagopalan teaches wherein the maintenance of the pipe part comprises at least one of replacing the pipe part, adjusting an alignment between faces of the pipe part (if the inspection detector detects positional misalignment/pipe alignment error, instructions are sent to the workers for placement and thickness of the shims needed to correct positional misalignment/pipe alignment error, the workers, remove the clamp, place the shims, and replace the clamp, [0845]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Feng so that the maintenance of the pipe part comprises at least one of replacing the pipe part, adjusting an alignment between faces of the pipe part because Rajagopalan suggests that this is needed in order for the pipes to function properly [0845].
However, Feng and Rajagopalan do not teach that the pipe part is a pipe flange. However, the combination of Rajagopalan and Hoshikawa teaches that the pipe part is a pipe flange, as discussed in the rejection for Claim 1. Hoshikawa teaches that if it is determined that the flange portion 66 of suction nozzle 60 is abnormal, the suction nozzle 60 is discarded and is thus prevented from being used in electronic component mounting device [0079]. Thus, it would have been obvious to one of ordinary skill in the art that this means that the defective flange is prevented from being used in electronic component mounting device and is replaced with a non-defective flange in electronic component mounting device. Thus, Hoshikawa teaches wherein the maintenance of the flange comprises at least one of replacing the flange, installing missing parts of the flange, adjusting an alignment between faces of the flange, or repairing the flange to prevent leakage from the flange [0079]. Thus, the combination of Rajagopalan and Hoshikawa teaches wherein the maintenance of the pipe flange comprises at least one of replacing the pipe flange, installing missing parts of the pipe flange, adjusting an alignment between faces of the pipe flange, or repairing the pipe flange to prevent leakage from the pipe flange.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Feng and Rajagopalan so that the maintenance of the pipe flange comprises at least one of replacing the pipe flange, installing missing parts of the pipe flange, adjusting an alignment between faces of the pipe flange, or repairing the pipe flange to prevent leakage from the pipe flange because Hosikawa suggests that this prevents the defective flange from being used in electronic component mounting device, and is replaced with a non-defective flange in the electronic component mounting device so that the electronic component mounting device operates correctly [0079].
15. As per Claim 10, Claim 10 is similar in scope to Claim 1, except that Claim 10 is directed to a non-transitory computer-readable medium storing one or more instructions executable by a computer system to perform operations comprising the method of Claim 1. Feng teaches the processor is programmed to perform the method [0005]. Thus, it would have been obvious to one of ordinary skill in the art that there is a non-transitory computer-readable medium storing one or more instructions (this program) executable by a computer system to perform operations comprising the method [0005]. Thus, Claim 10 is rejected under the same rationale as Claim 1.
16. As per Claims 11-15, these claims are similar in scope to Claims 2-5 and 8 respectively, and therefore are rejected under the same rationale.
17. As per Claims 16-20, these claims are similar in scope to Claims 10 and 12-15 respectively, and therefore are rejected under the same rationale.
18. Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Feng (US 20240134007A1), Rajagopalan (US 20180117718A1), and Hoshikawa (US 20220256750A1) in view of Kim (US 20240265503A1).
Feng, Rajagopalan, and Hoshikawa are relied upon for the teachings as discussed above relative to Claim 6. The combination of Feng, Rajagopalan, and Hoshikawa teach wherein training the first ML model using the plurality of RGB images and the plurality of thermal images of the plurality of pipe flanges, as discussed in the rejection for Claim 6.
However, Feng, Rajagopalan, and Hoshikawa do not teach extracting, using a second ML model, one or more features from the plurality of RGB images and the plurality of thermal images; generating a fused training image data set using the extracted one or more features; and training the first ML model based on the fused training image data set. However, Kim teaches extracting, using a second ML model, one or more features from the plurality of RGB images and the plurality of thermal images; generating a fused training image data set using the extracted one or more features (image fusion artificial neural network model can use each RGB visible light image and thermal image as input data, and can output a feature map, [0190]); and training the first ML model based on the fused training image data set (in addition, other artificial neural network models that take fused images as inputs may be models trained to perform inference such as anomaly detection, [0160]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Feng, Rajagopalan, and Hoshikawa to include extracting, using a second ML model, one or more features from the plurality of RGB images and the plurality of thermal images; generating a fused training image data set using the extracted one or more features; and training the first ML model based on the fused training image data set because Kim suggests that this minimizes the amount of computation [0008, 0190, 0160].
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONI HSU whose telephone number is (571)272-7785. The examiner can normally be reached M-F 10am-6:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JH
/JONI HSU/Primary Examiner, Art Unit 2611