DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Information Disclosure Statement
The information disclosure statement filed February 28, 2025 fails to comply with the provisions of 37 CFR 1.97, 1.98 and MPEP § 609 because the reference Schmidt et al. US2003-0343066, publication date: 10/26/2023 does not exist. The reference has not been considered because it was not found. It has been placed in the application file, but the information referred to therein has not been considered as to the merits. Applicant is advised that the date of any re-submission of any item of information contained in this information disclosure statement or the submission of any missing element(s) will be the date of submission for purposes of determining compliance with the requirements based on the time of filing the statement, including all certification requirements for statements under 37 CFR 1.97(e). See MPEP § 609.05(a).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 5, 9-10, 15-16 and 19-21 are rejected under 35 U.S.C. 103 as being unpatentable over Park et al. Thermal anomaly detection in walls via CNN-based segmentation as provided in the applicant’s information disclosure statement and further in view of Ahmed et al. US2020/0394430 hereinafter referred to as Ahmed.
As per Claim 1, Park teaches a computer-implemented method, comprising: obtaining, by a computing device comprising one or more processors,
a visible light image which depicts a structure and a thermal image which depicts the structure; (Park, p2, “Proposed method for thermal anomaly area detection”, “Fig. 1 is the schematic diagram of the proposed framework. Thermal and visible images of building envelopes are acquired simultaneously with an infrared camera”)
implementing, by the computing device, one or more first machine-learned models to identify a class associated with the structure in the visible light image, based on the visible light image; (Park, p2, “Proposed method for thermal anomaly area detection”, “A deep-learning model is trained as a preliminary step for image segmentation. A mask of wall domain is obtained from a visible image as a result of segmentation”)
implementing, by the computing device, identify one or more objects associated with the structure in the visible light image, based on the visible light image and the class associated with the structure; (Park, p2, “Proposed method for thermal anomaly area detection”, “A deep-learning model is trained as a preliminary step for image segmentation. A mask of wall domain is obtained from a visible image as a result of segmentation”, wall is considered class and object)
determining, by the computing device, a temperature associated with each object among the one or more objects, based on the thermal image; (Park, p2, “Proposed method for thermal anomaly area detection”, “A deep-learning model is trained as a preliminary step for image segmentation. A mask of wall domain is obtained from a visible image as a result of segmentation. After acquiring the mask, the temperature of the wall domain is extracted from the thermal image, and the thermal anomaly areas are detected for each wall domain.”)
evaluating, by the computing device, for each object among the one or more objects, whether a temperature value associated with a respective object among the one or more objects satisfies a temperature criteria associated with the respective object; (Park, p3, “Proposed method for thermal anomaly area detection”, “Then, thermal anomalies are detected by applying a temperature threshold for each domain”) and
providing, by the computing device, an output based on the evaluating. (Park, p3, Figure 1, “Anomaly area”)
Park does not explicitly teach implementing, by the computing device, one or more second machine-learned models to identify one or more objects associated with the structure in the visible light image, based on the visible light image and the class associated with the structure;
Ahmed teaches implementing, by the computing device, one or more second machine-learned models to identify one or more objects associated with the structure in the visible light image, based on the visible light image and the class associated with the structure; (Ahmed, Abstract, “using a first trained neural network, and classifying, by the processor, shapes of objects located within the identified region using a second trained neural network to determine whether any of the shapes correspond to a shape of a prohibited object known by the second neural network”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Ahmed into Park because by utilizing a neural network to identify objects based on shapes along with the neural network to identify objects utilizing thermal images will increase the accuracy of identification of objects by using both neural networks.
Therefore it would have been obvious to one of ordinary skill to combine the two references to obtain the invention in Claim 1.
As per Claim 3, Park in view of Ahmed teaches the computer-implemented method of claim 1, wherein the one or more second machine-learned models identify one or more objects in the visible light image, based on the visible light image, the thermal image, and the class associated with the structure. (Park, p2, “Proposed method for thermal anomaly area detection” and Ahmed, Abstract)
The rationale applied to the rejection of claim 1 has been incorporated herein.
As per Claim 5, Park in view of Ahmed teaches the computer-implemented method of claim 1, wherein the temperature criteria is a threshold temperature value associated with the respective object. (Park, p3, “Proposed method for thermal anomaly area detection”, “Then, thermal anomalies are detected by applying a temperature threshold for each domain”)
The rationale applied to the rejection of claim 1 has been incorporated herein.
As per Claim 9, Park in view of Ahmed teaches the computer-implemented method of claim 1, wherein implementing, by the computing device, the one or more second machine-learned models to identify the one or more objects associated with the structure in the visible light image, based on the visible light image and the class associated with the structure, comprises: providing at least one of a bounding shape or an indication of a key point to indicate a location of the respective object in at least one of the visible light image or the thermal image. (Park, p3, Figure 1, displays anomaly area vs normal wall)
The rationale applied to the rejection of claim 1 has been incorporated herein.
As per Claim 10, Park in view of Ahmed teaches the computer-implemented method of claim 9, further comprising: receiving an input changing a position of the bounding shape or the indication of the key point to indicate a changed location of the respective object in the at least one of the visible light image or the thermal image; and updating the one or more second machine-learned models based on the input. (Park, p2, “Proposed method for thermal anomaly area detection” and Ahmed, Abstract)
The rationale applied to the rejection of claim 1 has been incorporated herein.
As per Claim 15, Park in view of Ahmed teaches the computer-implemented method of claim 1, wherein the visible light image and the thermal image are captured at substantially a same time and from substantially a same perspective view. (Park, p2, “Proposed method for thermal anomaly area detection”, “Fig. 1 is the schematic diagram of the proposed framework. Thermal and visible images of building envelopes are acquired simultaneously with an infrared camera”)
The rationale applied to the rejection of claim 1 has been incorporated herein.
As per Claim 16, Park in view of Ahmed teaches the computer-implemented method of claim 1, further comprising performing an image registration operation by mapping a first set of coordinates associated with the one or more objects in the visible light image to a second set of coordinates associated with the one or more objects in the thermal image, and determining, by the computing device, the temperature associated with each object among the one or more objects, based on the thermal image, comprises determining a temperature associated with the second set of coordinates associated with the one or more objects in the thermal image. (Park, p2, “Proposed method for thermal anomaly area detection” and Ahmed, Abstract. Location of objects indicated by pixels are expressed in coordinates value)
The rationale applied to the rejection of claim 1 has been incorporated herein.
As per Claim 19, Park in view of Ahmed teaches the computer-implemented method of claim 1, further comprising: determining a confidence value associated with each of the one or more objects, and in response to a confidence value of at least one object being less than a threshold value, prompting a user to adjust at least one parameter associated with at least one of the visible light image or the thermal image. (Park, p2, “Proposed method for thermal anomaly area detection” and Ahmed, Abstract)
The rationale applied to the rejection of claim 1 has been incorporated herein.
As per Claim 20, Claim 20 claims a computing device performing the computer-implemented method as claimed in Claim 1. Therefore the rejection and rationale are analogous to that made in Claim 1.
As per Claim 21, Claim 21 claims a non-transitory computer readable medium storing instructions which, when executed by a processor, (Ahmed, Figure 1, Components 141 and 142) cause the processor to perform operations performing the computer-implemented method as claimed in Claim 1. Therefore the rejection and rationale are analogous to that made in Claim 1.
Claims 2 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Park et al. Thermal anomaly detection in walls via CNN-based segmentation as provided in the applicant’s information disclosure statement and further in view of Ahmed et al. US2020/0394430 hereinafter referred to as Ahmed as applied to Claim 1 and further in view of Kursun US2021/0173910.
As per Claim 2, Park in view of Ahmed teaches the computer-implemented method of claim 1,
wherein the one or more first machine-learned models identify the class (Park, p2, “Proposed method for thermal anomaly area detection”, “A deep-learning model is trained as a preliminary step for image segmentation. A mask of wall domain is obtained from a visible image as a result of segmentation. After acquiring the mask, the temperature of the wall domain is extracted from the thermal image, and the thermal anomaly areas are detected for each wall domain.”)
Park in view of Ahmed does not explicitly teach associated with the structure in the visible light image, based on the visible light image and the thermal image.
Kursun teaches associated with the structure in the visible light image, based on the visible light image and the thermal image. (Kursun, Paragraph [0004], “combining the received image data and thermal data to produce thermal image data; normalizing the thermal image data based on environmental, device, and channel characteristics; enriching the thermal image data with multimodal biometric data to produce an enriched biometric thermal image; processing the enriched biometric thermal image using a trained neural network model, wherein the processing comprises identification of isothermal curves and normalized thermal gradient coloring on the enriched biometric thermal image; performing an authentication check on the enriched biometric thermal image using the trained neural network model”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Kursun into Park in view of Ahmed because by combining the thermal image with the RGB image will increase the data contained in the combined image to be used for further classification.
Therefore it would have been obvious to one of ordinary skill to combine the three references to obtain the invention in Claim 2.
As per Claim 8, Park in view of Ahmed teaches the computer-implemented method of claim 1, further comprising:
wherein implementing, by the computing device, the one or more first machine-learned models to identify the class associated with the structure in the visible light image, is based on the visible light image forming part of the blended image, implementing, by the computing device, the one or more second machine-learned models to identify the one or more objects associated with the structure in the visible light image, is based on the visible light image forming part of the blended image and the class associated with the structure, and determining, by the computing device, the temperature associated with each object among the one or more objects, is based on the thermal image forming part of the
Park in view of Ahmed does not explicitly teach blending, by the computing device, the visible light image and the thermal image to generate a blended image, (Kursun, Paragraph [0004], “combining the received image data and thermal data to produce thermal image data; normalizing the thermal image data based on environmental, device, and channel characteristics; enriching the thermal image data with multimodal biometric data to produce an enriched biometric thermal image; processing the enriched biometric thermal image using a trained neural network model, wherein the processing comprises identification of isothermal curves and normalized thermal gradient coloring on the enriched biometric thermal image; performing an authentication check on the enriched biometric thermal image using the trained neural network model”)
Kursun teaches blending, by the computing device, the visible light image and the thermal image to generate a blended image, (Kursun, Paragraph [0004], “combining the received image data and thermal data to produce thermal image data; normalizing the thermal image data based on environmental, device, and channel characteristics; enriching the thermal image data with multimodal biometric data to produce an enriched biometric thermal image; processing the enriched biometric thermal image using a trained neural network model, wherein the processing comprises identification of isothermal curves and normalized thermal gradient coloring on the enriched biometric thermal image; performing an authentication check on the enriched biometric thermal image using the trained neural network model”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Kursun into Park in view of Ahmed because by combining the thermal image with the RGB image will increase the data contained in the combined image to be used for further classification.
Therefore it would have been obvious to one of ordinary skill to combine the three references to obtain the invention in Claim 8.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Park et al. Thermal anomaly detection in walls via CNN-based segmentation as provided in the applicant’s information disclosure statement and further in view of Ahmed et al. US2020/0394430 hereinafter referred to as Ahmed as applied to Claim 1 and further in view of Desai et al. US2022/0366167 hereinafter referred to as Desai.
As per Claim 4, Park in view of Ahmed teaches the computer-implemented method of claim 1,
Park in view of Ahmed does not explicitly teach wherein the one or more first machine-learned models include an image classification model and the one or more second machine-learned models include an object detection model, and the method further includes selecting the object detection model from among a plurality of object detection models based on the class identified by the image classification model.
Desai teaches wherein the one or more first machine-learned models include an image classification model and the one or more second machine-learned models include an object detection model, and the method further includes selecting the object detection model from among a plurality of object detection models based on the class identified by the image classification model. (Desai, Paragraph [0046], “The sub-classification determination engine 230 determines a sub-classification for each aircraft identified by the machine learning model 210 in the aerial image. In some embodiments, the sub-classification determination engine 230 utilizes a heuristics-approach to determine a sub-classification for each aircraft. In some embodiments, the sub-classification may correspond to a model of an aircraft. In other embodiments, the sub-classification may correspond to a make and model for an aircraft”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Desai into Park in view of Ahmed because by utilizing a sub-classification model will allow the object detection algorithm of Park and Ahmed to identify the object with more details and accuracy.
Therefore it would have been obvious to one of ordinary skill to combine the three references to obtain the invention in Claim 4.
Claims 6 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Park et al. Thermal anomaly detection in walls via CNN-based segmentation as provided in the applicant’s information disclosure statement and further in view of Ahmed et al. US2020/0394430 hereinafter referred to as Ahmed as applied to Claim 1 and further in view of Kuybeda et al. US2022/011165 hereinafter referred to as Kuybeda.
As per Claim 6, Park in view of Ahmed teaches the computer-implemented method of claim 1,
Park in view of Ahmed does not explicitly teach wherein the temperature value corresponds to a difference between a temperature of the respective object and an ambient temperature value, and the temperature criteria is a threshold temperature difference value associated with the respective object.
Kuybeda teaches wherein the temperature value corresponds to a difference between a temperature of the respective object and an ambient temperature value, and the temperature criteria is a threshold temperature difference value associated with the respective object. (Kuybeda, Paragraph [0030], use of ambient temperature and the object temperature to identify object)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Kuybeda into Park in view of Ahmed because by determining the ambient temperature along with the object temperature will reduce false-positives in object identification based on temperature.
Therefore it would have been obvious to one of ordinary skill to combine the three references to obtain the invention in Claim 6.
As per Claim 7, Park in view of Ahmed and Kuybeda teaches the computer-implemented method of claim 6, wherein the ambient temperature value corresponds to one of a lowest maximum temperature among maximum temperatures of the one or more objects, a lowest temperature in the thermal image, or a value received via an input from a user providing the ambient temperature value. (Kuybeda, Paragraph [0030])
The rationale applied to the rejection of claim 6 has been incorporated herein.
Claims 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Park et al. Thermal anomaly detection in walls via CNN-based segmentation as provided in the applicant’s information disclosure statement and further in view of Ahmed et al. US2020/0394430 hereinafter referred to as Ahmed as applied to Claim 1 and further in view of Yau et al. US2017/0372468 hereinafter referred to as Yau.
As per Claim 17, Park in view of Ahmed teaches the computer-implemented method of claim 1,
Park in view of Ahmed does not explicitly teach wherein in response to the evaluating indicating a temperature value associated with at least one object does not satisfy the temperature criteria, providing, by the computing device, the output based on the evaluating comprises generating a notification indicating the at least one object requires servicing.
Yau teaches wherein in response to the evaluating indicating a temperature value associated with at least one object does not satisfy the temperature criteria, providing, by the computing device, the output based on the evaluating comprises generating a notification indicating the at least one object requires servicing. (Yau, Paragraph [0033], “Then, by using different colored contours for different temperatures, the computer server maps the thermal images configured or directly obtained from different parts of the actual object to the three-dimensional model of the monitored object, thereby obtaining the three-dimensional thermal imaging of the monitored object (step S4). Finally, according to the obtained three-dimensional thermal imaging, predictive maintenance or the like is performed by further applying an expert system for fault identification and remedy measures for maintenance (step S5). For example, the computer server compares the temperatures of different parts of the monitored machine with the predefined threshold. If any unexpected hot spots or cold spots exist in the three-dimensional thermal image, an alert will be sent to the operator for notification”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Yau into Park in view of Ahmed because by providing an application to the determination of the temperature of the object and the object itself will allow users to perform tasks such as maintenance and fault identification.
Therefore it would have been obvious to one of ordinary skill to combine the three references to obtain the invention in Claim 17.
As per Claim 18, Park in view of Ahmed teaches the computer-implemented method of claim 1,
Park in view of Ahmed does not explicitly teach wherein in response to the evaluating indicating a temperature value associated with at least one object does not satisfy the temperature criteria, providing, by the computing device, the output based on the evaluating comprises automatically implementing a remedial or preventive action with respect to the at least one object.
Yau teaches wherein in response to the evaluating indicating a temperature value associated with at least one object does not satisfy the temperature criteria, providing, by the computing device, the output based on the evaluating comprises automatically implementing a remedial or preventive action with respect to the at least one object. (Yau, Paragraph [0033], “Then, by using different colored contours for different temperatures, the computer server maps the thermal images configured or directly obtained from different parts of the actual object to the three-dimensional model of the monitored object, thereby obtaining the three-dimensional thermal imaging of the monitored object (step S4). Finally, according to the obtained three-dimensional thermal imaging, predictive maintenance or the like is performed by further applying an expert system for fault identification and remedy measures for maintenance (step S5). For example, the computer server compares the temperatures of different parts of the monitored machine with the predefined threshold. If any unexpected hot spots or cold spots exist in the three-dimensional thermal image, an alert will be sent to the operator for notification”)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement the teachings of Yau into Park in view of Ahmed because by providing an application to the determination of the temperature of the object and the object itself will allow users to perform tasks such as maintenance and fault identification.
Therefore it would have been obvious to one of ordinary skill to combine the three references to obtain the invention in Claim 18.
Allowable Subject Matter
Claims 11-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MING HON whose telephone number is (571)270-5245. The examiner can normally be reached M-F 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached on 571-270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MING Y HON/Primary Examiner, Art Unit 2666
