DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Applicant(s) Response to Official Action
[1] The response filed on October 16, 2025 has been entered and made of record.
Response to Arguments
[2] Presented arguments have been fully considered but are held unpersuasive. Examiner’s response to the presented arguments follows below.
Summary of Arguments:
Applicant argues that Bhate does not disclose:
a first neural network model which is configured to detect a first object class in the inspection image and generate first neural network model output data [Remarks: Page 10].
a second neural network model configured to perform a task on the inspection image different to detecting the first object class and generate second neural network model output data [Remarks: Page 11].
Examiner’s Response:
Examiner respectfully disagrees. Bhate does disclose:
a first neural network model which is configured to detect a first object class in the inspection image and generate first neural network model output data because the scenario classifier (114) detects the class (i.e. class) and probability (i.e. output data) See Figs. 3 and 5.
a second neural network model configured to perform a task on the inspection image different to detecting the first object class and generate second neural network model output data because the second neural network (118) identifies defects and damages which is different form the class identified by the first neural network model See col. 3, ll. 41-45.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
[3] Claims 85-87 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Bhate et al. (“Bhate”) [US 11,663,815].
Regarding claim 85, Bhate discloses the claim limitations as follows:
A computer-implemented method of automated artificial intelligence ("Al") visual inspection using a multi-model architecture (i.e. fig. 1), the method comprising: providing inspection image data as input (i.e. 110) to a first neural network model (i.e. 114) configured to detect a first object class in the inspection image data (i.e. Class and Probability) [figs. 3 and 5]; performing a first object detection task (i.e. determining Class and Probability) using the first neural network model (i.e. 114), the first object detection task including generating first neural network model output data (i.e. Class and Probability) [figs. 3 and 5]; storing the first neural network model output data (i.e. Class and Probability) in a memory as inspection image annotation data [figs. 3 and 5; col. 11, ll. 45-50]; determining whether the first neural network model output data (i.e. Class and Probability) satisfies a second model triggering condition stored in the memory (i.e. Class 1…Class n associated with largest probability value) [figs. 3 and 5; col. 11, ll. 45-50; col. 9, ll. 5-27]; if the first neural network model output data satisfies the second model triggering condition (i.e. Class 1…Class n associated with largest probability value): providing the inspection image data (i.e. 110) as input to a second neural network model (i.e. 118) [fig. 3; col. 3, ll. 41-45]; performing a second task, different to the first object detection task, (i.e. detecting defects and damages) using the second neural network model (i.e. 118), the second object detection task including generating second neural network output data (i.e. identification of defects and damages) [col. 3, ll. 41-45]; and storing the second neural network output data in the memory as a subset of the inspection image annotation data (i.e. identification of defects and damages) [col. 3, ll. 41-54; col. 11, ll. 45-50].
Regarding claim 86, Bhate discloses the claim limitations as follows:
The method of claim 85, further comprising generating an annotated inspection image using the inspection image data (i.e. 110) and the inspection image annotation data (i.e. bounding box around defects and damages in 110) and displaying the annotated inspection image in a user interface (i.e. 110 with bounding box around defects and damages are displayed) [col. 3, ll. 41-54].
Regarding claim 87, Bhate discloses the claim limitations as follows:
The method of claim 85, wherein at least one of the first neural network model and the second neural network model is an instance segmentation neural network model (i.e. location of the damage is segmented using a bounding box) [col. 3, ll. 40-50].
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
[4] Claims 68, 69, 71 and 73-84 are rejected under 35 U.S.C. 103 as being unpatentable over Bhate et al. (“Bhate”) [US 11,663,815].
Regarding claim 68, Bhate discloses the following claim limitations:
A system for automated artificial intelligence ("Al") visual inspection using a multi-model architecture [fig. 1], the system comprising: a camera device (i.e. 102) for acquiring inspection image data (i.e. input frame) of a target object being inspected [col. 3, ll. 10-15]; an Al visual inspection device (i.e. 106 and 108) [fig. 1] comprising: a memory (i.e. machine readable medium) storing a second model triggering condition (i.e. 116) for triggering use of a second neural network model (i.e. 118) [fig. 3; col. 11, ll. 45-50]; a processor (i.e. CPU and GPU) in communication with the memory [col. 3, ll. 30-40], the processor configured to: execute a first neural network model (i.e. 114) configured to detect a first object class in the inspection image (i.e. Class 1…Class n) and generate first neural network model output data including a first list of detected objects (i.e. probability of Class) [figs. 3 and 5]; execute a model triggering determination module (i.e. 116) configured to determine whether the first neural network model output data (i.e. 114 probability values) satisfies the second model triggering condition (i.e. highest probability model corresponding to models 118) [col. 9, ll. 5-27]; execute the second neural network model (i.e. 118) upon satisfaction of the second model triggering condition (i.e. highest probability model corresponding to models 118), the second neural network model configured to perform a task on the inspection image different to detecting the first object class and generate second neural network model output data (i.e. damage detector) [col. 9, ll. 30-45]; send via a communication interface (i.e. 120) neural network model output data to an operator device (i.e. fig. 1), the neural network model output data including(i.e. output of 118); the operator device configured to display the received neural network model output data (i.e. display on 104) [fig. 1, col. 3, ll. 40-58].
Bhate does not explicitly disclose the following claim limitations:
Displaying the first neural network model output data.
However, Bhate discloses the following claim limitations:
Graphic representation of the first neural network model output data (i.e. output of 114) [fig. 5]
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to modify the teachings of Bhate and display the first neural network model output data, the reasoning being to provide feedback to an operator for fine tuning the neural network [col. 5, ll. 24-30].
Regarding claim 69, Bhate discloses the claim limitations as follows:
The system of claim 68, wherein the first neural network model output data includes an object class label of a detected object (i.e. Class 1...Class n with probability), the second model triggering condition includes a required object class label (i.e. Class 1…Class n associated with largest probability value), and the processor determines whether the object class label of the detected object matches the required object class label (i.e. Select Class 2 because it is the class label for the class with highest probability) [fig. 5; col. 9, ll. 4-27].
Regarding claim 71, Bhate discloses the claim limitations as follows:
The system of claim 68, wherein the first neural network model output data includes a confidence level of a detected object (i.e. Class 1...Class n with probability), the second model triggering condition includes satisfying a minimum confidence level (i.e. Class 1…Class n associated with largest probability value), and the processor determines whether the confidence level of the detected object meets the minimum confidence level (i.e. Select Class 2 because it is the class label for the class with highest probability) [fig. 5; col. 9, ll. 4-27].
Regarding claim 73, Bhate discloses the claim limitations as follows:
The system of claim 68, wherein the first neural network model output data includes object attribute data describing at least two attributes of a detected object (i.e. Class 1...Class n with probability), and wherein the at least two attributes include any two or more of an object location, an object class label (i.e. Class 1...Class n), an object confidence level (i.e. probability), and an object size [fig. 5].
Regarding claim 74, Bhate discloses the claim limitations as follows:
The system of claim 73, wherein the second model triggering condition includes a requirement for each of the at least two attributes of the detected object (i.e. Class 1…Class n associated with largest probability value), and wherein the processor is further configured to determine whether the object attribute data satisfies the requirement for each of the at least two attributes of the detected object (i.e. Select Class 2 because it is the class label for the class with highest probability) [fig. 5; col. 9, ll. 4-27].
Regarding claim 75, Bhate discloses the claim limitations as follows:
The system of claim 68, wherein the first neural network output data includes an identifier (i.e. Class 1...Class n with probability) that identifies that the second model triggering condition is to be used by the processor (i.e. Class 1…Class n associated with largest probability value), wherein the identifier comprises model identification data identifying the first neural network model (i.e. Class 1..Class n with probability), and wherein the processor determines the second model triggering condition is to be used based on the identifier (i.e. Select Class 2 because it is the class label for the class with highest probability) [fig. 5; col. 9, ll. 4-27].
Regarding claim 76, Bhate discloses the claim limitations as follows:
The system of claim 75, wherein upon determining the second model triggering condition is to be used the processor retrieves the second model triggering condition from the memory using the identifier (i.e. Class 1…Class n associated with largest probability value) in order to determine whether the first neural network model output data satisfies the second model triggering condition (i.e. Select Class 2 because it is the class label for the class with highest probability) [fig. 5; col. 9, ll. 4-27].
Regarding claim 77, Bhate discloses the claim limitations as follows:
The system of claim 68, wherein the inspection image provided to the second neural network model comprises a subset (i.e. sub-datasets) of the inspection image (i.e. combined data set), the subset of the inspection image determined from the first neural network model output data (i.e. scenario ID/class), and wherein the second object detection task is performed using the subset of the inspection image (i.e. 118 uses sub-datasets) [col. 7, ll. 27-40].
Regarding claim 78, Bhate discloses the claim limitations as follows:
The system of claim 68, wherein the processor is further configured to generate a list of neural network models to be executed by the processor based on the first neural network model output data (i.e. 118(1)...118(n)), the list of neural network models to be executed including the second neural network model (i.e. Model 2 corresponding to Class 2) when the processor determines that the second model triggering condition is satisfied (i.e. Select Class 2 because it is the class label for the class with highest probability) [figs. 3 and 5; col. 9, ll. 4-27].
Regarding claim 79, Bhate discloses the claim limitations as follows:
The system of claim 78, wherein the processor executes each of the neural network models in the list in series (i.e. training all 118), the execution of a respective one of the neural network models including providing at least a subset (i.e. sub-datasets) of the inspection image (i.e. combined data set) to the respective one of the neural network models (i.e. 118(1)…118(n)) and generating neural network model output data using the respective one of the neural network models [fig. 2; col. 7, ll. 20-42].
Regarding claim 80, Bhate discloses the claim limitations as follows:
The system of claim 78, wherein the processor is further configured to dynamically update the list to include an additional neural network model to be executed (i.e. modify 118 according to scenarios, where scenarios define the number of models), the additional neural network model to be executed determined by the processor based on neural network output data generated by a previously executed neural network model satisfying a model triggering condition of the additional neural network model stored in the memory (i.e. select class with highest probability) [figs. 3 and 5; col. 7, ll. 27-50; col. 9, ll. 4-27].
Regarding claim 81, Bhate discloses the claim limitations as follows:
The system of claim 78, wherein the list of neural network models to be executed comprises a plurality of separate lists of neural network models to be executed (i.e. each of 118(1)…118(n) is a list with one model), each respective one of the plurality of separate lists of neural network models to be executed corresponding to a single neural network model (i.e. each of 118(1)…118(n) is a list with one model)[fig. 3].
Regarding claim 82, Bhate discloses the claim limitations as follows:
The system of claim 68, wherein the operator device is configured to generate a user interface for receiving input data (i.e. 110) setting the second model triggering condition (i.e. Class 1…Class n associated with largest probability value), and wherein the second model triggering condition is generated by either the operator device or the Al visual inspection device (i.e. 116) according to the input data (i.e. 110) [fig. 1; col. 9, ll. 5-27].
Regarding claim 83, Bhate discloses the claim limitations as follows:
The system of claim 68, wherein at least one of the first neural network model and the second neural network model is an image segmentation neural network model (i.e. location of the damage is segmented using a bounding box) [col. 3, ll. 40-50].
Regarding claim 84, Bhate discloses the claim limitations as follows:
The system of claim 83, wherein the image segmentation neural network model is an instance segmentation neural network model (i.e. location of the damage is segmented using a bounding box) [col. 3, ll. 40-50].
[5] Claim 70 is rejected under 35 U.S.C. 103 as being unpatentable over Bhate et al. (“Bhate”) [US 11,663,815] in view of Collberg [US 2021/0406604].
Regarding claim 70, Bhate discloses the claim limitations as set forth in claim 68.
Bhate does not explicitly disclose the following claim limitations:
The system of claim 68, wherein the first neural network model output data includes object location data of a detected object, the second model triggering condition includes an object location requirement, and the processor determines whether the object location data of the detected object meets the object location requirement.
However, in the same field of endeavor Collberg discloses the deficient claim limitations, as follows:
The system of claim 68, wherein the first neural network model output data (i.e. first object recognition) includes object location data of a detected object (i.e. location of the recognized object),
the second model triggering condition (i.e. second object recognition) includes an object location requirement (i.e. object recognitions are within predetermined image area), and the processor determines whether the object location data of the detected object meets the object location requirement (i.e. object recognitions are within predetermined image area) [paras. 0017; 0065-0066].
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to modify the teachings of Bhate with Collberg and use location based triggering, the reasoning being to reduce processing of the second model to a narrow subset of cases [para. 0005; 0057].
[6] Claim 72 is rejected under 35 U.S.C. 103 as being unpatentable over Bhate et al. (“Bhate”) [US 11,663,815] in view of Chen et al. [IDS cited NPL titled “A Hybrid Deep Learning Based Framework for Component Defect Detection of Moving Trains”].
Regarding claim 72, Bhate discloses the claim limitations as set forth in claim 68.
Bhate does not explicitly disclose the following claim limitations:
The system of claim 68, wherein the first neural network model output data includes object size data of a detected object, the second model triggering condition includes satisfying a minimum object size, and the processor determines whether the object size data meets the minimum object size.
However, in the same field of endeavor Chen discloses the deficient claim limitations, as follows:
The system of claim 68, wherein the first neural network model output data includes object size data of a detected object (i.e. clustering based on size), the second model triggering condition includes satisfying a minimum object size (i.e. large size s_1), and the processor determines whether the object size data meets the minimum object size [fig. 3; page 4, section B].
It would have been obvious to one with ordinary skill in the art before the effective filing date of the invention to modify the teachings of Bhate with Chen and use size based triggering, the reasoning being to boost detection performance[page 4, section B].
Conclusion
[7] Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
[8] Any inquiry concerning this communication or earlier communications from the examiner should be directed to SATH V PERUNGAVOOR whose telephone number is (571)272-7455. The examiner can normally be reached M-F, 8 am-5 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, COLLEEN FAUZ can be reached at (571) 272-1667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SATH V PERUNGAVOOR/Supervisory Patent Examiner, Art Unit 2488