DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-14 are rejected under 35 U.S.C. 103 as being unpatentable over TAIRA (US 20240362888 A1, claims the priority of PCT/JP2022/045978, Dec. 14 2022, and JP 2022-004404, Jan. 14 2022, and Patent Family WO 2023136030 A1, Date Published: 2023-07-20), and in view of SUDA (JP 2022098590 A).
Re Claim 1, TAIRA discloses an article inspection apparatus that inspects a quality state of an inspection object article by applying a predetermined image processing algorithm to an inspection image obtained by imaging a predetermined article type of the inspection object article (see TAIRA: e.g., Fig. 1, -- an inspector visually determines discontinuities or defects of a subject, and in a case where defects are found, the subject is classified as a defective product.--, in [0003], -- an information processing apparatus comprising: a processor, in which the processor is configured to: acquire first discontinuity information obtained by analyzing an image of a subject with a first criterion, the first discontinuity information including information indicating a feature of a discontinuity; acquire second discontinuity information obtained by analyzing the image of the subject with a second criterion that is stricter than the first criterion, the second discontinuity information including information indicating the feature of the discontinuity; and record the second discontinuity information in association with the first discontinuity information.--, in [0006]-[0010], and, -- the processor can calculate a condition for at least one of a specific period, a manufacturing number, a lot number, an inspection time, an inspection number, or an image capturing time as the recording condition and can record the discontinuity information under the calculated recording condition.
[0014] According to a sixth aspect, in the information processing apparatus described in the fifth aspect, the processor is configured to associate the first discontinuity information and the second discontinuity information with each other based on the recording condition.
[0015] According to a seventh aspect, in the information processing apparatus described in any one of the first to sixth aspects, the image is a radiation transmission image. The radiation transmission image can be used for a non-destructive inspection of the subject… n the information processing apparatus described in any one of the first to seventh aspects, the first criterion is a criterion for at least one of a type, a number, a position in the subject, a size, a shape, a presence density, or a distance from another discontinuity of the discontinuity, and the processor is configured to perform the recording by using information indicating at least one of the type, the number, the position in the subject, the size, the shape, the presence density, or the distance from the other discontinuity of the discontinuity as the feature of the discontinuity. The second aspect defines a specific aspect of the “first criterion” for the “discontinuity”. A value of the criterion may differ depending on a type of the discontinuity.--, in [0015]-[0016], and, -- the processor is configured to acquire the second discontinuity information based on the second criterion for a discontinuity in the first discontinuity information and for a discontinuity that has a lower influence on quality and/or performance of the subject than the discontinuity in the first discontinuity information.--, in [0018]; and, -- [0117] Further, similar to the first criterion, the second criterion is a criterion for at least one of the type, the number, the position in the inspection target object OBJ (subject), the size, the shape, the presence density, or the distance from another discontinuity of the discontinuity, but is a criterion that is stricter than the first criterion. The “second criterion being stricter than the first criterion” means that the discontinuity information is acquired by using the second criterion for a discontinuity that has a lower influence on quality and/or performance of the inspection target object OBJ (subject) than the discontinuity obtained by being analyzed by the first criterion. For example, discontinuities such as air bubbles, pores, or fissuring may be considered “defects” and result in the inspection target object OBJ being deemed defective, for example, in a case where the discontinuities are large or have a high density. However, for example, in a case where the discontinuities are small or have a low density, the discontinuities can be considered to have a low influence on the quality and/or the performance of the inspection target object OBJ (considered acceptable as a product). The processing unit 22 (processor) can use the criterion set by the user via the operation unit 14 as the first criterion and the second criterion.--, in [0117], [0123]; and, see: -- a state of the convolution operation in the learning model 240 shown in FIGS. 17A and 17B. In the first convolutional layer of the intermediate layer 252, the convolution operations between an image set (a learning image set during learning and a recognition image set during recognition such as detection) composed of a plurality of images and a filter F.sub.1 are performed….[0148] The output layer 254 is a layer that performs the position detection of the region of interest appearing in the input image (patch image or the like) based on the feature amount output from the intermediate layer 252 and that outputs the result. In a case of performing segmentation (class classification of the discontinuity or the like), the output layer 254 uses the “feature map” obtained from the intermediate layer 252 to understand the position of the region of interest appearing in the image at the pixel level. That is, it is possible to detect whether or not each pixel of the image belongs to the region of interest (whether or not it belongs to a specific class) and output the detection result. On the other hand, in a case of performing the object detection (detection of the discontinuity), it is not necessary to make a determination at the pixel level, and the output layer 254 outputs the positional information of the target object (discontinuity).--, in [0141]-[0148], and, -- [0153] In a case of using the learning model 240 having the above-described configuration, it is preferable to, during the learning process, calculate the loss function (error function) by comparing the result output by the output layer 254 and the correct answer of recognition for the image set, and reduce (minimize) the loss function by performing processing (error backpropagation) of updating the weight parameter in the intermediate layer 252 from the layer on the output side toward the layer on the input side.
[0154] The learning model 240 after the learning end performs at least one of detection, classification, or measurement as the image analysis, and the processing unit 22 (processor) can acquire the discontinuity information based on the result of the analysis through the learning model 240.
[Correction of Analysis Result]
[0155] The processing unit 22 may correct the result of the analysis by the learning model 240 according to the operation of the user via the operation unit 14 or the like. This correction may be, for example, addition of the discontinuity information, deletion of unnecessary information, amendment of an error, grouping of information, or the like.--, in [0153]-[0155]);
the article inspection apparatus comprising:
an image processing algorithm storage unit that stores, in advance, a plurality of image processing algorithms including the predetermined image processing algorithm (see TAIRA: e.g., Fig. 7, Figs. 2, 8-9, and, --[0015] According to a seventh aspect, in the information processing apparatus described in any one of the first to sixth aspects, the image is a radiation transmission image. The radiation transmission image can be used for a non-destructive inspection of the subject… n the information processing apparatus described in any one of the first to seventh aspects, the first criterion is a criterion for at least one of a type, a number, a position in the subject, a size, a shape, a presence density, or a distance from another discontinuity of the discontinuity, and the processor is configured to perform the recording by using information indicating at least one of the type, the number, the position in the subject, the size, the shape, the presence density, or the distance from the other discontinuity of the discontinuity as the feature of the discontinuity. The second aspect defines a specific aspect of the “first criterion” for the “discontinuity”. A value of the criterion may differ depending on a type of the discontinuity.--, in [0015]-[0016], and, -- the processor is configured to acquire the second discontinuity information based on the second criterion for a discontinuity in the first discontinuity information and for a discontinuity that has a lower influence on quality and/or performance of the subject than the discontinuity in the first discontinuity information.--, in [0018]; and, -- [0117] Further, similar to the first criterion, the second criterion is a criterion for at least one of the type, the number, the position in the inspection target object OBJ (subject), the size, the shape, the presence density, or the distance from another discontinuity of the discontinuity, but is a criterion that is stricter than the first criterion. The “second criterion being stricter than the first criterion” means that the discontinuity information is acquired by using the second criterion for a discontinuity that has a lower influence on quality and/or performance of the inspection target object OBJ (subject) than the discontinuity obtained by being analyzed by the first criterion. For example, discontinuities such as air bubbles, pores, or fissuring may be considered “defects” and result in the inspection target object OBJ being deemed defective, for example, in a case where the discontinuities are large or have a high density. However, for example, in a case where the discontinuities are small or have a low density, the discontinuities can be considered to have a low influence on the quality and/or the performance of the inspection target object OBJ (considered acceptable as a product). The processing unit 22 (processor) can use the criterion set by the user via the operation unit 14 as the first criterion and the second criterion.--, in [0117], [0123]; and, see: -- a state of the convolution operation in the learning model 240 shown in FIGS. 17A and 17B. In the first convolutional layer of the intermediate layer 252, the convolution operations between an image set (a learning image set during learning and a recognition image set during recognition such as detection) composed of a plurality of images and a filter F.sub.1 are performed….[0148] The output layer 254 is a layer that performs the position detection of the region of interest appearing in the input image (patch image or the like) based on the feature amount output from the intermediate layer 252 and that outputs the result. In a case of performing segmentation (class classification of the discontinuity or the like), the output layer 254 uses the “feature map” obtained from the intermediate layer 252 to understand the position of the region of interest appearing in the image at the pixel level. That is, it is possible to detect whether or not each pixel of the image belongs to the region of interest (whether or not it belongs to a specific class) and output the detection result. On the other hand, in a case of performing the object detection (detection of the discontinuity), it is not necessary to make a determination at the pixel level, and the output layer 254 outputs the positional information of the target object (discontinuity).--, in [0141]-[0148], and, -- [0153] In a case of using the learning model 240 having the above-described configuration, it is preferable to, during the learning process, calculate the loss function (error function) by comparing the result output by the output layer 254 and the correct answer of recognition for the image set, and reduce (minimize) the loss function by performing processing (error backpropagation) of updating the weight parameter in the intermediate layer 252 from the layer on the output side toward the layer on the input side.
[0154] The learning model 240 after the learning end performs at least one of detection, classification, or measurement as the image analysis, and the processing unit 22 (processor) can acquire the discontinuity information based on the result of the analysis through the learning model 240.
[Correction of Analysis Result]
[0155] The processing unit 22 may correct the result of the analysis by the learning model 240 according to the operation of the user via the operation unit 14 or the like. This correction may be, for example, addition of the discontinuity information, deletion of unnecessary information, amendment of an error, grouping of information, or the like.--, in [0153]-[0155]);
an image processing algorithm evaluation unit that evaluates suitability of each of the plurality of image processing algorithms applied to the inspection image based on performance information representing inspection performance in a case where the plurality of image processing algorithms are applied to a plurality of acquired images (see TAIRA: e.g., -- [0153] In a case of using the learning model 240 having the above-described configuration, it is preferable to, during the learning process, calculate the loss function (error function) by comparing the result output by the output layer 254 and the correct answer of recognition for the image set, and reduce (minimize) the loss function by performing processing (error backpropagation) of updating the weight parameter in the intermediate layer 252 from the layer on the output side toward the layer on the input side. {herein, the “calculate the loss function (error function)” read on claimed limitation “evaluates suitability of each of the plurality of image processing algorithms applied to the inspection image based on performance information representing inspection performance”};
[0154] The learning model 240 after the learning end performs at least one of detection, classification, or measurement as the image analysis, and the processing unit 22 (processor) can acquire the discontinuity information based on the result of the analysis through the learning model 240.
[Correction of Analysis Result]
[0155] The processing unit 22 may correct the result of the analysis by the learning model 240 according to the operation of the user via the operation unit 14 or the like. This correction may be, for example, addition of the discontinuity information, deletion of unnecessary information, amendment of an error, grouping of information, or the like.--, in [0153]-[0155]);
TAIRA however does not explicitly disclose that calculates a plurality of evaluation values representing evaluation results for each of the plurality of image processing algorithms;
SUDA discloses that calculates a plurality of evaluation values representing evaluation results for each of the plurality of image processing algorithms (see SUDA: e.g., -- an AI (artificial intelligence) model and an inspection device which corrects an image by the AI model where the accuracy of inspection by correcting noise removal and resolution enhancement may be improved by the AI model generated by learning using AI.--, in abstract, and, The correction method according to the present embodiment is a machine such as deep learning using an image (transmission image or cross-sectional image) of the object to be inspected captured by the inspection device 1 as teacher data and learning data in advance. It is configured to perform learning (referred to as "AI learning" in the following description) to generate an AI model, and in the inspection, the generated AI model is used to correct a transmission image or a cross-sectional image. First, the process of AI learning (deep learning) will be described with reference to FIGS. 6 to 9. In the following description, the "teacher image" is an image of an object to be inspected with a sufficient exposure time and radiation quality (brightness and hardness of radiation) with little noise, or a sufficient resolution. It shows an image of the object to be inspected (resolution that can obtain a predetermined inspection accuracy in the inspection), and the "learning image" is the exposure time, radiation quality, or radiation quality set in the actual inspection. It shall indicate an image of the subject to be inspected, captured at resolution, containing noise, or at low resolution…. Further, in the AI learning process, as shown in FIG. 7, 3/4 of the entire image area Ra is used for learning, and the remaining 1/4 area is verified (performance evaluation). Used for. The arrangement of the verification area and the learning area shown in FIG. 7 is an example, and any partial area Rs can be assigned for verification or learning. Further, the ratio of the verification area and the learning area in one image is not limited to the combination of 3/4 and 1/4, and can be appropriately set. --, in page 8/15 of the English version of JP-2022098590-A, as provided with the Office Action; also see: -- In AI learning, a learning image (which may be a transparent image or a cross-sectional image) is input by a method such as deep learning, and a teacher's image (similar to the learning image, a transparent image and a cross-sectional image). Train the AI model so that the image is close to (may be). Then, the learning processing unit 37 updates the AI model based on the result of AI learning (step S216). Further, the learning processing unit 37 may be configured to display the learning progress status on the monitor 12, and if the learning progress status is displayed, the display is updated (step S218). Progress includes performance evaluation of AI models.
The performance evaluation of the AI model is performed by calculating the loss. The AI model trained by AI learning is applied to each image (input image) for learning and verification, and a corrected image (high-quality output image) is generated. The loss is calculated by taking the difference between the evaluation values of the generated output image and the teacher's image (for example, the mean square error of the luminance value). The loss here is the quality at which the difference between the teacher image and the image corrected by the AI model is lost or impaired (or irrecoverable) --, in page 10/15 of the English version of JP-2022098590-A, as provided with the Office Action);
TAIRA and SUDA are combinable as they are in the same field of endeavor: learning models applied in object inspection. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify TAIRA’s apparatus using SUDA’ teachings by including calculates a plurality of evaluation values representing evaluation results for each of the plurality of image processing algorithms to TAIRA’s performance evaluation {loss functions} in order to performance evaluation of the AI learning models applied to each image for object inspections (see SUDA: e.g. in abstract, page8, 10/15 of the English version of JP-2022098590-A, as provided with the Office Action);
TAIRA as modified by SUDA further disclose an image processing algorithm setting unit that sets the predetermined image processing algorithm used for determining the quality state of the inspection object article based on the plurality of evaluation values calculated by the image processing algorithm evaluation (see TAIRA: e.g., -- [0153] In a case of using the learning model 240 having the above-described configuration, it is preferable to, during the learning process, calculate the loss function (error function) by comparing the result output by the output layer 254 and the correct answer of recognition for the image set, and reduce (minimize) the loss function by performing processing (error backpropagation) of updating the weight parameter in the intermediate layer 252 from the layer on the output side toward the layer on the input side. {herein, the “calculate the loss function (error function)” read on claimed limitation “evaluates suitability of each of the plurality of image processing algorithms applied to the inspection image based on performance information representing inspection performance”};
[0154] The learning model 240 after the learning end performs at least one of detection, classification, or measurement as the image analysis, and the processing unit 22 (processor) can acquire the discontinuity information based on the result of the analysis through the learning model 240.
[Correction of Analysis Result]
[0155] The processing unit 22 may correct the result of the analysis by the learning model 240 according to the operation of the user via the operation unit 14 or the like. This correction may be, for example, addition of the discontinuity information, deletion of unnecessary information, amendment of an error, grouping of information, or the like.--, in [0153]-[0155]; also see SUDA: e.g., -- an AI (artificial intelligence) model and an inspection device which corrects an image by the AI model where the accuracy of inspection by correcting noise removal and resolution enhancement may be improved by the AI model generated by learning using AI.--, in abstract, and, The correction method according to the present embodiment is a machine such as deep learning using an image (transmission image or cross-sectional image) of the object to be inspected captured by the inspection device 1 as teacher data and learning data in advance. It is configured to perform learning (referred to as "AI learning" in the following description) to generate an AI model, and in the inspection, the generated AI model is used to correct a transmission image or a cross-sectional image. First, the process of AI learning (deep learning) will be described with reference to FIGS. 6 to 9. In the following description, the "teacher image" is an image of an object to be inspected with a sufficient exposure time and radiation quality (brightness and hardness of radiation) with little noise, or a sufficient resolution. It shows an image of the object to be inspected (resolution that can obtain a predetermined inspection accuracy in the inspection), and the "learning image" is the exposure time, radiation quality, or radiation quality set in the actual inspection. It shall indicate an image of the subject to be inspected, captured at resolution, containing noise, or at low resolution…. Further, in the AI learning process, as shown in FIG. 7, 3/4 of the entire image area Ra is used for learning, and the remaining 1/4 area is verified (performance evaluation). Used for. The arrangement of the verification area and the learning area shown in FIG. 7 is an example, and any partial area Rs can be assigned for verification or learning. Further, the ratio of the verification area and the learning area in one image is not limited to the combination of 3/4 and 1/4, and can be appropriately set. --, in page 8/15 of the English version of JP-2022098590-A, as provided with the Office Action; also see: -- In AI learning, a learning image (which may be a transparent image or a cross-sectional image) is input by a method such as deep learning, and a teacher's image (similar to the learning image, a transparent image and a cross-sectional image). Train the AI model so that the image is close to (may be). Then, the learning processing unit 37 updates the AI model based on the result of AI learning (step S216). Further, the learning processing unit 37 may be configured to display the learning progress status on the monitor 12, and if the learning progress status is displayed, the display is updated (step S218). Progress includes performance evaluation of AI models.
The performance evaluation of the AI model is performed by calculating the loss. The AI model trained by AI learning is applied to each image (input image) for learning and verification, and a corrected image (high-quality output image) is generated. The loss is calculated by taking the difference between the evaluation values of the generated output image and the teacher's image (for example, the mean square error of the luminance value). The loss here is the quality at which the difference between the teacher image and the image corrected by the AI model is lost or impaired (or irrecoverable) --, in page 10/15 of the English version of JP-2022098590-A, as provided with the Office Action; and, --whether the correction target of the AI model is a transparent image or a cross sectional image, noise is removed from the transparent image or the cross-sectional image or the resolution is increased, the synogram is interpolated, or the subject is inspected. Whether to learn the image of a specific area (board surface or specific component), create an AI model for each FOV, create an AI model common to FOV, or create a teacher image and a learning image. The learning conditions such as the image quality and the learning end condition can be set for the control unit 10 from the user interface including the monitor 12 and the keyboard / mouse. In addition, the learning status can be confirmed and the learning process can be completed via this user interface. When the AI model obtained as described above is applied to an inspection, the AI model described above is applied to the acquired transmission image or reconstructed image (cross-sectional image) in step S120 shown in FIG. 3 for correction. (Correct noise, increase resolution, interpolate synogram, etc.). Hereinafter, the process of correcting the transmission image or the reconstructed image (cross-sectional image) by the AI model will be described with reference to FIG. FIG. 10A shows a process when the correction by the AI model is applied to the transparent image. The control unit 10 captures a transmitted image of the object to be inspected under the imaging conditions for inspection (step S121a), and corrects the captured transmitted image using the AI model (step S122a).--, in in page 12/15 of the English version of JP-2022098590-A, as provided with the Office Action).
Re Claim 2, TAIRA as modified by SUDA further disclose wherein the image processing algorithm evaluation unit calculates the plurality of evaluation values such that numerical values of the evaluation values become larger as the inspection performance represented by the performance information is superior (see TAIRA: e.g., -- [0153] In a case of using the learning model 240 having the above-described configuration, it is preferable to, during the learning process, calculate the loss function (error function) by comparing the result output by the output layer 254 and the correct answer of recognition for the image set, and reduce (minimize) the loss function by performing processing (error backpropagation) of updating the weight parameter in the intermediate layer 252 from the layer on the output side toward the layer on the input side. {herein, the “calculate the loss function (error function)” read on claimed limitation “evaluates suitability of each of the plurality of image processing algorithms applied to the inspection image based on performance information representing inspection performance”};
[0154] The learning model 240 after the learning end performs at least one of detection, classification, or measurement as the image analysis, and the processing unit 22 (processor) can acquire the discontinuity information based on the result of the analysis through the learning model 240.
[Correction of Analysis Result]
[0155] The processing unit 22 may correct the result of the analysis by the learning model 240 according to the operation of the user via the operation unit 14 or the like. This correction may be, for example, addition of the discontinuity information, deletion of unnecessary information, amendment of an error, grouping of information, or the like.--, in [0153]-[0155]; also see SUDA: e.g., -- an AI (artificial intelligence) model and an inspection device which corrects an image by the AI model where the accuracy of inspection by correcting noise removal and resolution enhancement may be improved by the AI model generated by learning using AI.--, in abstract, and, The correction method according to the present embodiment is a machine such as deep learning using an image (transmission image or cross-sectional image) of the object to be inspected captured by the inspection device 1 as teacher data and learning data in advance. It is configured to perform learning (referred to as "AI learning" in the following description) to generate an AI model, and in the inspection, the generated AI model is used to correct a transmission image or a cross-sectional image. First, the process of AI learning (deep learning) will be described with reference to FIGS. 6 to 9. In the following description, the "teacher image" is an image of an object to be inspected with a sufficient exposure time and radiation quality (brightness and hardness of radiation) with little noise, or a sufficient resolution. It shows an image of the object to be inspected (resolution that can obtain a predetermined inspection accuracy in the inspection), and the "learning image" is the exposure time, radiation quality, or radiation quality set in the actual inspection. It shall indicate an image of the subject to be inspected, captured at resolution, containing noise, or at low resolution…. Further, in the AI learning process, as shown in FIG. 7, 3/4 of the entire image area Ra is used for learning, and the remaining 1/4 area is verified (performance evaluation). Used for. The arrangement of the verification area and the learning area shown in FIG. 7 is an example, and any partial area Rs can be assigned for verification or learning. Further, the ratio of the verification area and the learning area in one image is not limited to the combination of 3/4 and 1/4, and can be appropriately set. --, in page 8/15 of the English version of JP-2022098590-A, as provided with the Office Action; also see: -- In AI learning, a learning image (which may be a transparent image or a cross-sectional image) is input by a method such as deep learning, and a teacher's image (similar to the learning image, a transparent image and a cross-sectional image). Train the AI model so that the image is close to (may be). Then, the learning processing unit 37 updates the AI model based on the result of AI learning (step S216). Further, the learning processing unit 37 may be configured to display the learning progress status on the monitor 12, and if the learning progress status is displayed, the display is updated (step S218). Progress includes performance evaluation of AI models.
The performance evaluation of the AI model is performed by calculating the loss. The AI model trained by AI learning is applied to each image (input image) for learning and verification, and a corrected image (high-quality output image) is generated. The loss is calculated by taking the difference between the evaluation values of the generated output image and the teacher's image (for example, the mean square error of the luminance value). The loss here is the quality at which the difference between the teacher image and the image corrected by the AI model is lost or impaired (or irrecoverable) --, in page 10/15 of the English version of JP-2022098590-A, as provided with the Office Action; and, --whether the correction target of the AI model is a transparent image or a cross sectional image, noise is removed from the transparent image or the cross-sectional image or the resolution is increased, the synogram is interpolated, or the subject is inspected. Whether to learn the image of a specific area (board surface or specific component), create an AI model for each FOV, create an AI model common to FOV, or create a teacher image and a learning image. The learning conditions such as the image quality and the learning end condition can be set for the control unit 10 from the user interface including the monitor 12 and the keyboard / mouse. In addition, the learning status can be confirmed and the learning process can be completed via this user interface. When the AI model obtained as described above is applied to an inspection, the AI model described above is applied to the acquired transmission image or reconstructed image (cross-sectional image) in step S120 shown in FIG. 3 for correction. (Correct noise, increase resolution, interpolate synogram, etc.). Hereinafter, the process of correcting the transmission image or the reconstructed image (cross-sectional image) by the AI model will be described with reference to FIG. FIG. 10A shows a process when the correction by the AI model is applied to the transparent image. The control unit 10 captures a transmitted image of the object to be inspected under the imaging conditions for inspection (step S121a), and corrects the captured transmitted image using the AI model (step S122a).--, in in page 12/15 of the English version of JP-2022098590-A, as provided with the Office Action);
the image processing algorithm setting unit selects a superior part of the image processing algorithms from among the plurality of image processing algorithms based on the numerical values of the plurality of evaluation values, and sets the predetermined image processing algorithm (see TAIRA: e.g., -- [0153] In a case of using the learning model 240 having the above-described configuration, it is preferable to, during the learning process, calculate the loss function (error function) by comparing the result output by the output layer 254 and the correct answer of recognition for the image set, and reduce (minimize) the loss function by performing processing (error backpropagation) of updating the weight parameter in the intermediate layer 252 from the layer on the output side toward the layer on the input side. {herein, the “calculate the loss function (error function)” read on claimed limitation “evaluates suitability of each of the plurality of image processing algorithms applied to the inspection image based on performance information representing inspection performance”};
[0154] The learning model 240 after the learning end performs at least one of detection, classification, or measurement as the image analysis, and the processing unit 22 (processor) can acquire the discontinuity information based on the result of the analysis through the learning model 240.
[Correction of Analysis Result]
[0155] The processing unit 22 may correct the result of the analysis by the learning model 240 according to the operation of the user via the operation unit 14 or the like. This correction may be, for example, addition of the discontinuity information, deletion of unnecessary information, amendment of an error, grouping of information, or the like.--, in [0153]-[0155]; also see SUDA: e.g., -- an AI (artificial intelligence) model and an inspection device which corrects an image by the AI model where the accuracy of inspection by correcting noise removal and resolution enhancement may be improved by the AI model generated by learning using AI.--, in abstract, and, The correction method according to the present embodiment is a machine such as deep learning using an image (transmission image or cross-sectional image) of the object to be inspected captured by the inspection device 1 as teacher data and learning data in advance. It is configured to perform learning (referred to as "AI learning" in the following description) to generate an AI model, and in the inspection, the generated AI model is used to correct a transmission image or a cross-sectional image. First, the process of AI learning (deep learning) will be described with reference to FIGS. 6 to 9. In the following description, the "teacher image" is an image of an object to be inspected with a sufficient exposure time and radiation quality (brightness and hardness of radiation) with little noise, or a sufficient resolution. It shows an image of the object to be inspected (resolution that can obtain a predetermined inspection accuracy in the inspection), and the "learning image" is the exposure time, radiation quality, or radiation quality set in the actual inspection. It shall indicate an image of the subject to be inspected, captured at resolution, containing noise, or at low resolution…. Further, in the AI learning process, as shown in FIG. 7, 3/4 of the entire image area Ra is used for learning, and the remaining 1/4 area is verified (performance evaluation). Used for. The arrangement of the verification area and the learning area shown in FIG. 7 is an example, and any partial area Rs can be assigned for verification or learning. Further, the ratio of the verification area and the learning area in one image is not limited to the combination of 3/4 and 1/4, and can be appropriately set. --, in page 8/15 of the English version of JP-2022098590-A, as provided with the Office Action; also see: -- In AI learning, a learning image (which may be a transparent image or a cross-sectional image) is input by a method such as deep learning, and a teacher's image (similar to the learning image, a transparent image and a cross-sectional image). Train the AI model so that the image is close to (may be). Then, the learning processing unit 37 updates the AI model based on the result of AI learning (step S216). Further, the learning processing unit 37 may be configured to display the learning progress status on the monitor 12, and if the learning progress status is displayed, the display is updated (step S218). Progress includes performance evaluation of AI models.
The performance evaluation of the AI model is performed by calculating the loss. The AI model trained by AI learning is applied to each image (input image) for learning and verification, and a corrected image (high-quality output image) is generated. The loss is calculated by taking the difference between the evaluation values of the generated output image and the teacher's image (for example, the mean square error of the luminance value). The loss here is the quality at which the difference between the teacher image and the image corrected by the AI model is lost or impaired (or irrecoverable) --, in page 10/15 of the English version of JP-2022098590-A, as provided with the Office Action; and, --whether the correction target of the AI model is a transparent image or a cross sectional image, noise is removed from the transparent image or the cross-sectional image or the resolution is increased, the synogram is interpolated, or the subject is inspected. Whether to learn the image of a specific area (board surface or specific component), create an AI model for each FOV, create an AI model common to FOV, or create a teacher image and a learning image. The learning conditions such as the image quality and the learning end condition can be set for the control unit 10 from the user interface including the monitor 12 and the keyboard / mouse. In addition, the learning status can be confirmed and the learning process can be completed via this user interface. When the AI model obtained as described above is applied to an inspection, the AI model described above is applied to the acquired transmission image or reconstructed image (cross-sectional image) in step S120 shown in FIG. 3 for correction. (Correct noise, increase resolution, interpolate synogram, etc.). Hereinafter, the process of correcting the transmission image or the reconstructed image (cross-sectional image) by the AI model will be described with reference to FIG. FIG. 10A shows a process when the correction by the AI model is applied to the transparent image. The control unit 10 captures a transmitted image of the object to be inspected under the imaging conditions for inspection (step S121a), and corrects the captured transmitted image using the AI model (step S122a).--, in in page 12/15 of the English version of JP-2022098590-A, as provided with the Office Action).
Re Claim 3, TAIRA as modified by SUDA further disclose wherein the plurality of evaluation values are calculated so as to be a maximum value in a case where the performance information represents inspection performance that is superior to a predetermined level or more (see TAIRA: e.g., -- [0153] In a case of using the learning model 240 having the above-described configuration, it is preferable to, during the learning process, calculate the loss function (error function) by comparing the result output by the output layer 254 and the correct answer of recognition for the image set, and reduce (minimize) the loss function by performing processing (error backpropagation) of updating the weight parameter in the intermediate layer 252 from the layer on the output side toward the layer on the input side. {herein, the “calculate the loss function (error function)” read on claimed limitation “evaluates suitability of each of the plurality of image processing algorithms applied to the inspection image based on performance information representing inspection performance”};
[0154] The learning model 240 after the learning end performs at least one of detection, classification, or measurement as the image analysis, and the processing unit 22 (processor) can acquire the discontinuity information based on the result of the analysis through the learning model 240.
[Correction of Analysis Result]
[0155] The processing unit 22 may correct the result of the analysis by the learning model 240 according to the operation of the user via the operation unit 14 or the like. This correction may be, for example, addition of the discontinuity information, deletion of unnecessary information, amendment of an error, grouping of information, or the like.--, in [0153]-[0155]; also see SUDA: e.g., -- an AI (artificial intelligence) model and an inspection device which corrects an image by the AI model where the accuracy of inspection by correcting noise removal and resolution enhancement may be improved by the AI model generated by learning using AI.--, in abstract, and, The correction method according to the present embodiment is a machine such as deep learning using an image (transmission image or cross-sectional image) of the object to be inspected captured by the inspection device 1 as teacher data and learning data in advance. It is configured to perform learning (referred to as "AI learning" in the following description) to generate an AI model, and in the inspection, the generated AI model is used to correct a transmission image or a cross-sectional image. First, the process of AI learning (deep learning) will be described with reference to FIGS. 6 to 9. In the following description, the "teacher image" is an image of an object to be inspected with a sufficient exposure time and radiation quality (brightness and hardness of radiation) with little noise, or a sufficient resolution. It shows an image of the object to be inspected (resolution that can obtain a predetermined inspection accuracy in the inspection), and the "learning image" is the exposure time, radiation quality, or radiation quality set in the actual inspection. It shall indicate an image of the subject to be inspected, captured at resolution, containing noise, or at low resolution…. Further, in the AI learning process, as shown in FIG. 7, 3/4 of the entire image area Ra is used for learning, and the remaining 1/4 area is verified (performance evaluation). Used for. The arrangement of the verification area and the learning area shown in FIG. 7 is an example, and any partial area Rs can be assigned for verification or learning. Further, the ratio of the verification area and the learning area in one image is not limited to the combination of 3/4 and 1/4, and can be appropriately set. --, in page 8/15 of the English version of JP-2022098590-A, as provided with the Office Action; also see: -- In AI learning, a learning image (which may be a transparent image or a cross-sectional image) is input by a method such as deep learning, and a teacher's image (similar to the learning image, a transparent image and a cross-sectional image). Train the AI model so that the image is close to (may be). Then, the learning processing unit 37 updates the AI model based on the result of AI learning (step S216). Further, the learning processing unit 37 may be configured to display the learning progress status on the monitor 12, and if the learning progress status is displayed, the display is updated (step S218). Progress includes performance evaluation of AI models.
The performance evaluation of the AI model is performed by calculating the loss. The AI model trained by AI learning is applied to each image (input image) for learning and verification, and a corrected image (high-quality output image) is generated. The loss is calculated by taking the difference between the evaluation values of the generated output image and the teacher's image (for example, the mean square error of the luminance value). The loss here is the quality at which the difference between the teacher image and the image corrected by the AI model is lost or impaired (or irrecoverable) --, in page 10/15 of the English version of JP-2022098590-A, as provided with the Office Action; and, --whether the correction target of the AI model is a transparent image or a cross sectional image, noise is removed from the transparent image or the cross-sectional image or the resolution is increased, the synogram is interpolated, or the subject is inspected. Whether to learn the image of a specific area (board surface or specific component), create an AI model for each FOV, create an AI model common to FOV, or create a teacher image and a learning image. The learning conditions such as the image quality and the learning end condition can be set for the control unit 10 from the user interface including the monitor 12 and the keyboard / mouse. In addition, the learning status can be confirmed and the learning process can be completed via this user interface. When the AI model obtained as described above is applied to an inspection, the AI model described above is applied to the acquired transmission image or reconstructed image (cross-sectional image) in step S120 shown in FIG. 3 for correction. (Correct noise, increase resolution, interpolate synogram, etc.). Hereinafter, the process of correcting the transmission image or the reconstructed image (cross-sectional image) by the AI model will be described with reference to FIG. FIG. 10A shows a process when the correction by the AI model is applied to the transparent image. The control unit 10 captures a transmitted image of the object to be inspected under the imaging conditions for inspection (step S121a), and corrects the captured transmitted image using the AI model (step S122a).--, in in page 12/15 of the English version of JP-2022098590-A, as provided with the Office Action), and
the image processing algorithm setting unit displays a predetermined number of the image processing algorithms from among the plurality of image processing algorithms in order of superiority of the plurality of evaluation values in a selectable manner on a display device (see TAIRA: e.g., -- [0153] In a case of using the learning model 240 having the above-described configuration, it is preferable to, during the learning process, calculate the loss function (error function) by comparing the result output by the output layer 254 and the correct answer of recognition for the image set, and reduce (minimize) the loss function by performing processing (error backpropagation) of updating the weight parameter in the intermediate layer 252 from the layer on the output side toward the layer on the input side. {herein, the “calculate the loss function (error function)” read on claimed limitation “evaluates suitability of each of the plurality of image processing algorithms applied to the inspection image based on performance information representing inspection performance”};
[0154] The learning model 240 after the learning end performs at least one of detection, classification, or measurement as the image analysis, and the processing unit 22 (processor) can acquire the discontinuity information based on the result of the analysis through the learning model 240.
[Correction of Analysis Result]
[0155] The processing unit 22 may correct the result of the analysis by the learning model 240 according to the operation of the user via the operation unit 14 or the like. This correction may be, for example, addition of the discontinuity information, deletion of unnecessary information, amendment of an error, grouping of information, or the like.--, in [0153]-[0155]; also see SUDA: e.g., -- an AI (artificial intelligence) model and an inspection device which corrects an image by the AI model where the accuracy of inspection by correcting noise removal and resolution enhancement may be improved by the AI model generated by learning using AI.--, in abstract, and, The correction method according to the present embodiment is a machine such as deep learning using an image (transmission image or cross-sectional image) of the object to be inspected captured by the inspection device 1 as teacher data and learning data in advance. It is configured to perform learning (referred to as "AI learning" in the following description) to generate an AI model, and in the inspection, the generated AI model is used to correct a transmission image or a cross-sectional image. First, the process of AI learning (deep learning) will be described with reference to FIGS. 6 to 9. In the following description, the "teacher image" is an image of an object to be inspected with a sufficient exposure time and radiation quality (brightness and hardness of radiation) with little noise, or a sufficient resolution. It shows an image of the object to be inspected (resolution that can obtain a predetermined inspection accuracy in the inspection), and the "learning image" is the exposure time, radiation quality, or radiation quality set in the actual inspection. It shall indicate an image of the subject to be inspected, captured at resolution, containing noise, or at low resolution…. Further, in the AI learning process, as shown in FIG. 7, 3/4 of the entire image area Ra is used for learning, and the remaining 1/4 area is verified (performance evaluation). Used for. The arrangement of the verification area and the learning area shown in FIG. 7 is an example, and any partial area Rs can be assigned for verification or learning. Further, the ratio of the verification area and the learning area in one image is not limited to the combination of 3/4 and 1/4, and can be appropriately set. --, in page 8/15 of the English version of JP-2022098590-A, as provided with the Office Action; also see: -- In AI learning, a learning image (which may be a transparent image or a cross-sectional image) is input by a method such as deep learning, and a teacher's image (similar to the learning image, a transparent image and a cross-sectional image). Train the AI model so that the image is close to (may be). Then, the learning processing unit 37 updates the AI model based on the result of AI learning (step S216). Further, the learning processing unit 37 may be configured to display the learning progress status on the monitor 12, and if the learning progress status is displayed, the display is updated (step S218). Progress includes performance evaluation of AI models.
The performance evaluation of the AI model is performed by calculating the loss. The AI model trained by AI learning is applied to each image (input image) for learning and verification, and a corrected image (high-quality output image) is generated. The loss is calculated by taking the difference between the evaluation values of the generated output image and the teacher's image (for example, the mean square error of the luminance value). The loss here is the quality at which the difference between the teacher image and the image corrected by the AI model is lost or impaired (or irrecoverable) --, in page 10/15 of the English version of JP-2022098590-A, as provided with the Office Action; and, --whether the correction target of the AI model is a transparent image or a cross sectional image, noise is removed from the transparent image or the cross-sectional image or the resolution is increased, the synogram is interpolated, or the subject is inspected. Whether to learn the image of a specific area (board surface or specific component), create an AI model for each FOV, create an AI model common to FOV, or create a teacher image and a learning image. The learning conditions such as the image quality and the learning end condition can be set for the control unit 10 from the user interface including the monitor 12 and the keyboard / mouse. In addition, the learning status can be confirmed and the learning process can be completed via this user interface. When the AI model obtained as described above is applied to an inspection, the AI model described above is applied to the acquired transmission image or reconstructed image (cross-sectional image) in step S120 shown in FIG. 3 for correction. (Correct noise, increase resolution, interpolate synogram, etc.). Hereinafter, the process of correcting the transmission image or the reconstructed image (cross-sectional image) by the AI model will be described with reference to FIG. FIG. 10A shows a process when the correction by the AI model is applied to the transparent image. The control unit 10 captures a transmitted image of the object to be inspected under the imaging conditions for inspection (step S121a), and corrects the captured transmitted image using the AI model (step S122a).--, in in page 12/15 of the English version of JP-2022098590-A, as provided with the Office Action).
Re Claim 4, TAIRA as modified by SUDA further disclose wherein the plurality of evaluation values are calculated so as to be a maximum value in a case where the performance information represents inspection performance that is superior to a predetermined level or more (see TAIRA: e.g., -- [0153] In a case of using the learning model 240 having the above-described configuration, it is preferable to, during the learning process, calculate the loss function (error function) by comparing the result output by the output layer 254 and the correct answer of recognition for the image set, and reduce (minimize) the loss function by performing processing (error backpropagation) of updating the weight parameter in the intermediate layer 252 from the layer on the output side toward the layer on the input side. {herein, the “calculate the loss function (error function)” read on claimed limitation “evaluates suitability of each of the plurality of image processing algorithms applied to the inspection image based on performance information representing inspection performance”};
[0154] The learning model 240 after the learning end performs at least one of detection, classification, or measurement as the image analysis, and the processing unit 22 (processor) can acquire the discontinuity information based on the result of the analysis through the learning model 240.
[Correction of Analysis Result]
[0155] The processing unit 22 may correct the result of the analysis by the learning model 240 according to the operation of the user via the operation unit 14 or the like. This correction may be, for example, addition of the discontinuity information, deletion of unnecessary information, amendment of an error, grouping of information, or the like.--, in [0153]-[0155]; also see SUDA: e.g., -- an AI (artificial intelligence) model and an inspection device which corrects an image by the AI model where the accuracy of inspection by correcting noise removal and resolution enhancement may be improved by the AI model generated by learning using AI.--, in abstract, and, The correction method according to the present embodiment is a machine such as deep learning using an image (transmission image or cross-sectional image) of the object to be inspected captured by the inspection device 1 as teacher data and learning data in advance. It is configured to perform learning (referred to as "AI learning" in the following description) to generate an AI model, and in the inspection, the generated AI model is used to correct a transmission image or a cross-sectional image. First, the process of AI learning (deep learning) will be described with reference to FIGS. 6 to 9. In the following description, the "teacher image" is an image of an object to be inspected with a sufficient exposure time and radiation quality (brightness and hardness of radiation) with little noise, or a sufficient resolution. It shows an image of the object to be inspected (resolution that can obtain a predetermined inspection accuracy in the inspection), and the "learning image" is the exposure time, radiation quality, or radiation quality set in the actual inspection. It shall indicate an image of the subject to be inspected, captured at resolution, containing noise, or at low resolution…. Further, in the AI learning process, as shown in FIG. 7, 3/4 of the entire image area Ra is used for learning, and the remaining 1/4 area is verified (performance evaluation). Used for. The arrangement of the verification area and the learning area shown in FIG. 7 is an example, and any partial area Rs can be assigned for verification or learning. Further, the ratio of the verification area and the learning area in one image is not limited to the combination of 3/4 and 1/4, and can be appropriately set. --, in page 8/15 of the English version of JP-2022098590-A, as provided with the Office Action; also see: -- In AI learning, a learning image (which may be a transparent image or a cross-sectional image) is input by a method such as deep learning, and a teacher's image (similar to the learning image, a transparent image and a cross-sectional image). Train the AI model so that the image is close to (may be). Then, the learning processing unit 37 updates the AI model based on the result of AI learning (step S216). Further, the learning processing unit 37 may be configured to display the learning progress status on the monitor 12, and if the learning progress status is displayed, the display is updated (step S218). Progress includes performance evaluation of AI models.
The performance evaluation of the AI model is performed by calculating the loss. The AI model trained by AI learning is applied to each image (input image) for learning and verification, and a corrected image (high-quality output image) is generated. The loss is calculated by taking the difference between the evaluation values of the generated output image and the teacher's image (for example, the mean square error of the luminance value). The loss here is the quality at which the difference between the teacher image and the image corrected by the AI model is lost or impaired (or irrecoverable) --, in page 10/15 of the English version of JP-2022098590-A, as provided with the Office Action; and, --whether the correction target of the AI model is a transparent image or a cross sectional image, noise is removed from the transparent image or the cross-sectional image or the resolution is increased, the synogram is interpolated, or the subject is inspected. Whether to learn the image of a specific area (board surface or specific component), create an AI model for each FOV, create an AI model common to FOV, or create a teacher image and a learning image. The learning conditions such as the image quality and the learning end condition can be set for the control unit 10 from the user interface including the monitor 12 and the keyboard / mouse. In addition, the learning status can be confirmed and the learning process can be completed via this user interface. When the AI model obtained as described above is applied to an inspection, the AI model described above is applied to the acquired transmission image or reconstructed image (cross-sectional image) in step S120 shown in FIG. 3 for correction. (Correct noise, increase resolution, interpolate synogram, etc.). Hereinafter, the process of correcting the transmission image or the reconstructed image (cross-sectional image) by the AI model will be described with reference to FIG. FIG. 10A shows a process when the correction by the AI model is applied to the transparent image. The control unit 10 captures a transmitted image of the object to be inspected under the imaging conditions for inspection (step S121a), and corrects the captured transmitted image using the AI model (step S122a).--, in in page 12/15 of the English version of JP-2022098590-A, as provided with the Office Action), and
the image processing algorithm setting unit displays a predetermined number of the image processing algorithms from among the plurality of image processing algorithms in order of superiority of the plurality of evaluation values in a selectable manner on a display device (see TAIRA: e.g., -- [0153] In a case of using the learning model 240 having the above-described configuration, it is preferable to, during the learning process, calculate the loss function (error function) by comparing the result output by the output layer 254 and the correct answer of recognition for the image set, and reduce (minimize) the loss function by performing processing (error backpropagation) of updating the weight parameter in the intermediate layer 252 from the layer on the output side toward the layer on the input side. {herein, the “calculate the loss function (error function)” read on claimed limitation “evaluates suitability of each of the plurality of image processing algorithms applied to the inspection image based on performance information representing inspection performance”};
[0154] The learning model 240 after the learning end performs at least one of detection, classification, or measurement as the image analysis, and the processing unit 22 (processor) can acquire the discontinuity information based on the result of the analysis through the learning model 240.
[Correction of Analysis Result]
[0155] The processing unit 22 may correct the result of the analysis by the learning model 240 according to the operation of the user via the operation unit 14 or the like. This correction may be, for example, addition of the discontinuity information, deletion of unnecessary information, amendment of an error, grouping of information, or the like.--, in [0153]-[0155]; also see SUDA: e.g., -- an AI (artificial intelligence) model and an inspection device which corrects an image by the AI model where the accuracy of inspection by correcting noise removal and resolution enhancement may be improved by the AI model generated by learning using AI.--, in abstract, and, The correction method according to the present embodiment is a machine such as deep learning using an image (transmission image or cross-sectional image) of the object to be inspected captured by the inspection device 1 as teacher data and learning data in advance. It is configured to perform learning (referred to as "AI learning" in the following description) to generate an AI model, and in the inspection, the generated AI model is used to correct a transmission image or a cross-sectional image. First, the process of AI learning (deep learning) will be described with reference to FIGS. 6 to 9. In the following description, the "teacher image" is an image of an object to be inspected with a sufficient exposure time and radiation quality (brightness and hardness of radiation) with little noise, or a sufficient resolution. It shows an image of the object to be inspected (resolution that can obtain a predetermined inspection accuracy in the inspection), and the "learning image" is the exposure time, radiation quality, or radiation quality set in the actual inspection. It shall indicate an image of the subject to be inspected, captured at resolution, containing noise, or at low resolution…. Further, in the AI learning process, as shown in FIG. 7, 3/4 of the entire image area Ra is used for learning, and the remaining 1/4 area is verified (performance evaluation). Used for. The arrangement of the verification area and the learning area shown in FIG. 7 is an example, and any partial area Rs can be assigned for verification or learning. Further, the ratio of the verification area and the learning area in one image is not limited to the combination of 3/4 and 1/4, and can be appropriately set. --, in page 8/15 of the English version of JP-2022098590-A, as provided with the Office Action; also see: -- In AI learning, a learning image (which may be a transparent image or a cross-sectional image) is input by a method such as deep learning, and a teacher's image (similar to the learning image, a transparent image and a cross-sectional image). Train the AI model so that the image is close to (may be). Then, the learning processing unit 37 updates the AI model based on the result of AI learning (step S216). Further, the learning processing unit 37 may be configured to display the learning progress status on the monitor 12, and if the learning progress status is displayed, the display is updated (step S218). Progress includes performance evaluation of AI models.
The performance evaluation of the AI model is performed by calculating the loss. The AI model trained by AI learning is applied to each image (input image) for learning and verification, and a corrected image (high-quality output image) is generated. The loss is calculated by taking the difference between the evaluation values of the generated output image and the teacher's image (for example, the mean square error of the luminance value). The loss here is the quality at which the difference between the teacher image and the image corrected by the AI model is lost or impaired (or irrecoverable) --, in page 10/15 of the English version of JP-2022098590-A, as provided with the Office Action; and, --whether the correction target of the AI model is a transparent image or a cross sectional image, noise is removed from the transparent image or the cross-sectional image or the resolution is increased, the synogram is interpolated, or the subject is inspected. Whether to learn the image of a specific area (board surface or specific component), create an AI model for each FOV, create an AI model common to FOV, or create a teacher image and a learning image. The learning conditions such as the image quality and the learning end condition can be set for the control unit 10 from the user interface including the monitor 12 and the keyboard / mouse. In addition, the learning status can be confirmed and the learning process can be completed via this user interface. When the AI model obtained as described above is applied to an inspection, the AI model described above is applied to the acquired transmission image or reconstructed image (cross-sectional image) in step S120 shown in FIG. 3 for correction. (Correct noise, increase resolution, interpolate synogram, etc.). Hereinafter, the process of correcting the transmission image or the reconstructed image (cross-sectional image) by the AI model will be described with reference to FIG. FIG. 10A shows a process when the correction by the AI model is applied to the transparent image. The control unit 10 captures a transmitted image of the object to be inspected under the imaging conditions for inspection (step S121a), and corrects the captured transmitted image using the AI model (step S122a).--, in in page 12/15 of the English version of JP-2022098590-A, as provided with the Office Action).
Re Claims 5-6, TAIRA as modified by SUDA further disclose wherein the image processing algorithm evaluation unit evaluates suitability of the inspection of each of the plurality of image processing algorithms for the inspection image, based on a learning model created in advance through training using, as teacher data, inspection images of a plurality of types of product groups, which are other articles, and an image processing algorithm suitable for the inspection images of the product groups (see TAIRA: e.g., -- [0153] In a case of using the learning model 240 having the above-described configuration, it is preferable to, during the learning process, calculate the loss function (error function) by comparing the result output by the output layer 254 and the correct answer of recognition for the image set, and reduce (minimize) the loss function by performing processing (error backpropagation) of updating the weight parameter in the intermediate layer 252 from the layer on the output side toward the layer on the input side. {herein, the “calculate the loss function (error function)” read on claimed limitation “evaluates suitability of each of the plurality of image processing algorithms applied to the inspection image based on performance information representing inspection performance”};
[0154] The learning model 240 after the learning end performs at least one of detection, classification, or measurement as the image analysis, and the processing unit 22 (processor) can acquire the discontinuity information based on the result of the analysis through the learning model 240.
[Correction of Analysis Result]
[0155] The processing unit 22 may correct the result of the analysis by the learning model 240 according to the operation of the user via the operation unit 14 or the like. This correction may be, for example, addition of the discontinuity information, deletion of unnecessary information, amendment of an error, grouping of information, or the like.--, in [0153]-[0155]; also see SUDA: e.g., -- an AI (artificial intelligence) model and an inspection device which corrects an image by the AI model where the accuracy of inspection by correcting noise removal and resolution enhancement may be improved by the AI model generated by learning using AI.--, in abstract, and, The correction method according to the present embodiment is a machine such as deep learning using an image (transmission image or cross-sectional image) of the object to be inspected captured by the inspection device 1 as teacher data and learning data in advance. It is configured to perform learning (referred to as "AI learning" in the following description) to generate an AI model, and in the inspection, the generated AI model is used to correct a transmission image or a cross-sectional image. First, the process of AI learning (deep learning) will be described with reference to FIGS. 6 to 9. In the following description, the "teacher image" is an image of an object to be inspected with a sufficient exposure time and radiation quality (brightness and hardness of radiation) with little noise, or a sufficient resolution. It shows an image of the object to be inspected (resolution that can obtain a predetermined inspection accuracy in the inspection), and the "learning image" is the exposure time, radiation quality, or radiation quality set in the actual inspection. It shall indicate an image of the subject to be inspected, captured at resolution, containing noise, or at low resolution…. Further, in the AI learning process, as shown in FIG. 7, 3/4 of the entire image area Ra is used for learning, and the remaining 1/4 area is verified (performance evaluation). Used for. The arrangement of the verification area and the learning area shown in FIG. 7 is an example, and any partial area Rs can be assigned for verification or learning. Further, the ratio of the verification area and the learning area in one image is not limited to the combination of 3/4 and 1/4, and can be appropriately set. --, in page 8/15 of the English version of JP-2022098590-A, as provided with the Office Action; also see: -- In AI learning, a learning image (which may be a transparent image or a cross-sectional image) is input by a method such as deep learning, and a teacher's image (similar to the learning image, a transparent image and a cross-sectional image). Train the AI model so that the image is close to (may be). Then, the learning processing unit 37 updates the AI model based on the result of AI learning (step S216). Further, the learning processing unit 37 may be configured to display the learning progress status on the monitor 12, and if the learning progress status is displayed, the display is updated (step S218). Progress includes performance evaluation of AI models.
The performance evaluation of the AI model is performed by calculating the loss. The AI model trained by AI learning is applied to each image (input image) for learning and verification, and a corrected image (high-quality output image) is generated. The loss is calculated by taking the difference between the evaluation values of the generated output image and the teacher's image (for example, the mean square error of the luminance value). The loss here is the quality at which the difference between the teacher image and the image corrected by the AI model is lost or impaired (or irrecoverable) --, in page 10/15 of the English version of JP-2022098590-A, as provided with the Office Action; and, --whether the correction target of the AI model is a transparent image or a cross sectional image, noise is removed from the transparent image or the cross-sectional image or the resolution is increased, the synogram is interpolated, or the subject is inspected. Whether to learn the image of a specific area (board surface or specific component), create an AI model for each FOV, create an AI model common to FOV, or create a teacher image and a learning image. The learning conditions such as the image quality and the learning end condition can be set for the control unit 10 from the user interface including the monitor 12 and the keyboard / mouse. In addition, the learning status can be confirmed and the learning process can be completed via this user interface. When the AI model obtained as described above is applied to an inspection, the AI model described above is applied to the acquired transmission image or reconstructed image (cross-sectional image) in step S120 shown in FIG. 3 for correction. (Correct noise, increase resolution, interpolate synogram, etc.). Hereinafter, the process of correcting the transmission image or the reconstructed image (cross-sectional image) by the AI model will be described with reference to FIG. FIG. 10A shows a process when the correction by the AI model is applied to the transparent image. The control unit 10 captures a transmitted image of the object to be inspected under the imaging conditions for inspection (step S121a), and corrects the captured transmitted image using the AI model (step S122a).--, in in page 12/15 of the English version of JP-2022098590-A, as provided with the Office Action).
Re Claims 7-8, TAIRA as modified by SUDA further disclose wherein the image processing algorithm setting unit extracts at least one image processing algorithm candidate according to evaluation values of the plurality of image processing algorithms based on the learning model in the image processing algorithm evaluation unit, and in a case where the extracted image processing algorithm candidates are plural, the extracted image processing algorithm candidates are displayed on a display device in a selectable manner (see TAIRA: e.g., -- [0029] According to a twenty-first aspect, in the information processing apparatus described in any one of the first to twentieth aspects, the processor is configured to output the recorded information and/or information extracted from the recorded information to a recording device and/or a display device. The extracted information may be displayed by being associated with or superimposed on the image of the subject, a design drawing, or the like.
[0030] According to a twenty-second aspect, in the information processing apparatus described in the twenty-first aspect, the processor is configured to extract information from the recorded information under a condition designated by an operation of a user, and output the extracted information. According to the twenty-first aspect, the user can extract and refer to the information according to a desired condition.--, in [0029]-[0030]; and, -- [0067] FIG. 2 is a block diagram showing an example of a function of the processing unit 22 (processor, information processing apparatus). As shown in FIG. 2, the processing unit 22 comprises an extraction section 220, a measurement section 222, and a classification section 224 and acquires time-series defect information and discontinuity information from the image of the inspection target object (subject). In addition, the processing unit 22 comprises a condition calculation section 226, a recording control section 228, an information extraction section 230, and an output control section 232, and records or outputs the discontinuity information. As will be described below, the processing unit 22 may detect, classify, and measure the discontinuity by using an image analyzer (learning model) constructed by a machine learning algorithm (refer to the section of “Acquisition of Discontinuity Information Using Image Analyzer”)
[Extraction Section]
[0068] The extraction section 220 functions as an image processing unit and performs image processing (for example, color conversion processing, monochrome conversion processing, edge enhancement processing, conversion processing into three-dimensional data, and the like) on the captured image data to detect changes in color, brightness value, or the like of the inspection target object OBJ, thereby detecting discontinuities (for example, scratches, fissuring (cracks), wear, rust, and the like) of the inspection target object OBJ. The extraction section 220 analyzes the image with the first criterion and the second criterion, which will be described below, to detect the defect and the discontinuity based on, for example, a color change, an edge detection result, and the like. As a result, the position and the shape of the discontinuity are specified.
[0069] The extraction section 220 may detect the discontinuity by, for example, incorporating, into the product data D200, product image data including an image of a product (new product) in which the same defect of the inspection target object OBJ is not detected, and comparing the product image data with the captured image data of the inspection target object OBJ.--, in [0067]-[0069]).
Re Claims 9-10, TAIRA as modified by SUDA further disclose a learning unit that has product group storage means for storing images of a product group with which an optimal image processing algorithm among the plurality of image processing algorithms is associated, as learning data, and that generates the learning model based on the learning data (see TAIRA: e.g., --[0074] The classification section 224 assigns a discontinuity classification to the extracted discontinuity based on classification information stored in the storage unit 24. For example, the classification section 224 calculates a degree of similarity between the discontinuity extracted by the processing unit 22 and at least one of a discontinuity image corresponding to a discontinuity extracted in the past or information indicating the feature of the discontinuity image, and the classification section 224 assigns the discontinuity classification based on the degree of similarity. Here, the degree of similarity calculated by the classification section 224 is calculated by a known method. For example, the classification section 224 can calculate the degree of similarity by performing block matching between a discontinuity image extracted by the processing unit 22 and the discontinuity image extracted in the past. In addition, for example, the classification section 224 can calculate a degree of similarity between an extracted discontinuity candidate image and the discontinuity image (or the information indicating the feature of the discontinuity image) stored in the storage unit 24, and can assign a classification result assigned to the discontinuity image having the highest degree of similarity as the discontinuity classification in the discontinuity candidate image.--, in [0074]).
Re Claims 11-12, TAIRA as modified by SUDA further disclose wherein the learning model is created using a deep learning classification method (see SUDA: e.g., -- an AI (artificial intelligence) model and an inspection device which corrects an image by the AI model where the accuracy of inspection by correcting noise removal and resolution enhancement may be improved by the AI model generated by learning using AI.--, in abstract, and, The correction method according to the present embodiment is a machine such as deep learning using an image (transmission image or cross-sectional image) of the object to be inspected captured by the inspection device 1 as teacher data and learning data in advance. It is configured to perform learning (referred to as "AI learning" in the following description) to generate an AI model, and in the inspection, the generated AI model is used to correct a transmission image or a cross-sectional image. First, the process of AI learning (deep learning) will be described with reference to FIGS. 6 to 9. In the following description, the "teacher image" is an image of an object to be inspected with a sufficient exposure time and radiation quality (brightness and hardness of radiation) with little noise, or a sufficient resolution. It shows an image of the object to be inspected (resolution that can obtain a predetermined inspection accuracy in the inspection), and the "learning image" is the exposure time, radiation quality, or radiation quality set in the actual inspection. It shall indicate an image of the subject to be inspected, captured at resolution, containing noise, or at low resolution…. Further, in the AI learning process, as shown in FIG. 7, 3/4 of the entire image area Ra is used for learning, and the remaining 1/4 area is verified (performance evaluation). Used for. The arrangement of the verification area and the learning area shown in FIG. 7 is an example, and any partial area Rs can be assigned for verification or learning. Further, the ratio of the verification area and the learning area in one image is not limited to the combination of 3/4 and 1/4, and can be appropriately set. --, in page 8/15 of the English version of JP-2022098590-A, as provided with the Office Action; and, -- In AI learning, a learning image (which may be a transparent image or a cross-sectional image) is input by a method such as deep learning, and a teacher's image (similar to the learning image, a transparent image and a cross-sectional image). Train the AI model so that the image is close to (may be). Then, the learning processing unit 37 updates the AI model based on the result of AI learning (step S216). Further, the learning processing unit 37 may be configured to display the learning progress status on the monitor 12, and if the learning progress status is displayed, the display is updated (step S218). Progress includes performance evaluation of AI models.
The performance evaluation of the AI model is performed by calculating the loss. The AI model trained by AI learning is applied to each image (input image) for learning and verification, and a corrected image (high-quality output image) is generated. The loss is calculated by taking the difference between the evaluation values of the generated output image and the teacher's image (for example, the mean square error of the luminance value). The loss here is the quality at which the difference between the teacher image and the image corrected by the AI model is lost or impaired (or irrecoverable) --, in page 10/15 of the English version of JP-2022098590-A, as provided with the Office Action; and, --whether the correction target of the AI model is a transparent image or a cross sectional image, noise is removed from the transparent image or the cross-sectional image or the resolution is increased, the synogram is interpolated, or the subject is inspected. Whether to learn the image of a specific area (board surface or specific component), create an AI model for each FOV, create an AI model common to FOV, or create a teacher image and a learning image. The learning conditions such as the image quality and the learning end condition can be set for the control unit 10 from the user interface including the monitor 12 and the keyboard / mouse. In addition, the learning status can be confirmed and the learning process can be completed via this user interface. When the AI model obtained as described above is applied to an inspection, the AI model described above is applied to the acquired transmission image or reconstructed image (cross-sectional image) in step S120 shown in FIG. 3 for correction. (Correct noise, increase resolution, interpolate synogram, etc.). Hereinafter, the process of correcting the transmission image or the reconstructed image (cross-sectional image) by the AI model will be described with reference to FIG. FIG. 10A shows a process when the correction by the AI model is applied to the transparent image. The control unit 10 captures a transmitted image of the object to be inspected under the imaging conditions for inspection (step S121a), and corrects the captured transmitted image using the AI model (step S122a).--, in in page 12/15 of the English version of JP-2022098590-A, as provided with the Office Action).
Re Claims 13-14, TAIRA as modified by SUDA further disclose wherein the quality state of the inspection object article is inspected by irradiating the inspection object article with X-rays to acquire an X-ray inspection image and by applying the predetermined image processing algorithm to the X-ray inspection image (see TAIRA: e.g., --[0085] The captured image data is image data (for example, an X-ray image (radiation transmission image), a visible light image, or an infrared light image) obtained by imaging the inspection target object OBJ.
[0086] The imaging condition data is stored for each piece of the captured image data of the inspection target object OBJ and includes information indicating an imaging date and time, an imaging target location of each piece of the captured image data, a distance between the inspection target object OBJ and the camera during imaging, and an angle with respect to the camera.
[0087] The illumination condition data includes information indicating a type of radiation or light used for imaging of the inspection target object OBJ (for example, X-rays, visible light rays, infrared light rays, transmitted light rays, reflected light rays), an irradiation intensity, and an irradiation angle.--, in [0085]-[0087]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEIWEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on Monday-Friday 8:30am-4:30pm east.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
Prosecution Insights