DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see page 1, filed 2/2/2026, with respect to specification objection have been fully considered and are persuasive. The objection of the specification has been withdrawn.
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on all references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. The remarks state that the applied references do not perform the feature of the amendment. The reference of Fiebag cures the deficiency of the previously applied references and will be briefly discussed below.
Regarding the Fiebag reference, synthetic images can be created in order to improve the training of a machine learning algorithm and the classification of a detected defect on a circuit board, which is taught in ¶ [56] of Fiebag. This is created in addition to the training of the Machine learning algorithm in order to train the model for classification of the defect of the PCB, which is taught in ¶ [46]-[49]. This is an example of training a machine learning algorithm with different types of data. This feature, in combination with the previously applied references, performs the features of the claims.
Thus, based on the above, the features of the claims are disclosed below.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: imaging device, inspection module, defect detection module and processing device in claims 1-5.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, 4, 5, 7-12, 14-18 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gurumurthy (US Pub 2023/0125477) in view of Hida (US Pub 2021/0374928) and Fiebag (US Pub 2025/0272816).
Re claim 1: Gurumurthy discloses a printed circuit board (PCB) inspection system comprising:
an imaging device (interpretation: The imaging device 102 may include any type of camera or imaging sensor that can acquire images of PCBs and provide the images to a computer-readable storage device (such as system storage 612 discussed below with reference to FIG. 6). In some examples, the imaging device 102 is a camera operating in the visual spectrum. In other examples, the imaging device 102 is a camera operating in another region of the electromagnetic spectrum, such as the infrared, ultraviolet, or x-ray bands, for example. In other examples, the imaging device may be a scanning electron microscope. In some examples, the imaging device 102 acquires images of the PCBs under inspection, or regions of the PCBs under inspection, as the PCBs pass the field of view of imaging device 102 on a movable platform, such as a conveyor, which is taught on page 5, ¶ [20]. This interpretation and its equivalents are utilized for this claim term hereinafter in the Office Action.) configured to capture an image of a PCB under inspection, the PCB under inspection including at least one feature of interest (e.g. a camera is used to capture images of a PCB in order to detect features, such as defects. This is taught in ¶ [63], [66] and [67].);
[0063] In at least one embodiment, one or more trained neural networks 114 perform inferencing 110 using video 112 (e.g., images, digital photos, bit streams) containing information about a manufactured object (e.g., PCB) captured by a camera (e.g., microscope, x-ray sensor, CCD image sensor, CMOS image sensor, NMOS image sensor, infrared sensor, webcam). In at least one embodiment, video 112 comprises a bit stream of an image stream output by a camera (e.g., digital microscope, digital camera, digital sensor). In at least one embodiment, one or more trained neural networks 114 perform inferencing directly on a bit stream of image data prior to decoding said bit stream. In at least one embodiment, one or more trained neural networks 114 perform inferencing on a bit stream of image data after decoding said bit stream. In at least one embodiment, one or more trained neural networks 114 take images of a PCB and identify defects (e.g., solder bridging, open solder joints, component shifts, scratches) in a PCB. In at least one embodiment, video 112 comprises images, such as a PCB image. In at least one embodiment video 112 comprises bit streams of image data, such as bit streams representing images of a PCB. In at least one embodiment, video 112 is input data used by trained neural network 114. In at least one embodiment, one or more trained neural networks 114 perform inferencing 110 by using an image, such as a PCB image, to detect defects in a PCB identified by said one or more trained neural networks 114. In at least one embodiment, one or more trained neural networks 114 perform inferencing 110 by using an image, such as a PCB image, to count components on a PCB identified by said one or more trained neural networks 114. In at least one embodiment, one or more trained neural networks 114 perform inferencing to count components on a PCB comprising hundreds of components of many types including capacitors, resistors, and integrated circuits, and of sizes of 0.6 mm×0.3 mm and smaller. In at least one embodiment, input data such as an image or a series of images of a manufactured object (e.g., PCB) comprises an image containing information about components on a PCB, such as label, shape, location, or color information. In at least one embodiment, input data comprises any type of data on which one or more trained neural networks 114 are trained 102 to operate by training framework 106.
[0066] FIG. 2 depicts AOI system 200 for detecting defects on PCBs according to at least one embodiment. In at least one embodiment, AOI system 200 comprises one or more circuits to use one or more neural networks to detect one or more defects in one or more manufactured objects based, at least in part, on one or more combined images of one or more manufactured objects. In at least one embodiment, one or more combined images are generated based, at least in part, on multiple images from multiple video streams. In at least one embodiment, a system 200 comprises components to analyze (e.g., inspect) a manufactured object 206, such components including a single-board computer 202 (e.g., embedded computing board, portable artificial intelligence (AI) computer, NVIDIA® Jetson Nano™), one or more scopes 204, a manufactured object (e.g., PCB) 206 with a feature 207 (e.g., defect, component, label), display 208, a captured image 210 (e.g., a visual representation of manufactured object 206), and a sample object 212 (e.g., ideal manufactured object free of defects). In at least one embodiment, AOI system 200 comprises one or more neural networks to perform inferencing at the edge (e.g., performing inferencing on a low-power single board computer) for real-time detection of defects on a PCB. In at least one embodiment, AOI system 200 comprises one or more neural networks to perform a combination of cloud inferencing and inferencing on the edge.
[0067] In at least one embodiment, single-board computer 202 comprises one or more machine learning models such as deep learning models (e.g., neural networks (NNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs)). In at least one embodiment, single-board computer 202 runs multiple machine learning models in parallel. In at least one embodiment, single-board computer 202 comprises a graphics processing unit (GPU), a central processing unit (CPU), a hardware AI (e.g., deep learning) accelerator, and memory (e.g., random access memory (RAM), read-only memory (ROM), embedded multimedia card (eMMC), low-power double data rate (LPDDR)). In at least one embodiment, a single-board computer comprises a vision-based machine learning framework (e.g., machine learning model) that takes as input streaming data (e.g., live images, live video) captured from a camera comprising a scope. In at least one embodiment, single-board computer 202 comprises a machine learning framework for accelerating inferences made by neural networks. In at least one embodiment, a single-board computer comprises an operating system (e.g., Linux®). In at least one embodiment, one or more machine learning models label live videos frame by frame based on one or more detected features 207 and converts labels into input data used for training other machine learning models to detect defects in a manufactured object, wherein one or more machine learning models may comprise an object detection model. In at least one embodiment, training one or more neural networks to detect defects in a manufactured object comprises two video streams of a manufactured object and one video stream of an ideal manufactured object for validation. In at least one embodiment, one or more neural networks may comprise an object detection model that may be a single-stage, two-stage model, or some combination thereof or another type of model. In at least one embodiment, an object detection model may comprise a single-stage object detection model (e.g., one-stage objection detection model) such as RetinaNet (by Facebook AI Research), You Only Look Once (YOLO) [J. Redmon et. al., “You Only Look Once: Unified, Real-Time Object Detection,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788.], Single Shot MultiBox Detector (SSD) [Liu W. et al., “SSD: Single Shot MultiBox Detector,” Computer Vision—ECCV 2016, 2016.], or some combination thereof. In at least one embodiment, one or more neural networks comprise a single-stage object detection model and a base or compound-scaled model of EfficientNet (by Google) (e.g., EfficientNet scaled to different block layers, EfficientNet B1-B7) to pretrain a neural network to detect defects or count components on a PCB. In at least one embodiment, use of EfficientNet-B4 provides optimum levels of accuracy and computational efficiency for AOI system 200. In at least one embodiment, one or more neural networks may comprise a two-stage object detection model (e.g., Fast Region-based Convolutional Network (Fast R-CNN) (by Microsoft), Mask RCNN (by Facebook AI Research)). In at least one embodiment, a machine learning model performs deep learning inference on multiple video streams, for example, a machine learning model may comprise a deep learning model that performs deep learning inferencing and video decoding on six to eight video streams in parallel.
the processing device (interpretation: The processing device comprises an inspection module including an autoencoder and a defect detection module. The autoencoder is trained to recognize the at least one feature of interest based on a combination of model data and training data. Once trained, the autoencoder is configured to receive and process the test image to produce a reconstructed image. The defect detection module is configured to compare the reconstructed image to the test image to generate a reconstruction error value, and to produce an output indicating detection of a defect in the feature of interest under inspection based on the reconstruction error value transgressing a threshold value, which is taught in ¶ [16]. This interpretation and its equivalents are utilized for this claim term hereinafter in the Office Action.) comprising
an inspection module (interpretation: The autoencoder 120 includes an encoder 402 that maps input data 404 to a code 406, and a decoder 408 that reconstructs the input data 404 from the code 406. An optimal autoencoder would perform as close to perfect construction of the input data 404 as possible, with performance of the autoencoder being evaluated according to a reconstruction quality function. The autoencoder 120 operates in the training mode, in which the encoder 402 learns an encoding function and the decoder 408 learns a decoding function, thereby constructing the code 406. Once trained, the autoencoder 120 operates in the inference mode in which the autoencoder 120 applies the code 406 to the test images 104. For each processed test image 104, the autoencoder 120 produces a reconstruction 410 of that test image based on the code 406. In the training mode, the input data 404 includes the training data 132 and the model data 134, and in the inference mode, the input data 404 includes the test images 104, which is taught in ¶ [27]. This interpretation and its equivalents are utilized for this claim term hereinafter in the Office Action.) trained to recognize the at least one feature of interest based on a combination of model data and training data, the autoencoder configured to receive and process the image to produce a reconstructed image (e.g. the system discloses analyzing a sample object, which is a manufactured object free of defects considered as model data. In addition, an image with defects is analyzed in a manufactured object considered as training data, which is taught in ¶ [66] above. ¶ [56] discloses a neural network that is trained to recognize features within an image. An autoencoder is used to receive an image and produce a reconstructed image, which is taught in ¶ [59].), and
[0056] FIG. 1 is a block diagram illustrating training architecture 100 for training 102 a neural network 108, 114 to infer 110 from input video various types of information such as features of a manufactured product or a number of components present on a manufactured product, according to at least one embodiment. In at least one embodiment, training data 104 (e.g., training dataset) based on images (e.g., live video, recorded video, photos, image streams, digital images) is used as input by training framework 106 to train 102 one or more neural networks 108 using machine learning models (e.g., artificial neural networks, neural networks, deep learning models based on convolutional neural networks, Siamese convolutional neural networks). In at least one embodiment, training data 104 is a set of data based on images to provide a set of examples on which one or more neural networks 108 learns to perform a function, such as detecting defects on a PCB.
[0057] In at least one embodiment, training data 104 is a set of data, such as video (e.g., digital video, analog video, streaming video, electronic representation of moving visual images) on which one or more neural networks 108 are to be trained to operate on. In at least one embodiment, training data 104 comprises video of a manufactured object (e.g., manufactured product, PCB, mobile phone, liquid crystal display panel, silicon wafer, packaged food product). In at least one embodiment, training data 104 comprises a set of images with labels or classifications, which may further comprise characteristics or features found on PCBs (e.g., solder welds, capacitors, resistors, transistors, integrated circuits, traces, etchings, holes, scratches, layers, alphanumeric markings, input/output connectors, power connectors, colors, reflections). In at least one embodiment, training data 104 comprises images that provide information about locations of characteristics found on PCBs. In at least one embodiment, training data 104 is one or more other types of data for which one or more neural networks 108 are trained 102 by training framework 106 to perform operations such as counting components on a PCB.
[0058] In at least one embodiment, training framework 106 is a set of software instructions that, when executed on one or more computing devices, govern training 102 of one or more untrained neural networks 108 using training data 104, such as videos of PCBs described above. In at least one embodiment, training framework 106 trains one or more neural networks to facilitate learning by one or more neural networks 108 based on training data 104. In at least one embodiment, training framework 106 trains one or more neural networks using deep learning.
[0059] In at least one embodiment, training framework 106 trains one or more neural networks 108 without supervision (e.g., unsupervised training of neural networks). In at least one embodiment, training framework 106 trains one or more neural networks 108 without supervision and uses only training data 104. In at least one embodiment, training of one or more neural networks 108 without supervision comprises training data 104 without labels (e.g., scores, classifications). In at least one embodiment, unsupervised training comprises one or more circuits of one or more processors using one or more neural networks 108 to automatically find structure in training data 104 by using a type of machine learning method such as clustering (e.g., discovering similar groups or classifications within data), anomaly detection (e.g., defining a norm in data and identifying data abnormalities), association (e.g., discovering relationships between variables or features in data), autoencoder (e.g., encoding input data with target values and attempting to regenerate said input data with backpropagation), or some combination thereof. In at least one embodiment, one or more circuits of one or more processors using one or more neural networks 108 organize training data 104. In at least one embodiment, training framework 106 trains one or more neural networks 108 using any available supervision in conjunction with training data 104.
a defect detection module (interpretation: In some examples, the defect detection module 112 compares the current test image 104 with the corresponding reconstructed image 410. As discussed above, where the test image 104 contains no defects in the features of interest, assuming the autoencoder 120 has been well trained, the reconstructed image 410 will be a very accurate/close reconstruction of the original image. However, when there is a defect in one or more features of interest, the reconstruction quality worsens. Accordingly, the defect detection module 112 may produce a reconstruction error value based on the comparison between the current test image 104 and the corresponding reconstructed image 410. If the reconstruction error value is above or below a certain threshold value, the defect detection module may provide an output indicating that a defect has been detected, which is taught in ¶ [37] on page 11. This interpretation and its equivalents are utilized for this claim term hereinafter in the Office Action.) configured to compare the reconstructed image to the image to generate a reconstruction error value, and to produce an output indicating detection of a defect in the feature of interest based on the reconstruction error value transgressing a threshold value (e.g. the combined or reconstructed image is generated as an output in order to be compared to an ideal image. The comparison yields a number that is compared to a threshold to determine if the comparison between images has a value above or below a threshold. This is taught in ¶ [78]-[81]. The features of the defects are displayed with a bounding box around the features, which is explained in ¶ [81].).
[0078] In at least one embodiment of a neural network architecture 300, one or more circuits are to perform one or more convolution operations to generate multiple images to be combined to generate one or more combined images. In at least one embodiment, combined images are generated based, at least in part, on images of a top side of one or more manufactured objects and images of a bottom side of one or more manufactured objects. In at least one embodiment, an image may not have a corresponding display image that visually resembles an object said image represents. In at least one embodiment, one or more combined images comprise an image generated based, at least in part, on a first output of one or more convolution operations, a second output of one or more convolution operations, a first output of one or more fully connected layers (e.g., embedded layers), a second output of one or more convolution operations, and a second output of one or more fully connected layers. In at least one embodiment, one or more circuits are to obtain output of one or more fully connected layers of one or more neural networks to generate one or more combined images before providing one or more combined images as input to a loss function 314 (e.g., contrastive loss function). In at least one embodiment, a loss function 314 compares combined images of a manufactured object with one or more combined images of a sample object.
[0079] In at least one embodiment, a neural network comprises a feed forward network further comprising convolutional layers 306, 308, fully connected layers 310, 312, and a loss function 314 (e.g., contrastive loss function). As depicted in FIG. 3, according to at least one embodiment, convolutional layers comprise 306a-306N, 308a-308N, wherein a letter N represents a number of convolutional layers in a neural network. In at least one embodiment, a neural network architecture 300 uses one more circuits of one or more processors to perform convolutions on image data with convolutional layers 306a-306N and as further discussed in FIG. 4. In at least one embodiment, fully connected layers comprise 310a-310N, 312a-312N, wherein a letter N represents several fully connected layers in a neural network. In at least one embodiment, a neural network architecture 300 uses one more circuits of one or more processors to input image data into fully connected layers 310a-310N, 312a-312N and as further discussed in FIG. 4. In at least one embodiment, a neural network may comprise any neural network suitable for comparing two or more images. In at least one embodiment, one or more circuits are to use a loss function 314 to compare one or more combined images of a manufactured object with one or more other images of a sample object. In at least one embodiment, one or more circuits of one or more processors use a neural network architecture 300 to perform feature extraction on image data by using convolutional layers and to perform classification on image data by using fully connected layers.
[0080] In at least one embodiment, manufactured object images 302 comprise image or video data from one or more images or video captured by one or more cameras, which further comprises one or more scopes. In at least one embodiment, multiple images of one manufactured object are generated based, at least in part, on multiple images of different lighting conditions or different viewing angles. In at least one embodiment, manufactured object images 302 comprise two video streams of a single manufactured object, wherein one video stream captures a first set of features due to a first set of image capturing parameters (e.g., camera angle, camera position, lighting conditions, type of scope, type of sensor) and a second video stream captures a second set of features due to a second set of image capturing parameters, for example, manufactured object images 302 may comprise a first video stream of a soldering joint or component on a PCB under white light positioned from above and a second video stream under a combination of red, green, and blue (RGB) light positioned from a side, which a neural network uses to infer a set of defects of a weld that a single video stream may not fully reveal.
[0081] In at least one embodiment, a loss function 314 comprises one or more loss functions (e.g., contrastive loss, triplet loss, regression loss, mean squared error loss, mean squared logarithmic error loss, log loss, cross entropy loss, likelihood loss, ranking loss, margin loss, hinge loss). In at least one embodiment, a neural network architecture 300 comprises one or more neural networks that perform a contrastive loss function that, at least in part, calculates a distance between a pair of representations of images and produces an output 322 based, at least in part, on said distance. In at least one embodiment, one or more neural networks use a loss function 314 to compare an output of a) one neural network applied to images of a sample object to the b) output of another neural network applied to images of a manufactured object. In at least one embodiment, neural network architecture 300 comprises a combination of more than one type of loss function. In at least one embodiment, one or more circuits of one or more processors are used to train a neural network architecture 300 to predict a distance between two inputs. In at least one embodiment, one or more circuits of one or more processors of network architecture 300 uses one or more neural networks to infer a count of components or defects on a manufactured object based, at least in part, on using a loss function 314 to compare distances between features extracted from images of a manufactured object and features extracted from a sample object such as a golden sample of a manufactured object, for example, one or more neural networks counts a component in a manufactured object if one or more features respresenting said component match (e.g., calculate a distance less than a threshold value for a distance) one or more features representing said component in a sample object. In at least one embodiment, one or more neural networks of a network architecture 300 identifies a defect if one or more features on a manufactured object do not match (e.g., calculate a distance greater than a threshold value for a distance) with features on a sample object. In at least one embodiment, one or more neural networks of a network architecture 300 identifies features on a manufactured object by drawing and displaying bounding boxes around said features on display 208.
However, Gurumurthy fails to specifically teach the features of a data bus coupled to the imaging device and configured to transfer the image; and
a processing device coupled to the data bus.
However, this is well known in the art as evidenced by Gurumurthy. Similar to the primary reference, Gurumurthy discloses using a data bus and processor with an autonomous vehicle (same field of endeavor or reasonably pertinent to the problem).
Gurumurthy discloses a data bus coupled to the imaging device and configured to transfer the image (e.g. in another embodiment, a data bus is used to connect sensors or cameras and send sensor data from the camera to a controller, which is taught in ¶ [138], [152] and [153].); and
[0138] In at least one embodiment, controller(s) 1036 provide signals for controlling one or more components and/or systems of vehicle 1000 in response to sensor data received from one or more sensors (e.g., sensor inputs). In at least one embodiment, sensor data may be received from, for example and without limitation, global navigation satellite systems (“GNSS”) sensor(s) 1058 (e.g., Global Positioning System sensor(s)), RADAR sensor(s) 1060, ultrasonic sensor(s) 1062, LIDAR sensor(s) 1064, inertial measurement unit (“IMU”) sensor(s) 1066 (e.g., accelerometer(s), gyroscope(s), a magnetic compass or magnetic compasses, magnetometer(s), etc.), microphone(s) 1096, stereo camera(s) 1068, wide-view camera(s) 1070 (e.g., fisheye cameras), infrared camera(s) 1072, surround camera(s) 1074 (e.g., 360 degree cameras), long-range cameras (not shown in FIG. 10A), mid-range camera(s) (not shown in FIG. 10A), speed sensor(s) 1044 (e.g., for measuring speed of vehicle 1000), vibration sensor(s) 1042, steering sensor(s) 1040, brake sensor(s) (e.g., as part of brake sensor system 1046), and/or other sensor types. In at least one embodiment, stereo camera(s) 1068 capture two or more images of one object, convolve said images using trained neural network 808 and then combine said images.
[0152] FIG. 10C is a block diagram illustrating an example system architecture for autonomous vehicle 1000 of FIG. 10A, according to at least one embodiment. In at least one embodiment, system architecture for autonomous vehicle 1000 of FIG. 10A comprises stereo camera(s) 1068 that can capture two or more images of one object, convolve said images using trained neural network 808 and then combine said images. In at least one embodiment, each of components, features, and systems of vehicle 1000 in FIG. 10C is illustrated as being connected via a bus 1002. In at least one embodiment, bus 1002 may include, without limitation, a CAN data interface (alternatively referred to herein as a “CAN bus”). In at least one embodiment, a CAN may be a network inside vehicle 1000 used to aid in control of various features and functionality of vehicle 1000, such as actuation of brakes, acceleration, braking, steering, windshield wipers, etc. In at least one embodiment, bus 1002 may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). In at least one embodiment, bus 1002 may be read to find steering wheel angle, ground speed, engine revolutions per minute (“RPMs”), button positions, and/or other vehicle status indicators. In at least one embodiment, bus 1002 may be a CAN bus that is ASIL B compliant.
[0153] In at least one embodiment, in addition to, or alternatively from CAN, FlexRay and/or Ethernet protocols may be used. In at least one embodiment, there may be any number of busses forming bus 1002, which may include, without limitation, zero or more CAN busses, zero or more FlexRay busses, zero or more Ethernet busses, and/or zero or more other types of busses using different protocols. In at least one embodiment, two or more busses may be used to perform different functions, and/or may be used for redundancy. For example, a first bus may be used for collision avoidance functionality and a second bus may be used for actuation control. In at least one embodiment, each bus of bus 1002 may communicate with any of components of vehicle 1000, and two or more busses of bus 1002 may communicate with corresponding components. In at least one embodiment, each of any number of system(s) on chip(s) (“SoC(s)”) 1004 (such as SoC 1004(A) and SoC 1004(B)), each of controller(s) 1036, and/or each computer within vehicle may have access to same input data (e.g., inputs from sensors of vehicle 1000), and may be connected to a common bus, such CAN bus.
a processing device coupled to the data bus (e.g. the different CPUs, GPUs and Socs are connected to the data bus, which is seen in figure 10C. The inference and/or training logic are also a part of these processors, which the processors are disclosed in ¶ [195] and [196].).
[0195] In at least one embodiment, vehicle 1000 may include CPU(s) 1018 (e.g., discrete CPU(s), or dCPU(s)), that may be coupled to SoC(s) 1004 via a high-speed interconnect (e.g., PCIe). In at least one embodiment, CPU(s) 1018 may include an X86 processor, for example. CPU(s) 1018 may be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and SoC(s) 1004, and/or monitoring status and health of controller(s) 1036 and/or an infotainment system on a chip (“infotainment SoC”) 1030, for example. In at least one embodiment, SoC(s) 1004 includes one or more interconnects, and an interconnect can include a peripheral component interconnect express (PCIe).
[0196] In at least one embodiment, vehicle 1000 may include GPU(s) 1020 (e.g., discrete GPU(s), or dGPU(s)), that may be coupled to SoC(s) 1004 via a high-speed interconnect (e.g., NVIDIA's NVLINK channel). In at least one embodiment, GPU(s) 1020 may provide additional artificial intelligence functionality, such as by executing redundant and/or different neural networks, and may be used to train and/or update neural networks based at least in part on input (e.g., sensor data) from sensors of a vehicle 1000.
Therefore, in view of Gurumurthy second embodiment, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of a data bus coupled to the imaging device and configured to transfer the image; and a processing device coupled to the data bus, incorporated in the device of Gurumurthy, in order to perform transfer a larger amount of data across a data bus, which accelerate the execution and efficiency of performed applications (as stated in Gurumurthy ¶ [241]).
However, the combination above fails to specifically teach the feature of an autoencoder trained to recognize the at least one feature of interest based on a combination of model data and training data.
However, this is well known in the art as evidenced by Hida. Similar to the primary reference, Hida discloses training an autoencoder (same field of endeavor or reasonably pertinent to the problem).
Hida discloses an autoencoder trained to recognize the at least one feature of interest based on a combination of model data and training data (e.g. an autoencoder is trained is trained to extract features from a defect free image. The autoencoder is trained on defect free images and images with defects, which is taught in ¶ [63]-[66].).
[0063] The defect image data is images of anomalies or defects that may be attached to the set of images of defect-free physical samples at S202. The defect image data may include zero, less than 10, or less than 20% (in area) image of defect-free physical samples (i.e. in some implementations defect image data may incorporate a small amount of defect-free image). The training data image manager 330 accesses the stored set of images of defect-free physical samples from the first storage location 310, and defect image data from the second storage location 320 (when required), to produce training data images for input to the generator neural network 340.
[0064] The training data image manager 330 may control one or more from among number, transparency, colour, size, and shape, of defect image data to attach to the parent images, in order to cause an amount of difference between the parent image and the training data image, which amount of difference is controlled according to a training condition such as number of completed training epochs.
[0065] The generator neural network 340 comprises an encoder 341 and a decoder 342. The generator neural network 340 may be an autoencoder comprising an encoder 341 and a decoder 342. Specifically, it may be a convolutional autoencoder. The convolutional autoencoder has a multilayer neural network architecture comprising one or more convolutional layers, followed by sub-sampling layers (pooling) and one or more fully connected layers. An example is a skip connection such as UNet (https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/).
[0066] The combination of the autoencoder 341 and decoder 342 realises the generator neural network 340. The decoder 342 effectively reverses the action of the autoencoder, since it takes a decoded representation of the input image and generates an image from the decoded representation. A role of the autoencoder is to extract features of images—in particular, the autoencoder is trained to extract the features of a defect-free image. The autoencoder restores input images and extracts features of input images. Skip connection such as UNet is an example of a functional part of an autoencoder.
Therefore, in view of Hida, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of an autoencoder trained to recognize the at least one feature of interest based on a combination of model data and training data, incorporated in the device of Gurumurthy, in order to train an autoencoder with a plurality off epochs with a maximum difference between images, which can improve the effectiveness of removing defects (as stated in Hida ¶ [37]).
However, the combination above fails to specifically teach the features of wherein the model data comprises synthetically created representations of PCB features.
However, this is well known in the art as evidenced by Fiebag. Similar to the primary reference, Fiebag discloses determining defects on a PCB board (same field of endeavor or reasonably pertinent to the problem).
Fiebag discloses wherein the model data comprises synthetically created representations of PCB features (e.g. the invention discloses training an algorithm using training data within a first phase, which is taught in ¶ [46]-[50]. In addition, in another phase, synthetic images of the PCB and regions of the circuit board can be used to improve the classification performance of the adaptive algorithm ML. This is taught in ¶ [54]-[56].).
[0046] For training or post-training of the adaptive algorithm ML images of circuit boards indicated as defective by the image processing OR by a first defect indicator F1 and for example the spatial information on the defects detected by the image processing OR are picked off and for example, as shown in FIG. 4, stored in a cloud, for example in the memory M shown in FIG. 4.
[0047] For this purpose, the inspection system Q may be connected to a gateway GW, the gateway producing a connection between the cloud and the inspection system Q. Moreover, the information from the manual diagnosis of the operator from the quality database DB may be stored in the memory M. For this purpose, the database DB may also be connected to the gateway GW. The diagnostic information, that for example is stored in the database DB, represents the knowledge of the operator O who was involved in the diagnosis of the circuit board(s) indicated as defective by the image recognition OR. This diagnostic information is used to label the images of the circuit board(s). In other words, the image captured of a circuit board and the spatial information about a defect determined by the image acquisition are linked to the diagnostic information and if appropriate stored in the memory M in the cloud. In the cloud the adaptive algorithm ML may then be trained on the basis of this data labeled with the diagnostic information. The images labeled with the diagnostic information then form the so-called “ground truth” for the adaptive algorithm. If appropriate, a preprocessing “data prep” of the received or stored images initially also takes place in the cloud. This preprocessing may for example include padding and augmentation of the images or image data, as is explained below.
[0048] On conclusion of the training, the adaptive algorithm may evaluate an image of a circuit board, for example using the “ground truth”—i.e., on the basis of images or image data labeled with diagnostic information. The diagnostic information may in this case be present in the form of a third defect indicator F3.
[0049] After the training the trained adaptive algorithm ML may be exported, cf. “Export” in FIG. 4. The trained adaptive algorithm ML may then be transferred to the inspection system Q. An adaptive algorithm ML already present in the inspection system Q may thus be updated by the newly trained adaptive algorithm ML. Consequently, the adaptive algorithm ML may be adjusted. For example, in the presence of a specified number of captured images and/or in the presence of a specified amount of diagnostic information it is possible to update the adaptive algorithm on the inspection system Q. The diagnostic information in this case for example corresponds to the third defect indicator, the value of which was established by an operator during the diagnosis.
[0050] FIG. 5 depicts a schematic representation of a first operational sequence of image processing for the training (Phase 1) and during of the use of an adaptive algorithm ML (Phase 2).
[0054] The labeling process may include two stages: the operator O performs a pre-labeling of the data and stores this information in the database DB. In addition, a review of the labeling carried out may be undertaken and the pre-labeling overwritten if appropriate. FIG. 5 depicts “Labeling review” by the “Labeling operator (database)”.
[0055] Because the individual images or image regions are of different sizes, in the next step it is necessary to bring the images or image regions to a uniform size. The different size of the images or image regions is physically conditioned, since the image regions contain different electronic components or test regions with one or more soldering points. For example, a uniform image size may be defined for processing. For example, a square-shaped size may be defined (for example 512 by 512 pixels may be selected). The uniform image size may be configurable, so that it is adjustable, for example to different test regions on a circuit board and thus different circuit board types. In order to achieve scaling to a uniform image size, an “up-down sampling” takes place for example and artificial pixels may be generated and artificial pixel values padded. This is known as padding. For example, if an image region has an original size of 200 by 600 pixels, artificial pixel values are generated and are appended to the image, in order for example to obtain a square-shaped size. The generated artificial pixel values may be randomly generated from a generated color distribution, for example permissible RGB values. The color distribution from which the pixel values for the artificial pixels are generated may for example be given by intervals in the RGB color space. The color distribution is in this case adjustable and should be in a similar color distribution to the basic color of the circuit boards. For example, an artificial pixel value may be selected, for example randomly, from an interval for the red channel, for example [37, 42], an interval for the green channel, for example [63, 73], and an interval for the blue channel, for example [36, 41]. The boundary values of the respective interval or color distribution in general may be determined using the minimum values and/or maximum values of captured images of different circuit boards, for example the aforementioned ground truth dataset. The generated color distribution or the color ranges from which the artificial pixel values for the padding are selected may correspond to the color values of the circuit boards from a dataset of images of the circuit board, for example from the circuit board manufacture in the facility.
[0056] Following on from the padding, an augmentation of the images or image region may take place. Alternatively, the augmentation may also take place first. The augmentation is a method to augment the original dataset of images in order to generate the maximum variance and to minimize the influence of noise and slight changes to new genuine images. The images or image regions are rotated, turned, mirrored randomly and/or individual RGB color channels are changed and/or the brightness of the images or image regions is varied for this purpose. This thus produces a plurality of new, synthetic images or image regions. This step is advantageous for the training of the adaptive algorithm and results in an improvement in the classification of the adaptive algorithm ML.
Therefore, in view of Fiebag, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the model data comprises synthetically created representations of PCB features, incorporated in the device of Gurumurthy, as modified by Hida, in order to include synthetic image in the training of an algorithm, which can improve the classification performance of the algorithm (as stated in Fiebag ¶ [56]).
Re claim 2: Gurumurthy discloses the PCB inspection system of claim 1, wherein the autoencoder includes:
a model-based process configured to operate on the model data to generate a first plurality of neural network nodes corresponding to a plurality of model-based features derived from the model data (e.g. nodes are generated in order to process features from the model input into the layers of the neural networks. The node containing the first layer is used to process the sample image data before passing the processed data to the fully connected layer, which is taught in ¶ [79] above and figure 3.); and
a machine learning process configured to operate on the training data to generate a second plurality of neural network nodes corresponding to a plurality of machine learning-based features derived from the training data (e.g. the invention discloses operating on data containing defects in order to determine the defects used in training the neural network. The images containing the defects are passed between generated nodes before the data is output, which is taught in ¶ [79]-[81] above.).
Re claim 4: Gurumurthy discloses the PCB inspection system of claim 2, wherein the autoencoder includes a plurality of network layers, the first and second pluralities of neural network nodes being distributed among the plurality of layers (e.g. plurality of nodes are distributed throughout the neural network. There are various layers within the nodes within the system, which is taught in ¶ [79] above.).
Re claim 5: Gurumurthy discloses the PCB inspection system of claim 1, wherein the defect detection module is configured to produce the output indicating the detection of the defect based on the reconstruction error value exceeding the threshold value (e.g. the system determines a defect is detected based on the exceeding of a threshold by reconstructed images that are compared to create a value. The value is compared to the threshold, which is taught in ¶ [78]-[81] above.).
Re claim 7: However, Gurumurthy 1st embodiment fails to specifically teach the features of the PCB inspection system of claim 1, further comprising: at least one processor-readable storage medium coupled to the data bus and configured to store one or more of the training data, the model data, or the image.
However, this is well known in the art as evidenced by Gurumurthy 2nd embodiment. Similar to the primary reference, Gurumurthy discloses using a data bus and processor with an autonomous vehicle (same field of endeavor or reasonably pertinent to the problem).
Gurumurthy 2nd embodiment discloses further comprising: at least one processor-readable storage medium coupled to the data bus and configured to store one or more of the training data, the model data, or the image (e.g. the inference and/or training logic code is stored within a data storage inside a training logic hardware structure, which is taught in ¶ [99]-[101]. The hardware can be internal or external to the CPU, which is taught in ¶ [108], [109] and [179], shown in figure 10C.).
[0099] FIG. 7A illustrates inference and/or training logic 715 used to perform inferencing and/or training operations associated with one or more embodiments. Details regarding inference and/or training logic 715 are provided below in conjunction with FIGS. 7A and/or 7B. In at least one embodiment, inference and/or training logic 715 can comprise one or more circuits to use one or more neural networks to detect one or more defects in one or more manufactured objects based, at least in part, on one or more combined images of the one or more manufactured objects such as described above and/or perform other operations discussed herein.
[0100] In at least one embodiment, inference and/or training logic 715 may include, without limitation, code and/or data storage 701 to store forward and/or output weight and/or input/output data, and/or other parameters to configure neurons or layers of a neural network trained and/or used for inferencing in aspects of one or more embodiments. In at least one embodiment, training logic 715 may include, or be coupled to code and/or data storage 701 to store graph code or other software to control timing and/or order, in which weight and/or other parameter information is to be loaded to configure, logic, including integer and/or floating point units (collectively, arithmetic logic units (ALUs)). In at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on an architecture of a neural network to which such code corresponds. In at least one embodiment, code and/or data storage 701 stores weight parameters and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input/output data and/or weight parameters during training and/or inferencing using aspects of one or more embodiments. In at least one embodiment, any portion of code and/or data storage 701 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory.
[0101] In at least one embodiment, any portion of code and/or data storage 701 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and/or code and/or data storage 701 may be cache memory, dynamic randomly addressable memory (“DRAM”), static randomly addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, a choice of whether code and/or code and/or data storage 701 is internal or external to a processor, for example, or comprising DRAM, SRAM, flash or some other storage type may depend on available storage on-chip versus off-chip, latency requirements of training and/or inferencing functions being performed, batch size of data used in inferencing and/or training of a neural network, or some combination of these factors.
[0108] In at least one embodiment, inference and/or training logic 715 illustrated in FIG. 7A may be used in conjunction with an application-specific integrated circuit (“ASIC”), such as a TensorFlow® Processing Unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, inference and/or training logic 715 illustrated in FIG. 7A may be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware or other hardware, such as field programmable gate arrays (“FPGAs”).
[0109] FIG. 7B illustrates inference and/or training logic 715, according to at least one embodiment. In at least one embodiment, inference and/or training logic 715 may include, without limitation, hardware logic in which computational resources are dedicated or otherwise exclusively used in conjunction with weight values or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, inference and/or training logic 715 illustrated in FIG. 7B may be used in conjunction with an application-specific integrated circuit (ASIC), such as TensorFlow® Processing Unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corp. In at least one embodiment, inference and/or training logic 715 illustrated in FIG. 7B may be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware or other hardware, such as field programmable gate arrays (FPGAs). In at least one embodiment, inference and/or training logic 715 includes, without limitation, code and/or data storage 701 and code and/or data storage 705, which may be used to store code (e.g., graph code), weight values and/or other information, including bias values, gradient information, momentum values, and/or other parameter or hyperparameter information. In at least one embodiment illustrated in FIG. 7B, each of code and/or data storage 701 and code and/or data storage 705 is associated with a dedicated computational resource, such as computational hardware 702 and computational hardware 706, respectively. In at least one embodiment, each of computational hardware 702 and computational hardware 706 comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in code and/or data storage 701 and code and/or data storage 705, respectively, result of which is stored in activation storage 720. In at least one embodiment, each of computational hardware 702 and computational hardware 706 comprises one or more circuits to use one or more neural networks to detect one or more defects in one or more manufactured objects based, at least in part, on one or more combined images of the one or more manufactured objects such as described above and/or perform other operations discussed herein.
[0179] In at least one embodiment, one or more of SoC(s) 1004 may include data store(s) 1016 (e.g., memory). In at least one embodiment, data store(s) 1016 may be on-chip memory of SoC(s) 1004, which may store neural networks to be executed on GPU(s) 1008 and/or a DLA. In at least one embodiment, data store(s) 1016 may be large enough in capacity to store multiple instances of neural networks for redundancy and safety. In at least one embodiment, data store(s) 1016 may comprise L2 or L3 cache(s).
Therefore, in view of Gurumurthy second embodiment, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of further comprising: at least one processor-readable storage medium coupled to the data bus and configured to store one or more of the training data, the model data, or the image, incorporated in the device of Gurumurthy, in order to perform transfer a larger amount of data across a data bus, which accelerate the execution and efficiency of performed applications (as stated in Gurumurthy ¶ [241]).
Re claim 8: Gurumurthy discloses a computer program product including one or more non-transitory machine-readable mediums having instructions encoded thereon that when executed by at least one processor cause a process to be carried out for detecting defects in one or more features of interest on a printed circuit board (PCB) (e.g. a camera is used to capture images of a PCB in order to detect features, such as defects. This is taught in ¶ [63], [66] and [67] above.), the process comprising:
processing an image of the PCB using an autoencoder to generate a reconstructed image, trained to recognize the at least one feature of interest based on a combination of model data and training data (e.g. the system discloses analyzing a sample object, which is a manufactured object free of defects considered as model data. In addition, an image with defects is analyzed in a manufactured object considered as training data, which is taught in ¶ [66] above. ¶ [56] above discloses a neural network that is trained to recognize features within an image. An autoencoder is used to receive an image and produce a reconstructed image, which is taught in ¶ [59] above.);
comparing the reconstructed image to the image; based on the comparison, generating a reconstruction error value representing a measure of difference between the image and the reconstructed image (e.g. the combined or reconstructed image is generated as an output in order to be compared to an ideal image. The comparison yields a number that is compared to a threshold to determine if the comparison between images has a value above or below a threshold. This is taught in ¶ [78]-[81] above.); and
generating an output indicating detection of a defect in at least one of the one or more features of interest based on the reconstruction error value transgressing a threshold (e.g. the comparison yields a number that is compared to a threshold to determine if the comparison between images has a value above or below a threshold. This is taught in ¶ [78]-[81] above. The features of the defects are displayed with a bounding box around the features, which is explained in ¶ [81] above.).
However, Gurumurthy fails to specifically teach the features of the autoencoder having been trained to recognize the at least one feature of interest based on a combination of model data and training data.
However, this is well known in the art as evidenced by Hida. Similar to the primary reference, Hida discloses training an autoencoder (same field of endeavor or reasonably pertinent to the problem).
Hida discloses the autoencoder having been trained to recognize the at least one feature of interest based on a combination of model data and training data (e.g. an autoencoder is trained is trained to extract features from a defect free image. The autoencoder is trained on defect free images and images with defects, which is taught in ¶ [63]-[66] above.).
Therefore, in view of Hida, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of the autoencoder having been trained to recognize the at least one feature of interest based on a combination of model data and training data, incorporated in the device of Gurumurthy, in order to train an autoencoder with a plurality off epochs with a maximum difference between images, which can improve the effectiveness of removing defects (as stated in Hida ¶ [37]).
However, the combination above fails to specifically teach the features of wherein the model data comprises synthetically created representations of PCB features.
However, this is well known in the art as evidenced by Fiebag. Similar to the primary reference, Fiebag discloses determining defects on a PCB board (same field of endeavor or reasonably pertinent to the problem).
Fiebag discloses wherein the model data comprises synthetically created representations of PCB features (e.g. the invention discloses training an algorithm using training data within a first phase, which is taught in ¶ [46]-[50] above. In addition, in another phase, synthetic images of the PCB and regions of the circuit board can be used to improve the classification performance of the adaptive algorithm ML. This is taught in ¶ [54]-[56] above.).
Therefore, in view of Fiebag, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the model data comprises synthetically created representations of PCB features, incorporated in the device of Gurumurthy, as modified by Hida, in order to include synthetic image in the training of an algorithm, which can improve the classification performance of the algorithm (as stated in Fiebag ¶ [56]).
Re claim 9: However, Gurumurthy fails to specifically teach the features of the computer program product of claim 8, wherein the process further comprises: prior to processing the image, training the autoencoder using the combination of the model data and the training data.
However, Gurumurthy fails to specifically teach the features of the autoencoder having been trained to recognize the at least one feature of interest based on a combination of model data and training data.
However, this is well known in the art as evidenced by Hida. Similar to the primary reference, Hida discloses training an autoencoder (same field of endeavor or reasonably pertinent to the problem).
Hida discloses wherein the process further comprises: prior to processing the image, training the autoencoder using the combination of the model data and the training data (e.g. an autoencoder is trained is trained to extract features from a defect free image. The autoencoder is trained, beforehand, on defect free images and images with defects, which is taught in ¶ [63]-[66] above.).
Therefore, in view of Hida, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the process further comprises: prior to processing the image, training the autoencoder using the combination of the model data and the training data, incorporated in the device of Gurumurthy, in order to train an autoencoder with a plurality off epochs with a maximum difference between images, which can improve the effectiveness of removing defects (as stated in Hida ¶ [37]).
Re claim 10: Gurumurthy discloses the computer program product of claim 9, wherein training the autoencoder comprises:
processing the model data to generate a first plurality of neural network nodes corresponding to a plurality of model-based features derived from the model data (e.g. nodes are generated in order to process features from the model input into the layers of the neural networks. The node containing the first layer is used to process the sample image data before passing the processed data to the fully connected layer, which is taught in ¶ [79] above and figure 3.);
processing the training data to generate a second plurality of neural network nodes corresponding to a plurality of machine learning-based features derived from the training data (e.g. the invention discloses operating on data containing defects in order to determine the defects used in training the neural network. The images containing the defects are passed between generated nodes before the data is output, which is taught in ¶ [79]-[81] above.); and
creating a plurality of interconnections between the first and second pluralities of neural network nodes based on correlations between the model-based features and the machine learning-based features (e.g. as seen in figure 3, a plurality of interconnections are formed between the first and second network nodes based on these nodes receiving the different types of data and comparing them in the loss function step, which is taught in ¶ [79]-[81] above.).
Re claim 11: Gurumurthy discloses the computer program product of claim 10, wherein training the autoencoder further comprises:
forming a plurality of layers in a latent space representation of the autoencoder, the first and second pluralities of neural network nodes being distributed among the plurality of layers (e.g. plurality of nodes are distributed throughout the neural network. There are various layers within the nodes within the system, which is taught in ¶ [79] above.).
Re claim 12: Gurumurthy discloses the computer program product of claim 9, wherein training the autoencoder further comprises annotating at least a portion of at least one of the model data or the training data (e.g. the invention discloses training with the use of labels for the training data, which is taught in ¶ [60] and [61].).
[0060] In at least one embodiment, training framework 106 uses training data 104 with supervision, where supervision is in a form of classification or labels. In at least one embodiment, training framework 106 uses training data 104 to train one or more neural networks 108 using any other form of supervision to facilitate training 102 of said one or more neural networks 108. In at least one embodiment, training framework 106 does not use supervision for some or all training data 104.
[0061] In at least one embodiment, one or more neural networks 108 are trained by training framework 106 using supervision. In at least one embodiment, supervision comprises multiple types of assistance utilized to facilitate training 102 of one or more neural networks 108 by training framework 106, as described above. In at least one embodiment, supervision comprises input information that describes one or more aspects of training data 104, such as objects or styles, or a classification for said training data 104, to assist training one or more neural networks 108 by training framework 106. In at least one embodiment, supervision is strong, wherein input information provides direct identification of an object, style, or other aspect of an item, such as an image, in training data 104. In at least one embodiment, supervision is weak, wherein input information provides partial identification of an object, style, or other aspect of an input training data 104 item. In at least one embodiment, strong supervision is input information such as bounding boxes, where one or more objects are outlined in an input training data 104 item. In at least one embodiment, weak supervision comprises input information such as points, where individual locations in an input training data 104 item are identified as being within an object or objects. In at least one embodiment, weak supervision comprises input information such as lines, where each point in a line within an input training data 104 item are identified by said weak supervision as being within an object or objects. In at least one embodiment, weak supervision comprises input information such as tags or labels, where a tag or label identifies that an input training data 104 item contains a specific object or objects or is of a specific classification.
Re claim 14: Gurumurthy discloses a method of detecting defects in one or more features of interest on a printed circuit board (PCB) (e.g. a camera is used to capture images of a PCB in order to detect features, such as defects. This is taught in ¶ [63], [66] and [67] above.), the method comprising:
processing an image of the PCB using an autoencoder to generate a reconstructed image, trained to recognize the at least one feature of interest based on a combination of model data and training data (e.g. the system discloses analyzing a sample object, which is a manufactured object free of defects considered as model data. In addition, an image with defects is analyzed in a manufactured object considered as training data, which is taught in ¶ [66] above. ¶ [56] above discloses a neural network that is trained to recognize features within an image. An autoencoder is used to receive an image and produce a reconstructed image, which is taught in ¶ [59] above.);
comparing the reconstructed image to the image; based on the comparison, generating a reconstruction error value representing a measure of difference between the image and the reconstructed image (e.g. the combined or reconstructed image is generated as an output in order to be compared to an ideal image. The comparison yields a number that is compared to a threshold to determine if the comparison between images has a value above or below a threshold. This is taught in ¶ [78]-[81] above.); and
generating an output indicating detection of a defect in at least one of the one or more features of interest based on the reconstruction error value transgressing a threshold (e.g. the comparison yields a number that is compared to a threshold to determine if the comparison between images has a value above or below a threshold. This is taught in ¶ [78]-[81] above. The features of the defects are displayed with a bounding box around the features, which is explained in ¶ [81] above.).
However, Gurumurthy fails to specifically teach the features of the autoencoder having been trained to recognize the at least one feature of interest based on a combination of model data and training data.
However, this is well known in the art as evidenced by Hida. Similar to the primary reference, Hida discloses training an autoencoder (same field of endeavor or reasonably pertinent to the problem).
Hida discloses the autoencoder having been trained to recognize the at least one feature of interest based on a combination of model data and training data (e.g. an autoencoder is trained is trained to extract features from a defect free image. The autoencoder is trained on defect free images and images with defects, which is taught in ¶ [63]-[66] above.).
Therefore, in view of Hida, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of the autoencoder having been trained to recognize the at least one feature of interest based on a combination of model data and training data, incorporated in the device of Gurumurthy, in order to train an autoencoder with a plurality off epochs with a maximum difference between images, which can improve the effectiveness of removing defects (as stated in Hida ¶ [37]).
However, the combination above fails to specifically teach the features of wherein the model data comprises synthetically created representations of PCB features.
However, this is well known in the art as evidenced by Fiebag. Similar to the primary reference, Fiebag discloses determining defects on a PCB board (same field of endeavor or reasonably pertinent to the problem).
Fiebag discloses wherein the model data comprises synthetically created representations of PCB features (e.g. the invention discloses training an algorithm using training data within a first phase, which is taught in ¶ [46]-[50] above. In addition, in another phase, synthetic images of the PCB and regions of the circuit board can be used to improve the classification performance of the adaptive algorithm ML. This is taught in ¶ [54]-[56] above.).
Therefore, in view of Fiebag, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the model data comprises synthetically created representations of PCB features, incorporated in the device of Gurumurthy, as modified by Hida, in order to include synthetic image in the training of an algorithm, which can improve the classification performance of the algorithm (as stated in Fiebag ¶ [56]).
Re claim 15: Gurumurthy discloses the method of claim 14, wherein generating the output indicating detection of the defect includes generating the output indicating detection of the defect based on the reconstruction error value exceeding the threshold (e.g. the combined or reconstructed image is generated as an output in order to be compared to an ideal image. The comparison yields a number that is compared to a threshold to determine if the comparison between images has a value above or below a threshold. This is taught in ¶ [78]-[81] above.).
Re claim 16: However, Gurumurthy fails to specifically teach the features of the method of claim 14, further comprising: training the autoencoder using the combination of the model data and the training data.
However, Gurumurthy fails to specifically teach the features of the autoencoder having been trained to recognize the at least one feature of interest based on a combination of model data and training data.
However, this is well known in the art as evidenced by Hida. Similar to the primary reference, Hida discloses training an autoencoder (same field of endeavor or reasonably pertinent to the problem).
Hida discloses further comprising: training the autoencoder using the combination of the model data and the training data (e.g. an autoencoder is trained is trained to extract features from a defect free image. The autoencoder is trained, beforehand, on defect free images and images with defects, which is taught in ¶ [63]-[66] above.).
Therefore, in view of Hida, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of further comprising: training the autoencoder using the combination of the model data and the training data, incorporated in the device of Gurumurthy, in order to train an autoencoder with a plurality off epochs with a maximum difference between images, which can improve the effectiveness of removing defects (as stated in Hida ¶ [37]).
Re claim 17: Gurumurthy discloses the method of claim 16, wherein training the autoencoder includes:
processing the model data to generate a first plurality of neural network nodes corresponding to a plurality of model-based features derived from the model data (e.g. nodes are generated in order to process features from the model input into the layers of the neural networks. The node containing the first layer is used to process the sample image data before passing the processed data to the fully connected layer, which is taught in ¶ [79] above and figure 3.);
processing the training data to generate a second plurality of neural network nodes corresponding to a plurality of machine learning-based features derived from the training data (e.g. the invention discloses operating on data containing defects in order to determine the defects used in training the neural network. The images containing the defects are passed between generated nodes before the data is output, which is taught in ¶ [79]-[81] above.); and
creating a plurality of interconnections between the first and second pluralities of neural network nodes based on correlations between the model-based features and the machine learning-based features (e.g. as seen in figure 3, a plurality of interconnections are formed between the first and second network nodes based on these nodes receiving the different types of data and comparing them in the loss function step, which is taught in ¶ [79]-[81] above.).
Re claim 18: Gurumurthy discloses the method of claim 17, wherein training the autoencoder further comprises:
forming a plurality of layers in a latent space representation of the autoencoder, the first and second pluralities of neural network nodes being distributed among the plurality of layers (e.g. plurality of nodes are distributed throughout the neural network. There are various layers within the nodes within the system, which is taught in ¶ [79] above.).
Re claim 20: Gurumurthy discloses the method of claim 14, further comprising: acquiring the image using an imaging device (e.g. a camera is used to capture images of a PCB in order to detect features, such as defects. This is taught in ¶ [63], [66] and [67] above.).
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gurumurthy, as modified by Hida and Fiebag, as applied to claim 1 above, and further in view of Qiu (US Pub 2023/0136914).
Re claim 6: However, Gurumurthy fails to specifically teach the features of the PCB inspection system of claim 1, wherein the at least one feature of interest includes a wire bond.
However, this is well known in the art as evidenced by Qiu. Similar to the primary reference, Qiu discloses detecting faults on PCBs (same field of endeavor or reasonably pertinent to the problem).
Qiu discloses wherein the at least one feature of interest includes a wire bond (e.g. faults can be sensed on bond wires on PCB connectors, which is taught in ¶ [21].).
[0021] The sensor device detects open and short circuit faults without the necessity of access points or pogo pins. Further, the sensor device enables unpowered testing in that it does not require the DUT to be powered on to perform measurements. Also, the sensor device may be used for testing PCB connectors, such as FFC headers, stacking connectors, PCI Express connectors, and mezzanine connectors, for example, which do not have conventional ICT solutions. Additionally, since PCB connectors form continuous circuits with connected components of the PCBA, such as ICs and bond wires, the sensor device may also detect faults on these structures when the ICs are connected to the PCB connectors.
Therefore, in view of Qiu, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the at least one feature of interest includes a wire bond, incorporated in the device of Gurumurthy, as modified by Hida and Fiebag, in order to detect defects in bond wires, which can prevent failed electronic products including a defective component from being scrapped (as stated in Qiu ¶ [01]).
Claim(s) 3, 13 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gurumurthy, as modified by Hida and Fiebag, as applied to claims 1, 8 and 14 above, and further in view of Che (US Pub 2020/0372339).
Re claim 3: Gurumurthy discloses the PCB inspection system of claim 2, wherein the autoencoder includes a neural network corresponding to interconnections between the first and second pluralities of neural network nodes (e.g. as seen in figure 3, a plurality of interconnections are formed between the first and second network nodes based on these nodes receiving the different types of data and comparing them in the loss function step, which is taught in ¶ [79]-[81] above.);
wherein the autoencoder is configured to encode the image to produce an encoded image (e.g. the autoencoder is used to encode the input data with target values, which is taught in ¶ [59] above.), and
to apply the code to the encoded image to produce the reconstructed image (e.g. the target values are used on the encoded image to generate a reconstructed input image data using backpropagation, which is taught in ¶ [59] above.).
However, Gurumurthy fails to specifically teach the features of the neural network defining a code.
However, this is well known in the art as evidenced by Che. Similar to the primary reference, Che discloses an autoencoder determining a defect in an image (same field of endeavor or reasonably pertinent to the problem).
Che discloses the neural network defining a code (e.g. the distribution of the latent variable z and the prediction from another model is used by the decoder to generate a reconstructed input. The distribution can be used as the code applied to create the reconstruction, which is taught in ¶ [28]-[30].).
[0028] FIG. 2 is a simplified diagram of a verifier module being used to verify a discriminative model according to some embodiments described herein. As shown in FIG. 2, an input x is provided to a discriminative model 200, which produces, as an output, a prediction y′. The input x is also provided to encoder 131, which generates a latent variable z. In some examples, encoder 131 is a stochastic encoder. In some examples, encoder 131 is a convolutional neural network. In some examples, encoder 131 includes two, three, four, or more convolutional layers. In some examples, latent variable z is an encoded vector. In some examples, latent variable z is a 128-dimensional vector.
[0029] The latent variable z and prediction y′ from discriminative model 200 are then provided to decoder 132, which generates a reconstructed input x′ based on the distribution p(x|z,y) learned by decoder 132 during training as is described in further detail below. In some examples, decoder 132 is a stochastic decoder. In some examples, decoder 132 is a convolutional neural network. In some examples, decoder 132 includes two, three, four, or more deconvolutional layers.
[0030] Input x, latent variable z, and reconstructed input x′ are then passed to anomaly detection module 133 to generate reliability prediction 150. In some examples, anomaly detection module 133 evaluates the loss in the generation of latent variable z and reconstructed input x′. In some examples, the loss is a log-likelihood loss. In some examples, the loss corresponds to the evidence lower bound for encoder 131 and decoder 132. In some examples, the loss includes a reconstruction loss, such as an L.sup.2 loss between input x and reconstructed input x′. In some examples, the loss includes an estimate of the Kullback-Leibler (KL) variational autoencoder (VAE) loss) for encoder 131 and latent variable z. The KL VAE loss is described in further detail in Doersch, “Tutorial on Variational Autoencoders,” 2016, available at https://arxiv.org/abs/1606.05908, which is incorporated by reference. Anomaly detection module 133 then compares the loss and/or the log of the loss to a threshold δ. When the loss and/or the log of the loss is greater than or equal to threshold δ, prediction y′ is considered reliable. When the loss and/or the log of the loss is less than threshold δ, prediction y′ is considered unreliable. In some examples, prediction y′ may be considered unreliable when input x is out-of-distribution, input x is adversarial, and/or prediction y′ is likely to be incorrect.
Therefore, in view of Che, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of the neural network defining a code, incorporated in the device of Gurumurthy, as modified by Hida and Fiebag, in order to generate a reconstructed image based on a distribution that is used to calculate a loss, which is used to improve the performance of the overall module (as stated in Che ¶ [62]).
Re claim 13: Gurumurthy discloses the computer program product of claim 8, wherein the autoencoder comprises
wherein processing the image comprises: encoding the image using the autoencoder to produce an encoded image (e.g. the autoencoder is used to encode the input data with target values, which is taught in ¶ [59] above.); and
applying the code to the encoded image to generate the reconstructed image (e.g. the target values are used on the encoded image to generate a reconstructed input image data using backpropagation, which is taught in ¶ [59] above.).
However, Gurumurthy fails to specifically teach the features of wherein the autoencoder comprises a code constructed based on correlations determined between model-based features derived from the model data and machine learning-based features derived from the training data.
However, this is well known in the art as evidenced by Che. Similar to the primary reference, Che discloses an autoencoder determining a defect in an image (same field of endeavor or reasonably pertinent to the problem).
Che discloses wherein the autoencoder comprises a code constructed based on correlations determined between model-based features derived from the model data and machine learning-based features derived from the training data (e.g. the distribution of the latent variable z and the prediction from another model is used by the decoder to generate a reconstructed input. The distribution can be used as the code applied to create the reconstruction, which is taught in ¶ [28]-[30] above.).
Therefore, in view of Che, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the autoencoder comprises a code constructed based on correlations determined between model-based features derived from the model data and machine learning-based features derived from the training data, incorporated in the device of Gurumurthy, as modified by Hida and Fiebag, in order to generate a reconstructed image based on a distribution that is used to calculate a loss, which is used to improve the performance of the overall module (as stated in Che ¶ [62]).
Re claim 19: Gurumurthy discloses the method of claim 14, wherein the autoencoder comprises
wherein processing the image comprises:
encoding the image using the autoencoder to produce an encoded image (e.g. the autoencoder is used to encode the input data with target values, which is taught in ¶ [59] above.); and
applying the code to the encoded image to generate the reconstructed image (e.g. the target values are used on the encoded image to generate a reconstructed input image data using backpropagation, which is taught in ¶ [59] above.).
However, Gurumurthy fails to specifically teach the features of a code constructed based on correlations determined between model-based features derived from the model data and machine learning-based features derived from the training data.
However, this is well known in the art as evidenced by Che. Similar to the primary reference, Che discloses an autoencoder determining a defect in an image (same field of endeavor or reasonably pertinent to the problem).
Che discloses a code constructed based on correlations determined between model-based features derived from the model data and machine learning-based features derived from the training data (e.g. the distribution of the latent variable z and the prediction from another model is used by the decoder to generate a reconstructed input. The distribution can be used as the code applied to create the reconstruction, which is taught in ¶ [28]-[30] above.).
Therefore, in view of Che, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of a code constructed based on correlations determined between model-based features derived from the model data and machine learning-based features derived from the training data, incorporated in the device of Gurumurthy, as modified by Hida and Fiebag, in order to generate a reconstructed image based on a distribution that is used to calculate a loss, which is used to improve the performance of the overall module (as stated in Che ¶ [62]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Liu discloses a neural network that includes an autoencoder.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHAD S DICKERSON whose telephone number is (571)270-1351. The examiner can normally be reached Monday-Friday 10AM-6PM EST..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abderrahim Merouan can be reached at 571-270-5254. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHAD DICKERSON/ Primary Examiner, Art Unit 2682