DETAILED ACTION
This Office Action is in response to the Applicants' communication filed on February 9, 2026, which amends the independent claims 1, 9, and 15, amends the dependent claims 2-8, , and presents arguments, is hereby acknowledged. Claims 1-20 are currently pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s arguments filed on February 9, 2026, have been fully considered.
Applicant argues that by this response, the independent claims 1, 9, and 15 are amended to overcome the 101 rejections.
Examiner replies that the 101 rejections for claims 1-20 have been withdrawn.
Applicant argues that by this response, the independent claims 1, 9, and 15 are hereby amended to add new limitations of “use one or more neural networks to apply one or more image transformations to one or more images to generate one or more modified versions of the one or more images; invert the one or more image transformations for one or more identified features in the one or more images” in order to overcome the 35 U.S.C. §103 rejection.
Examiner replies that the amended claims with new limitation may overcome the cited portions of the prior arts. However, a newly found art, Bozchalooi, etc.( US 20220137634 A1) teaches that use one or more neural networks to apply one or more image transformations to one or more images to generate one or more modified versions of the one or more images (See Bozchalooi: Figs. 2-3, and [0036], “During testing and operation, the memory augmented neural network 300 generates a large number (>100) of image variants 306 based on the transform bank 204. The image variants 306 are passed one at a time through the neural network (NN) 308, where they are processed as described above in relation to FIG. 2 to form feature point sets for each image variant that can be stored as feature variant sets (FVS) 310. As described above, set mean (SM) 312 and inverse covariance (IC) 314 are determined based on each feature variant set 310 for each test image 302 input to the transform bank 204. The set mean 312 and inverse covariance 314 are input to Mahalanobis distance and KL divergence testing block (MKLD) 318 to determine which feature variant set 210 stored in memory 216 the current feature variant set 310 matches most closely”); and another art has been found, Zhang, etc. (US 20170351952 A1) teaches that invert the one or more image transformations for one or more identified features in the one or more images (See Zhang: Figs. 1-4, and [0058], “The component(s), e.g., component(s) 100 shown in FIG. 1, executed by the computer subsystem(s), e.g., computer subsystem 36 and/or computer subsystem(s) 102, include neural network 104. The neural network is configured for determining inverted features of input images in a training set for a specimen input to the neural network. For example, as shown in FIG. 2, image 200 may be input to neural network 202, which determines inverted features 204 for the image. In this manner, a neural network is used to approximate the inversion function f.sup.−1( ), and the neural network generates the inverted features from the input image. In the context of semiconductor applications such as inspection, metrology, and defect review, the neural network described herein can be used to solve inverse problems in imaging formation (e.g., diffraction, interference, partial coherence, blurring, etc.) to regenerate optically-corrected features. “Inverted features” (where inverted is related to the context of an inversion neural network) are generally defined herein as features after inverting a physical process and “features” are defined as generally referring to measurable properties including, but not limited to, intensity, amplitude, phase, edge, gradients, etc.”, Note that the measurable properties of intensity, amplitude, phase, edge, gradients, etc. are mapped to the identified features, and the inversion function f.sup.−1( ) is mapped to “invert the one or more image transformations”). The remaining arguments of the applicant are mooted in view of the newly found arts..
Examiner respectfully further replies that the Applicant's arguments have been fully considered and a new ground of rejections have been made. Accordingly, new grounds of rejection are set forth below. Since the new grounds of rejection are necessitated by Applicant's amendments to the claims, the present action is made final.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4 and 6-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yu, etc. (US 20220261593 A1) in view of Chadha, etc. (US 20210150282 A1), further in view of Bozchalooi, etc.( US 20220137634 A1), and Zhang, etc. (US 20170351952 A1).
Regarding claim 1, Yu teaches that one or more processor (See Yu: Figs. 1 and 12, and [0256], "FIG. 12 is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SOC) or some combination thereof formed with a processor that may include execution units to execute an instruction, according to at least one embodiment") comprising:
circuitry (See Yu: Fig. 17, and [0327], “FIG. 17 illustrates exemplary integrated circuits and associated graphics processors that may be fabricated using one or more IP cores, according to various embodiments described herein. In addition to what is illustrated, other logic and circuits may be included in at least one embodiment, including additional graphics processors/cores, peripheral interface controllers, or general-purpose processor cores”) to:
use one or more neural networks to apply one or more image transformations to one or more images to generate one or more modified versions of the one or more images;
invert the one or more image transformations for one or more identified features in the one or more images; and
use the one or more neural networks (See Yu: Figs. 9 and 12, and [0265], "Inference and/or training logic 815 are used to perform inferencing and/or training operations associated with one or more embodiments. Details regarding inference and/or training logic 815 are provided herein in conjunction with FIGS. 8A and/or 8B. In at least one embodiment, inference and/or training logic 815 may be used in system FIG. 12 for inferencing or predicting operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and/or architectures, or neural network use cases described herein") to identify one or more objects within the one or more images (See Yu: Figs. 1 and 12, and [0072], "In at least one embodiment, semantic correspondence 112 refers to one or more processes that determine regions of images 102, 104 that are semantically related. In at least one embodiment, regions of images 102, 104 that are semantically related refer to regions of images that belong to a same class of object, such as car, person, plant, and/or variations thereof. In at least one embodiment, semantic correspondence 112 refers to one or more processes that determine correspondence between pixels or sets of pixels of images 102, 104 (e.g., pixels or sets of pixels that are semantically related) by extracting and comparing features of said images 102, 104. In at least one embodiment, for semantic correspondence 112, a framework 106 obtains and processes one or more images (e.g., image 102 and/or image 104) to determine correspondence between objects of said one or more images 102, 104 (e.g., determining if one or more objects of a first image are depicted in a second image). In at least one embodiment, a framework 106 extracts features of an image, identifies objects of said image by analyzing said features, and compares identified objects from said image with other identified objects of other images to determine correspondence between objects of said image and said other images. In at least one embodiment, referring to FIG. 1, semantic correspondence 112 comprises one or more processes that determine correspondence between a car object depicted in a first image 102 and a first car object and a second car object depicted in a second image 104. In at least one embodiment, referring to FIG. 1, semantic correspondence 112 comprises one or more processes that determine correspondence between a first car object and/or a second car object depicted in a second image 104 and a car object depicted in a first image 102") based, at least in part, on the one or more features (See Yu: Figs. 1 and 12, and [0066], "In at least one embodiment, a student network obtains and processes training images 102, 104 by determining features of said training images 102, 104. In at least one embodiment, a student network processes determined features to perform object detection 108, in which said student network generates box proposals indicating potential locations of objects of training images 102, 104 and classifications of said objects. In at least one embodiment, a student network processes determined features to perform instance segmentation 110, in which said student network generates masks indicating pixels of objects of training images 102, 104 to determine instances of said objects in training images. In at least one embodiment, a student network processes determined features to perform semantic segmentation, in which said student network generates one or more segmentation maps indicating pixels of objects of training images 102, 104") of the one or more modified versions of the one or more images with the inverted one or more image transformations.
However, Yu fails to explicitly disclose that the use one or more neural networks to apply one or more image transformations to one or more images to generate one or more modified versions of the one or more images; invert the one or more image transformations for one or more identified features in the one or more images; and one or more features of the one or more modified versions of the one or more images with the inverted one or more image transformations.
However, Chadha teaches that one or more features of the one or more modified versions of the one or more images (See Chadha: Fig. 2, and [0030], "The input image 205 and the modified input images 210 may be processed by an object detection machine learning model. In FIG. 2, the machine learning model is an object detection neural network 220, but other types of machine learning (e.g., deep learning) models may be used. The object detection neural network 220 may perform various processes such as object recognition, image classification, object localization, object segmentation, etc. for each of the input image 205 and the modified input images 210") with the inverted one or more image transformations.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Yu to have one or more features of the one or more modified versions of the one or more images as taught by Chadha in order to provide a better chance of detecting an object or classifying the image (See Chadha: Fig. 1, and [0030], "In some cases, an image detection process may utilize test time augmentation techniques, in which a test image may be manipulated to provide a better chance of detecting an object or classifying the image. For example, a test image may be manipulated such as to output different scaled test images. The test image and the scaled test images may be input in the image detection model, various confidence scores may be assigned to objects/images, the confidence scores may be filtered, and remaining confidence scores may correspond to detected objects or classified images. However, these techniques may discard various confident predictions because the filtering methods may not account for variations of sampled predictions at different scales (or other image manipulations). As such, the model may fail to identify some objects or misclassify an image"). Yu teaches a method and system that may detect and identify objects in the image using neural network with two input images, one source image and one reference image; while Chadha teaches a system and method that may detect and identify objects in the input image by processing the input image and the modified input image. Therefore, it is obvious to one of ordinary skill in the art to modify Yu by Chadha to detect and identify the objects in the input source image using the modified input image as the reference input image to have better detection results. The motivation to modify Yu by Chadha is "Use of known technique to improve similar devices (methods, or products) in the same way".
However, Yu, modified by Chadha, fails to explicitly disclose that the use one or more neural networks to apply one or more image transformations to one or more images to generate one or more modified versions of the one or more images; invert the one or more image transformations for one or more identified features in the one or more images; and with the inverted one or more image transformations.
However, Bozchalooi teaches that use one or more neural networks to apply one or more image transformations to one or more images to generate one or more modified versions of the one or more images (See Bozchalooi: Figs. 2-3, and [0036], “During testing and operation, the memory augmented neural network 300 generates a large number (>100) of image variants 306 based on the transform bank 204. The image variants 306 are passed one at a time through the neural network (NN) 308, where they are processed as described above in relation to FIG. 2 to form feature point sets for each image variant that can be stored as feature variant sets (FVS) 310. As described above, set mean (SM) 312 and inverse covariance (IC) 314 are determined based on each feature variant set 310 for each test image 302 input to the transform bank 204. The set mean 312 and inverse covariance 314 are input to Mahalanobis distance and KL divergence testing block (MKLD) 318 to determine which feature variant set 210 stored in memory 216 the current feature variant set 310 matches most closely”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Yu to have to use one or more neural networks to apply one or more image transformations to one or more images to generate one or more modified versions of the one or more images as taught by Bozchalooi in order to enable improving training and operation of a neural network by generating additional images of an object in a training data set and increasing number of images and corresponding ground truth in a training dataset determined or learned by the neural network (See Bozchalooi: Fig. 6, and [0011], " Techniques discussed herein improve the training and operation of a neural network by generating additional images of an object in a training dataset. Image transformations that alter input images and corresponding ground truth data to increase the number of images and corresponding ground truth in a training dataset can be determined or learned by a neural network"). Yu teaches a method and system that may detect and identify objects in the image using neural network with two input images, one source image and one reference image; while Bozchalooi teaches a system and method that may generate a variety of modified versions of the input image with a transformation bank and train the NN (neural network) accordingly. Therefore, it is obvious to one of ordinary skill in the art to modify Yu by Bozchalooi to use the transformation bank to generate a plurality of modified version of the input image to train or test the NN in order to improve the training and operation of the NN. The motivation to modify Yu by Bozchalooi is "Use of known technique to improve similar devices (methods, or products) in the same way".
However, Yu, modified by Chadha and Bozchalooi, fails to explicitly disclose that invert the one or more image transformations for one or more identified features in the one or more images; and with the inverted one or more image transformations.
However, Zhang teaches that invert the one or more image transformations for one or more identified features in the one or more images (See Zhang: Figs. 1-4, and [0058], “The component(s), e.g., component(s) 100 shown in FIG. 1, executed by the computer subsystem(s), e.g., computer subsystem 36 and/or computer subsystem(s) 102, include neural network 104. The neural network is configured for determining inverted features of input images in a training set for a specimen input to the neural network. For example, as shown in FIG. 2, image 200 may be input to neural network 202, which determines inverted features 204 for the image. In this manner, a neural network is used to approximate the inversion function f.sup.−1( ), and the neural network generates the inverted features from the input image. In the context of semiconductor applications such as inspection, metrology, and defect review, the neural network described herein can be used to solve inverse problems in imaging formation (e.g., diffraction, interference, partial coherence, blurring, etc.) to regenerate optically-corrected features. “Inverted features” (where inverted is related to the context of an inversion neural network) are generally defined herein as features after inverting a physical process and “features” are defined as generally referring to measurable properties including, but not limited to, intensity, amplitude, phase, edge, gradients, etc.”, Note that the measurable properties of intensity, amplitude, phase, edge, gradients, etc. are mapped to the identified features, and the inversion function f.sup.−1( ) is mapped to “invert the one or more image transformations”); and
with the inverted one or more image transformations (See Zhang: Figs. 2-4, and [0099], "In an additional embodiment, the one or more computer subsystems are configured to input a runtime image for the specimen or another specimen into the trained neural network such that: the trained neural network determines the inverted features for the runtime image; the forward physical model reconstructs the runtime image from the inverted features determined for the runtime image; and the residue layer determines differences between the runtime image and the reconstructed runtime image, where the inverted features are features of an optically corrected version of the runtime image, and the differences between the runtime image and the reconstructed runtime image are features of a residue image").
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Yu to have invert the one or more image transformations for one or more identified features in the one or more images; and with the inverted one or more image transformations as taught by Zhang in order to ensure determining the accurate size of defects (See Zhang: Fig. 1, and [0005], " The higher resolution data for the defects generated by defect review is more suitable for determining attributes of the defects such as profile, roughness, more accurate size information, etc. "). Yu teaches a method and system that may detect and identify objects in the image using neural network with two input images, one source image and one reference image; while Zhang teaches a system and method that may use the network to get the inverted features from the input image and use the inverted features to reconstruct the input image with accurate defect size determination. Therefore, it is obvious to one of ordinary skill in the art to modify Yu by Zhang to invert the image transformations to obtain the inverted features of the input image in order to reconstruct the input images more accurately using the inverted features. The motivation to modify Yu by Zhang is "Use of known technique to improve similar devices (methods, or products) in the same way".
Regarding claim 2, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 1 as outlined above. Further, Chadha teaches that the one or more processors of claim 1, wherein the circuitry is further to generate additional one or more images based, at least in part, on one or more features of the one or more images and the one or more features of the one or more modified versions of the one or more images (See Chadha: Fig. 2, and [0032], "According to aspects described herein, post processing and filtering may include confidence distribution normalization 245 and non-max suppression (NMS) thresholding per modification 250. According to confidence distribution normalization 245, the system may normalize the confidence distributions for each of the range of confidence scores corresponding to the modified input images 210 based on the range of confidence scores for the input image 205").
Regarding claim 3, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 1 as outlined above. Further, Chadha teaches that the one or more processors processor of claim 1, wherein the circuitry is further to generate the one or more features of the one or more modified versions of the one or more images using one or more convolutional layers of the one or more neural networks (See Chadha: Fig. 2, and [0031], "The object detection neural network 220 may identify sets of objects (e.g., an object identified by a bounding box 225) for each processed image and confidence scores associated with the identified objects. That is, the object detection neural network 220 may identify a range (e.g., distribution) of confidence scores for each processed image, where each confidence score corresponds to an identified object from one of the processed images. Certain post processing and filtering algorithms 230 may perform certain procedures to output detected objects. For example, the post processing and filtering algorithms 230 may resize bounding boxes to original scale and filter confidence scores and associated objects". Note that the scale, bounding box, confidence scores, etc., may be the additional features for the modified images).
Regarding claim 4, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 1 as outlined above. Further, Chadha teaches that the one or more processors of claim 1, wherein the circuitry is further to generate the modified versions of the one or more images using at least one of random roll, affine transform, or pixel removing (See Chadha: Fig. 2, and [0036], "According to NMS thresholding per modification 250, the system may identify objects corresponding to overlapping bounding boxes per modified input image 210 and select the highest score from the overlapping boxes for consideration. This technique may be performed prior to resizing images/bounding boxes to the original scale (or changing other parameters of the modified input images 210 to correspond to the input image 205, such as removing transformations, removing brightness modifications, etc.). The object detection neural network 220 may identify bounding boxes that outline detected objects. In some cases, the object detection neural network 220 may identify bounding boxes that correspond to the same object in an image, and as such, may overlap to some degree. Further, each bounding box may have a corresponding confidence score associated therewith that corresponds to the confidence assigned by the object detection neural network 220 in detection of an object. In some systems, each of the bounding boxes (whether overlapping or not) for the modified input images 210 may be rescaled along with rescaling of the modified input images 210 to the original scale corresponding to the input image 205. Thereafter, the system may identify overlapping bounding boxes (rescaled and original) and select a highest score (or filter the lowest score) from the overlapping bounding boxes. According to the techniques described herein, the system selects the highest (or filters one or more of the lowest) confidence scores (and the corresponding bounding boxes/objects) for each modified input image 210 (before any rescaling or removing modifications). As such, the most relevant confidence score per modified input image 210 may be identified before rescaling. In some cases, this may avoid utilizing processing resources for rescaling or removing modifications for a set of bounding boxes, since some may be removed from consideration prior to the rescaling/removing of modifications. For example, when the image manipulation techniques include generating multiple scaled images (as illustrated in FIG. 2), the system may perform NMS thresholding per scaled image, including the original input image 205 and the manipulated input images 210").
Regarding claim 6, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 1 as outlined above. Further, Yu teaches that the one or more processors of claim 1, wherein the one or more images depict a scene (See Yu: Figs. 18A-B, and [0335], "In at least one embodiment, graphics processor 1840 includes one or more shader core(s) 1855A-1855N (e.g., 1855A, 1855B, 1855C, 1855D, 1855E, 1855F, through 1855N-1, and 1855N) as shown in FIG. 18B, which provides for a unified shader core architecture in which a single core or type or core can execute all types of programmable shader code, including shader program code to implement vertex shaders, fragment shaders, and/or compute shaders. In at least one embodiment, a number of shader cores can vary. In at least one embodiment, graphics processor 1840 includes an inter-core task manager 1845, which acts as a thread dispatcher to dispatch execution threads to one or more shader cores 1855A-1855N and a tiling unit 1858 to accelerate tiling operations for tile-based rendering, in which rendering operations for a scene are subdivided in image space, for example to exploit local spatial coherence within a scene or to optimize use of internal caches").
Regarding claim 7, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 1 as outlined above. Further, Yu teaches that the one or more processors of claim 1, wherein the circuitry is further to send information of the one or more identified objects to one or more autonomous vehicles (See Yu: Figs. 11A-D, and [0152], "FIG. 11A illustrates an example of an autonomous vehicle 1100, according to at least one embodiment. In at least one embodiment, autonomous vehicle 1100 (alternatively referred to herein as "vehicle 1100") may be, without limitation, a passenger vehicle, such as a car, a truck, a bus, and/or another type of vehicle that accommodates one or more passengers. In at least one embodiment, vehicle 1100 may be a semi-tractor-trailer truck used for hauling cargo. In at least one embodiment, vehicle 1100 may be an airplane, robotic vehicle, or other kind of vehicle").
Regarding claim 8, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 1 as outlined above. Further, Yu teaches that the one or more processors of claim 1, wherein at least one of the one or more images is three-dimensional (3D) image (See Yu: Figs. 11A-D, and [0164], "In at least one embodiment, one or more camera may be mounted in a mounting assembly, such as a custom designed (three-dimensional ("3D") printed) assembly, in order to cut out stray light and reflections from within vehicle 1100 (e.g., reflections from dashboard reflected in windshield mirrors) which may interfere with camera image data capture abilities. With reference to wing-mirror mounting assemblies, in at least one embodiment, wing-mirror assemblies may be custom 3D printed so that a camera mounting plate matches a shape of a wing-mirror. In at least one embodiment, camera(s) may be integrated into wing-mirrors. In at least one embodiment, for side-view cameras, camera(s) may also be integrated within four pillars at each corner of a cabin").
Regarding claim 9, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 1 as outlined above. Further, Yu, Chadha, Bozchalooi, and Zhang teach that a method (See Yu: Fig. 1 and 12, and [0256], "FIG. 12 is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SOC) or some combination thereof formed with a processor that may include execution units to execute an instruction, according to at least one embodiment") comprising:
using one or more neural networks (See Yu: Figs. 9 and 12, and [0265], "Inference and/or training logic 815 are used to perform inferencing and/or training operations associated with one or more embodiments. Details regarding inference and/or training logic 815 are provided herein in conjunction with FIGS. 8A and/or 8B. In at least one embodiment, inference and/or training logic 815 may be used in system FIG. 12 for inferencing or predicting operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and/or architectures, or neural network use cases described herein") to apply one or more image transformations to one or more images to generate one or more modified versions of the one or more images (See Bozchalooi: Figs. 2-3, and [0036], “During testing and operation, the memory augmented neural network 300 generates a large number (>100) of image variants 306 based on the transform bank 204. The image variants 306 are passed one at a time through the neural network (NN) 308, where they are processed as described above in relation to FIG. 2 to form feature point sets for each image variant that can be stored as feature variant sets (FVS) 310. As described above, set mean (SM) 312 and inverse covariance (IC) 314 are determined based on each feature variant set 310 for each test image 302 input to the transform bank 204. The set mean 312 and inverse covariance 314 are input to Mahalanobis distance and KL divergence testing block (MKLD) 318 to determine which feature variant set 210 stored in memory 216 the current feature variant set 310 matches most closely”);
inverting the one or more image transformations for one or more identified features in the one or more images (See Zhang: Figs. 1-4, and [0058], “The component(s), e.g., component(s) 100 shown in FIG. 1, executed by the computer subsystem(s), e.g., computer subsystem 36 and/or computer subsystem(s) 102, include neural network 104. The neural network is configured for determining inverted features of input images in a training set for a specimen input to the neural network. For example, as shown in FIG. 2, image 200 may be input to neural network 202, which determines inverted features 204 for the image. In this manner, a neural network is used to approximate the inversion function f.sup.−1( ), and the neural network generates the inverted features from the input image. In the context of semiconductor applications such as inspection, metrology, and defect review, the neural network described herein can be used to solve inverse problems in imaging formation (e.g., diffraction, interference, partial coherence, blurring, etc.) to regenerate optically-corrected features. “Inverted features” (where inverted is related to the context of an inversion neural network) are generally defined herein as features after inverting a physical process and “features” are defined as generally referring to measurable properties including, but not limited to, intensity, amplitude, phase, edge, gradients, etc.”, Note that the measurable properties of intensity, amplitude, phase, edge, gradients, etc. are mapped to the identified features, and the inversion function f.sup.−1( ) is mapped to “invert the one or more image transformations”); and
using the one or more neural networks to identify (See Bozchalooi: Figs. 2-3, and [0035], “FIG. 3 is a diagram of a memory augmented neural network 300 configured for testing and operation. Following training as discussed above in relation to FIG. 2, and omitting discussion of elements discussed with respect to FIG. 2 to avoid redundancy, a memory augmented neural network can be configured for testing and operation by adding Mahalanobis distance and Kullback-Leibler (KL) divergence testing block (MKLD) 318 and fully connected layers (FC) 320 to produce output states (OUT) 322. Mahalanobis distance is discussed in relation to FIG. 4, and KL divergence is discussed in relation to FIG. 5. The memory augmented neural network 300 can be tested by submitting test images (TI) 302 to a trained transform bank (TB) 204 from FIG. 2, which produces image variations (IV) 306 based on the test images 302. The test images 302 include ground truth and can be from the same dataset that was used to train the memory augmented object detection system 200. In some examples a training dataset can be divided into two portions, where the first portion is used to train a neural network and the second portion is used to test the neural network. Because the memory augmented neural network 300 increases the number of test images 302 in the same fashion as the training images 202, the memory augmented neural network 300 improves testing by generating large numbers of image variants 306 based on a small number of test images 302, thereby reducing the need to generate large numbers of test images and corresponding ground truth"):
one or more objects within the one or more images (See Yu: Figs. 1 and 12, and [0072], "In at least one embodiment, semantic correspondence 112 refers to one or more processes that determine regions of images 102, 104 that are semantically related. In at least one embodiment, regions of images 102, 104 that are semantically related refer to regions of images that belong to a same class of object, such as car, person, plant, and/or variations thereof. In at least one embodiment, semantic correspondence 112 refers to one or more processes that determine correspondence between pixels or sets of pixels of images 102, 104 (e.g., pixels or sets of pixels that are semantically related) by extracting and comparing features of said images 102, 104. In at least one embodiment, for semantic correspondence 112, a framework 106 obtains and processes one or more images (e.g., image 102 and/or image 104) to determine correspondence between objects of said one or more images 102, 104 (e.g., determining if one or more objects of a first image are depicted in a second image). In at least one embodiment, a framework 106 extracts features of an image, identifies objects of said image by analyzing said features, and compares identified objects from said image with other identified objects of other images to determine correspondence between objects of said image and said other images. In at least one embodiment, referring to FIG. 1, semantic correspondence 112 comprises one or more processes that determine correspondence between a car object depicted in a first image 102 and a first car object and a second car object depicted in a second image 104. In at least one embodiment, referring to FIG. 1, semantic correspondence 112 comprises one or more processes that determine correspondence between a first car object and/or a second car object depicted in a second image 104 and a car object depicted in a first image 102") based, at least in part, on the one or more features of the one or more modified versions of the one or more images (See Chadha: Fig. 2, and [0030], "The input image 205 and the modified input images 210 may be processed by an object detection machine learning model. In FIG. 2, the machine learning model is an object detection neural network 220, but other types of machine learning (e.g., deep learning) models may be used. The object detection neural network 220 may perform various processes such as object recognition, image classification, object localization, object segmentation, etc. for each of the input image 205 and the modified input images 210") with the inverted one or more image transformations (See Zhang: Figs. 2-4, and [0099], "In an additional embodiment, the one or more computer subsystems are configured to input a runtime image for the specimen or another specimen into the trained neural network such that: the trained neural network determines the inverted features for the runtime image; the forward physical model reconstructs the runtime image from the inverted features determined for the runtime image; and the residue layer determines differences between the runtime image and the reconstructed runtime image, where the inverted features are features of an optically corrected version of the runtime image, and the differences between the runtime image and the reconstructed runtime image are features of a residue image").
Regarding claim 10, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 9 as outlined above. Further, Chadha teaches that the method of claim 9, further comprising:
generating one or more additional images based, at least in part, on the one or more features of the one or more modified versions of the one or more images (See Chadha: Fig. 2, and [0032], "According to aspects described herein, post processing and filtering may include confidence distribution normalization 245 and non-max suppression (NMS) thresholding per modification 250. According to confidence distribution normalization 245, the system may normalize the confidence distributions for each of the range of confidence scores corresponding to the modified input images 210 based on the range of confidence scores for the input image 205").
Regarding claim 11, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 9 as outlined above. Further, Chadha teaches that the method of claim 9, further comprising generating one or more features of the one or more images using one or more convolutional layers of the one or more neural networks (See Chadha: Fig. 2, and [0031], "The object detection neural network 220 may identify sets of objects (e.g., an object identified by a bounding box 225) for each processed image and confidence scores associated with the identified objects. That is, the object detection neural network 220 may identify a range (e.g., distribution) of confidence scores for each processed image, where each confidence score corresponds to an identified object from one of the processed images. Certain post processing and filtering algorithms 230 may perform certain procedures to output detected objects. For example, the post processing and filtering algorithms 230 may resize bounding boxes to original scale and filter confidence scores and associated objects". Note that the scale, bounding box, confidence scores, etc., may be the additional features for the modified images).
Regarding claim 12, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 9 as outlined above. Further, Chadha teaches that the method of claim 9, wherein the one or more modified versions of the one or more images are generated by at least using one of random roll, color modification, or affine transformation (See Chadha: Fig. 2, and [0036], "According to NMS thresholding per modification 250, the system may identify objects corresponding to overlapping bounding boxes per modified input image 210 and select the highest score from the overlapping boxes for consideration. This technique may be performed prior to resizing images/bounding boxes to the original scale (or changing other parameters of the modified input images 210 to correspond to the input image 205, such as removing transformations, removing brightness modifications, etc.). The object detection neural network 220 may identify bounding boxes that outline detected objects. In some cases, the object detection neural network 220 may identify bounding boxes that correspond to the same object in an image, and as such, may overlap to some degree. Further, each bounding box may have a corresponding confidence score associated therewith that corresponds to the confidence assigned by the object detection neural network 220 in detection of an object. In some systems, each of the bounding boxes (whether overlapping or not) for the modified input images 210 may be rescaled along with rescaling of the modified input images 210 to the original scale corresponding to the input image 205. Thereafter, the system may identify overlapping bounding boxes (rescaled and original) and select a highest score (or filter the lowest score) from the overlapping bounding boxes. According to the techniques described herein, the system selects the highest (or filters one or more of the lowest) confidence scores (and the corresponding bounding boxes/objects) for each modified input image 210 (before any rescaling or removing modifications). As such, the most relevant confidence score per modified input image 210 may be identified before rescaling. In some cases, this may avoid utilizing processing resources for rescaling or removing modifications for a set of bounding boxes, since some may be removed from consideration prior to the rescaling/removing of modifications. For example, when the image manipulation techniques include generating multiple scaled images (as illustrated in FIG. 2), the system may perform NMS thresholding per scaled image, including the original input image 205 and the manipulated input images 210").
Regarding claim 13, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 9 as outlined above. Further, Yu teaches that the method of claim 9, wherein the one or more neural networks are deployed in one or more autonomous vehicles (See Yu: Figs. 11A-D, and [0152], "FIG. 11A illustrates an example of an autonomous vehicle 1100, according to at least one embodiment. In at least one embodiment, autonomous vehicle 1100 (alternatively referred to herein as "vehicle 1100") may be, without limitation, a passenger vehicle, such as a car, a truck, a bus, and/or another type of vehicle that accommodates one or more passengers. In at least one embodiment, vehicle 1100 may be a semi-tractor-trailer truck used for hauling cargo. In at least one embodiment, vehicle 1100 may be an airplane, robotic vehicle, or other kind of vehicle").
Regarding claim 14, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 9 as outlined above. Further, Yu teaches that the method of claim 9, wherein at least one of the one or more images is a bird eye view image (See Yu: Figs. 7 and 11A-D, and [0116], "In at least one embodiment, a system performing at least a part of process 700 includes executable code to obtain 702 training data comprising two or more images. In at least one embodiment, training data comprises images depicting one or more objects and associated bounding box annotations. In at least one embodiment, a bounding box annotation comprises one or more indications of a location (e.g., coordinates, borders) of an object of an image and a classification (e.g., category) of said object of said image. In at least one embodiment, training data comprises images such as those captured from a view of an autonomous vehicle, such as an autonomous car, unmanned aerial vehicle (UAV), and/or variations thereof").
Regarding claim 15, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 1 as outlined above. Further, Yu, Chadha, Bozchalooi, and Zhang teach that a system (See Yu: Fig. 1 and 12, and [0256], "FIG. 12 is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SOC) or some combination thereof formed with a processor that may include execution units to execute an instruction, according to at least one embodiment") comprising:
one or more processors (See Yu: Fig. 1 and 12, and [0256], "FIG. 12 is a block diagram illustrating an exemplary computer system, which may be a system with interconnected devices and components, a system-on-a-chip (SOC) or some combination thereof formed with a processor that may include execution units to execute an instruction, according to at least one embodiment")to:
use one or more neural networks to apply one or more image transformations to one or more images to generate one or more modified versions of the one or more images (See Bozchalooi: Figs. 2-3, and [0036], “During testing and operation, the memory augmented neural network 300 generates a large number (>100) of image variants 306 based on the transform bank 204. The image variants 306 are passed one at a time through the neural network (NN) 308, where they are processed as described above in relation to FIG. 2 to form feature point sets for each image variant that can be stored as feature variant sets (FVS) 310. As described above, set mean (SM) 312 and inverse covariance (IC) 314 are determined based on each feature variant set 310 for each test image 302 input to the transform bank 204. The set mean 312 and inverse covariance 314 are input to Mahalanobis distance and KL divergence testing block (MKLD) 318 to determine which feature variant set 210 stored in memory 216 the current feature variant set 310 matches most closely”);
invert the one or more image transformations for one or more identified features in the one or more images (See Zhang: Figs. 1-4, and [0058], “The component(s), e.g., component(s) 100 shown in FIG. 1, executed by the computer subsystem(s), e.g., computer subsystem 36 and/or computer subsystem(s) 102, include neural network 104. The neural network is configured for determining inverted features of input images in a training set for a specimen input to the neural network. For example, as shown in FIG. 2, image 200 may be input to neural network 202, which determines inverted features 204 for the image. In this manner, a neural network is used to approximate the inversion function f.sup.−1( ), and the neural network generates the inverted features from the input image. In the context of semiconductor applications such as inspection, metrology, and defect review, the neural network described herein can be used to solve inverse problems in imaging formation (e.g., diffraction, interference, partial coherence, blurring, etc.) to regenerate optically-corrected features. “Inverted features” (where inverted is related to the context of an inversion neural network) are generally defined herein as features after inverting a physical process and “features” are defined as generally referring to measurable properties including, but not limited to, intensity, amplitude, phase, edge, gradients, etc.”, Note that the measurable properties of intensity, amplitude, phase, edge, gradients, etc. are mapped to the identified features, and the inversion function f.sup.−1( ) is mapped to “invert the one or more image transformations”); and
use the one or more neural networks (See Yu: Figs. 9 and 12, and [0265], "Inference and/or training logic 815 are used to perform inferencing and/or training operations associated with one or more embodiments. Details regarding inference and/or training logic 815 are provided herein in conjunction with FIGS. 8A and/or 8B. In at least one embodiment, inference and/or training logic 815 may be used in system FIG. 12 for inferencing or predicting operations based, at least in part, on weight parameters calculated using neural network training operations, neural network functions and/or architectures, or neural network use cases described herein") to identify one or more objects within the one or more images (See Yu: Figs. 1 and 12, and [0072], "In at least one embodiment, semantic correspondence 112 refers to one or more processes that determine regions of images 102, 104 that are semantically related. In at least one embodiment, regions of images 102, 104 that are semantically related refer to regions of images that belong to a same class of object, such as car, person, plant, and/or variations thereof. In at least one embodiment, semantic correspondence 112 refers to one or more processes that determine correspondence between pixels or sets of pixels of images 102, 104 (e.g., pixels or sets of pixels that are semantically related) by extracting and comparing features of said images 102, 104. In at least one embodiment, for semantic correspondence 112, a framework 106 obtains and processes one or more images (e.g., image 102 and/or image 104) to determine correspondence between objects of said one or more images 102, 104 (e.g., determining if one or more objects of a first image are depicted in a second image). In at least one embodiment, a framework 106 extracts features of an image, identifies objects of said image by analyzing said features, and compares identified objects from said image with other identified objects of other images to determine correspondence between objects of said image and said other images. In at least one embodiment, referring to FIG. 1, semantic correspondence 112 comprises one or more processes that determine correspondence between a car object depicted in a first image 102 and a first car object and a second car object depicted in a second image 104. In at least one embodiment, referring to FIG. 1, semantic correspondence 112 comprises one or more processes that determine correspondence between a first car object and/or a second car object depicted in a second image 104 and a car object depicted in a first image 102") based, at least in part, on the one or more features of the one or more modified versions of the one or more images (See Chadha: Fig. 2, and [0030], "The input image 205 and the modified input images 210 may be processed by an object detection machine learning model. In FIG. 2, the machine learning model is an object detection neural network 220, but other types of machine learning (e.g., deep learning) models may be used. The object detection neural network 220 may perform various processes such as object recognition, image classification, object localization, object segmentation, etc. for each of the input image 205 and the modified input images 210") with the inverted one or more image transformations (See Zhang: Figs. 2-4, and [0099], "In an additional embodiment, the one or more computer subsystems are configured to input a runtime image for the specimen or another specimen into the trained neural network such that: the trained neural network determines the inverted features for the runtime image; the forward physical model reconstructs the runtime image from the inverted features determined for the runtime image; and the residue layer determines differences between the runtime image and the reconstructed runtime image, where the inverted features are features of an optically corrected version of the runtime image, and the differences between the runtime image and the reconstructed runtime image are features of a residue image").
Regarding claim 16, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 15 as outlined above. Further, Chadha teaches that the system of claim 15, wherein the one or more processors are further to generate additional one or more images based, at least in part, on one or more features of the one or more images and one or more features of one or more modified versions of the one or more images (See Chadha: Fig. 2, and [0032], "According to aspects described herein, post processing and filtering may include confidence distribution normalization 245 and non-max suppression (NMS) thresholding per modification 250. According to confidence distribution normalization 245, the system may normalize the confidence distributions for each of the range of confidence scores corresponding to the modified input images 210 based on the range of confidence scores for the input image 205").
Regarding claim 17, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 15 as outlined above. Further, Yu teaches that the system of claim 15, wherein the one or more processors are further to generate one or more features of one or more modified versions of the one or more images using one or more attention modules of the one or more neural networks (See Yu: Fig. 1, and [0068], "In at least one embodiment, a framework 106 for object detection 108, instance segmentation 110, and semantic correspondence 112 formulates instance segmentation learning as a teacher-student self-distillation problem where a teacher is guided by multiple instance learning and CRF smoothing. In at least one embodiment, boxes contain rich information about object masks as they tightly enclose objects. In at least one embodiment, for a framework 106 for object detection 108, instance segmentation 110, and semantic correspondence 112, foreground/background localization naturally emerges as network attention under box supervision. In at least one embodiment, CRF can promote inductive bias on contrast-sensitive smoothness. In at least one embodiment, for a framework 106 for object detection 108, instance segmentation 110, and semantic correspondence 112, self-training/distillation with pseudo-labels can considerably benefit weakly and semi-supervised learning tasks").
Regarding claim 18, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 15 as outlined above. Further, Yu teaches that the system of claim 15, wherein one or more first cell positions of one or more features of the one or more images match with one or more second cell positions of one or more features of the one or more modified versions of the one or more images (See Yu: Figs. 11A-D, and [0194], "In at least one embodiment, accelerator(s) 1114 can have a wide array of uses for autonomous driving. In at least one embodiment, a PVA may be used for key processing stages in ADAS and autonomous vehicles. In at least one embodiment, a PVA's capabilities are a good match for algorithmic domains needing predictable processing, at low power and low latency. In other words, a PVA performs well on semi-dense or dense regular computation, even on small data sets, which might require predictable run-times with low latency and low power. In at least one embodiment, such as in vehicle 1100, PVAs might be designed to run classic computer vision algorithms, as they can be efficient at object detection and operating on integer math").
Regarding claim 19, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 15 as outlined above. Further, Chadha teaches that the system of claim 15, wherein the one or more processors are further to generate the modified versions of the one or more images using at least one of random roll, affine transformation, or pixel removing (See Chadha: Fig. 2, and [0036], "According to NMS thresholding per modification 250, the system may identify objects corresponding to overlapping bounding boxes per modified input image 210 and select the highest score from the overlapping boxes for consideration. This technique may be performed prior to resizing images/bounding boxes to the original scale (or changing other parameters of the modified input images 210 to correspond to the input image 205, such as removing transformations, removing brightness modifications, etc.). The object detection neural network 220 may identify bounding boxes that outline detected objects. In some cases, the object detection neural network 220 may identify bounding boxes that correspond to the same object in an image, and as such, may overlap to some degree. Further, each bounding box may have a corresponding confidence score associated therewith that corresponds to the confidence assigned by the object detection neural network 220 in detection of an object. In some systems, each of the bounding boxes (whether overlapping or not) for the modified input images 210 may be rescaled along with rescaling of the modified input images 210 to the original scale corresponding to the input image 205. Thereafter, the system may identify overlapping bounding boxes (rescaled and original) and select a highest score (or filter the lowest score) from the overlapping bounding boxes. According to the techniques described herein, the system selects the highest (or filters one or more of the lowest) confidence scores (and the corresponding bounding boxes/objects) for each modified input image 210 (before any rescaling or removing modifications). As such, the most relevant confidence score per modified input image 210 may be identified before rescaling. In some cases, this may avoid utilizing processing resources for rescaling or removing modifications for a set of bounding boxes, since some may be removed from consideration prior to the rescaling/removing of modifications. For example, when the image manipulation techniques include generating multiple scaled images (as illustrated in FIG. 2), the system may perform NMS thresholding per scaled image, including the original input image 205 and the manipulated input images 210").
Regarding claim 20, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 15 as outlined above. Further, Yu teaches that the system of claim 15, wherein at least one of the one or more images is three-dimensional (3D) image (See Yu: Figs. 11A-D, and [0164], "In at least one embodiment, one or more camera may be mounted in a mounting assembly, such as a custom designed (three-dimensional ("3D") printed) assembly, in order to cut out stray light and reflections from within vehicle 1100 (e.g., reflections from dashboard reflected in windshield mirrors) which may interfere with camera image data capture abilities. With reference to wing-mirror mounting assemblies, in at least one embodiment, wing-mirror assemblies may be custom 3D printed so that a camera mounting plate matches a shape of a wing-mirror. In at least one embodiment, camera(s) may be integrated into wing-mirrors. In at least one embodiment, for side-view cameras, camera(s) may also be integrated within four pillars at each corner of a cabin").
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Yu, etc. (US 20220261593 A1) in view of Chadha, etc. (US 20210150282 A1), further in view of Bozchalooi, etc.( US 20220137634 A1), Zhang, etc. (US 20170351952 A1), and Kato, etc. (US 20090257665 A1).
Regarding claim 5, Yu, Chadha, Bozchalooi, and Zhang teach all the features with respect to claim 1 as outlined above. However, Yu, modified by Chadha, Bozchalooi, and Zhang, fails to explicitly disclose that the one or more processors of claim 1, wherein the one or more features of the one or more images are generated based, at least in part, on inverting the one or more modified versions of the one or more images.
However, Kato teaches that the one or more processors of claim 1, wherein the one or more features of the one or more images are generated based, at least in part, on inverting the one or more modified versions of the one or more images (See Kato: Fig. 1, and [0003], "The encoding device according to Non Patent Document 1 inverts the input image by choosing from four types of options, namely, no inversion, vertical inversion, horizontal inversion, and vertical and horizontal inversion. (a) of FIG. 1 shows the input image; (b) of FIG. 1 shows an image resulting from the non-inversion of the input image; (c) of FIG. 1 shows an image obtained by inverting the input image in a vertical direction; (d) of FIG. 1 shows an image obtained by inverting the input image in a horizontal direction; and (e) of FIG. 1 shows an image obtained by inverting the input image in a vertical and horizontal direction. Furthermore, in inter-predictive encoding which makes predictions between frames, the encoding device inverts the reference image which is a locally decoded image of an image which has been previously encoded in the same direction as the input image and performs encoding by using the inverted reference image to generate a predictive signal for the inverted input image constituting an encoding target").
Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention was effectively filed to modify Yu to have the one or more processors of claim 1, wherein the one or more features of the one or more images are generated based, at least in part, on inverting the one or more modified versions of the one or more images as taught by Kato in order to provide a better chance of detecting an object or classifying the image (See Kato: Fig. 7, and [0007], "The present invention was conceived in order to solve the above problems and an object thereof is to provide, by generating a precise predictive signal, a moving image encoding device which more efficiently encodes image information, a moving image encoding method, a moving image encoding program, a moving image decoding device which more efficiently decodes encoded image information, a moving image decoding method, and a moving image decoding program"). Yu teaches a method and system that may detect and identify objects in the image using neural network with two input images, one source image and one reference image; while Kato teaches a system and method that may modify input images in various ways including inverting the input image to generate the exact image prediction signal and encoding and decoding the image efficiently. Therefore, it is obvious to one of ordinary skill in the art to modify Yu by Kato to modify the input images in various ways including inverting the input images. The motivation to modify Yu by Kato is "Use of known technique to improve similar devices (methods, or products) in the same way".
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights