Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to under 37 CFR 1.83(a) because “DEFECT 202” and “NON-PRODUCT 206”in FIG. 2 should be “DEFECT 206” and “NON-PRODUCT 202” as described in the specification ([0013]). Any structural detail that is essential for a proper understanding of the disclosed invention should be shown in the drawing. MPEP § 608.02(d). Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-5, 7, 9, 11-13, 15, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Jia (PCT Patent Pub. No.: WO 2024/206624 A1) hereinafter Jia, in view of Bowman (US Patent Pub. No.: US 2026/0057647 A1), hereinafter Bowman, further in view of Jagadeesan (US Patent No.: US 12,586,369 B1), hereinafter Jagadeesan, further in view of Yu (US Patent Pub. No.: US 2024/0378230 A1), hereinafter Yu.
Regarding claim 1, Jia teaches a system comprising: one or more image data sources (The operations include obtaining a training input image that depicts an object. [0008]); a computer system comprising at least one hardware processor and a non-transitory computer-readable medium storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations (The computer system includes one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computer system to perform operations. [0008]) comprising: accessing one or more training images (The operations include obtaining a training input image that depicts an object. [0008]), each training image including a depiction of a product (The operations include obtaining a training input image that depicts an object. [0008]) and also having a corresponding mask (For example, a mask can be applied to the input image to generate a masked image 16. [0032]); passing the one or more training images (The operations include obtaining a training input image that depicts an object. [0008]) and corresponding masks (For example, a mask can be applied to the input image to generate a masked image 16. [0032]) to a generative artificial intelligence (GAI) model with instructions to causing the GAI model to generate a plurality of images (The operations include processing the object embedding and the text embedding with a generative model to generate a synthetic image. [0008]), based on one or more parameters (The operations include modifying one or more parameter values of at least the generative model based on the reconstruction loss function. [0008]); repeating using different one or more parameters, until iteration criteria are satisfied (Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations. [0075]); generating a plurality of synthesized images using the GAI model (Referring still to Figure 1, the computing system can process the object embedding 20 and the text embedding 24 with a machine-learned generative model 26 to generate a synthetic image 28. [0034]) and one or more parameters (The operations include modifying one or more parameter values of at least the generative model based on the reconstruction loss function. [0008]).
Jia does not teach the following limitations as further recited, but Bowman further teaches wherein the corresponding mask provides a label for one or more portions of the training image (For instance, to identify and measure scratches on an object, the training manager 308 may use synthetic images labeled with bounding boxes or segmentation masks to guide the training process, with each bounding box or each pixel in an image is classified as belonging to a certain class (in this case, for example, "scratch" or "no scratch"). [0047]); passing the one or more training images and corresponding masks (For instance, to identify and measure scratches on an object, the training manager 308 may use synthetic images labeled with bounding boxes or segmentation masks to guide the training process, with each bounding box or each pixel in an image is classified as belonging to a certain class (in this case, for example, "scratch" or "no scratch"). [0047]) to a generative artificial intelligence (GAI) model with instructions to causing the GAI model to generate a plurality of images that depict a defect in the product (Fig. 6 620 Generate, using the generative model, one or more images including a representation of the object, having the type of defect, under one or more environmental conditions); training a segmentation model using a first set of the plurality of synthesized defect images (The input to the defect detection models may be images and the output is a classification or a segmentation map. [0046]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jia to incorporate the teachings of Bowman to pass the training images and corresponding masks with a label for portions of the training image to a generative artificial intelligence (GAI) model with instructions to causing the GAI model to generate images that depict a defect in the product and train a segmentation model using a first set of the plurality of synthesized defect images in order to generate a diverse array of synthetic images, such as can be used to train defect detection models.
The combination of Jia and Bowman does not teach the following limitations as further recited, but Jagadeesan further teaches saving the one or more parameters as a checkpoint (The checkpointing module 140 can perform checkpointing at any training epoch. Checkpointing is a technique used to save the current state of the AI model 132 and analyze the performance of the AI model 132. Column 6 line 44); repeating the passing, saving, inputting, and calculating for a different iteration (In some embodiments, checkpointing includes applying the AI model 132 in training to the training subset 182 to determine a train loss value, applying the AI model 132 in training to the holdout subset 184 to determine a test loss value, and determining whether both the train loss value and test loss value converge at a same epoch. Column 6 line 53); selecting a checkpoint (Responsive to determining that convergence occurred at a particular epoch, the system 100 selects the AI model trained at the particular epoch. Column 6 line 65) based on a comparison corresponding to each checkpoint (In some embodiments, checkpointing includes applying the AI model 132 in training to the training subset 182 to determine a train loss value, applying the AI model 132 in training to the holdout subset 184 to determine a test loss value, and determining whether both the train loss value and test loss value converge at a same epoch. Column 6 line 53).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Jagadeesan to select a checkpoint with saved parameters based on a comparison corresponding to each checkpoint in order to improve model training efficiency because such a modification is the result of combining prior art elements according to known methods to yield predictable results.
The combination of Jia, Bowman, and Jagadeesan does not teach the following limitations as further recited, but Yu further teaches inputting each of the plurality of images that depict a defect in the product into an embedding machine learning model to generate synthesized image embeddings, comprising a different embedding for each image in the plurality of images (At block 302, the input natural language's text embedding is generated. If an input image is used, then its image embedding is generated. [0041]); passing each of the one or more training images into the embedding machine learning model to generate training image embeddings, comprising a different embedding for each image in the one or more training images (At block 303, image embeddings are obtained from a retrieval system (e.g., block 204 in FIG. 2). [0043]. For example, embedding generated at block 302 is in the same format (e.g., 512-dimension feature vector) as the image embeddings obtained at block 303. [0041]); calculating a mean of the synthesized image embeddings (At block 203, for each entity, mean embeddings are generated from image embeddings related to that entity. For example, mean embedding involves averaging feature vectors of images, which creates a single vector that represents the overall "meaning" of the image. [0037]); calculating a mean (At block 307, similarity calculation is performed between the mean embedding of the selected images and the entity mean embeddings (from block 308) via similarity calculation (e.g., similarity calculation used in block 304). For example, the selected images are compared with the entities' images, which involves comparing "mean" of user selected images and "mean" of entities' images. [0048]) of the training image embeddings (At block 204, entity mean embeddings are also stored in the information retrieval system. For example, information retrieval system may implement a vector database. [0038]); selecting a checkpoint (At block 305, user selects relevant images from the search results generated from block 304. For example, a user interface is provided for user selection in UI. In various embodiments, similarity calculation at block 304 and image selection at block 305 may be performed by different systems (e.g., a server in a datacenter for block 304, and a personal computing device for block 305). [0046]) based on a comparison of the mean of the training image embeddings and the mean of the synthesized image embedding corresponding to each checkpoint (At block 307, similarity calculation is performed between the mean embedding of the selected images and the entity mean embeddings (from block 308) via similarity calculation (e.g., similarity calculation used in block 304). For example, the selected images are compared with the entities' images, which involves comparing "mean" of user selected images and "mean" of entities' images. [0048]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Yu to select a checkpoint based on a comparison of the mean of the calculated training image embeddings and the mean of the calculated synthesized image embedding corresponding to each checkpoint in order to provide more efficient image-to-image search because such a modification is the result of combining prior art elements according to known methods to yield predictable results.
Regarding claim 3, Bowman in the combination teaches the system of claim 1, wherein the GAI model (Fig. 6 620 Generate, using the generative model, one or more images including a representation of the object, having the type of defect, under one or more environmental conditions) is a Large Language Model (LLM) (systems implemented using language models such as large language models (LLMs) or vision language models (VLMs) [0021]).
Regarding claim 4, Jia in the combination teaches the system of claim 1, wherein the GAI model is a diffusion model (The embeddings are then passed to the generative model 26 (e.g., which can be an augmented text-to-image diffusion model, to generate the final output synthetic image 28. [0029]).
Regarding claim 5, Bowman in the combination teaches the system of claim 4, wherein the operations further comprise: cropping the one or more training images based on corresponding masks (For instance, to identify and measure scratches on an object, the training manager 308 may use synthetic images labeled with bounding boxes or segmentation masks to guide the training process, with each bounding box or each pixel in an image is classified as belonging to a certain class (in this case, for example, "scratch" or "no scratch"). [0047]); and wherein the plurality of synthesized defect images are each combined with a background image (An augmentation module 430 in accordance with at least one embodiment may augment the simulated environment models and defect models to generate comprehensive synthetic images. [0045]) prior to being used to train (The training manager 308 may train defect detection models using the synthetic images generated from the image generation module 306 as training data. [0046]) the segmentation model (The output of a CNN model include bounding boxes labeled with predicted classifications. In one embodiment, the output may be a segmentation map for a segmentation task, where each pixel of the image is classified with labels. [0046]).
Regarding claim 7, Yu in the combination teaches the system of claim 1, wherein the comparison includes performing a cosine similarity function (At block 304, similarity calculation is performed between the input text/image embedding (from block 302) and image embeddings (from block 303) stored in the information retrieval system via non-metric space searching algorithm (e.g., NMSLIB), which allows for fast similarity calculation. As mentioned above, cosine similarity or other similarity calculations may be used. [0044]) on the mean of the training image embeddings (At block 204, entity mean embeddings are also stored in the information retrieval system. For example, information retrieval system may implement a vector database. [0038]) and the mean of the synthesized image embeddings corresponding to each checkpoint (At block 203, for each entity, mean embeddings are generated from image embeddings related to that entity. For example, mean embedding involves averaging feature vectors of images, which creates a single vector that represents the overall "meaning" of the image. [0037]).
Method claims 9, 11-13 and 15 are drawn to the method of using the corresponding apparatus claimed in claims 1, 3-5 and 7. Therefore method claims 9, 11-13 and 15 correspond to apparatus claims 1, 3-5 and 7 and are rejected for the same reasons of obviousness as used above.
Claims 17-20 are drawn to a non-transitory machine-readable storage medium having executable instructions stored for executing the method of using the corresponding apparatus as claimed in claims 1-4. Therefore, claims 17-20 correspond to apparatus claims 1-4, and are rejected for the same reasons of obviousness as used above.
Claims 2 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Jia (PCT Patent Pub. No.: WO 2024/206624 A1) hereinafter Jia, in view of Bowman (US Patent Pub. No.: US 2026/0057647 A1), hereinafter Bowman, further in view of Jagadeesan (US Patent No.: US 12,586,369 B1), hereinafter Jagadeesan, further in view of Yu (US Patent Pub. No.: US 2024/0378230 A1), hereinafter Yu, further in view of Junjiang (Chinese Patent Pub. No.: CN 111259898 B), hereinafter Junjiang.
Regarding claim 2, Jia, Bowman, Jagadeesan and Yu teach all of the elements of the claimed invention as stated in claim 1 except for expressly teaching the following limitations as further recited. However, Junjiang teaches wherein the operations further comprise: validating the segmentation model ((4e) Inputting the data image in the verification set into a crop semantic segmentation network with current parameters, calculating a cross entropy loss function value of the network on the verification set, and comparing the cross entropy loss function value with a set optimal cross entropy loss function value of the verification set: if the cross entropy loss function value is smaller than the set verification set optimal cross entropy loss function value, updating the verification set optimal cross entropy loss function value to be the cross entropy loss function value on the verification set obtained by the calculation in the current round, and storing the current network model, namely the optimal model in the current training. Page 9 9th paragraph) using a second set of images (dividing the enhanced data set according to a proportion to obtain a training set and a validation set, wherein the training set and the validation set both comprise data images and label images, the division proportion of the training set and the validation set in the data set is generally empirical and can be set as 8: a scale of 2 sets the scale of the training set and the validation set. Page 8 14th paragraph).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jia, Bowman, Jagadeesan and Yu to incorporate the teachings of Junjiang to validate the segmentation model using a second set of defect images in order to obtain an optimal segmentation model because such a modification is the result of combining prior art elements according to known methods to yield predictable results.
Bowman further teaches using a set of defect images (Through the integration of these synthetic defects, a synthetic defect generation system can generate a diverse array of synthetic images, such as can be used to train defect detection models. Abstract).
Method claim 10 is drawn to the method of using the corresponding apparatus claimed in claim 2. Therefore method claim 10 corresponds to apparatus claim 2 and is rejected for the same reasons of obviousness as used above.
Claims 6 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Jia (PCT Patent Pub. No.: WO 2024/206624 A1) hereinafter Jia, in view of Bowman (US Patent Pub. No.: US 2026/0057647 A1), hereinafter Bowman, further in view of Jagadeesan (US Patent No.: US 12,586,369 B1), hereinafter Jagadeesan, further in view of Yu (US Patent Pub. No.: US 2024/0378230 A1), hereinafter Yu, further in view of Wang (Improving Text Embeddings with Large Language Models, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916, August 11-16, 2024), hereinafter Wang.
Regarding claim 6, Jia, Bowman, Jagadeesan and Yu teach all of the elements of the claimed invention as stated in claim 1 except for the following limitations as further recited. However, Wang teaches wherein the embedding machine learning model is part of an LLM (This implies that extensive auto-regressive pre-training enables LLMs to acquire good text representations, and only minimal fine-tuning is required to transform them into effective embedding models. Page 11904 left column 2nd paragraph).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jia, Bowman, Jagadeesan and Yu to incorporate the teachings of Wang to utilize an embedding machine learning model which is part of an LLM in order to generate more diverse synthetic data by LLMs.
Method claim 14 is drawn to the method of using the corresponding apparatus claimed in claim 6. Therefore method claim 14 corresponds to apparatus claim 6 and is rejected for the same reasons of obviousness as used above.
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Jia (PCT Patent Pub. No.: WO 2024/206624 A1) hereinafter Jia, in view of Bowman (US Patent Pub. No.: US 2026/0057647 A1), hereinafter Bowman, further in view of Jagadeesan (US Patent No.: US 12,586,369 B1), hereinafter Jagadeesan, further in view of Yu (US Patent Pub. No.: US 2024/0378230 A1), hereinafter Yu, further in view of Xie (The Unsupervised Feature Selection Algorithms Based on Standard Deviation and Cosine Similarity for Genomic Data Analysis, Frontiers in Genetics, May 2021 | Volume 12), hereinafter Xie.
Regarding claim 8, Jia, Bowman, Jagadeesan and Yu teach all of the elements of the claimed invention as stated in claim 7 except for the following limitations as further recited. However, Xie teaches wherein the operations further comprise: at each iteration, computing a standard deviation (The Unsupervised Feature Selection Algorithms Based on Standard Deviation and Cosine Similarity for Genomic Data Analysis. Abstract); and wherein the selecting is based on a combination of output of the cosine similarity function and the standard deviation (To tackle this challenging task, this paper will focus on the feature selection problem for genomic data analysis under an unsupervised learning scenario. It will propose the unsupervised feature selection technique based on the standard deviation and the cosine similarity of variables. Page 3 left column last paragraph).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Jia and Yu to incorporate the teachings of Xie so that the selecting is based on a combination of output of the cosine similarity function and the computed standard deviation for each corresponding checkpoint in order to detect the features with both high discernibility and high independence with low computational load when the detected feature subset is sparse while representative because such a modification is the result of combining prior art elements according to known methods to yield predictable results.
Jia further teaches of the synthesized image (The method includes processing, by the computing system, the object embedding and the text embedding with a machine-learned generative model to generate a synthetic image, wherein the synthetic image depicts the object in combination with the desired content. [0007]).
Yu further teaches of the image embeddings (At block 203, for each entity, mean embeddings are generated from image embeddings related to that entity. For example, mean embedding involves averaging feature vectors of images, which creates a single vector that represents the overall "meaning" of the image. [0037]).
Jagadeesan further teaches for the corresponding checkpoint (The checkpointing module 140 can perform checkpointing at any training epoch. Checkpointing is a technique used to save the current state of the AI model 132 and analyze the performance of the AI model 132. Column 6 line 44).
Method claim 16 is drawn to the method of using the corresponding apparatus claimed in claim 8. Therefore method claim 16 corresponds to apparatus claim 8 and is rejected for the same reasons of obviousness as used above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEI ZHAO whose telephone number is (703)756-1922. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VU LE can be reached at (571)272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LEI ZHAO/Examiner, Art Unit 2668
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights