DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) filed on June 6, 2025, August 12, 2025, and December 3, 2025, have been considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 2, 7-8, 11, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Witzgall et al. (US Pub No 20230010033), hereinafter Witzgall, in view of Ghiasi et al. (Ghiasi, et al., "Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation", IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp 2917 – 2927), hereinafter Ghiasi.
As to Claim 1, Witzgall teaches a system (see Title, “Method and System For Accelerating Rapid Class Augmentation for Object Detection in Deep Neural Networks”, comprising:
a processing system; and memory coupled to the processing system, the memory comprising computer executable instructions that, when executed by the processing system, causes the system to perform operations comprising (see paragraph [0094], “It is submitted that one skilled in the art would understand the various computing environments, including computer readable mediums, which may be used to implement the methods described herein. Selection of computing environment and individual components may be determined in accordance with… processing requirements, security requirements and the like. It is submitted that one or more steps or combinations of steps of the methods described herein may be developed locally or remotely, i.e., on a remote physical computer”):
training an object detection model, which has been pre-trained with a first set of object classes, with a new object class (see paragraph [0010], “a computer-implemented process for augmenting an object detection architecture for detecting objects in an image, includes: training an object detection architecture trained to detect for n object classes to detect for n+c object classes”, where the ‘n object class’ is interpreted as the first set of object classes, the ‘c object class’ is interpreted as the new object class), by:
applying, to each of a plurality of first images that each depicts an object corresponding to an object class among the first set of object classes to generate a plurality of augmented images, a set of data augmentations (see paragraph [0047], “One of the changes from the original model is that the images are uniformly resized to 480 (height) by 640 (width) regardless of their original aspect size. Resizing could cause some distortion in object sizes. To compensate, images may be padded in one-dimension after resizing to minimize this distortion”);
and training the object detection model using the plurality of augmented images(see paragraph [0048], “The features produced from prediction heads 25 a, 25 b, and 25 c are used to train XRCA-YOLOv3 box-class weights to predict bounding box coordinates and object classification”).
Witzgall fails to teach the augmented set of data is obtained combining each first image with at least one second image among a plurality of second images that each depicts a second object corresponding to the new object class.
However, in an analogous art, Ghiasi teaches a method of image augmentation to create training data for a model (see pg. 2, Section 1., paragraph 1, “The key idea behind the Copy-Paste augmentation is to paste objects from one image to another image. This can lead to a combinatorial number of new training data”),
wherein an augmented set of data is obtained combining each first image with at least one second image among a plurality of second images that each depicts a second object (see pg. 2, Section 1., paragraph 1, “The key idea behind the Copy-Paste augmentation is to paste objects from one image to another image”), and see pg.3, Section 3., “We randomly select two images”, thus implying there is a plurality of images),
corresponding to another class (see pg. 2, caption, under Figure 2, “We use a simple copy and paste method to create new images for training instance segmentation models”, and see Fig. 2, shown below, where a first image containing object belonging to a first class and a second image containing objects of a second class are combined)
PNG
media_image1.png
572
1328
media_image1.png
Greyscale
Figure 2 of Ghiasi
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the image augmentation taught by Ghiasi with the teachings of object detection model taught by Witzgall in order to combine each first image with at least one second image that depicts a second object corresponding to the new object class. The motivation for doing so would be to increase efficiency by automating the creation of training data. Ghiasi teaches on pg. 1, Section 1, “At the same time, annotating large datasets for instance segmentation [40, 21] is usually expensive and time consuming. For example, 22 worker hours were spent per 1000 instance masks for COCO [40]. It is therefore imperative to develop new methods to improve the data-efficiency. Here, we focus on data augmentation [50] as a simple way to significantly improve the data-efficiency”. Thus, it would have been obvious to combine the image augmentation taught by Ghiasi with the teachings of Witzgall in order to obtain the invention as claimed in Claim 1.
As to Claim 2, Witzgall in view of Ghiasi teaches wherein the set of data augmentations includes one of or a combination of two or more of: numerical representation similarity-based data augmentations; image insertion -based data augmentations; or image assemblage -based data augmentations (see Ghiasi, pg. 2, Section 1., paragraph 1, “The key idea behind the Copy-Paste augmentation is to paste objects from one image to another image”, where pasting objects has been interpreted as an image insertion-based augmentation). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the image augmentation taught by Ghiasi with the teachings of object detection model taught by Witzgall. The motivation for doing so would be to increase efficiency by automating the creation of training data (see Ghiasi, pg. 1, Section 1). Thus, it would have been obvious to combine the image augmentation taught by Ghiasi with the teachings of Witzgall in order to obtain the invention as claimed in Claim 2.
As to Claim 7, Witzgall in view of Ghiasi teaches wherein training the object detection model includes retaining weights associated with the first set of object classes (see Witzgall, paragraph [0033], “Accordingly, an XRCA optimization model consists of three components: 1) a weight matrix, 2) an inverse feature covariance matrix, 3) and a null-class vector. The weight matrix maps a network's features to its existing classes”) .
and randomizing weights of the new object class (see Witzgall, paragraph [0089], “In the SGD implementation, the augmented new class vector algorithm is initialized with a scaled random (but pre-selected, for repeatability) weight”).
As to Claim 8, Witzgall in view of Ghiasi teaches the object detection model has a general architecture including a backbone portion and a head portion, the backbone portion being configured to detect or identify general features in an image and to generate a numerical representation for each general feature (see Witzgall, paragraph [0011], “In a second exemplary embodiment, an object detection architecture for detecting objects in an image, includes: an object detection backbone including a feature extractor, the feature extractor including one or more prediction heads for predicting features in the image”, and see paragraph [0075], “Note that when an image is passed through the SSD feature extraction backbone 50, which consists of various convolutional layers and down samplings, it creates six different multi-scale feature maps (FMAPs)”, where the ‘feature maps’ are numerical representations of features),
the head portion being configured to perform at least one of classification, confidence determination, or bounding box detection (see paragraph [0012], “one or more prediction head models trained to classify n known objects using training data, including the multiple feature maps”).
As to Claim 11, Witzgall in view of Ghiasi teaches a computer-implemented method (see paragraph Witzgall, paragraph [0010], “In a first exemplary embodiment, a computer-implemented process for augmenting an object detection architecture for detecting objects in an image”, comprising:
training an object detection model, which has been pre-trained with a first set of object classes, with a new object class (see paragraph [0010], “a computer-implemented process for augmenting an object detection architecture for detecting objects in an image, includes: training an object detection architecture trained to detect for n object classes to detect for n+c object classes”, where the ‘n object class’ is interpreted as the first set of object classes, the ‘c object class’ is interpreted as the new object class), by:
applying, to each of a plurality of first images that each depicts an object corresponding to an object class among the first set of object classes, data augmentations (see paragraph [0047], “One of the changes from the original model is that the images are uniformly resized to 480 (height) by 640 (width) regardless of their original aspect size. Resizing could cause some distortion in object sizes. To compensate, images may be padded in one-dimension after resizing to minimize this distortion”);
and training the object detection model using the plurality of augmented images (see paragraph [0048], “The features produced from prediction heads 25 a, 25 b, and 25 c are used to train XRCA-YOLOv3 box-class weights to predict bounding box coordinates and object classification”).
Witzgall fails to teach the augmented set of data is obtained combining each first image with at least one second image among a plurality of second images that each depicts a second object corresponding to the new object class.
However, in an analogous art, Ghiasi teaches a method of image augmentation to create training data for a model (see pg. 2, Section 1., paragraph 1, “The key idea behind the Copy-Paste augmentation is to paste objects from one image to another image. This can lead to a combinatorial number of new training data”),
wherein an augmented set of data is obtained combining each first image with at least one second image among a plurality of second images that each depicts a second object (see pg. 2, Section 1., paragraph 1, “The key idea behind the Copy-Paste augmentation is to paste objects from one image to another image”), and see pg.3, Section 3., “We randomly select two images”, thus implying there is a plurality of images),
corresponding to another class (see pg. 2, caption, under Figure 2, “We use a simple copy and paste method to create new images for training instance segmentation models”, and see Fig. 2, where a first image containing object belonging to a first class and a second image containing objects of a second class are combined).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the image augmentation taught by Ghiasi with the teachings of object detection model taught by Witzgall in order to combine each first image with at least one second image that depicts a second object corresponding to the new object class. The motivation for doing so would be to increase efficiency by automating the creation of training data (see Ghiasi, pg. 1, Section 1.) Thus, it would have been obvious to combine the image augmentation taught by Ghiasi with the teachings of Witzgall in order to obtain the invention as claimed in Claim 11.
As to Claim 18, Claim 18 claims the same limitation claimed as Claim 7 and is dependent on a similarly rejected independent claim. Therefore, the rejection and rationale are similar to that of Claim 7.
As to Claim 19, Claim 19 claims the same limitation claimed as Claim 8 and is dependent on a similarly rejected independent claim. Therefore, the rejection and rationale are similar to that of Claim 8.
As to Claim 20, Witzgall in view of Ghiasi teaches wherein training the object detection model includes training at least the backbone using an adapter (see paragraph [0088], “As discussed above with respect to the XRCA-YOLOv3 implementation, there are benefits to using XRCA optimization for progressive learning compared to SGD optimization. To facilitate the comparison, both optimizers use the same feature extraction backbone architecture and labeling conventions, and both are given the same new class training data (comprised of both positive object and hard negative training examples).”, where the backbone is trained by the XRCA optimization adapter)
that freezes pre-trained weights corresponding to the first set of object classes (see paragraph [0033], “Accordingly, an XRCA optimization model consists of three components: 1) a weight matrix, 2) an inverse feature covariance matrix, 3) and a null-class vector. The weight matrix maps a network's features to its existing classes”, and thus the weights corresponding to first set of classes are retained or ‘frozen’)
and stores additional weight changes corresponding to the new object class in a matrix T(see paragraph [0033], “The null-class vector is used to initialize a new class with the prior information on what the new class is not.”, and see paragraph [0040], “In this stage, the old XRCA model matrix wk (number of features F×number of old classes NbC) is augmented with a new class initialization vector Δwk of size F by 1, to form the new augmented model”, where the new class data with weight changes is stored in the matrix wk ).
Claims 3 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Witzgall et al. (US Pub No 20230010033), hereinafter Witzgall, in view of Ghiasi et al. (Ghiasi, et al., "Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation", IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp 2917 – 2927), hereinafter Ghiasi, and further in view of Guignard et al., US Pat No 11954943), hereinafter Guignard.
As to Claim 3, Witzgall in view of Ghiasi fails to teach applying the numerical representation similarity -based data augmentations comprises: selecting the at least one second image based on a similarity between a numerical representation of each of the at least one second image and a numerical representation of each first image, wherein each numerical representation is representative of at least one of color, texture, contrast, brightness, or pixel values; and combining each first image with at least one second image based on numerical representation-based selection of images to generate the plurality of augmented images.
However, in an analogous art, Guignard teaches a method of creating synthetic image data (see abstract, “The embodiments are directed to generating synthetic data”),
which comprises selecting the at least one second image based on a similarity between a numerical representation of each of the at least one second image and a numerical representation of each first image and combining each first image with at least one second image based on numerical representation-based selection of images to generate the plurality of augmented images (see Col. 3, lines 52-60, “the step of insertion can consist, for each 2D image, in inserting a generated synthetic object in the 2D image only when the generated synthetic object and the 2D image present the same brightness conditions. In this case, a detection of the background of each 2D image is carried out and a classification is made. The combination between one synthetic object and one 2D image is performed by comparison of the intensity of the background brightness and that of the lighting of the object”),
wherein each numerical representation is representative of at least one of color, texture, contrast, brightness, or pixel values (see Col. 4, lines 10-12 “an average brightness is calculated for each image. Each 2D image is associated with a brightness score”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the numerical representation taught by Guignard with the synthetic image generation method taught by Ghiasi. The motivation for doing so would be to create multiple sets of training data that represent different conditions to prevent over-training. Guignard teaches in Col. 1, lines 57-61 , “In theory, the samples should be varied: hands of different ethnicities and sizes, different environments and changing light conditions. If these variations are not respected, the result of the learning called “model” may be over-trained “, and in Col. 3, lines 34-40, “Thus, the 2D images can be used as training dataset for a machine learning. Several thousand of 2D images can be created in few minutes. The method of the invention is faster than the manual method according to the prior art. Creating thousands of poses of synthetic objects allows for a large amount of images available under various circumstances”. Thus, it would have been obvious to combine the numerical representation taught by Guignard with the synthetic image generation method taught by Witzgall in view of Ghiasi in order to obtain the invention as claimed in Claim 3.
As to Claim 14, Claim 14 claims the same limitation claimed as Claim 3 and is dependent on a similarly rejected independent claim. Therefore, the rejection and rationale are similar to that of Claim 3.
Claims 4 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Witzgall et al. (US Pub No 20230010033), hereinafter Witzgall, in view of Ghiasi et al. (Ghiasi, et al., "Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation", IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp 2917 – 2927), hereinafter Ghiasi, further in view of Ge et al., (Ge, et al., "DALL-E for Detection: Language-driven Context Image Synthesis for Object Detection", arXiv:2206.09592v1, June 20, 2022, 28 Pages), hereinafter Ge, and further in view of Guignard et al., (US Pat No 11954943), hereinafter Guignard.
As to Claim 4, Witzgall in view of Ghiasi teaches applying the image insertion -based data augmentations comprises: retrieving a plurality of third images that depict the second object corresponding to the new object class, the plurality of third images including the plurality of second images; for each of the plurality of third images, modifying the third image to remove its background, leaving the second object depicted in the third image; and for each first image of the plurality of first images, selecting one of the modified third images for insertion in the first image based on a similarity between a numerical representation of the one of the modified third images and a numerical representation of the first image; identifying a background portion of the first image over which to insert the one of the modified third images; and inserting the one of the modified third images in the first image to overlay the identified background portion of the first image to generate one of the plurality of augmented images.
However, in an analogous art, Ge teaches retrieving a plurality of images that depict the object corresponding to an object class, the plurality of third images including the plurality of second images (see pg. 4, caption under Figure 2, “Our pipeline consists of foreground generation and context background generation. (a) Foreground generation (top row): (1) we fill the interest class name (e.g., dog) into fixed prompt templates to produce foreground sentences. (2) We then feed the sentences to DALL-E (or Stable diffusion) to generate high quality foreground images with easy to separate background”, where the generated foreground image of an object class is interpreted as the retrieved image);
modifying the third image to remove its background, leaving the second object depicted in the third image (see pg. 4, caption under Figure 2, “(3) We use off-the-shelf image segmentation methods to extract foreground segments from foreground images”);
and for each first image of the plurality of first images, selecting one of the modified third images for insertion into a first image (see pg.5, Section 3.3., “At each step a group of foreground object masks is selected and pasted into a sampled back ground image,”);
identifying a background portion of the first image over which to insert the one of the modified third images and inserting the one of the modified third images in the first image to overlay the identified background portion of the first image to generate one of the plurality of augmented images
(see pg. 5, Section 3.3, “The foreground mask, after 2D geometric augmentation such as rotation and scaling, is pasted on a random location in the image”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the image retrieval and insertion taught by Ge with the object detection system taught by WITZGALL IN VIEW OF GHIASI. The motivation for doing so would be to create high quality images for training. Ge teaches on pg. 2, Section 1., “We again leverage DALL-E to generate diverse set of high quality context images. At the core of our approach lies utilizing an interplay between language description of con text and language-driven image generation. Given a small number of images that represent the context environment, we use image captioning to generate a high-level language description of the context automatically. The language description of the context is used within a text-to-image generation pipeline (e.g., DALL-E) to generate a diverse set of images. These diverse sets of generated images are used as context images.”
Ge fails to explicitly teach that the image is selected on insertion based on a similarity between a numerical representation.
However, Guignard teaches a method of creating synthetic images which comprises selecting the an image based on a similarity between a numerical representation and a numerical representation of each first image and combining each first image with at least one second image based on numerical representation-based selection of images to generate the plurality of augmented images (see Col. 3, lines 52-60, “the step of insertion can consist, for each 2D image, in inserting a generated synthetic object in the 2D image only when the generated synthetic object and the 2D image present the same brightness conditions. In this case, a detection of the background of each 2D image is carried out and a classification is made. The combination between one synthetic object and one 2D image is performed by comparison of the intensity of the background brightness and that of the lighting of the object”), and see Col. 4, lines 10-12 “an average brightness is calculated for each image. Each 2D image is associated with a brightness score”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the numerical representation taught by Guignard with the synthetic image generation method taught by Ghiasi. The motivation for doing so would be to create multiple sets of training data that represent different conditions to prevent over-training (see Guignard Col. 1, lines 57-61). Thus, it would have been obvious to combine the numerical representation taught by Guignard with the synthetic image generation method taught by Witzgall, Ghiasi, and Ge in order to obtain the invention as claimed in Claim 4.
As to Claim 15, Claim 15 claims the same limitation claimed as Claim 4 and is dependent on a similarly rejected independent claim. Therefore, the rejection and rationale are similar to that of Claim 4.
Claims 5-6 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Witzgall et al. (US Pub No 20230010033), hereinafter Witzgall, in view of Ghiasi et al. (Ghiasi, et al., "Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation", IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp 2917 – 2927), hereinafter Ghiasi, further in view of Zeng et al. (US Pub No 20250166237), hereinafter Zeng, and further in view of Guignard et al., (US Pat No 11954943), hereinafter Guignard.
As to Claim 5, Witzgall in view of Ghiasi fails to teach wherein applying the image assemblage -based data augmentations comprises: for each first image of the plurality of first images, retrieving at least one fourth image among the plurality of second images based on a similarity between a numerical representation of each of the at least one fourth image and a numerical representation of the first image; and generating a fifth image as an image assemblage that combines the first image with the at least one fourth image.
However, in an analogous art, Zeng teaches a method of generating images (see abstract),
which comprises retrieving an image among the plurality of second images based on a similarity between a numerical representation of each of the at least one fourth image and a numerical representation of the first image (see paragraph [0075], “In at least one embodiment, neural network 205 generates images for an image set 206 and connects a subject from each text description data set to said images forming text image pairs (e.g., to generate training data)”, where the Examiner has interpreted the generated images as the retrieved images,);
and generating a fifth image as an image assemblage that combines the first image with the at least one fourth image (see paragraph [0121], “In at least one embodiment, neural network 205 receives as input the subject text description data set and generates 708 a collage or set of images 206 corresponding to the one or more subjects of the subject description data set. In at least one embodiment, neural network 205 parses each subject of the subject set 203 {xn, xn+1, xn+2, . . . N} from the subject text description data set and generates the collage or set of images 206 for image set {zl, zl+1, zl+2 . . . . L}. Each of the images that make up the image set 206 are images of the input 201 subject in different poses” where the Examiner has interpreted the ‘collage’ as the image assemblage, and the collage includes input image 201, and see paragraph [0084], “In at least one embodiment inputs 402 are text, an image, or a combination of text and images (e.g., inputs 101, 201)”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the image assemblage taught by Zeng with the object detection system taught by Witzgall in view of Ghiasi. The motivation for doing so would be to improve models by generating images models can use for training. Zeng teaches in paragraph [0002], “For example, it can be challenging to generate training data (e.g., labeled images) that can be used to train a neural network to identify objects (e.g., animals) in images…If neural networks are not trained sufficiently and/or sufficient processing power is not available to train them, neural networks may not generate accurate outputs…Accordingly, there exists a need to improve neural networks that generate images as well as ways to improve training of these neural networks”).
Zeng fails to teach retrieving the image among the based on a similarity between a numerical representation of each of the at least one image and a numerical representation of the first image.
However, in an analogous art, Guignard teaches selecting an image for augmentation based on a similarity between a numerical representation of a first and second image(see Col. 3, lines 52-60, “the step of insertion can consist, for each 2D image, in inserting a generated synthetic object in the 2D image only when the generated synthetic object and the 2D image present the same brightness conditions. In this case, a detection of the background of each 2D image is carried out and a classification is made. The combination between one synthetic object and one 2D image is performed by comparison of the intensity of the background brightness and that of the lighting of the object”),
Abd see Col. 4, lines 10-12 “an average brightness is calculated for each image. Each 2D image is associated with a brightness score”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the numerical representation taught by Guignard with the synthetic image generation method taught by Ghiasi. The motivation for doing so would be to create multiple sets of training data that represent different conditions to prevent over-training (see Guignard Col. 1, lines 57-61). Thus it would have been obvious to combine the numerical selection taught by Guignard with the image assemblage method taught by Witzgall in view of Ghiasi and Zeng in order to obtain the invention as claimed in Claim 5.
As to Claim 6, Witzgall in view of Ghiasi and Guignard fails to teach wherein retrieving at least one fourth image comprises: for each first image, generating a first prompt comprising the first image; providing the first prompt to a large language model ("LLM") -driven text-to-image retrieval system or an LLM-driven text-to-image generation system; and receiving, from the LLM-driven text-to-image retrieval system or the LLM-driven text-to-image generation system, the at least one fourth image.
However, Zeng teaches : for each first image, generating a first prompt comprising the first image (see paragraph [0106], “In at least one embodiment, neural network 600 receives one or more inputs 601 (e.g., inputs 402) and one or more prompts 602. In at least one embodiment, inputs 601 can be text, images, or a combination of text and images”);
providing the first prompt to an LLM-driven text-to-image generation system (see paragraph [0068], “In at least one embodiment a subject of the inputs 201 is an object of two more outputs 209 (e.g., a text includes fox, and images each include that fox). In at least one embodiment, neural network 202 can include a diffusion neural network, transformer neural network, large language model, or other language processing model”;
and receiving, from the LLM-driven text-to-image retrieval system or the LLM-driven text-to-image generation system, the at least one fourth image (see paragraph [0075, “In at least one embodiment, neural network …generates images for an image set 206 and connects a subject from each text description data set to said images forming text image pairs (e.g., to generate training data)”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the LLM taught by Zeng with the object detection system taught by Witzgall, Ghiasi, and Guignard. The motivation for doing so would be to use LLM to retrieve images that coincide with user preferences. Zeng teaches in paragraph (see paragraph [0068], “In at least one embodiment, inputs 101 can be provided by a user (e.g., through a user interface), a server, a different neural network, or other source of text inputs (e.g., received from a large language model, user interface). In at least one embodiment, inputs 101 indicate content of, at least, two or more different images”. Thus, it would have been obvious to combine the image assemblage generation taught by Zeng with the teachings of Witzgall, Ghiasi, and Guignard in order to obtain the invention as claimed in Claim 6.
As to Claim 16, Claim 16 claims the same limitation claimed as Claim 5 and is dependent on a similarly rejected independent claim. Therefore, the rejection and rationale are similar to that of Claim 5.
As to Claim 17, Claim 17 claims the same limitation claimed as Claim 6 and is dependent on a similarly rejected independent claim. Therefore, the rejection and rationale are similar to that of Claim 6.
Claims 9-10 and 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Witzgall et al. (US Pub No 20230010033), hereinafter Witzgall, in view of Ghiasi et al. (Ghiasi, et al., "Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation", IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp 2917 – 2927), and further in view of Kaul et al. (P. Kaul et al., "Label, Verify, Correct: A Simple Few Shot Object Detection Method," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 14217-14227), hereinafter Kaul, and further in view of Bagherinezhad et al. (US Pub No 20190325269), hereinafter Bagherinezhad.
As to Claim 9, Witzgall in view of Ghiasi fails to explicitly teach performing automatic labeling of images including at least one of the plurality of first images or the plurality of second images, by: for each object class for each image, applying a second object detection model to the image to detect and to label an object corresponding to the object class; cropping the image, leaving the object depicted in the image; using a classifier model to detect and to label an object in the cropped image; determining whether labels by the second object detection model and by the classifier model agree; and performing one of: based on a determination that the labels by the second object detection model and by the classifier model agree within a natural language tolerance, adding the label to the image; or based on a determination that the labels by the second object detection model and by the classifier model do not agree within the natural language tolerance, identifying adversarial image examples for the object class.
However, in an analogous art Kaul teaches applying an object detection model to the image to detect and to label an object corresponding to the object class (see pg. 14220, Section 4.1, “Specifically, the detector from Novel Training (as described in Section 3.2), is used to perform inference on the training images (D) to generate a set of candidate detections, each containing a class label and predicted bounding box coordinates”);
cropping the image, leaving the object depicted in the image (see pg. 14221, Section 4.2, “In detail, to compute the feature of a given annotation/candidate detection, we first use the bounding box to crop the relevant image”);
using a model to detect and to label an object in the cropped image (see pg. 14221, Section 4.2, “This crop is then resized and passed as input to the self-supervised model”;
determining whether labels by the object detection model and by the classifier model agree; and performing one of: based on a determination that the labels by the second object detection model and by the classifier model agree within a natural language tolerance, adding the label to the image (see pg. 14220, Section 4.2, “We adopt a simple verification policy: a given candidate detection is accepted (or verified) if our kNN classifier, using cosine similarity, predicts the same class as the predicted class label from the detector”, where the cosine similarity is interpreted as the ‘natural language tolerance’)
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the object detection model and label verification taught by Kaul with the object detection method taught Witzgall in view of Ghiasi. The motivation for doing so would be to improve the accuracy of detection and labelling of objects within an image. Kaul teaches on pg. 14218, Section 1., “To summarise, our contributions are as follows: (i) we carefully examine the problem of few-shot object detection with the modern two-stage object detector, e.g. Faster R CNN, and identify the issue of “supervision collapse”; (ii) we introduce a novel verification and correction procedure to pseudo-labelling, which significantly improves the precision of pseudo-annotations, both class labels and bounding box coordinates”. Thus, it would have been obvious to combine the object detector and label verification taught by Kaul with the teachings of Witzgall and Zeng.
Kaul fails to explicitly teach based on a determination that the labels by the second object detection model and by the classifier model do not agree within the natural language tolerance, identifying adversarial image examples for the object class. Instead the label is simply not accepted (see pg. 4, Section 4.2).
However, in an analogous art, Bagherinezhad teaches a model for object classification (see )
comprising a first neural network and a second neural network (see paragraph [0007], “The computing system may then apply a first neural network to the set of crops to obtain a set of respective outputs. In particular embodiments, the computing system may then train a second neural network using the set of crops as training examples and using the set of respective outputs as labels for the set of crops”)
which based on a determination that the labels by the first neural network and second neural network do not agree, identifies adversarial image examples for the object class (see paragraph [0056], “The embodiments disclosed herein experiment with using the Label Refinery in conjunction with the network being trained in order to generate adversarial examples on which the two networks disagree”).
Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the adversarial image identification taught by Bagherinezhad with the label verification method taught by Kaul. The motivation for doing so would be to use the identified adversarial image examples to train and improve the object detection network. Bagherinezhad teaches in paragraph [0064], “These models may be further improved by training with adversarial inputs (Table 3). All network architectures that were tried using Label Refineries gained significant accuracy improvement over their previous state-of-the-art. Alex Net and ResNetXnor-50”. Thus, it would have been obvious to combine the adversarial image identification taught by Bagherinezhad with the teachings of Witzgall, Ghiasi, and Kaul in order to obtain the invention as claimed in Claim 9.
As to Claim 10, Witzgall, Ghiasi, Kaul, Bagherinezhad teach that a classification model that is finetuned on crops of a labeled dataset of objects (see Kaul, pg. 14221, Section 5.2., “We apply RandomCrop and ColorJitter augmentations when finetuning”).
Thus it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the object detection model and label verification taught by Kaul with the object detection method taught Witzgall in view of Ghiasi. The motivation for doing so would be to improve the accuracy of detection and labelling of objects within an image (see Kaul, pg. 2, Section 1.). Thus, it would have been obvious to combine the teachings of Kaul with the teachings of Witzgall, Ghiasi and Bagherinezhad in order to obtain the invention as claimed in Claim 10.
As to Claim 12, Claim 12 claims the same limitation claimed as Claim 9 and is dependent on a similarly rejected independent claim. Therefore, the rejection and rationale are similar to that of Claim 9.
As to Claim 13, Claim 13 claims the same limitation claimed as Claim 10 and is dependent on a similarly rejected independent claim. Therefore, the rejection and rationale are similar to that of Claim 10.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Turgutlu (US Pub No 20230274478), teaches a method for inserting new objects into images through utilizing a large language model. The new object can belong to an unknown class.
Tullberg (US Pub No 20190005353) teaches a two step system for object detection comprising detecting an object, cropping the image to obtain a cropped image of the object, and then inputting the cropped image into an image classifier in order to obtain a label.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOUMYA THOMAS whose telephone number is (571)272-8639. The examiner can normally be reached M-F 8:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.T./ Examiner, Art Unit 2664
/JENNIFER MEHMOOD/ Supervisory Patent Examiner, Art Unit 2664
Prosecution Insights