DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5, 7-11 and 13-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over HOFFMAN et al. (US 20250308225 A1), referred herein as HOFFMAN in view of LIU et al. (US 20240282016 A1), referred herein as LIU.
Regarding Claim 1, HOFFMAN in view of SON teaches a data augmentation method, comprising (HOFFMAN Abst: Systems and methods… to generate a plurality of augmented images):
obtaining first field images captured in a first field and second field images captured in a second field (HOFFMAN [0004] plurality of first images that each depicts an object corresponding to an object class among the first set of object classes… a plurality of second images that each depicts a second object),
wherein a number of the first field images is greater than a number of the second field images (HOFFMAN [0024] The trained object detection model 125 may be used in either or both of image editing system 180 or video editing system 190 with results of labeled objects corresponding to both the old object classes and the new object class(es) being stored in image repository(ies) 185 or video repository(ies) 195, respectively); video is made up of many still images (frames) displayed in quick succession, while a single image (like a photograph) is just one still picture. video repository is interpreted as the first field images.
cropping first object images and second object images respectively from the first field images and the second field images (HOFFMAN [0028] at operation 215, cropping the image, leaving the object depicted in the image);
HOFFMAN disclosed new object class, but does not teach new object images. However, LIU teaches
training an object generation model with the first object images and the second object images (LIU [0025] computing system(s) 105, orchestrator(s) 110, and/or AI system 120 may perform methods for implementing training of a pre-trained object detection model for detecting new object classes, as described in detail with respect to FIGS. 2-4);
generating new object images by the object generation model (LIU [0039] FIG. 3 schematically shows a data flow of an example training phase of the machine learning diffusion model 128. A set of training images 300 is generated to train the image encoder 200. In some examples, the set of training images 300 includes images of faces of different people; FIG. 3: training phase);
HOFFMAN in view of LIU further teaches
synthesizing the new object images into the second field images as new object field images (HOFFMAN [0035] inserting the one of the modified second images in the first image to overlay the identified background portion of the first image to generate one of the plurality of augmented images (at operation 325). In some cases, selecting the one of the modified second images (at operation 315) may be based on a similarity between a numerical representation of the one of the modified second images and a numerical representation of the first image. In some cases, operations 315-325 may be repeated for the other first images of the plurality of first images, as denoted by long-dashed arrow forming a loop from operation 325 back to operation 315 and through operations 315-325.); and
training an object discrimination model with the second field images and the new object field images (HOFFMAN [0025] computing system(s) 105, orchestrator(s) 110, and/or AI system 120 may perform methods for implementing training of a pre-trained object detection model for detecting new object classes, as described in detail with respect to FIGS. 2-4).
LIU discloses a systems and methods for generating a synthesized image of a user with a trained machine learning diffusion model, which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified HOFFMAN to incorporate the teachings of LIU, and apply a trained machine learning diffusion model into the systems and methods provided for implementing training of a pre-trained object detection model for detecting new object classes.
Doing so would be capable of achieving more diversified generated content.
Regarding Claim 2, HOFFMAN in view of LIU teaches the data augmentation method of claim 1, and further teaches further comprising:
performing a style transfer on the new object field images into style-transferred new object field images (LIU [0012] The synthesized images can have different artistic styles, and the people depicted in the synthesized images can have different clothes and body gestures);
training the object discrimination model with the second field images, the new object field image, and the style-transferred new object field images (LIU [0040] the set of training images 300 includes a plurality of synthesized training images 302 that are generated based at least on an initial training image of the set of training images 300. Each of the plurality of synthesized training images 302 is modified relative to the initial training image by one or more of rotating, horizontally flipping, and/or cropping the initial training image to generate the synthesized training image).
Regarding Claim 3, HOFFMAN in view of LIU teaches the data augmentation method of claim 1, and further teaches further comprising:
grouping the first object image and the second object image into a plurality of groups (HOFFMAN [0028] sequence flow 200A includes, for each object class for each image, applying a second object detection model (e.g., second object detection model 135a or 135b of FIG. 1) to the image (in this case, image 210) to detect and to label an object corresponding to the object class (at operation 205). Labeling results by the second object detection model may be as depicted by bounding box 210a as shown, e.g., in FIG. 2A. Sequence flow 200A further includes, at operation 215, cropping the image, leaving the object depicted in the image, as depicted by image 220 in FIG. 2A. At operation 225, sequence flow 200A further includes using a classifier model to detect and to label an object in the cropped image);
training the object generation models corresponding to the plurality of groups with the first object images and the second object images (LIU [0043] The set of training embeddings 306 (and the set of synthesized word embeddings 310 when applicable) are fed to the text encoder 202. The text encoder 202 generates a set of training feature vectors based at least on the set of training embeddings 306 (and the set of synthesized word embeddings 310 when applicable). The set of training feature vectors 312 are fed to the diffusion model 204 and the diffusion model 204 generates a set of predicted synthesized images 314 based at least on the set of set of training feature vectors 312.); and
generating the new object images corresponding to the plurality of groups by the object generation models (LIU [0043] The set of training embeddings 306 (and the set of synthesized word embeddings 310 when applicable) are fed to the text encoder 202. The text encoder 202 generates a set of training feature vectors based at least on the set of training embeddings 306 (and the set of synthesized word embeddings 310 when applicable). The set of training feature vectors 312 are fed to the diffusion model 204 and the diffusion model 204 generates a set of predicted synthesized images 314 based at least on the set of set of training feature vectors 312).
Regarding Claim 5, HOFFMAN in view of LIU teaches the data augmentation method of claim 1, and further teaches wherein the object generation model is trained with a text instruction, the first object images and the second object images (LIU [0017] the description text corresponding to the first object images and/or the second object images may be directly generated with an image-to-text generative AI model as text instructions. The object images and the text instructions are used as model input to train the object generation model so that the object generation model generates new object images that meet the needs; HOFFMAN [0039] training the object detection model (at operation 410) includes applying, to each of a plurality of first images that each depicts an object corresponding to an object class among the first set of object classes, a set of data augmentations to generate a plurality of augmented images (at operation 415) and training the object detection model using the plurality of augmented images (at operation 420)).
Regarding Claim 7 HOFFMAN in view of LIU teaches the data augmentation method of claim 1, and further teaches wherein synthesizing the new object images into the second field images is replacing or not replacing the second object images in the second field images with the new object images (HOFFMAN [0035] An example augmented image 340 is shown in FIG. 3A, in which a modified second image 335 (in this case, an image of a bear without its original background) is inserted in a background portion of the first image 330; LIU [0040] Each of the plurality of synthesized training images 302 is modified relative to the initial training image by one or more of rotating, horizontally flipping, and/or cropping the initial training image to generate the synthesized training image).
Regarding Claim 8 HOFFMAN in view of LIU teaches the data augmentation method of claim 1, and further teaches further comprising: synthesizing the new object images into third field images as the new object field images (HOFFMAN [0036] turning to FIG. 3B, sequence flow 300B includes applying the image assemblage-based data augmentations, which includes, for each third image of the plurality of third images, retrieving at least one fourth image based on a similarity between a numerical representation of each of the at least one fourth image and a numerical representation of the third image (at operation 380); [0042] [0042] In some examples, applying the image insertion-based data augmentations (with reference to the example of FIG. 4) includes retrieving a plurality of third images that depict the second object corresponding to the new object class, the plurality of third images including the plurality of second images).
Regarding Claims 9-11 and 13-15, HOFFMAN in view of SON teaches a data augmentation system, comprising (HOFFMAN Abst: Systems and methods… to generate a plurality of augmented images):
an image database for storing (HOFFMAN FIG. 1.185: image repository(ies); 195: video repository(ies))
a first image processing server for receiving the first field images and the second field images from the image database, wherein the first image processing server comprises (HOFFMAN FIG. 1.165: image server(s))
an image cropping module
a data augmentation server for receiving the first object images and the second object images from the first image processing server, and the data augmentation server comprises (HOFFMAN FIG. 1.150a: AI system, data augmentation system):
an object generation module configured to train (HOFFMAN FIG. 1.155a: AI system, LLM-Driven Text-to-image system)
a second image processing server for receiving the new object images from the data augmentation server, and the second image processing server comprises (HOFFMAN FIG. 1.180: image editing system; 190: video editing system):
an image synthesis module configured to synthesize (LIU FIG. 2.128: trained machine learning diffusion model)
and an object discrimination server for training (HOFFMAN [0025] AI system 120 may perform methods for implementing training of a pre-trained object detection model for detecting new object classes, as described in detail with respect to FIGS. 2-4).
The metes and bounds of the claim substantially correspond to the claimed limitations set forth in claims 1-3, 5, 7 and 8; thus they are rejected on similar grounds and rationale as their corresponding limitations.
Claim(s) 4 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over HOFFMAN et al. (US 20250308225 A1), referred herein as HOFFMAN in view of LIU et al. (US 20240282016 A1), referred herein as LIU and of Pakhomov et al. (US 20240135514 A1), referred herein as Pakhomov.
Regarding Claim 4, HOFFMAN in view of LIU teaches the data augmentation method of claim 1, but does not teach the claimed limitations herein. However, Pakhomov teaches
obtaining object position information of the second object images in the second field images (Pakhomov [0561] As further illustrated in FIG. 45A, the scene-based image editing system 106 determines relative positions 4514 for the objects 4504a-4504c within the digital image 4502.);
The prior art further teaches
synthesizing the new object images into the second field images as new object field images based on the object position information (HOFFMAN [0035] An example augmented image 340 is shown in FIG. 3A, in which a modified second image 335 (in this case, an image of a bear without its original background) is inserted in a background portion of the first image 330 (in this case, an image of living space in a residential building)).
Pakhomov discloses a systems and methods for generating a synthesized image of a user with a trained machine learning diffusion model, which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified HOFFMAN to incorporate the teachings of Pakhomov, and apply the trained machine learning diffusion model into the systems and methods provided for implementing training of a pre-trained object detection model for detecting new object classes.
Doing so would be capable of achieving more diversified generated content.
Regarding Claim 12, HOFFMAN in view of LIU teaches the data augmentation system of claim 9. The metes and bounds of the claim substantially correspond to the claimed limitations set forth in claim 4; thus they are rejected on similar grounds and rationale as their corresponding limitations.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over HOFFMAN et al. (US 20250308225 A1), referred herein as HOFFMAN in view of LIU et al. (US 20240282016 A1), referred herein as LIU and of BROKMAN et al. (US 20260065071 A1), referred herein as BROKMAN.
Regarding Claim 6, HOFFMAN in view of LIU teaches the data augmentation method of claim 5, but does not teach the claimed limitations herein. However, BROKMAN teaches
wherein the text instruction is generated by an image-to-text model with the first object images and the second object images (BROKMAN [0011] training a machine learning attribution model configured to provide data attribution to an output generation of a generative artificial intelligence (AI) model; [0044] the generative AI model is a diffusion model. Optionally, the diffusion model is an image-to-text diffusion model; [0107] the order of the concept groupings of the fine-tuning images 73 as based on the concepts 71 is the same as the order of the concept groupings of the output generated images 93 as based on the concepts of the prompt concepts 8. Note the number of fine-tuning images 73 in the concept group need not be the same as the number of output generated images 93 in the associated concept group, as shown in FIG. 5).
BROKMAN discloses a computer-implemented method of training a machine learning attribution model configured to provide data attribution to an output generation of a generative artificial intelligence (AI) model, which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified HOFFMAN to incorporate the teachings of BROKMAN, and apply the groups of images from the training data in generative AI models into the systems and methods provided for implementing training of a pre-trained object detection model for detecting new object classes.
Doing so would be capable of increasing accuracy and efficiency in determining the influence and/or contribution of input training images on the generation outputs.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Samantha (Yuehan) Wang whose telephone number is (571)270-5011. The examiner can normally be reached Monday-Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Samantha (YUEHAN) WANG/
Primary Examiner
Art Unit 2617
Prosecution Insights