Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1 and 9 are objected to because of the following informalities:
Both claims 1 and 9 recite “input of image information…”, which should read either “an input of image information…” or “input image information” for grammatical correctness. Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “input device”, “calculation device” and “storage device” in claim 9, and “output device” in claim 13.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4, 7, 9-10, 12, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (WIPO PG Pub 2025029057, claiming original priority from Korean application KR101230100706 filed 01 August 2023, and hereinafter “Lee”) in view of Sung et al. (US PG Pub 20200065992, hereinafter “Sung”).
Regarding claim 1, Lee discloses a method for data augmentation based on outpainting (para. 0043), the method comprising:
receiving, by an image processing apparatus, input of image information and prompt information (para. 0048, “In other words, the computing device can expand the input image in a direction determined based on the movement of the object requested by the user. To expand the input image, the computing device can perform outpainting using the first-generation model (1210) described above. In order for the first-generation model (1210) to perform the outpainting, the computing device can input a prompt instructing the first-generation model (1210) to perform the outpainting”); and
creating, by the image processing apparatus, data by using outpainting techniques and performing the data augmentation on the basis of the image information and the prompt information (para. 0048, “In other words, the computing device can expand the input image in a direction determined based on the movement of the object requested by the user. To expand the input image, the computing device can perform outpainting using the first-generation model (1210) described above. In order for the first-generation model (1210) to perform the outpainting, the computing device can input a prompt instructing the first-generation model (1210) to perform the outpainting”); and
converting and storing the prompt information into a database (para. 0134, wherein storing the programming instructions within the non-transitory medium would necessarily store instructions for generating the new image, including the user prompt, within a data structure).
Specifically, Lee discloses a method and system using outpainting for image editing and object identification, comprising detecting image boundaries and identifying whether objects are in proximity or contact to image or image segment boundaries, before outpainting in the direction of the object of interest.
Lee does not explicitly disclose wherein the data is training data, or:
training, by the image processing apparatus, an object detection model on the basis of the created training data;
evaluating, by the image processing apparatus, performance of the object detection model; and
converting and storing, by the image processing apparatus, performance evaluation results into a database.
However, Sung discloses:
wherein the data is training data (para. 0047, “ A recognition model training apparatus may generate augmented data by applying an augmentation process to input data, and may train the recognition model based on the augmented data. Also, a data recognition apparatus may perform a recognition based on augmented data that is generated by applying an augmentation process to input data”);
training, by the image processing apparatus, an object detection model on the basis of the created training data (para. 0047, “ A recognition model training apparatus may generate augmented data by applying an augmentation process to input data, and may train the recognition model based on the augmented data. Also, a data recognition apparatus may perform a recognition based on augmented data that is generated by applying an augmentation process to input data”);
evaluating, by the image processing apparatus, performance of the object detection model (paras. 0116 & 0118, “For example, in operation 940, the recognition model training apparatus calculates a contribution score of each augmentation process based on the output 939 calculated from the augmented data, using the recognition model. The contribution score is a score calculated from the output 939, and indicates a degree to which a corresponding augmentation process contributes to an accuracy of a recognition. For example, when a loss between the output 939 and a label decreases, a contribution score determined by the recognition model training apparatus increases”; wherein, under reasonable interpretation of the disclosure, the recognition method’s accuracy is used as a metric for the success of both the trained model as well as the augmentation procedure);
and converting and storing, by the image processing apparatus, performance evaluation results into a database (para. 0117, “. Also, the recognition model training apparatus calculates, from the augmented data, a contribution score of the augmentation process 981 selected for each training. The recognition model training apparatus generates a contribution histogram by classifying and accumulating contribution scores of selected augmentation processes 981 based on augmentation processes…[the] contribution histogram is normalized.”).
Specifically, Sung discloses a method and system of generating augmented data for training an object detection and recognition machine learning model, the augmentation generation method and system comprising any one of many trained ML models including artificial neural networks. Thus, one having ordinary skill in the art prior to the effective filing date of the claimed invention would have found it obvious to have utilized the outpainting-based augmentation method of Lee within the method and system of Sung as both a simple substitution of the data augmentation method as recognizable by an ordinarily skilled artisan, as well as an application of a known technique to a known device ready for improvement to yield the predictable result of an improved augmentation method for the object detection algorithm. More specifically, the outpainting method of Lee would enable low-variance in the environmental characteristics of the image input, allowing high-fidelity training data for accurate training and validation of the object recognition model.
Regarding claim 9, Examiner is interpreting the “input device” to be an interface capable of collecting user input and data (including algorithms) and funneling them to be processed downstream by a processor and stored within a processor-readable medium (according to paras. 0062-0064 of the specification), “calculation device” to be a processor (according to para. 0067 of the specification), and “storage device” as either an internal or external device such as a hard disk capable of storing a database (according to paras 0065-0066 of the specification).
Claim 9 is rejected, mutatis mutandis, for reasons similar to claim 1. Lee further discloses an input device (para. 0005, “According to one embodiment of the present disclosure, a computing device includes an input/output interface for receiving a user input requesting processing of an input image and outputting a reconstructed image processed according to the user input”), a calculation device (para. 0005, “and at least one processor for executing the instructions, wherein, when the at least one processor receives a user input requesting movement of at least one object included in the input image, the at least one processor expands the input image in a direction determined based on the movement of the object, determines a generation required area, which is an area requiring generation of a partial image of the at least one object, based on the expanded input image, and generates an image for the generation required area using at least one generative model, and then outputs a reconstructed image based on the input image and the partial image of the at least one object”), and a storage device (para. 0134, wherein storing the programming instructions within the non-transitory medium would necessarily store instructions for generating the new image, including the user prompt, within a data structure). Although Lee does not explicitly disclose storing the evaluation results, Sung does explicitly disclose this aspect (para. 0117). Thus, it would have been obvious to modify the disclosure of Lee with the saved evaluation results of Sung according to the rationale of claim 1.
Regarding claims 2 and 10, Lee in view of Sung discloses all limitations of claims 1 and 9, respectively. Lee further discloses an object recognition algorithm for determine areas for potential outpainting, wherein the image information of each image input into the method of Lee includes a type of an object contained within that image, and a position of the object contained in the image (para. 0089-0090, “As described above, the 'object proposal region' may mean a region in which an object (10) is judged or predicted to exist within an image… In the fourth operation (1d), the computing device can infer (determine) the first object suggestion area (100a) in the form of a bounding box”, and para. 0101, “Or, for example, the computing device may identify the object (10) by performing object recognition on an input image (e.g., identify the type of the object), and determine the region requiring generation of an image of the object (10) based on the identification result of the object”).
Regarding claims 4 and 12, Lee in view of Sung discloses all limitations of claim 1. Lee further discloses wherein the generated data is reference data and comprises information about a type of an object contained in the image and a position of the object contained in the image (para. 0096, “if the computing device infers (determines) a first object proposal region (100a) from the expanded input image in the fourth operation (1d), it can transform the object (10) and infer (determine) a second object proposal region (100b) corresponding to the transformed object (10) in the fifth operation” and para. 0103, “Meanwhile, according to one embodiment of the present disclosure, in the fifth operation (1e), the computing device may infer (determine) the entire second object proposal area (100b) as a region requiring generation. Accordingly, the computing device may generate a new image for the entire second object proposal area (100b) using the second generation model (1220) for object image generation.”. Paraphrasing, the newly created image regions (the training data, in context of the combination in view of Sung) has been created as the result of steps 1d and 1e, which utilize image object location and classification to generate the augmented image. As a result, the newly outpainted image objects will already have an associated location and type associated with them). Although Lee does not explicitly disclose wherein the data is training data, Sung explicitly discloses wherein the data is training data for training an object detection algorithm (para. 0047, “ A recognition model training apparatus may generate augmented data by applying an augmentation process to input data, and may train the recognition model based on the augmented data. Also, a data recognition apparatus may perform a recognition based on augmented data that is generated by applying an augmentation process to input data”). Thus, the combination of the disclosures of Lee and Sung would be obvious to the ordinarily skilled artisan according to the rationale of claim 1.
Regarding claims 7 and 15, Lee in view of Sung discloses all limitations of claim 1. Sung further explicitly discloses wherein the image processing apparatus further records the created training data in the database (para. 0133, “The local memory 1125 may be one or more temporary or local buffers/memories, while the memory 1130 may store a database from which kernel elements, feature maps, weight elements, voice elements, and/or image elements may be loaded from and into the local memory 1825.”).
Claims 3 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Sung and in further view of Cai et al. (“DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Models”, full citation in PTO-892 form, hereinafter “Cai”).
Regarding claims 3 and 11, Lee modified in view of Sung discloses all limitations of claims 1 and 9, respectively. Lee further discloses wherein the outpainting is performed by using an artificial neural network model (para. 0035, “A 'generative model' can refer to a neural network model that implements generative AI technology”).
Lee in view of Sung does not disclose wherein the artificial neural network model is a Stable Diffusion-based model
However, Cai discloses a neural network model based on the Stable Diffusion architecture (Abstract, “DiffDreamer”) which performs outpainting of an image (Abstract, “We demonstrate that image-conditioned diffusion models [(DiffDreamer)] can effectively perform long-range scene extrapolation while preserving consistency significantly better than prior GAN-based methods.”). Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have utilized the Stable Diffusion-based architecture of Cai within the method of Lee as modified by Sung as a simple substitution in the art, replacing the generic neural network of Lee while providing a measurable, predictable improvement of higher consistency compared to other potential architectures (Cai, Abstract, “while preserving consistency significantly better than prior GAN-based methods”).
Claims 5 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Sung and in further view of Cha et al. (US PG Pub 20250005901, claiming provisional priority with an effective filing date of 30 June 2023, and hereinafter “Cha”).
Regarding claim 13, Examiner is interpreting the “output device” to be an interface capable of collecting modified/augmented data (including algorithms) and funneling them to be returned to a user through a screen (according to para. 0070 of the specification).
Regarding claims 5 and 13, Lee in view of Sung discloses all limitations of claims 1 and 9, respectively. Lee in view of Sung further discloses an output device for outputting output information to a user (para. 0005, “According to one embodiment of the present disclosure, a computing device includes an input/output interface for receiving a user input requesting processing of an input image and outputting a reconstructed image processed according to the user input”).
Lee in view of Sung does not disclose listing, by an image processing apparatus, the prompt information in order of higher to lower performance evaluation results in the database, and then outputting the prompt information in order of higher to lower performance.
However, Cha discloses, under the broadest reasonable interpretation, the ordering and returning of prompt information in order of higher to lower performance evaluation results in the database, and the subsequent outputting the prompt information in the order of higher to lower performance (para. 0010, “The method may include generating, at the realism assessment system, a first realism score based on a comparison of characteristics of the first object with the second object by comparing the one or more Hu Moments for the first object and the second object. The method may include generating, at the realism assessment system, a second realism score based on a comparison of characteristics of the first object with the third object by comparing the one or more Hu Moments for the first object and the third object, the second realism score being greater than the first realism score. The method may include determining, at the realism assessment system, the second synthetic digital image has a greater likelihood of accurately representing the real-world content than the first synthetic digital image based on the second realism score being greater than the first realism score. The method may include ranking each synthetic digital image of the plurality of synthetic digital images based on their computed realism score and presenting, at a computing device, the synthetic digital images in order of their ranking.”, wherein the metric of “determined realism” from artificially generated content is the basis for the ordering of the generated images, and the content is returned in that order to the user). Specifically, Cha discloses a method and system for object detection and extraction from digital images for realism evaluation. The evaluation of realism degree as a metric could easily be changed, by an ordinarily skilled artisan, to any of the object detection accuracy metrics of Sung, and rankings could then be delivered to a user terminal according to the output device of Lee. Thus, it would have been obvious to the ordinarily skilled artisan to have adapted the ordering methodology of Cha’s within the method and system of Lee in view of Sung as the application of a known technique to a known device ready for improvement to yield the predictable result of an optimized list of prompts based on their judged success according to a pre-specified metric.
Claims 6, 8, 14, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Lee in view of Sung and in further view of Rombach et al. (“High-Resolution Image Synthesis with Latent Diffusion Models”, full citation in PTO-892 form, hereinafter “Rombach”).
Regarding claims 6 and 14, Lee in view of Sung discloses all limitations of claims 1 and 9, respectively. Lee in view of Sung does not disclose wherein the evaluating of the performance of the object detection model comprises evaluating the performance of the object detection model by using at least one of Precision, Recall, intersection over union (IoU), and mean average precision (mAP) as an evaluation index.
However, Rombach discloses wherein the evaluating of the performance of the object detection model comprises evaluating the performance of the object detection model by using Precision and Recall as evaluation indices (pg. 10689, section 4.2, para. 1, “We train unconditional models of 2562 images on CelebA-HQ, FFHQ, LSUN-Churches and-Bedrooms and evaluate the i) sample quality and ii) their coverage of the data manifold using ii) FID and Precision-and-Recall”). Specifically, Rombach discloses a method and system of image synthesis of high-resolution images using a diffusion model. Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have adapted the metrics of Rombach to use within the method and system of Lee in view of Sung as the application of a known method to a known device ready for improvement to yield the predictable result of a more accurate metric for object detection as a result of the created dataset, balancing dataset quality and distribution and enabling more accurate object detection.
Regarding claims 8 and 16, Lee and Sung disclose all limitations of claims 1 and 9, respectively. Lee and Sung do not disclose performing, by the image processing apparatus, de-noising for removing noise contained in the created training data.
However, Rombach discloses performing, by the image processing apparatus, de-noising for removing noise contained in the created training data (pg. 10687, section 3.3, para. 1, “This can be implemented with a conditional denoising autoencoder ǫθ(zt, t, y) and paves the way to controlling the synthesis process through inputs y such as text [66], semantic maps [32,59] or other image-to-image translation tasks [33].”).
Thus, it would have been obvious to one having ordinary skill in the art prior to the effective filing date of the claimed invention to have adapted the denoising of Rombach to use within the method and system of Lee in view of Sung according to the methodology of claim 8.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROHAN TEJAS MUKUNDHAN whose telephone number is (571)272-2368. The examiner can normally be reached Monday - Friday 9AM - 6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 5712723838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROHAN TEJAS MUKUNDHAN/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights