DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The action is responsive to the following communication: an application filed on 12/12/2024 where:
Claims 1-23 are currently pending.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 8-10, 12, 14-20 and 22-23 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Lin et al. (US 2020/0327675, hereinafter Lin).
Regarding claim 1, Lin teaches: An information processing apparatus (fig. 1, computer system 101, [0019], The computing environment 100 includes a computing system 102) comprising:
one or more memories storing instructions (fig. 7, memory 704, [0076]); and
one or more processors executing the instructions (fig. 7, processor 702, [0076]) to:
obtain an input image (fig. 1 and fig. 2, receive and image 202, The image manipulation application 104 obtain image 108);
extract an object from the input image (see fig. 1 and fig. 2, object 120, [0020-0021], The contour detection module 114 can be configured for detecting foreground objects 120 in the incomplete image 108 and extracting the contour of the foreground objects 120);
identify a missing portion of the object extracted by the extracting, based on a shape of an outline of the object (see fig. 1 and fig. 2, [0020-0021], If a foreground object 120 overlaps with the hole area 122, the contour of the foreground object 120 might also be corrupted, i.e. a portion of the object contour is missing, resulting in an incomplete contour); and
generate a completed image in which the missing portion identified in the input image is completed (see fig.1 and 2, [0020-22], The image manipulation application 104 can employ the contour completion module 116 trained to generate the missing portion of the contour. Fig. 1, complete image 124).
Regarding claim 8, Lin teaches: The information processing apparatus according to claim 1, wherein, in the generating, in the case where cut-out processing of the object is executed on the input image, the completed image is generated ([0021], extracting the contour of the foreground objects 120. And fig. 1, completed image 124).
Regarding claim 9, Lin teaches: The information processing apparatus according to claim 1, wherein, in the generating, in the case where the missing portion is in a predetermined region, the completed image is generated ([0024], The training may also involve generating training data 112 for the image completion model. The training data 112 can include training samples each including an input and an output. The input in a training sample can include an incomplete image, a completed contour and other types of data, such as a hole mask indicating the location of the hole area 122. The output can include the corresponding completed image.).
Regarding claim 10, Lin teaches: The information processing apparatus according to claim 1, wherein, in the generating, a plurality of the completed images different from each other are generated for one of the input image ([0046] and fig. 4, The contour completion module 116 accepts the incomplete contour 410 as an input along with other inputs such as the incomplete image 402 and generates a completed contour 412 using the coarse contour generative model 314 and the refined contour generative model 318 contained in the contour completion model.).
Regarding claim 12, Lin teaches: The information processing apparatus according to claim 1, wherein, in the extracting, the object is extracted from the completed image generated in the generating, in the identifying, the missing portion of the object extracted from the completed image is identified, and in the generating, a re-completed image in which the missing portion identified in the completed image is completed is generated ([0046] and fig. 4, The contour completion module 116 accepts the incomplete contour 410 as an input along with other inputs such as the incomplete image 402 and generates a completed contour 412 using the coarse contour generative model 314 and the refined contour generative model 318 contained in the contour completion model.).
Regarding claim 14, Lin teaches: The information processing apparatus according to claim 1, wherein the one or more processors execute the instructions further to receive a user input, and in the generating, the completion region is set based on the user input ([0020], Accordingly, the image manipulation application 104 can receive the incomplete image 108 by a user uploading or specifying the incomplete image 108).
Regarding claim 15, Lin teaches: The information processing apparatus according to claim 1, wherein, in the generating, in the case where a plurality of the input images are obtained in the obtaining and another input image is arranged on the missing portion, the completed image is not generated ([0051-0052], In one example, the generated hole areas have two types: arbitrarily shaped holes that can appear in any region of the input image and arbitrarily shaped holes that are restricted so that they have no overlaps with the foreground objects. [0052] For the first type of hole areas, it is likely that a hole area overlaps with a foreground object. This scenario is designed to handle the cases when unwanted objects are inside the foreground objects or partially occlude the foreground objects. The second type of hole areas are generated to simulate the cases where the unwanted regions or distracting objects are behind the foreground objects. To generate the second type of hole areas, the model training system 106 can generate, or cause to be generated, randomly and arbitrarily shaped holes. The model training system 106 then removes the parts of holes that have overlap with the foreground objects. If the holes overlaps the foreground objects then complete the object if it doesn’t then it is restricted and not need to complete).
Regarding claim 16, Lin teaches: The information processing apparatus according to claim 1, wherein, in the generating, a plurality of the completed images varying in the completion region are generated ([0030], The segmentation model can also be trained using incomplete images with hole areas generated randomly.).
Regarding claim 17, Lin teaches: The information processing apparatus according to claim 1, wherein, in the extracting, the object with a predetermined attribute is extracted ([0029-0030], For instance, semantic segmentation can be utilized to segment objects, such as a human or a car, based on understanding the content of the image.).
Regarding claim 18, Lin teaches: The information processing apparatus according to claim 1, wherein, in the extracting, a user input specifying the object is obtained, and the object to be extracted is set based on the user input ([0027], For instance, the image manipulation application 104 can receive the incomplete image 108 by a user operating in a user interface presented by the image manipulation application 104 to select or otherwise specify the incomplete image 108.).
Regarding claim 19, Lin teaches: The information processing apparatus according to claim 1, wherein, in the generating, the completed image is generated by using a generative model machine-learned in advance ([0022], In one example, the contour completion module 116 can apply a machine learning model, such as a GAN, to the incomplete contour.).
Regarding claim 20, Lin teaches: The information processing apparatus according to claim 1, wherein the one or more processors execute the instructions further to display the generated completed image on an editing screen of layout data ([0043], the image manipulation application 104 can satisfy the request for image inpainting by sending the completed image 124 to the requesting application or causing it to be presented to a user, such as through a display device of the computing system 102 or other devices accessible to the user.).
Claims 22 and 23 are rejected for reasons similar to claim 1 above
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-7 are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (US 2020/0327675, hereinafter Lin) in view of Johnson et al. (US 2023/0362347, hereinafter Johnson).
Regarding claim 2, Lin teaches: The information processing apparatus according to claim 1, wherein, in the identifying, in the case where the outline of the extracted object (see fig. 1, object 120).
Lin does not explicitly teach: includes a straight line portion, the straight line portion is identified as the missing portion.
However, Johnson teaches: includes a straight line portion, the straight line portion is identified as the missing portion ([0041], Inpainting to be semantically meaningful may include missing portions of a feature or object in an image, such as a couch, should be filled in with similar texture. Inpainting to be geometrically accurate may include where a feature or an object that has straight lines should continue to be straight.).
Therefore, the Applicant's claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Lin to include includes a straight line portion, the straight line portion is identified as the missing portion as taught by Johnson. The motivation/suggestion would have been to further enhance/improve the information processing apparatus since doing so would allow for completing a missing part of an image; therefore, allowing the image to be semantically meaningful and geometrically accurate.
Regarding claim 3, Lin and Johnson teach: The information processing apparatus according to claim 2, wherein, in the identifying, a portion of the outline matching an end portion of the input image is determined as the straight line portion (Johnson, [0041]).
Regarding claim 4, Lin and Johnson teach: The information processing apparatus according to claim 3, wherein, in the straight line portion, a variation of positions of pixels corresponding to the straight line portion with respect to a straight line calculated based on the positions of the pixels corresponding to the straight line portion is equal to or less than a predetermined threshold (Johnson, [0041]).
Regarding claim 5, Lin and Johnson teach: The information processing apparatus according to claim 2, wherein, in the generating, a region that is adjacent to the straight line portion and that is outside the input image is set as a completion region for the input image (Johnson, [0041]).
Regarding claim 6, Lin and Johnson teach: The information processing apparatus according to claim 2, wherein, in the generating, in the case where a plurality of the objects are extracted from the input image in the extracting and the plurality of objects are parts forming the same object, a completion region for the input image is set between the plurality of objects forming the same object (Lin, [0030], The segmentation model can also be trained using incomplete images with hole areas generated randomly).
Regarding claim 7, Lin and Johnson teach: The information processing apparatus according to claim 6, wherein, in the generating, in the case where the missing portions of the plurality of objects extracted in the extracting face each other, the plurality of objects are determined to be the parts forming the same object (Lin, see fig. 4, [0029-30]).
Claims 11 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (US 2020/0327675, hereinafter Lin) in view of Couleaud et al. (US 2025/0078347, hereinafter Couleaud).
Regarding claim 11, Lin teaches: The information processing apparatus according to claim 1.
Lin does not explicitly teach: wherein, in the generating, a text prompt is obtained, and the completed image is generated based on the obtained text prompt.
However, Couleaud teaches: wherein, in the generating, a text prompt is obtained (see fig. 7, text prompt 708 and input image 700, [0087]), and the completed image is generated based on the obtained text prompt (see fig. 7, output image 712, [0087]).
Therefore, the Applicant's claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Lin to include wherein, in the generating, a text prompt is obtained, and the completed image is generated based on the obtained text prompt as taught by Couleaud. The motivation/suggestion would have been to further enhance/improve the information processing apparatus since doing so would allow for outputting the edit/modification image with text prompt; therefore, allowing the user to modify the input image with text prompt without the user needing to upload an actual image.
Regarding claim 13, Lin teaches: The information processing apparatus according to claim 1.
Lin does not explicitly teach: wherein the one or more processors execute the instructions further to receive a user input, and in the generating, whether to generate the completed image or not is determined depending on the user input.
However, Couleaud teaches: wherein the one or more processors execute the instructions further to receive a user input, and in the generating, whether to generate the completed image or not is determined depending on the user input (see fig. 7, generate image 710, and [0087], . Based on receiving user selection of option 710, the image processing system may, using the techniques described herein, generate for display image 714 at output portion 712 of GUI 700. If user doesn’t click/press on generate image 710 then image is not generated).
Therefore, the Applicant's claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Lin to include wherein the one or more processors execute the instructions further to receive a user input, and in the generating, whether to generate the completed image or not is determined depending on the user input as taught by Couleaud. The motivation/suggestion would have been to further enhance/improve the information processing apparatus since doing so would allow for outputting the edit/modification image with text prompt; therefore, allowing the user to modify the input image or not based on the user input as needed.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Lin et al. (US 2020/0327675, hereinafter Lin) in view of Vazquez (US 2018/0184167).
Regarding claim 21, Lin teaches: The information processing apparatus according to claim 20.
Lin does not explicitly teach: wherein the editing screen of the layout data is a screen for editing the layout data relating to a printed product or a document.
However, Vazquez teaches: wherein the editing screen of the layout data is a screen for editing the layout data relating to a printed product (fig. 6, [0101] and [0121] Printer 834 enables a user with the capability to print information to a sheet of paper) or a document.
Therefore, the Applicant's claimed invention would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Lin to include wherein the editing screen of the layout data is a screen for editing the layout data relating to a printed product or a document as taught by Vazquez. The motivation/suggestion would have been to further enhance/improve the information processing apparatus since doing so would allow for outputting the edit/modification image; therefore, allowing the user with the capability to print the edit/modification image to a sheet of paper.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW H LAM whose telephone number is (571)270-7969 and fax number is 571-270-8969. The examiner can normally be reached on 9AM-5PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Benny Tieu can be reached on 571-272-7490. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW H LAM/
Primary Examiner, Art Unit 2682
Prosecution Insights