DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on November 7, 2023 and September 2, 2025 were filed after the mailing date of the application on November 7, 2023. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Examiner’s Note
The Examiner makes note that wherever the claims recite a “unit”, they recite that the unit is implemented by the processor. Thus, the claims are not being interpreted under 35 U.S.C. 112(f).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-8 and 12-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Park (US 20240087265A1), Niu (US 20220292269A1), He (US 20190266262A1), Paula (US 20230409624A1), and Sasaki (US 20180240093A1).
As per Claim 1, Park teaches an image generating and retrieving apparatus having a processor, the image generating and retrieving apparatus comprising: an image generation unit that acquires, by the processor, a new generated image by image generation processing from an input text and an input image that have been input (subsequent to the upload of the input photograph 503, receive indications of user edits via receiving the natural language characters within the text field 513, [0110], user has entered in the natural language characters of “looks like French Alps in July” (user edit request) the model finds an output image scene transformation that matches (is within a threshold distance of) the natural language characters, “French Alps” contain many pointed mountains, yet there is a set of pixels representing only a single mount in the input photograph 503, at the output image 505, warp or change pixel values that represent the single mountain to include other pixel values representing additional mountains, as illustrated in the output image 505, [0111]); a text registration unit that calculates a text feature amount from the input text by the processor; an image registration unit that calculates an image feature amount from the input image and the generated image by the processor; a retrieval unit that calculates a similarity from the text feature amount and the image feature amount by the processor, and retrieves a similar image similar to a retrieval target by using the similarity (goal of the CLIP model 304 is that the vectors representing the natural language text be very close (via Cosine similarity) to the vectors for the corresponding image, which represents minimal loss at training, measure or determine how close the embedded representation (series of numbers) for each text is to the embedded representation for each image, the text encoder and image encoder get fit at the same time by simultaneously maximizing the cosine similarity of matches and minimizing the cosine similarity of non-matches, across all text-image pairs, once the model is fit, the image is passed into the image encoder to retrieve the text description that best fits the image—or, vice versa, pass a text description into the model to retrieve an image, [0069]).
However, Park does not expressly teach an image text database that holds the input text, the input image, the generated image; retrieving the similar image from the image text database. However, Niu teaches an image text database that holds the input text, the input image, the generated image (image-text pair database, [0033]); a retrieval unit that calculates a similarity from the text feature amount and the image feature amount by the processor, and retrieves a similar image similar to a retrieval target from the image text database by using the similarity (calculating similarity between a vector representation of the image and a vector representation of the text, calculating cosine similarity between the two vector representations, [0057], [0033]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Park to include an image text database that holds the input text, the input image, the generated image; retrieving the similar image from the image text database because Niu suggests that this way, the user can efficiently retrieve similar images and texts as desired [0034, 0043].
However, Park and Niu do not expressly teach that the image text database holds the image feature amount, the text feature amount, and an input/output relationship in the image generation processing as an image generation process. However, He teaches an image text database that holds the image feature amount (image vector), the text feature amount (text vector), and an input/output relationship in the image generation processing as an image generation process (generated text vector can be stored in a database, image vector can be stored in a database, [0042], image vector to which the text vector 392 is compared by the cosine similarity 350, [0038]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Park and Niu so that the image text database holds the image feature amount, the text feature amount, and an input/output relationship in the image generation processing as an image generation process because He suggests that this way, the image feature amount and the text feature amount can be efficiently retrieved in order to determine a similar image [0042, 0038].
However, Park, Niu, and He do not teach visualizing the image generation process held in the image text database by the processor. However, Paula teaches visualizing the image generation process held in the image text database by the processor (generate efficiently searchable data structures, such as graphs, to enable identification of content that is semantically similar to images, text, [0006], data set comprises text and image, parsing the data set, in accordance with the text protocol and the image protocol, to extract object data sets representing at least a portion of a textual or an image feature, generating for each of the object data sets an enriched object data set, using the generated enriched object data sets, generating a graph, [0041]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Park, Niu, and He to include visualizing the image generation process held in the image text database by the processor because Paula suggests that this is an efficient way to identify images that are similar to an image [0006].
However, Park, Niu, He, and Paula do not teach a display unit that visualizes the image generation process by the processor. However, Sasaki teaches a display unit that visualizes the graph that are images in which the similarities of the respective images are graphically displayed (display control module 54 displays the graphs that are images in which the similarities of the respective commodity candidates are graphically displayed, [0083], Fig. 5). Since Paula teaches visualizing in a graph the image generation process held in the image text database by the processor [0006, 0041], this teaching of displaying from Sasaki can be implemented on the graph of Paula so that a display unit visualizes the image generation process held in the image text database by the processor.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Park, Niu, He, and Paula to include a display unit that visualizes the image generation process by the processor because Sasaki suggests that this way, the user can easily determine which image the image is most similar to [0083].
As per Claim 2, Park teaches further comprising: a text input unit that receives the input text by the processor; an image input unit that receives the input image by the processor [0110]; and an image generation assisting unit that assists the image generation processing of the generated image by the processor [0111].
As per Claim 3, Park does not expressly teach wherein the retrieval unit retrieves, by the processor, the image text database using a retrieval query given by the input text or the input image. However, Niu teaches wherein the retrieval unit retrieves, by the processor, the image text database using a retrieval query given by the input text or the input image (image-text pair database, [0033], retrieval may be performed in the text database using text1 to obtain texts with similarity to the text 1 greater than or equal to a preset similarity threshold, [0043]). This would be obvious for the reasons given in the rejection for Claim 1.
However, Park and Niu do not expressly teach rearranges, by the processor, the retrieved similar images according to the similarity. However, He teaches rearranges, by the processor, the retrieved similar images according to the similarity (top-ranked text-image pairs, where such rankings can be based on the distance in the multi-dimensional space, [0030]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Park and Niu to include rearranging, by the processor, the retrieved similar images according to the similarity because He suggests that this is an efficient way to determine the image that is the most similar [0030].
As per Claim 4, Park, Niu, and He do not teach visualizing, by the processor, the image generation process as a graph indicating the input/output relationship by arranging the generated image and the similar image. However, Paula teaches visualizes, by the processor, the image generation process as a graph indicating the input/output relationship by arranging the generated image and the similar image [0006, 0041]. This would be obvious for the reasons given in the rejection for Claim 1.
However, Park, Niu, He, and Paula do not teach wherein the display unit visualizes the graph. However, Sasaki teaches wherein the display unit visualizes, by the processor, a graph indicating the relationship between the images [0083] (Fig. 5). Since Paula teaches visualizes, by the processor, the image generation process as a graph indicating the input/output relationship by arranging the generated image and the similar image [0006, 0041], this teaching of displaying from Sasaki can be implemented on the graph of Paula so that the display unit visualizes, by the processor, the image generation process as a graph indicating the input/output relationship by arranging the generated image and the similar image. This would be obvious for the reasons given in the rejection for Claim 1.
As per Claim 5, Park and Niu do not teach wherein the image generation unit acquires, by the processor, a predetermined number of the generated images as a generated image set, and the retrieval unit retrieves, by the processor, the similar image by a query image or a query text for the generated image set by filtering processing using the similarity, and outputs, by the processor, a retrieval result as a filtering result. However, He teaches the top-ranked text-image pairs, where such rankings are based on the distance in the multi-dimensional space [0030]. He teaches voting for the attributes of the physical object visually depicted by the input image based on the identified attributes enumerated by the text of the aggregated text-image pairs, wherein votes are weighted based on similarity between a voted-for attribute and other identified attributes enumerated by the text of the aggregated text-image pairs; and selecting, based on the voting, the attributes of the physical object visually depicted by the input image [0054]. The image vector to which the text vector 392 is compared by the cosine similarity 350 [0038]. Thus, the text-image pairs are ranked based on the similarity, and the lower ranked text-image pairs are filtered out based on the similarity, and the top ranked text-image pair is output based on the similarity. Thus, He teaches wherein the image generation unit acquires, by the processor, a predetermined number of the generated images as a generated image set, and the retrieval unit retrieves, by the processor, the similar image by a query image or a query text for the generated image set by filtering processing using the similarity, and outputs, by the processor, a retrieval result as a filtering result [0030, 0054, 0038].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Park and Niu so that the image generation unit acquires, by the processor, a predetermined number of the generated images as a generated image set, and the retrieval unit retrieves, by the processor, the similar image by a query image or a query text for the generated image set by filtering processing using the similarity, and outputs, by the processor, a retrieval result as a filtering result because He suggests that this is an efficient way to determine the image that is the most similar [0030, 0054, 0038].
As per Claim 6, Park and Niu do not teach wherein the retrieval unit calculates, by the processor, a score by calculating the similarity using a query set preset for the generated image in the filtering processing, and outputs, by the processor, the filtering result on a basis of the score. However, He teaches wherein the retrieval unit calculates, by the processor, a score (cosine similarity 350) by calculating the similarity using a query set preset for the generated image in the filtering processing, and outputs, by the processor, the filtering result on a basis of the score [0030, 0054, 0038]. This would be obvious for the reasons given in the rejection for Claim 5.
As per Claim 7, Park and Niu do not teach wherein the image generation unit acquires, by the processor, a predetermined number of the generated images from the input text, and the retrieval unit recognizes, by the processor, whether a concept of a term included in the input text is included in the generated image by filtering processing using image recognition, and outputs a recognition result as a filtering result. However, He teaches wherein the image generation unit acquires, by the processor, a predetermined number of the generated images from the input text, and the retrieval unit recognizes, by the processor, whether a concept of a term included in the input text is included in the generated image by filtering processing using image recognition, and outputs a recognition result as a filtering result [0030, 0054, 0038]. This would be obvious for the reasons given in the rejection for Claim 5.
As per Claim 8, Park and Niu do not teach wherein the retrieval unit executes, by the processor, object detection processing on the generated image, calculates, by the processor, a score from a matching degree between an object detection result in the object detection processing and the term included in the input text, and outputs, by the processor, the filtering result on a basis of the score. However, He teaches wherein the retrieval unit executes, by the processor, object detection processing on the generated image, calculates, by the processor, a score (cosine similarity 350) from a matching degree between an object detection result in the object detection processing and the term included in the input text, and outputs, by the processor, the filtering result on a basis of the score [0030, 0054, 0038]. This would be obvious for the reasons given in the rejection for Claim 5.
As per Claim 12, Park teaches an image generating and retrieving system, comprising: the image generating and retrieving apparatus; an input device that specifies a retrieval condition [0110]; and a display device that displays a retrieval result (user edits are combined to the extracted attributes, the edited attributes and features are passed to a trained generator network to produce the output image 315, [0067], presentation module 120 can cause presentation of a visual rendering that reflects an output image, [0056]).
As per Claim 13, Park teaches wherein the display device includes an operation screen for performing the image generation processing and the retrieval (Fig. 5 is an example user interface for editing an image, [0107], Fig. 5).
However, Park, Niu, and He do not teach visualizing the image generation process. However, Paula teaches visualizing the image generation process [0006, 0041], as discussed in the rejection for Claim 1.
However, Park, Niu, He, and Paula do not teach the operation screen includes a generation process visualization field for displaying the image generation process visualized by the display unit. However, Sasaki teaches the operation screen includes a visualization field for displaying the graph that are images in which the similarities of the respective images are graphically visualized by the display unit [0083] (Fig. 5). Since Paula teaches visualizing the image generation process in a graph [0006, 0041], this teaching of displaying from Sasaki can be implemented on the graph of Paula so that the operation screen includes a generation process visualization field for displaying the image generation process visualized by the display unit. This would be obvious for the reasons given in the rejection for Claim 1.
As per Claim 14, Park teaches wherein the operation screen further includes: a text input field (513) for inputting the input text via the input device [0110]; an image generation button (interface elements such as graphics buttons, [0056]) for acquiring the generated image via the input device [0111]; a generated image display field for displaying the generated image [0067, 0056]; a retrieval condition field for specifying the retrieval condition via the input device [0110]; an image retrieval button [0056] for performing the retrieval via the input device [0111]; and an image retrieval result field for displaying the retrieval result [0067, 0056].
As per Claim 15, Claim 15 is similar in scope to Claim 1, and therefore is rejected under the same rationale.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Park (US 20240087265A1), Niu (US 20220292269A1), He (US 20190266262A1), Paula (US 20230409624A1), and Sasaki (US 20180240093A1) in view of Zhang (US 20230230198A1).
Park, Niu, He, Paula, and Sasaki are relied upon for the teachings as discussed above relative to Claim 2. Park teaches wherein the image generation unit acquires, by the processor, the generated image from the input text by the image generation processing [0110-0111], the retrieval unit retrieves, by the processor, the similar image using the generated image as a query to acquire a related text of the similar image [0069].
However, Park does not expressly teach retrieving the similar image from the image text database. However, Niu teaches retrieving the similar image from the image text database [0057, 0033], as discussed in the rejection for Claim 1.
However, Park, Niu, He, Paula, and Sasaki do not teach the image generation assisting unit extracts, by the processor, a plurality of keywords from the related text, and adds, by the processor, the keyword selected by the user from the plurality of keywords to the input text to acquire the generated image again by the image generation processing. However, Zhang teaches the image generation assisting unit extracts, by the processor, a plurality of keywords from the related text, and adds, by the processor, the keyword selected by the user from the plurality of keywords to the input text (selectively determines elements of a previous style vector to update based on the additional textual feedback, determines a similarity between a semantic feature change for each style element of the previous style vector and a desired semantic change based on the additional textual feedback, from a modified style vector with the updated style elements, a generative neural network can flexibly generate modified images that reflect iterative feedback plus prior feedback, [0026]) to acquire the generated image again by the image generation processing (user simulator to give text feedback based on the generated images, the user simulator can provide text feedback that identifies target attributes not satisfied by the generated image, the text feedback to feed back into the interactive generation system for further image manipulation, the interaction process stops when the user simulator finds the generated image matches all the target attributes, [0119]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Park, Niu, He, Paula, and Sasaki so that the image generation assisting unit extracts, by the processor, a plurality of keywords from the related text, and adds, by the processor, the keyword selected by the user from the plurality of keywords to the input text to acquire the generated image again by the image generation processing because Zhang suggests that this way the generative neural network can flexibly generate modified images that reflect iterative feedback plus prior feedback so that the final generated image is the desired image [0026, 0119].
Allowable Subject Matter
Claims 10-11 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The prior art taken singly or in combination do not teach or suggest the combination of all the limitations of Claim 10 and base Claim 1 and intervening Claim 2, and in particular, do not teach wherein the image generation assisting unit acquires, by the processor, a corrected text input by a user and extracts an existing element included in the initial text and a newly added additional element, and reflects a detection region obtained by detecting an object of the existing element from the generated image in a mask image as a holding region, the retrieval unit retrieves, by the processor, the similar image from the image text database using the corrected text as a query, and
the image generation assisting unit detects, by the processor, objects of the existing element and the additional element from the similar image, respectively, updates the mask image according to an appearance frequency of the additional element, calculates a relative size of the additional element with respect to the existing element, and generates the mask image according to the relative size. Claim 11 depends from Claim 10, and therefore also contains allowable subject matter.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONI HSU whose telephone number is (571)272-7785. The examiner can normally be reached M-F 10am-6:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JH
/JONI HSU/Primary Examiner, Art Unit 2611
Prosecution Insights