DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 6-9, 11, 17, 20-22, 25, 29, 36, 37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benedetto et al. (U.S. Pub. No. 20240193351) in view of Bean (U.S. Pub. No. 20240320867).
Regarding claim 1, Benedetto discloses a method comprising (para 23, “Systems and method for generating an image using an image generation artificial intelligence (IGAI) process are described.”): at an electronic device in communication with a display generation component and one or more input devices (para 79, “The graphics subsystem 720 periodically outputs pixel data for an image from the graphics memory 718 to be displayed on the display device 710. The display device 710 can be any device capable of displaying visual information in response to a signal from the device 700, including a cathode ray tube (CRT) display, a liquid crystal display (LCD), a plasma display, and an organic light emitting diode (OLED) display. The device 700 can provide the display device 710 with an analog or digital signal, for example”; also, para 77, “User input devices 708 communicate user inputs from one or more users to the device 700. Examples of the user input devices 708 include keyboards, mouse, joysticks, touch pads, touch screens, still or video recorders/cameras, tracking devices for recognizing gestures, and/or microphones.”): receiving, via the one or more input devices (para 77, “User input devices 708 communicate user inputs from one or more users to the device 700. Examples of the user input devices 708 include keyboards, mouse, joysticks, touch pads, touch screens, still or video recorders/cameras, tracking devices for recognizing gestures, and/or microphones.”), a prompt for use in creating automatically- generated visual media that is generated at least partially using one or more autonomous processes (para 46, “In the example illustrated in FIG. 4A-1, the user begins to enter text input in the search field rendered in the user interface 110 at a display screen 105 of the client device 100. In the example illustrated in FIG. 4A-1, the user has provided an initial text prompt, "Dark sky". The initial text prompt defines the prompt 1 provided by the user.”; also, para 24, “An image generation artificial intelligence (IGAI) process is used to receive the user prompt, analyze the user prompt to determine if the user prompt includes an image or keywords or both, determine the context of the user prompt, and identify image features that match the context and, where available, a style preferred by the user, and generate a single image with the identified image features that provides a visual representation of the user prompt.”); and in response to receiving the prompt, displaying, via the display generation component, a user interface that includes prompt information (para 46, “In response to detecting the text input of prompt 1 entered by the user, the text analyzer 311 dynamically parses the text input and identifies the keyword kw 1, "dark" included in prompt 1. The text analyzer 311 forwards the identified keyword kw 1 to the word variation module 313.”; also, para 46, “The identified keyword variations for keyword kw 1 are returned to the client device for rendering in the user interface 110 for user selection.”), wherein displaying the user interface includes concurrently displaying (para 46, “An example set of keyword variations identified for the keyword "dark" is shown as checkboxes in box 106 of FIG. 4A-1.”; also, para 46, “The adjusted prompt, prompt 2 is forwarded to the client device for rendering in the search field at the user interface, where the user provides the text input.”): a representation of the prompt (para 46, “In the example illustrated in FIG. 4A-1, the user has provided an initial text prompt, "Dark sky".”; also, para 46, “The adjusted prompt, prompt 2 is forwarded to the client device for rendering in the search field at the user interface, where the user provides the text input.”); and automatically-generated visual media (para 48, “The adjusted context and the updated prompt 2 is also fed into the ML engine 320 as input so that appropriate image features influencing the content of prompt 2 can be identified for generating the image for the user prompt.”; also, para 62, “The images returned to the client device 100, in response to the user prompt include image features that are influenced by content of the respective user prompt, represent contextually relevant, visual representation of the user prompt.”). Benedetto does not disclose a first visual indication that a first portion of the prompt has been identified as a first recognized concept.
However, in a similar field of endeavor, Bean discloses a first visual indication that a first portion of the prompt has been identified as a first recognized concept (para 74, “In some embodiments, the user interface 610 is updated to indicate that one or more alternative and/or complementary words have been identified for words in the initial text prompt and/or the subsequent text prompt. For example, and as illustrated, the user interface 610 may be updated to display a first indication 632 that one or more alternative words have been identified for the word "Car" in the initial text prompt and a second indication 634 the one or more complementary words have been identified for the word "Road" in the initial text prompt. As further illustrated, the first indication 632 and the second indication 634 may include a rectangular boundary around the respective words.”; also, para 74, “Additional or alternative methods of indicating that alternative and/or complementary words have been identified may include changing the typeface of the words, highlighting the words with a distinct color or shading, adding a selectable option to each word, and the like.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of a method performed at an electronic device that provides a display device with the signal by which an image is displayed and that receives user inputs communicated to it from input devices, in which a text prompt entered by the user in a search field is received for use in creating an image generated by an image generation artificial intelligence process, in which the entered text is parsed in response to its receipt so that a keyword within the prompt is identified, in which the user interface concurrently renders the prompt in the search field together with the material returned for that keyword, and in which the identified keyword drives the image features of the generated image, with the features of Bean's invention of updating the user interface to display an indication, such as a rectangular boundary, a changed typeface, or a distinct color or shading, marking the particular word of the entered text prompt for which the system has identified related words. The combination would have been obvious because Benedetto and Bean are both interfaces of the same assignee for editing a text prompt that drives an image generation model, and Benedetto leaves the user with no way to see which of the words he typed the analyzer actually acted on. Benedetto states expressly that of the words "dark" and "sky" in the entered prompt, only "dark" is selected for treatment while "sky" is ignored, so the interface acts on some typed words and not others without telling the user which. Bean answers that exact need by drawing the boundary around the very word for which related words were found, and applying Bean's marking to the word Benedetto has already identified yields the predictable result of a prompt whose acted-upon words are visible to the user, which is what Benedetto's own keyword variation feature depends on the user understanding.
Regarding claim 2, Benedetto as modified by Bean discloses the method of claim 1, wherein Benedetto further discloses displaying the user interface that includes the prompt information (para 47, “FIG. 4A-2 illustrates an example set of keyword variations identified for the second keyword kw 2 and rendered in box 107 by the word variation module 313. “; also, para 46, “The adjusted prompt, prompt 2 is forwarded to the client device for rendering in the search field at the user interface, where the user provides the text input.”) further includes influence the generation of the automatically-generated visual media (para 48, “The adjusted context and the updated prompt 2 is also fed into the ML engine 320 as input so that appropriate image features influencing the content of prompt 2 can be identified for generating the image for the user prompt.”; also, para 62, “The images returned to the client device 100, in response to the user prompt include image features that are influenced by content of the respective user prompt, represent contextually relevant, visual representation of the user prompt.”), second portion of the prompt is different than the first portion of the prompt (para 47, “When the user provides additional text in the search field, the text analyzer 311 detects the additional text and dynamically parses the additional text to identify a second keyword kw 2, "field" in prompt 2.”; also, para 49, “For example, in the example illustrated in FIGS. 4A-1 and 4A-2, keywords 'dark' and 'sky' are identified in the user prompt and each is assigned a relative weight based on the context.”). Benedetto does not discloses displaying a second visual indication that a second portion of the prompt has been identified as a second recognized concept, wherein the second visual indication is displayed concurrently with the first visual indication in the user interface, the second visual indication is different than the first visual indication.
However, in a similar field of endeavor, Bean discloses displaying a second visual indication that a second portion of the prompt has been identified as a second recognized concept (para 74, “For example, and as illustrated, the user interface 610 may be updated to display a first indication 632 that one or more alternative words have been identified for the word "Car" in the initial text prompt and a second indication 634 the one or more complementary words have been identified for the word "Road" in the initial text prompt.”; also, para 74, “In some embodiments, the user interface 610 is updated to indicate that one or more alternative and/or complementary words have been identified for words in the initial text prompt and/or the subsequent text prompt.”; also, para 55, “In some embodiments, the additional words may be further distinguished according to the category or class of word to which they belong. For example, a content word 332 (e.g., "Sports") and/or a content phrase 334 (e.g., "Mountain Road") may be highlighted or otherwise visually distinguished to identify their correspondence to the visual content class 330.”), wherein the second visual indication is displayed concurrently with the first visual indication in the user interface (para 74, “As further illustrated, the first indication 632 and the second indication 634 may include a rectangular boundary around the respective words.”, the second visual indication is different than the first visual indication (para 55, “In some embodiments, the additional words may be further distinguished according to the category or class of word to which they belong. For example, a content word 332 (e.g., "Sports") and/or a content phrase 334 (e.g., "Mountain Road") may be highlighted or otherwise visually distinguished to identify their correspondence to the visual content class 330. As further examples, a style word 342 (e.g., "Photorealism"), a perspective phrase 352 (e.g., "Atmospheric perspective"), and a lighting phrase 362 (e.g., "Dusk Lighting"), may each be individually distinguished to indicate their correspondence to the visual style class 340, the visual perspective class 350, and the lighting class 360 respectively.”; also, para 54, “As illustrated, the user interface 310 and/or the text field 312 may use one or more visual indicators to highlight or otherwise distinguish the subset of words representing the additional description of the generated image. For example, such words may be highlighted in bold or surrounded by a border. While illustrated as bold text enclosed in dashed boxes, additional or alternative visual indications may instead be used. For example, the additional words may be highlighted and/or presented in a different color font compared to the words included in the initial text prompt.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of a user interface that renders the prompt in the search field together with the material returned for a second keyword the analyzer has identified in that prompt, in which the second identified keyword is distinct from the first and each identified keyword is assigned its own weight, and in which the updated prompt and its identified content drive the image features of the image the generation engine produces, with the features of Bean's invention of updating the interface to display a second indication marking the second identified portion of the entered text concurrently with the first indication marking the first, where the marked portion is a single word or a multiple word phrase, and of distinguishing each marked portion individually according to the class of word to which it belongs. The combination would have been obvious because Benedetto already tracks more than one identified keyword at a time and weights them differently according to their relevance, so the interface must convey to the user not merely that portions of the text were identified but which portion carries which treatment. Bean supplies exactly that convention by marking each identified portion separately, by treating a phrase such as a two word content phrase as one marked portion in the same way as a single word, and by giving portions of different classes their own distinguishing appearance, and the predictable result is an interface in which a user reading his own prompt can tell the two identified portions apart and can tell what each one contributes.
Regarding claim 3, Benedetto as modified by Bean discloses the method of claim 1, wherein Benedetto further discloses the prompt includes starting media for use in creating the automatically-generated visual media (para 57, “In this alternate implementation, the analysis module 310 is configured to receive a source image as a user prompt. The source image may be provided in the search field provided at a user interface 110 rendered at the display screen 105 of the client device 100 of the user.”; also, para 57, “In alternate implementations, the source image is provided in an image input field and the text input at a text input field at the user interface, wherein the source image and the text input represent the user prompt.”), and the starting media influences the generation of the automatically-generated visual media (para 34, “the user provides the image in addition to text string in the user prompt, the text content generated for the source image is combined with the text string provided in the user prompt to generate an aggregate prompt. The analysis module 310 then analyzes the aggregate prompt to identify keywords and sequence of keywords to determine the context of the user prompt. The context with the keywords and the keyword sequence are forwarded to the ML engine 320 for identifying image features to include in an image generated for the user prompt.”; also, para 66, “The image features identified for the adjusted prompt are used to influence a change in corresponding image features of the source image so that image feature reflects a style specified by the content of the adjusted prompt, as illustrated in operation 580.”).
Regarding claim 4, Benedetto as modified by Bean discloses the method of claim 1, wherein Benedetto further discloses the first recognized concept includes a keyword or set of related keywords (para 46, “In response to detecting the text input of prompt 1 entered by the user, the text analyzer 311 dynamically parses the text input and identifies the keyword kw 1, "dark" included in prompt 1.”; also, para 51, “From the above example of prompt 1 entered by the user, the text analyzer 311 can identify “Horizon zero Dawn” as keyword sequence 1 (kw-s1), and “zero dawn” as keyword sequence 2 (kw-s2).”).
Regarding claim 6, Benedetto as modified by Bean discloses the method of claim 1, wherein Bean further discloses the first recognized concept includes a style prompt (para 55, “As further examples, a style word 342 (e.g., "Photorealism"), a perspective phrase 352 (e.g., "Atmospheric perspective"), and a lighting phrase 362 (e.g., "Dusk Lighting"), may each be individually distinguished to indicate their correspondence to the visual style class 340, the visual perspective class 350, and the lighting class 360 respectively.”; also, para 86, “The initial prompt may be a text prompt including a first set of words describing visual features such as visual content, visual styles, visual perspectives, lighting, and the like.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of identifying a keyword within the entered prompt and marking it as a concept that drives the image features of the generated image, with the features of Bean's invention of treating a style word within the prompt as one of the classes of word the system recognizes and distinguishes. The combination would have been obvious because Benedetto already obtains a style for the generated image and applies it to the output, but takes that style from a separate query, a stored user profile, or the user's usage history rather than from the prompt itself, so the user cannot set the style by writing it. Bean recognizes the style word inside the prompt alongside the content, perspective and lighting words, and adding that recognition to Benedetto yields the predictable result of a single prompt in which the style is expressed and acted upon in the same way as every other concept the system identifies.
Regarding claim 7, Benedetto as modified by Bean discloses the method of claim 1, wherein Benedetto further comprising: while displaying the user interface that includes the prompt information (para 46, “The identified keyword variations for keyword kw 1 are returned to the client device for rendering in the user interface 110 for user selection.”): detecting one or more inputs corresponding to a request to modify one or more recognized concepts that will influence the generation of the automatically-generated visual media (para 46, “User selection of a keyword variation is used to replace the identified keyword kw 1 in prompt 1 to generate an adjusted prompt, prompt 2.”); and in response to detecting the one or more inputs (para 66, “User selection of a particular keyword variation is received at the server and, in response, the aggregated user prompt is updated to include the keyword variation in place of the one or more keywords to generate an adjusted prompt, as illustrated in operation 575.”): modifying the one or more recognized concepts that will influence the generation of the automatically-generated visual media in accordance with the one or more inputs (para 46, “FIG. 4A-2 shows an example of the adjusted prompt, prompt 2, that includes the keyword variation 'night' replacing the keyword 'dark' in prompt 1.”; also, para 66, “The adjusted prompt with the keyword variation is then forwarded to a ML engine, which engages an AI model to identify outputs that are influenced by the content of the adjusted prompt.”).
Regarding claim 8, Benedetto as modified by Bean discloses the method of claim 7, wherein Benedetto further discloses the one or more inputs corresponding to the request to modify the one or more recognized concepts that will influence the generation of the automatically-generated visual media include an input to modify the prompt (para 47, “When the user provides additional text in the search field, the text analyzer 311 detects the additional text and dynamically parses the additional text to identify a second keyword kw 2, "field" in prompt 2.”); and in response to receiving the one or more inputs (para 66, “User selection of a particular keyword variation is received at the server and, in response, the aggregated user prompt is updated to include the keyword variation in place of the one or more keywords to generate an adjusted prompt, as illustrated in operation 575.”): updating the representation of the prompt to a representation of a second prompt in accordance with modifications to the prompt indicated by the one or more inputs (para 46, “The adjusted prompt, prompt 2 is forwarded to the client device for rendering in the search field at the user interface, where the user provides the text input.”; also, para 47, “The process of receiving additional keywords in the search field, identifying and providing keyword variations for each identified keyword, and updating prompt 2 continues so long as the user provides additional text input in the search field.”); and displaying, in the user interface, one or more second visual indications corresponding to one or more portions of the second prompt that have been identified as recognized concepts.
However, in a similar field of endeavor, Bean discloses displaying, in the user interface, one or more second visual indications corresponding to one or more portions of the second prompt that have been identified as recognized concepts (para 74, “In some embodiments, the user interface 610 is updated to indicate that one or more alternative and/or complementary words have been identified for words in the initial text prompt and/or the subsequent text prompt.”; also, para 71, “In response to receiving the subsequent text prompt, the user interface 610 and/or the text field 612 may be updated to display the subsequent text prompt.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of receiving further text in the search field, identifying the keywords of the resulting second prompt, and rendering the updated prompt back in the search field, with the features of Bean's invention of updating the interface to mark the words of a subsequent text prompt for which the system has identified related words. The combination would have been obvious because Benedetto states that the cycle of receiving additional keywords, identifying them and updating the prompt continues for as long as the user keeps typing, so the interface has to re-express which words are being acted on each time the prompt changes rather than only once. Bean marks the words of the subsequent prompt in the same way it marks the words of the initial prompt, and carrying that convention through Benedetto's loop yields the predictable result that the marking stays current with the prompt the user is actually editing.
Regarding claim 9, Benedetto as modified by Bean discloses the method of claim 7, wherein Benedetto further disclosesa; and in response to receiving the one or more inputs (para 66, “User selection of a particular keyword variation is received at the server and, in response, the aggregated user prompt is updated to include the keyword variation in place of the one or more keywords to generate an adjusted prompt, as illustrated in operation 575.”): one or more inputs corresponding to the request to modify the one or more recognized concepts that will influence the generation of the automatically-generated visual media include an input corresponding to a request to add a second recognized concept that will influence the generation of the automatically-generated visual media, and displaying a second visual indication corresponding to the second recognized concept concurrently with the first visual indication corresponding to the first recognized concept.
However, in a similar field of endeavor, Bean discloses the one or more inputs corresponding to the request to modify the one or more recognized concepts that will influence the generation of the automatically-generated visual media include an input corresponding to a request to add a second recognized concept that will influence the generation of the automatically-generated visual media (para 76, “The user interface 610 may further display one or more options to replace a word in the initial text prompt and/or the subsequent text prompt with an alternative word and/or one or more options to add a word to the initial text prompt and/or the subsequent text prompt with a complementary word.”), and displaying a second visual indication corresponding to the second recognized concept concurrently with the first visual indication corresponding to the first recognized concept (para 74, “For example, and as illustrated, the user interface 610 may be updated to display a first indication 632 that one or more alternative words have been identified for the word "Car" in the initial text prompt and a second indication 634 the one or more complementary words have been identified for the word "Road" in the initial text prompt. As further illustrated, the first indication 632 and the second indication 634 may include a rectangular boundary around the respective words.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of receiving a user selection that changes the recognized concepts of the prompt and updating the prompt in response, with the features of Bean's invention of offering an option to add a complementary word to the prompt and of displaying an indication for that added word alongside the indication already displayed for the first. The combination would have been obvious because Benedetto's own editing options are confined to swapping one identified keyword for a variation, which lets the user change a concept but never enrich the prompt with an additional one, and Benedetto expressly seeks to better understand the user's intentions for the prompt. Bean's complementary words are the concepts commonly used with a word already in the prompt, and adding Bean's add option together with Bean's per-word marking yields the predictable result that a user can grow the set of concepts driving the image while continuing to see every concept the system is acting on.
Regarding claim 11, Benedetto as modified by Bean discloses the method of claim 1, wherein Bean further comprising: while displaying the first visual indication corresponding to the first recognized concept (para 74, “For example, and as illustrated, the user interface 610 may be updated to display a first indication 632 that one or more alternative words have been identified for the word "Car" in the initial text prompt and a second indication 634 the one or more complementary words have been identified for the word "Road" in the initial text prompt. As further illustrated, the first indication 632 and the second indication 634 may include a rectangular boundary around the respective words”; also, para 75, “For example, in response to a user selection 604 of the first indication 632, the user interface 610 may be updated to display an alternative word menu 650 including the alternative words related to the word "Car".”), receiving, via the one or more input devices, a second input corresponding to a request to add a second recognized concept, wherein the second recognized concept is in a same category as the first recognized concept (para 76, “The user interface 610 may further display one or more options to replace a word in the initial text prompt and/or the subsequent text prompt with an alternative word and/or one or more options to add a word to the initial text prompt and/or the subsequent text prompt with a complementary word.”; para 77, “Based on the words displayed the table, the user may choose to replace a word in the subsequent text prompt, or add a word to the subsequent text prompt, by manually editing the text in the text field 612, dragging and dropping a word from the table to the text field 612, and the like.”; also, para 73, “For example, in addition to receiving complementary words (e.g., "Supercar" or "Mountain"), the user interface 610 may receive or otherwise identify alternative words for one or more of the complementary words (e.g., "Minivan", "Truck", "Sedan", etc. as alternatives for "Supercar" and "Beach", "Snowy", "City", etc. as alternatives for "Mountain").”; also, para 37, “For example, the text-to-image generator 142 may be configured to accept, as inputs, one or more classes of visual features within which the resulting images are to be varied, such as a visual content class, a visual style class, a visual perspective class, a lighting class, and the like.”); and in response to receiving the second input: ceasing display of the first visual indication corresponding to the first recognized concept (para 74, “As further illustrated, the first indication 632 and the second indication 634 may include a rectangular boundary around the respective words.”; 76, “For example, in response to a user input selecting a word from the alternative word menu 650, the initial text prompt and/or the subsequent text prompt may be updated to indicate that the selected word has replaced the original word in the initial text prompt and/or the subsequent text prompt.”); and displaying a second visual indication corresponding to the second recognized concept that will influence the generation of the automatically-generated visual media (para 74, “In some embodiments, the user interface 610 is updated to indicate that one or more alternative and/or complementary words have been identified for words in the initial text prompt and/or the subsequent text prompt.”; also, para 83, “As further described above, the user interface 710 and/or the text field 712 may visually distinguish the additional words using one or more methods, such as bold font, a different font color, text highlighting, and the like.”; also, para 78, “After receiving one or more subsequent user inputs and selections updating and/or modifying the initial text prompt and/or the subsequent text prompt, the user interface 610 may submit a new image generation request to the image generation system. In response, the image 670 may be replaced with a subsequent image generated from the modified text prompt.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of identifying a keyword within the entered prompt, offering the user variations of that keyword that are alternatives of the same kind, and replacing the identified keyword in the displayed prompt with the variation the user selects, with the features of Bean's invention of marking the word for which alternatives were found, offering those alternatives from the mark itself, offering an option to add a further word of the same kind to the prompt, and applying its marking to the words of the prompt as it currently stands. The combination would have been obvious because Benedetto's whole editing mechanism is the substitution of a same-kind alternative for a word it has identified, and its interface says nothing about which word is being acted on at any moment, so a user who has just swapped a word cannot tell whether the system is now acting on what he chose or on what he discarded. Bean marks the word the system is treating as a concept and re-marks the prompt as the prompt changes, and applying that convention through Benedetto's substitution yields the predictable result that the mark leaves the word that was replaced and appears on the word that replaced it, so that what the interface shows and what drives the image remain the same thing.
Regarding claim 17, Benedetto as modified by Bean discloses the method of claim 1, wherein Benedetto further discloses displaying the user interface further comprises displaying a text entry region configured to receive text corresponding to additional prompt information (para 47, “When the user provides additional text in the search field, the text analyzer 311 detects the additional text and dynamically parses the additional text to identify a second keyword kw 2, "field" in prompt 2.”; also, para 47, “The process of receiving additional keywords in the search field, identifying and providing keyword variations for each identified keyword, and updating prompt 2 continues so long as the user provides additional text input in the search field.”) corresponding to one or more concepts that will influence the generation of the automatically-generated visual media (para 48, “The adjusted context and the updated prompt 2 is also fed into the ML engine 320 as input so that appropriate image features influencing the content of prompt 2 can be identified for generating the image for the user prompt.”).
Regarding claim 20, Benedetto as modified by Bean discloses the method of claim 17, wherein Benedetto further discloses the text entry region includes the representation of the prompt; and the method further comprises (para 46, “In the example illustrated in FIG. 4A-1, the user begins to enter text input in the search field rendered in the user interface 110 at a display screen 105 of the client device 100. In the example illustrated in FIG. 4A-1, the user has provided an initial text prompt, "Dark sky".”): while displaying the user interface with the text entry region including the representation of the prompt (para 46, “The identified keyword variations for keyword kw 1 are returned to the client device for rendering in the user interface 110 for user selection.”; also, para 47, “The process of receiving additional keywords in the search field, identifying and providing keyword variations for each identified keyword, and updating prompt 2 continues so long as the user provides additional text input in the search field.”; also, para 46, “The identified keyword variations for keyword kw 1 are returned to the client device for rendering in the user interface 110 for user selection.”): receiving, via the one or more input devices, an input corresponding to a request to change one or more recognized concepts that will influence the generation of the automatically-generated visual media (para 46, “User selection of a keyword variation is used to replace the identified keyword kw 1 in prompt 1 to generate an adjusted prompt, prompt 2.”); and in response to receiving the input (para 66, “User selection of a particular keyword variation is received at the server and, in response, the aggregated user prompt is updated to include the keyword variation in place of the one or more keywords to generate an adjusted prompt, as illustrated in operation 575.”): displaying, via the display generation component, an updated representation of the prompt in accordance with the request to change the one or more recognized concepts (para 46, “The adjusted prompt, prompt 2 is forwarded to the client device for rendering in the search field at the user interface, where the user provides the text input.”).
Regarding claim 21, Benedetto as modified by Bean discloses the method of claim 1, wherein Benedetto further comprising, in response to receiving the prompt, displaying, via the display generation component, a representation of the automatically- generated visual media in the user interface that is influenced by the prompt information (para 62, “The images returned to the client device 100, in response to the user prompt include image features that are influenced by content of the respective user prompt, represent contextually relevant, visual representation of the user prompt.”; also, para 38, “Once the user has completed their input to the user prompt, the resulting normalized image is forwarded to the client device 100 for rendering.”).
Regarding claim 22, Benedetto as modified by Bean discloses the method of claim 21, wherein Benedetto further discloses comprising: representation of the automatically-generated visual media (para 38, “It should be noted that the analysis module 310, the AI model 320 and the image normalizing module 330 can process the user prompt on-the-fly as the user is providing the input for the user prompt.”; also, para 62, “The images returned to the client device 100, in response to the user prompt include image features that are influenced by content of the respective user prompt, represent contextually relevant, visual representation of the user prompt”), receiving, via the one or more input devices, a first input corresponding to a request to modify one or more recognized concepts that will influence the generation of the automatically-generated visual media (para 38, “As additional prompts are provided by the user, the additional prompts are analyzed using the analysis module 310 to identify the additional keywords and additional keyword sequences from the additional keywords.”); and in response to receiving the first input (para 38, “The context of the user prompt is refined by taking into consideration the additional keywords.”): displaying, via the display generation component, an updated representation of the automatically-generated visual media in accordance with the modifications of the one or more recognized concepts (para 38, “The additional keywords, additional keyword sequences and the refined context are used to dynamically adjust the image generated for the user prompt. The dynamic adjustment to the generated image continues so long as additional input is being provided by the user at the user prompt.”). Benedetto does not disclose while displaying the first visual indication corresponding to the first recognized concept.
However, in a similar field of endeavor, Bean discloses while displaying the first visual indication corresponding to the first recognized concept (para 74, “For example, and as illustrated, the user interface 610 may be updated to display a first indication 632 that one or more alternative words have been identified for the word "Car" in the initial text prompt and a second indication 634 the one or more complementary words have been identified for the word "Road" in the initial text prompt. As further illustrated, the first indication 632 and the second indication 634 may include a rectangular boundary around the respective words.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of processing the prompt on the fly while the generated image is displayed, receiving further input that changes the recognized concepts, and dynamically adjusting the displayed image to match the refined set of concepts, with the features of Bean's invention of keeping an indication on the prompt marking the word for which the system has identified related words. The combination would have been obvious because Benedetto adjusts the image continuously as the user works, so the user is comparing a changing image against a prompt whose acted-upon words he cannot see, and has no way to attribute a change in the image to the concept that produced it. Bean keeps the marking on the prompt itself, and carrying it through Benedetto's adjustment loop yields the predictable result that the marked concepts and the image they are driving are both on screen while the user edits, which is what lets the user judge which concept to change next.
Regarding claim 25, Benedetto as modified by Bean discloses the method of claim 21, wherein Bean further comprising: while displaying a second representation of the automatically-generated visual media, receiving, via the one or more input devices (para 59, “For example, and as illustrated, the user interface 410 may be updated to display a first image 452, a second image 454, a third image 456, and a fourth image 458 generated by the image generation system from the initial text prompt (e.g., "A car") in response to a user input requesting that four images be generated from the initial text prompt.”; also, para 64, “Likewise, the one or more selectable prompt suggestion options 420, and/or the one or more selectable image result variability options 425 may be configured to receive one or more user inputs modifying the initial selections.”), a sequence of one or more inputs corresponding to a request to display different automatically-generated visual media based on the prompt (para 61, “As further illustrated, the user interface 410 includes one or more selectable image result variability options 425 to vary the visual features represented in each of the images generated from the initial text prompt.”; also, para 64, “For example, after updating the subsequent text prompt to indicate the desired left-side perspective, the user may update the selections in the one or more selectable image result variability options 425 to vary future image results within the lighting class 460. Subsequently, the submission button 414 may be selected a subsequent time.”); in response to receiving the sequence of one or more inputs corresponding to the request to display different automatically-generated visual media based on the prompt: displaying, via the display generation component, a representation of a second automatically-generated visual media wherein (para 64, “In response, the images may be replaced with subsequent images generated from the modified text prompt and the modified text prompt may again be replaced with a subsequent text prompt.”): the second automatically-generated visual media is different from the automatically-generated visual media (para 37, “In some embodiments, the text-to-image generator 142 generates multiple images from an initial text prompt with varying visual features between each image.”; also, para 37, “As the text-to-image generator 142 begins generating images from the initial text prompt, the text-to-image generator 142 may apply a different variable from a selected class to each generated image.”; also, para 61, “Depending on the selected options from the one or more selectable image result variability options 425, visual features corresponding to the selected categories or classes may vary between each image generated from the initial text prompt.”); the second automatically-generated visual media was generated based on the prompt (para 37, “In some embodiments, the text-to-image generator 142 generates multiple images from an initial text prompt with varying visual features between each image.”; also, para 61, “As further illustrated, the user interface 410 includes one or more selectable image result variability options 425 to vary the visual features represented in each of the images generated from the initial text prompt.”); and the second automatically-`generated visual media is displayed at a location that was previously occupied by the second representation of the automatically-generated visual media (para 64, “In response, the images may be replaced with subsequent images generated from the modified text prompt and the modified text prompt may again be replaced with a subsequent text prompt.”; also, para 59, “For example, and as illustrated, the user interface 410 may be updated to display a first image 452, a second image 454, a third image 456, and a fourth image 458 generated by the image generation system from the initial text prompt (e.g., "A car") in response to a user input requesting that four images be generated from the initial text prompt.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of forwarding the completed user prompt for image generation and returning the rendered image to the client device, with the features of Bean's invention of displaying the images a prompt produced, offering selectable options that vary the visual features between the images made from that prompt, and replacing the displayed images with the images a subsequent submission returns. The combination would have been obvious because Benedetto returns one rendering of a prompt and offers the user no way to see another, so a user who is satisfied with the concepts he has assembled but not with the particular image they produced can move only by altering concepts he wanted to keep. Bean holds the prompt fixed and moves the variation into separate selectable options, and adding that to the combination yields the predictable result that the user can ask for further images of the concepts already marked and see them arrive in place of the ones he was judging.
Regarding claim 29, Benedetto as modified by Bean discloses the method of claim 1, wherein Benedetto further discloses comprising: receiving the input (para 38, “Once the user has completed their input to the user prompt, the resulting normalized image is forwarded to the client device 100 for rendering.”): displaying the user interface including the representation of the prompt and the first visual indication: receiving, via the one or more input devices, an input corresponding to a request to generate the automatically-generated visual media; initiating a process to generate the automatically-generated visual media.
However, in a similar field of endeavor, Bean discloses while displaying the user interface including the representation of the prompt and the first visual indication (para 74, “For example, and as illustrated, the user interface 610 may be updated to display a first indication 632 that one or more alternative words have been identified for the word "Car" in the initial text prompt and a second indication 634 the one or more complementary words have been identified for the word "Road" in the initial text prompt.”): receiving, via the one or more input devices, an input corresponding to a request to generate the automatically-generated visual media (para 53, “The submission button 314 may include a selectable option to submit the initial text prompt entered in the text field 312 and an indication of the selected options from the one or more selectable prompt suggestion category options 320 as the user input 302 to an image generation system.”); initiating a process to generate the automatically-generated visual media (para 53, “As described further above, such an image generation system may receive the user input 302 from the user interface 310 and generate an image and a subsequent text prompt for display by the user interface 310.”; also, para 78, “After receiving one or more subsequent user inputs and selections updating and/or modifying the initial text prompt and/or the subsequent text prompt, the user interface 610 may submit a new image generation request to the image generation system.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of forwarding the completed user prompt for image generation and returning the rendered image to the client device, with the features of Bean's invention of a submission button that the user selects, while the marked prompt is on screen, to submit the prompt to the image generation system and start the generation. The combination would have been obvious because Benedetto generates on the fly as the user types and so commits the user to whatever the analyzer has picked up at each keystroke, which leaves no point at which the user can review the concepts the interface has marked and then decide that the prompt is ready. Bean places an explicit submission control in the same interface that carries the marked prompt, and adding it to Benedetto yields the predictable result that generation begins when the user asks for it rather than continuously, which is what makes the marking worth reading before the image is made.
Regarding claim 36, Benedetto discloses an electronic device that is in communication with a display generation component (para 79, “The graphics subsystem 720 periodically outputs pixel data for an image from the graphics memory 718 to be displayed on the display device 710. The display device 710 can be any device capable of displaying visual information in response to a signal from the device 700, including a cathode ray tube (CRT) display, a liquid crystal display (LCD), a plasma display, and an organic light emitting diode (OLED) display. The device 700 can provide the display device 710 with an analog or digital signal, for example.”) and one or more input devices, the electronic device comprising (para 77, “User input devices 708 communicate user inputs from one or more users to the device 700. Examples of the user input devices 708 include keyboards, mouse, joysticks, touch pads, touch screens, still or video recorders/cameras, tracking devices for recognizing gestures, and/or microphones.”): one or more processors (para 76, “The device 700 includes a CPU 702 for running software applications and optionally an operating system. The CPU 702 includes one or more homogeneous or heterogeneous processing cores. For example, the CPU 702 is one or more general-purpose microprocessors having one or more processing cores.”); memory (para 77, “A memory 704 stores applications and data for use by the CPU 702.”); and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for (para 77, “A memory 704 stores applications and data for use by the CPU 702. A data storage 706 provides non-volatile storage and other computer readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, compact disc-ROM (CD-ROM), digital versatile disc-ROM (DVD-ROM), Blu-ray, high definition-DVD (HD-DVD), or other optical storage devices, as well as signal transmission and storage media.”; also, para 76, “The device 700 includes a CPU 702 for running software applications and optionally an operating system.”): receiving, via the one or more input devices, a prompt for use in creating automatically-generated visual media that is generated at least partially using one or more autonomous processes (para 46, “In the example illustrated in FIG. 4A-1, the user begins to enter text input in the search field rendered in the user interface 110 at a display screen 105 of the client device 100. In the example illustrated in FIG. 4A-1, the user has provided an initial text prompt, "Dark sky". The initial text prompt defines the prompt 1 provided by the user.”; also, para 24, “An image generation artificial intelligence (IGAI) process is used to receive the user prompt, analyze the user prompt to determine if the user prompt includes an image or keywords or both, determine the context of the user prompt, and identify image features that match the context and, where available, a style preferred by the user, and generate a single image with the identified image features that provides a visual representation of the user prompt.”); and in response to receiving the prompt, displaying, via the display generation component, a user interface that includes prompt information, wherein displaying the user interface includes concurrently displaying (para 46, “In response to detecting the text input of prompt 1 entered by the user, the text analyzer 311 dynamically parses the text input and identifies the keyword kw 1, "dark" included in prompt 1.”; also, para 46, “The identified keyword variations for keyword kw 1 are returned to the client device for rendering in the user interface 110 for user selection. An example set of keyword variations identified for the keyword "dark" is shown as checkboxes in box 106 of FIG. 4A-1.”): a representation of the prompt (para 46, “The adjusted prompt, prompt 2 is forwarded to the client device for rendering in the search field at the user interface, where the user provides the text input.”); and a influence generation of the automatically- generated visual media (para 48, “The adjusted context and the updated prompt 2 is also fed into the ML engine 320 as input so that appropriate image features influencing the content of prompt 2 can be identified for generating the image for the user prompt.”; also, para 62, “The images returned to the client device 100, in response to the user prompt include image features that are influenced by content of the respective user prompt, represent contextually relevant, visual representation of the user prompt.”). Benedetto does not disclose first visual indication that a first portion of the prompt has been identified as a first recognized concept.
However, in a similar field of endeavor, Bean discloses first visual indication that a first portion of the prompt has been identified as a first recognized concept (para 74, “In some embodiments, the user interface 610 is updated to indicate that one or more alternative and/or complementary words have been identified for words in the initial text prompt and/or the subsequent text prompt. For example, and as illustrated, the user interface 610 may be updated to display a first indication 632 that one or more alternative words have been identified for the word "Car" in the initial text prompt and a second indication 634 the one or more complementary words have been identified for the word "Road" in the initial text prompt. As further illustrated, the first indication 632 and the second indication 634 may include a rectangular boundary around the respective words.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of an electronic device having a processor and a memory holding the applications it runs, which provides a display device with the signal by which an image is displayed and receives user inputs communicated to it from input devices, which receives a text prompt entered by the user in a search field for use in creating an image generated by an image generation artificial intelligence process, which parses the entered text in response to its receipt so that a keyword within the prompt is identified, which concurrently renders the prompt in the search field together with the material returned for that keyword, and in which the identified keyword drives the image features of the generated image, with the features of Bean's invention of updating the user interface to display an indication marking the particular word of the entered text prompt for which the system has identified related words. The combination would have been obvious because Benedetto and Bean are both interfaces of the same assignee for editing a text prompt that drives an image generation model, and Benedetto leaves the user with no way to see which of the words he typed the analyzer acted on, expressly selecting one word of the entered prompt for treatment while ignoring another. Bean answers that need by drawing the boundary around the very word for which related words were found, and applying Bean's marking to the word Benedetto has already identified yields the predictable result of a prompt whose acted-upon words are visible to the user.
Regarding claim 37, Benedetto discloses a non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to perform a method comprising (para 96, “One or more embodiments can also be fabricated as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data, which can be thereafter be read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical and non-optical data storage devices. The computer readable medium can include computer readable tangible medium distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.”; also, para 76, “The device 700 includes a CPU 702 for running software applications and optionally an operating system.”): receiving, via one or more input devices, a prompt for use in creating automatically- generated visual media that is generated at least partially using one or more autonomous processes (para 46, “In the example illustrated in FIG. 4A-1, the user begins to enter text input in the search field rendered in the user interface 110 at a display screen 105 of the client device 100. In the example illustrated in FIG. 4A-1, the user has provided an initial text prompt, "Dark sky". The initial text prompt defines the prompt 1 provided by the user.”; also, para 24, “An image generation artificial intelligence (IGAI) process is used to receive the user prompt, analyze the user prompt to determine if the user prompt includes an image or keywords or both, determine the context of the user prompt, and identify image features that match the context and, where available, a style preferred by the user, and generate a single image with the identified image features that provides a visual representation of the user prompt.”); and in response to receiving the prompt, displaying, via a display generation component, a user interface that includes prompt information, wherein displaying the user interface includes concurrently displaying (para 46, “In response to detecting the text input of prompt 1 entered by the user, the text analyzer 311 dynamically parses the text input and identifies the keyword kw 1, "dark" included in prompt 1.”; also, para 46, “The identified keyword variations for keyword kw 1 are returned to the client device for rendering in the user interface 110 for user selection. An example set of keyword variations identified for the keyword "dark" is shown as checkboxes in box 106 of FIG. 4A-1.”; also, para 79, “The device 700 can provide the display device 710 with an analog or digital signal, for example.”): a representation of the prompt (para 46, “The adjusted prompt, prompt 2 is forwarded to the client device for rendering in the search field at the user interface, where the user provides the text input.”); and influence generation of the automatically-generated visual media (para 48, “The adjusted context and the updated prompt 2 is also fed into the ML engine 320 as input so that appropriate image features influencing the content of prompt 2 can be identified for generating the image for the user prompt.”; also, para 62, “The images returned to the client device 100, in response to the user prompt include image features that are influenced by content of the respective user prompt, represent contextually relevant, visual representation of the user prompt.”). Benedetto does not disclose a first visual indication that a first portion of the prompt has been identified as a first recognized concept.
However, in a similar field of endeavor, Bean discloses a first visual indication that a first portion of the prompt has been identified as a first recognized concept (para 74, “In some embodiments, the user interface 610 is updated to indicate that one or more alternative and/or complementary words have been identified for words in the initial text prompt and/or the subsequent text prompt. For example, and as illustrated, the user interface 610 may be updated to display a first indication 632 that one or more alternative words have been identified for the word "Car" in the initial text prompt and a second indication 634 the one or more complementary words have been identified for the word "Road" in the initial text prompt. As further illustrated, the first indication 632 and the second indication 634 may include a rectangular boundary around the respective words.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of computer readable code on a computer readable tangible medium, executed by a processor, that receives a text prompt entered by the user in a search field for use in creating an image generated by an image generation artificial intelligence process, that parses the entered text in response to its receipt so that a keyword within the prompt is identified, that concurrently renders the prompt in the search field together with the material returned for that keyword, and in which the identified keyword drives the image features of the generated image, with the features of Bean's invention of updating the user interface to display an indication marking the particular word of the entered text prompt for which the system has identified related words. The combination would have been obvious because Benedetto and Bean are both interfaces of the same assignee for editing a text prompt that drives an image generation model, and Benedetto leaves the user with no way to see which of the words he typed the analyzer acted on, expressly selecting one word of the entered prompt for treatment while ignoring another. Bean answers that need by drawing the boundary around the very word for which related words were found, and applying Bean's marking to the word Benedetto has already identified yields the predictable result of a prompt whose acted-upon words are visible to the user.
Claim(s) 5 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benedetto et al. (U.S. Pub. No. 20240193351) as modified by Bean (U.S. Pub. No. 20240320867), further in view of Parasnis et al. (U.S. Doc. 11809688).
Regarding claim 5, Benedetto as modified by Bean discloses the method of claim 1, first recognized concept includes visual media.
However, in a similar field of endeavor, Parasnis discloses wherein first recognized concept includes visual media (col 14, “After the user selects the asset Mocha Coke asset, the selected asset is included in the prompt tool 110 with a special formatting (e.g., different background shading) to illustrate that a specific option has been selected and is not just text that was typed.”; also, col 14, “The content-generation tool is aware that the selected asset has a plurality of images, and the asset may also have a custom model associated with this asset.”; also, col 7, “At a high level, selecting the start-training option 606 means telling the content-generation tool to learn about this particular product, so when an input in the prompt includes the "Superstar Shoes," the content-generation tool will generate images with the Superstar Shoes.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, in which a keyword identified within the entered prompt is marked in the interface as a concept that drives the generated image, with the features of Parasnis's invention of a concept carried in the prompt that is an asset holding a plurality of images rather than a word alone, whose images are what the generator has been trained on so that a prompt naming that asset produces images of that particular subject. The combination would have been obvious because Benedetto's concepts are confined to text, so a user who wants the image to reproduce a particular thing he already has a picture of has no way to express that thing as one of the concepts in the prompt. Benedetto does receive a source image, but it converts that image into a text description before anything is identified, so what its analyzer identifies is a keyword drawn from that description rather than the picture itself, and the picture is an input to the identification rather than one of the concepts identified. Parasnis carries the asset itself into the prompt as a distinguished element, keeps its images available to the generator, and states that the generator is trained on those images so that naming the asset in the prompt yields images of that subject, and adding that to the combination yields the predictable result that a concept in the prompt can be a picture as readily as a word, which is what lets the generated image render that specific subject rather than a generic one.
Regarding claim 12, Benedetto as modified by Bean discloses method of claim 1, wherein Bean further discloses displaying the user interface further includes: displaying a first plurality of representations of concepts representing recognized concepts of a first category (para 56, “Additionally, or alternatively, the user interface 310 may be updated to display one or more new text fields, each corresponding to a respective category or class of words selected in the one or more selectable prompt suggestion category options 320 and displaying the words associated with the respective category or class.”; also, para 55, “For example, a content word 332 (e.g., "Sports") and/or a content phrase 334 (e.g., "Mountain Road") may be highlighted or otherwise visually distinguished to identify their correspondence to the visual content class 330.”); while displaying the first plurality of representations of concepts: receiving, via the one or more input devices, an input corresponding to a request to display a second plurality of representations of concepts representing recognized concepts of a second category, the second category different from the first category (para 64, “For example, after updating the subsequent text prompt to indicate the desired left-side perspective, the user may update the selections in the one or more selectable image result variability options 425 to vary future image results within the lighting class 460.”; also, para 88, For example, categories of visual features, such as visual content, visual style, visual perspective, lighting, and the like, may be displayed to a user with an option to receive additional words describing a selected category of visual features represented in the image.”); and in response to receiving the input corresponding to the request to display the second plurality of representations of concepts, displaying, in the user interface, the second plurality of representations of concepts (para 56, “Additionally, or alternatively, the user interface 310 may be updated to display one or more new text fields, each corresponding to a respective category or class of words selected in the one or more selectable prompt suggestion category options 320 and displaying the words associated with the respective category or class.”; also, para 88, “For example, categories of visual features, such as visual content, visual style, visual perspective, lighting, and the like, may be displayed to a user with an option to receive additional words describing a selected category of visual features represented in the image”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of returning to the user, for rendering in the interface, the concepts identified for the entered prompt, with the features of Bean's invention of dividing the concepts it offers into named classes, displaying the words of a selected class in a field of their own, letting the user change which class he wants returned after he has worked with the first, and displaying the words of the newly selected class in response. The combination would have been obvious because Benedetto returns whatever concepts its analyzer produces for the words the user typed, with no way for the user to ask for concepts of a particular kind, so a user who wants a lighting concept must wait for one to be offered. Bean names the classes and lets the user select among them, and adding that to the combination yields the predictable result that the concepts on offer are the concepts of the kind the user asked for.
Parasnis discloses within a database of recognized concepts and within the database (col 24, “From, operation 3406, the method 3400 flows to operation 3408 where, in response to detecting a selection of asset in the menu, a list of products previously added to a data store is obtained.”; also, col 14, “As discussed above with reference to FIGS. 5-7, the content generation tool stores assets and metadata that can be used to create the new content. In addition, the metadata store may include details on tone, color, palette, etc., to be used for the user assets.”; also, col 20, “Each block type has an associated internal curated prompt corpus that is used to augment user provided prompts. For example, for an image block type, the prompt corpus can consist of categories such as camera, lighting, etc. Within the camera category, there are prompts such as "Nikon" or "Sony." In the lighting category, there can be prompts such as "Sunrise" or "glorious sunset."”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, in which the concepts offered to the user are divided into named classes and the user selects which class he wants returned, with the features of Parasnis's invention of holding the prompt concepts in a stored corpus divided into named categories, each category holding its own enumerated concepts. The combination would have been obvious because Bean names its classes and displays the words of the class the user selects, but leaves the concepts of a class to be produced for the occasion, so the same request can return different concepts at different times and the user cannot learn what is available to him. Parasnis keeps the concepts in a corpus organized by category rather than generating them anew, and adding that to the combination yields the predictable result that the concepts the interface offers for a category are that category's stored concepts, drawn from a database of recognized concepts.
Claim(s) 10, and 13-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benedetto et al. (U.S. Pub. No. 20240193351) as modified by Bean (U.S. Pub. No. 20240320867), further in view of Murray et al. (U.S. Pub. 20160179804).
Regarding claim 10, Benedetto as modified by Bean discloses the method of claim 7, wherein Benedetto further discloses: receiving the one or more inputs (para 66, “User selection of a particular keyword variation is received at the server and, in response, the aggregated user prompt is updated to include the keyword variation in place of the one or more keywords to generate an adjusted prompt, as illustrated in operation 575.”): one or more inputs corresponding to the request to modify the one or more recognized concepts that will influence the generation of the automatically-generated visual media include an input corresponding to a request to delete the first recognized concept, ceasing display of the first visual indication.
However, in a similar field of endeavor, Murray discloses the one or more inputs corresponding to the request to modify the one or more recognized concepts that will influence the generation of the automatically-generated visual media include an input corresponding to a request to delete the first recognized concept (para 44, “The bulk keyword management application 325 also allows the user to dissociate keywords 548 from the media items 550a-550i. Upon selection of one or more media items 550a-550i by the user, the corresponding keywords 548 associated with the one or more selected media items 550a-550i may be displayed in the keyword entry application 544. The user may then select a first link 568 labeled "X" or "remove" associated with the keyword 548, which dissociates the keyword 548 from the corresponding one or more media items 550a-550i.”; also, para 44, “The bulk keyword management application 325 receives the input from the first link 568 and subsequently dissociates the keyword 548 from the corresponding media items 550a-550i.”), ceasing display of the first visual indication (para 39, “The keyword entry application 544 may display all keywords 548 associated with each of media items 550a-550i.”; also, para 44, “In another embodiment, the user may select the first link 568 associated with the keyword 548 without first having to select one or more media items 550a-550i. All media items 550a-550i associated with the selected keyword 548 associated with the first link 568 would then be dissociated”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, in which a keyword identified within the entered prompt is marked in the interface as a concept that drives the generated image, with the features of Murray's invention of drawing each such keyword as its own discrete element carrying its own removal control, in a region defined to display every keyword that is currently associated. The combination would have been obvious because Benedetto and Bean reach a marked word only through the running text, so a user who no longer wants a concept must edit the sentence around it and is left to guess what became of the mark, and Bean says only that the user may remove or modify words in the text field. Murray makes the keyword itself the thing the user acts on and ties what is displayed to what is currently associated, and adding that to the combination yields the predictable result that removing the concept and removing its marking are one operation, which is what keeps the interface and the set of concepts driving the image in agreement.
Regarding claim 13, Benedetto as modified by Bean discloses the method of claim 1, further comprising: displaying the user interface that includes the prompt information, displaying a selectable option for removing a plurality of recognized concepts that will influence generation of the automatically-generated visual media; while displaying the selectable option, detecting, via the one or more input devices, an input directed towards the selectable option; and in response to detecting the input, removing the plurality of recognized concepts from being used to influence generation of the automatically-generated visual media: ceasing display of one or more visual indications corresponding to the plurality of the recognized concepts; and ceasing using the plurality of the recognized concepts to influence the generation of the automatically-generated visual media.
However, in a similar field of endeavor, Murray discloses while displaying the user interface that includes the prompt information, displaying a selectable option for removing a plurality of recognized concepts (para 45, “The bulk keyword management tool 540 may include a second link 562 to remove all keywords associated with the media items 550b-550d.”; also, para 39, “The keyword management tool 540 may include a keyword entry application 544. The keyword entry application 544 may include a keyword entry means 546 and one or more keywords 548 associated with at least one media item 550a-550i”); while displaying the selectable option, detecting, via the one or more input devices, an input directed towards the selectable option (para 45, “The bulk keyword management tool 540 may also include a third link 566 to remove all numeric keywords associated with the media items 550b-550d.”; also, para 44, “The bulk keyword management application 325 receives the input from the first link 568 and subsequently dissociates the keyword 548 from the corresponding media items 550a-550i.”): ceasing display of one or more visual indications corresponding to the plurality of the recognized (para 39, “The keyword entry application 544 may display all keywords 548 associated with each of media items 550a-550i.”; also, para 44, “The bulk keyword management application 325 receives the input from the first link 568 and subsequently dissociates the keyword 548 from the corresponding media items 550a-550i.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, in which a keyword identified within the entered prompt is marked in the interface as a concept that drives the generated image, with the features of Murray's invention of a single control, displayed together with the keywords themselves, whose stated function is to remove the keywords as a group. The combination would have been obvious because Benedetto's user reaches the concepts one at a time through the text, so abandoning a line of work and starting the prompt again means undoing every concept separately and is slow enough that a user will keep a prompt he has outgrown. Murray treats the assembled keywords as a set that one control can clear, and adding that to the combination yields the predictable result that the user can drop the whole set of concepts the interface is carrying and begin again, which is what makes starting over as cheap as continuing.
Bean discloses that will influence generation of the automatically-generated visual media (para 53, “The submission button 314 may include a selectable option to submit the initial text prompt entered in the text field 312 and an indication of the selected options from the one or more selectable prompt suggestion category options 320 as the user input 302 to an image generation system.”; also, para 53, “As described further above, such an image generation system may receive the user input 302 from the user interface 310 and generate an image and a subsequent text prompt for display by the user interface 310.”); and in response to detecting the input, removing the plurality of recognized concepts from being used to influence generation of the automatically-generated visual media (para 57, “For example, after reviewing the subsequent text prompt, a user may proceed to type or otherwise input additional words into the text field 312, remove or modify one or more of the additional words in the text field 312, and the like.”; also, para 78, “After receiving one or more subsequent user inputs and selections updating and/or modifying the initial text prompt and/or the subsequent text prompt, the user interface 610 may submit a new image generation request to the image generation system. In response, the image 670 may be replaced with a subsequent image generated from the modified text prompt.”); and ceasing using the plurality of the recognized concepts to influence the generation of the automatically-generated visual media (para 57, “For example, after reviewing the subsequent text prompt, a user may proceed to type or otherwise input additional words into the text field 312, remove or modify one or more of the additional words in the text field 312, and the like.”; also, para 78, “After receiving one or more subsequent user inputs and selections updating and/or modifying the initial text prompt and/or the subsequent text prompt, the user interface 610 may submit a new image generation request to the image generation system. In response, the image 670 may be replaced with a subsequent image generated from the modified text prompt.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, further in view of Murray, in which the concepts identified within the entered prompt are marked in the interface and a single control removes the marked concepts as a group, with the features of Bean's invention of submitting the prompt as it stands after words have been removed from it and replacing the displayed image with one generated from that modified prompt. The combination would have been obvious because clearing the marked concepts from the interface accomplishes nothing if the generator continues to work from the concepts the user has just discarded, so the interface state and the input to the model have to be the same thing. Bean states that what is sent on the next request is the modified prompt and that the image the user is shown is generated from that modified prompt, and adding that to the combination yields the predictable result that concepts removed by the control stop influencing the image as well as leaving the display.
Regarding claim 14, Benedetto as modified by Bean discloses the method of claim 13, wherein: removing the plurality of recognized concepts includes removing all of the recognized concepts that will influence the generation of the automatically-generated visual media.
However, in a similar field of endeavor, Murray discloses removing the plurality of recognized concepts includes removing all of the recognized concepts (para 45, “The bulk keyword management tool 540 may include a second link 562 to remove all keywords associated with the media items 550b-550d.”; also, para 44, “In another embodiment, the user may select the first link 568 associated with the keyword 548 without first having to select one or more media items 550a-550i. All media items 550a-550i associated with the selected keyword 548 associated with the first link 568 would then be dissociated.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, in which a keyword identified within the entered prompt is marked in the interface as a concept that drives the generated image, with the features of Murray's invention of a control whose stated scope is every keyword the tool is holding rather than a selection within them. The combination would have been obvious because a user who wants to abandon a prompt entirely is served by a control that is not qualified by a selection, and a control that clears only part of the set leaves him to judge what remains before he can begin again. Murray states the scope of its control as all of the keywords, and adding that to the combination yields the predictable result that one action returns the prompt interface to an empty state, which is what makes the result of the action certain to the user before he takes it.
Bean discloses that will influence the generation of the automatically-generated visual content media (para 57, “For example, after reviewing the subsequent text prompt, a user may proceed to type or otherwise input additional words into the text field 312, remove or modify one or more of the additional words in the text field 312, and the like.”; also, para 78, “After receiving one or more subsequent user inputs and selections updating and/or modifying the initial text prompt and/or the subsequent text prompt, the user interface 610 may submit a new image generation request to the image generation system. In response, the image 670 may be replaced with a subsequent image generated from the modified text prompt.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, further in view of Murray, in which the concepts identified within the entered prompt are marked in the interface and a single control removes every marked concept the interface is holding, with the features of Bean's invention of submitting the prompt as it stands once words have been taken out of it and replacing the displayed image with one generated from that modified prompt. The combination would have been obvious because emptying the interface of every concept accomplishes nothing if the generator keeps working from the concepts the user has just discarded, and a user who clears the prompt in order to start again expects the image to stop reflecting what he cleared. Bean states that what is submitted on the next request is the modified prompt and that the image shown is generated from that modified prompt, and adding that to the combination yields the predictable result that concepts removed by the control stop influencing the generated media as well as leaving the display.
Claim(s) 15-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benedetto et al. (U.S. Pub. No. 20240193351) as modified by Bean (U.S. Pub. No. 20240320867), further in view of Li et al. (U.S. Pub. 20250285343).
Regarding claim 15, Benedetto as modified by Bean discloses the method of claim 1, wherein Benedetto further comprising: receiving the first input (para 66, “User selection of a particular keyword variation is received at the server and, in response, the aggregated user prompt is updated to include the keyword variation in place of the one or more keywords to generate an adjusted prompt, as illustrated in operation 575.”): displaying the first visual indication corresponding to the first recognized concept, receiving, via the one or more input devices, a first input corresponding to a request to add a second recognized concept corresponding to a representation of a person that will influence the generation of the automatically-generated visual media; displaying, in the user interface, a second visual indication corresponding to the second recognized concept that will influence the generation of the automatically-generated visual media.
However, in a similar field of endeavor, Bean discloses while displaying the first visual indication corresponding to the first recognized concept (para 74, “For example, and as illustrated, the user interface 610 may be updated to display a first indication 632 that one or more alternative words have been identified for the word "Car" in the initial text prompt and a second indication 634 the one or more complementary words have been identified for the word "Road" in the initial text prompt. As further illustrated, the first indication 632 and the second indication 634 may include a rectangular boundary around the respective words.”); receiving, via the one or more input devices, a first input corresponding to a request to add a second recognized concept (para 76, “The user interface 610 may further display one or more options to replace a word in the initial text prompt and/or the subsequent text prompt with an alternative word and/or one or more options to add a word to the initial text prompt and/or the subsequent text prompt with a complementary word.”; also, para 77, “Based on the words displayed the table, the user may choose to replace a word in the subsequent text prompt, or add a word to the subsequent text prompt, by manually editing the text in the text field 612, dragging and dropping a word from the table to the text field 612, and the like.”); that will influence the generation of the automatically-generated visual media (para 78, “After receiving one or more subsequent user inputs and selections updating and/or modifying the initial text prompt and/or the subsequent text prompt, the user interface 610 may submit a new image generation request to the image generation system. In response, the image 670 may be replaced with a subsequent image generated from the modified text prompt.”; also, para 53, “The submission button 314 may include a selectable option to submit the initial text prompt entered in the text field 312 and an indication of the selected options from the one or more selectable prompt suggestion category options 320 as the user input 302 to an image generation system.”); displaying, in the user interface, a second visual indication corresponding to the second recognized concept that will influence the generation of the automatically-generated visual media (para 74, “In some embodiments, the user interface 610 is updated to indicate that one or more alternative and/or complementary words have been identified for words in the initial text prompt and/or the subsequent text prompt.”; also, para 83, “As further described above, the user interface 710 and/or the text field 712 may visually distinguish the additional words using one or more methods, such as bold font, a different font color, text highlighting, and the like.”; also, para 78, “After receiving one or more subsequent user inputs and selections updating and/or modifying the initial text prompt and/or the subsequent text prompt, the user interface 610 may submit a new image generation request to the image generation system. In response, the image 670 may be replaced with a subsequent image generated from the modified text prompt.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of identifying a keyword within the entered prompt that drives the image features of the generated image, receiving a user selection that changes the concepts of the prompt, and updating the prompt in response, with the features of Bean's invention of displaying an indication marking each identified word within the prompt, offering an option to add a further word to the prompt, applying its markings to the prompt as it currently stands, and submitting that prompt so that the displayed image is generated from it. The combination would have been obvious because Benedetto's editing is confined to swapping one identified keyword for a variation, which lets the user change a concept but never add one, its interface says nothing about which words the analyzer took from the prompt, and Benedetto expressly seeks to better understand the user's intentions for the prompt. Bean offers the add option, marks the word for which related words were found, re-marks the prompt as the prompt changes, and does not leave the added word inert, because it resubmits the modified prompt and replaces the displayed image with one generated from that modified prompt. Applying those conventions to Benedetto yields the predictable result that a concept added after the first is marked in the interface alongside the concept already marked and governs the image generated from the prompt that now carries it.
Li discloses corresponding to a representation of a person (para 18, “In one example, the system provides an improved method for avatar creation that provides a user experience for creating a generated avatar in which the user can upload at least one image such as of themselves, along with a style goal (e.g., a style image).”; also, para 100, “In some implementations, the first instruction string further comprises instructions to the multimodal model (e.g., the LMM 126a) to identify the at least one subject (e.g., a person) among a plurality of objects (e.g., people, tables, chairs, a dish, windows, shutters, and the like in a restaurant) depicted in the at least one subject image based on a size threshold (e.g., 25% of the photo).”; also, para 59, “In some implementations, instead of image style transfer based on one style image and one subject image as the above-discussed example, the system can generate one avatar based on a plurality of style images (e.g., a museum exhibit room image, a morning lake image, a living room with a fireplace image, and the like) and/or a plurality of subject images (e.g., a person and a cat).”; also, para 18, “The system applies a large language model (LLM, e.g., GPT-4V) to interpret the subject image to extract key features to be retained in the generated avatar (i.e., user avatar features).”; also, para 45, “The key identifying features of the subject are to be retained in the generated avatar. The LMM 126a then converts the key identifying features of the subject into a subject textural description 150b.”; also, para 18, “The LMM rewrites the key subject features and the style text description into an image generation prompt that is sent to a large visual model (LVM, e.g., DALLE-3) to create the avatar for the subject in the style.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, in which a concept identified within the entered prompt is marked in the interface, a concept added after the first receives its own marking, and the image is generated from the prompt as it stands, with the features of Li's invention of supplying a particular person as the added subject by uploading a picture of that person and reducing that person to a description written into the prompt. The combination would have been obvious because Benedetto and Bean take every concept from words the user has already typed, so a particular individual the user has in mind cannot enter the prompt at all, and a word such as a name tells the generator nothing about what that individual looks like. Li accepts the individual as a picture and converts the features that identify that individual into prompt text, and adding that to the combination yields the predictable result that the concept the user adds to the prompt can be a person rather than only a word the interface suggested, while the marking and the resubmission already in the combination treat that added concept exactly as they treat the concepts already there.
Regarding claim 16, Benedetto as modified by Bean discloses the method of claim 1, wherein Benedetto further comprising: receiving the first input (para 66, “User selection of a particular keyword variation is received at the server and, in response, the aggregated user prompt is updated to include the keyword variation in place of the one or more keywords to generate an adjusted prompt, as illustrated in operation 575.”): displaying the first visual indication corresponding to the first recognized concept, receiving, via the one or more input devices, a first input corresponding to a request to add a second recognized concept corresponding to a representation of an animal that will influence the generation of the automatically-generated visual media; displaying, in the user interface, a second visual indication corresponding to the second recognized concept that will influence the generation of the automatically-generated visual media.
However, in a similar field of endeavor, Bean discloses while displaying the first visual indication corresponding to the first recognized concept (para 74, “For example, and as illustrated, the user interface 610 may be updated to display a first indication 632 that one or more alternative words have been identified for the word "Car" in the initial text prompt and a second indication 634 the one or more complementary words have been identified for the word "Road" in the initial text prompt. As further illustrated, the first indication 632 and the second indication 634 may include a rectangular boundary around the respective words.”); receiving, via the one or more input devices, a first input corresponding to a request to add a second recognized concept (para 76, “The user interface 610 may further display one or more options to replace a word in the initial text prompt and/or the subsequent text prompt with an alternative word and/or one or more options to add a word to the initial text prompt and/or the subsequent text prompt with a complementary word.”; also, para 77, “Based on the words displayed the table, the user may choose to replace a word in the subsequent text prompt, or add a word to the subsequent text prompt, by manually editing the text in the text field 612, dragging and dropping a word from the table to the text field 612, and the like.”); that will influence the generation of the automatically-generated visual media (para 78, “After receiving one or more subsequent user inputs and selections updating and/or modifying the initial text prompt and/or the subsequent text prompt, the user interface 610 may submit a new image generation request to the image generation system. In response, the image 670 may be replaced with a subsequent image generated from the modified text prompt.”; also, para 53, “The submission button 314 may include a selectable option to submit the initial text prompt entered in the text field 312 and an indication of the selected options from the one or more selectable prompt suggestion category options 320 as the user input 302 to an image generation system.”); displaying, in the user interface, a second visual indication corresponding to the second recognized concept that will influence the generation of the automatically-generated visual media (para 74, “In some embodiments, the user interface 610 is updated to indicate that one or more alternative and/or complementary words have been identified for words in the initial text prompt and/or the subsequent text prompt.”; also, para 83, “As further described above, the user interface 710 and/or the text field 712 may visually distinguish the additional words using one or more methods, such as bold font, a different font color, text highlighting, and the like.”; also, para 78, “After receiving one or more subsequent user inputs and selections updating and/or modifying the initial text prompt and/or the subsequent text prompt, the user interface 610 may submit a new image generation request to the image generation system. In response, the image 670 may be replaced with a subsequent image generated from the modified text prompt.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of identifying a keyword within the entered prompt that drives the image features of the generated image, receiving a user selection that changes the concepts of the prompt, and updating the prompt in response, with the features of Bean's invention of displaying an indication marking each identified word within the prompt, offering an option to add a further word to the prompt, applying its markings to the prompt as it currently stands, and submitting that prompt so that the displayed image is generated from it. The combination would have been obvious because Benedetto's editing is confined to swapping one identified keyword for a variation, which lets the user change a concept but never add one, its interface says nothing about which words the analyzer took from the prompt, and Benedetto expressly seeks to better understand the user's intentions for the prompt. Bean offers the add option, marks the word for which related words were found, re-marks the prompt as the prompt changes, and does not leave the added word inert, because it resubmits the modified prompt and replaces the displayed image with one generated from that modified prompt. Applying those conventions to Benedetto yields the predictable result that a concept added after the first is marked in the interface alongside the concept already marked and governs the image generated from the prompt that now carries it.
Li discloses corresponding to a representation of an animal (para 52, “In FIG. 1C, the user adds a subject image 150c (i.e., a cat), and switches to a different style request (i.e., a Christmas-themed flat design style).”; also, para 70, “After the user drops a subject image of a baby elephant into the prompt enter box 225c, the chat pane 225 shows a prompt enter box 225d with an instruction of "Select one of the following images as a style image for generating avatar" and several style templates for the user to select in FIG. 2C.”; also, para 45, “In the following examples, the user requests to create stylized avatars of themselves, families, friends, pets, and the like."; also, para 52, " The key identifying features of the subjects are to be retained in the generated avatar. The LMM 126a then converts the key identifying features of the subjects into a subject textural description 150d.”; also, para 53, “Concurrently, the request processing unit 122 and the LMM 126a process the style text 152d (e.g., a Christmas-themed flat design style) as they did with the style text 152a, and the LVM 126b can process the textual prompt to generate an avatar with Christmas theme flat design style 154c.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, in which a concept identified within the entered prompt is marked in the interface, a concept added after the first receives its own marking, and the image is generated from the prompt as it stands, with the features of Li's invention of supplying a particular animal as the added subject by dropping a picture of that animal into the entry region and reducing that animal to a description written into the prompt. The combination would have been obvious because Benedetto and Bean take every concept from words the user has already typed, so a particular animal the user has in mind cannot enter the prompt at all, and a word such as a species name tells the generator nothing about which animal is meant. Li accepts the animal as a picture and converts the features that identify it into prompt text, and adding that to the combination yields the predictable result that the concept the user adds to the prompt can be an animal rather than only a word the interface suggested, while the marking and the resubmission already in the combination treat that added concept exactly as they treat the concepts already there.
Claim(s) 23, 27, and 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benedetto et al. (U.S. Pub. No. 20240193351) as modified by Bean (U.S. Pub. No. 20240320867), further in view of Karpman et al. (U.S. Doc. No. 11995803).
Regarding claim 23, Benedetto as modified by Bean discloses the method of claim 21, wherein Benedetto further comprising: receiving the first input (para 38, “Once the user has completed their input to the user prompt, the resulting normalized image is forwarded to the client device 100 for rendering.”: disclose while displaying the user interface including the representation of the automatically- generated visual media: receiving, via the one or more input devices, a first input corresponding to a request to generate the automatically-generated visual media based on the prompt corresponding to the one or more recognized concepts; ceasing display of the representation of the automatically-generated visual media; and displaying, via the display generation component, a second representation of the automatically-generated visual media, wherein the second representation of the automatically-generated visual media is of higher fidelity than the representation of the automatically-generated visual media.
However, in a similar field of endeavor, Bean discloses while displaying the user interface including the representation of the automatically- generated visual media: receiving, via the one or more input devices, a first input corresponding to a request to generate the automatically-generated visual media based on the prompt corresponding to the one or more recognized concepts (para 81, “After the image 770 has been generated in response to the user input 702, the user interface 710 may be updated to display the generated image 770”; also, para 84, “For example, after reviewing the subsequent text prompt describing the image 770, the user may manually enter additional words and/or edit existing words in the text field 712 before selecting the submission button 714 a subsequent time.”; also; para 83, “In some embodiments, the user interface 710 and/or the text field 712 visually distinguish the additional words in the subsequent text prompt from the words in the initial text prompt. For example, and as illustrated, the text field 712 includes a first indication 732 associated with the word "Sports" and a second indication 734 associated with the word "Mountain".”; also, para 84, “While not illustrated, the text field 712 may be configured to receive one or more additional user inputs modifying the initial text prompt and/or the subsequent text prompt for submission to the image generation system to receive a subsequent image.”); ceasing display of the representation of the automatically-generated visual media (para 84, “In response, the image 770 may be replaced with a subsequent image generated from the modified text prompt.”; also, para 78, “After receiving one or more subsequent user inputs and selections updating and/or modifying the initial text prompt and/or the subsequent text prompt, the user interface 610 may submit a new image generation request to the image generation system. In response, the image 670 may be replaced with a subsequent image generated from the modified text prompt.”) and displaying, via the display generation component, a second representation of the automatically-generated visual media (para 84, “In response, the image 770 may be replaced with a subsequent image generated from the modified text prompt.”; also, para 81, “After the image 770 has been generated in response to the user input 702, the user interface 710 may be updated to display the generated image 770.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of forwarding the completed user prompt for image generation and returning the rendered image to the client device, with the features of Bean's invention of displaying the generated image, marking within the displayed prompt the words the system has picked out of it, receiving a further submission of that same marked prompt from the user while the image is displayed, and replacing the displayed image with the image the further submission produces. The combination would have been obvious because Benedetto renders an image and stops, so a user looking at a result has no way to ask for anything further to be done with it, and nothing in Benedetto says what becomes of the result on screen if he could. Bean keeps the submission control available with the result on screen and states what happens to the displayed image when the user submits again, and adding that to the combination yields the predictable result that a request made while the image is displayed is answered at the place the image is displayed, the first image giving way to the second.
Karpman discloses wherein the second representation of the automatically-generated visual media is of higher fidelity than the representation of the automatically-generated visual media (col 23, “Results interface 414 displays (a preview of) the image(s) 1 to 6 generated by the text-to-image diffusion model 112 at 416 in response to the image generation request and a description field 418 that displays the text prompt submitted to the model.”; also, col 23, “Upon receiving the final, upsampled image(s) from the communication interface 122, the software application layer 124 can: cease to display the rendering message; replace display of the low-resolution preview (e.g., base) images with final, upsampled versions; and activate the regenerate affordance,”; also, col 5, “In some implementations, each high-resolution diffusion model 116 in the set of high-resolution diffusion models 116 defines a deep learning network configured to receive a low-resolution base image (e.g., 64 pixels by 64 pixels, 256 pixels by 256 pixels) and generate a higher-resolution version (e.g., copy) of the base image (e.g., 256 pixels by 256 pixels, 1024 pixels by 1024 pixels).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, in which the generated image is displayed with the prompt that made it, a further request may be made while it is displayed, and the displayed image is replaced by the image that request produces, with the features of Karpman's invention of making the first of those two displayed images a low-resolution preview and the second an upsampled copy of that same image. The combination would have been obvious because generation at full resolution takes long enough that a user given nothing to look at cannot tell whether the concepts he marked are taking the image where he wanted it, and Bean's own generator already applies super-resolution to an image when the prompt asks for a higher resolution version of it. Karpman shows the low-resolution version first and then swaps in the high-resolution version of that same image, and adding that to the combination yields the predictable result that the representation the user was looking at gives way to a higher-fidelity representation of the same media rather than to an unrelated one.
Regarding claim 27, Benedetto as modified by Bean discloses the method of claim 25, receiving the sequence of one or more inputs corresponding to the request to display different automatically-generated visual media based on the prompt, the representation of the second automatically-generated visual media had not been generated, the method further comprising in response to receiving the sequence of one or more inputs corresponding to the request to display different automatically-generated visual media based on the prompt, generating the representation of the second automatically-generated visual media based on the prompt.
However, in a similar field of endeavor, Karpman discloses wherein prior to receiving the sequence of one or more inputs corresponding to the request to display different automatically-generated visual media based on the prompt, the representation of the second automatically-generated visual media had not been generated (col 23, “Results interface 414 displays (a preview of) the image(s) 1 to 6 generated by the text-to-image diffusion model 112 at 416 in response to the image generation request and a description field 418 that displays the text prompt submitted to the model.”; also, col 23, “While the system is executing the set of the high-resolution diffusion models 116 to upsample base images corresponding to the image generation request, the software application layer 124 can keep the regenerate affordance in a disabled (e.g., deactivated) state, indicated visually to the user by greying out the regenerate affordance and/or displaying a "generating" message on the affordance and display a "rendering" message elsewhere on the results interface.”), the method further comprising in response to receiving the sequence of one or more inputs corresponding to the request to display different automatically-generated visual media based on the prompt, generating the representation of the second automatically-generated visual media based on the prompt (col 23, “The results interface 414 may also include a regenerate affordance (e.g., a button, a link) that enables the user to submit another image generation request (or resubmit an image generation task using the same prompt).”; also, col 23, “Upon receiving the final, upsampled image(s) from the communication interface 122, the software application layer 124 can: cease to display the rendering message; replace display of the low-resolution preview (e.g., base) images with final, upsampled versions; and activate the regenerate affordance,”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, in which selectable options vary the visual features between the images a prompt produces and a subsequent submission replaces the displayed images with the images it returns, with the features of Karpman's invention of running the generation only after the request has been submitted, reporting that generation is under way, and displaying the result when it arrives. The combination would have been obvious because a system that prepared the alternatives in advance would have to decide, before the user asked, how many further images of a prompt he would want and to spend the model on all of them, while a prompt admits more images than can usefully be prepared and the user may ask for none. Karpman runs the task when the request is made and tells the user it is running, and adding that to the combination yields the predictable result that the image the user is shown next is one the system produced because he asked for it rather than one held in reserve.
Regarding claim 30, Benedetto as modified by Bean discloses the method of claim 1, wherein Benedetto further comprising: first input: displaying (para 38, “Once the user has completed their input to the user prompt, the resulting normalized image is forwarded to the client device 100 for rendering.”), via the automatically-generated visual media is influenced by one or more concepts of the prompt, receiving, via the one or more input devices, a first input corresponding to a request to regenerate the automatically-generated visual media; display generation component, a second automatically- generated visual media based on the prompt corresponding to the one or more recognized concepts, wherein the second automatically-generated visual media is different from the automatically-generated visual media.
However, in a similar field of endeavor, Bean discloses based on the prompt corresponding to the one or more recognized concepts (para 74, “For example, and as illustrated, the user interface 610 may be updated to display a first indication 632 that one or more alternative words have been identified for the word "Car" in the initial text prompt and a second indication 634 the one or more complementary words have been identified for the word "Road" in the initial text prompt. As further illustrated, the first indication 632 and the second indication 634 may include a rectangular boundary around the respective words.”; also, para 78, “After receiving one or more subsequent user inputs and selections updating and/or modifying the initial text prompt and/or the subsequent text prompt, the user interface 610 may submit a new image generation request to the image generation system. In response, the image 670 may be replaced with a subsequent image generated from the modified text prompt.”); wherein the second automatically-generated visual media is different from the automatically-generated visual media (para 37, “In some embodiments, the text-to-image generator 142 generates multiple images from an initial text prompt with varying visual features between each image.”; also, para 37, “As the text-to-image generator 142 begins generating images from the initial text prompt, the text-to-image generator 142 may apply a different variable from a selected class to each generated image.”; also, para 61, “Depending on the selected options from the one or more selectable image result variability options 425, visual features corresponding to the selected categories or classes may vary between each image generated from the initial text prompt”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of forwarding the completed user prompt for image generation and returning the rendered image to the client device, in which a keyword identified within the entered prompt drives the image features of that image, with the features of Bean's invention of marking within the displayed prompt the words it has picked out as concepts, submitting that same prompt on the next request so that the image the user is shown is generated from it, and of a generator that produces images of one prompt with varying visual features between them. The combination would have been obvious because Benedetto returns a single rendering and treats it as the answer to the prompt, so a user who does not care for that rendering is given no reason to believe the prompt could yield anything else, and Benedetto's interface never shows him which of his words the analyzer took. Bean marks those words, sends the prompt that carries them, and states that the images one prompt produces differ from one another, and adding that to the combination yields the predictable result that asking again returns a picture generated from the concepts the interface is showing him and other than the one already shown.
Karpman discloses while the automatically-generated visual media is influenced by one or more concepts of the prompt, receiving, via the one or more input devices, a first input corresponding to a request to regenerate the automatically-generated visual media (col 23, “Results interface 414 displays (a preview of) the image(s) 1 to 6 generated by the text-to-image diffusion model 112 at 416 in response to the image generation request and a description field 418 that displays the text prompt submitted to the model.”; also, col 23, “The results interface 414 may also include a regenerate affordance (e.g., a button, a link) that enables the user to submit another image generation request (or resubmit an image generation task using the same prompt).”; also, col 23, “While the system is executing the set of the high-resolution diffusion models 116 to upsample base images corresponding to the image generation request, the software application layer 124 can keep the regenerate affordance in a disabled (e.g., deactivated) state, indicated visually to the user by greying out the regenerate affordance and/or displaying a "generating" message on the affordance and display a "rendering" message elsewhere on the results interface.”); display generation component, a second automatically- generated visual media (col 23, “Concurrently (e.g., after the base diffusion model has generated the base image and while the executing the high-resolution diffusion model(s), after the communication interface 122 has output the generated image(s)), the software application layer 124 can transition display of the waiting screen to the results interface.”; also, col 23, “Upon receiving the final, upsampled image(s) from the communication interface 122, the software application layer 124 can: cease to display the rendering message; replace display of the low-resolution preview (e.g., base) images with final, upsampled versions; and activate the regenerate affordance”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of forwarding the completed user prompt for image generation and returning the rendered image to the client device, in which a keyword identified within the entered prompt drives the image features of that image, with the features of Karpman's invention of a control displayed alongside the finished image and the prompt that made it, by which the user resubmits the generation task using that same prompt. The combination would have been obvious because Benedetto gives a user who is satisfied with the concepts he has assembled but not with the particular image they produced no move at all except to alter concepts he wanted to keep, since every path Benedetto offers runs through changing a keyword. Karpman separates the decision to ask again from the decision to change the prompt, and holds its control inactive until the image is finished and then activates it, and adding that to the combination yields the predictable result that the user can ask for the work to be done again on the concepts already marked in the interface.
Claim(s) 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benedetto et al. (U.S. Pub. No. 20240193351) as modified by Bean (U.S. Pub. No. 20240320867), further in view of Lee et al. “Diffusion Explainer: Visual Explanation for Text-to-image Stable Diffusion”, 05/04/2023, 2305.03509v1.
Regarding claim 24, Benedetto as modified by Bean discloses the method of claim 21, displaying the representation of the automatically-generated visual media includes animating the representation of the automatically- generated visual media over time.
However, in a similar field of endeavor, Lee discloses wherein displaying the representation of the automatically-generated visual media includes animating the representation of the automatically- generated visual media over time (sec 4.1, “Diffusion Explainer is an interactive visualization tool that explains how Stable Diffusion generates a high-resolution image from a text prompt, selected from the Prompt Selector (Fig. 1A). It incorporates an animation of random noise gradually refined and a Timestep Con troller (Fig. 1D) that enables users to visit each refinement timestep.”; also, sec 4.1, “The Refinement Comparison View visualizes the incremental im age generation process for two related text prompts to allow users to discover how prompts affect image generation (G3).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, in which a keyword identified within the entered prompt is marked in the interface as a concept that drives the generated image and the generated image is displayed alongside that prompt, with the features of Lee's invention of displaying the generated image as an animation that runs from noise to the finished picture. The combination would have been obvious because Benedetto shows the user nothing between the moment he asks for an image and the moment the finished image arrives, and the wait carries no information about whether the concepts he marked are taking the image where he wanted it. Lee shows the same process as a running animation and treats the animation as what tells the viewer how the prompt is steering the result, and adding that to the combination yields the predictable result that the representation the user is watching while he works the prompt moves over time, which is what lets him judge the effect of a concept before the image settles.
Claim(s) 28 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benedetto et al. (U.S. Pub. No. 20240193351) as modified by Bean (U.S. Pub. No. 20240320867), further in view of Williams et al. (U.S. Doc. 11650709).
Regarding claim 28, Benedetto as modified by Bean discloses the method of claim 21, displaying the representation of the automatically-generated visual media in the user interface includes displaying a selectable option for generating a three-dimensional representation of the automatically- generated visual media, and the method further comprises: detecting, via the one or more input device, an input directed towards the selectable option; and in response to detecting the input, generating a three-dimensional representation of the automatically-generated visual media.
However, in a similar field of endeavor, Williams discloses wherein displaying the representation of the automatically-generated visual media in the user interface includes displaying a selectable option for generating a three-dimensional representation of the automatically- generated visual media, and the method further comprises (col 16, “The screen 902 also displays 2D elements 904, 906, 908 that are graphics (e.g., object images) for viewing by the user.”; also, col 16, “In response to the cursor 910 being in an associated position with respect to the 2D element 906, the processing unit 130 then provides an indicator 930 (e.g., generated by the graphic generator 430) for display by the screen 902. The indicator 930 indicates to the user that the 2D element 906 has an associated 3D model that may be retrieved.”; also, col 19, “Optionally, the method 1100 further includes providing an indication that the 2D element is selectable to access the 3D model.”; also col 19, “Optionally, in the method 1100, the visual information comprises a graphic that is displayed in association with the 2D element.”): detecting, via the one or more input device, an input directed towards the selectable option (col 16, “If the user selects the 2D element 906 (e.g., by double-clicking on the 2D element 906, by pressing and holding, by providing an audio command, etc.), the processing unit 130 then obtains a 3D model that is associated with the 2D element 906.”; also, col 16, “The screen 902 further displays a cursor 910, which is controllable by the user via a user input device.”); and in response to detecting the input, generating a three-dimensional representation of the automatically-generated visual media (col 18, “In some embodiments, the model may be dynamically generated and communicated to the system when the user performs the designated action.”; also, col 17, “In some embodiments, the object 950 may be a 3D version of the 2D object 906 selected by the user. In some embodiments, the object 950 may be generated by the processing unit 130 based on the 3D model.”; also, col 19, “Optionally, in the method 1100, the 2D element indicates an object, and the 3D model is a three-dimensional version of the object.”; also, col 19, “Optionally, in the method 1100, the 3D model is a 3D rendering or 3D depiction of the 2D element.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, in which a keyword identified within the entered prompt is marked in the interface as a concept that drives the generated image and the generated image is displayed alongside that prompt, with the features of Williams's invention of displaying, in association with a displayed image, a graphic signifying that the image is selectable to reach a three-dimensional model of it, and of generating, when the user selects it, a three-dimensional rendering or depiction of that same displayed image. The combination would have been obvious because Benedetto's output is a flat picture and a user who wants the scene he described as a three-dimensional object has to rebuild it in another tool from nothing, and because a system that converted every image it produced would spend the geometry step on images the user has already rejected. Williams gates the conversion on a control shown with the image, generates the model when that control is used, and defines what it generates as a depiction of the element the control was shown with, so the operation attaches to whatever image the interface is displaying. Adding that to the combination, where the image the interface is displaying is the image the generator produced from the prompt, yields the predictable result that the user is offered on the generated image itself the choice to have a three-dimensional version of that generated image made.
Claim(s) 32 and 34 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benedetto et al. (U.S. Pub. No. 20240193351) as modified by Bean (U.S. Pub. No. 20240320867), further in view of Smetanin et al. (U.S. Pub. No. 20240296606).
Regarding claim 32, Benedetto as modified by Bean discloses the method of claim 1, wherein Benedetto further comprising: second prompt corresponds to a second recognized concept (para 46, “User selection of a keyword variation is used to replace the identified keyword kw 1 in prompt 1 to generate an adjusted prompt, prompt 2.”; also, para 47, “When the user provides additional text in the search field, the text analyzer 311 detects the additional text and dynamically parses the additional text to identify a second keyword kw 2, "field" in prompt 2.”; also, para 47, “As with the keyword variations for the keyword kw 1, the keyword variations for the second keyword kw 2 is identified in accordance to the updated context of prompt 2 and the style of the user.”); and one or more input devices, a first input corresponding to a request to generate second automatically-generated visual media based on a second prompt, in response to receiving the first input: in accordance with a determination that the second recognized concept satisfies one or more criteria, initiating a process to generate the second visual generative visual content media based on the second recognized concept; and in accordance with a determination that the second recognized concept does not satisfy the one or more criteria, forgoing initiating the process to generate the second automatically-generated visual media based on the second recognized concept.
However, in a similar field of endeavor, Bean discloses receiving, via the one or more input devices, a first input corresponding to a request to generate second automatically-generated visual media based on a second prompt (para 78, “After receiving one or more subsequent user inputs and selections updating and/or modifying the initial text prompt and/or the subsequent text prompt, the user interface 610 may submit a new image generation request to the image generation system. In response, the image 670 may be replaced with a subsequent image generated from the modified text prompt.”; also, para 53, “The submission button 314 may include a selectable option to submit the initial text prompt entered in the text field 312 and an indication of the selected options from the one or more selectable prompt suggestion category options 320 as the user input 302 to an image generation system.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of parsing the entered prompt and identifying a keyword within it as a concept that drives the generated image, with the features of Bean's invention of a submission control by which the user asks for an image to be generated from a prompt he has since revised. The combination would have been obvious because Benedetto generates as the user types and so has no moment at which a revised prompt is presented as a fresh request, which is the moment at which anything can be decided about the revised prompt. Bean gives the revised prompt its own submission to the image generation system, and adding that to the combination yields the predictable result that a second prompt carrying a second identified concept reaches the generator as a request of its own.
Smetanin discloses in response to receiving the first input: in accordance with a determination that the second recognized concept satisfies one or more criteria, initiating a process to generate the second visual generative visual content media based on the second recognized concept (para 109, “The automated image generation system 234 receives an image generation request originating from the first user device comprising the selected or entered text prompt at block 506. Prior to feeding the text prompt to the automated image generator 404, the content moderation engine 408 analyzes the text prompt to check for objectionable text objects or objectionable meaning/context (block 508). The content moderation engine 408 may be configured to check for specific words or phrases that are not allowable, or may implement a machine learning model that is trained to predict whether the text prompt may include objectionable content, or may lead to objectionable visual output (e.g., based on a predicted meaning or context of the relevant words).”; also, para 127, “As mentioned above, prior to the generating of images by the automated image generator 404, the content moderation engine 408 may analyze the text prompt 802 and the text prompt may only be transmitted to the automated image generator 404 if the content moderation engine 408 detects that the text prompt does not contain objectionable text objects, or is otherwise permissible, depending on the checking, filtering or moderation methodology employed by the content moderation engine 408.”; also, para 111, “At block 510, once the above-described check has been performed, and the text prompt has either been determined not to be a restricted prompt or automatically modified to obviate a restriction, the automated image generator 404 receives the text prompt and generates one or more images based on the text prompt.”); and in accordance with a determination that the second recognized concept does not satisfy the one or more criteria, forgoing initiating the process to generate the second automatically-generated visual media based on the second recognized concept (para 110, “In some examples, if the content moderation engine 408 determines that the text prompt is not allowable within the interaction system 100 (e.g., it is determined to be a restricted prompt), the text prompt may be rejected and a notification of the rejection may be presented to the first user on the first user device.”; also, para 100, “The automated image generation system 234 may be configured to prohibit the user from generating images based on objectionable, sensitive or unwanted content.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, in which the concepts identified within the entered prompt are marked in the interface and a revised prompt is submitted to the image generation system as a request of its own, with the features of Smetanin's invention of examining the words a submitted prompt carries against a standard before the prompt reaches the generator, passing the prompt on when it meets the standard and refusing it when it does not. The combination would have been obvious because Benedetto already separates the prompt into the individual concepts it has identified, which is the granularity at which Smetanin's check is stated to operate, and Benedetto has no answer at all for a prompt whose subject the operator will not render. Smetanin places the check between the request and the generator and states both outcomes, and adding that to the combination yields the predictable result that the request reaches the generator when the identified concept passes the standard and is stopped before the generator when it does not.
Regarding claim 33, Benedetto as modified by Bean and Smetanin discloses the method of claim 32, wherein Smetanin further discloses forgoing initiating the process to generate the second automatically-generated visual media based on the second recognized concept in response to receiving the first input further comprises: initiating a process to generate third automatically-generated visual media based on a third recognized concept corresponding to the second prompt that satisfies the one or more criteria and not based on the second recognized concept that does not satisfy the one or more criteria (para 110, “In some examples, the content moderation engine 408 may be configured to modify the text prompt, e.g., to remove objectionable or flagged words, or modify certain words automatically, and then feed the modified prompt to the automated image generator 404.”; also, para 102, “For instance, the content moderation engine 408 may reject the entire input prompt or modify it by replacing specific words or phrases with more appropriate ones. In other words, the restricted prompt may be adapted such that it is no longer classified as "restricted," and the adapted prompt can be passed to the automated image generator 404.”; also, para 111, “At block 510, once the above-described check has been performed, and the text prompt has either been determined not to be a restricted prompt or automatically modified to obviate a restriction, the automated image generator 404 receives the text prompt and generates one or more images based on the text prompt.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, further in view of Smetanin, in which the concepts identified within the entered prompt are marked in the interface and a request carrying a concept that fails the operator's standard is refused before generation, with the features of Smetanin's further teaching that the offending prompt may instead be adapted by removing the flagged word and the adapted prompt passed to the generator. The combination would have been obvious because a flat refusal returns the user nothing for a prompt that may have failed on one of several concepts, leaving him to guess which of his concepts to sacrifice and to submit the prompt again. Smetanin states the alternative in terms, keeping the remainder of the same prompt and generating from it while treating the flagged word alone as what must be given up, and adding that to the combination yields the predictable result that the user receives an image answering the concepts the operator will render instead of receiving nothing at all.
Regarding claim 34, Benedetto as modified by Bean and Smetanin discloses the method of claim 32, wherein Smetanin further discloses forgoing initiating the process to generate the second automatically-generated visual media based on the second recognized concept in response to receiving the first input further comprises: forgoing generating the second automatically-generated visual media (para 110, “In some examples, if the content moderation engine 408 determines that the text prompt is not allowable within the interaction system 100 (e.g., it is determined to be a restricted prompt), the text prompt may be rejected and a notification of the rejection may be presented to the first user on the first user device.”; also, para 100, “The automated image generation system 234 may be configured to prohibit the user from generating images based on objectionable, sensitive or unwanted content.”; also, para 127, “As mentioned above, prior to the generating of images by the automated image generator 404, the content moderation engine 408 may analyze the text prompt 802 and the text prompt may only be transmitted to the automated image generator 404 if the content moderation engine 408 detects that the text prompt does not contain objectionable text objects, or is otherwise permissible, depending on the checking, filtering or moderation methodology employed by the content moderation engine 408.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Benedetto's invention of receiving a prompt, identifying a keyword within it as a concept that will drive the image, and generating the image when the user asks for it, with the features of Smetanin's invention of withholding the prompt from the generator entirely when the prompt fails the standard. The combination would have been obvious because a system that made the image and then declined to show it would have spent the model on output it will not deliver and would have to hold that output somewhere. Smetanin refuses at the request rather than suppressing a result, stating that the prompt is only transmitted to the generator when it is permissible, and adding that to the combination yields the predictable result that nothing at all is generated for the concept that failed the standard.
Claim(s) 35 is/are rejected under 35 U.S.C. 103 as being unpatentable over Benedetto et al. (U.S. Pub. No. 20240193351) as modified by Bean (U.S. Pub. No. 20240320867) and Smetanin et al. (U.S. Pub. No. 20240296606), further in view of Pasumarthi et al. (U.S. Pub. 20250094619).
Regarding claim 35, Benedetto as modified by Bean and Smetanin discloses the method of claim 32, process to generate the second visual generative visual media based on the second recognized concept further comprises: displaying, via the display generation component, a visual indication indicating a portion of the second prompt that does not satisfy the one or more criteria in the user interface including a representation of the second prompt.
However, in a similar field of endeavor, Pasumarthi discloses wherein forgoing initiating the process to generate the second visual generative visual media based on the second recognized concept further comprises: displaying, via the display generation component, a visual indication indicating a portion of the second prompt that does not satisfy the one or more criteria in the user interface including a representation of the second prompt (para 46, “At step 210, a prompt query intended to be sent to an LLM is entered through a user interface (e.g., the user interface 122). The received prompt query can be displayed on the user interface.”; also, para 42, “The highlighter 126 can automatically highlight, in runtime, the extracted named entities within the prompt query 102 displayed on the UI 122. The highlighter 126 can use different highlighting properties or visual cues to differentiate named entities that are security compliant from named entities that are security noncompliant.”; also, para 63, “In this example, the security noncompliant named entities are further bounded within rectangular text boxes so that they are visually distinguished from the security compliant named entities.”; also, para 44, “In a first option, the prompt query 102 that is deemed to be security noncompliant is prohibited from being sent to the LLMs 170. The user 110 can be prompted to change or modify the prompt query 102, e.g., by removing the named entities that are highlighted to be security noncompliant, and then resubmit the modified prompt query 102 for evaluation.”; also, para 50, “In such circumstances, the prompt query can be prevented from sending to the LLM.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Benedetto in view of Bean, further in view of Smetanin, in which the concepts identified within the entered prompt are marked in the interface and a request carrying a concept that fails the operator's standard is refused before generation, with the features of Pasumarthi's invention of marking, inside the displayed prompt itself, the particular spans that failed the standard, and of marking them differently from the spans that passed, while the prompt is held back from the model. The combination would have been obvious because Benedetto's interface already identifies the individual portions of the prompt and already marks them where they sit, so the interface knows which portion carries the concept that failed and has somewhere to put the notice, and because a refusal that does not say which portion was the problem leaves the user to strike words one at a time to find it. Pasumarthi treats the marking as the instruction to the user, directing him to remove the spans that are highlighted as noncompliant and resubmit, and adding that to the combination yields the predictable result that the user is shown which portion of his own prompt to change rather than being told only that the request was refused.
Allowable Subject Matter
Claim 18-19, 26, and 31 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement of reasons for allowance:
Claim 18 recites that where a single text input is determined to correspond to a plurality of recognized concepts, one visual representation corresponding to that input is displayed indicating the plurality of recognized concepts of the text. The art performs the determination and the art performs the marking, but never in the same element. The closest art parses one entered text into several recognized units, identifying two keywords in a two word prompt and two overlapping keyword sequences in a three word prompt, and it acts on each of those units separately, opening its own set of substitutes for each; what it never does is gather them into one displayed thing. The same art, where it does display a single marked unit, marks it because the system treated that unit as one concept, a phrase assigned to one class or a span for which one set of substitutes was found, which is the converse of what the claim requires. Elsewhere the art's marking scheme is expressly one indication per word, a first indication on one word and a second indication on another, which is the arrangement of claim 2 rather than of claim 18. Nothing found conditions the display of a representation on a determination that one input carried several concepts, and nothing found displays a representation whose function is to signify that the text behind it was read as more than one thing. What the claim captures is a way of showing the user that one span of his text carried several concepts, and the art does not reach it.
Claim 19 recites that an input directed to the representation of claim 18 produces a second, different visual representation of the same text. That added step is itself present in the art, which on an input directed at a displayed representation of a concept re-renders that same concept differently, shading it, highlighting it, or drawing a box around it to show that it has been acted upon. Claim 19 is indicated as allowable because it depends from claim 18 and therefore carries claim 18's requirement of a single representation indicating a plurality of concepts of one text input, which the art does not reach.
Claim 26 recites that the input of the parent claim is a movement input directed towards the displayed representation of the generated media. The art contains movement inputs directed at displayed generated images, and it contains regeneration in place from an unchanged prompt, but never the two together. The closest art detects a horizontal swipe over an area occupied by generated images and replaces those images with others where they sit, but the images that arrive were generated in response to other requests and each carries its own prompt, so what the swipe produces is a different prompt's image rather than different media generated from the prompt at hand. Other art does regenerate from the same prompt and does put the new image in the place the old one occupied, but the input that produces it is the supplying of a value in an input field, and the one movement input that art provides propagates the generating formula into neighboring positions and therefore adds images at new positions rather than replacing the one dragged. Still other art states in a single passage that a user may reposition a generated object or may instead edit its prompt and cause it to be regenerated, treating the two as separate operations with separate consequences, the movement input merely relocating the same media. Nothing found directs a movement input at the displayed representation and, in consequence, displays different media generated from the same prompt in the place that representation occupied.
Claim 31 recites displaying a gallery of automatically-generated visual content, receiving an input directed towards a selectable option to display the user interface that includes the prompt information, and in response both ceasing display of the gallery and displaying that prompt interface. The art puts a gallery and a route into creation on one screen but does not surrender the gallery. The closest art carries, within the very interface that receives the prompt, a browsable menu of images previously produced by the generator paired with the prompts that produced them, and keeps a separate history of the user's own generations. The same art states in terms that it transitions the display of one interface to another when it means to, transitioning the creation interface to a waiting screen and the waiting screen back again, so its silence as to the gallery is deliberate rather than accidental. That art expressly keeps the gallery on screen while a generation is being processed, continuing to display it as an overlay so the user may browse it during the wait, which is the opposite of ceasing to display it. The only closing of an overlay the art describes returns the user to the creation interface from a settings panel on a swipe, rather than on a selectable option to display the prompt interface. No reference states that a gallery of generated content stops being displayed when the prompt interface opens. The claim treats the two views as alternatives rather than as layers, and the art does not.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jai Li whose telephone number is (571)272-1170. The examiner can normally be reached Mon-Thu between 06:00-16:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAI W LI/Junior Patent Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights