DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This Action is responsive to the Amendment filed on 6/3/2026. Claims 1-20 are pending in the case.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 10 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Claim 10 recites "after identifying the one or more characters by analyzing the pixels of the textual area indicated by the user marking, and (3) when a length of the one or more translated characters exceeds a length of the editable text box, to automatically expand the editable text box or to insert additional white space in the new graphic design image to accommodate the one or more translated characters while maintaining at least one design context attribute". There is no mention in the original specification of this limitations. Thus, the limitations include subject matter that was not described in the original specification.
If the examiner has overlooked the portion of the original specification that describes these features, then applicant should point it out in the response to this office action.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-9 and 11-20 are rejected under 35 U.S.C. 103 as being unpatentable over Kraus et al. (US 20220138402 A1, hereinafter Kraus) in view of Brdiczka et al. (US 20230305690 A1, hereinafter Brdiczka).
As to independent claim 1, Kraus teaches a data processing system comprising:
a processor (processor 814), and a machine-readable storage medium storing executable instructions (“Computing device 800 typically includes a variety of computer-readable media” paragraph 0077), which when executed by the processor, cause the processor alone or in combination with other processors to perform the following operations:
receiving, via a user interface of a client device, an indication of a user marking of a textual area in a graphic design image, the textual area showing one or more characters (“a user may invoke a text style suggestion service by selecting a particular text” paragraph 0020, Fig. 2 textual area 215 selected);
constructing, via a prompt construction unit, a first prompt by appending the graphic design image (graphic design image 210) and the user marking of the textual area (textual area 215) in the graphic design image to a first instruction string, the first instruction string including instructions to a character design and semantics of the one or more characters, and a position of the textual area in the graphic design image (“Upon selection, the text style tool looks up, extracts, or otherwise collects metadata about the selected text element and the user's design, such as position in the design, font features (e.g., density, ornateness, etc.), derived features like variance in background color around the text element, the semantic role of the text element in the design, and/or other features. The text style tool sends the extracted metadata or a representation thereof to the text style suggestion service,” paragraph 0020, “Additionally or alternative, in some embodiments, style selector 182 includes one or more neural networks or some other machine-learning model configured to predict a compatibility score for each candidate style based on encoded contextual metadata from the request)” paragraph 0048, 0082-0083);
providing as an input the first prompt to the variance, contrast, point size).” Paragraph 0020-0021, “Additionally or alternative, in some embodiments, style selector 182 includes one or more neural networks or some other machine-learning model configured to predict a compatibility score for each candidate style based on encoded contextual metadata from the request)” paragraph 0048, “A neural network can include a variety of deep learning models, including convolutional neural networks, recurrent neural networks, deep neural networks, and deep stacking networks, to name a few examples. Although some implementations are described with respect to neural networks, such as generative adversarial, and or other type of machine learning models” Paragraph 0082-0083);
providing the one or more characters, the one or more design context attributes, and the new graphic design image to the client device (“The identified compatible text styles are mapped to different positions of style wheel 116 (e.g., which may correspond to style wheel 250 of FIG. 2), and the user may scroll through the positions of style wheel 116 to iterate through the compatible style suggestions.” Paragraph 0045); and
causing the user interface of the client device to display at least one of the new graphic design image or an editable text box over the textual area in the graphic design image, wherein the editable text box shows the one or more characters based on the one or more design context attributes (“Upon selection of a particular text style suggestion (e.g., corresponding to the position of indicator 255 of style wheel 250 of FIG. 2), style preview component previews and/or applies the text suggestion to the user's design.” Paragraph 0045).
Kraus does not appear to expressly teach the first instruction string including instructions to a multimodal generative model to identify the one or more characters within the textual area by analyzing pixels of the textual area indicted by the user marking; and
providing as an input the first prompt to the multimodal generative model and receiving as an output the one or more characters, the one or more design context attributes, and the new graphic design image created based on the at least one changed design context attribute from the multimodal generative model.
Brdiczka teaches the first instruction string including instructions to a multimodal generative model to identify the one or more characters within the textual area by analyzing pixels of the text as indicated according to the text field (“As mentioned, the co-pilot design system 106 also uses the language abstraction model 304 to extract digital design elements associated with digital design document 402. For example, the co-pilot design system 106 utilizes the language abstraction model 304 to extract information associated with a text element by identifying metadata associated with the text (e.g., to identify a font, font color, font size, text placement, text box size, etc.). Additionally or alternatively, the language abstraction model 304 uses optical character recognition and/or a trained machine-learning model to identify a character count, wording, or semantic similarity.” Paragraph 0081, “as shown at an act 404 of FIG. 4, the co-pilot design system 106 receives one or more user interactions indicating at least one of an image, shape, or text.” Paragraph 0083, “the additional digital design document 824 comprises a text element 826 with text “Gourmet Cooking Class” as indicated according to the text field 822.” Paragraph 0150, “In some embodiments, generating the additional design language representation of the design variation comprises utilizing the generative language model to generate one or more of an image token, a shape token, or a text token of the additional design language representation.” Paragraph 0177); and
providing as an input the first prompt to the multimodal generative model (providing the text element 826 and image to generative language model)and receiving as an output the one or more characters, the one or more design context attributes, and the new graphic design image created based on the at least one changed design context attribute from the multimodal generative model (“as shown in FIG. 8E, the co-pilot design system 106 generates an additional digital design document 828 comprising a design variation (e.g., darkened text, changed frame, and different font) to the text element 826. Specifically, the co-pilot design system 106 generates the additional digital design document 828 in response to detecting a user interaction with the iteration control 832.” Paragraph 0154,0177).
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kraus to comprise the first instruction string including instructions to a multimodal generative model to identify the one or more characters within the textual area by analyzing pixels of the textual area indicted by the user marking; and providing as an input the first prompt to the multimodal generative model and receiving as an output the one or more characters, the one or more design context attributes, and the new graphic design image created based on the at least one changed design context attribute from the multimodal generative model. One would have been motivated to make such a combination to provide higher accuracy in mixed-font revision.
As to dependent claim 2, Kraus teaches the data processing system of claim 1, wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
receiving, via the editable text box, one or more new characters; and causing the editable text box to display the one or more new characters based on the one or more design context attributes (“Upon selection of a particular text style suggestion (e.g., corresponding to the position of indicator 255 of style wheel 250 of FIG. 2), style preview component previews and/or applies the text suggestion to the user's design.” Paragraph 0045).
As to dependent claim 3, Kraus teaches the data processing system of claim 2, Krause further teaches wherein the first instruction string further includes instructions to identify one or more new white spaces with a consistent color or style in the graphic design image (“Upon selection, the text style tool looks up, extracts, or otherwise collects metadata about the selected text element and the user's design, such as position in the design, font features (e.g., density, ornateness, etc.), derived features like variance in background color around the text element,” paragraph 0020) and the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
when determining that a length of the one or more new characters exceeds a length of the editable text box, causing the user interface of the client device to display an editable text box over the textual area and the one or more new white spaces in the graphic design image, wherein the editable text box shows the one or more new characters (As shown in Fig. 2, the editable text box over textual area 215 is expandable).
As to dependent claim 4, Kraus teaches the data processing system of claim 1, Krause further taches wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
causing the user interface of the client device to display the editable text box over the textual area in the new graphic design image (Fig. 2, editable text box over textual area 215).
As to dependent claim 5, Krause teaches the data processing system of claim 1, Krause further teaches wherein the design context attributes further include at least one of a visual hierarchy of the one or more characters in the graphic design image, a balance of the one or more characters with respect to white space in the graphic design image, a theme of the graphic design image, a formality of the graphic design image, or a color scheme of the graphic design image (“Upon selection, the text style tool looks up, extracts, or otherwise collects metadata about the selected text element and the user's design, such as position in the design, font features (e.g., density, ornateness, etc.), derived features like variance in background color around the text element, the semantic role of the text element in the design, and/or other features.” Paragraph 0020, 0034).
As to dependent claim 6, Krause teaches the data processing system of claim 1, Krause further teaches wherein the first instruction string further includes instructions to change the character design of the one or more characters based on the semantics of the one or more characters in the new graphic design image, and wherein the character design includes at least one of a font, color, size, style, angle, or transparency level (“Generally, any given phrase may have multiple ways to emphasize different words in a way that focuses the audience's attention on different semantic aspects. For example, “The BEACH you should be sitting on now” focuses attention on the subject of the phrase, whereas “The beach you should be sitting on NOW” focuses attention on the temporal aspect of the phrase.” Paragraph 0059).
As to dependent claim 7, Krause teaches the data processing system of claim 1, Krause further teaches wherein the first instruction string further includes instructions to change the position of the textual area in the graphic design image based on the semantics of the one or more characters in the new graphic design image (“There are many elements of visual stylization that can be applied to text or a text field, including font face, font color, point size, line spacing (leading), letter spacing (tracking and kerning), text position, line or field length, text background, and other effects, to name a few.” Paragraph 0015).
As to dependent claim 8, Krause teaches the data processing system of claim 1, Krause further teaches wherein the first instruction string further includes instructions to identify a background style of the graphic design image, and to change the character design of the one or more characters to match with the background style (“Upon selection, the text style tool looks up, extracts, or otherwise collects metadata about the selected text element and the user's design, such as position in the design, font features (e.g., density, ornateness, etc.), derived features like variance in background color around the text element, the semantic role of the text element in the design, and/or other features. The text style tool sends the extracted metadata or a representation thereof to the text style suggestion service,” paragraph 0020), and
wherein the editable text box shows the one or more characters in the changed character design that matches the background style (“There are many elements of visual stylization that can be applied to text or a text field, including font face, font color, point size, line spacing (leading), letter spacing (tracking and kerning), text position, line or field length, text background, and other effects, to name a few.” Paragraph 0015,0020).
As to dependent claim 9, Krause teaches the data processing system of claim 1, Krause further teaches wherein the first instruction string further includes instructions to identify a background of the graphic design image, and to change the background of the graphic design image to a higher contrast to the character design (“Upon selection, the text style tool looks up, extracts, or otherwise collects metadata about the selected text element and the user's design, such as position in the design, font features (e.g., density, ornateness, etc.), derived features like variance in background color around the text element, the semantic role of the text element in the design, and/or other features. The text style tool sends the extracted metadata or a representation thereof to the text style suggestion service,” paragraph 0020), and
wherein the user interface of the client device displays the at least one of the new graphic design image or the editable text box over the textual area in the changed background (“The text style suggestion service returns the selected text styles, and in some embodiments, a contextual analysis is performed to curate style suggestions that are compatible with the context of the user's design, for example, by selecting or modifying styles to avoid collisions with other design elements, match an existing design alignment, and ensure a threshold amount of readability (e.g., quantified based on font density, background color variance, contrast, point size). The user may iterate through the selected style suggestions to preview and/or apply the suggestions to the user's design.” Paragraph 0020, 0052).
As to dependent claim 11, Krause teaches the data processing system of claim 1, Krause further teaches wherein the first instruction string further includes instructions to re-write the one or more characters based on a user preference into one or more rewritten characters (“The user may iterate through the selected style suggestions to preview and/or apply the suggestions to the user's design.” Paragraph 0020, Fig. 2), and
wherein the user interface of the client device displays the at least one of the new graphic design image or the editable text box showing the rewritten one or more characters (“The user may iterate through the selected style suggestions to preview and/or apply the suggestions to the user's design.” Paragraph 0020, Fig. 2).
As to dependent claim 12, Krause teaches the data processing system of claim 1, Krause further teaches wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
causing the user interface of the client device to display an instruction for a user to edit the one or more characters displayed in the editable text box (“The user may iterate through the selected style suggestions to preview and/or apply the suggestions to the user's design.” Paragraph 0020, Fig. 2, characters 215 edited within the editable text box).
As to dependent claim 13, Krause teaches the data processing system of claim 12, Krause further teaches wherein the machine-readable storage medium further includes instructions configured to cause the processor alone or in combination with other processors to perform operations of:
upon receiving a character edited or added in the editable text, retrieving a corresponding character in the character design and displaying the corresponding character in the character design in the editable text in real-time or substantially real-time (“The user may iterate through the selected style suggestions to preview and/or apply the suggestions to the user's design.” Paragraph 0020, Fig. 2).
As to dependent claim 14, Krause teaches the data processing system of claim 1, Krause does not appear to expressly teach wherein the multimodal generative model is configured to process (i) the graphic design image and (ii) the user marking to generate, as the output, both (a) the one or more characters identified from the textual area and (b) the new graphic design image; and
wherein the multimodal generative model comprises an image encoder configured to encode the graphic design image and the user marking, and a text decoder configured to decode the one or more characters and an instruction to change the at least one design context attribute.
Brdiczka teaches wherein the multimodal generative model is configured to process (i) the graphic design image (“In some embodiments, generating the additional design language representation of the design variation comprises utilizing the generative language model to generate one or more of an image token, a shape token, or a text token of the additional design language representation.” Paragraph 0177) and (ii) the user interacting to generate (“as shown at an act 404 of FIG. 4, the co-pilot design system 106 receives one or more user interactions indicating at least one of an image, shape, or text.” Paragraph 0083, “the additional digital design document 824 comprises a text element 826 with text “Gourmet Cooking Class” as indicated according to the text field 822.” Paragraph 0150), as the output, both (a) the one or more characters identified from the textual area and (b) the new graphic design image (“as shown in FIG. 8E, the co-pilot design system 106 generates an additional digital design document 828 comprising a design variation (e.g., darkened text, changed frame, and different font) to the text element 826. Specifically, the co-pilot design system 106 generates the additional digital design document 828 in response to detecting a user interaction with the iteration control 832.” Paragraph 0154), and
wherein the multimodal generative model comprises an image encoder configured to encode the graphic design image and the user marking, and a text decoder configured to decode the one or more characters and an instruction to change the at least one design context attribute (“generating the additional design language representation of the design variation comprises utilizing the generative language model to predict, based on the design language representation, one or more attribute values for an attribute of the at least one of the image, the shape, or the text. In one or more embodiments, the attribute comprises an image attribute comprising at least one of a frame, opacity, filter, or blur; a shape attribute comprising at least one of a frame, color, or opacity; or a text attribute comprising at least one of a character count, frame, font, layout, opacity, or color.” Paragraph 0177).
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kraus to comprise wherein the multimodal generative model is configured to process (i) the graphic design image and (ii) the user marking to generate, as the output, both (a) the one or more characters identified from the textual area and (b) the new graphic design image, and wherein the multimodal generative model comprises an image encoder configured to encode the graphic design image and the user marking, and a text decoder configured to decode the one or more characters and an instruction to change the at least one design context attribute One would have been motivated to make such a combination to simplify the creative process for the users and increases user productivity and utilization.
Claims 15-20 are substantially the same as claims 1-3 and are therefore rejected under similar rational.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Kraus et al. in view of Brdiczka et al. and further in view of Agarwal et al. (US 20210073340 A1, hereinafter Agarwal).
As to dependent claim 10, Krause teaches the data processing system of claim 1, Krause does not appear to expressly teach wherein the first instruction string further includes instructions (1) to determine a language of the one or more characters, (2) to translate the one or more characters into another language based on a user preference, after identifying the one or more characters by analyzing the pixels of the textual area indicated by the user marking, and (3) when a length of the one or more translated characters exceeds a length of the editable text box, to automatically expand the editable text box or to insert additional white space in the new graphic design image to accommodate the one or more translated characters while maintaining at least one design context attribute, and
wherein the user interface of the client device displays the at least one of the new graphic design image or the editable text box showing the one or more translated characters.
Agarwal teaches wherein the first instruction string further includes instructions (1) to determine a language of the one or more characters, (2) to translate the one or more characters into another language based on a user preference (“GUI interface 120 also provides an area below 128 for allowing the user to select one or more target languages.” Paragraph 0033) after identifying the one or more characters by analyzing the pixels of the textual area indicated by the user marking (“In FIG. 1A, the document 100 includes images 102 and text elements 104. Each text element 104 is characterized by a particular size and contains text content 106. The size of a text element 104 may correspond to a length 110, which may be measured in pixels, millimeters (mm), inches, and so forth.” Paragraph 0032), and (3) when a length of the one or more translated characters exceeds a length of the editable text box, to automatically expand the editable text box or to insert additional white space in the new graphic design image to accommodate the one or more translated characters while maintaining at least one design context attribute (“In some embodiments, not only will problematic text be identified, but will also be automatically rectified. For example, the system may increase the length of a text element based on the determined reference length of the longest target language translation.” Paragraph 0054), and
wherein the user interface of the client device displays the at least one of the new graphic design image or the editable text box showing the translated one or more characters (“Fig. 1C, GUI 140 displaying results of translated characters).
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kraus to comprise wherein the first instruction string further includes instructions (1) to determine a language of the one or more characters, (2) to translate the one or more characters into another language based on a user preference, after identifying the one or more characters by analyzing the pixels of the textual area indicated by the user marking, and (3) when a length of the one or more translated characters exceeds a length of the editable text box, to automatically expand the editable text box or to insert additional white space in the new graphic design image to accommodate the one or more translated characters while maintaining at least one design context attribute, and wherein the user interface of the client device displays the at least one of the new graphic design image or the editable text box showing the translated one or more characters. One would have been motivated to make such a combination to assist a document designer in designing a document that is ready for translation into multiple target languages (Agarwal [0004]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
PEREZ-SORROSAL et al. US 20250166257 A1 teaches a method including: (i) identifying, by a processor, a generative machine learning model trained on image data, (ii) generating, by the generative machine learning model executed by the processor, an image based on at least one parameter, (iii) editing, by an image-editing algorithm executed by the processor, the image to comprise a specified string of text in a selected area of the image, and (iv) causing display, by the processor, of the edited image (Abstract).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAHELET SHIBEROU whose telephone number is (571)270-7493. The examiner can normally be reached Monday-Friday 9:00 AM-5:00 PM Eastern Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kieu Vu can be reached at 571-272-4057. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MAHELET SHIBEROU/Primary Examiner, Art Unit 2171
Prosecution Insights