DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s amendment was received on 5/12/26 and has been entered and made of record. Currently, claims 1-25 are pending.
Response to Arguments
Applicant's arguments filed 5/12/26 have been fully considered but they are not persuasive.
The applicant asserts the combination of Hamedi (US 2020/0210764) and Xie et al. (US 2023/0394855) do not disclose nor render obvious at least the feature of generating a prompt to a large language model that is based on one or more emption attributes derived from an image. The Examiner respectfully disagrees as the combination of Hamedi and Xie disclose the above mentioned feature and are believed to be combinable. Particularly, Hamedi discloses extracting image features 210 from an image 202. The features extracted from the image 202 can be or can include any stylistic features that may relate to any visual characteristic of the image, such as expressions and emotions of people within the image (para 78). Xie discloses generating a caption for an image using an image and associated features as a prompt. “Visual clues” are used to prompt a large language model to generate a caption. The visual clues can be, among other things, image tags and object attributes parsed from the input image (paras 18-20 and 26). Xie does not specifically state how the image tags or visual clues are determined. The process described by Hamedi would provide the system of Xie with the exact method performed to obtain image tags. Both Hamedi and Xie are in the same field of endeavor, utilizing artificial intelligence to extract information from image data. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to combine the large language model generating a description of an image, as described by Xie, with the system of Hamedi. The suggestion/motivation for doing so would have been to eliminate the need for manual captioning thereby saving time and increasing system efficiency.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-25 are rejected under 35 U.S.C. 103(a) as being unpatentable over Hamedi (US 2020/0210764) in view of Xie et al. (US 2023/0394855).
Regarding claims 1, 13, and 25, Hamedi discloses a non-transitory computer-readable medium storing instructions, a computer-implemented method, and a system comprising:
a processor configured to execute a plurality of instructions to cause the system to:
extract, from an image, one or more stylistic visual attributes of the image using a first trained machine learning model (see paras 49-50, 76-78, and 92, high-level stylistic features of an image are extracted, such as type of object shown, dominant color scheme, brightness or contrast of the image, etc.); and
map the one or more stylistic visual attributes to one or more emotion attributes using a second trained machine learning model (see paras 76-78 and 93, stylistic features can propagate through a plurality of layers of a trained machine learning model to generate emotion attributes).
Hamedi does not disclose expressly generate a prompt to a large language model (LLM), the prompt being based on the one or more emotion attributes; provide the generated prompt to the LLM; and obtain, from the LLM, a generated description of the image.
Xie discloses generate a prompt to a large language model (LLM), the prompt being based on the one or more emotion attributes (see Fig. 2 and paras 15 and 18, a prompt based on visual cues from an image are generated);
provide the generated prompt to the LLM (see Fig. 2 and paras 15 and 18, a prompt based on visual cues from an image are generated and provided to a large language model); and
obtain, from the LLM, a generated description of the image (see paras 19-21 and 27, the large language model generates a description of the image).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to combine the large language model generating a description of an image, as described by Xie, with the system of Hamedi.
The suggestion/motivation for doing so would have been to eliminate the need for manual captioning thereby saving time and increasing system efficiency.
Therefore, it would have been obvious to combine Xie with Hamedi to obtain the invention as specified in claims 1, 13, and 25.
Regarding claims 2 and 14, Hamedi further discloses wherein the first trained machine learning model is a trained deep neural network (see Fig.4 and paras 80 and 93, a multi-layered machine learning model is used to extract visual attributes from images).
Regarding claims 3 and 15, Hamedi further discloses wherein the second trained machine learning model is a trained neural network (see Fig.4 and paras 80 and 93, one or more multi-layered machine learning model is used to extract visual attributes from images).
Regarding claims 4 and 16, Hamedi further discloses wherein the prompt includes at least one of the one or more emotion attributes (see para 78, stylistic features are fed to a multi-layered machine learning model to generate an emotional attribute).
Regarding claims 5 and 17, Xie further discloses wherein the processor is further configured to execute the instructions to cause the system to incorporate a generic description of the image into the prompt (see Fig. 2 and paras 19-21, generic descriptions can be utilized).
Regarding claims 6 and 18, Xie further discloses wherein the processor is further configured to execute the instructions to cause the system to retrieve the generic description of an object from a description database (see Fig. 2 and paras 19-21, generic descriptions can be utilized).
Regarding claims 7 and 19, Xie further discloses wherein the processor is further configured to execute the instructions to cause the system to provide the image to a descriptor text generator to obtain the generic description for incorporation into the prompt (see paras 18-21, the large language model generates a description of the image based on image tags and object attributes).
Regarding claims 8 and 20, Hamedi further discloses wherein the image comprises an object (see paras 50-51, 76, and 78, the image can contain objects, high-level stylistic features of an image are extracted, such as type of object shown).
Regarding claim 9, Xie further discloses wherein the generated prompt further comprises a name of the object in the image (see Fig. 2 and para 19, a name, such as “man”, is an object name).
Regarding claims 10 and 21, Hamedi further discloses wherein the processor is further configured to execute the instructions to cause the system to incorporate physical attributes of the object into the prompt (see para 78, physical attributes, such as facial attributes are utilized).
Regarding claims 11 and 23, Hamedi further discloses wherein the visual attributes are extracted from a plurality of multiple images of the object (see paras 50 and 74, a plurality of images can be used, the visual attributes are then extracted from each one of the multiple images).
Regarding claims 12 and 24, Hamedi further discloses wherein the visual attributes are common visual attributes to each of the multiple images (see paras 50 and 78, high-level stylistic features of the images are extracted, such as type of object shown, dominant color scheme, brightness or contrast of the image, etc.).
Regarding claim 22, Hamedi further discloses extracting the physical attributes of the object from an object attribute database (see paras 76-78, object attributes are selected from a predetermined list).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARK R MILIA whose telephone number is (571) 272-7408. The examiner can normally be reached Monday-Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Akwasi Sarpong can be reached at 571-270-3438. The fax number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARK R MILIA/ Primary Examiner, Art Unit 2681
Prosecution Insights