DETAILED ACTION
Response to Arguments
Applicant's arguments filed with respect to claims 1-2, and 4-21 have been fully considered but are moot in view of the new ground(s) of rejection. The rejections are necessitated due to claim amendments.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine,
manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2, and 4-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. When reviewing independent claim 1, and based upon consideration of all of the relevant factors with respect to the claim as a whole, claims 1-20 are held to claim an abstract idea without reciting elements that amount to significantly more than the abstract idea and is/are therefore rejected as ineligible subject matter under 35 U.S.C. 101.
The Examiner will analyze Claim 1, and similar rationale applies to independent Claim 12 and Claim 20.
The rationale, under MPEP § 2106, for this finding is explained below. The claimed invention (1) must be directed to one of the four statutory categories, and (2) must not be wholly directed to subject matter encompassing a judicially recognized exception, as defined below. The following two step analysis is used to evaluate these criteria.
Step 1: Is the claim directed to one of the four patent-eligible subject matter categories: process, machine, manufacture, or composition of matter?
When examining the claim under 35 U.S.C. 101, the Examiner interprets that the claims is related to a machine since the claim is directed to a computing device for managing and analyzing digital images.
Step 2a, Prong 1: Does the claim wholly embrace a judicially recognized exception, which includes laws of nature, physical phenomena, and abstract ideas, or is it a particular practical application of a judicial exception?
The Examiner interprets that the judicial exception applies since Claim 1 limitation of select an image from a plurality of images; determine a processing priority for the selected image; generate a summary for the selected image are directed to an abstract. The claim is related to mental process by a person selecting a photo, based on priority, from a photo album, and writing down what is depicted in the selected photo.
customize the generated summary to provide a customized summary based identification of the one or more subjects in the image, identified relationship between a user profile and to the one or more subjects in the image it could also be done in human mind (mental process). The user can observe, evaluate, judge and generate an opinion using image subjects, user profile information, relationship and context.
update metadata associated with the selected image based on the customized summary it’s nothing more than indexing (attaching label) the image/photo based on the evaluation. In court decision, TLI communication LLC v. AV automotive L.L.C, the court has treated “attaching classification data to images for organized storage” as an abstract idea.
If/when the claim recites a judicial exception (i.e., an abstract idea enumerated in MPEP § 2106.04(a), a law of nature, or a natural phenomenon), the claim requires further analysis in Prong Two.
Step 2a, Prong 2: Does the claim recite additional elements that integrate the judicial exception into a practical application?
The additional claim limitations store the selected image and associated metadata in memory is a generic computer storage step.
query [inputting]a generative artificial intelligence (AI) model with image content and associated metadata is nothing more than insignificant extra solution activity.
A computing device is used to generally apply the abstract idea without limiting how it functions.
Step 2b: If a judicial exception into a practical application is not recited in the claim, the Examiner must interpret if the claim recites additional elements that amount to significantly more than the judicial exception.
Furthermore, the generic computer components or machine learning algorithm of the processor/memory recited as performing generic computer or machine learning functions that are well-understood, routine and conventional activities amount to no more than implementing the abstract idea with a computerized system.
The Examiner finds that Claims 2, 4-10, and 13-19 does not state significantly more since the claim only recites additional steps summarizing an image and indexing/updating the metadata of summary.
Thus, claims 1, 2, and 4-20 recite the same abstract idea and therefore are not drawn to the eligible subject matter as they are directed to the abstract idea without significantly more.
Therefore, all claims are rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 12 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (Pub. No. US 2021/0029389) in view of VALLIANI et al. (Pub. No. US 2017/0132821).
Claims 1, 12 and 20, Zhang teaches a computing device, comprising: a processor configured to: select an image from a plurality of images [Para. 55 “selecting visual media of interest for a social network's news feed”, it’s clear that selecting implies having multiple visual medias to choose from]; query (accepts a context as an input) a generative artificial intelligence (AI) model (personalized language model) with image content (recognized people or objects) and associated metadata to generate a summary (caption) for the selected image [Para. 63 “The personalized language model may be a model that accepts a context as an input (e.g., encompassing information describing the visual media such as recognized people or objects, metadata, sensor data, etc.). The context may be processed in view of rules and/or probabilities that generate sentences or phrases using a narrative style of the user”; Para. 70 “The communications server may provide the identified context to the personalized language model, which may use the context to generate a description of the visual media 308. The communications server may transmit the description, as a proposed caption 312, to the sending client.”], wherein the summary includes semantic information (contextual information) and one or more subjects extracted from the image (visual media) [Para. 22 “The visual media may be analyzed to determine contextual information (such as location information, people and objects in the video, time, etc.). For example, object recognition may be applied to a photo or to frames in a video stream. In another example, facial recognition may be applied to the visual media to identify people in the visual media”, para. 49 “In this media, a new object has been identified: a tent object 110. The system may provide this new context to the personalized language model and may generate a new caption for the new media (e.g., “First order of business: pitching the tent!”)”]; customize the generated summary (caption) to provide a customized summary based on identification of the one or more subjects within the image [Para. 105 “the visual media may be analyzed or processed to identify further information. Analyzing the visual media may include performing object and/or facial recognition on the visual media, analyzing audio data associated with the visual media, performing event detection, analyzing manually-applied tags, etc.”; Para. 106, 109, and 114. the term “customize” per se has not patentable weight. As long as the prior art teaches the rest portion of the limitation, it doesn’t have mention that it is customized]; update metadata associated with the selected image based on the customized summary [Para. 47 “In further embodiments, the system may automatically associate the caption with the visual media, but the user may be presented with an option to later alter or eliminate the caption, or replace the caption with another automatically or manually-generated caption”; Para. 125 “at block 476, the system may index the visual media using the caption. The contextual tags used to generate the caption may be associated with the visual media”]; and store the selected image and the updated metadata in memory [Para. 19, 103, 55 and 74].
However, Zhang doesn’t explicitly teach customize the generated summary to provide a customized summary based on identification of the one or more subjects within the image and identified relationships between a user profile (user profile 240), and the one or more subjects.
Vaiiliani teaches doesn’t explicitly teach customize the generated summary (caption) to provide a customized summary based (using) on identification (identify the name) of the one or more subjects (girl) within the image (picture) and identified relationships (daughter of a person viewing the picture) between a user profile (user profile 240), and the one or more subjects (girl) [Para. 68, 78, and 80].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Zhang’s personalized caption generation system so that it performs Valliani’s complete teaching of the claim limitation, namely using the caption engine to recognize a depicted person, obtain identity and relation with user context from user-profile information, and combine the object and user context data to select a caption scenario and generate the caption. This medication improves Zhang by making the caption responsive to the viewer’s relationship with depicted people, thereby increasing semantic relevance and personalization.
Claims 1, 2, 7, 12, 13, 16, 20 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Schult (Pub. No. US 2014/0095587) in view of Zhang et al. (Pub. No. US 2019/0200050) further in view of Hall (Pub. No. US 2014/0351687).
Regarding claims 1, 12, and 20 Schult teaches a computing device, comprising: a processor configured to: select an image from a plurality of images (device library of photos) [Para. 581 “Wireless computing appliance 6005 may be a smart phone, an I-pad, notebook, nook, or android device with a display and input mechanism such as a physical "dockable" keyboard or touch screen.”; and Para. 585]; Query (conducts search) with image content (facial recognition pattern of the photo) and associated metadata (caption text attributed to the photo) to generate (create) a summary (caption) for the selected image (photo) [Para. 470 and 460], wherein the summary (caption) includes semantic information (relationship between them) and one or more subjects (Jane and Joan) extracted from the image (photo) [Para. 463 and 473]; customize the generated (create) summary (caption) to provide (sends a recommendation) a customized summary based on identification (identify) of the one or more subjects (jane and Joan) within the image and identified relationships (data indicating the importance of both to the user) between a user (user Jim) and the one or more subjects (Jane and Joan) [Para. 460, 462, 463, and 473]; update (auto-populates) metadata (caption text attributed to the photo) associated with the selected image (photo) based on the customized summary (caption) [Para. 459, 463, and 473]; and store (cache) the selected image (photo) and associated the updated (latest “version”) metadata (caption text attributed to the photo) in memory (local cache memory) [Para. 459, 473, 537, and 538].
However, Schult doesn’t explicitly teach having a generative artificial intelligence (AI) model.
Zhang teaches query (provide the identified context) a generative (rules and/or probabilities that generate sentence or phrases) artificial intelligence (AI) model (personalized language model) with image content (information describing the visual media such as recognized people or objects) and associated metadata to generate a summary for the selected image [Para. 101, 62, 68, and 69].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Schult’s facial recognition caption recommendation system so that it performs Zhang’s complete teaching of the limitation, by adding logic to provide the identified context , containing information describing the visual media such as recognized people or object and metadata, to a personalized language model whose rules that generates or phrases generate a caption for the visual media. This modification improves Schult by generating detailed captions from combined visual and metadata context, thereby increasing caption detail and personalization.
Schult in view of Zhang doesn’t explicitly teach identified relationships between a user profile and the one or more subjects.
Hall teaches generated summary to provide (display) a summary (caption) based on (based on social graph proximity) identification (tagged in the image) of the one or more subjects (users tagged in the image) within the image (image and identified relationships (first-degree friends) between a user profile (user node corresponding to the particular user: user-profile page) and the one or more subjects (users tagged in the image) [Para. 16, 39, 40, 43, 44, and 52].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Schult’s facial recognition caption-recommendation system, modified by Zhang, so that it performs Hall’s complete teaching of the limitation, by configuring the generated summary to provide a summary based on identification (tagged in the image) of the one or more subjects (users tagged in the image) within the image and identified relationship between a suer profile and the one or more subjects. This medication improves Schult by making the summary responsive to the identified relationships between user profile and the subjects, thereby prioritizing personally relevant image subjects and increasing contextual personalization.
Regarding claims 2 and 13, Schult in view Zhang in view of Hall teaches all claim limitations as stated above. Furthermore, Dolson teaches wherein the processor is configured to determine the processing priority (score) for the selected image (photo) based on image quality (photo quality metrics) [Para. 25].
Regarding claims 7, 16, and 21, Schult teaches wherein the processor is configured to customize the generated summary (caption or suggested description) further based on one or more of: metadata associated with the selected image (photo data or metadata) [Para. 463 “If there is any photo data or metadata already associated with the user's contribution like place name, time, date, or the like, the caption or suggested description for the contribution may contain the added information such as "Jane and sister Joan at Dune State Park in July"”].
Claims 4-6, 8, 14, 15 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Schult (Pub. No. US 2014/0095587) in view of Zhang et al. (Pub. No. US 2019/0200050) further in view of Hall (Pub. No. US 2014/0351687) further in view of XIE et al. (Pub. No. US 2023/0394855).
Regarding claims 4 and 14, Schult teaches the processor is further configured to determine whether existing metadata (photo data or metadata) is available for the selected image and enhance based on the existing metadata (photo data or metadata) [Para. 463 “If there is any photo data or metadata already associated with the user's contribution like place name, time, date, or the like, the caption or suggested description for the contribution may contain the added information such as "Jane and sister Joan at Dune State Park in July”].
However, Shult in view Zhang in view of Hall doesn’t explicitly teach receive query results from the generative AI model in response to querying the generative AI model.
Xie teaches receive query result (plurality of image story caption candidates) from the generative AI model (generative language model 140) in response to querying (directly prompt) the generative AI model [Para 40 “the collected visual information 120 (tags 122, initial image caption 124, and object information 126) is then formatted into structured visual clues 130, which is used to directly prompt generative language model 140”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Schult in view Zhang in view of Hall caption generation workflow by incorporating Xie’s candite caption output from a prompted generative language model to Schult’s system receives model generated candidate captions and then uses Schult’s existing place, time, data, or other photo metadata to enhance the caption or descriptive field. This medication improvs Schult in view Dolson by grounding model generated caption results in available photo metadata, thereby reducing context poor caption fields and improving metadata accuracy for later image retrieval.
Regarding claim 5, Schult teaches the processor is configured to generate (recommendation) the summary (photo caption; photo description) for the selected image (photo) [Para. 473 “At step 4715, the system sends a recommendation to the client for a photo caption or for a photo description”].
However, Shult in view Zhang in view of Hall doesn’t explicitly teach the rest of claim limitations.
Xie teaches wherein the processor is configured to query (directly prompt) the generative AI model (generative language model 140) by querying a remote generative AI model (remote resource; cloud resource) [Para. 40 and para. 85].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Schult in view Zhang in view of Hall caption generation workflow by incorporating Xie’s candite caption output from a prompted generative language model to Schult’s system receives model generated candidate captions and then uses Schult’s existing place, time, data, or other photo metadata to enhance the caption or descriptive field. This medication improvs Schult in view Dolson by grounding model generated caption results in available photo metadata, thereby reducing context poor caption fields and improving metadata accuracy for later image retrieval.
Regarding claims 6 and 15, Schult in view Zhang in view of Hall doesn’t explicitly teach the claim limitations.
However, Xie teaches wherein the processor is configured to query (directly prompt) the generative AI model (model 140) by querying a local generative AI model (on-premises computing solution) [Para. 40, and para. 79].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Schult in view Zhang in view of Hall caption generation workflow by incorporating Xie’s candite caption output from a prompted generative language model to Schult’s system receives model generated candidate captions and then uses Schult’s existing place, time, data, or other photo metadata to enhance the caption or descriptive field. This medication improvs Schult in view Dolson by grounding model generated caption results in available photo metadata, thereby reducing context poor caption fields and improving metadata accuracy for later image retrieval.
Regarding claims 8 and 17, Schult in view Zhang in view of Hall doesn’t explicitly teach the claim limitations.
However, Xie teaches receive a user-generated query (caption focus 132) about a specific image (image 104); and customize the generated summary (image story caption candidates 144) further based on the received user-generated query (caption focus 132) [Para. 21 and 23].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Schult in view Zhang in view of Hall caption generation workflow by incorporating Xie’s candite caption output from a prompted generative language model to Schult’s system receives model generated candidate captions and then uses Schult’s existing place, time, data, or other photo metadata to enhance the caption or descriptive field. This medication improvs Schult in view Dolson by grounding model generated caption results in available photo metadata, thereby reducing context poor caption fields and improving metadata accuracy for later image retrieval.
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Schult (Pub. No. US 2014/0095587) in view of Zhang et al. (Pub. No. US 2019/0200050) further in view of Hall (Pub. No. US 2014/0351687) further in view of Gulli et al. (Pub. No. US 2007/0078846).
Regarding claims 9 and 18, Schult in view Zhang in view of Hall teaches all claim limitations above. Furthermore, Schult teaches determine the processing priority (score) for the selected image and generates (recommendation) the summary (photo caption; photo description) for the selected image in response to determining that the selected image is not a redundant image [Para. 473 “At step 4715, the system sends a recommendation to the client for a photo caption or for a photo description.”];
However, Schult in view Zhang in view of Hall doesn’t explicitly teach the rest of claim limitations.
Gulli teaches identify and group similar images (similar images; clustered together); identify redundant images (near duplicates) in the grouped images (cluster); and not determine a processing priority for the selected image or generate the summary for the selected image in response to determining that the selected image is a redundant image (remaining images) [Para. 43 “In certain embodiments, images that have a similarity number that is higher than a set threshold, are deemed to be near duplicates of each other”; Para. 44 “At block 261, according to certain embodiments, similar images are then clustered together”, “Accordingly, images A, B, and C can be clustered together, and a representative chosen (e.g., image A) to represent the cluster of images in an image search result display. The representative can be selected from the cluster based on, for instance, the quality of the image, compression value, and so forth the remaining images in the cluster (images B and C) can be made accessible by clicking on a data pointer (such as a hyperlink), or can be eliminated”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Schult in view Zhang in view of Hall phot processing workflow by adding Guilli’s similarity clustering and representative image selection to generate caption while redundant/unselected images are eliminated from that processing path. This modification improves Schult in view of Dolson by presenting duplicate from being scored and captioned separately, thereby reducing redundant computation and avoiding inconsistent metadata/captions for substantially the same image.
Claims 10 are rejected under 35 U.S.C. 103 as being unpatentable over Schult (Pub. No. US 2014/0095587) in view of Zhang et al. (Pub. No. US 2019/0200050) further in view of Hall (Pub. No. US 2014/0351687) further in view of Huerta et al. (Pub. No. US 2023/0122787).
Regarding claim 10, Schult in view Zhang in view of Hall doesn’t explicitly teach the claim limitations.
However, Huerta teaches wherein, in response to determining that a nearby device (remote connection is already established and active; near, active, and connected) is available and connected, the processor is further configured to offload (transmitting the data packet) tasks to the nearby device (remote computing device) and receive corresponding processing results (receiving a data analysis; data analysis 204) [Para. 27, 6, 41, 44 and 46].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Schult in view Zhang in view of Hall to teach the claim limitations, feature as taught by Huerta; because the modification enables the system to select the main subject more accurately from plurality of subject in the image.
Claims 11 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Schult (Pub. No. US 2014/0095587) in view of Zhang et al. (Pub. No. US 2019/0200050) further in view of Hall (Pub. No. US 2014/0351687) further in view of Stanley-Marbell et al. (Pub. No. US 2015/0351033).
Regarding claim 11 and 19, Schult in view Zhang in view of Hall doesn’t explicitly teach the claim limitations.
However, Stanley teaches monitor availability and use of battery (battery charge level; energy usage; battery power level) and processing resources (processor utilization metrics; cpu usage) of the computing device (mobile device) [Para. 5 “in some implementations, a mobile device can be configured to monitor environmental, system and user events.”; Para. 53 “Mobile device 100 can monitor the battery charge level and determine when the charge level changes by a threshold amount and generate a “battery Level” event to record the charge level of the battery.”; Para. 55 “For example, when a user invokes a function (e.g., invocation of an application, illumination of the display, transmission of data, etc.) of mobile device 100, mobile device 100 can monitor the energy usage over a period of time that the function is executing to estimate how much energy each activity or function uses”; and Para. 158 “For example, sampling daemon 102 can be configured to interface with a voter that monitors the thermal conditions of mobile device 100, a voter that monitors CPU usage of mobile device 100 and/or a voter that monitors battery power level of mobile device 100”]; and adjust processing schedules (set timers; application launch forecast; background task) to balance tradeoffs (tradeoff between battery lifetime, power requirement, thermal management and performance) between performance (performance; user experience) and resource consumption (energy budgets) based on a result of the monitoring (current mobile device conditions and budges; voter feedback) [Para 34, 177, 178, and 192].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Schult in view Zhang in view of Hall to teach the claim limitations, feature as taught by Stanley; because the modification enables the system to select the main subject more accurately from plurality of subject in the image.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOLOMON G BEZUAYEHU whose telephone number is (571)270-7452. The examiner can normally be reached on Monday-Friday 10 AM-8 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 888-786-0101 (IN USA OR CANADA) or 571-272-4000.
/SOLOMON G BEZUAYEHU/
Primary Examiner, Art Unit 2666
