DETAILED ACTION
This action is in response to the Amendment dated 24 June 2026. Claims 1, 10 and 11 are amended. Claim 12 has been added. No claims have been cancelled. Claims 1-12 remain pending and have been considered below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4-8 and 10-12 are rejected under 35 U.S.C. 103 as being unpatentable over Goyal et al. (US 2018/0300046 A1) in view of Gupta et al. (US 2021/0217152 A1).
As for independent claim 1, Goyal teaches an apparatus comprising:
a processor configured to: specify a recommended operation that is an operation to be performed with respect to target content that is electronic content operated by a user by inputting state information indicating a state of the target content into a learning model that is trained to predict and output an operation to be performed with respect to the electronic content based on a state of the electronic content using training data including state information indicating the state of the electronic content and operation information indicating an operation performed in the past with respect to the electronic content [(e.g. see Goyal paragraphs 0021, 0034, 0045) ”a cognitive system is further provided to make recommendations for augmenting/replacing an image section or object of interest based upon a prior knowledge of the user. This prior knowledge is used to train a machine learning system or model to make recommendations for editing/replacing portions of a current photographic or video frame(s) image. Thus, for example, various recommendations for editing/replacing the photograph (or portion thereof) with identified similar (or most relevant) image content, may be automatically presented to the user based upon that user's prior historical usage, e.g., how that user has edited/replaced similar images in photographs in the past … FIG. 3 depicts an example embodiment of a method 300 for ingesting data relating to past historical user actions and building a learned recommendation model for providing a cognitive ability of the system of FIG. 2. … over time, as the user takes pictures and makes edits to them, the cognitive training/recommender system 280 implements machine learning techniques that ingest, reason and learn the user's preferences (e.g., types of edits made to photographs/video frames) which are stored in the knowledgebase and used for mapping to object editing recommendations. In one embodiment, over time, the selection of the image(s) that require ‘work’ can be achieved through the system's historical references. Once a pattern of selection from the user is determined based on historical information accessed from the knowledgebase 260, the system 200 automatically presents available image editing and/or replacement options to the user via display interface 258 As the system learns the preferences of the user the versions presented will be tailored to their selection and quality needs. It is understood that, over time, the image set presented may change as the system learns which types of images and their make up the user is most likely to select”].
notify the user of the recommended operation [(e.g. see Goyal paragraphs 0021, 0045) ”A generated option(s) or recommendation(s) using the cognitive ability of the system may be presented to the user with suggestions to take any action with respect to the digital image … the system 200 automatically presents available image editing and/or replacement options to the user via display interface”].
Goyal does not specifically teach wherein the state information includes attributes of the target content and an operation performed so far with respect to the target content. However, in the same field of invention, Gupta teaches:
wherein the state information includes attributes of the target content and an operation performed so far with respect to the target content [(e.g. see Gupta paragraphs 0020, 0057) ”auto-complete image suggestions for a user image are identified using final edit settings associated with images deemed similar to the user image. In particular, a current edit state of a user image can be compared to edit states of pre-edited images (e.g., edited by a professional editor) referenced in an image-editing index. An edit state generally refers to a state of edits of an image and may indicate, for example, whether edits associated with various edit controls have been applied to the image. For images having edit states with a same or substantially similar to the edit state of a user image, image representations are compared to determine similarity … the image suggestion manager 206 generally manages image suggestions also referred to as auto-complete image suggestions. In particular, the image suggestion manager 206 can generate an image suggestion(s) for a particular input or user image. An image for which an image suggestion(s) is generated can be obtained in any of a number of ways. For example, a user may select or upload a particular image (e.g., to edit). In embodiments, the image suggestion manager 206 can generate image suggestions for an initial image and/or as an image is being edited. Generating image suggestions for an image being edited can be performed dynamically. That is, as an edit is made to an image, the image suggestion manager 206 can facilitate generation of any image suggestions relevant to the current state of the edited imaged”].
Therefore, considering the teachings of Goyal and Gupta, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to add wherein the state information includes attributes of the target content and an operation performed so far with respect to the target content, as taught by Gupta, to the teachings of Goyal because it allows the user to edit images in a more efficient manner by reducing the time and tedium for modifications (e.g. see Gupta paragraphs 0001, 0002).
As for dependent claim 2, Goyal and Gupta teach the apparatus as described in claim 1 and Goyal further teaches:
wherein the processor is further configured to: after the user performs an operation with respect to the target content, specify the recommended operation with respect to the target content after the operation by inputting state information indicating a state of the target content after the operation into the learning model [(e.g. see Goyal paragraphs 0035, 0036, 0043) ”a receipt of a user input specifying or selecting an area within the digital photograph or video frame including an object of interest for replacement and/or addition … viewing and image editing preferences are obtained as set by the user for editing operations performed with respect to the photograph or video frame(s) … That is, at 325, if it is determined that there is enough data for training and using the recommender model, then the process will proceed to step 330 to implement machine learning technique at the cognitive model trainer module 280 to generate/update a mapping that can be used to map current user selections to candidate object editing and/or replacement suggestions/recommendations to the user for new digital photographs/video taken”].
and notify the user of the recommended operation with respect to the target content after the operation [(e.g. see Goyal paragraph 0045) ”over time, as the user takes pictures and makes edits to them, the cognitive training/recommender system 280 implements machine learning techniques that ingest, reason and learn the user's preferences (e.g., types of edits made to photographs/video frames) which are stored in the knowledgebase and used for mapping to object editing recommendations. In one embodiment, over time, the selection of the image(s) that require ‘work’ can be achieved through the system's historical references. Once a pattern of selection from the user is determined based on historical information accessed from the knowledgebase 260, the system 200 automatically presents available image editing and/or replacement options to the user via display interface 258 As the system learns the preferences of the user the versions presented will be tailored to their selection and quality needs”].
As for dependent claim 4, Goyal and Gupta teach the apparatus as described in claim 1 and Goyal further teaches:
wherein the processor is configured to: before the user performs the recommended operation, predict an after-operation state that is a state of the target content after the recommended operation is executed [(e.g. see Goyal paragraph 0036) ” further viewing and image editing preferences are obtained as set by the user for editing operations performed with respect to the photograph or video frame(s). For example, the user may tend to always apply red-eye reduction to all face objects, and/or may always open a particular editing program that the user uses to overlay a hand-drawn logo or indicia onto the photograph image/video frame(s). In one embodiment, the system records the user selection of a section or object of interest within the image, e.g., a face, and then records editing actions such as applying red-eye reduction. The user may always look in his/her social media account to look for other photographs having the same object for potential replacement. The user may further always post the digital photograph/video in a social media website. All these actions with respect to editing/replacing image sections/objects of a photograph are received into the system at 320”].
specify a subsequent recommended operation that is an operation to be performed with respect to the target content of the after-operation state, by inputting after-operation state information indicating the after-operation state into the learning model [(e.g. see Goyal paragraphs 0043, 0044) ”the process will proceed to step 330 to implement machine learning technique at the cognitive model trainer module 280 to generate/update a mapping that can be used to map current user selections to candidate object editing and/or replacement suggestions/recommendations … the knowledgebase stores all updates with respect to the particular editing/replacing actions taken of any particular image section(s)/object(s) of interest. The system then returns to 305 to repeat method for continuously training the model, over time, whenever further photographs/video and corresponding editing/replacement actions are subsequently taken”].
and notifying the user of the subsequent recommended operation [(e.g. see Goyal paragraph 0045) ”Once a pattern of selection from the user is determined based on historical information accessed from the knowledgebase 260, the system 200 automatically presents available image editing and/or replacement options to the user via display interface 258 As the system learns the preferences of the user the versions presented will be tailored to their selection and quality needs. It is understood that, over time, the image set presented may change as the system learns which types of images and their make up the user is most likely to select”].
As for dependent claim 5, Goyal and Gupta teach the apparatus as described in claim 4 and Goyal further teaches:
wherein the processor is configured to: acquire the after-operation state information by rewriting the state information before performing the recommended operation based on a detail of the recommended operation [(e.g. see Goyal paragraphs 0043, 0044) ”then the process will proceed to step 330 to implement machine learning technique at the cognitive model trainer module 280 to generate/update a mapping that can be used to map current user selections to candidate object editing and/or replacement suggestions/recommendations … the knowledgebase stores all updates with respect to the particular editing/replacing actions taken of any particular image section(s)/object(s) of interest. The system then returns to 305 to repeat method for continuously training the model, over time, whenever further photographs/video and corresponding editing/replacement actions are subsequently taken”].
As for dependent claim 6, Goyal and Gupta teach the apparatus as described in claim 4 and Goyal further teaches:
wherein the processor is configured to: in accordance with selection of the subsequent recommended operation by the user, control display of a screen to sequentially execute processing related to the recommended operation that changes the state of the target content to the after-operation state as a target of the subsequent recommended operation and processing related to the subsequent recommended operation with respect to the target content [(e.g. see Goyal paragraphs 0036, 0053, 0057, 0058) ”further viewing and image editing preferences are obtained as set by the user for editing operations performed with respect to the photograph or video frame(s). For example, the user may tend to always apply red-eye reduction to all face objects, and/or may always open a particular editing program that the user uses to overlay a hand-drawn logo or indicia onto the photograph image/video frame(s) … an operator/editor will have the option to select one or more image objects from the photograph according to setting to be replaced/or overlayed to modify the original photograph. In the embodiments of system 200 of FIG. 2, the recommender model training system 280, will access the knowledgebase, and based on the current user settings and preferences, and use the recommender model to suggest to the user particular edit to make and candidate images/photographs … display of the image object navigation block that will provide an option for an operator/editor to display each candidate digital photographs as an overlay around the selected object of interest … While moving from one version of image object to another version of image object for the same image object, the user may desire to and has the option to replace the existing image object in focus”].
As for dependent claim 7, Goyal and Gupta teach the apparatus as described in claim 5 and Goyal further teaches:
wherein the processor is configured to: in accordance with selection of the subsequent recommended operation by the user, control display of a screen to sequentially execute processing related to the recommended operation that changes the state of the target content to the after-operation state as a target of the subsequent recommended operation and processing related to the subsequent recommended operation with respect to the target content [(e.g. see Goyal paragraphs 0036, 0053, 0057, 0058) ”further viewing and image editing preferences are obtained as set by the user for editing operations performed with respect to the photograph or video frame(s). For example, the user may tend to always apply red-eye reduction to all face objects, and/or may always open a particular editing program that the user uses to overlay a hand-drawn logo or indicia onto the photograph image/video frame(s) … an operator/editor will have the option to select one or more image objects from the photograph according to setting to be replaced/or overlayed to modify the original photograph. In the embodiments of system 200 of FIG. 2, the recommender model training system 280, will access the knowledgebase, and based on the current user settings and preferences, and use the recommender model to suggest to the user particular edit to make and candidate images/photographs … display of the image object navigation block that will provide an option for an operator/editor to display each candidate digital photographs as an overlay around the selected object of interest … While moving from one version of image object to another version of image object for the same image object, the user may desire to and has the option to replace the existing image object in focus”].
As for dependent claim 8, Goyal and Gupta teach the apparatus as described in claim 1 and Goyal further teaches:
wherein the processor is configured to: display the recommended operation on a display unit in an aspect corresponding to the recommended operation based on operation type information in which an operation and a type of the operation are associated with each other [(e.g. see Goyal paragraphs 0021, 0045) ”as the user takes pictures and makes edits to them, the cognitive training/recommender system 280 implements machine learning techniques that ingest, reason and learn the user's preferences (e.g., types of edits made to photographs/video frames) which are stored in the knowledgebase and used for mapping to object editing recommendations … A generated option(s) or recommendation(s) using the cognitive ability of the system may be presented to the user with suggestions to take any action with respect to the digital image”].
As for independent claim 10, Goyal and Gupta teach a non-transitory computer readable medium. Claim 10 discloses substantially the same limitations as claim 1. Therefore, it is rejected with the same rational as claim 1.
As for independent claim 11, Goyal and Gupta teach a method. Claim 11 discloses substantially the same limitations as claim 1. Therefore, it is rejected with the same rational as claim 1.
As for dependent claim 12, Goyal and Gupta teach the apparatus as described in claim 4, but Goyal does not specifically teach the following limitation. However, Gupta teaches:
wherein the processor is configured to notify the user of the recommended operation and the subsequent recommended operations simultaneously [(e.g. see Gupta paragraph 0076 and Fig. 5B) ”the user can select an auto-complete image suggestion icon 504. Upon selecting the auto-complete image suggestion icon 504, a set of auto-complete image suggestions may be presented. For example, as shown in FIG. 5B, a set of auto-complete image suggestions 506 are presented. As described herein, such auto-complete image suggestions can have various edits that are determined via similar pre-edited images. If selected, the applicable edits can be applied to image 502 to automatically complete a set of edits in connection with the image”].
The motivation to combine is the same as that used for claim 1.
Claims 3 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Goyal et al. (US 2018/0300046 A1) in view of Gupta et al. (US 2021/0217152 A1), as applied to claim 1 above, and further in view of Adaska et al. (US 2019/0392345 A1).
As for dependent claim 3, Goyal and Gupta the apparatus as described in claim 1 and Goyal further teaches:
wherein the learning model outputs a plurality of recommended operations [(e.g. see Goyal paragraph 0021, 0048) ”A generated option(s) or recommendation(s) using the cognitive ability of the system may be presented to the user with suggestions to take any action with respect to the digital image … user preference to set an amount of options to for the system to suggest or recommend”].
Goyal and Gupta do not specifically teach and prediction accuracy of each recommended operation and the processor is configured to: notify the user of the plurality of recommended operations in an aspect corresponding to the prediction accuracy of each recommended operation. However, in the same field of invention, Adaska teaches:
and prediction accuracy of each recommended operation and the processor is configured to: notify the user of the plurality of recommended operations in an aspect corresponding to the prediction accuracy of each recommended operation [(e.g. see Adaska paragraph 0043) ”The resulting values (e.g., output in respective output nodes of an output layer) may form the rows of an output probability vector 415. Each row of this output probability vector 415 may correspond to a respective suggested action to perform, where a higher probability value indicates an action or application more likely to be performed or loaded. The machine-learned model 400-a may output one or more actions 420 (e.g., tasks to perform, applications to load, etc.) based on the output probability vector 415 … the machine-learned model 400-a may output a configured number of actions 420 with the highest probability values (e.g., ranked from highest to lowest probability)”].
Therefore, considering the teachings of Goyal, Gupta and Adaska, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to add and prediction accuracy of each recommended operation and the processor is configured to: notify the user of the plurality of recommended operations in an aspect corresponding to the prediction accuracy of each recommended operation, as taught by Adaska, to the teachings of Goyal and Gupta because it allows a user to streamline the workflow within an application and reduce the time taken to accomplish tasks (e.g. see Adaska paragraph 0015).
As for dependent claim 9, Goyal and Gupta teach the apparatus as described in claim 1, but do not specifically teach the following limitation. However, Adaska teaches:
wherein the state information includes a plurality of state items, and the processor is configured to: calculate a degree of contribution indicating a degree of contribution to specifying of the recommended operation for each state item based on a difference between output data of the trained learning model in a case where first state information of the target content is input and output data of the trained learning model in a case where second sate information obtained by changing a value of one of a plurality of state items included in the first state information is input and notify the user of the state item that has contributed to specifying of the recommended operation based on the degree of contribution of each state item [(e.g. see Adaska paragraphs 0022, 0046) ”passing training data 155 through a training model to calculate training output 165 based on a current set of weights in the training model. For example, the training data 155 may be an example of an input vector (or a sequence of input vectors), and the values of the input vector may be assigned to a set of input nodes in an input node layer of the training model. These input node values may pass through a number of hidden layers of the training model, where the values are modified based on weights between the nodes of the training model. Following the hidden layer(s), the training model may calculate output node values in an output layer, where each output node corresponds to a particular suggested action to perform. The set of output nodes may correspond to an output vector (e.g., the training output 165), where the values of the output nodes are probabilities each corresponding to a specific probability that a respective action should be suggested by the action suggestion system. The model training 160 may compare the training output 165 to actual action selection data (e.g., historical task selection data 140 or on-the-fly action selection data from the user device 105) and may update the weights of the training model based on the comparison … The machine-learned model 400-b may receive a data context from one or more applications and may generate a sequence of input vectors 425 based on the data context. For example, the sequence of input vectors 425 may be based on a time sequence of data contexts and/or actions performed by a user. These input vectors may be similar to the input vector 405 described with reference to FIG. 4A. The trained model 410-b may process the sequence of input vectors 425 and output a sequence of output probability vectors 430. The sequence of output probability vectors 430 may correspond to an ordered or unordered sequence of actions 435 (e.g., an ensemble of actions) to perform. For example, if the sequence of output probability vectors 430 includes three vectors, the first vector may indicate a first suggested action for a user to perform, the second vector may indicate a second suggested action for the user to perform, and the third vector may indicate a third suggested action for the user to perform (e.g., to be performed in the specified order). This suggested sequence of actions 435 may be sent for presentation (e.g., display) in a user interface as suggested actions to perform, and in some cases may be ranked by probability or by a suggested order for performing the actions. Based on the sequence of input vectors 425 and sequence of output vectors, the machine-learned model 400-b presents a sequence of next actions to perform”].
The motivation to combine is the same as that used for claim 3.
Response to Arguments
Applicant's arguments, filed 24 June 2026, have been fully considered but they are not persuasive.
Applicant argues that [“Goyal is silent on ‘the state information includes attributes of the target content and an operation performed so far with respect to the target content’ as recited in the amended claim 1.” (Pages 8-9).].
The argument described above, in paragraph number 6, with respect to the newly added limitations to the independent claims has been considered, but is moot in view of the new grounds of rejection.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTOPHER J FIBBI whose telephone number is (571)-270-3358. The examiner can normally be reached Monday - Thursday (8am-6pm).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Bashore can be reached at (571)-272-4088. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHRISTOPHER J FIBBI/Primary Examiner, Art Unit 2174
Prosecution Insights