Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4, 7-16, 19 and 20 are rejected under 35 U.S.C. 102(a) (1) as being unpatentable by Kovac et al (“Kovac” US 2019/0197402 A1), published on June 27, 2019.
As to claim 1, Kovac teaches “identifying user input of a user, wherein the user input is associated with a computer-controlled agent of a virtual environment” in par. 0036 (A game developer (corresponding to a user) can provide different input properties (corresponding to user input) to a deep neural network, DNN (corresponding to model) to infer AI actions (model output) in a game environment (corresponding to a virtual environment) including AI-controlled agents (corresponding to a computer-controlled agent)).
Kovac teaches “generating, based on the user input and agent information associated with the computer-controlled agent, model output associated with a multimodal machine learning model” in par. 0036 (A game developer (corresponding to a user) can provide different input properties (corresponding to user input) to a deep neural network, DNN (corresponding to model) to infer AI actions (model output) in a game environment (corresponding to a virtual environment)).
Kovac teaches “wherein the model output is a multimodal output that includes natural language output and programmatic output” in par. 0223 (“… the encoded AI behavior specification includes a specification of an AI problem specified using an AI planning language…”).
Kovac teaches “evaluating the model output according to a set of constraints to determine whether to present the model output to the user;
based on determining not to present the model output to the user:
generating replacement model output that replaces at least a part of the model output” in paragraphs [0101-0102] (re-iteration to generate updated solver with respect to current problem (as to evaluate the model output according to a set of constraints to determine whether to present the model output to the user; every change in problem data structure, the existing problem output is replaced by the new one to solve the new problem).
Kovac teaches “and executing replacement model output to control the computer-controlled agent within the virtual environment” in par. 0036 (“A DNN can be trained and used to infer AI actions based on the specified AI behavior for an NPC such as a sentry guard”; DNN corresponding to a model that produces output (AI actions based on the specified AI behavior) controlling AI agents).
As to claim 2, Kovac teaches “receiving, from the user, an indication to change at least a part of the agent information based on a behavior of the computer-controlled agent associated with the model output” in par. 0036 (“The encoded and/or compiled AI behavior is trained, resulting in a deep neural network (DNN) that implements and realizes the AI behavior. For example, based on different input properties, the DNN can be used to infer an appropriate action result”. Different input properties correspond to indication to change at least a part of the agent information based on a behavior of the computer-controlled agent associated with the model output).
Kovac teaches “updating the agent information based on the received indication to generate updated agent information; generating replacement model output based on the updated agent information; and executing at least a part of the replacement model output to control the computer- controlled agent within the virtual environment” in par. 0036 (“The encoded and/or compiled AI behavior is trained, resulting in a deep neural network (DNN) that implements and realizes the AI behavior. For example, based on different input properties, the DNN can be used to infer an appropriate action result”).
As to claim 3, Kovac teaches “wherein the agent information comprises a set of agent attributes that define one or more of: a trait of the computer-controlled agent; a persona of the computer-controlled agent; a goal of the computer-controlled agent; or a mood of the computer-controlled agent” in par. 0036 (AI behavior specifications are defined by the user and then compiled to define AI-agents behavior that correspond to a trait of the computer-controlled agent; a persona of the computer-controlled agent; a goal of the computer-controlled agent; or a mood of the computer-controlled agent).
As to claim 4, Kovac teaches “wherein the agent information comprises at least one of: background information associated with the virtual environment; historical information associated with the user; a set of attributes associated with the user; or virtual environment state information for the virtual environment” in par. 0036 (“A game environment may also be created for the AI logic and AI tool to access environmental properties of the game. A game developer can then define the AI behaviors that can be used by the AI agents of the AI scene”. “environmental properties of the game” corresponds to background information associated with the virtual environment).
As to claim 7, Kovac teaches “wherein generating the model output comprises: providing, to a machine learning service, an indication of the user input in association with the agent information; and receiving, from the machine learning service, the model output” in par. 0038 (AI behavior specification is provided to DNN to produce model output).
As to claim 8, Kovac teaches “generating, based on agent information associated with a computer-controlled agent of a virtual environment, model output associated with a multimodal machine learning model, wherein the model output is a multimodal output that includes natural language output and programmatic output” in par. 0103 (solution plan stores agent or NPC behaviors. Solver utilizes a deep neural network machine learning model) and in par. 0223 (“…In various embodiments, a created AI behavior is encoded into an encoded AI behavior specification. In some embodiments, the encoded AI behavior specification includes a specification of an AI problem specified using an AI planning language…”).
Kovac teaches “controlling the computer-controlled agent within the virtual environment based on the generated model output” in par. 0103 (solution plan is used to reiterate of execution in order to generate design agents’ behaviors).
Kovac teaches “receiving, from a user, an indication to change at least a part of the agent information based a behavior of the computer-controlled agent associated with the model output; updating the agent information based the received indication to generate updated agent information; generating replacement model output that replaces at least a part of the model output based on the updated agent information; and controlling the computer-controlled agent within the virtual environment based on the replacement model output” in par. 0103 (iteration process to re-plan agent behaviors, update agent information, create new agent behavior in the virtual environment).
As to claim 9, Kovac teaches “storing the updated agent information in a game agent data store for use in controlling [[a]] the computer-controlled agent in an interaction with a player of the virtual environment” in par. 0103 (solution plan repository stores the updated agent information in a game agent data store for use).
As to claim 10, Kovac teaches “wherein the indication to change at least a part of the agent information is received as a change to a prompt of the agent information” in par. 0036 (game developer can change agent behavior using AI tool).
As to claim 11, Kovac teaches “wherein the indication to change at least a part of the agent information comprises a change to a set of constraints for the computer-controlled agent” in par. 0141 (agent belief’s set can be updated to control agent behavior).
As to claim 12, Kovac teaches “wherein the agent information comprises a set of agent attributes that define one or more of. a trait of the computer-controlled agent; a persona of the computer-controlled agent; a goal of the computer-controlled agent; or a mood of the computer-controlled agent” in par. 0036 (AI behavior specifications are defined by the user and then compiled to define AI-agents behavior that correspond to a trait of the computer-controlled agent; a persona of the computer-controlled agent; a goal of the computer-controlled agent; or a mood of the computer-controlled agent).
As to claim 13, Kovac teaches “wherein the agent information comprises at least one of. background information associated with the virtual environment; historical information associated with [[the]] a player; a set of attributes associated with the player; or virtual environment state information for the virtual environment” in par. 0036 (“A game environment may also be created for the AI logic and AI tool to access environmental properties of the game. A game developer can then define the AI behaviors that can be used by the AI agents of the AI scene”. “environmental properties of the game” corresponds to background information associated with the virtual environment).
As to claim 14, Kovac teaches “identifying user input of a player, wherein the user input is associated with a computer- controlled agent of a game application” in par. 0039 (“…Goals may include game objectives such as traveling from one location to another, speaking with the game player to inform her or him of relevant information…”).
Kovac teaches “"generating, based on the user input and agent information associated with the computer- controlled agent, model output associated with a multimodal machine learning model; and controlling the computer-controlled agent within the game application based on the generated model output, thereby causing the computer-controlled agent to interact with the player” in par. 0039 (“…The solution is solved by applying a trained machine learning model such as one utilizing a deep convolutional neural network (DCNN)…”).
As to claim 14, Kovac teaches “identifying user input of a player, wherein the user input is associated with a computer- controlled agent of a game application” in par. 0039 (“…Goals may include game objectives such as traveling from one location to another, speaking with the game player to inform her or him of relevant information…”).
Kovac teaches “generating, based on the user input and agent information associated with the computer- controlled agent, model output associated with a multimodal machine learning model, wherein the model output is a multimodal output that includes natural language output and programmatic output” in par. 0223 (“… the encoded AI behavior specification includes a specification of an AI problem specified using an AI planning language…”).
Kovac teaches “evaluating the model output according to a set of constraints to determine whether to present the model output to the user;
based on determining not to present the model output to the user:
generating replacement model output that replaces at least a part of the model output” in paragraphs [0101-0102] (re-iteration to generate updated solver with respect to current problem (as to evaluate the model output according to a set of constraints to determine whether to present the model output to the user; every change in problem data structure, the existing problem output is replaced by the new one to solve the new problem).
Kovac teaches “and controlling the computer-controlled agent within the game application based on the replacement model output, thereby causing the computer-controlled agent to interact with the player” in par. 0039 (“…The solution is solved by applying a trained machine learning model such as one utilizing a deep convolutional neural network (DCNN)…”).
As to claim 15, Kovac teaches “wherein the agent information comprises a set of agent attributes that define one or more of: a trait of the computer-controlled agent; a persona of the computer-controlled agent; a goal of the computer-controlled agent; or a mood of the computer-controlled agent” in par. 0036 (AI behavior specifications are defined by the user and then compiled to define AI-agents behavior that correspond to a trait of the computer-controlled agent; a persona of the computer-controlled agent; a goal of the computer-controlled agent; or a mood of the computer-controlled agent).
As to claim 16, Kovac teaches “wherein the agent information comprises at least one of: background information associated with [[the]] a virtual environment; historical information associated with the player; a set of attributes associated with the player; or virtual environment state information for the virtual environment” in par. 0036 (“A game environment may also be created for the AI logic and AI tool to access environmental properties of the game. A game developer can then define the AI behaviors that can be used by the AI agents of the AI scene”. “environmental properties of the game” corresponds to background information associated with the virtual environment).
As to claim 19, Kovac teaches “wherein generating the model output comprises: providing, to a machine learning service, an indication of the user input in association with the agent information; and receiving, from the machine learning service, the model output” in par. 0038 (AI behavior specification is provided to DNN to produce model output).
As to claim 20, Kovac teaches “executing the programmatic output of the model output to control the computer-controlled agent; displaying the natural language output of the model output in association with the computer- controlled agent; or generating audio output for the computer-controlled agent based on the natural language output of the model output” in par. 0158 (“…the Futurable life simulation game utilizes the process of FIG. 19 to allow players to interact with NPC agents using natural languages. When a player talks or inputs a sentence to an NPC, the game system recognizes the provided input and utilizes a conversational AI server to process the voice input and provide an answer…”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 6 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Kovac et al (“Kovac” US 2019/0197402 A1), published on June 27, 2019 in view of Back et al (“Back” US 2021/0174801 A1), published on June 10, 2021.
As to claim 6, it appears Kovac does not explicitly teach “generating an indication of feedback associated with the model output, wherein the indication of feedback is used to finetune the multimodal machine learning model using reinforcement learning”.
However, Back teaches “generating an indication of feedback associated with the model output, wherein the indication of feedback is used to finetune the multimodal machine learning model using reinforcement learning” in par. 0027 (“the processor is further configured to execute the instructions to receive user feedback for a part of the output content, and modify the part of the output content by using the NN model”) and in par. 0188 (disclosed on reinforcement learning).
Kovac and Back are analogous art because they are in the same field of endeavor, artificial intelligence. It would have been obvious to one of ordinary skill in the art before the effective filling date of the claim invention to finetune the machine learning model (disclosed by Kovac) to include “generating an indication of feedback associated with the model output, wherein the indication of feedback is used to finetune the multimodal machine learning model using reinforcement learning” as suggested by Back in order to improve a learning machine model (see Back par. 0188).
As to claim 18, it appears Kovac does not explicitly teach “generating an indication of feedback associated with the model output, wherein the indication of feedback is used to finetune the multimodal machine learning model using reinforcement learning”.
However, Back teaches “generating an indication of feedback associated with the model output, wherein the indication of feedback is used to finetune the multimodal machine learning model using reinforcement learning” in par. 0027 (“the processor is further configured to execute the instructions to receive user feedback for a part of the output content, and modify the part of the output content by using the NN model”) and in par. 0188 (disclosed on reinforcement learning).
Kovac and Back are analogous art because they are in the same field of endeavor, artificial intelligence. It would have been obvious to one of ordinary skill in the art before the effective filling date of the claim invention to finetune the machine learning model (disclosed by Kovac) to include “generating an indication of feedback associated with the model output, wherein the indication of feedback is used to finetune the multimodal machine learning model using reinforcement learning” as suggested by Back in order to improve a learning machine model (see Back par. 0188).
Response to Arguments
Regarding Applicant’s argument on page 10 of the remarks, Applicant argues “Kovacs, in the cited paragraph or elsewhere in the disclosure, does not disclose or suggest any feature corresponding to the set of constraints, nor does Kovacs disclose or suggest the use of those constraints to prompt regeneration of the model output”. Applicant’s argument is respectfully considered, but is not persuasive. According to paragraphs [0101-0102], the artificial intelligence loop is run until a goal is achieved. Kovacs discloses “…solver 1021 generates a plan and corresponding actions to achieve one or more goals defined by the problem…” as to suggest that a model output (the generated plan) is determined based on a set of constraint (how to achieve one or more goals). When the generated plan is not qualified for a goal yet, a new plan is then generated in order to achieve a goal. It is noted that criteria for goal achievement correspond to a set of constraint in the claim language.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicants’ disclosure:
. Copper (US 10,713,597 B2)
. Machacek et al (US 11,237,805 B2)
THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Loc Tran whose telephone number is (571)272-8485. The examiner can normally be reached on Mon - Fri (8:00 am - 5:00 pm).
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amy Ng can be reached on 571-272-3676. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LOC TRAN/
Primary Examiner, Art Unit 2164