Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier.
Such claim limitations are:
a reinforcement learning agent configured to (claims 1, 10, and 19)…
a filtering agent configured to (claims 7 and 16)…
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 7, 10, 16, and 19 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement.
Claims 1, 7, 10, 16, and 19 contain subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. As discussed above, the disclosure does not provide adequate structure to perform the claimed functions of:
a reinforcement learning agent configured to (claims 1, 10, and 19).
a filtering agent configured to (claims 7 and 16).
The specification does not demonstrate that applicant has made an invention that achieves the claimed function because the invention is not described with sufficient detail such that one of ordinary skill in the art can reasonably conclude that the inventor had possession of the claimed invention.
Claims 2-6, 8-9, 11-15, 17-18, and 20 are further rejected for dependence, either directly or indirectly, on claims 1, 10, and 19.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 7, 10, 16, and 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 1, 7, 10, 16, and 19, various claim limitations reciting a reinforcement learning agent configured to (claims 1, 10, and 19)…; a filtering agent configured to (claims 7 and 16)… invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed functions and to clearly link the structure, material, or acts to the functions. The specification is devoid of adequate structure to perform the claimed functions. There is no clear disclosure of the particular structure, either explicitly or inherently, to perform the functions of the claims. As would be recognized by those of ordinary skill in the art, the functions can be performed in any number of ways including in hardware, in software, or a combination of the two. The specification does not provide sufficient details such that one of ordinary skill in the art would understand which structure or structures perform(s) the claimed functions.
Further, the “filtering agent” is only mentioned once in the specification in paragraph [0008], which merely repeats the claimed functions of the filtering agent, but it does not provide structure to perform the claimed functions. Further, the “reinforcement learning agent” is mentioned in specification paragraphs [0032] and [0050], which merely repeat the claimed functions of the reinforcement learning agent, but it does not provide structure to perform the claimed functions.
Therefore, the claims are indefinite and are rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph. For purposes of examination, any computer software that performs the claimed functions will be deemed to read on the claims.
Claims 2-6, 8-9, 11-15, 17-18, and 20 are further rejected for dependence, either directly or indirectly, on claims 1, 10, and 19.
Applicant may:
(a) Amend the claims so that the claim limitations will no longer be interpreted as limitations under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed functions, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the functions recited in the claims, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the functions so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed functions, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed functions, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed functions. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claims 2, 7-8, 11, 16-17, and 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “optimal sequence of in-game actions” in claims 2, 11, and 19 is a relative term which renders the claim indefinite. The term “optimal sequence” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term “sequence” has been rendered indefinite by the use of the term “optimal”.
The term “optimal metas, policies, and strategies for the in-game actions” in claims 7 and 16 is a relative term which renders the claim indefinite. The term “optimal metas, policies, and strategies” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term “metas, policies, and strategies” has been rendered indefinite by the use of the term “optimal”.
The term “optimal results” in claims 8 and 17 is a relative term which renders the claim indefinite. The term “optimal results” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The term “results” has been rendered indefinite by the use of the term “optimal”.
Examiner’s Note: For the purposes of Examination, the term “optimal” will be interpreted as any optimization process or desired outcome.
Claim 20 is further rejected for dependence on claim 19.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1,
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 1 is directed to a system for discovering at least one new technique for playing a game, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“search at least one Internet platform for data related to at least one game scenario”
“determine at least one reward framework based on results from the search of the at least one Internet platform, the at least one reward framework being determined by at least one metric for a characteristic of the at least one game scenario”
“playing the at least one game scenario within a game, the playing comprising taking a plurality of sequential in-game actions available in the at least one game scenario”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)) and insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“A system for discovering at least one new technique for playing a game, the system comprising: a determining agent comprising at least one processor and at least one memory storing instructions which, when executed by the at least one processor, cause the processor to:..”
“a reinforcement learning agent configured to perform a training and exploration loop comprising a plurality of iterations”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
The limitations:
“transmitting results of each of the plurality of sequential in-game actions to the determining agent”
“receiving a reward for successful progression through the at least one game scenario according to at least one reward framework determined by the determining agent, the reward comprising a quantitative score”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply” and “insignificant extra-solution activity”. Specifically the transmitting, and receiving limitations recite the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Mere instructions to apply and insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 2,
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 2 is directed to a system for discovering at least one new technique for playing a game, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“determine an optimal sequence of in-game actions for the at least one game scenario based on a plurality of scores comprising each of the quantitative scores from each of the plurality of iterations”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“the determining agent being further configured to…”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 3,
Claim 3 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 3 is directed to a system for discovering at least one new technique for playing a game, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“the at least one Internet platform comprising any one of: a video streaming platform, a game developer website, an Internet forum, a game wiki, and a news source”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 4,
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 4 is directed to a system for discovering at least one new technique for playing a game, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“the characteristic of the at least one game scenario being any one of: level of difficulty or skill of the at least one game scenario; novelty of the in-game action or the sequence of actions; surprise due to a result of the action or the sequence of actions; popularity of the in-game action or sequence of actions; humor due to the result of the action or the sequence of actions; and enjoyment of the result of the action or sequence of actions”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 5,
Claim 5 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 5 is directed to a system for discovering at least one new technique for playing a game, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“the at least one metric for the characteristic being any one of: frequency of a key word or phrase associated with the at least one in-game scenario; image data from image stills or video frames associated with the at least one in-game scenario; a trendline indicating a change in the frequency with which the key word or phrase are used; and game data from networked games indicating the frequency with which the action or the sequence of actions are used in the in-game scenario”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 6,
Claim 6 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 6 is directed to a system for discovering at least one new technique for playing a game, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“a memory storage for storing a sequence of game states, the actions and the sequence of actions, and information associated with the metrics for the at least one characteristic of the in-game scenario”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “insignificant extra-solution activity”. Specifically, the storing limitation recites the well-understood, routine, and conventional activity of storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015). Insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 7,
Claim 7 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 7 is directed to a system for discovering at least one new technique for playing a game, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“determine optimal metas, policies, and strategies for the in-game actions as defined by the at least one metric for the characteristic”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“a filtering agent configured to…”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 8,
Claim 8 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 8 is directed to a system for discovering at least one new technique for playing a game, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“the determining agent further comprising a deep neural network having an input layer, a plurality of hidden layers, and an output layer, the plurality of hidden layers being configured to process input received at the input layer and transmit a first output to the output layer, the plurality of hidden layers being trained and tuned using weights and biases for optimal results based on the at least one metric for the characteristic”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 9,
Claim 9 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 9 is directed to a system for discovering at least one new technique for playing a game, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“evaluate subsequent trends related to the results”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“share results from the plurality of iterations with a network of users”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “insignificant extra-solution activity”. Specifically, the sharing limitation recites the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 10,
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 10 is directed to a method for configuring a reinforcement learning system for discovering at least one new technique for playing a game, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“search at least one Internet platform for data related to at least one game scenario”
“determine at least one reward framework based on results from the search of the at least one Internet platform, the at least one reward framework being determined by at least one metric for a characteristic of the at least one game scenario”
“playing the at least one game scenario within a game, the playing comprising taking a plurality of sequential in-game actions available in the at least one game scenario”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)) and insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“configuring a determining agent comprising at least one processor and at least one memory with instructions which, when executed by the at least one processor, cause the processor to:..”
“configuring a reinforcement learning agent to perform a training and exploration loop comprising a plurality of iterations”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
The limitations:
“transmitting results of each of the plurality of sequential in-game actions to the determining agent”
“receiving a reward for successful progression through the at least one game scenario according to at least one reward framework determined by the determining agent, the reward comprising a quantitative score”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply” and “insignificant extra-solution activity”. Specifically, the transmitting and receiving limitations recite the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Mere instructions to apply and insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 11,
Claim 11 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 11 is directed to a method for configuring a reinforcement learning system for discovering at least one new technique for playing a game, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“determine an optimal sequence of in-game actions for the at least one game scenario based on a plurality of scores comprising each of the quantitative scores from each of the plurality of iterations”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“the configuring of the determining agent further comprising instructions to”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 12,
Claim 12 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 12 is directed to a method for configuring a reinforcement learning system for discovering at least one new technique for playing a game, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 10.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“the at least one Internet platform comprising any one of: a video streaming platform, a game developer website, an Internet forum, a game wiki, and a news source”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 13,
Claim 13 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 13 is directed to a method for configuring a reinforcement learning system for discovering at least one new technique for playing a game, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 10.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“the characteristic of the at least one game scenario being any one of: level of difficulty or skill of the at least one game scenario; novelty of the in-game action or the sequence of actions; surprise due to a result of the action or the sequence of actions; popularity of the in-game action or sequence of actions; humor due to the result of the action or the sequence of actions; and enjoyment of the result of the action or sequence of actions”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 14,
Claim 14 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 14 is directed to a method for configuring a reinforcement learning system for discovering at least one new technique for playing a game, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 10.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“the at least one metric for the characteristic being any one of: frequency of a key word or phrase associated with the at least one in-game scenario; image data from image stills or video frames associated with the at least one in-game scenario; a trendline indicating a change in the frequency with which the key word or phrase are used; and game data from networked games indicating the frequency with which the action or the sequence of actions are used in the in-game scenario”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 15,
Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 15 is directed to a method for configuring a reinforcement learning system for discovering at least one new technique for playing a game, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 10.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“storing a sequence of game states, the actions and the sequence of actions, and information associated with the metrics for the at least one characteristic of the in-game scenario in a memory storage”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “insignificant extra-solution activity”. Specifically, the storing limitation recites the well-understood, routine, and conventional activity of storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015). Insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 16,
Claim 16 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 16 is directed to a method for configuring a reinforcement learning system for discovering at least one new technique for playing a game, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“determine optimal metas, policies, and strategies for the in-game actions as defined by the at least one metric for the characteristic”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“implementing a filtering agent configured to…”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 17,
Claim 17 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 17 is directed to a method for configuring a reinforcement learning system for discovering at least one new technique for playing a game, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 10.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“the determining agent further comprising a deep neural network having an input layer, a plurality of hidden layers, and an output layer, the plurality of hidden layers being configured to process input received at the input layer and transmit a first output to the output layer, the plurality of hidden layers being trained and tuned using weights and biases for optimal results based on the at least one metric for the characteristic”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 18,
Claim 18 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 18 is directed to a method for configuring a reinforcement learning system for discovering at least one new technique for playing a game, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“evaluate subsequent trends related to the results”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“share results from the plurality of iterations with a network of users”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “insignificant extra-solution activity”. Specifically, the sharing limitation recites the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 19,
Claim 19 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 19 is directed to a method for discovering at least one new technique for playing a game, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“searching at least one Internet platform for data related to at least one game scenario”
“determining at least one reward framework based on results from the search of the at least one Internet platform, the at least one reward framework being determined by at least one metric for a characteristic of the at least one game scenario”
“determining an optimal sequence of in-game actions for the at least one game scenario based on a plurality of scores comprising each of the quantitative scores from each of the plurality of iterations”
“playing the at least one game scenario within a game, the playing comprising taking a plurality of sequential in-game actions available in the at least one game scenario”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)) and insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“implementing a determining agent comprising at least one processor and at least one memory storing instructions which, when executed by the at least one processor, cause the processor to execute a method”
“implementing a reinforcement learning agent configured to perform a training and exploration loop comprising a plurality of iterations”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
The limitations:
“transmitting results of each of the plurality of sequential in-game actions to the determining agent”
“receiving a reward for successful progression through the game scenario according to at least one reward framework determined by the determining agent, the reward comprising a quantitative score”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply” and “insignificant extra-solution activity”. Specifically, the transmitting and receiving limitations recite the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Mere instructions to apply and insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 20,
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 20 is directed to a method for discovering at least one new technique for playing a game, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“evaluating subsequent trends related to the results”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“sharing results from the plurality of iterations with a network of users”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “insignificant extra-solution activity”. Specifically, the sharing limitation recites the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7, 9-16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Firoiu et al. (Beating the World’s Best at Super Smash Bros. Melee with Deep Reinforcement Learning) (“Firoiu”) in view of Fan et al. (MINEDOJO: Building Open-Ended Embodied Agents with Internet-Scale Knowledge) (“Fan”).
Regarding claim 1, Firoiu teaches a system for discovering at least one new technique for playing a game (Firoiu Section 2 The SSBM Environment “The metagame is constantly evolving as new mechanics are discovered and refined and top players push each other to ever greater levels of skill.”; Section 4.1 In-game AI “For each algorithm, we found little variance between experiments with different initializations. However, the two algorithms found qualitatively different policies from each other. Actor-Critics pursued a standard strategy of attacking and counter-attacking, similar to the way humans play. Q-learners on the other hand would consistently find the unintuitive strategy of tricking the in-game AI into killing itself. This multi-step tactic is fairly impressive” Firoiu provides metagame discovery including finding an unintuitive strategy of tricking in-game AI, corresponding to discovering at least one new technique for playing a game.), the system comprising: a determining agent comprising at least one processor and at least one memory storing instructions which, when executed by the at least one processor, cause the processor to (Firoiu Section 3.3 Training “With the help of a GPU, the trainer continually performs (minibatched) stochastic gradient descent on its set of experiences while periodically saving snap shots of the neural network weights for the agents to load.” Firoiu provides a GPU (i.e., processing hardware), corresponding to a determining agent comprising processor and memory.): search at least one Internet platform for data related to at least one game scenario (Firoiu Section 2 The SSBM Environment “We focus on Super Smash Bros. Melee (SSBM), a fast-paced multi-player fighting game released in 2001 for the Nintendo Gamecube.”: Section 4.2 Self-play “Table1: Some results against ranked SSBM players. Rankings from http://wiki.teamliquid.net/smash/SSBM_Rank.S2J is considered by some to be the best Captain Falcon player in the world.” Firoiu provides searching an internet wiki for rankings related to Super Smash Bros. Melee (SSBM), corresponding to searching the internet for data related to a game scenario, wherein the wiki corresponds to the internet platform.); …and a reinforcement learning agent configured to perform a training and exploration loop comprising a plurality of iterations (Firoiu Section 3.3 Training “The many parallel agents [reinforcement learning agent] periodically send their experiences to a trainer, which maintains a circular queue of the most recent experiences… With the help of a GPU, the trainer continually performs (minibatched) stochastic gradient descent on its set of experiences while periodically saving snap shots of the neural network weights for the agents to load. In RL, however, we are optimizing more than just a loss function- we are optimizing a policy through policy iteration. [iterative training process]”; Section 5.2 Exploration vs Exploitation “Our main method for quantitatively measuring the tendency of an agent to explore different actions is through the average entropy of its policy [exploration loop].”), each of the plurality of iterations comprising: playing the at least one game scenario within a game (Firoiu Section 2 The SSBM Environment “We focus on Super Smash Bros. Melee (SSBM), a fast-paced multi-player fighting game released in 2001 for the Nintendo Gamecube.”; Section 4 Results “Unless otherwise stated, all agents, human or AI, played as Captain Falcon on the stage Battlefield” Firoiu provides the RL agents playing Super Smash Bros. Melee and playing as Captain Falcon on the stage Battlefield, which is the agent playing the in-game scenario.), the playing comprising taking a plurality of sequential in-game actions available in the at least one game scenario (Firoiu Section 4.1 In-game AI “We began by testing the RL algorithms against the in-game AI. After appropriate parameter tuning, both Q learners and actor-critics proved capable of defeating this AI at its highest difficulty setting, and reached similar average reward levels within a day… Actor-Critics pursued a standard strategy of attacking and counter-attacking, similar to the way humans play. Q-learners on the other hand would consistently find the unintuitive strategy of tricking the in-game AI into killing itself. This multi-step tactic is fairly impressive” Firoiu provides the RL agent taking in-game actions, such as tricking the game AI.); transmitting results of each of the plurality of sequential in-game actions to the determining agent (Firoiu Section 3.3 Training “This means that generating experiences (state action-reward sequences) is a major bottleneck… The many parallel agents periodically send their experiences to a trainer, which maintains a circular queue of the most recent experiences. With the help of a GPU, the trainer continually performs (minibatched) stochastic gradient descent on its set of experiences while periodically saving snap shots of the neural network weights for the agents to load.” Section 4 Results “Q-learners on the other hand would consistently find the un intuitive strategy of tricking the in-game AI into killing itself. This multi-step tactic is fairly impressive; it involves moving to the edge of the stage and allowing the enemy to attempt a 2-attack string, the first of which hits (resulting in a small negative reward) while the second misses and causes the enemy to suicide (resulting in a large positive reward).” Firoiu teaches the agents transmitting their actions for a reward determination, which requires a transmission of the results of the actions to the reward component.); and receiving a reward for successful progression through the at least one game scenario according to at least one reward framework determined by the determining agent (Firoiu Section 4.1 In-game AI “Q-learners on the other hand would consistently find the unintuitive strategy of tricking the in-game AI into killing itself. This multi-step tactic is fairly impressive; it involves moving to the edge of the stage and allowing the enemy to attempt a 2-attack string, the first of which hits (resulting in a small negative reward) while the second misses and causes the enemy to suicide (resulting in a large positive reward).” Firoiu teaches a large positive reward for successful in-game progression (i.e., defeating the game AI).), the reward comprising a quantitative score (Firoiu Figure 2; Section 4.1 In-game AI “Q-learners on the other hand would consistently find the unintuitive strategy of tricking the in-game AI into killing itself. This multi-step tactic is fairly impressive; it involves moving to the edge of the stage and allowing the enemy to attempt a 2-attack string, the first of which hits (resulting in a small negative reward) while the second misses and causes the enemy to suicide (resulting in a large positive reward).” Firoiu teaches the small negative or large positive rewards as quantitative scores, as shown in Figure 2.).
Firoiu fails to explicitly teach …determine at least one reward framework based on results from the search of the at least one Internet platform, the at least one reward framework being determined by at least one metric for a characteristic of the at least one game scenario;
However, Fan teaches determine at least one reward framework based on results from the search of the at least one Internet platform (Fan Section 4 Agent Learning with Large-scale Pre-training “Agents developed in popular RL benchmarks [91, 110] often rely on meticulously crafted dense and task-specific reward functions to guide random explorations. However, these rewards are hard or even infeasible to define for our diverse and open-ended tasks in MINEDOJO. To address this challenge, our key insight is to learn a dense, language-conditioned reward function [at least one reward framework] from in-the-wild YouTube videos and their transcripts [at least one Internet platform]. Therefore, we introduce MINECLIP, a contrastive video-language model that learns to correlate video snippets and natural language descriptions (Fig.4).” Fan teaches crafting reward functions based on YouTube videos and their corresponding transcripts, corresponding to a reward framework based on the results of a search of at least one internet platform.), the at least one reward framework being determined by at least one metric for a characteristic of the at least one game scenario (Fan Section 4 Agent Learning with Large-scale Pre-training “To address this challenge, our key insight is to learn a dense, language-conditioned reward function [at least one reward framework] from in-the-wild YouTube videos and their transcripts. Therefore, we introduce MINECLIP, a contrastive video-language model that learns to correlate video snippets and natural language descriptions [at least one metric for a characteristic of the at least one game scenario] (Fig.4).”; Figure 4 “MINECLIP is a contrastive video-language model pre-trained on MINEDOJO’s massive Youtube database. It computes the correlation between an open-vocabulary language goal string and a 16-frame video snippet. The correlation score can be used as a learned dense reward function to train a strong multi-task RL agent.” Fan teaches using image data from image stills or video frames associated with the at least one in-game scenario (video snippets) as at least one metric for a characteristic of the at least one game scenario.).
Firoiu and Fan are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to reinforcement learning agents in video games. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Firoiu with the above teachings of Fan. Doing so would significantly improve RL training efficiency, making a practical module for open-ended agent learning in video games (Fan Section 4 Agent Learning with Large-scale Pre-training “Because the learned reward model incurs a non-trivial computational overhead, we introduce several techniques to significantly improve RL training efficiency, making MINECLIP a practical module for open-ended agent learning in Minecraft”).
Regarding claim 2, Firoiu in view of Fan teaches the determining agent being further configured to determine an optimal sequence of in-game actions for the at least one game scenario based on a plurality of scores comprising each of the quantitative scores from each of the plurality of iterations (Firoiu Figure 2; Section 4.1 In-game AI “We began by testing the RL algorithms against the in-game AI. After appropriate parameter tuning, both Q learners and actor-critics proved capable of defeating this AI at its highest difficulty setting [an optimal sequence of in-game actions], and reached similar average reward levels within a day… Q-learners on the other hand would consistently find the unintuitive strategy of tricking the in-game AI into killing itself. This multi-step tactic is fairly impressive; it involves moving to the edge of the stage and allowing the enemy to attempt a 2-attack string, the first of which hits (resulting in a small negative reward) while the second misses and causes the enemy to suicide (resulting in a large positive reward). [quantitative scores]” Fan teaches defeating this AI at its highest difficulty setting based on the quantitative reward scores shown in Figure 2, corresponding to an optimal sequence of in-game actions based on quotative scores.).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Firoiu in view of Fan for the same reasons disclosed above in the rejection of claim 1.
Regarding claim 3, Firoiu in view of Fan teaches the at least one Internet platform comprising any one of: a video streaming platform, a game developer website, an Internet forum, a game wiki, and a news source (Firoiu Section 2 The SSBM Environment “We focus on Super Smash Bros. Melee (SSBM), a fast-paced multi-player fighting game released in 2001 for the Nintendo Gamecube.”: Section 4.2 Self-play “Table1: Some results against ranked SSBM players. Rankings from http://wiki.teamliquid.net/smash/SSBM_Rank.S2J [a game wiki] is considered by some to be the best Captain Falcon player in the world.” Firoiu provides searching an internet wiki for rankings related to Super Smash Bros. Melee (SSBM), corresponding to searching the internet for data related to a game scenario, wherein the wiki corresponds to the internet platform.)
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Firoiu in view of Fan for the same reasons disclosed above in the rejection of claim 1.
Regarding claim 4, Firoiu in view of Fan teaches the characteristic of the at least one game scenario being any one of: level of difficulty or skill of the at least one game scenario; novelty of the in-game action or the sequence of actions; surprise due to a result of the action or the sequence of actions; popularity of the in-game action or sequence of actions; humor due to the result of the action or the sequence of actions; and enjoyment of the result of the action or sequence of actions (Fan Section 2 MINEDOJO Simulator & Benchmark Suite “We provide unified observation and action spaces across all tasks, facilitating the development of multi-task and continually learning agents that can constantly adapt to new scenarios and novel tasks. [novelty of the in-game action or the sequence of actions]” Fan teaches agents adapt to new scenarios and novel tasks, corresponding to the characteristic being a novelty of the in-game action or the sequence of actions.).
Firoiu and Fan are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to reinforcement learning agents in video games. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Firoiu with the above teachings of Fan. Doing so would significantly improve RL training efficiency, making a practical module for open-ended agent learning in video games (Fan Section 4 Agent Learning with Large-scale Pre-training “Because the learned reward model incurs a non-trivial computational overhead, we introduce several techniques to significantly improve RL training efficiency, making MINECLIP a practical module for open-ended agent learning in Minecraft”).
Regarding claim 5, Firoiu in view of Fan teaches the at least one metric for the characteristic being any one of: frequency of a key word or phrase associated with the at least one in-game scenario; image data from image stills or video frames associated with the at least one in-game scenario; a trendline indicating a change in the frequency with which the key word or phrase are used; and game data from networked games indicating the frequency with which the action or the sequence of actions are used in the in-game scenario (Fan Section 4 Agent Learning with Large-scale Pre-training “To address this challenge, our key insight is to learn a dense, language-conditioned reward function [at least one reward framework] from in-the-wild YouTube videos and their transcripts. Therefore, we introduce MINECLIP, a contrastive video-language model that learns to correlate video snippets and natural language descriptions [image data from image stills or video frames associated with the at least one in-game scenario] (Fig.4).”; Figure 4 “MINECLIP is a contrastive video-language model pre-trained on MINEDOJO’s massive Youtube database. It computes the correlation between an open-vocabulary language goal string and a 16-frame video snippet. The correlation score can be used as a learned dense reward function to train a strong multi-task RL agent.” Fan teaches using image data from image stills or video frames associated with the at least one in-game scenario (video snippets) as at least one metric for a characteristic of the at least one game scenario, as shown in Figure 4.).
Firoiu and Fan are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to reinforcement learning agents in video games. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Firoiu with the above teachings of Fan. Doing so would significantly improve RL training efficiency, making a practical module for open-ended agent learning in video games (Fan Section 4 Agent Learning with Large-scale Pre-training “Because the learned reward model incurs a non-trivial computational overhead, we introduce several techniques to significantly improve RL training efficiency, making MINECLIP a practical module for open-ended agent learning in Minecraft”).
Regarding claim 6, Firoiu in view of Li in further view of OpenAI (2023) teaches further comprising a memory storage (Fan Section 4.2 RL with MINECLIP Reward “We leverage Self-Imitation Learning [65] to store the trajectories with high MINECLIP reward values in a buffer [a memory storage], and alternate between PPO and self-imitation gradient steps.” Fan teaches a memory storage.) for storing a sequence of game states (Fan Section 2.1 Task Suite I: Programming Tasks “I is the initial conditions of the agent and the world, such as the initial inventory, spawn terrain, and weather. fS: st → {0,1} is the success criterion, a deterministic function that maps the current world state [game states] st to a Boolean success label. fR: st → R is an optional dense reward function.” Fan teaches a storage including world states corresponding to game states.), the actions and the sequence of actions (Fan Section 4.2 RL with MINECLIP Reward “In each episode, the agent is prompted with a language goal and takes a sequence of actions [sequence of actions] to fulfill this goal. When calculating the MINECLIP rewards, we concatenate the agent’s latest 16 egocentric RGB frames in a temporal window to form a video snippet.” Fan teaches a storage including a sequence of actions.), and information associated with the metrics for the at least one characteristic of the in-game scenario (Fan Section 1 Introduction “We develop a new learning algorithm for embodied agents that makes use of the internet-scale domain knowledge we have collected from the web. Using the massive volume of YouTube videos from MINEDOJO, we train a video-text contrastive model in the spirit of CLIP [69], which associates natural language subtitles with their time-aligned video segments.” Fan teaches a storage including collecting Youtube videos (image data from image stills or video frames associated with the at least one in-game scenario), corresponding to the information associated with the metrics for the at least one characteristic of the in-game scenario.).
Firoiu and Fan are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to reinforcement learning agents in video games. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Firoiu with the above teachings of Fan. Doing so would significantly improve RL training efficiency, making a practical module for open-ended agent learning in video games (Fan Section 4 Agent Learning with Large-scale Pre-training “Because the learned reward model incurs a non-trivial computational overhead, we introduce several techniques to significantly improve RL training efficiency, making MINECLIP a practical module for open-ended agent learning in Minecraft”).
Regarding claim 7, Firoiu in view of Li in further view of OpenAI (2023) teaches further comprising a filtering agent configured to determine optimal metas (Fan Section 4 Agent Learning with Large-scale Pre-training “Because the learned reward model incurs a non-trivial computational overhead, we introduce several techniques to significantly improve RL training efficiency, making MINECLIP a practical module for open-ended agent learning in Minecraft [optimal metas] (Sec.4.2)” Fan teaches improving RL training efficiency for gameplay techniques in Minecraft, corresponding to optimal metas.), policies (Fan Section 4.2 RL with MINECLIP Reward “We train a language-conditioned policy network that takes as input raw pixels and predicts discrete control. The policy is trained with PPO [76] on the MINECLIP rewards. In each episode, the agent is prompted with a language goal and takes a sequence of actions to fulfill this goal. When calculating the MINECLIP rewards, we concatenate the agent’s latest 16 egocentric RGB frames in a temporal window to form a video snippet. MINECLIP handles all task prompts zero-shot without any further finetuning.” Fan teaches training using PPO (Proximal Policy Optimization, corresponding to optimal policies.), and strategies (Fan Section 1 Introduction “We hope that this enormous knowledge base can help the agent acquire diverse skills, develop complex strategies, discover interesting objectives, and learn actionable representations automatically.” Fan includes developing complex strategies in Minecraft corresponding to optimal strategies) for the in-game actions as defined by the at least one metric for the characteristic (Fan Section 4 Agent Learning with Large-scale Pre-training “To address this challenge, our key insight is to learn a dense, language-conditioned reward function from in-the-wild YouTube videos and their transcripts. Therefore, we introduce MINECLIP, a contrastive video-language model that learns to correlate video snippets and natural language descriptions [at least one metric for a characteristic of the at least one game scenario] (Fig.4).” Fan teaches video snippets and corresponding descriptions to define in-game actions, corresponding to the metrics, as shown in Figure 4.).
Firoiu and Fan are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to reinforcement learning agents in video games. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Firoiu with the above teachings of Fan. Doing so would significantly improve RL training efficiency, making a practical module for open-ended agent learning in video games (Fan Section 4 Agent Learning with Large-scale Pre-training “Because the learned reward model incurs a non-trivial computational overhead, we introduce several techniques to significantly improve RL training efficiency, making MINECLIP a practical module for open-ended agent learning in Minecraft”).
Regarding claim 9, Firoiu in view of Fan teaches the determining agent being further configured to …evaluate subsequent trends related to the results (Firoiu Section 4.1 In-game AI “We began by testing the RL algorithms against the in-game AI. After appropriate parameter tuning, both Q learners and actor-critics proved capable of defeating this AI at its highest difficulty setting, and reached similar average reward levels within a day.”; Figure 2: “Learning curves for Actor-Critic (purple) and “DQN” (yellow) against the in-game AI. Y-axis is average reward, X-axis is hours.” Firoiu provides the learning curves of Figure 2, corresponding to evaluating trends related to the results.).
Further, Fan teaches …share results from the plurality of iterations with a network of users (Fan Section 1 Introduction “In summary, this paper proposes an open-ended task suite, internet-scale domain knowledge, and agent learning with recent advances on large pre-trained models [11]. We have open-sourced MINEDOJO’s simulator, knowledge bases, algorithm implementations, pretrained model checkpoints, and task curation tools at https://minedojo.org/ [share results]. We hope that MINEDOJO will serve as an effective starter framework for the community to develop new algorithms and advance towards generally capable embodied agent.” Fan teaches open sourcing MINEDOJO’s simulator, knowledge bases, algorithm implementations, pretrained model checkpoints, and task curation tools, corresponding to sharing the results from the iterations with a plurality of users.).
Firoiu and Fan are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to reinforcement learning agents in video games. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Firoiu with the above teachings of Fan. Doing so would significantly improve RL training efficiency, making a practical module for open-ended agent learning in video games (Fan Section 4 Agent Learning with Large-scale Pre-training “Because the learned reward model incurs a non-trivial computational overhead, we introduce several techniques to significantly improve RL training efficiency, making MINECLIP a practical module for open-ended agent learning in Minecraft”).
Regarding claim 10, it is the method embodiment of claim 1 with similar limitations to claim 1 and is rejected using the same reasoning found above in the rejection of claim 1.
Regarding claim 11, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Firoiu in view of Fan for the same reasons disclosed above in the rejection of claim 2.
Regarding claim 12, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Firoiu in view of Fan for the same reasons disclosed above in the rejection of claim 3.
Regarding claim 13, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Firoiu in view of Fan for the same reasons disclosed above in the rejection of claim 4.
Regarding claim 14, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Firoiu in view of Fan for the same reasons disclosed above in the rejection of claim 5.
Regarding claim 15, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Firoiu in view of Fan for the same reasons disclosed above in the rejection of claim 6.
Regarding claim 16, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Firoiu in view of Fan for the same reasons disclosed above in the rejection of claim 7.
Regarding claim 18, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Firoiu in view of Fan for the same reasons disclosed above in the rejection of claim 9.
Regarding claim 19, it is the method embodiment of claim 1 with similar limitations to claim 1 and is rejected using the same reasoning found above in the rejection of claim 1. Further, Firoiu teaches …determining an optimal sequence of in-game actions for the at least one game scenario based on a plurality of scores comprising each of the quantitative scores from each of the plurality of iterations (Firoiu Section 3.3 Training “The many parallel agents periodically send their experiences to a trainer, which maintains a circular queue of the most recent experiences… In RL, however, we are optimizing more than just a loss function- we are optimizing a policy through policy iteration… The method of Lagrange multipliers shows that the optimal direction for ∆θ is given by the solution to Hx = g (which we then rescale to satisfy the constraint).” Firoiu teaches RL optimization including optimal direction parameters for RL agents navigating in-game actions, corresponding to an optimal sequence of in-game actions.).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Firoiu in view of Fan for the same reasons disclosed above in the rejection of claim 1.
Regarding claim 20, the rejection of claim 19 is incorporated herein. Further, the limitations in this claim are taught by Firoiu in view of Fan for the same reasons disclosed above in the rejection of claim 9.
Claims 8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Firoiu et al. (Beating the World’s Best at Super Smash Bros. Melee with Deep Reinforcement Learning) (“Firoiu”) in view of Fan et al. (MINEDOJO: Building Open-Ended Embodied Agents with Internet-Scale Knowledge) (“Fan”) in further view of Gisslén et al. (U.S. Patent Publication No. 2022/0266145) (“Gisslén”).
Regarding claim 8, Firoiu in view of Fan teaches the determining agent further comprising a deep neural network having an input layer, a plurality of hidden layers, and an output layer, the plurality of hidden layers being configured to process input received at the input layer and transmit a first output to the output layer (Firoiu Section 3.3 Training “All of our neural networks (Q, actor, and critic) used architectures with two fully-connected hidden layers of size 128” [neural network having an input layer, a plurality of hidden layers, and an output layer] Section 6 Conclusion “Finally, we demonstrate an agent based on deep learning [deep neural network] which sets the state of the art in this environment, surpassing the abilities of ten highly-ranked human players” Firoiu teaches a deep neural network having input/output layers and hidden layers.), but fails to explicitly teach the plurality of hidden layers being trained and tuned using weights and biases for optimal results based on the at least one metric for the characteristic.
However, Gisslén teaches the plurality of hidden layers being trained and tuned using weights and biases for optimal results based on the at least one metric for the characteristic (Gisslén [0040] “The adversarial RL system 100 uses adversarial deep RL techniques for training the first RL agent 102”; [0075] “In this example, the Generator 102 includes a Generator network based on a feed forward neural network with at least 2 hidden layers and 512 neural units per layer [the plurality of hidden layers being trained], with a hyperparameter γ of 0.990. The RL technique used for training the Generator 102 is based on, without limitation, for example a Proximal Policy Optimization (PPO) algorithm [optimal results] and the like with a learning rate of 2e-4. The Generator 104 receives a first reward 102b from the computer game environment 106 of the Platform game 300 along with observation data in the form of a state vector. The observation data [metric for the characteristic] provided by the computer game environment 106 of the Platform game 300 to the Generator 102 consists of a game state array or vector including data representative of the relative position to the overall goal, angle relative to the overall goal, overall goal distance, previous block position, size, and rotation, and auxiliary diversity input value and the like.”; [0080] “During training, the Generator 102 also receives a negative/positive reward from the Generator reward function… with α; β being weighting factors [weights and biases] that are adjusted by the designer of the game.” Gisslén teaches a deep neural network including hidden layers trained on weights and biases for optimal results based on metrics.).
Firoiu, Fan and Gisslén are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to reinforcement learning agents in video games. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Firoiu in view of Fan with the above teachings of Gisslén. Doing so would provide improved generalization for interacting with a generated PCG computer game environment (Gisslén [0001] “The present application relates to apparatus, systems and method(s) for using adversarial reinforcement-learning (RL) techniques to train a first RL agent to perform procedural content generation (PCG) for a computer game environment of a video game and train a second RL agent with improved generalization for interacting with a generated PCG computer game environment.”).
Regarding claim 17, the rejection of claim 10 is incorporated herein. Further, the limitations in this claim are taught by Firoiu in view of Fan in further view of Gisslén for the same reasons disclosed above in the rejection of claim 8.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KURT NICHOLAS PRESSLY whose telephone number is (703)756-4639. The examiner can normally be reached M-F 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KURT NICHOLAS PRESSLY/Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights