DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1 – 9 have been amended.
Claims 1 – 9 are pending.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
This subject matter eligibility analysis follows the latest guidance for Patent Subject Matter Eligibility Guidance.
Claims 1 - 9 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Step 1:
Claims 1 – 6 are drawn to a method.
Claims 7 are drawn to a non-transitory CRM
Claims 8 and 9 are drawn to a system.
Thus, initially, under Step 1 of the analysis, it is noted that the claims are directed towards eligible categories of subject matter.
Step 2A:
Prong 1: Does the Claim recite an Abstract idea, Law of Nature, or Natural Phenomenon?
Claims 8 are exemplary because they require substantially the same operative limitations of the remaining claims (reproduced below.) Examiner has underlined the claim limitations which recite the abstract idea, discussed in detail in the paragraphs that follow.
[Claim 8] A system comprising:
a processor; and
a memory connected to the processor, wherein the memory comprises a program, when executed by the processor, configured to perform a method comprising:
generating game state text and action text, which are controlled natural language (CNL) data expressed in a prescribed format, from data of game states and actions that are included in history data concerning a computer game;
generating training data to produce generated training data, wherein the generated training data comprises pairs of game state text and action text corresponding to pairs of one game state and an action selected in the one game state; and
generating a trained model by training, using the generated training data, a neural network model that is capable of learning distributed representations of words or word sequences,
wherein the trained model predicts an action to be selected by a user in the computer game that proceeds in accordance with a plurality of actions selected by the user while updating a plurality of game states of the computer game
The claims recite italicized limitations that fall within at least one of the groupings of abstract ideas enumerated in the 2019 PEG, namely, Mental Processes and Certain Methods of Organizing human activity.
More specifically, under this grouping, the italicized limitations represent Mental Processes such as converting historical actions such as game state and player actions information , analyzing that information to model and predict player actions and Methods of Organizing Human Activity such as user behavior prediction and/or decision making in games.
Prong 2: Does the Claim recite additional elements that integrate the exception in to a practical application of the exception?
Although the claims recite additional limitations, these limitations do not integrate the exception into a practical application of the exception. For example, the claims require additional limitations as follow, (emphasis added): Processors, memory and neural networks
These additional limitations do not represent an improvement to the functioning of a computer, or to any other technology or technical field, (MPEP 2106.05(a)). Nor do they apply the exception using a particular machine, (MPEP 2106.05(b)). Furthermore, they do not effect a transformation. (MPEP 2106.05(c)). Rather, these additional limitations amount to an instruction to “apply” the judicial exception using a computer as a tool to perform the abstract idea. Therefore, since the additional limitations, individually or in combination, are indistinguishable from a computer used as a tool to perform the abstract idea, the analysis continues to Step 2B, below.
Step 2B:
Under Step 2B, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because they amount to conventional and routine computer implementation and mere instructions for implementing the abstract idea on generic computing devices.
For example, as pointed out above, the claimed invention recites additional elements facilitating implementation of the abstract idea. Applicant has claimed computer processors, memory and neural networks. However, all of these elements viewed individually and as a whole, are indistinguishable from conventional computing elements known in the art. Therefore, the additional elements fail to supply additional elements that yield significantly more than the underlying abstract idea.
As the Alice court cautioned, citing Flook, patent eligibility cannot depend simply on the draftsman' s art. Here, amending the claims with generic computing elements does not (in this Examiner' s opinion), confer eligibility.
Regarding the Berkheimer decision, Applicant specification establishes that these additional elements are generic:
[0028] Fig. 1 is a block diagram showing the hardware configuration of the learning device 10 in one embodiment of the present invention. The learning device 10 includes a processor 11, an input device 12, a display device 13, a storage device 14, and a communication device 15. These individual constituent devices are connected via a bus 16. Note that interfaces are interposed as needed between the bus 16 and the individual constituent devices. The learning device 10 includes a configuration similar to that of an ordinary server, PC, or the like.
[0067] Fig. 7 is a block diagram showing the hardware configuration of the determining device 50 in one embodiment of the present invention. The determining device 50 includes a processor 51, an input device 52, a display device 53, a storage device 54, and a communication device 55. These individual constituent devices are connected via a bus 56. Note that interfaces are interposed as needed between the bus 56 and the individual constituent devices. The determining device 50 includes a configuration similar to that of an ordinary server, PC, or the like.
[0091]
Furthermore, the learning method in this embodiment is widely applicable to turn-based battle games, and makes it possible to expand AI that simulate human playing tendencies to a variety of genres. Furthermore, the method of generating a trained model by using fine tuning, which is an example of this embodiment, is a method that is compatible with the case where replay logs are continuously expanded, which makes it suitable for game titles that will be run on a long-term basis. Furthermore, with the training model generated in this embodiment, since explanations of cards are interpreted as natural language text as well as card names, it is possible to perform inference with relatively high accuracy even with new cards that have been newly released. Furthermore, with the method of generating a trained model in this embodiment, without depending on any specific transformer neural network technology or fine tuning method, it is possible to use an arbitrary natural language learning system based on a transformer neural network that support learning for next sentence prediction. Therefore, it is possible to switch the natural language learning system when a neural-network-based natural language learning system having improved accuracy has emerged or depending on the support status of external libraries.
Therefore, these elements fail to supply additional elements that yield significantly more than the underlying abstract idea. Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea).
Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation.
Moreover, the claims do not recite improvements to another technology or technical field. Nor, do the claims improve the functioning of the underlying computer itself -- they merely recite generic computing elements. Furthermore, they do not effect a transformation of a particular article to a different state or thing: the underlying computing elements remain the same.
Concerning preemption, the Federal Circuit has said in Ariosa Diagnostics, Inc., V. Sequenom, Inc., (Fed Cir. June 12, 2015):
The Supreme Court has made clear that the principle of preemption is the basis for the judicial exceptions to patentability. Alice, 134 S. Ct at 2354 (“We have described the concern that drives this exclusionary principal as one of pre-emption”). For this reason, questions on preemption are inherent in and resolved by the § 101 analysis. The concern is that “patent law not inhibit further discovery by improperly tying up the future use of these building blocks of human ingenuity.” Id. (internal quotations omitted). In other words, patent claims should not prevent the use of the basic building blocks of technology—abstract ideas, naturally occurring phenomena, and natural laws. While preemption may signal patent ineligible subject matter, the absence of complete preemption does not demonstrate patent eligibility. In this case, Sequenom’s attempt to limit the breadth of the claims by showing alternative uses of cffDNA outside of the scope of the claims does not change the conclusion that the claims are directed to patent ineligible subject matter. Where a patent’s claims are deemed only to disclose patent ineligible subject matter under the Mayo framework, as they are in this case, preemption concerns are fully addressed and made moot. (Emphasis added.)
For these reasons, it appears that the claims are not patent-eligible under 35 USC §101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2 and 5 – 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yao et al “Keep Calm and Explore: Language Models for Action Generation in Text-Based Games, 2020 in view of Kim et al (US 10,558,852).
As per claim 1, Yao discloses:
generating game state text and action text, which are controlled natural language (CNL) data expressed in a prescribed format, from data of game states and actions that are included in history data concerning a computer game (Yao discloses the generation of game states and actions expressed in a prescribed format from history concerning a game. Regarding the limitations directed towards CNL, the Examiner notes that Yao discloses CALM being trained on human transcripts comprising NL wherein CALM processes these and generated “compact sets of action candidate” i.e. A restrictive language data) (Yao 3.2 and 4.1) and
generating training data to produce generated training data, wherein the generated training data comprises pairs of game state text and action text corresponding to pairs of one game state and an action selected in the one game state; and (Yao discloses the training of “CALM” wherein top actions are generated based for every unique state of the game, there is generated the top 30 actions (i.e. pairs of actions)) (Yao 4.2, “Generating Top Actions”)
generating a trained model by training, using of the generated training data . (Yao further discloses the generation of the trained model based upon the CALM generated top 30 actions for each state (Yao 5.3 Analysis, 5. CALM (random agent))
wherein the trained model is configured to predict an action to be selected by a user in the computer game that proceeds in accordance with a plurality of actions selected by the user while updating a plurality of game states of the computer game. (Yao discloses the use of a trained model that is configured to predict and action to be selected by a user and update the game states) (Yao 4.2 “Generating top Actions”; 5.2 “Evaluating game play on Jericho” “)
Yao fails to disclose:
a neural network model that is capable of learning distributed representations of words or word sequences
However, in a similar field of endeavor wherein a user actions are predicted Kim teaches that neural networks have bee used in many areas such as natural language processing tasks (Kim 1:46 – 51) and that the use of recurrent neural networks are useful for predicting next user actions and predicting the likelihood of target behaviors based upon logs of prior user behavior (Kim 1:58 – 67; 5:17 – 33). Kim teaches the use of training based upon “navigational sequences” (Kim 5:35 – 28), that refer to a sequence of actions performed by the user that include identifiers for the actions performed and the sequence or order of the actions during the navigational session. Kim teaches the use actions can include text based actions such as engaging in a chat, or writing a review (i.e. text) (Kim 3:46 – 54). The Examiner further notes that the neural network of Kim is indeed capable to perform the functional language of “learning distributed representations of words or word sequences”.
It would be obvious to one of ordinary skill in the art, at the time of filing, to modify Yao in view of Kim to utilize a known technique to modify similar devices in the same way by utilizing a neural network that is trained on textual data that is capable of recognizing words or word sequences. As Kim teaches For instance, RNNs have been especially useful in Natural Language Processing tasks where RNNs have shown superior performance on machine translation, document classification, and sentiment analysis.
As per claim 2, Yao discloses:
wherein the generating training data comprises generating, as game state text corresponding to one game state, a plurality of items of game state text having different orders of a plurality of text elements that are included in the game state text, and generating the training data comprises pairs of each of the plurality of items of generated game state text and action text corresponding to an action selected in the one game state. (Yao discloses the training of “CALM” wherein top actions are generated based for every unique state of the game, there is generated the top 30 actions (i.e. pairs of actions)) (Yao 4.2, “Generating Top Actions”)
As per claim 5, wherein generating the training data comprises generating game state text and action text expressed by using grammar, syntax, and vocabulary for mechanical conversion into a distributed representation, based on a rule-based system created in advance, from game state data and action data. (As best can be understood by the usage of the phrase “mechanical conversion” (which the examiner will interpret as merely a conversion process, Yao discloses the converting of a user’s commands expressed by using grammar, syntax and vocabulary into game state and text pair commands, such as in Fig 2, wherein “you see a locked case” is converted to the candidate pair of “at,2 unlock case” (Yao fig 2)
As per claim 6,
determining a plurality of actions selectable by the user in a game state subject to prediction; generating pairs of game state text and action text from pairs of game state data and action data for a plurality of individual actions that are determined; and determining an action that is predicted to be selected by the user by using the individual generated pairs of game state text and action text as well as the trained model recited in Claim 1. (Yao, see fig 2, wherein action text and game stat is generated, context is considered to generate a plurality of action candidates (i.e. at,1 – at,3 and the DRNN calculates a q-value and selected a predicted action to selected ‘at “unlock case”’)
As per claim 7, A non-transitory computer readable medium storing a program that causes a computer to execute the steps of the method according to Claim 1 (Yao entire disclosure)
Dependent claim(s) 8 is/are anticipated by Yao based on the same analysis set forth for claim(s) 1, which are similar in claim scope.
Dependent claim(s) 9 is/are anticipated by Yao based on the same analysis set forth for claim(s) 6, which are similar in claim scope.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yao et al “Keep Calm and Explore: Language Models for Action Generation in Text-Based Games, 2020 in view of Kim et al (US 10,558,852) in view of Devlin et al; “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” 2019.
As per claim 3, Yao fails to disclose:
wherein generating the trained model comprises generating the trained model by training a pretrained natural language model with the generated training data, the pretrained natural language model having learned in advance grammatical structures and text-to- text relationships concerning a natural language.
However in a similar field of endeavor, Devlin discloses the generation of a pretrained model based upon training a pretrained natural language model that has learned in advance grammatical structures and text to text relationships by utilizing masked LM and next sentence prediction (NSP) (Devlin page 4174m Task #1, Task #2) Devlin further states (“3.2 Fine-tuning BERT Fine-tuning is straightforward since the self-attention mechanism in the Transformer allows BERT to model many downstream tasks— whether they involve single text or text pairs—by swapping out the appropriate inputs and outputs. For applications involving text pairs, a common pattern is to independently encode text pairs before applying bidirectional cross attention, such as Parikh et al. (2016); Seo et al. (2017). BERT instead uses the self-attention mechanism to unify these two stages, as encoding a concatenated text pair with self-attention effectively includes bidirectional cross attention between two sentences. For each task, we simply plug in the task specific inputs and outputs into BERT and finetune all the parameters end-to-end. At the input, sentence A and sentence B from pre-training are analogous to (1) sentence pairs in paraphrasing, (2) hypothesis-premise pairs in entailment, (3) question-passage pairs in question answering, and (4) a degenerate text-∅ pair in text classification or sequence tagging. At the output, the token representations are fed into an output layer for token level tasks, such as sequence tagging or question answering, and the [CLS] representation is fed into an output layer for classification, such as entailment or sentiment analysis.” (Devlin page 4175)
It would be obvious to one of ordinary skill in the art, at the time of filing, to modify Yao in view of Devlin to utilize pretrained that was trained to learn in advance grammar and text to text relationships associated with natural language. This would be beneficial as it would enable the text based game to more efficiently predict the next most correct action based upon the context of the game state and the natural language used.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yao et al “Keep Calm and Explore: Language Models for Action Generation in Text-Based Games, 2020, in view of Kim et al (US 10,558,852) in view of Kano et al (US 2021/0279638).
As per claim 4, Yao fails to disclose:
wherein the generated training data including first pairs and second pairs, the first pairs being pairs of game state text and action text corresponding to pairs of one game state and an action selected in the one game state, generated based on the data of game states and actions that are included in the history data, and the second pairs being pairs of the one game state text and action text corresponding to an action…(Yao discloses the generation of training data based upon a plurality of pairs of observable game states and actions. Yao discloses the training of “CALM” wherein top actions are generated based for every unique state of the game, there is generated the top 30 actions (i.e. pairs of actions)) (Yao 4.2, “Generating Top Actions”)
Yao fails to disclose:
…that is selected at random from the plurality of actions selectable by the user and that is not included in the first pairs; and
wherein generating the trained model comprises generating the trained model by performing training with the first pairs as correct data and performing training with the second pairs as incorrect data.
However, in a similar field of endeavor wherein machine learning models are trained, Kano discloses a system to improve the accuracy of machine learning (Kano 0006). Kano states:
“[0022] In the present exemplary embodiment, a filter model is trained by using correct pairs of text and title, which are used as “positive examples”, and incorrect pairs, which are used as “negative examples”. Negative examples, which are incorrect pairs, are obtained by changing input-output pairs, for example, through random sampling. In the present exemplary embodiment, negative examples are generated by changing input-output pairs.
In the present exemplary embodiment, the filter model 22 learns how appropriate pairs of text and summary are. The difference between the present exemplary embodiment and the related art is that, while a classification model is used to increase the training data in the related art, the negative-example generating unit 30 generates the negative example 32 from the training data 26 in the present exemplary embodiment. As long as combinations between input and output are changed, the generation process performed by the negative-example generating unit 30 is any. Pairs of text and summary in the training data 26 may be subjected to random sampling to generate new pairs, thus generating the negative example 32.
[0043] The actual pairs of text and summary in the training data 26 are used as the positive example 28, and the pairs, which are obtained through random sampling, are used as the negative example 32. Thus, the filter model 22 is trained. After training, the filter model 22 makes discrimination again only on the positive example 28 in the training data 26, that is, on the training data 26 itself. A bottom n % of data in descending order of predicted probability is removed from the training data for the summary model 24, that is, the supervised data that is input to the summary model 24. (Kano 0022, 0042 – 0043).”
It would be obvious to one of ordinary skill in the art, at the time of filing, to modify Yao in view of Kano to utilize negative sampling of randomly selected pairs of that are incorrect to train the machine learning model. As Kano states, this would be beneficial as it would improve the accuracy of the machine learning model, “To improve the accuracy of machine learning, it is necessary to prepare, in advance, a sufficient amount of supervised data formed of correct input-output pairs (hereinafter referred to as “positive examples”). In a machine learning model” (Kano 0006)
Response to Arguments
Applicant’s arguments with respect to claim(s) 1 - 9 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Please see above rejection in view of newly found reference to Kim et al (US 10,558,852)
Regarding the rejection of the claims under 35 U.S.C. 101 the Applicant states essentially that unlike the claims at issue in Recentive, the present claims “recite unconventional and non-generic machine learning techniques for generating a trained model for predicting an action to be selected by a user in a computer game. Thus, the claimed invention provides a technological improvement to both machine learning and computer game technology under the first and second steps of the Alice test. In particular, the amended independent claims require the following non-generic and non-conventional machine-learning techniques: (1) generating game state text and action text, which are controlled natural language (CNL) data expressed in a prescribed format, from data of game states and actions that are included in history data concerning a computer game, (2) generating training data that includes pairs of game state text and action text corresponding to pairs of one game state and an action selected in the one game state, and (3) generating a trained model by training, using the training data, a neural network model that is capable of learning distributed representations of words or word sequences.” (Remarks page 12 – 13). The Examiner respectfully disagrees and notes that while the are a number of steps, the claims essentially are using generated data such as data relating to pairs of game state and action text and training a neural network to generate a trained model. The Examiner notes that the Applicant is reciting aspects of machine learning that are inherent to generic machine learning techniques, specifically that of taking data, training a model on it and producing a trained model. The Applicant has also pointed out the functional language of what the model is capable of leaning, to which the Examiner notes all models have the capability of being trained to learn.
The Applicant further goes on to state under Prong Two Step 2A that the judicial exception is integrated into a practical application. Applicant states:
In view of MPEP § 2106.05, the originally-filed specification clearly identifies a technological problem related to training neural network models, where "neural network technology" has made it possible to recognize context effectively "in the case of learning causal relationships or order relationships as in turn-based battle games, but it has been difficult to use this type of technology for the purpose of learning game history data [for other computer game genres]." See Published Application, pars. [0010] and [0138] (emphasis added). Based on this articulated technical problem, the originally-filed specification describes a technological solution using "controlled natural language (CNL) data" in the training process of a neural network model, where the CNL data is expressed in a prescribed format in which grammar and vocabulary are controlled so as to satisfy prescribed requirements for pairs of game state text and action text. See Published Application, pars. [0080]- [0082]. As such, the originally-filed specification clearly provides evidence indicating a technical improvement to machine learning and computer game technology. (Remarks pages 13 – 17)
The Examiner respectfully disagrees and fails to see how the mere usage of a type of data such as a controlled natural language (CNL) wherein data is of a certain specified format, provides an improvement to the functioning of a computer as specified in MPEP 2106.05 or even as the applicant points out to machine learning. Rather this merely appears to be applying generic machine learning to a particular context or merely applying a different type of data such as CNL data to a machine learning model to train it.
Regarding Step 2B, the Applicant argues the claimed machine learning system provides an unconventional combination that uses unconventional techniques to generate a trained model with specific and non- generic machine learning technology. The Examiner respectfully disagrees and directs attention to the above rejection addressing the newly amended claim limitations and the identification of additional elements wherein the additional elements such as computer processors, memory and neural networks are clearly shown to be generic and conventional additional elements that fails to amount to significantly more than the judicial exception. The Examiner further points to the above rejection showing from the Applicant’s own specification that the additional elements are generic and conventional. The Examiner maintains the rejection.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROSS A WILLIAMS whose telephone number is (571)272-5911. The examiner can normally be reached Mon-Fri 8am - 4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kang Hu can be reached at (571)270-1344. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RAW/Examiner, Art Unit 3715
7/11/2026
/KANG HU/Supervisory Patent Examiner, Art Unit 3715
Prosecution Insights