DETAILED ACTION
This Office Action is responsive to the Applicant’s submission, filed on December 30, 2025, amending claims 1, 5, 8, 12, 15 and 19. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-5, 8-12 and 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No. 2019/0354759 to Somers et al. (“Somers”), and also over the article entitled, “StarCraft II: A New Challenge for Reinforcement Learning” by Vinyals et al. (“Vinyals”).
Regarding claims 1, 8 and 15, Somers describes systems and methods for generating models to control AI units based on replays of gameplay (e.g. of video games) (see e.g. paragraphs 0021-0022). Like claimed, Somers particularly teaches:
obtaining, by a device comprising a memory storing instructions and a processor in communication with the memory, a to-be-trained image set, the to-be-trained image set comprising N to-be-trained images, N being an integer greater than or equal to 1 (see e.g. paragraphs 0044-0045 and 0076-0078: Somers teaches generating video gameplay logs, which include game state information, inputs provided by people through an input device, and/or images indicating positional information during play of a video game. The video gameplay logs can particularly include images received from video output during gameplay, wherein the images can include a collection of image frames for each state of the game – see e.g. paragraphs 0046 and 0048. Such a collection of images frames is considered a to-be-trained image set like claimed, which comprises N to-be-trained images, N being an integer greater than or equal to 1. Somers discloses that the gameplay logs are provided through a network to an AI model training system that trains AI models based on the video gameplay logs – see e.g. paragraphs 0048 and 0056-0057. The AI model training system can include a computer that comprises a memory storing instructions and a processor in communication with the memory – see e.g. paragraphs 0146-0148. The to-be-trained image set in the gameplay logs is thus understandably obtained by a computing device that comprises a memory storing instructions and a processor in communication with the memory to train the AI models based on the gameplay logs.);
extracting, by the device, a to-be-trained feature from each to-be-trained image, the to-be-trained feature representing an image feature of a first region, and extracting another to-be-trained feature representing an attribute feature related to an interaction operation (see e.g. paragraph 0064-0066: Somers discloses that the machine learning system trains the AI models using model inputs provided by the video gameplay log. The model inputs are generated for each state of the gameplay where a person provides control inputs – see e.g. paragraph 0080. The model inputs can include an image of a map, positions of objects in the game environment or other spatial data – see e.g. paragraphs 0066-0067 and 0081. The image can particularly be an image frame that has been simplified or converted into a data structure – see e.g. paragraphs 0081, 0087-0091 and 0096 . Such a simplified or converted image frame is a considered a “to-be-trained” feature extracted from each to-be-trained image, wherein the extracted to-be-trained feature represents an image feature of a first region. Somers further discloses that the model inputs can include non-positional, non-spatial situational data – see e.g. paragraphs 0068 and 0082. This non-positional, non-spatial situational data is considered another to-be-trained feature, which is related to an interaction operation.);
constructing, at the same time by the device, a first to-be-trained label related to a micro control task and a second to-be-trained label related to a big picture task that correspond to each to-be-trained image, the first to-be-trained label representing a label related to operation content corresponding to the micro control task, and the second to-be-trained label representing a label related to an operation intention corresponding to the big picture task (see e.g. paragraphs 0064-0066: as noted above, Somers discloses that the machine learning system trains the AI models using model inputs provided by the video gameplay log. Somers discloses that the model inputs include human control inputs, i.e. inputs provided through an input device, wherein each human control input is associated with a respective image – see e.g. paragraphs 0069-0070 and 0083. The identifier for each human control input is considered a “first to-be-trained label” like claimed, which relates to a micro control task that corresponds to each to-be-trained image, and wherein the identifier of the first to-be-trained label represents a label related to operation content corresponding to the micro control task. Somers further discloses that the model inputs can be grouped or divided into different bins based on the roles or actions occurring during gameplay, e.g. offense or defense, wherein the bins are categorized with a plurality of labels – see e.g. paragraphs 0084, 0111, 0115. The label of each of the bins into which model inputs, including the images, are grouped is considered a “second to-be-trained” label like claimed, which relates to a big picture task and represents a label related to an operation intention corresponding to the big picture task. The human control inputs and the bins into which the model inputs are placed are identified during a process for generating training data based on video gameplay logs (see e.g. paragraphs 0076 and 0080-0084). Consequently, the first-to-be-trained label, i.e. the identifier of a human control input, is considered to be constructed at the same time – while constructing training data – as the second to-be-trained label, i.e. the identifier of the bin into the model inputs are grouped.);
training, by the device, a neural network model to obtain a combined model according to the to-be-trained feature in each to-be-trained image, and according to the first to-be-trained label related to the micro control task and the second to-be-trained label related to the big picture task that corresponds to the each to-be-trained image (see e.g. paragraphs 0061, 0112-0114 and 0116: Somers discloses that the AI model training system trains a plurality of AI models to be executed under different game states, and a strategy AI model that selects one of the other AI models based on a current game state. Each of the AI models can be implemented with, e.g. a neural network – see e.g. paragraphs 0058-0059 and 0105-0108. Particularly, as noted above, Somers discloses that the machine learning system trains the AI models using model inputs provided by the video gameplay log – see e.g. paragraphs 0064-0066. As further noted above, Somers discloses that these model inputs include: (i) a to-be-trained feature, e.g. a simplified image or data structure, extracted from each to-be-trained image – see e.g. paragraph 0081; (ii) a to-be-trained feature, e.g. non-positional, non-spatial data, representing an attribute feature related to an interaction operation – see e.g. paragraphs 0068 and 0082; and (iii) a first to-be-trained label, i.e. human control input, related to a micro control task – see e.g. paragraphs 0069-0070 and 0083. Somers further discloses that the AI models are also trained according to the second-to-be-trained label, i.e. the bin associated with each model input, which is related to the big picture task that corresponds to each to-be-trained image – see e.g. paragraphs 0111-0116. The plurality of AI models including the strategy AI model is considered a “combined model” like claimed.);
obtaining, by the device, a to-be-trained video, the to-be-trained video comprising a plurality of frames of interaction images (see e.g. paragraphs 0026 and 0133: Somers discloses that the AI models can continue to learn, train and improve as new training data is collected. In particular, Somers discloses that additional training data regarding specific scenarios, i.e. bins, can be collected and provided to the machine learning systems to refine the associated AI models – see e.g. paragraph 0119. Somers suggests that such training data is collected, in part, by obtaining video comprising a plurality of frames of gameplay – see e.g. paragraphs 0044-0048 and 0077-0081. Such video is considered a “to-be-trained video” like claimed, which comprises a plurality of frames of interaction images.);
extracting, by the device, target scene data corresponding to the to-be-trained video by using the combined model, the target scene data comprising related data in a target scene (see e.g. paragraphs 0026, 0119 and 0133: as noted above, Somers discloses that the AI models can continue to learn, train and improve as new training data is collected, whereby additional training data regarding specific scenarios, i.e. bins, is collected and provided to the machine learning systems to refine the associated AI models. Like further noted above, Somers suggests that such training data is collected, in part, by obtaining to-be-trained video comprising a plurality of frames of gameplay – see e.g. paragraphs 0044-0048 and 0077-0081. Somers further suggests that the training data is collected by extracting target scene data corresponding to the to-be-trained video, wherein the target scene data comprises related data in a target scene, e.g. image frames from the video, user inputs and non-positional, non-spatial situational data – see e.g. paragraphs 0066-0070 and 0080-0083. Moreover, Somers suggests that the data can be extracted in part by using the combined model – see e.g. paragraphs 0028, 0048, 0070, 0128 and 0133);
training, by the device, the same combined model only at a level of the micro control task without training at a level of the big picture task to obtain a target model parameter according to the target scene data (see e.g. paragraphs 0061, 0112-0114 and 0116: Like noted above, Somers discloses that the AI models composing the combined model include a plurality of AI models that are executed in different game states, and also include a strategy AI model that selects one of the other AI models based on a current game state. The models that execute under different game states are trained to generate one or more controls, e.g. directional inputs and/or button presses, that are similar to controls that a human might provide – see e.g. paragraphs 0112-0114 and 0128-0129. Accordingly, such models correspond to micro control tasks, e.g. the different control inputs. The strategy AI model is trained to identify a current game state and select one of the other models to execute based on the current game state – see e.g. paragraphs 0061 and 0116. Accordingly, the strategy AI model corresponds to big picture tasks. As noted above, Somers discloses that the AI models can continue to learn, train and improve as new training data is collected, whereby additional training data regarding specific scenarios, i.e. bins, is collected and provided to the machine learning systems to refine the associated AI models – see e.g. paragraphs 0026, 0119 and 0133. Somers particularly teaches that only a selected one or more of the AI models can be trained with the additional, targeted training data; for example, a model for cross-crease passes in a hockey game can be trained with additional training data if the model is deemed ineffective – see e.g. paragraph 0119. Accordingly, in such circumstances, the combined model is trained only at a level of the micro control task, i.e. by training only one or more of the AI models that are executed in different game states, without training at a level of the big picture task, i.e. without training the strategy AI model. It is appreciated that a target model parameter, i.e. a parameter of an AI model that is executed in a particular game state, would be obtained via such training according to the target scene data, i.e. according to the image frames, user inputs and non-positional, non-spatial situational data extracted from the new training data.); and
updating, by the device, the combined model according to the target model parameter to obtain a reinforced combined model (see e.g. paragraphs 0026, 0119 and 0133: as noted above, Somers discloses that the AI models can continue to learn, train and improve as new training data is collected. Accordingly, it is apparent that the AI models are updated according to target model parameters based on the new training data. The updated AI models and the strategy AI model are considered a reinforced combined model like claimed.).
Accordingly, Somers teaches a method similar to that of claim 1. Somers discloses that such teachings can be implemented by a device comprising a memory storing instructions and a processor in communication with the memory (see e.g. paragraphs 0048, 0056-0057, 0146-0148 and 0153-0155). Such a device for implementing the above-described tasks taught by Somers is considered an apparatus similar to that of claim 8. The memory of such a device comprising software to implement the above-described tasks is considered a non-transitory computer-readable storage medium similar to that of claim 15. Somers, however, does not explicitly disclose that a to-be-trained feature set is extracted from each to-be-trained image like required by claims 1, 8 and 15, the to-be-trained feature set comprising a first to-be-trained feature, a second to-be-trained feature and a third to-be-trained feature, the first to-be-trained feature representing an image feature of a first region, the second to-be-trained feature representing an image feature of a second region, and the third to-be-trained feature representing an attribute feature related to an interaction operation, wherein a range of the first region is smaller than a range of the second region, and wherein the neural network model is trained to obtain the combined model according to the to-be-trained feature set. Moreover, Somers does not explicitly disclose that the first to-be-trained label represents a hierarchical label and includes “predicting a to-be-executed key first and then a release parameter of the to-be-executed key,” as is recited in claims 1, 8 and 15. Somers also does not explicitly disclose that, when the combined model is trained only at the level of the micro control task, a target model parameter according to the target scene data, the first to-be-trained label, and a first predicted label is obtained, wherein the first predicted label represents a label that is obtained through prediction and that is related to the operation content corresponding to the micro control task, and the first predicted label is a predicted value and the first to-be-trained label is a true value, as is further required by claims 1, 8 and 15.
Vinyals generally describes, inter alia, a reinforcement learning environment based on the game StarCraft II, and further provides a dataset of game replay data from human expert players that can be used to train neural networks to predict game outcomes and player actions (see e.g. the Abstract). In particular, like Somers, Vinyals teaches generating AI models based on replays of gameplay (i.e. of StarCraft II) (see e.g. section 5 “Supervised Learning from Replays”). To do so, Vinyals teaches:
obtaining a to-be-trained image set, the to-be-trained image set comprising N to-be-trained images, N being an integer greater than or equal to 1, and extracting a to-be-trained feature set from each to-be-trained image, the to-be-trained feature set comprising a first to-be-trained feature, a second to-be-trained feature, and a third to-be-trained feature, the first to-be-trained feature representing an image feature of a first region, the second to-be-trained feature representing an image feature of a second region, the third to-be-trained feature representing an attribute feature related to an interaction operation, and wherein a range of the first region is smaller than a range of the second region (Vinyals teaches employing replay data from games of StarCraft II to train networks to predict a game outcome and the action by the player at each timestep:
In the rest of this section, we provide baselines using the architectures described in Section 4, but using a set of 800K games to learn both a value function (i.e., predicting the winner of the game from game observations), and a policy (i.e., predicting the action taken from game observations). The games contain all possible matchups in StarCraft II (i.e., we do not restrict the agent to play a single race).
(Section 5 “Supervised Learning from Replays”).
We train dual-headed networks that predict both the game outcome (1 = win vs. 0 = loss or tie), and the action taken by the player at each time step. Sharing the body of the network makes it necessary to balance the weights for the two loss functions, but it also allows value and policy predictions to inform one another. We did not make ties a separate game outcome class in the supervised training setup, since the number of ties in the dataset is very low (< 1%) compared to victory and defeat.
(Section 5 “Supervised Learning from Replays”).
Vinyals teaches that the game replay data used to train the networks comprises “observations” sampled at a fixed step multiplier of 8 frames from the games:
There are many schemes one might employ to train networks to imitate human behaviour from replays. Here we use a simple approach that connects straightforwardly with the RL work in Section 4. When training our policy we sampled observations at a fixed step multiplier of 8 frames. We take the first action issued within each 8 frames as the learning target for the policy. If no action was taken during that period, we take the target to be a ‘no-op’, i.e., a special action which has no effect.
(Section 5.2 “Policy Predictions”).
Vinyals discloses that these sampled observations comprise two sets of feature layers: a minimap, which is a coarse representation of the state of the entire StarCraft II world; and a screen, which is a detailed view of a subset of the world corresponding to the player’s on-screen view, and in which most actions are executed:
Thus, the main observations come as sets of feature layers which are rendered at N × M pixels (where N and M are configurable, though in our experiments we always used N = M). Each of these layers represents something specific in the game, for example: unit type, hit points, owner, or visibility. Some of these (e.g., hit points, height map) are scalars, while others (e.g., visibility, unit type, owner) are categorical. There are two sets of feature layers: the minimap is a coarse representation of the state of the entire world, and the screen is a detailed view of a subsection of the world corresponding to the player’s on-screen view, and in which most actions are executed. Some features (e.g., owner or visibility) exist for both the screen and minimap, while others (e.g., unit type and hit points) exist only on the screen. See the environment documentation for a complete description of all observations provided.
(Section 3.2 “Observations;” footnote omitted).
Vinyals further discloses that the observations also comprise non-spatial observations provided by the human interface for the game, such as an amount of gas and minerals collected:
In addition to the screen and minimap, the human interface for the game provides various non-spatial observations. These include the amount of gas and minerals collected, the set of actions currently available (which depends on game context, e.g., which units are selected), detailed information about selected units, build queues, and units in a transport vehicle. These observations are also exposed by PySC2, and are fully described in the environment documentation. The audio channel is not exposed as a wave form but important notifications will be exposed as part of the observations.
(Section 3.2 “Observations”).
These observations sampled at each frame can be considered a “to-be-trained image” like claimed, from which is extracted a “to-be-trained feature set” comprising a first to be trained feature, a second to-be-trained feature and a third to be trained feature, wherein the first to-be-trained feature, i.e. the minimap, represents an image feature of a first region, the second to-be-trained feature, i.e. the extracted screen features, represents an image feature of a second region, and the third to-be-trained feature, i.e. the non-spatial features, represent an attribute feature related to an interaction operation, and wherein a range of the first region is smaller than a range of the second region. Moreover, Vinyals further discloses that RGB pixels could alternatively be used to train the neural networks:
This paper introduces an interface intended to make RL in StarCraft straightforward: observations and actions are defined in terms of low resolution grids of features; rewards are based on the score from the StarCraft II engine against the built-in computer opponent; and several simplified minigames are also provided in addition to the full game maps. Future releases will extend the interface for the full challenge of StarCraft II: observations and actions will expose RGB pixels; agents will be ranked by the final win/loss outcome in multi-player games; and evaluation will be restricted to full game maps used in competitive human play.
(Section 1 “Introduction”).
In future releases we will expose a rendered API allowing agents to play from RGB pixels. This will allow us to study the effects of learning from raw pixels versus learning from feature layers and make closer comparisons to human play. In the mean time, we played the game with feature layers to verify that agents are not severely handicapped. Though the game-play experience is obviously altered we found that a resolution of N, M ≥ 64 is sufficient to allow a human player to select and individually control small units such as Zerglings. The reader is encouraged to try this using pysc2 play.
(Section 3.2 “Observations;” footnote omitted).
The RGB pixels used to train the networks can alternatively be considered a to-be-trained image set like claimed, from which is extracted the above-noted first, second and third to-be-trained features.);
constructing a first to-be trained label related to a micro control task that corresponds to each to-be-trained image, the first to-be-trained label representing a hierarchical label related to operation content corresponding to the micro control task and including predicting a to-be-executed key first and then a release parameter of the to-be-executed key (As noted above, Vinyals teaches that the game replay data used to train the networks comprises “observations” sampled at a fixed step multiplier of 8 frames from the games. Vinyals further discloses that the first action issued within the set of 8 frames is taken as the learning target:
There are many schemes one might employ to train networks to imitate human behaviour from replays. Here we use a simple approach that connects straightforwardly with the RL work in Section 4. When training our policy we sampled observations at a fixed step multiplier of 8 frames. We take the first action issued within each 8 frames as the learning target for the policy. If no action was taken during that period, we take the target to be a ‘no-op’, i.e., a special action which has no effect.
(Section 5.2 “Policy Predictions”).
Vinyals further discloses that each action is represented as a composition of a function identifier and a sequence of arguments which that function identifier requires:
More formally, an action α is represented as a composition of a function identifier α0 and a sequence of arguments which that function identifier requires: α1, α2,…, αL. For instance, consider selecting multiple units by drawing a rectangle. The intended action is then select_rect(select_add,(x1, y1),(x2, y2)). The first argument select_add is binary. The other arguments are integers that define coordinates — their allowed range is the same as the resolution of the observations. This action is fed to the environment in the form [select_rect, [[select_add], [x1, y1], [x2, y2]]].
To represent the full action space we define approximately 300 action-function identifiers with 13 possible types of arguments (ranging from binary to specifying a point on the discretised 2D screen). See the environment documentation for a more detailed specification and description of the actions available through PySC2, and Figure 3 for an example of a sequence of actions.
In StarCraft, not all the actions are available in every game state. For example, the move command is only available if a unit is selected. Human players can see which actions are available in the “command card” on the screen. Similarly, we provide a list of available actions via the observations given to the agent at each step. Taking an action that is not available is considered an error, so agents should filter their action choices so that only legal actions are taken.
(Section 3.3 “Actions”).
The action taken as the learning target can be considered a first-to-be-trained label like claimed, which is related to a micro-control task that corresponds to a to-be-trained image, i.e. a frame. Because the action is represented as a composition of a function identifier (which can be considered a parent) and a sequence of arguments (which can be considered children of the parent), the action can be considered a “hierarchical label” like further claimed, which is related to operation content corresponding to the micro control task. Moreover, Vinyals demonstrates that the action can be indicative of a to-be-executed key first, e.g. a mouse key, and then a release parameter, e.g. a location, of the to-be-executed key:
PNG
media_image1.png
394
505
media_image1.png
Greyscale
(FIG. 3; emphasis added).
Accordingly, Vinyals teaches constructing a first to-be trained label, i.e. an action, related to a micro control task that corresponds to each to-be-trained image, the first to-be-trained label representing a hierarchical label related to operation content corresponding to the micro control task and including predicting a to-be-executed key first and then a release parameter of the to-be-executed key.); and
training the model at a level of a micro control task to obtain a target model parameter according to target scene data, a first to-be-trained label and a first predicted label, wherein the first predicted label represents a label that is obtained through prediction and that is related to operation content corresponding to the micro control task, the first predicted label being a predicted value, and the first to-be-trained label being a true value (Like noted above, Vinyals teaches employing replay data from games of StarCraft II to train networks to predict a game outcome and the action by the player at each timestep. As further noted above, the replay data particularly comprises screen, minimap and non-spatial features from frames of the StarCraft II game. Like further noted above, Vinyals discloses that actions in the game can be represented by a function identifier and a sequence of arguments. Vinyals teaches that an AI model is trained to predict a target action and its arguments based on the screen, minimap and non-spatial features observed from every 8th frame of the game replay, the target action being the first action issued within each set of 8 frames
The same network trained to predict values had a separate output designed to predict the action issued by the user. We sometimes refer to this part of the network as the policy since it can be readily deployed to play the game.
There are many schemes one might employ to train networks to imitate human behaviour from replays. Here we use a simple approach that connects straightforwardly with the RL work in Section 4. When training our policy we sampled observations at a fixed step multiplier of 8 frames. We take the first action issued within each 8 frames as the learning target for the policy. If no action was taken during that period, we take the target to be a ‘no-op’, i.e., a special action which has no effect.
When humans play StarCraft II, only a subset of all possible actions are available at any given time. For example, “building a marine” is enabled only if barracks are currently selected. Networks should not need to learn to avoid illegal actions since this information is readily available. Thus, during training, we filter out actions that would not be available to a human player. To do so, we take the union of all available actions for the past 8 frames and apply a mask that sets the probability of all unavailable actions to near zero.
Note that, as previously mentioned, we trained the policy to play all possible matchups. Thus, in principle, the agent can play any race. However, for consistency with the reinforcement learning agents studied in Section 4, we report in-game metrics in the single Terran versus Terran matchup.
Table 2 shows how different architectures perform in terms of accuracy at predicting the action identifier, the screen, and the minimap argument. As expected, both FullyConv and arFullyConv architectures perform much better for spatial arguments. As well, the arFullyConv architecture outperforms FullyConv, presumably because it knows which action identifier the argument will be used for.
(Section 5.2 “Policy Predictions”).
Atari-net Agent The first baseline is a simple adaptation of the architecture successfully used for the Atari [4] benchmark and DeepMind Lab environments [3]. It processes screen and minimap feature layers with the same convolutional network as in [21] — two layers with 16, 32 filters of size 8, 4 and stride 4, 2 respectively. The non-spatial features vector is processed by a linear layer with a tanh non-linearity. The results are concatenated and sent through a linear layer with a ReLU activation. The resulting vector is then used as input to linear layers that output policies over the action function id a0 and each action-function argument
a
I
I
=
0
L
independently. For spatial actions (screen or minimap coordinates) we independently model policies to select (discretised) x and y coordinates.
FullyConv Agent Convolutional networks for reinforcement learning (such as the Atari-net baseline above) usually reduce the spatial resolution of the input with each layer and ultimately finish with a fully connected layer that discards spatial structure completely. This allows spatial information to be abstracted away before actions are inferred. In StarCraft, though, a major challenge is to infer spatial actions (i.e. clicking on the screen and minimap). As these spatial actions act within the same space as the inputs, it might be detrimental to discard the spatial structure of the input.
Here we propose a fully convolutional network agent, which predicts spatial actions directly through a sequence of resolution-preserving convolutional layers. The network we propose has no stride and uses padding at every layer, thereby preserving the resolution of the spatial information in the input. For simplicity, we assume the screen and minimap inputs have the same resolution. We pass screen and minimap observations through separate 2-layer convolutional networks with 16, 32 filters of size 5 × 5, 3 × 3 respectively. The state representation is then formed by the concatenation of the screen and minimap network outputs, as well as the broadcast vector statistics, along the channel dimension. Note that this is likely non-optimal since the screen and minimap do not have the same spatial extent — future work could improve on this arrangement. To compute the baseline and policies over categorical (non-spatial) actions, the state representation is first passed through a fully-connected layer with 256 units and ReLU activations, followed by fully-connected linear layers. Finally, a policy over spatial actions is obtained using 1 × 1 convolution of the state representation with a single output channel. See Figure 4 for a visual representation of this computation.
(Section 4.3 “Agent Architectures”).
PNG
media_image2.png
276
541
media_image2.png
Greyscale
Figure 4 (emphasis added).
As supervised learning is employed to train the model – see e.g. Section 5 “Supervised Learning from Replays” – it is apparent that the model is trained by inputting the screen, minimap and non-spatial features into the model, which processes the features and outputs a predicted action, and whereby the predicted action is then compared with the learning target, i.e. the actual action issued within the frames, to adjust the weights of the model. Vinyals thus teaches training the model at a level of a micro control task, i.e. at the action level, to obtain a target model parameter, i.e. model weights, according to: (i) target scene data, i.e. the extracted screen, minimap and non-spatial features extracted at every 8th frame; (ii) a first to-be-trained label, i.e. the actual target action issued within the frames; and (iii) a first predicted label, i.e. the predicted action output by model based on the extracted screen, minimap and non-spatial features. Here, the first predicted label represents a label that is obtained through prediction, i.e. by the model, and that is related to operation content corresponding to a micro control task, the first predicted label being a predicted value, i.e. a predicted action, and the first to-be-trained label being a true value, i.e. the actual target action.).
It would have been obvious to one of ordinary skill in the art, having the teachings of Somers and Vinyals before the effective filing date of the claimed invention, to train the combined model taught by Somers based on replays of StarCraft II like taught by Vinyals, wherein a to-be-trained feature set is extracted from each to-be-trained image in the replay data, the to-be-trained feature set comprising a first to-be-trained feature, a second to-be-trained feature and a third to-be-trained feature, the first to-be-trained feature representing an image feature of a first region, the second to-be-trained feature representing an image feature of a second region, and the third to-be-trained feature representing an attribute feature related to an interaction operation, wherein a range of the first region is smaller than a range of the second region, wherein the neural network model is trained to obtain the combined model according to the to-be-trained feature set, and wherein the first to-be-trained label related to each micro control task represents a hierarchical label related to operation content corresponding to the micro control task and can include a to-be-executed key first and then a release parameter of the to-be-executed key. It would have been advantageous to one of ordinary skill to utilize such a combination because StarCraft II is a popular game, as is suggested by Vinyals (see e.g. section 1 “Introduction,” which recites that “[o]ver the past two decades, StarCraft I and StarCraft II have been pioneering and enduring e-sports, with millions of casual and highly competitive professional players.”). Additionally, it would have been obvious to one of ordinary skill in the art, having the teachings of Somers and Vinyals before the effective filing date of the claimed invention, to further modify the method, apparatus and non-transitory computer-readable storage medium taught by Somers such that supervised learning like taught by Vinyals is employed when training the same combined model only at the level of the micro control task, i.e. without training at a level of the big picture task, with the new training data, whereby a target model parameter is obtained according to the target scene data, the first to-be-trained label, and a first predicted label, wherein the first predicted label represents a label that is obtained through prediction and that is related to the operation content corresponding to the micro control task, and the first predicted label is a predicted value and the first to-be-trained label is a true value. It would have been advantageous to one of ordinary skill to utilize such a combination because supervised learning has been successful in other applications, as is taught by Vinyals (see e.g. section 5 “Supervised Learning from Replays,” which states that “[t]he use of supervised data such as replays or human demonstrations has been successful in robotics, the game of Go, and Atari” (internal citations omitted)). Accordingly, Somers and Vinyals are considered to teach a method like that of claim 1, an apparatus like that of claim 8 and a non-transitory computer-readable storage medium like that of claim 15.
As per claims 2, 9 and 16, it would have been obvious, as is described above, to train the combined model taught by Somers based on replays of StarCraft II like taught by Vinyals, wherein a to-be-trained feature set is extracted from each to-be-trained image in the replay data, the to-be-trained feature set comprising a first to-be-trained feature, a second to-be-trained feature and a third to-be-trained feature. Vinyals particularly teaches:
the first to-be-trained feature is a two-dimensional vector feature, and the first to-be-trained feature comprises at least one of character position information, moving object position information, fixed object position information, or defensive object position information in the first region (As noted above, the minimap features taught by Vinyals that are extracted for each frame of replay data used to train the above-described networks are considered a “first to-be-trained feature” like claimed. Vinyals teaches that the minimap features are provided as a set of feature layers, which are each rendered at N x M pixels, and represent features of the minimap such as character position information, e.g. player identity, or fixed object position information, e.g. terrain height:
Thus, the main observations come as sets of feature layers which are rendered at N × M pixels (where N and M are configurable, though in our experiments we always used N = M). Each of these layers represents something specific in the game, for example: unit type, hit points, owner, or visibility. Some of these (e.g., hit points, height map) are scalars, while others (e.g., visibility, unit type, owner) are categorical. There are two sets of feature layers: the minimap is a coarse representation of the state of the entire world, and the screen is a detailed view of a subsection of the world corresponding to the player’s on-screen view, and in which most actions are executed. Some features (e.g., owner or visibility) exist for both the screen and minimap, while others (e.g., unit type and hit points) exist only on the screen. See the environment documentation for a complete description of all observations provided.
(Section 3.2 “Observations;” footnote omitted).
PNG
media_image3.png
386
612
media_image3.png
Greyscale
Figure 2 (emphasis added).
Accordingly, the minimap features, i.e. “first-to-be-trained feature,” comprises a two-dimensional vector, i.e. an N x M feature layer, that comprises at least one of character position information, moving object position information, fixed object position information, or defensive object position information in the minimap.);
the second to-be-trained feature is a two-dimensional vector feature, and the second two-be-trained feature comprises at least one of character position information, moving object position information, fixed object position information, defensive object position information, obstacle position information, or output object position information in the second region (As described above, the screen features taught by Vinyals that are extracted for each frame of replay data used to train the above-described networks are considered a “second to-be trained feature” like claimed. Vinyals teaches that the screen features are provided as a set of feature layers, which are each rendered at N x M pixels, and represent features of the screen such as character position information, e.g. player identity, or fixed object position information, e.g. terrain height:
Thus, the main observations come as sets of feature layers which are rendered at N × M pixels (where N and M are configurable, though in our experiments we always used N = M). Each of these layers represents something specific in the game, for example: unit type, hit points, owner, or visibility. Some of these (e.g., hit points, height map) are scalars, while others (e.g., visibility, unit type, owner) are categorical. There are two sets of feature layers: the minimap is a coarse representation of the state of the entire world, and the screen is a detailed view of a subsection of the world corresponding to the player’s on-screen view, and in which most actions are executed. Some features (e.g., owner or visibility) exist for both the screen and minimap, while others (e.g., unit type and hit points) exist only on the screen. See the environment documentation for a complete description of all observations provided.
(Section 3.2 “Observations;” footnote omitted).
Accordingly, the screen features, i.e. “second-to-be-trained feature,” comprises a two-dimensional vector, i.e. an N x M feature layer, that comprises at least one of character position information, moving object position information, fixed object position information, defensive object position information, obstacle object position information, or output object position information in the screen corresponding to the player’s on-screen view);
the third to-be-trained feature is a one-dimensional vector feature, and the third to-be-trained feature comprises at least one of a character hit point value, a character output value, time information, or score information (As described above, the non-spatial features that are extracted for each frame of replay data used to train the above-described networks are considered a “third to-be trained feature” like claimed. Vinyals teaches that the non-spatial features are provided via a one-dimensional vector, as Vinyals discloses that “[t]he non-spatial features vector is processed by a linear layer with a tanh non-linearity” – see section 4.3 “Agent Architectures.” Figure 4, which is presented above, also demonstrates that the non-spatial features are input into the neural networks as a one-dimensional vector. Vinyals discloses that the non-spatial features can comprise score information, e.g. an amount of gas and minerals collected:
In addition to the screen and minimap, the human interface for the game provides various non-spatial observations. These include the amount of gas and minerals collected, the set of actions currently available (which depends on game context, e.g., which units are selected), detailed information about selected units, build queues, and units in a transport vehicle. These observations are also exposed by PySC2, and are fully described in the environment documentation. The audio channel is not exposed as a wave form but important notifications will be exposed as part of the observations.
(Section 3.2 “Observations”).
Accordingly, the non-spatial features, i.e. “third to-be-trained feature,” is a one-dimensional vector feature, and the third to-be-trained feature comprises at least one of a character hit point value, a character output value, time information, or score information, e.g. an amount of gas and minerals collected.); and
correspondence relationship exists between the first to-be-trained feature, the second to-be-trained feature, and the third to-be-trained feature (As noted above: the minimap features extracted for each frame of replay data used to train the networks are considered a “first to-be-trained feature” like claimed; the extracted screen features are considered a “second to-be trained feature” like claimed; and the extracted non-spatial features are considered a “third to-be trained feature” like claimed. As these observations represent the state of the game during a particular timestep during gameplay – see section 3.2 “Observations” and section 5.1 “Value Predictions” – they can be considered to correspond to each other.).
Accordingly, the above-described combination of Somers and Vinyals is further considered to teach a method like that of claim 2, an apparatus like that of claim 9, and a non-transitory computer-readable storage medium like that of claim 16.
As per claims 3, 10 and 17, Somers teaches that the first to-be-trained label (i.e. human control input) comprises at least one of key type information or key parameter information, wherein the key parameter information comprises at least one of a direction-type parameter, a position-type parameter, or a target-type parameter, and wherein the direction-type parameter represents a moving direction of a character, the position-type parameter represents a position of a character, and the target-type parameter represents a to-be-targeted object of the character (see e.g. paragraphs 0069-0070 and 0083). As described above, it would have been obvious to train the combined model taught by Somers based on replays of StarCraft II like taught by Vinyals. Vinyals also teaches that a first to-be-trained label corresponding to a to-be-trained image can comprise at least one of key type information or key parameter information, wherein the key parameter information comprises at least one of a direction-type parameter, a position-type parameter, or a target-type parameter, and wherein the direction-type parameter represents a moving direction of a character, the position-type parameter represents a position of a character, and the target-type parameter represents a to-be-targeted object of the character (As noted above, Vinyals teaches that the game replay data used to train the networks comprises screen, minimap and non-spatial features sampled at a fixed step multiplier of 8 frames from the games, and wherein the first action issued within the set of 8 frames is taken as the learning target:
There are many schemes one might employ to train networks to imitate human behaviour from replays. Here we use a simple approach that connects straightforwardly with the RL work in Section 4. When training our policy we sampled observations at a fixed step multiplier of 8 frames. We take the first action issued within each 8 frames as the learning target for the policy. If no action was taken during that period, we take the target to be a ‘no-op’, i.e., a special action which has no effect.
(Section 5.2 “Policy Predictions”).
As also noted above, Vinyals further discloses that each action is represented as a composition of a function identifier and a sequence of arguments. The function identifier and arguments of the target action obtained for the frames of replay data can be considered a “first to-be-trained label” like claimed. In particular, the function identifier corresponds to particular key or mouse inputs – see e.g. section 3.3 “Actions” and Figure 3 – and can thus be considered “key type information” like claimed. The arguments can be considered “key parameter information” like claimed.). Accordingly, the above-described combination of Somers and Vinyals is further considered to teach a method like that of claim 3, an apparatus like that of claim 10, and a non-transitory computer-readable storage medium like that of claim 17.
As per claims 4, 11 and 18, Somers teaches that the second to-be-trained label (i.e. bin label) comprises at least one of operation intention information or character position information, wherein the operation intention information represents an intention (e.g. offense, defense) with which a character interacts with an object, and the character position information represents a position of the character in the first region (see e.g. paragraphs 0084 and 0111-0115). Accordingly, the above-described combination of Somers and Vinyals is further considered to teach a method like that of claim 4, an apparatus like that of claim 11, and a non-transitory computer-readable storage medium like that of claim 18.
As per claims 5, 12 and 19, it would have been obvious, as is described above, to train the combined model taught by Somers using supervised learning based on replays of StarCraft II like taught by Vinyals, wherein a to-be-trained feature set is extracted from each to-be-trained image in the replay data, the to-be-trained feature set comprising a first to-be-trained feature, a second to-be-trained feature and a third to-be-trained feature. Vinyals further teaches that training the neural network model to obtain the combined model according to the to-be-trained feature set in each to-be-trained image and the first to-be-trained label related to the micro control task can include:
processing the to-be-trained feature set in the each to-be-trained image to obtain a target feature set comprising a first target feature, a second target feature, and a third target feature (As noted above, the minimap features extracted for each frame of replay data used to train the networks are considered a “first to-be-trained feature” like claimed; the extracted screen features are considered a “second to-be trained feature” like claimed; and the extracted non-spatial features are considered a “third to-be trained feature” like claimed. Vinyals teaches that all of these features, i.e. the to-be-trained feature set, for each frame of replay data are pre-processed before being input into the neural networks described above:
Input pre-processing All the baseline agents share the same pre-processing of input feature layers. We embed all feature layers containing categorical values into a continuous space, which is equivalent to using a one-hot encoding in the channel dimension followed by a 1 × 1 convolution. We also re-scale numerical features with a logarithmic transformation as some of them such as hit-points or minerals might attain substantially high values.
(Section 4.3 “Agent Architectures”).
The results of such pre-processing can be considered a target feature set comprising a first target feature, a second target feature and a third target feature, i.e. the pre-processed minimap, screen and non-spatial features, respectively. Vinyals further teaches that each of the neural networks additionally processes the features input thereto to obtain corresponding target features:
Atari-net Agent The first baseline is a simple adaptation of the architecture successfully used for the Atari [4] benchmark and DeepMind Lab environments [3]. It processes screen and minimap feature layers with the same convolutional network as in [21] — two layers with 16, 32 filters of size 8, 4 and stride 4, 2 respectively. The non-spatial features vector is processed by a linear layer with a tanh non-linearity. The results are concatenated and sent through a linear layer with a ReLU activation. The resulting vector is then used as input to linear layers that output policies over the action function id a0 and each action-function argument
a
I
I
=
0
L
independently. For spatial actions (screen or minimap coordinates) we independently model policies to select (discretised) x and y coordinates.
FullyConv Agent Convolutional networks for reinforcement learning (such as the Atari-net baseline above) usually reduce the spatial resolution of the input with each layer and ultimately finish with a fully connected layer that discards spatial structure completely. This allows spatial information to be abstracted away before actions are inferred. In StarCraft, though, a major challenge is to infer spatial actions (i.e. clicking on the screen and minimap). As these spatial actions act within the same space as the inputs, it might be detrimental to discard the spatial structure of the input.
Here we propose a fully convolutional network agent, which predicts spatial actions directly through a sequence of resolution-preserving convolutional layers. The network we propose has no stride and uses padding at every layer, thereby preserving the resolution of the spatial information in the input. For simplicity, we assume the screen and minimap inputs have the same resolution. We pass screen and minimap observations through separate 2-layer convolutional networks with 16, 32 filters of size 5 × 5, 3 × 3 respectively. The state representation is then formed by the concatenation of the screen and minimap network outputs, as well as the broadcast vector statistics, along the channel dimension. Note that this is likely non-optimal since the screen and minimap do not have the same spatial extent — future work could improve on this arrangement. To compute the baseline and policies over categorical (non-spatial) actions, the state representation is first passed through a fully-connected layer with 256 units and ReLU activations, followed by fully-connected linear layers. Finally, a policy over spatial actions is obtained using 1 × 1 convolution of the state representation with a single output channel. See Figure 4 for a visual representation of this computation.
(Section 4.3 “Agent Architectures”).
The features resulting from such further processing of the minimap, screen and non-spatial features input to the network can alternatively be considered a first target feature, a second target feature, and a third target feature, respectively.);
obtaining a first predicted label that corresponds to the target feature set by using a long short-term memory (LSTM) layer, the first predicted label representing a label that is obtained through prediction and that is related to operation content (Like noted above, Vinyals teaches employing replay data from games of StarCraft II to train networks to predict a game outcome and the action by the player at each timestep:
In the rest of this section, we provide baselines using the architectures described in Section 4, but using a set of 800K games to learn both a value function (i.e., predicting the winner of the game from game observations), and a policy (i.e., predicting the action taken from game observations). The games contain all possible matchups in StarCraft II (i.e., we do not restrict the agent to play a single race).
(Section 5 “Supervised Learning from Replays”).
We train dual-headed networks that predict both the game outcome (1 = win vs. 0 = loss or tie), and the action taken by the player at each time step. Sharing the body of the network makes it necessary to balance the weights for the two loss functions, but it also allows value and policy predictions to inform one another. We did not make ties a separate game outcome class in the supervised training setup, since the number of ties in the dataset is very low (< 1%) compared to victory and defeat.
(Section 5 “Supervised Learning from Replays”).
Vinyals teaches that the action predicted by the network comprises both a function identifier and one or more arguments:
The networks proposed in Section 4 produce the action identifier and its arguments independently. However, the accuracy of predicting a point on the screen can be improved by conditioning on the base action, e.g., building an extra base versus moving an army. Thus, in addition to the Atari-net and FullyConv architecture, we have arFullyConv which uses the auto-regressive policy introduction introduced in Section 4.2, i.e. using the function identifier α0 and previously sampled arguments α<ᶩ to model a policy over the current argument αᶩ.
(Section 5.1 “Value Predictions”).
The function identifier and/or arguments predicted by the network is considered a “first predicted label” like claimed, which represents a label that is obtained through prediction and that is related to the operation content. Vinyals particularly teaches in one embodiment the predictions can be made using an LSTM layer:
FullyConv LSTM Agent Both of the above baselines are feed-forward architectures and therefore have no memory. While this is sufficient for some tasks, we cannot expect it to be enough for the full complexity of StarCraft. Here we introduce a baseline architecture based on a convolutional LSTM. We follow the fully-convolutional agent’s pipeline described above and simply add a convolutional LSTM module after the minimap and screen feature channels are concatenated with the non-spatial features.
(Section 4.3 “Agent Architectures”).
Accordingly, Vinyals teaches obtaining a first predicted label, i.e. a predicted function identifier and/or arguments, that correspond to the target feature set, i.e. the processed minimap, screen and non-spatial features, by using an LTSM layer, the first predicted label representing a label that is obtained through prediction and that is related to operation content.).
obtaining a model core parameter through training according to the first predicted label and a first to-be-trained label, the first predicted label being a predicted value, and the first to-be-trained label being a true value (Like described above, Vinyals teaches employing replay data from games of StarCraft II to train networks to predict the action by the player at each timestep. Particularly, like also described above, the replay data comprises to-be-trained images from which to-be-trained feature sets are extracted, i.e. sets of screen, minimap and non-spatial features from frames of the StarCraft II game:
Networks are trained for 200k steps of gradient descent on all possible match-ups in StarCraft II. We trained with mini-batches of 64 observations taken at random from all replays uniformly across time. Observations are sampled with a step multiplier of 8, consistent with the RL setup. The resolution of both screen and minimap is 64 × 64. Each observation consists of the screen and minimap spatial feature layers as well as player stats such as food cap and number of collected minerals that human players see on the screen. We use 90% of the replays as training set, and a fixed test set of 0.5M frames drawn from the rest of the 10% of the replays. The agent performance is evaluated continuously against this test set as training progresses.
(Section 5.1 “Value Predictions).
Moreover, as further described above, Vinyals teaches that the frames are also associated with a learning target, i.e. an action issued within the frames, which is considered a true value:
There are many schemes one might employ to train networks to imitate human behaviour from replays. Here we use a simple approach that connects straightforwardly with the RL work in Section 4. When training our policy we sampled observations at a fixed step multiplier of 8 frames. We take the first action issued within each 8 frames as the learning target for the policy. If no action was taken during that period, we take the target to be a ‘no-op’, i.e., a special action which has no effect.
(Section 5.2 “Policy Predictions”).
The target function identifier and/or arguments of the action can be considered a “first to-be-trained label” like claimed. Because supervised learning is employed to train the neural networks – see e.g. Section 5 “Supervised Learning from Replays” – it is apparent that the training entails, for each frame of gameplay used to train the network: inputting the corresponding to-be-trained feature set into the neural network, which processes the to-be-trained feature set and thereby outputs a predicted action, i.e. the first predicted label, and whereby the predicted action is compared with the true action of the learning target, i.e. the first to-be-trained label, to adjust the weights of the neural network. The weight(s) obtained through such training are considered a core model parameter like claimed.); and
and generating a combined model according to the core model parameter (As described above, Vinyals teaches that each network used to predict play actions is trained by: inputting a to-be-trained feature set into the neural network, which processes the to-be-trained feature set and thereby outputs a predicted action, i.e. the first and second predicted labels, and whereby the predicted action is compared with the true action of the learning target, i.e. the first and second to-be-trained labels, to adjust the weights, i.e. a model core parameter, of the neural network. Consequently, the combined model – the trained neural network – is generated according to the model core parameter.).
Like noted above, Somers further teaches that the combined model includes a strategy AI model trained to select one of the other AI models (represented by a bin label indicating a game state) based on a current game state (e.g. based on offense or defense) (see e.g. paragraphs 0061 and 0116). It follows that the strategy AI model of the combined model would similarly be trained by the above-noted supervised learning process, i.e. by: (i) processing the to-be-trained feature set in the each to-be-trained image to obtain the target feature set, the target feature set comprising the first target feature, the second target feature and the third target feature; (ii) obtaining a second predicted label (i.e. a predicted game state/AI model) that corresponds to the target feature set by using the LSTM layer, the second predicted label representing a label that is obtained through prediction and that is related to the operation intention; (iii) obtaining a model core parameter through training according to the second predicted label and the second to-be-trained label (i.e. actual game state) of each to-be-trained image, the second predicted label being a predicted value, and the second to-be-trained label being a true value; and (iv) generating the combined model according to the model core parameter. Accordingly, Somers and Vinyals are further considered to teach a method like that of claim 5, an apparatus like that of claim 12, and a non-transitory computer-readable storage medium like that of claim 19.
Claims 6, 13 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Somers and Vinyals, which is described above, and also over U.S. Patent Application Publication No. 2020/0051211 to Shiokawa et al. (“Shiokawa”).
As described above, Somers and Vinyals teach a method like that of claim 5, an apparatus like that of claim 12, and a non-transitory computer-readable storage medium like that of claim 19, which entail processing a to-be-trained feature set extracted from an image – the to-be-trained feature set comprising first, second and third to-be-trained features – to obtain a target feature set comprising first, second and third target features. Like in each of claims 6, 13, and 20, Vinyals further suggests: (i) processing the third to-be-trained feature (i.e. non-spatial features) in the each to-be-trained image by using a fully connected layer (e.g. by concatenating with screen and minimap network outputs to form a state representation and inputting to a fully connected layer) to obtain the third target feature; (ii) processing the second to-be-trained feature (i.e. screen features) in the each to-be-trained image by using a convolutional layer to obtain the second target feature; and (iii) processing the first to-be-trained feature (i.e. minimap features) in the each to-be-trained image by using the convolutional layer to obtain the first target feature:
Here we propose a fully convolutional network agent, which predicts spatial actions directly through a sequence of resolution-preserving convolutional layers. The network we propose has no stride and uses padding at every layer, thereby preserving the resolution of the spatial information in the input. For simplicity, we assume the screen and minimap inputs have the same resolution. We pass screen and minimap observations through separate 2-layer convolutional networks with 16, 32 filters of size 5 × 5, 3 × 3 respectively. The state representation is then formed by the concatenation of the screen and minimap network outputs, as well as the broadcast vector statistics, along the channel dimension. Note that this is likely non-optimal since the screen and minimap do not have the same spatial extent — future work could improve on this arrangement. To compute the baseline and policies over categorical (non-spatial) actions, the state representation is first passed through a fully-connected layer with 256 units and ReLU activations, followed by fully-connected linear layers. Finally, a policy over spatial actions is obtained using 1 × 1 convolution of the state representation with a single output channel. See Figure 4 for a visual representation of this computation.
(Section 4.3 “Agent Architectures”).
Somers and Vinyals, however, do not explicitly disclose that the first, second and third target features are one-dimensional vector features, as is required by claims 6, 13 and 20.
Shiokawa nevertheless generally teaches processing (e.g. unfolding) features output by a network layer (e.g. a convolutional layer) so as to become a one-dimensional vector feature (see e.g. paragraph 0097).
It would have been obvious to one of ordinary skill in the art, having the teachings of Somers, Vinyals and Shiokawa before him prior to the effective filing date of the claimed invention, to modify the network architecture taught by Somers and Vinyals such that the features (e.g. the first, second and third target features) output by the network layers are processed so as to become one-dimensional vector features like taught by Shiokawa. It would have been advantageous to one of ordinary skill to utilize such one-dimensional vector features because it can facilitate further processing of the features by the network architecture, as is suggested by Shiokawa (see e.g. paragraph 0097). Accordingly, Somers, Vinyals and Shiokawa are considered to teach, to one of ordinary skill in the art, a method like that of claim 6, an apparatus like that of claim 13, and a non-transitory computer-readable storage medium like that of claim 20.
Response to Arguments
The Examiner acknowledges the Applicant’s amendments to claims 1, 5, 8, 12, 15 and 19. Regarding the 35 U.S.C. § 103 rejections, the Applicant argues that none of the cited references teaches the feature of “training, by the device, the same combined model only at a level of the micro control task without training at a level of the big picture task to obtain a target model parameter according to the target scene data,” as is now recited in each of independent claims 1, 8 and 15.
The Examiner, however, respectfully disagrees. Like described above, Somers describes an AI model training system that trains a plurality of AI models to be executed under different game states, and that trains a strategy AI model that selects one of these AI models for execution based on a current game state (see e.g. paragraphs 0061, 0112-0114 and 0116-0117). The combination of the AI models that are executed under different game states and the strategy AI model is considered a “combined model” like claimed.
The models that execute under different game states are trained to generate one or more controls, e.g. directional inputs and/or button presses, that are similar to the controls that a human might provide during particular gameplay scenarios (see e.g. paragraphs 0112-0114 and 0128-0129). Accordingly, such models correspond to micro control tasks, e.g. to the different control inputs. As noted, the strategy AI model is trained to identify a current game state (e.g. offense or defense in a hockey game) and select one of the other models to execute based on the current game state (see e.g. paragraphs 0061 and 0116). The strategy AI model thus corresponds to big picture tasks.
Somers discloses that after the strategy AI model and the plurality of AI models associated with different game states are trained, the models are evaluated, and only a selected one or more of the models (i.e. models that are identified as ineffective) are further trained with additional, targeted training data (see e.g. paragraphs 0118-0119). For example, a model for offense in a hockey game can be trained with additional training data if the model is deemed ineffective (see e.g. paragraph 0119). Accordingly, in such circumstances, the combined model is trained only at a level of the micro control task, i.e. by training only one or more of the AI models that are executed in different game states, without training at a level of the big picture task, i.e. without training the strategy AI model. The Examiner thus respectfully maintains that Somers teaches “training, by the device, the same combined model only at a level of the micro control task without training at a level of the big picture task to obtain a target model parameter according to the target scene data,” as is now claimed.
The Applicant’s arguments filed on December 30, 2025 have thus been fully considered, but are not persuasive.
Conclusion
Applicant's amendment necessitated any new grounds of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BLAINE T BASOM whose telephone number is (571)272-4044. The examiner can normally be reached Monday-Friday, 9:00 am - 5:30 pm, EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached at (571)270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BTB/
5/8/2026
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights