DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/29/2022, 02/12/2025, and 08/26/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments/Remarks
101 Arguments
Applicant asserts:
Applicant argues, on page 12-15, that that neither "training, by the computing system and based on the plurality of graph-based representations, a graph neural network model to generate an action prediction for each player in each frame of the tracking data, wherein the graph neural network model includes a multi-head self- attention neural network model, and wherein the training the graph neural network includes: generating, using the multi-head self-attention neural network model, a plurality of adjacency matrices based on one or more nodes of a sub-graph of the plurality of graph-based representations for each frame of the tracking data," as recited in claim 1, nor "generating, by the computing system and based on the plurality of target graph-based representations, an action prediction for each player in each target frame using the multi-head self-attention neural network model of the trained graph neural network model," as claim 1 further recites, can be performed in the human mind.
Examiner response:
Examiner respectfully disagrees and notes that “training, by the computing system and based on the plurality of graph-based representations, a graph neural network model to generate an action prediction for each player in each frame of the tracking data, wherein the graph neural network model includes a multi-head self- attention neural network model,” and “using the multi-head self-attention neural network model of the trained graph neural network model,” is not interpreted as an abstract idea, but as merely applying a generic computing component to implement mental steps with generating an action prediction. The claims do not describe how the act of “generating a plurality of adjacency matrices based on one or more nodes of a sub-graph” and “generating an action prediction for each player”. Examiner notes that the claims do not describe what occurs to the generating of adjacency matrices and action predictions in a way that would differentiate this process from the way a person could perform these actions in their mind. Applicant’s argument is therefore considered unpersuasive.
Applicant asserts:
Applicant argues, on page 15, that the "heterogenous structure [(e.g., the heterogenous graph of claim 1)] provides an improvement over conventional approaches by both decoupling the spatial domain from the temporal domain, as well as simplifying [an] overall learning process for identifying interactions between entities."
Examiner response:
Examiner respectfully disagrees and notes that MPEP 2106.04(d)(1) states “if the specification explicitly sets forth an improvement but in a conclusory manner (i.e., a bare assertion of an improvement without the detail necessary to be apparent to a person of ordinary skill in the art), the examiner should not determine the claim improves technology.” Applicant’s argument regarding the heterogenous structure does not show how it is improving the neural network model itself and is a bare assertion of improvement. Applicant’s argument is therefore considered unpersuasive.
103 Arguments
Applicant asserts:
Applicant argues, on page 15-16, that the amendments would overcome the rejection, so the rejection should be removed. Specifically, the prior art does not teach “one or more temporal edges that represent one or more connections of each entity to a same respective entity across the plurality of frames.”
Examiner response:
Examiner respectfully disagrees. Arguments are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. An updated 103 rejection is shown below.
Claims 2-7 depend directly or indirectly to from claim 1, and are therefore rejected for at least the same reasons.
Claims 8, while of different scope, includes features of claim 1 that have been rejected under prior art. Therefore, claim 8 is rejected as well.
Claims 9-14 depend directly or indirectly to from claim 8, and are therefore rejected for at least the same reasons.
Claim 15, while of different scope, includes features of claim 1 that have been rejected under prior art. Therefore, claim 15 is rejected as well.
Claims 16-20 depend directly or indirectly to from claim 8, and are therefore rejected for at least the same reasons. The prior art rejections have been updated to include the amended limitations and to clarify the reasoning given for the limitations that were not amended.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
In reference to claim 1:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a process
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“converting, by the computing system, the tracking data into a plurality of graph-based representations, wherein at least one of the plurality of graph-based representations includes a heterogenous graph including one or more nodes representing one or more entities, one or more temporal edges, or one or more spatial edges, wherein the one or more temporal edges represent one or more connections of each entity to a same respective entity across the plurality of frames;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could convert the tracking data into a plurality of graph-based representations including one or more nodes representing one or more entities, one or more temporal edges, or one or more spatial edges, wherein the one or more temporal edges represent one or more connections of each entity to a same respective entity across the plurality of frames by drawing them out.
“and wherein the training the graph neural network includes: generating, using the multi-head self-attention neural network model, a plurality of adjacency matrices based on one or more nodes of a sub-graph of the plurality of graph-based representations for each frame of the tracking data;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could generate a plurality of adjacency matrices based on one or more nodes of a sub-graph of the plurality of graph-based representations for each frame of the tracking data.
“converting, by the computing system, the target tracking data to a plurality of target graph-based representations, wherein each of the plurality of target graph-based representations corresponds to a target frame of the plurality of target frames and includes one or more target nodes, one or more target temporal edges, [[and]]or one or more target spatial edges;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could convert the target tracking data into a plurality of graph-based representations including nodes, temporal edges, and spatial edges by drawing them out.
“and generating, by the computing system and based on the plurality of target graph-based representations, an action prediction for each player in each target frame [using the multi-head self-attention neural network model of the trained graph neural network model.]” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could mentally generate an action prediction for each player in each target frame.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
“retrieving, by a computing system, tracking data from a data store, the tracking data comprising a plurality of frames of data for a plurality of events across a plurality of seasons;” (insignificant extra-solution activity mere data gathering MPEP 2106.05(g))
“training, by the computing system and based on the plurality of graph-based representations, a graph neural network model to generate an action prediction for each player in each frame of the tracking data, wherein the graph neural network model includes a multi-head self-attention neural network model,” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“receiving, by the computing system, target tracking data for a target event, the target tracking data comprising a plurality of target frames;” (insignificant extra-solution activity mere data gathering MPEP 2106.05(g))
“using the multi-head self-attention neural network model of the trained graph neural network model.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
The claim does not include additional elements that are integrated into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
“retrieving, by a computing system, tracking data from a data store, the tracking data comprising a plurality of frames of data for a plurality of events across a plurality of seasons;” (well-understood, routine, conventional MPEP 2106.05(d))
“training, by the computing system and based on the plurality of graph-based representations, a graph neural network model to generate an action prediction for each player in each frame of the tracking data, wherein the graph neural network model includes a multi-head self-attention neural network model,” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“receiving, by the computing system, target tracking data for a target event, the target tracking data comprising a plurality of target frames;” (well-understood, routine, conventional MPEP 2106.05(d))
“using the multi-head self-attention neural network model of the trained graph neural network model.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
In reference to claim 2:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a process
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The method of claim 1, wherein the graph neural network model comprises a spatial dynamic graph generation network configured to update the plurality of target graph-based representations with spatial interaction data among players.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could physically draw spatial interaction among players.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 3:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a process
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The method of claim 2, wherein the spatial dynamic graph generation neural network model comprises the multi-head self-attention network, and wherein the multi-head self-attention neural network model comprises a plurality of heads, wherein each head of a plurality of heads corresponds to a respective action of a plurality of actions for classification.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could mentally classify a respective action of a plurality of actions.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 4:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a process
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The method of claim 3, wherein each head of the plurality of heads is configured to generate a respective adjacency matrix, and wherein, when at least two consecutive frames of the plurality of frames have a same set of entities, an adjacency matrix of the plurality of adjacency matrices is diagonal.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could physically draw an adjacency matrix.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 5:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a process
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The method of claim 2, wherein training the graph neural network model to generate the action prediction for each player in each frame of the tracking data, comprises: learning spatial relationships between each player in each frame of the tracking data;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could mentally learn the spatial relationships between each player.
“and learning a plurality of neural network weights.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could mentally learn the neural network weights.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 6:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a process
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The method of claim 2, wherein training the graph neural network model to generate the action prediction for each player in each frame of the tracking data, comprises: extracting temporal features from the tracking data” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could observe the tracking data and extract temporal features.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 7:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a process
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The method of claim 1, wherein training the graph neural network model to generate the action prediction for each player in each frame of the tracking data, comprises: learning to generate a probability distribution across each of a plurality of action classes for each player in each frame.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could mentally learn to make a probability distribution across all possible action classes.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 8:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a manufacture
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“converting the tracking data into a plurality of graph-based representations, wherein at least one of the plurality of graph-based representations includes a heterogenous graph including one or more nodes representing one or more entities, one or more temporal edges, or one or more spatial edges, wherein the one or more temporal edges represent one or more connections of each entity to a same respective entity across the plurality of frames;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could convert the tracking data into a plurality of graph-based representations one or more entities, one or more temporal edges, or one or more spatial edges, wherein the one or more temporal edges represent one or more connections of each entity to a same respective entity across the plurality of frames.
“and wherein the training the graph neural network model includes: generating, using the multi-head self- attention neural network model, a plurality of adjacency matrices based on one or more nodes of a sub-graph of the plurality of graph-based representations for each frame of the tracking data;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could generate a plurality of adjacency matrices based on nodes of a sub-graph, for each frame of the tracking data.
“converting the target tracking data to a plurality of target graph-based representations, wherein each of the plurality of target graph-based representations corresponds to a target frame of the plurality of target frames and includes one or more target nodes, one or more target temporal edges, or one or more target spatial edges;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could convert the target tracking data into a plurality of graph-based representations including nodes, temporal edges, and spatial edges by drawing them out.
“and generating, based on the plurality of target graph-based representations, an action prediction for each player in each target frame [using the multi-head self-attention neural network model of the trained graph neural network model.]” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could mentally generate an action prediction for each player in each target frame.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
“A system, comprising: a processor; and a memory having programming instructions stored thereon, which, when executed by the processor, causes the system to perform one or more operations, comprising:” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“retrieving tracking data from a data store, the tracking data comprising a plurality of frames of data for a plurality of events across a plurality of seasons;” (insignificant extra-solution activity mere data gathering MPEP 2106.05(g))
“training, based on the plurality of graph-based representations, a graph neural network model to generate an action prediction for each player in each frame of the tracking data, wherein the graph neural network model includes a multi-head self-attention neural network model,” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“receiving target tracking data for a target event, the target tracking data comprising a plurality of target frames;” (insignificant extra-solution activity mere data gathering MPEP 2106.05(g))
“using the multi-head self-attention neural network model of the trained graph neural network model.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
The claim does not include additional elements that are integrated into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
“A system, comprising: a processor; and a memory having programming instructions stored thereon, which, when executed by the processor, causes the system to perform one or more operations, comprising:” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“retrieving tracking data from a data store, the tracking data comprising a plurality of frames of data for a plurality of events across a plurality of seasons;” (well-understood, routine, conventional MPEP 2106.05(d))
“training, based on the plurality of graph-based representations, a graph neural network model to generate an action prediction for each player in each frame of the tracking data, wherein the graph neural network model includes a multi-head self-attention neural network model,” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“receiving target tracking data for a target event, the target tracking data comprising a plurality of target frames;” (well-understood, routine, conventional MPEP 2106.05(d))
“using the multi-head self-attention neural network model of the trained graph neural network model.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
In reference to claim 9:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a manufacture
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The system of claim 8, wherein the graph neural network model comprises a spatial dynamic graph generation network configured to update the plurality of target graph-based representations with spatial interaction data among players.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could physically draw spatial interaction among players.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 10:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a manufacture
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The system of claim 9, wherein the spatial dynamic graph generation neural network model comprises the multi-head self-attention neural network model, and wherein the multi-head self-attention network comprises a plurality of heads, wherein each head of the plurality of heads corresponds to a respective action of a plurality of actions for classification.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could mentally classify a respective action of a plurality of actions.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 11:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a manufacture
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The system of claim 10, wherein each head of the plurality of heads is configured to generate a respective adjacency matrix, and wherein, when at least two consecutive frames of the plurality of frames have a same set of entities, an adjacency matrix of the plurality of adjacency matrices is diagonal.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could physically draw an adjacency matrix and evaluate the matrix.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 12:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a manufacture
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The system of claim 9, wherein training the graph neural network model to generate the action prediction for each player in each frame of the tracking data, comprises: learning spatial relationships between each player in each frame of the tracking data;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could mentally learn the spatial relationships between each player.
“and learning a plurality of neural network weights.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could mentally learn the neural network weights.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 13:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a manufacture
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The system of claim 9, wherein training the graph neural network model to generate the action prediction for each player in each frame of the tracking data, comprises: extracting temporal features from the tracking data” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could observe the tracking data and extract temporal features.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 14:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a manufacture
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The system of claim 8, wherein training the graph neural network model to generate the action prediction for each player in each frame of the tracking data, comprises: learning to generate a probability distribution across each of a plurality of action classes for each player in each frame.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could mentally learn to make a probability distribution across all possible action classes.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 15:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a machine
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“converting, by the computing system, the tracking data into a plurality of graph- based representations, wherein at least one of the graph-based representations includes a heterogenous graph including one or more nodes representing one or more entities, one or more temporal edges, or one or more spatial edges, wherein the one or more temporal edges represent one or more connections of each entity to a same respective entity across the plurality of frames;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could convert the tracking data into a plurality of graph-based representations including one or more nodes representing one or more entities, one or more temporal edges, or one or more spatial edges, wherein the one or more temporal edges represent one or more connections of each entity to a same respective entity across the plurality of frames.“and wherein the training the graph neural network model includes: generating, using the multi-head self-attention neural network model, a plurality of adjacency matrices based on one or more nodes of a sub-graph of the plurality of graph-based representations for each frame of the tracking data;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could generate a plurality of adjacency matrices based on nodes of a sub-graph, for each frame of the tracking data.
“converting, by the computing system, the target tracking data to a plurality of target graph-based representations, wherein each of the plurality of target graph-based representations corresponds to a target frame of the plurality of target frames and includes one or more target nodes, one or more target temporal edges, or one or more target spatial edges;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could convert the target tracking data into a plurality of graph-based representations including nodes, temporal edges, and spatial edges by drawing them out.
“and generating, by the computing system and based on the plurality of target graph-based representations, an action prediction for each player in each target frame [using the multi-head self-attention neural network model of the trained graph neural network model.]” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(Ill)(c)). For example, a person could mentally generate an action prediction for each player in each target frame.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
“A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by one or more processors, causes a computing system to perform operations, comprising:” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“retrieving, by the computing system, tracking data from a data store, the tracking data comprising a plurality of frames of data for a plurality of events across a plurality of seasons;” (insignificant extra-solution activity mere data gathering MPEP 2106.05(g))
“training, based on the plurality of graph-based representations, a graph neural network model to generate an action prediction for each player in each frame of the tracking data, wherein the graph neural network model includes a multi-head self- attention neural network model,” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“receiving, by the computing system, target tracking data for a target event, the target tracking data comprising a plurality of target frames;” (insignificant extra-solution activity mere data gathering MPEP 2106.05(g))
“using the multi-head self-attention neural network model of the trained graph neural network model.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
The claim does not include additional elements that are integrated into a practical application.
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
“A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by one or more processors, causes a computing system to perform operations, comprising:” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“retrieving tracking data from a data store, the tracking data comprising a plurality of frames of data for a plurality of events across a plurality of seasons;” (well-understood, routine, conventional MPEP 2106.05(d))
“training, based on the plurality of graph-based representations, a graph neural network model to generate an action prediction for each player in each frame of the tracking data, wherein the graph neural network model includes a multi-head self- attention neural network model,” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
“receiving target tracking data for a target event, the target tracking data comprising a plurality of target frames;” (well-understood, routine, conventional MPEP 2106.05(d))
“using the multi-head self-attention neural network model of the trained graph neural network model.” is merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)).
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
In reference to claim 16:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a machine
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The non-transitory computer readable medium of claim 15, wherein the graph neural network model comprises a spatial dynamic graph generation network configured to update the plurality of target graph-based representations with spatial interaction data among players.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could physically draw spatial interaction among players.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 17:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a machine
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The non-transitory computer readable medium of claim 16, wherein the spatial dynamic graph generation network comprises the multi-head self-attention neural network model, and wherein the multi-head self-attention neural network model comprises a plurality of heads, wherein each head of the plurality of heads corresponds to a respective action of a plurality of actions for classification.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could mentally classify a respective action of a plurality of actions.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 18:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a machine
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The non-transitory computer readable medium of claim 16, wherein training the graph neural network model to generate the action prediction for each player in each frame of the tracking data, comprises: learning spatial relationships between each player in each frame of the tracking data;” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could mentally learn the spatial relationships between each player.
“and learning a plurality of neural network weights.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could mentally learn the neural network weights.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 19:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a machine
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The non-transitory computer readable medium of claim 16, wherein training the graph neural network model to generate the action prediction for each player in each frame of the tracking data, comprises: extracting temporal features from the tracking data” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could observe the tracking data and extract temporal features.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
In reference to claim 20:
Step 1 - Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a machine
Step 2A Prong 1 - Does the claim recite an abstract idea, law of nature, or natural phenomenon?
“The non-transitory computer readable medium of claim 15, wherein training the graph neural network model to generate the action prediction for each player in each frame of the tracking data, comprises: learning to generate a probability distribution across each of a plurality of action classes for each player in each frame.” which is an abstract idea because it is directed to a mental process, an observation, evaluation, judgement, or opinion. The limitation as drafted, and under a broadest reasonable interpretation, can be performed in the human mind, or by a human using a pen and paper (MPEP 2106.04(a)(2)(III)). For example, a person could mentally learn to make a probability distribution across all possible action classes.
Step 2A Prong 2 - Does the claim recite additional elements that integrate the judicial exception into a practical application?
No
Step 2B - Does the claim recite additional elements that amount to significantly more than the judicial exception?
No
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-6, 8-10, 12-13, and 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al; “Group activity prediction with sequential relational anticipation model” [12 Nov 2020] (hereinafter “Chen” in view of N. Danisik et al; "Football Match Prediction Using Players Attributes" [2018] (hereinafter “Danisik”) in further view of Ashesh Jain et al; “Structural-RNN: Deep Learning on Spatio-Temporal Graphs” [Apr 11, 2016] (hereinafter “Jain”) in further view of Bang et al; US 20200356628 A1 (hereinafter “Bang”)
Regarding Claim 1, Chen teaches A method, comprising:
retrieving, by a computing system, tracking data from a data store, the tracking data comprising a plurality of frames of data for a plurality of events [across a plurality of seasons]; (Chen Section 4.3 Paragraph 1 shows retrieving data; “our method adopt the tracklets of players provided by [19]”. Examiner notes that adopt is the same as retrieve; Chen Section 3.5 Paragraph 1 shows computing system; “Note that the generated group representation is computed by SRAM S from the partial observation”. Chen Section 4.3 Paragraph 1 shows tracking data; “we extend state-of-the-art action prediction method AAPNet to make use of tracking information”; Chen Section 1 Paragraph 1 shows a data store; “some methods attempt to predict actions performed by multiple people in standard databases such as UCF101”; Chen Fig 1. left portion of figure shows a plurality of frames; Chen Section 4.1 Paragraph 1 shows plurality of events; “Volleyball Dataset [19] consists of 4830 video clips distributed in 8 group activities, such as left spiking and right setting”. Examiner notes that events can be interpreted as actions or activity)
PNG
media_image1.png
282
727
media_image1.png
Greyscale
converting, by the computing system, the tracking data into a plurality of graph-based representations, [wherein at least one of the plurality of graph-based representations includes a heterogenous graph including one or more nodes representing one or more entities, one or more temporal edges, or one or more spatial edges, wherein the one or more temporal edges represent one or more connections of each entity to a same respective entity across the plurality of frames]; (Chen Section 3.1 Paragraph 1; “we follow [48] to explicitly model the pair-wise position relations and action relations of multiple people in the observed frames as two relation graphs” Examiner notes that converting tracking data (position relations and action relations) into graph-based representations (relation graphs));
training, by the computing system and based on the plurality of graph-based representations, a graph neural network model to generate an action prediction for each player in each frame of the tracking data, (Chen Section 3.2 Paragraph 1 and Fig. 1 shows training a graph neural network model to generate an action prediction as described in the caption of Fig. 1 of predicting group activity; the training is based on the plurality of graph-based representations (graphs Gp and Ga); “The observation encoder £ is proposed to summarize spatiotemporal information of the complex relational dynamics of multiple people in partial observations containing to frames. £ learns to map Ga, Gp, and X to a latent variable Zo… Specifically, it first performs spatial graph convolution [20] on the two graphs Gp t and Ga t for the t-th frame… Latent variable Zo will be integrated in the sequential decoder”; Chen Section 1 Paragraph 3 shows for each player in each frame; “the observation encoder naturally models the relational dynamics of people and complex interactions between people in the observed frames.”);
PNG
media_image2.png
305
705
media_image2.png
Greyscale
receiving, by the computing system, target tracking data for a target event the target tracking data comprising a plurality of target frames; (Chen Section 3.3 Paragraph 2 shows receiving target tracking data for a target event; “a partial observation of a volleyball activity is given, which contains run-up of ace spikers and waiting gestures of their opponents. Our model is supposed to predict it as ‘spiking’”. Examiner notes that target tracking data is partial observation and being given means it has been received; Chen left portion of Fig 2 shows that partial observation contains multiple frames);
converting, by the computing system, the target tracking data to a plurality of target graph-based representations, wherein each of the plurality of target graph-based representations corresponds to a target frame of the plurality of target frames and includes one or more target nodes, one or more target temporal edges, or one or more target spatial edges; (Chen Fig 2 shows a plurality of target frames/partial observations being converted into a plurality of target graph based representations in the observation encoder; A target graph (unrolling stage graphs) includes target nodes (nodes of individuals) and target spatial edges (predicted position); Chen Section 3.2 Paragraph 1 shows target frames being converted into graph-based representations; “The observation encoder £ is proposed to summarize spatiotemporal information of the complex relational dynamics of multiple people in partial observations containing to frames.” Examiner notes that “summarize” comprises of converting action features and positions into position relation graphs (See Section 3.1))
and generating, by the computing system and based on the plurality of target graph-based representations, an action prediction for each player in each target frame [using the multi-head self-attention neural network model of the trained graph neural network model]. (Chen Fig 2 and Section 4.3 Paragraph 4 shows generating an action prediction for each player in target frame via the trained graph neural network model (our model) and based on the plurality of target graph-based representations (trackelts is processed in architecture shown in Fig 2 to produce a plurality of unrolling stage graphs); “Given tracklets as input, our method is 6.5% higher than e-AAPNet at 50% observation ratio since the people’s actions and relations are predicted in our model”)
Chen fails to teach a plurality of seasons. However, Danisik teaches a plurality of seasons (Danisik Page 204 Paragraph 2 shows plurality of seasons; “We used data from the seasons 2011/2012-2015/2016”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen and Danisik. Chen teaches a sequential decoder to anticipate the representations for multiple people’s future positions and activity. Danisik teaches using data of players to predict results of a match. One of ordinary skill would have motivation to combine Chen and Danisik to incorporate seasonal data to see player improvements over time “As long as the players are evolving as a people, we need to have this set of attributes for each season separately.” (Danisik Page 202 Paragraph 5).
Chen in view of Danisik does not teach wherein at least one of the plurality of graph-based representations includes a heterogenous graph including one or more nodes representing one or more entities, one or more temporal edges, or one or more spatial edges, wherein the one or more temporal edges represent one or more connections of each entity to a same respective entity across the plurality of frames
However, Jain does teach wherein at least one of the plurality of graph-based representations includes a heterogenous graph including one or more nodes representing one or more entities, one or more temporal edges, or one or more spatial edges, wherein the one or more temporal edges represent one or more connections of each entity to a same respective entity across the plurality of frames (Jain Figure 2 and Page 3 Paragraph 4; “We represent a st-graph with G = (V,ES,ET), whose structure (V,ES) unrolls over time through edges ET. Figure 2a shows an example st-graph capturing human-object inter actions during an activity. The nodes v ∈ V and edges e ∈ ES ∪ET of the st-graph repeats over time. In particular, Figure 2b shows the same st-graph unrolled through time. In the unrolled st-graph, the nodes at a given time step t are connected with undirected spatio-temporal edge e = (u,v) ∈ ES, and the nodes at adjacent time steps (say the node u at time t and the node v at time t + 1) are connected with undirected temporal edge iff (u,v) ∈ ET” Examiner notes that at least one of the plurality of graph-based representations includes a heterogenous graph (st-graph) including one or more nodes representing entities (nodes Human and Object), one or more temporal edges (temporal edge Et), one or more spatial edges (spatio temporal edges ES), wherein the one or more temporal edges (Et) represent one or more connections of each entity to a same respective entity (Fig 2 shows edge X v, v connects human node at t to human node at t + 1) across the plurality of frames (each spatio temporal graph shows a frame of activity at time t; spatio temporal graph at time t is connected to spatio temporal graph at time t + 1))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Danisik, and Jain. Chen teaches a sequential decoder to anticipate the representations for multiple people’s future positions and activity. Danisik teaches using data of players to predict results of a match. Jain teaches combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural networks. One of ordinary skill would have motivation to combine Chen, Danisik, and Jain to utilize st-graphs to overcome the struggles to model long-term human motion “S-RNN’s good performance is attributed to its structural modeling of human motion through the underlying st-graph. S-RNN models each body part separately with node RNNs and captures interactions between them with edge RNNs in order to produce coherent motions.” (Jain Page 7 Paragraph 1).
Chen in view of Danisik in further view of Jain fails to teach wherein the [graph] neural network model includes a multi-head self-attention neural network model, and wherein the training the graph neural network includes: generating, using the multi-head self-attention neural network model, a plurality of adjacency matrices based on one or more nodes of a sub-graph of the plurality of graph-based representations each frame of the tracking data;
However, Bang does teach wherein the [graph] neural network model includes a multi-head self-attention neural network model, and wherein the training the graph neural network includes: generating, using the multi-head self-attention neural network model, a plurality of adjacency matrices based on one or more nodes of a sub-graph of the plurality of graph-based representations each frame of the tracking data; (Bang Paragraph 0075; “The weight determination module 522 then applies an attention mechanism on the given node and the set of neighbor nodes to determine respective attention weights 524 for the set of neighbor nodes.” Bang Paragraph 0083; “to stabilize the learning process of the attention neural networks, the weight determination module 522 may apply a multi-head attention mechanism on a given node and its set of neighbor nodes.” Examiner notes that the neural network model (attention neural network) includes a multi-head self-attention neural network (multi-head attention mechanism), wherein the training includes generating a plurality of adjacency matrices (attention weights) based on one or more nodes of a sub-graph (given node) of a the plurality of graph-based representations each frame of the tracking data)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Danisik, Jain, and Bang. Chen teaches a sequential decoder to anticipate the representations for multiple people’s future positions and activity. Danisik teaches using data of players to predict results of a match. Jain teaches combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural networks. Bang teaches applying a multi-head attention mechanism in a neural network. One of ordinary skill would have motivation to combine Chen, Danisik, Jain, and Bang to apply a multi-head attention mechanism to stabilize the learning process “to stabilize the learning process of the attention neural networks, the weight determination module 522 may apply a multi-head attention mechanism on a given node and its set of neighbor nodes.” (Bang Paragraph 0083).
PNG
media_image3.png
123
469
media_image3.png
Greyscale
Regarding Claim 2, 9, and 16, Chen teaches the method of claim 1, wherein the graph neural network model comprises a spatial dynamic graph generation network configured to update the plurality of target graph-based representations with spatial interaction data among players. (Chen Fig 2 shows the graph-based representations being updated within the unrolling stage, i.e. spatial dynamic graph, where updates are completed for each stage; generated unrolling stages/spatial dynamic graph indicates graph neural network model comprises a spatial dynamic graph generation network; Chen Section 3.3 Paragraph 6 shows how the stages are updated from previous stage variable with spatial interaction data among players/position information; “The position auto-encoder first encodes the positions Bk of multiple people to a latent variable Zk at stage k through graph convolution … and then decodes the positions Bk+1 for the next stage by … where Vep, Vea, Vdp, Vda are learnable parameters. Gpk and Gak; are the same graphs used in the activity auto-encoder. The anticipation of Bk+1 is conditioned on latent variables Zpk and Z0, in order to both keep track of the short-term position information of the previous unrolling stage and use the long-term spatiotemporal information in the partial observations.”)
Regarding Claim 3, 10, 17, Chen fails to teach the method of claim 2, wherein the spatial dynamic graph generation network comprises a multi-head self-attention neural network model, and wherein the multi-head self-attention neural network model comprises a plurality of heads, wherein each head of the plurality of heads corresponds to a respective action of a plurality of actions for classification
However, Bang teaches the method of claim 2, wherein the spatial dynamic graph generation network comprises a multi-head self-attention neural network model, and wherein the multi-head self-attention neural network model comprises a plurality of heads, wherein each head of the plurality of heads corresponds to a respective action of a plurality of actions for classification (Bang Paragraph 0083; “to stabilize the learning process of the attention neural networks, the weight determination module 522 may apply a multi-head attention mechanism on a given node and its set of neighbor nodes. The multi-head attention mechanism may consist of a plurality of attention mechanisms, each of the attention mechanism being applied on the given node and its set of neighbor nodes in a similar way as described above.” Examiner notes that neural network is the network that contains the multi-head self-attention network/multi-head attention mechanism; multi-head suggests a plurality of heads; where in each head of the plurality of heads correspond/is applied to a respective action of a plurality of actions for classification/given node)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Danisik, Jain, and Bang. Chen teaches a sequential decoder to anticipate the representations for multiple people’s future positions and activity. Danisik teaches using data of players to predict results of a match. Jain teaches combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural networks. Bang teaches applying a multi-head attention mechanism in a neural network. One of ordinary skill would have motivation to combine Chen, Danisik, Jain, and Bang to apply a multi-head attention mechanism to stabilize the learning process “to stabilize the learning process of the attention neural networks, the weight determination module 522 may apply a multi-head attention mechanism on a given node and its set of neighbor nodes.” (Bang Paragraph 0083).
Regarding Claim 5, 12, and 18, Chen teaches the method of claim 2, wherein training the graph neural network model to generate the action prediction for each player in each frame of the tracking data, comprises: learning spatial relationships between each player in each frame of the tracking data; (Chen Section 3.2 Paragraph 1 shows training GNN model to learn spatial relationships; “The observation encoder £ is proposed to summarize spatiotemporal information of the complex relational dynamics of multiple people in partial observations containing to frames. £ learns to map … to a latent variable Zo, by the spatio-temporal graph convolution network”)
and learning a plurality of neural network weights. (Chen Section 3.2 Paragraph 1 shows GNN learning weights; “Here, CT is ReLU activation, Wp and Wa are learnable weights”)
Regarding Claim 6, 13, 19, Chen teaches the method of claim 2, wherein training the graph neural network model to generate the action prediction for each player in each frame of the tracking data, comprises: extracting temporal features from the tracking data. (Chen Section 3.2 Paragraph 2 shows extracting temporal features; “The observation encoder £ is proposed to summarize spatiotemporal information of the complex relational dynamics of multiple people in partial observations containing t0 frames.” Examiner notes that to be able to summarize the spatiotemporal information in the partial observations it needs to be extracted.)
Regarding Claim 8 and 15, Chen teaches A method, comprising:
A system, comprising: a processor; and a memory having programming instructions stored thereon, which, when executed by the processor, causes the system to perform one or more operations, comprising: (Chen Section 3.5 Paragraph 1 shows computing system; “Note that the generated group representation is computed by SRAM S from the partial observation” Examiner notes that a computing system signify a processor and memory to execute instructions);
retrieving tracking data from a data store, the tracking data comprising a plurality of frames of data for a plurality of events [across a plurality of seasons]; (Chen Section 4.3 Paragraph 1 shows retrieving data; “our method adopt the tracklets of players provided by [19]”. Examiner notes that adopt is the same as retrieve; Chen Section 3.5 Paragraph 1 shows computing system; “Note that the generated group representation is computed by SRAM S from the partial observation”. Chen Section 4.3 Paragraph 1 shows tracking data; “we extend state-of-the-art action prediction method AAPNet to make use of tracking information”; Chen Section 1 Paragraph 1 shows a data store; “some methods attempt to predict actions performed by multiple people in standard databases such as UCF101”; Chen Fig 1. left portion of figure shows a plurality of frames; Chen Section 4.1 Paragraph 1 shows plurality of events; “Volleyball Dataset [19] consists of 4830 video clips distributed in 8 group activities, such as left spiking and right setting”. Examiner notes that events can be interpreted as actions or activity)
PNG
media_image1.png
282
727
media_image1.png
Greyscale
converting the tracking data into a plurality of graph-based representations, [wherein at least one of the plurality of graph-based representations includes a heterogenous graph including one or more nodes representing one or more entities, one or more temporal edges, or one or more spatial edges, wherein the one or more temporal edges represent one or more connections of each entity to a same respective entity across the plurality of frames]; (Chen Section 3.1 Paragraph 1; “we follow [48] to explicitly model the pair-wise position relations and action relations of multiple people in the observed frames as two relation graphs” Examiner notes that converting tracking data (position relations and action relations) into graph-based representations (relation graphs));
training, based on the plurality of graph-based representations, a graph neural network model to generate an action prediction for each player in each frame of the tracking data, (Chen Section 3.2 Paragraph 1 and Fig. 1 shows training a graph neural network model to generate an action prediction as described in the caption of Fig. 1 of predicting group activity; the training is based on the plurality of graph-based representations (graphs Gp and Ga); “The observation encoder £ is proposed to summarize spatiotemporal information of the complex relational dynamics of multiple people in partial observations containing to frames. £ learns to map Ga, Gp, and X to a latent variable Zo… Specifically, it first performs spatial graph convolution [20] on the two graphs Gp t and Ga t for the t-th frame… Latent variable Zo will be integrated in the sequential decoder”; Chen Section 1 Paragraph 3 shows for each player in each frame; “the observation encoder naturally models the relational dynamics of people and complex interactions between people in the observed frames.”);
PNG
media_image2.png
305
705
media_image2.png
Greyscale
receiving target tracking data for a target event the target tracking data comprising a plurality of target frames; (Chen Section 3.3 Paragraph 2 shows receiving target tracking data for a target event; “a partial observation of a volleyball activity is given, which contains run-up of ace spikers and waiting gestures of their opponents. Our model is supposed to predict it as ‘spiking’”. Examiner notes that target tracking data is partial observation and being given means it has been received; Chen left portion of Fig 2 shows that partial observation contains multiple frames);
converting the target tracking data to a plurality of target graph-based representations, wherein each of the plurality of target graph-based representations corresponds to a target frame of the plurality of target frames and includes one or more target nodes, one or more target temporal edges, or one or more target spatial edges; (Chen Fig 2 shows a plurality of target frames/partial observations being converted into a plurality of target graph based representations in the observation encoder; A target graph (unrolling stage graphs) includes target nodes (nodes of individuals) and target spatial edges (predicted position); Chen Section 3.2 Paragraph 1 shows target frames being converted into graph-based representations; “The observation encoder £ is proposed to summarize spatiotemporal information of the complex relational dynamics of multiple people in partial observations containing to frames.” Examiner notes that “summarize” comprises of converting action features and positions into position relation graphs (See Section 3.1))
and generating, based on the plurality of target graph-based representations, an action prediction for each player in each target frame [using the multi-head self-attention neural network model of the trained graph neural network model]. (Chen Fig 2 and Section 4.3 Paragraph 4 shows generating an action prediction for each player in target frame via the trained graph neural network model (our model) and based on the plurality of target graph-based representations (trackelts is processed in architecture shown in Fig 2 to produce a plurality of unrolling stage graphs); “Given tracklets as input, our method is 6.5% higher than e-AAPNet at 50% observation ratio since the people’s actions and relations are predicted in our model”)
Chen fails to teach a plurality of seasons. However, Danisik teaches a plurality of seasons (Danisik Page 204 Paragraph 2 shows plurality of seasons; “We used data from the seasons 2011/2012-2015/2016”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen and Danisik. Chen teaches a sequential decoder to anticipate the representations for multiple people’s future positions and activity. Danisik teaches using data of players to predict results of a match. One of ordinary skill would have motivation to combine Chen and Danisik to incorporate seasonal data to see player improvements over time “As long as the players are evolving as a people, we need to have this set of attributes for each season separately.” (Danisik Page 202 Paragraph 5).
Chen in view of Danisik does not teach wherein at least one of the plurality of graph-based representations includes a heterogenous graph including one or more nodes representing one or more entities, one or more temporal edges, or one or more spatial edges, wherein the one or more temporal edges represent one or more connections of each entity to a same respective entity across the plurality of frames
However, Jain does teach wherein at least one of the plurality of graph-based representations includes a heterogenous graph including one or more nodes representing one or more entities, one or more temporal edges, or one or more spatial edges, wherein the one or more temporal edges represent one or more connections of each entity to a same respective entity across the plurality of frames (Jain Figure 2 and Page 3 Paragraph 4; “We represent a st-graph with G = (V,ES,ET), whose structure (V,ES) unrolls over time through edges ET. Figure 2a shows an example st-graph capturing human-object inter actions during an activity. The nodes v ∈ V and edges e ∈ ES ∪ET of the st-graph repeats over time. In particular, Figure 2b shows the same st-graph unrolled through time. In the unrolled st-graph, the nodes at a given time step t are connected with undirected spatio-temporal edge e = (u,v) ∈ ES, and the nodes at adjacent time steps (say the node u at time t and the node v at time t + 1) are connected with undirected temporal edge iff (u,v) ∈ ET” Examiner notes that at least one of the plurality of graph-based representations includes a heterogenous graph (st-graph) including one or more nodes representing entities (nodes Human and Object), one or more temporal edges (temporal edge Et), one or more spatial edges (spatio temporal edges ES), wherein the one or more temporal edges (Et) represent one or more connections of each entity to a same respective entity (Fig 2 shows edge X v, v connects human node at t to human node at t + 1) across the plurality of frames (each spatio temporal graph shows a frame of activity at time t; spatio temporal graph at time t is connected to spatio temporal graph at time t + 1))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Danisik, and Jain. Chen teaches a sequential decoder to anticipate the representations for multiple people’s future positions and activity. Danisik teaches using data of players to predict results of a match. Jain teaches combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural networks. One of ordinary skill would have motivation to combine Chen, Danisik, and Jain to utilize st-graphs to overcome the struggles to model long-term human motion “S-RNN’s good performance is attributed to its structural modeling of human motion through the underlying st-graph. S-RNN models each body part separately with node RNNs and captures interactions between them with edge RNNs in order to produce coherent motions.” (Jain Page 7 Paragraph 1).
Chen in view of Danisik in further view of Jain fails to teach wherein the [graph] neural network model includes a multi-head self-attention neural network model, and wherein the training the graph neural network includes: generating, using the multi-head self-attention neural network model, a plurality of adjacency matrices based on one or more nodes of a sub-graph of the plurality of graph-based representations each frame of the tracking data;
However, Bang does teach wherein the [graph] neural network model includes a multi-head self-attention neural network model, and wherein the training the graph neural network includes: generating, using the multi-head self-attention neural network model, a plurality of adjacency matrices based on one or more nodes of a sub-graph of the plurality of graph-based representations each frame of the tracking data; (Bang Paragraph 0075; “The weight determination module 522 then applies an attention mechanism on the given node and the set of neighbor nodes to determine respective attention weights 524 for the set of neighbor nodes.” Bang Paragraph 0083; “to stabilize the learning process of the attention neural networks, the weight determination module 522 may apply a multi-head attention mechanism on a given node and its set of neighbor nodes.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Danisik, Jain, and Bang. Chen teaches a sequential decoder to anticipate the representations for multiple people’s future positions and activity. Danisik teaches using data of players to predict results of a match. Jain teaches combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural networks. Bang teaches applying a multi-head attention mechanism in a neural network. One of ordinary skill would have motivation to combine Chen, Danisik, Jain, and Bang to apply a multi-head attention mechanism to stabilize the learning process “to stabilize the learning process of the attention neural networks, the weight determination module 522 may apply a multi-head attention mechanism on a given node and its set of neighbor nodes.” (Bang Paragraph 0083).
Claim(s) 4 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al; “Group activity prediction with sequential relational anticipation model” [12 Nov 2020] (hereinafter “Chen”) in view of N. Danisik et al; "Football Match Prediction Using Players Attributes" [2018] (hereinafter “Danisik”) in further view of Ashesh Jain et al; “Structural-RNN: Deep Learning on Spatio-Temporal Graphs” [Apr 11, 2016] (hereinafter “Jain”) in further view of Bang et al; US 20200356628 A1 (hereinafter “Bang”) in further view of Filipe Manuel Clemente et al; “ANALYSIS OF SCORED AND CONCEDED GOALS BY A FOOTBALL TEAM THROUGHOUT A SEASON: A NETWORK ANALYSIS” [2018] (hereinafter “Clemente”)
Regarding Claim 4 and 11, Chen fails to the method of claim 3, wherein each head of the plurality of heads is configured to generate a respective adjacency matrix, and wherein, when at least two consecutive frames of the plurality of frames have a same set of entities, an adjacency matrix of the plurality of adjacency matrices is diagonal.
However, Bang teaches the method of claim 3, wherein each head of the plurality of heads is configured to generate a respective adjacency matrix. (Bang Paragraph 0075; “The weight determination module 522 then applies an attention mechanism on the given node and the set of neighbor nodes to determine respective attention weights 524 for the set of neighbor nodes.” Examiner notes that each head/attention mechanism of the plurality of heads is configured to generate/determine an adjacency matrix/attention weights)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Danisik, Jain, and Bang. Chen teaches a sequential decoder to anticipate the representations for multiple people’s future positions and activity. Danisik teaches using data of players to predict results of a match. Jain teaches combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural networks. Bang teaches applying a multi-head attention mechanism in a neural network. One of ordinary skill would have motivation to combine Chen, Danisik, Jain, and Bang to apply a multi-head attention mechanism to stabilize the learning process “to stabilize the learning process of the attention neural networks, the weight determination module 522 may apply a multi-head attention mechanism on a given node and its set of neighbor nodes.” (Bang Paragraph 0083).
Chen in view of Danisik in further view of Jain in further view of Bang does not teach and wherein, when at least two consecutive frames of the plurality of frames have a same set of entities, an adjacency matrix of the plurality of adjacency matrices is diagonal.
However, Clemente does teach and wherein, when at least two consecutive frames of the plurality of frames have a same set of entities, an adjacency matrix of the plurality of adjacency matrices is diagonal. (Clemente Fig 1 and Fig 3b and Page 105 Last Paragraph; “Thus, from each attacking play it is possible to compute the adjacency matrix. The adjacency matrix is used to build a finite network where the entries represent the linkages between the vertices (Couceiro, Clemente, & Martins, 2013) per each attacking play that resulted in the goal scored. To perform the connectivity between teammates the definition used was 0 (zero) value as the non-connectivity and 1 (one) as the connectivity.” Examiner notes that when at least two consecutive frames of the plurality of frames have a same set of entities (when an attacking play starts with a regular strategic position as starting frame to a frame that resulted in a score seen in fig 3b; a scoring play can be made without interactions between players meaning the starting frame will have the same set of entities as the scoring frame), an adjacency matrix of the plurality of adjacency matrices (adjacency matrix of plurality of attacking plays) is diagonal (adjacency matrix where there is no interaction between different players))
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Danisik, Jain, Bang, and Clemente. Chen teaches a sequential decoder to anticipate the representations for multiple people’s future positions and activity. Danisik teaches using data of players to predict results of a match. Jain teaches combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural networks. Bang teaches applying a multi-head attention mechanism in a neural network. Clemente teaches analyze the goals scored and conceded during a full season using network methods. One of ordinary skill would have motivation to combine Chen, Danisik, Jain, Bang, and Clemente to increase the possibility to identify efficacy processes hidden in the dataset “Our approach reduces the potential to understand the patterns of play but increases the possibility to identify some efficacy processes that can be masked by a big dataset of full passing sequences.” (Clemente Page 104 Paragraph 4).
Claim(s) 7, 14, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al; “Group activity prediction with sequential relational anticipation model” [12 Nov 2020] (hereinafter “Chen”) and in view of N. Danisik et al; "Football Match Prediction Using Players Attributes" [2018] (hereinafter “Danisik”) in further view of Ashesh Jain et al; “Structural-RNN: Deep Learning on Spatio-Temporal Graphs” [Apr 11, 2016] (hereinafter “Jain”) in further view of Bang et al; US 20200356628 A1 (hereinafter “Bang”) and further in view of Lu et al; “GAIM: graph attention interaction model for collective activity recognition” [2019] (hereinafter “Lu”).
Regarding Claim 7, 14, and 20, Chen fails to teach the method of claim 1, wherein training the graph neural network model to generate the action prediction for each player in each frame of the tracking data, comprises: learning to generate a probability distribution across each of a plurality of action classes for each player in each frame.
However, Lu teaches the method of claim 1, wherein training the graph neural network model to generate the action prediction for each player in each frame of the tracking data, comprises: learning to generate a probability distribution across each of a plurality of action classes for each player in each frame. (Lu Section III.D shows generating a probability distribution; The GRUs network are respectively fed into two SoftMax classifiers to predict the final labels for both individuals' actions and the collective activity. Examiner notes that SoftMax classifiers uses probability distributions to give probabilities for each class label)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Chen, Danisik, Jain, Bang, and Lu. Chen teaches a sequential decoder to anticipate the representations for multiple people’s future positions and activity. Danisik teaches using data of players to predict results of a match. Jain teaches combining the power of high-level spatio-temporal graphs and sequence learning success of Recurrent Neural networks. Bang teaches applying a multi-head attention mechanism in a neural network. Lu teaches generating a probability distribution. One of ordinary skill would have motivation to combine Chen, Danisik, Jain, Bang, and Lu to increase the possibility to identify efficacy processes hidden in the dataset “Our approach reduces the potential to understand the patterns of play but increases the possibility to identify some efficacy processes that can be masked by a big dataset of full passing sequences.” (Clemente Page 104 Paragraph 4).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL DUC TRAN whose telephone number is (571)272-6870. The examiner can normally be reached Mon-Fri 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.D.T./Examiner, Art Unit 2147
/ERIC NILSSON/Primary Examiner, Art Unit 2151
Prosecution Insights