DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments and amendments in the Amendment filed May 8, 2026 (herein “Amendment”), with respect to the objections to claims 29 and 30 and claims depending therefrom have been fully considered and are persuasive. The objections to claims 29 and 30 and claims depending therefrom has been withdrawn.
Applicant’s arguments and amendments in the Amendment, with respect to the rejection to claims 25, 39, 34, 38 and 40 and claims depending therefrom under 35 U.S.C. 112(b) have been fully considered and are persuasive. the rejection to claims 25, 39, 34, 38 and 40 and claims depending therefrom under 35 U.S.C. 112(b) has been withdrawn. It is noted however that the amendment to correct claim 40 is incomplete and not consistent with the correcting amendment made to claims 25 and 34, thus resulting in a new objections ground for the informality error.
Applicant’s arguments and amendments in the Amendment with respect to the rejections of claims 21, 30, 39 and claims depending therefrom under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, new grounds of rejection are made in view of Zhou et al., “UDE: A Unified Driving Engine for Human Motion Generation,” arXiv:2211.16016v1 [cs.CV] 29 Nov 2022, https://doi.org/10.48550/arXiv.2211.16016.
The Terminal Disclaimer filed on May 8, 2026 has obviated the non-statutory Double Patenting rejection in view of co-pending application no. 18/401,006, and therefore, this double patenting rejection is withdrawn.
Applicant’s arguments and amendments in the Amendment with respect to the non-statutory double patenting rejection in view of co-pending application nos. 19/314,642 and 19/309,109 have been fully considered and are persuasive in part, although the claim amendments are obvious in view of Zhou et al., “UDE: A Unified Driving Engine for Human Motion Generation,” arXiv:2211.16016v1 [cs.CV] 29 Nov 2022, https://doi.org/10.48550/arXiv.2211.16016, and therefore the non-statutory double patenting rejection in view of co-pending application nos. 19/314,642 and 19/309,109 have been maintained but updated to include reliance upon Zhou.
Applicant’s arguments and amendments in the Amendment with respect to the rejections of claims 21, 30, 39 and claims depending therefrom under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, new grounds of rejection are made in view of Zhou et al., “UDE: A Unified Driving Engine for Human Motion Generation,” arXiv:2211.16016v1 [cs.CV] 29 Nov 2022, https://doi.org/10.48550/arXiv.2211.16016. It is noted also that amendments to previously indicated as allowable dependent claims 34 and 40 have removed subject matter that had made the claims previously allowable. Therefore, the scope of these claims have changed, now subjecting them, and claims depending therefrom, also to rejection under 35 U.S.C. 103.
Claim Objections
Claim 40 is objected to because of the following informalities: the claim recites “a stacking of each of the one or more agents” but should instead recite “a stacking of each of the one or more of the labeled events of the agents.” Appropriate correction is required.
Claim 35, and therefore claims 36-38 which depend therefrom are objected to because of the following informalities: claim 35 recites “the tracking decoder,” but should instead recite “a tracking encoder.” Appropriate correction is required.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 21, 30 and 39 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 7 and 13 of copending Application No. 19/314,642 (herein “‘642 application”) in view of Salzmann et al., “Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data,” arXiv:2001.03093v5 [cs.RO], January 13, 2021, https://doi.org/10.48550/arXiv.2001.03093 (herein “Salzmann”) and in view of Zhou et al., “UDE: A Unified Driving Engine for Human Motion Generation,” arXiv:2211.16016v1 [cs.CV] 29 Nov 2022, https://doi.org/10.48550/arXiv.2211.16016 (herein “Zhou”).
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the ‘642 application recite most of the limitations of the present application with correspondence to the claims is set forth below:
Regarding claims 21, 30 and 39, claims 1, 7 and 13 of the ‘642 application correspond as follows, with deficiencies of claims 1, 7 and 13 of the ‘642 application noted in curly brackets {}:
Claims 21, 30 and 39 of the present application
Claims 1, 7 and 13 of the ‘642 application
[claim 21 only: A computer implemented method for tracking one or more individuals during a sporting event, the method comprising:]
[Claim 1 only: A computer implemented method for tracking one or more individuals during a sporting event, the method comprising:]
[claim 30 only: A system for tracking one or more individuals during a sporting event, the system comprising:]
[Claim 7 only: A system for tracking one or more individuals during a sporting event, the system comprising:]
[Claim 39 only: A non-transitory computer readable medium configured to store processor-readable instructions, wherein when executed by a processor, the instructions perform operations comprising:]
[Claim 13 only: A non-transitory computer readable medium configured to store processor-readable instructions, wherein when executed by a processor, the instructions perform operations comprising:]
receiving, as an input, sporting event data, the sporting event data including {geospatial} data and labeled event data based on the sporting event;
receiving, as an input, broadcast tracking data of a sporting event and labeled event data of the sporting event;
performing multi-object tracking of one or more agents based on the {geospatial} data to determine one or more vectors;
performing multi-object tracking of one or more agents of the broadcast tracking data to determine one or more vectors;
inputting the labeled event data and one or more vectors into a diffusion model, {the diffusion model incorporating a transformer based-neural network};
inputting the labeled event data and one or more vectors into a diffusion model;
and determining, using the diffusion model, one or more trajectory sequences for the one or more agents {by tokenizing the labeled event data using a linear projection to generate tokenized labeled event data; processing the tokenized labeled event data with stacked encoders; and outputting event embeddings based on the processing}.
determining, using the diffusion model, one or more trajectory sequences for the one or more agents;
Claims 1, 7 and 13 of the ‘642 application, while reciting “broadcast tracking data of a sporting event” does not recite the data specifically as being “geospatial data” however, Salzmann teaches “geospatial data” on page 7, LIDAR data incorporated into the event/agent representation vectors.
Further, claims 1, 7 and 13 of the ‘642 application do not, however Zhou teaches the diffusion model incorporating a transformer based-neural network; (Zhou Abstract and page 3, section 3, framework for determining human motion sequences including diffusion module and a unified token transformer) by tokenizing the labeled event data using a linear projection to generate tokenized labeled event data (Zhou page 4, section 3.3, left column, in the unified token transformer, for each embedding of motion data associated with a sequence (labeled event data) a linear projection is applied, and as shown in fig. 2, token sequences are output); processing the tokenized labeled event data with stacked encoders (Zhou page 3, section 3.3, the unified token transformer uses a stacked encoder architecture, where fig. 2 on page 4 illustrates the token processing within the stacked encoders); and outputting event embeddings based on the processing (Zhou page 5, section 3.4, fig. 2, in the diffusion motion decoder, motion sequence embeddings are extracted from the token sequence).
Therefore, taking the teachings of claims 1, 7 and 13 of the ‘642 application and Salzmann together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified claims 1, 7 and 13 of the ‘642 application to have the geographic data teachings cited above as disclosed by Salzmann at least because doing so would allow for modeling agents with different perception ranges. See Salzmann, bottom of page 5.
Further, taking the teachings of claims 1, 7 and 13 of the ‘642 application and Zhou together as a whole, it would have been obvious to a “PHOSITA” before the effective filing date of the claimed invention to have modified the teachings of claims 1, 7 and 13 of the ‘642 application with the processing as disclosed above by Zhou at least because doing so would allow for multi-modal motion generation with high diversity. See Zhou page 2, left column.
Claims 21–23, 30–32 and 39 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1–2, 4, 11–12, 14 and 18 of copending Application No. 19/309,109 (herein “‘109 application”) in view of Salzmann and in view of Zhou.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the ‘109 application recite most of the limitations of the present application with correspondence to the claims is set forth below:
Regarding claims 21, 30 and 39, claims 1, 11 and 18 of the ‘109 application correspond as follows, with deficiencies of claims 1, 11 and 18 of the ‘109 application noted in curly brackets {}:
Claims 21, 30 and 39 of the present application
Claims 1, 11 and 18 of the ‘109 application
[claim 21 only: A computer implemented method for tracking one or more individuals during a sporting event, the method comprising:]
[Claim 1 only: A method for generating trajectories for one or more players during a sporting event, the method comprising:]
[claim 30 only: A system for tracking one or more individuals during a sporting event, the system comprising:]
[Claim 11 only: A system for generating trajectories for one or more players during a sporting event, the system comprising:]
[Claim 39 only: A non-transitory computer readable medium configured to store processor-readable instructions, wherein when executed by a processor, the instructions perform operations comprising:]
[Claim 18 only: A non-transitory computer readable medium configured to store processor-readable instructions, wherein when executed by a processor, the instructions perform operations comprising:]
receiving, as an input, sporting event data, the sporting event data including {geospatial} data and {labeled} event data based on the sporting event;
receiving, as an input, broadcast footage of a sporting event; receiving event data of the sporting event;
performing multi-object tracking of one or more agents based on the received {geospatial} data to determine one or more vectors;
performing multi-object tracking of one or more agents of the broadcast tracking data to determine one or more vectors;
inputting the labeled event data and one or more vectors into a diffusion model;
inputting the one or more vectors and event data into a multimodal model, determining, by the multimodal model, a tensor, inputting … the tensor into a diffusion model
wherein the diffusion model includes a decoder;
the diffusion model incorporating a {transformer based-neural network}
wherein the diffusion model includes a decoder;
and determining, using the diffusion model, one or more trajectory sequences for the one or more agents {by tokenizing the labeled event data using a linear projection to generate tokenized labeled event data; processing the tokenized labeled event data with stacked encoders; and outputting event embeddings based on the processing}.
generating, by the diffusion model, one or more trajectories for the one or more players in the sporting event.
Claims 1, 11 and 18 of the ‘109 application, while reciting “broadcast tracking data of a sporting event” does not recite the data specifically as being “geospatial data” or that the event data is “labeled” however, Salzmann teaches “geospatial data” (Salzmann page 7, LIDAR data incorporated into the event/agent representation vectors), and labeled event data (Salzmann page 5, section 4, fig. 2, agents are classified (car, bus, pedestrian), thus labeled and are nodes in a spatiotemporal graph, along with edges that represent interactions (events)).
Further, claims 1, 11 and 18 of the ‘109 application do not however Zhou teaches the diffusion model incorporating a transformer based-neural network; (Zhou Abstract and page 3, section 3, framework for determining human motion sequences including diffusion module and a unified token transformer) by tokenizing the labeled event data using a linear projection to generate tokenized labeled event data (Zhou page 4, section 3.3, left column, in the unified token transformer, for each embedding of motion data associated with a sequence (labeled event data) a linear projection is applied, and as shown in fig. 2, token sequences are output); processing the tokenized labeled event data with stacked encoders (Zhou page 3, section 3.3, the unified token transformer uses a stacked encoder architecture, where fig. 2 on page 4 illustrates the token processing within the stacked encoders); and outputting event embeddings based on the processing (Zhou page 5, section 3.4, fig. 2, in the diffusion motion decoder, motion sequence embeddings are extracted from the token sequence).
Therefore, taking the teachings of claims 1, 11 and 18 of the ‘109 application and Salzmann together as a whole, it would have been obvious to a “PHOSITA” before the effective filing date of the claimed invention to have modified claims 1, 11 and 18 of the ‘109 application to have the geographic data teachings cited above as disclosed by Salzmann at least because doing so would allow for modeling agents with different perception ranges. See Salzmann, bottom of page 5.
Further, taking the teachings of claims 1, 11 and 18 of the ‘109 application and Zhou together as a whole, it would have been obvious to a “PHOSITA” before the effective filing date of the claimed invention to have modified the teachings of claims 1, 7 and 13 of the ‘642 application with the processing as disclosed above by Zhou at least because doing so would allow for multi-modal motion generation with high diversity. See Zhou page 2, left column.
Regarding claims 22 and 31, claims 4 and 14 of the ‘109 application recite the same limitations.
Regarding claims 23 and 32, claims 2 and 12 of the ‘109 application recite the same limitations.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 21–23, 30–32, 34 and 39–40 are rejected under 35 U.S.C. 103 as being unpatentable over Salzmann et al., “Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data,” arXiv:2001.03093v5 [cs.RO], January 13, 2021, https://doi.org/10.48550/arXiv.2001.03093 (herein “Salzmann” – an earlier version of this reference was cited by Applicant in the IDS filed 9/18/2024), Zhou et al., “UDE: A Unified Driving Engine for Human Motion Generation,” arXiv:2211.16016v1 [cs.CV] 29 Nov 2022, https://doi.org/10.48550/arXiv.2211.16016 (herein “Zhou”) in view of Zhu et al., "Event Tactic Analysis Based on Broadcast Sports Video," in IEEE Transactions on Multimedia, vol. 11, no. 1, pp. 49-67, Jan. 2009, doi: 10.1109/TMM.2008.2008918 (herein “Zhu”).
Regarding claims 21, 30 and 39, with claim 21 as exemplary, substantive differences
between the claims noted in curly brackets {}, and deficiencies of Salzmann noted in square brackets [], Salzmann teaches {A computer implemented method for tracking one or more agents during a [sporting] event, the method comprising: - claim 21 / A system for tracking one or more agents during a [sporting] event, the system comprising: a memory configured to store processor-readable instructions; and a processor operatively connected to the memory, and configured to execute the instructions to perform operations comprising: - claim 30 / A non-transitory computer readable medium configured to store processor-readable instructions, wherein when executed by a processor, the instructions perform operations comprising: - claim 39}(Salzmann abstract, pages 5 and 9, Trajectron++ model for predicting trajectories based on tracking pedestrians in a street scene, is implemented in PyTorch on a computer running Ubuntu (instructions), the computer containing a CPU and GPUs, where CPUs and GPUs are understood to have computer readable memory for executing processes)
receiving, as an input, [sporting] event data, the [sporting] event data including geospatial data (Salzmann page 7, first full paragraph, page 2, additional information of LIDAR data (geospatial data) is included (received input) into the Trajectron++ model framework and encoded as a vector to be added to the backbone of representation vectors ex, the LIDAR data being of the environment in which the dynamics (events) of the agents is taking place), and labeled event data based on the [sporting] event (Salzmann page 5, section 4, fig. 2, agents are classified (car, bus, pedestrian), thus labeled and are nodes in a created (receiving) spatiotemporal graph, along with edges that represent interactions (events));
performing multi-object tracking of one or more agents based on the received geospatial data to determine one or more vectors (Salzmann page 6, Modeling Agent History and Encoding Agent Interactions sections, each agent (multi-object) has its current state and its history encoded (tracking), the agents being “of the received geospatial data” since the agents are tracked in the environment the LIDAR data represents, and encodings are made from the agent interactions to produce a single node representation vector ex (vector));
inputting the labeled event data and one or more vectors into a [diffusion] model (Salzmann page 7, Producing Dynamically-Feasible Trajectories section, the backbone representation ex is fed (inputting) into a decoder (model));
[the diffusion model incorporating a transformer based-neural network;] and
determining, using the [diffusion] model, one or more trajectory sequences for the one or more agents (Salzmann page 7, Producing Dynamically-Feasible Trajectories section, the decoder produces trajectories in position space) [by: tokenizing the labeled event data using a linear projection to generate tokenized labeled event data; processing the tokenized labeled event data with stacked encoders; and outputting event embeddings based on the processing].
While Salzmann teaches receiving data tracking pedestrians in a street scene, nonetheless, Salzmann does not explicitly teach “sporting event.”
Further, Salzmann does not teach that its model is a “diffusion model” or that the diffusion model is “incorporating a transformer based-neural network.” Salzmann also does not explicitly teach “by: tokenizing the labeled event data using a linear projection to generate tokenized labeled event data; processing the tokenized labeled event data with stacked encoders; and outputting event embeddings based on the processing.”
Zhou teaches a/the diffusion model incorporating a transformer based-neural network; (Zhou Abstract and page 3, section 3, framework for determining human motion sequences including diffusion module and a unified token transformer) by tokenizing the labeled event data using a linear projection to generate tokenized labeled event data (Zhou page 4, section 3.3, left column, in the unified token transformer, for each embedding of motion data associated with a sequence (labeled event data) a linear projection is applied, and as shown in fig. 2, token sequences are output); processing the tokenized labeled event data with stacked encoders (Zhou page 3, section 3.3, the unified token transformer uses a stacked encoder architecture, where fig. 2 on page 4 illustrates the token processing within the stacked encoders); and outputting event embeddings based on the processing (Zhou page 5, section 3.4, fig. 2, in the diffusion motion decoder, motion sequence embeddings are extracted from the token sequence).
Zhu teaches “sporting event” (Zhu fig. 1, page 52, section III, attack event extraction performed on input broadcast soccer video of a soccer game (sporting event)).
Further, taking the teachings of Salzmann and Zhou together as a whole, it would have been obvious to a “PHOSITA” before the effective filing date of the claimed invention to have modified the event data processing teachings of Salzmann with the processing as disclosed above by Zhou at least because doing so would allow for multi-modal motion generation with high diversity. See Zhou page 2, left column.
Further, taking the teachings of Salzmann as modified by Gu and Zhu together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the pedestrian event data and LIDAR data of Salzmann to be sporting event data of the sporting event as disclosed by Zhu at least because doing so would allow for discovering tactic patterns amongst professional athletes, in order to improve team performance during a game. See Zhu abstract, section I.
Regarding claims 22 and 31, with claim 22 as exemplary, Salzmann does not explicitly teach, but Zhu teaches wherein the labeled event data includes a sequential stream of one or more major events throughout a sport event, the one or more major events including at least one of a pass, shot, tackle, foul, turnover, penalty, goal, score, or substitution from the sporting event (Zhu pages 52–53, section III A, fig. 2, web-casting text describing the event that has happened in a game with a timestamp and brief description such as “goal” or “headed in” for a scoring event, also called a “shot”), and wherein the geospatial data includes one or more of a sports broadcast footage, in-venue footage, global positioning system (GPS) data, near field communication (NFC) data, or radio-frequency identification (RFID) data (Zhu page 53, section B, broadcast soccer videos are analyzed and input into the analysis model).
Further, taking the teachings of Salzmann as modified by Gu and Zhu together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the pedestrian event data and LIDAR data of Salzmann to be major events in a sports game like a goal or score or shot as disclosed by Zhu at least because doing so would allow for discovering tactic patterns amongst professional athletes, in order to improve team performance during a game. See Zhu abstract, section I.
Regarding claims 23 and 32, with claim 23 as exemplary, Salzmann teaches wherein the one or more vector includes at least one of an agent two dimensional coordinates on a sporting event's field, an agent position, an agent team, an indicator indicating the agent is a ball, or player visibility information (given that the claim only requires “at least one,” Salzmann teaches on page 5, the spatiotemporal graph which is converted to the representation vector ex includes 2D world positions of agents (an agent position)).
Regarding claims 34 and 40, with claim 34 as exemplary, and deficiencies of Salzmann noted in square brackets [], Salzmann teaches wherein determining, using the [diffusion] model, one or more trajectory sequences for the one or more agents further includes (Salzmann page 7, Producing Dynamically-Feasible Trajectories section, the decoder produces trajectories in position space): receiving as input the labeled event data (Salzmann page 5, section 4, fig. 2, agents are classified thus labeled and are nodes in a created (receiving) graph), the labeled event data being an event stream of data represented as a two dimensional spatiotemporal grid, the grid representing a stacking of each of the one or more of the labeled events of the agents (Salzmann page 5, section 4, fig. 2, a scene including agents as nodes and edges as the agents’ interactions (events) is abstracted as a spatiotemporal graph (grid) arranging in a graph pattern (stacking) the agents and their interactions); and
applying a temporal convolution to the event data (Salzmann page 7, lines 1–6, for each modeled agent (including the graph), a local map is encoded using convolutional layers, where page 6 teaches the modeled agent consists of a time based sequence, thus the convolutional layers being temporal) and inputting an output to the encoder (Salzmann page 7, lines 7–10, additional information added to the local map generated by (an output) the convolutional network, the additional information being added via encoding (input to the encoder)).
Claims 24 and 33 are rejected under 35 U.S.C. 103 as being unpatentable over Salzmann in view of Zhou in view of Zhu, as set forth above regarding claims 21 and 30 from which claims 24 and 33 respectively depend, further in view of Alcorn et al., "baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotemporal Modeling," arXiv:2102.03291v3, September 28, 2021, https://doi.org/10.48550/arXiv.2102.03291 (herein “Alcorn”).
Regarding claims 24 and 33, with claim 24 as exemplary, Salzmann teaches wherein determining, using the diffusion model, one or more trajectory sequences for the one or more agents, as set forth above in the rejection rationale for claims 21 and 30, but does not explicitly teach, where Alcorn teaches further including: applying spatiotemporal axial attention to the labeled event data and the one or more vectors (Alcorn page 2, fig. 1, transformer model applies self-attention, given the locations of the players (spatio) through time (temporal) using a self-attention mask tensor (multiple dimensional/axial) on multi-entity (one or more vectors) sequential data (event data)).
Therefore, taking the teachings of Salzmann as modified by Gu and Zhu and Alcorn together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified to have the model of Salzmann to apply spatiotemporal axial attention teachings cited above as disclosed by Alcorn at least because doing so would allow for capturing idiosyncratic qualities of players, thus providing a more detailed analysis of game play for a particular sport. See Alcorn, bottom of page 2.
Claims 35–36 are rejected under 35 U.S.C. 103 as being unpatentable over Salzmann in view of Zhou in view of Zhu, as set forth above regarding claim 34 from which claim 35 depends, further in view of Hori et al., “Attention-Based Multimodal Fusion for Video Description,” arXiv:1701.03126v2 [cs.CV] 9 Mar 2017, https://doi.org/10.48550/arXiv.1701.03126 (herein “Hori”).
Regarding claim 35, Salzmann does not explicitly teach where Hori teaches applying, using the tracking decoder, attention to embed the one or more vectors with the event embeddings (Hori pages 4–5, section 4, attention based multi-modal fusion applied in a decoder (tracking decoder) to input sequence vectors which can be encoded motion features (event embeddings)).
Therefore, taking the teachings of Salzmann as modified above and Hori as a whole, it
would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the tracking processing of Salzmann to have the decoder network with attention-based multimodal fusion as disclosed by Hori at least because doing so would provide a robustness strategy for models producing summary output for multimodal input. See Hori page 2, top right column.
Regarding claim 36, Salzmann teaches the system further including: applying a temporal convolution to the one or more vectors (Salzmann page 7, lines 1–6, for each modeled agent (including the graph), a local map is encoded using convolutional layers, where page 6 teaches the modeled agent consists of a time based sequence, thus the convolutional layers being temporal) and inputting the output into the tracking decoder (Salzmann page 7, output of the convolutional neural network and encoding and adding to representation vectors which are then fed into a GRU decoder); receiving the event embeddings within the tracking decoder (Salzmann page 7, last paragraph, GRU decoder (tracking decoder) is fed (receiving) the representation vector (event embeddings)).
Salzmann does not explicitly teach where Hori teaches and applying attention to embed and fuse the one or more vectors and the event embeddings (Hori fig. 4, pages 4–5, decoder performing attention-based multimodal fusion on different modality embedding vectors, including both annotated video and other modality vectors, where page 2 left column teaches the additional modality as motion features derived from the video).
Therefore, taking the teachings of Salzmann as modified above and Hori as a whole, it
would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the tracking processing of Salzmann to have the decoder network with attention-based multimodal fusion as disclosed by Hori at least because doing so would provide a robustness strategy for models producing summary output for multimodal input. See Hori page 2, top right column.
Allowable Subject Matter
Claims 25 and 37, and therefore claims 26–29, and 38 which depend from claims 25 and 37 respectively, are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and that any claim objections and the obviousness non-statutory double patenting issues given above are addressed.
Specifically, claim 25 recites “tokenizing the labeled event data using a linear projection; applying sinusoidal positional embeddings to specify temporal occurrences of the event data,; processing the event data with stacked encoders; and outputting event embeddings” which is not taught or suggested in Salzmann, Gu, Zhu or Alcorn. Further, additionally cited, but not used in any of the above rejections, Strudel et al., US PgPub No. US 2024/0119261, while setting forth using a diffusion model that uses a sequence of discrete tokens, and applies a linear projection to convert tokens, does not teach that the linear projection is used for the tokenizing, much less tokenizing the labeled event data, as claimed. Further, none of Salzmann, Gu, Zhu, Alcorn, or Strudel, or any of the other cited art of record, whether considered alone or in an obvious combination to a PHOSITA teaches or suggests “applying sinusoidal positional embeddings to specify temporal occurrences of the event data,” and all other limitations from claim 25, and thus, this claim, and any other claims which depend therefrom, are allowable over the cited art of record.
Regarding claim 37, the closest prior art combination includes Salzmann in view of Zhou in view of Zhu, further in view of Hori as applied above regarding claims 30, 34, 35 and 36 from which claim 37 depends. While the combination of Salzmann, Zhou, Zhu and Hori does teach multi-modal attention (Hori) which would include temporal as well as spatial attention and aspects of cross attention given multiple modalities, none of Salzmann, Zhou, Zhu and Hori whether considered alone or in a combination obvious to a person having ordinary skill in the art teaches or suggests all of the limitations in claim 37 including intervening claims, and including the applying normalization and feedforward layers; and outputting joint encoding of each agent’s event and broadcast tracking streams as claimed.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE M KOETH whose telephone number is (571)272-5908. The examiner can normally be reached Monday-Thursday, 09:00-17:00, Friday 09:00-13:00, EDT/EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MICHELLE M. KOETH
Primary Examiner
Art Unit 2671
/MICHELLE M KOETH/Primary Examiner, Art Unit 2671
Prosecution Insights