DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1, 10 and 19, and therefore claims depending therefrom are objected to because of the following informalities: these claims all recite “inputting the labeled event data and one or more vectors,” but should instead recite “inputting the labeled event data and the one or more vectors” (emphasis added), because “one or more vectors” is earlier introduced in the previous line. Appropriate correction is required.
Further regarding claim 10, line 5 recites “the memory” but intends, as best understood by the Examiner, to refer back to the earlier recited “non-transitory computer readable medium,” and thus should instead recite “the non-transitory computer readable medium.”
Further regarding claims 4 and 13, these claims recite “the event data” in line 1 but should instead recite “the labeled event data.” Appropriate correction is required.
Further regarding claims 5 and 14, these claims recite “the received event data and one or more vectors,” but “the received event data” should be recited instead as “the received labeled event data,” or simply “the labeled event data,” and “one or more vectors” should instead be recited as “the one or more vectors.” Appropriate correction is required.
Further regarding claims 7 and 16, these claims both recite “the event data,” but this should instead be recited as “the labeled event data.” Appropriate correction is required.
Further regarding claim 9, this claim recites “The method of claim 6, further including: a second tracking decoder ..” however a method claim cannot “further include” structural elements, and therefore this should be rewritten instead in a wherein clause, for example, “wherein the diffusion model further includes a second tracking decoder …” Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4 and 13, 6 and 9, and 15 and 18, and therefore claim 7 which depends from claim 6, and claim 16 which depends from claim 15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 4 and 13, these claims recite “a stacking of each player’s events” thus assuming antecedent basis for “players” however, no antecedent basis exists, and it is unclear and indefinite whether it is intended to have “player’s” be “the one or more agents,” (broadly) or if these claims intend to further limit that the “the one or more agents” are “players.”
Regarding claims 6 and 15, these claims recite “decodes trajectory sets” which does not use a definite article “the” in an effort to refer back to antecedent basis, specifically the earlier recited “trajectory sequences” of claims 1 and 10. Therefore, it is unclear an indefinite whether the claimed “trajectory sets” of claims 6 and 15 intend to refer back to the earlier recited “trajectory sequences” or if the “trajectory sets” are a new limitation altogether.
Regarding claims 9 and 18, these claims recite “trajectories” which does not use a definite article “the” in an effort to refer back to antecedent basis, specifically the earlier recited “trajectory sequences” of claims 1 and 10. Therefore, it is unclear an indefinite whether the claimed “trajectories” of claims 6 and 15 intend to refer back to the earlier recited “trajectory sequences” or if the “trajectories” are a new limitation altogether.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1–3, 10–12 and 19–20 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 21, 30 and 39 of copending Application No. 18/421,539 (herein “‘539 application”) in view of Zhu et al., "Event Tactic Analysis Based on Broadcast Sports Video," in IEEE Transactions on Multimedia, vol. 11, no. 1, pp. 49-67, Jan. 2009, doi: 10.1109/TMM.2008.2008918 (herein “Zhu”). This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the ‘539 application recite most of the limitations of the present application with correspondence to the claims is set forth below:
Regarding claims 1, 10 and 19, claims 21, 30 and 39 of the ‘539 application correspond as follows, with deficiencies of claims 21, 30 and 39 of the ‘539 application noted in curly brackets {}:
Claims 1, 10 and 19 of the present application
Claims 21, 30 and 39 of the ‘539 application
[claim 1 only: A computer implemented method for tracking one or more individuals during a sporting event, the method comprising:]
[Claim 21 only: A computer implemented method for tracking one or more individuals during a sporting event, the method comprising:]
[claim 10 only: A system for tracking one or more individuals during a sporting event, the system comprising:]
[Claim 30 only: A system for tracking one or more individuals during a sporting event, the system comprising:]
[Claim 19 only: A non-transitory computer readable medium configured to store processor-readable instructions, wherein when executed by a processor, the instructions perform operations comprising:]
[Claim 39 only: A non-transitory computer readable medium configured to store processor-readable instructions, wherein when executed by a processor, the instructions perform operations comprising:]
receiving, as an input, geospatial data of a sporting event;
receiving, as an input, sporting event data, the sporting event data including geospatial data and labeled event data based on the sporting event;
receiving, as an input, labeled event data {based on sports broadcast footage} of the sporting event;
receiving, as an input, sporting event data … including … labeled event data based on the sporting event;
performing multi-object tracking of one or more agents of the geospatial data to determine one or more vectors;
performing multi-object tracking of one or more agents based on the received geospatial data to determine one or more vectors;
inputting the labeled event data and one or more vectors into a diffusion model;
inputting the labeled event data and the one or more vectors into a diffusion model
and determining, using the diffusion model, one or more trajectory sequences for the one or more agents.
determining, using the diffusion model, one or more trajectory sequences for the one or more agents.
Claims 21, 30 and 39 of the ‘539 application do not recite “based on sports broadcast footage,” however, Zhu teaches “based on sports broadcast footage” on pages 52–53, sections A–C, web-casting text analysis teaching that webcasting text from broadcasted sports games are labels provided by sports professionals for specific game events such as goal or a heading in scoring event, and that the web-casting text is extracted and aligned with time points in the game video.
Therefore, taking the teachings of claims 21, 30 and 39 of the ‘539 application and Zhu together as a whole, it would have been obvious to a person having ordinary skill in the art (herein “PHOSITA”) before the effective filing date of the claimed invention to have modified claims 21, 30 and 39 of the ‘539 application to have the sporting event data be based on sports broadcast footage of the sporting event as disclosed by Zhu at least because doing so would allow for discovering tactic patterns amongst professional athletes, in order to improve team performance during a game. See Zhu abstract, section I.
Regarding claims 2 and 11, claims 22 and 31 of the ‘539 application recite identical limitations.
Regarding claims 3 and 12, claims 23 and 32 of the ‘539 application recite identical limitations.
Regarding 20, claim 39 of the ‘539 application recites “the non-transitory computer readable medium” and “wherein the labeled event data” but does not teach where Zhu teaches includes a sequential stream of one or more major events throughout a sport event, the major events including at least one of a pass, shot, tackle, foul, turnover, penalty, goal, score, or substitution from the sporting event, and wherein the geospatial data includes one or more of the sports broadcast footage, in-venue footage, global positioning system (GPS) data, near field communication (NFC) data, or radio- frequency identification (RFID) data (Zhu pages 52–53, sections A–C, web-casting text analysis teaching that webcasting text from broadcasted sports games are labels provided by sports professionals for specific game events such as goal or a heading in scoring event, and that the web-casting text is extracted and aligned with time points in the game video, which is sports broadcast footage).
Therefore, taking the teachings of claim 39 of the ‘539 application and Zhu together as a whole, it would have been obvious to a “PHOSITA” before the effective filing date of the claimed invention to have modified claim 39 of the ‘539 application to have the sporting event data be sports broadcast footage including a goal and score event as disclosed by Zhu at least because doing so would allow for discovering tactic patterns amongst professional athletes, in order to improve team performance during a game. See Zhu abstract, section I.
Claims 6–9 and 15–18 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 21, 25, 27, 29–30, 34–35, and 38 of copending Application No. 18/421,539 (herein “‘539 application”) in view of Zhu and further in view of Gu et al., "Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 17092-17101, doi: 10.1109/CVPR52688.2022.01660 (herein “Gu”). This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the ‘539 application recite most of the limitations of the present application with correspondence to the claims is set forth below:
Regarding claims 6 and 15, claims 21 and 30 of the ‘539 application recite “the diffusion model incorporating a transformer based-neural networks and including: an encoder; and a tracking decoder” which are recited in claims 6 and 15, but do not recite the following limitations from claims 6 and 15, where Gu does teach “wherein the event encoder encodes the labeled event data and the tracking decoder conditionally decodes trajectory sets.” (Gu page 17094, fig. 2 caption, and section 3.2, temporal-social encoder maps social interaction data (labeled event data) into a state embedding, and a decoder outputs trajectories (trajectory sets) using a Markov chain and Gaussian transitions (conditionally)).
Therefore, taking the teachings of claims 21 and 30 of the ‘539 application and Gu together as a whole, it would have been obvious to a “PHOSITA” before the effective filing date of the claimed invention to have modified claims 21 and 30 of the ‘539 application to have the encoder and decoder teachings cited above as disclosed by Gu at least because doing so would allow for predicting trajectories with a flexible indeterminacy that is capable of adapting to dynamic environment. See Gu page 17093, left column.
Regarding claims 7 and 16, the limitations of these claims are identically recited within claims 25 and 34 of the ‘539 application.
Regarding claims 8 and 17, the limitations of these claims are identically recited within claims 27 and 35 of the ‘539 application.
Regarding claims 9 and 18, the limitations of these claims are identically recited by claims 29 and 38 of the ‘539 application.
Claims 4 and 13 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 24 and 33 of copending Application No. 18/421,539 (herein “‘539 application”) in view of Zhu and further in view of Salzmann et al., “Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data,” arXiv:2001.03093v5 [cs.RO], January 13, 2021, https://doi.org/10.48550/arXiv.2001.03093 (herein “Salzmann”).
Regarding claims 4 and 13, claims 21 and 31 of the ‘539 application recite “the event data” and “each player’s events” as provided above, but do not recite where Salzmann teaches “is represented as a two dimensional spatiotemporal grid, the grid representing a stacking.” (Salzmann page 5, fig. 2, a scene of agents and their events are represented in a spatiotemporal graph with nodes and edges connected together in a graph (stacking) format).
Therefore, taking the teachings of claims 21 and 31 of the ‘539 application and Salzmann together as a whole, it would have been obvious to a “PHOSITA” before the effective filing date of the claimed invention to have modified claims 21 and 31 of the ‘539 application to have the two dimensional spatiotemporal graph teachings cited above as disclosed by Salzmann at least because doing so would allow for modeling agents with different perception ranges. See Salzmann, bottom of page 5.
Claims 5 and 14 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 24 and 33 of copending Application No. 18/421,539 (herein “‘539 application”) in view of Zhu and further in view of Alcorn et al., "baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotemporal Modeling," arXiv:2102.03291v3, September 28, 2021, https://doi.org/10.48550/arXiv.2102.03291 (herein “Alcorn”). This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the ‘539 application recite most of the limitations of the present application with correspondence to the claims is set forth below:
Regarding claims 5 and 14, claims 24 and 33 of the ‘539 application recite “wherein determining, using the diffusion model, one or more trajectory sequences for the one or more agents further includes: applying spatiotemporal axial attention to the labeled event data and the one or more vectors,” which corresponds to the claimed limitations of claims 5 and 14 of “wherein the diffusion model applies spatiotemporal axial attention on the received event data and one or more vectors.” However, claims 24 and 33 of the ‘539 application do not recite, where Alcon teaches “where self-attention is applied across temporal and spatial axis, separately.” (Alcorn, page 2, fig. 1, a self-attention head in the disclosed model is applied to the locations of the players (spatial axis) through time (temporal axis)).
Therefore, taking the teachings of claims 24 and 33 of the ‘539 application and Alcorn together as a whole, it would have been obvious to a “PHOSITA” before the effective filing date of the claimed invention to have modified claims 24 and 33 of the ‘539 application to have the self-attention is applied across temporal and spatial axis teachings cited above as disclosed by Alcorn at least because doing so would allow for capturing idiosyncratic qualities of players, thus providing a more detailed analysis of game play for a particular sport. See Alcorn, bottom of page 2.
Claims 1, 10 and 19 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 7 and 13 of copending Application No. 19/314,642 (herein “‘642 application”) in view of Salzmann et al., “Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data,” arXiv:2001.03093v5 [cs.RO], January 13, 2021, https://doi.org/10.48550/arXiv.2001.03093 (herein “Salzmann”).
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the ‘642 application recite most of the limitations of the present application with correspondence to the claims is set forth below:
Regarding claims 1, 10 and 19, claims 1, 7 and 13 of the ‘642 application correspond as follows, with deficiencies of claims 1, 7 and 13 of the ‘642 application noted in curly brackets {}:
Claims 1, 10 and 19 of the present application
Claims 1, 7 and 13 of the ‘642 application
[claim 1 only: A computer implemented method for tracking one or more individuals during a sporting event, the method comprising:]
[Claim 1 only: A computer implemented method for tracking one or more individuals during a sporting event, the method comprising:]
[claim 10 only: A system for tracking one or more individuals during a sporting event, the system comprising:]
[Claim 7 only: A system for tracking one or more individuals during a sporting event, the system comprising:]
[Claim 19 only: A non-transitory computer readable medium configured to store processor-readable instructions, wherein when executed by a processor, the instructions perform operations comprising:]
[Claim 13 only: A non-transitory computer readable medium configured to store processor-readable instructions, wherein when executed by a processor, the instructions perform operations comprising:]
receiving, as an input, {geospatial} data of a sporting event;
receiving, as an input, broadcast tracking data of a sporting event
receiving, as an input, labeled event data based on sports broadcast footage of the sporting event;
receiving, as an input, … labeled event data of the sporting event;
performing multi-object tracking of one or more agents of the geospatial data to determine one or more vectors;
performing multi-object tracking of one or more agents of the broadcast tracking data to determine one or more vectors;
inputting the labeled event data and one or more vectors into a diffusion model;
inputting the labeled event data and one or more vectors into a diffusion model;
and determining, using the diffusion model, one or more trajectory sequences for the one or more agents.
determining, using the diffusion model, one or more trajectory sequences for the one or more agents;
Claims 1, 7 and 13 of the ‘642 application, while reciting “broadcast tracking data of a sporting event” does not recite the data specifically as being “geospatial data” however, Salzmann teaches “geospatial data” on page 7, LIDAR data incorporated into the event/agent representation vectors.
Therefore, taking the teachings of claims 1, 7 and 13 of the ‘642 application and Salzmann together as a whole, it would have been obvious to a “PHOSITA” before the effective filing date of the claimed invention to have modified claims 1, 7 and 13 of the ‘642 application to have the geographic data teachings cited above as disclosed by Salzmann at least because doing so would allow for modeling agents with different perception ranges. See Salzmann, bottom of page 5.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1–4, 6, 10–13, 15 and 19–20 are rejected under 35 U.S.C. 103 as being unpatentable over Salzmann et al., “Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data,” arXiv:2001.03093v5 [cs.RO], January 13, 2021, https://doi.org/10.48550/arXiv.2001.03093 (herein “Salzmann” – an earlier version of this reference was cited by Applicant in the IDS filed 9/18/2024), in view of Gu et al., "Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 17092-17101, doi: 10.1109/CVPR52688.2022.01660 (herein “Gu”) in view of Zhu et al., "Event Tactic Analysis Based on Broadcast Sports Video," in IEEE Transactions on Multimedia, vol. 11, no. 1, pp. 49-67, Jan. 2009, doi: 10.1109/TMM.2008.2008918 (herein “Zhu”).
Regarding claims 1, 10 and 19, with claim 1 as exemplary, substantive differences
between the claims noted in curly brackets {}, and deficiencies of Salzmann noted in square brackets [], Salzmann teaches {A computer implemented method for tracking one or more individuals during a [sporting] event, the method comprising: - claim 1 / A system for tracking one or more individuals during a [sporting] event, the system comprising: a non-transitory computer readable medium configured to store processor-readable instructions; and a processor operatively connected to the memory, and configured to execute the instructions to perform operations comprising: - claim 7 / A non-transitory computer readable medium configured to store processor-readable instructions, wherein when executed by a processor, the instructions perform operations comprising: - claim 13}(Salzmann abstract, pages 5 and 9, Trajectron++ model for predicting trajectories based on tracking pedestrians in a street scene, is implemented in PyTorch on a computer running Ubuntu (instructions), the computer containing a CPU and GPUs, where CPUs and GPUs are understood to have computer readable memory for executing processes)
receiving, as an input, geospatial data of a [sporting] event (Salzmann page 7, first full paragraph, page 2, additional information of LIDAR data (geospatial data) is included (received input) into the Trajectron++ model framework and encoded as a vector to be added to the backbone of representation vectors ex, the LIDAR data being of the environment in which the dynamics (events) of the agents is taking place);
receiving, as an input, labeled event data based on [sports broadcast footage] of the [sporting] event (Salzmann page 5, section 4, fig. 2, agents are classified (car, bus, pedestrian), thus labeled and are nodes in a created (receiving) spatiotemporal graph, along with edges that represent interactions (events));
performing multi-object tracking of one or more agents of the received geospatial data to determine one or more vectors (Salzmann page 6, Modeling Agent History and Encoding Agent Interactions sections, each agent (multi-object) has its current state and its history encoded (tracking), the agents being “of the received geospatial data” since the agents are tracked in the environment the LIDAR data represents, and encodings are made from the agent interactions to produce a single node representation vector ex (vector));
inputting the labeled event data and one or more vectors into a [diffusion] model (Salzmann page 7, Producing Dynamically-Feasible Trajectories section, the backbone representation ex is fed (inputting) into a decoder (model)); and
determining, using the [diffusion] model, one or more trajectory sequences for the one or more agents (Salzmann page 7, Producing Dynamically-Feasible Trajectories section, the decoder produces trajectories in position space).
While Salzmann teaches receiving data tracking pedestrians in a street scene, nonetheless, Salzmann does not explicitly teach “sporting event.”
Further, Salzmann does not teach “sports broadcast footage” or that its model is a “diffusion model.”
Gu teaches a diffusion model (Gu page 17094, section 3.2, trajectories are determined by way of a diffusion process).
Zhu teaches “sporting event” and “sports broadcast footage” (Zhu fig. 1, page 52, section III, attack event extraction performed on input broadcast soccer video (sports broadcast footage) of a soccer game (sporting event)).
Therefore, taking the teachings of Salzmann and Gu together as a whole, it would have been obvious to a “PHOSITA” before the effective filing date of the claimed invention to have modified the model of Salzmann to include diffusion and thus be a diffusion model as disclosed by Gu at least because doing so would allow for predicting trajectories with a flexible indeterminacy that is capable of adapting to dynamic environment. See Gu page 17093, left column.
Further, taking the teachings of Salzmann as modified by Gu and Zhu together as a whole, it would have been obvious to a person having ordinary skill in the art (herein “PHOSITA”) before the effective filing date of the claimed invention to have modified the pedestrian event data and LIDAR data of Salzmann to be sporting event data and based on sports broadcast data of the sporting event as disclosed by Zhu at least because doing so would allow for discovering tactic patterns amongst professional athletes, in order to improve team performance during a game. See Zhu abstract, section I.
Regarding claims 2, 11 and 20, with claim 2 as exemplary, Salzmann does not explicitly teach, but Zhu teaches wherein the labeled event data includes a sequential stream of one or more major events throughout a sport event, the major events including at least one of a pass, shot, tackle, foul, turnover, penalty, goal, score, or substitution from the sporting event (Zhu pages 52–53, section III A, fig. 2, web-casting text describing the event that has happened in a game with a timestamp and brief description such as “goal” or “headed in” for a scoring event, also called a “shot”), and wherein the geospatial data includes one or more of the sports broadcast footage, in-venue footage, global positioning system (GPS) data, near field communication (NFC) data, or radio-frequency identification (RFID) data (Zhu page 53, section B, broadcast soccer videos are analyzed and input into the analysis model).
Further, taking the teachings of Salzmann as modified by Gu and Zhu together as a whole, it would have been obvious to a person having ordinary skill in the art (herein “PHOSITA”) before the effective filing date of the claimed invention to have modified the pedestrian event data and LIDAR data of Salzmann to be major events in a sports game like a goal or score or shot, and broadcast soccer videos as disclosed by Zhu at least because doing so would allow for discovering tactic patterns amongst professional athletes, in order to improve team performance during a game. See Zhu abstract, section I.
Regarding claims 3 and 12, with claim 3 as exemplary, Salzmann teaches wherein the one or more vector includes at least one of an agent two dimensional coordinates on a sporting event's field, an agent position, an agent team, an indicator indicating the agent is a ball, or player visibility information (given that the claim only requires “at least one,” Salzmann teaches on page 5, the spatiotemporal graph which is converted to the representation vector ex includes 2D world positions of agents (an agent position)).
Regarding claims 4 and 13, with claim 4 as exemplary, Salzmann teaches wherein the event data is represented as a two dimensional spatiotemporal grid, the grid representing a stacking of each player's events (Salzmann page 5, section 4, fig. 2, a scene including agents as nodes and edges as the agents’ interactions (events) is abstracted as a spatiotemporal graph (grid), arranging in a graph pattern (stacking) the agents and their interactions).
Regarding claims 6 and 15, with claim 6 as exemplary, Salzmann does not explicitly teach but Gu teaches wherein the diffusion model includes: an event encoder; and a tracking decoder, wherein the event encoder encodes the labeled event data (Gu page 17094, fig. 2, section 3.1, temporal-social encoder maps a history path and social interaction into a state embedding, where different pedestrians are labeled by number as shown) and the tracking decoder conditionally decodes trajectory sets (Gu page 17094, fig. 2, and section 3.2, a decoder takes yk trajectories, and decodes using a Markov chain and Gaussian transitions (conditionally)).
Therefore, taking the teachings of Salzmann and Gu together as a whole, it would have been obvious to a “PHOSITA” before the effective filing date of the claimed invention to have modified the model of Salzmann to include diffusion with an encoder and decoder as disclosed by Gu at least because doing so would allow for predicting trajectories with a flexible indeterminacy that is capable of adapting to dynamic environment. See Gu page 17093, left column.
Claims 5 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Salzmann in view of Gu in view of Zhu, as set forth above regarding claims 1 and 10 from which claims 5 and 14 respectively depend, further in view of Alcorn et al., "baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotemporal Modeling," arXiv:2102.03291v3, September 28, 2021, https://doi.org/10.48550/arXiv.2102.03291 (herein “Alcorn”).
Regarding claims 5 and 14, with claim 5 as exemplary, Salzmann does not explicitly teach, but Alcorn teaches wherein the diffusion model applies spatiotemporal axial attention on the received event data and one or more vectors (Alcorn page 2, fig. 1, transformer model applies self-attention, given the locations of the players (spatio) through time (temporal) using a self-attention mask tensor (multiple dimensional/axial) on multi-entity (one or more vectors) sequential data (event data)), where self-attention is applied across temporal and spatial axis, separately (Alcorn pages 3–4, section 2.3 teaches the self-attention mask as a tensor (multidimensional/separately) applied to an agent value at a time step (temporal and spatial axis)).
Therefore, taking the teachings of Salzmann as modified by Gu and Zhu and Alcorn together as a whole, it would have been obvious to a “PHOSITA” before the effective filing date of the claimed invention to have modified to have the model of Salzmann apply self-attention across temporal and spatial axis teachings cited above as disclosed by Alcorn at least because doing so would allow for capturing idiosyncratic qualities of players, thus providing a more detailed analysis of game play for a particular sport. See Alcorn, bottom of page 2.
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Salzmann in view of Gu in view of Zhu, as set forth above regarding claims 6 and 15 from which claims 9 and 18 respectively depend, further in view of Lohit et al., "Recovering Trajectories of Unmarked Joints in 3D Human Actions Using Latent Space Optimization," 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2021, pp. 2341-2350, doi: 10.1109/WACV48630.2021.00239 (herein “Lohit”).
Regarding claims 9 and 18, with claim 9 as exemplary, and with deficiencies of Salzmann noted in square brackets [], Salzmann teaches a second tracking decoder (Salzmann fig. 2, page 5, the decoder as shown consisting of two branches, where each branch is considered its own decoder, therefore including a second decoder, and where each decodes tracking data). Salzmann as modified above by Gu and Zhu does not explicitly teach, but Lohit teaches a transpose temporal convolution, the temporal convolution being configured to expand trajectories to their initial temporal dimensionality (Lohit page 2344, section 4, temporal convolutional autoencoder with transposed convolutional layers that minimizes the loss between an input sequence (initial temporal dimensionality) and the output of the decoder, also the autoencoder is trained on only a subset of joints observed (subset of dimensionality) such that, as disclosed in section 5, the incomplete action sequence (trajectory) is projected to the range space of the generator which is the complete set of human action sequences, and thus an expansion to an initial dimensionality in time).
Therefore taking the teachings of Salzmann as modified above by Gu and Zhu and Lohit together as a whole, it would have been obvious to a “PHOSITA” before the effective filing date of the claimed invention to have modified the trajectory processing of Salzmann with the transpose temporal convolution operations disclosed in Lohit at least because doing so would provide robust activity analysis by recovering actions and dynamics of missing joints in human movement. Lohit Abstract.
Allowable Subject Matter
Claims 7–8 and 16–17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and that any claim objections and the obviousness non-statutory double patenting issues given above are addressed.
Specifically, claims 7 and 16, and therefore claims 8 and 17 which depend therefrom, recite “tokenizing the labeled vent data using a linear projection; applying sinusoidal positional embeddings to specify temporal occurrences of the event data,” which is not taught or suggested in Salzmann, Gu, Zhu or Alcorn. Further, additionally cited, but not used in any of the above rejections, Strudel et al., US PgPub No. US 2024/0119261, while setting forth using a diffusion model that uses a sequence of discrete tokens, and applies a linear projection to convert tokens, does not teach that the linear projection is used for the tokenizing, much less tokenizing the labeled event data, as claimed. Further, none of Salzmann, Gu, Zhu, Alcorn, or Strudel, or any of the other cited art of record, whether considered alone or in an obvious combination to a PHOSITA teaches or suggests “applying sinusoidal positional embeddings to specify temporal occurrences of the event data,” and all other limitations from claims 7 and 16, and thus, these claims, and claims 8 and 17 which depend therefrom, are allowable over the cited art of record.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Darwish et al., “STE: Spatio-Temporal Encoder for Action Spotting in Soccer Videos,” Proceedings of the 5th International ACM Workshop on Multimedia Content Analysis in Sports (MMSports’22), October 10, 2022, Lisboa, Portugal, ACM, 6 pages, directed towards extracting events from video of sports games.
Tjondronegoro et al., "Multi-modal summarization of key events and top players in sports tournament videos," 2011 IEEE Workshop on Applications of Computer Vision (WACV), Kona, HI, USA, 2011, pp. 471-478, doi: 10.1109/WACV.2011.5711541, directed towards a multi-modal analysis framework to automatically detect, annotate and visualize sports summaries in matches and tournaments.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE M KOETH whose telephone number is (571)272-5908. The examiner can normally be reached Monday-Thursday, 09:00-17:00, Friday 09:00-13:00, EDT/EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MICHELLE M. KOETH
Primary Examiner
Art Unit 2671
/MICHELLE M KOETH/ Primary Examiner, Art Unit 2671