DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “180” in Fig. 1 has been used to designate both the application programming interface (API) system and the application in the client device. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 10-17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Chang et al. (US 20200074181 A1) in view of Huke et al. (US 20240046763 A1), hereinafter Chang and Huke respectively.
Regarding claim 1, Chang teaches a method (Paragraph 0063 – “FIG. 55 depicts a flow chart that illustrates various steps executed”; Note: the steps make-up a method) comprising:
receiving, by a computing system (Paragraph 0494 – “an interactive game system 5300 may be implemented in the context of any suitable system (e.g., as a software application running on One or More Mobile Computing Devices 5310, as an overlay to an underlying software application running on the One or More Mobile Computing Devices 5310, as a data processing system utilizing one or more servers to perform particular processing steps, as part of the system 4800 described herein”), data for a game, the data comprising at least tracking data or event data (Paragraph 0500, 0502 – “the system is configured to retrieve a listing of available players from one or more database…the system is configured to determine spatiotemporal event data for at least the selected one or more players during the sporting or other event. Various embodiments of a system configured to retrieve and/or analyze spatiotemporal event data for at least the selected one or more players during the sporting or other event”);
determining, by the computing system, an occurrence of a first trigger event (spatio-temporal event [Wingdings font/0xE0] see quote below) within the game based on the data for the game (Paragraph 0521, 0523 – “a spatiotemporal event may include, for example: (1) a particular play during the sporting event; (2) a particular time period during the sporting event (e.g., a quarter, half, etc.); (3) a particular incident…the system is configured to analyze the spatiotemporal event data to identify one or more spatiotemporal events during the sporting event”; Note: the spatiotemporal event is equivalent to an occurrence of a first trigger event within a game. It is determined based on game event data);
providing, by the computing system, the data for the game and the first trigger event (spatio-temporal event [Wingdings font/0xE0] see quote below) to a system, wherein the system generates a first graphic (custom graphical user interface [Wingdings font/0xE0] see quote below) based on the data for the game and the occurrence of the first trigger event (Paragraph 0497, 0510 – “when executing an Interactive Game Module 5500, the interactive game system 5300 is configured to:…(2) identify events that occur during the sporting event that involve the selected one or more players; (3) determine a score for each event for each of the one or more players involved in the event (e.g., based on one or more scoring criteria)…(5) display a graphical user interface over (e.g., and/or in conjunction with) a video feed of the sporting event (e.g., on a mobile computing device 5310) that includes one or more indications related to, for example: the scoring data for the user, one or more event scores, scoring data for one or more other users…the system is configured to generate one or more custom user interfaces based at least in part on one or more of the one or more user provided inputs, the spatiotemporal event data, the spatiotemporal event score for each of the plurality of spatiotemporal events, and the one or more scoring criteria”; Note: it is implied that the game data and the first trigger event (spatiotemporal event) are provided to the interactive game system since the system uses them to generate a custom user interface, which is the first graphic);
receiving, from the system, the first graphic (custom graphical user interface [Wingdings font/0xE0] see quote below) based on the data for the game and the occurrence of the first trigger event (Paragraph 0510 – “the system is configured to generate one or more custom user interfaces based at least in part on one or more of the one or more user provided inputs, the spatiotemporal event data, the spatiotemporal event score for each of the plurality of spatiotemporal events, and the one or more scoring criteria”; Note: the custom user interface is equivalent to the first graphic, and it is generated based on game data and the spatiotemporal event (occurrence of the first trigger event));
and generating, using a first template of a plurality of templates at the computing system, a first visual element (templated video feed containing the custom graphical user interface [Wingdings font/0xE0] see quote and note below) including the first graphic for presentation within a user interface (Paragraph 0281, 0510 – “a system 4800 (e.g., the system described herein) may output one or more content feeds 4802-1, 4802-2 . . . 4802-N. The content feeds may include video, audio, text, and/or data (e.g., statistics of a game, player names)… the SDK 4804 may include one or more templates that define a manner by which the different content feeds 4802 may be laid out… the interactive game system is configured to generate (e.g., on a computing device associated with each respective user playing the interactive game) a custom graphical user interface over (e.g., in conjunction with) a video feed of the sporting event…the custom graphical user interface may augment an existing video feed of the sporting event”; Note: the custom graphical user interface (first graphic) is augmented over a video feed. Together, they make-up a content feed. A template is used to determine the layout of the content feed and produce a final result, which would be the first visual element),
wherein the first visual element is associated with a first marker (spatio-temporal index [Wingdings font/0xE0] see quote below), and wherein the first marker represents a first recommended position for a first interactive element (augmentations or advertisement [Wingdings font/0xE0] see quote below) to be presented within the user interface (Paragraph 0308, 0313, 0316 – “a wide variety of elements may be indexed temporally (e.g., in relation to individual video frames) and/or spatially (e.g., in relation to pixels, groups of pixels, or “real world” locations depicted in the video frames). Examples of elements that may be indexed include…locations of areas corresponding to the environment (e.g., floor areas, background areas, signage areas) where information, augmentations, graphics, animations, and advertising can be displayed in a frame, indicia of what information, augmentation elements, and the like that are available to augment a video feed in a content channel…spatio-temporal index identifies information associated with particular video frames of a video and/or particular locations depicted in the video frames… the spatio-temporal index may indicate a respective wall time for each video frame in a video feed (e.g., a real time at which the frame was captured/initially streamed)… the video transformation module 5208 determines the advertisement to display from the spatio-temporal index based on the particular frame and location. In embodiments, the video transformation module 5208 may retrieve the advertisement from the multimedia server (or another device). In other embodiments, the advertisement may be transmitted with the video feeds and the spatio-temporal index. In response to obtaining the advertisement, the video transformation module 5208 may determine a location at which the advertisement is to be displayed (e.g., in the manner discussed above), and may overlay the advertisement onto the video at the determined location”; Note: the video feed (visual element) is associated with a spatio-temporal index. The spatio-temporal index is a first marker that represents a first recommended position for a first interactive element (such as an augmentation or advertisement) to be presented within the user interface).
Chang does not teach the “first machine learning model” in the limitations: “providing, by the computing system, the data for the game and the first trigger event to a first machine learning model, wherein the first machine learning model is trained to generate a first graphic based on the data for the game and the occurrence of the first trigger event; receiving, from the first machine learning model, the first graphic based on the data for the game and the occurrence of the first trigger event”. However, Huke teaches providing, by the computing system, the data for the game and the first trigger event to a first machine learning model, wherein the first machine learning model is trained to generate a first graphic based on the data for the game and the occurrence of the first trigger event; and receiving, from the first machine learning model, the first graphic based on the data for the game and the occurrence of the first trigger event (Paragraph 0005 – “the method, system, and apparatus may use the generated probabilities in order to create and display probability graphics on a display device. The graphics may be generated and/or updated using artificial intelligence or machine learning. Display information may be updated in real-time as changes are made in the game (e.g. a player injury, substitutions, changing weather conditions, etc.). In some embodiments the changes may be detected by a detection system on the field”; Note: a machine learning model uses game data from detections and trigger events (changes) to generate or update graphics on display. Thus, it is implied that the game data and event are provided to the machine learning model). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the part of the system of Chang that generates a graphic could have been substituted for the machine-learning model of Huke because both the system part and machine-learning model serve the purpose of generating graphics for display based on game data and an event. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of receiving graphics generated based on game data and an event. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the system part of Chang for the machine-learning model of Huke according to known methods to yield the predictable result of receiving graphics generated based on game data and an event.
Regarding claim 2, Chang in view of Huke teaches the method of claim 1. Chang further teaches wherein the first trigger event is a pre-determined or dynamically-determined trigger event type (Paragraph 0521, 0523 – “a spatiotemporal event may include, for example: (1) a particular play during the sporting event; (2) a particular time period during the sporting event (e.g., a quarter, half, etc.); (3) a particular incident during the sporting event; (4) a particular action by one or more players during the sporting event (e.g., a pass, an attempted shot… the system is configured to analyze the spatiotemporal event data to identify one or more spatiotemporal events during the sporting event”; Note: the spatiotemporal event is equivalent to the first trigger event, and it is either a pre-determined type (a particular time period) or dynamically-determined (a particular play or action)).
Regarding claim 3, Chang in view of Huke teaches the method of claim 1. Chang further teaches wherein the first graphic includes one or more of text, a statistic, an image, a segment of broadcast video, or a link to the segment of broadcast video (Fig. 68, Paragraph 0511, 0534 – “the custom graphical user interface may include, for example, one or more indications related to: (1) scoring data for the user (e.g., the user's overall, accumulated score for the event); (2) one or more actions performed by one or more participants selected by the user (e.g., in conjunction with and/or substantially immediately after the one or more selected participants preform the one or more actions in the video feed of the sporting event)… In the user interface shown in FIG. 68, the custom user interface 6800 includes an indicium adjacent (e.g., around) the selected player in addition to an indication of a scoring event (e.g., a spatiotemporal event) associated with the selected player (e.g., +10 points for a rebound)”; Note: the custom user interface, which is equivalent to the first graphic, includes a text related to the scoring, as shown in Fig. 68).
PNG
media_image1.png
320
416
media_image1.png
Greyscale
Screenshot of Fig. 68 (taken from Chang)
Regarding claim 4, Chang in view of Huke teaches the method of claim 1. The interactive game system embodiment of Chang does not teach receiving, using the computing system, broadcast video data for the game; and determining, using a second machine learning model, a segment of the broadcast video data associated with the first graphic, wherein the first visual element includes the segment of the broadcast video data or a link to the segment of the broadcast video data. However, a different embodiment of Chang teaches receiving, using the computing system, broadcast video data for the game (Paragraph 0297-0298 – “The multimedia system 5000 may include a media datastore 5010, a communication unit 5030, and a dynamic video module 5020…the media metadata 5018 of a media record may include any suitable information relating to the content feed. For example, the media metadata 5018 may include an identifier of the game to which the content feed corresponds …The dynamic video module 5020 is configured to generate dynamic videos and to deliver dynamic videos to a user device 5080… the dynamic video module 5020 may utilize the media metadata 5018 of the media records 5012 stored in the media datastore 5010 to determine the selection, length, and/or sequence of the constituent media segments”; Note: the dynamic video module receives media records and media metadata, which are broadcast video data for a game); and determining, using a second machine learning model, a segment of the broadcast video data associated with the first graphic (Paragraph 0298-0299, 0510, 0512 – “The dynamic video module 5020 is configured to generate dynamic videos and to deliver dynamic videos to a user device 5080. The dynamic video module 5020 may select the media segments to include in the dynamic video 5002 in any suitable manner. In some embodiments, the dynamic video module 5020 may implement optimization and/or reinforcement learning-based approaches to determine the selection, length, and/or sequence of the constituent media segments… the dynamic video module 5020 is configured to provide personalized dynamic videos 5002. The dynamic video module 5020 may utilize user preferences (either predicted, indicated, or inferred) to customize the dynamic video…the interactive game system is configured to generate (e.g., on a computing device associated with each respective user playing the interactive game) a custom graphical user interface over (e.g., in conjunction with) a video feed of the sporting event (e.g., on the mobile computing device 5310). In various embodiments, the custom graphical user interface may augment an existing video feed of the sporting event….the system may be configured to: (1) receive a selection of one or more particular participants in an event; (2) in response to receiving the selection, retrieve and/or identify (e.g., from one or more remote servers) an existing augmented video feed that corresponds to the selection; and (3) provide the augmented video feed to a computing device associated with the user for display on the computing device”; Note: the dynamic video module, which is equivalent to a second machine learning model, determines segments of the media records associated with user preferences. User preferences include identified video feed of a user-selected participant. The identified video feed of a user-selected participant is part of the custom user interface, which is equivalent to the first graphic), wherein the first visual element includes the segment of the broadcast video data or a link to the segment of the broadcast video data (Paragraph 0298-0299, 0510, 0512 – “The dynamic video module 5020 is configured to generate dynamic videos and to deliver dynamic videos to a user device 5080. The dynamic video module 5020 may select the media segments to include in the dynamic video 5002 in any suitable manner. In some embodiments, the dynamic video module 5020 may implement optimization and/or reinforcement learning-based approaches to determine the selection, length, and/or sequence of the constituent media segments… the dynamic video module 5020 is configured to provide personalized dynamic videos 5002. The dynamic video module 5020 may utilize user preferences (either predicted, indicated, or inferred) to customize the dynamic video…the interactive game system is configured to generate (e.g., on a computing device associated with each respective user playing the interactive game) a custom graphical user interface over (e.g., in conjunction with) a video feed of the sporting event (e.g., on the mobile computing device 5310). In various embodiments, the custom graphical user interface may augment an existing video feed of the sporting event….the system may be configured to: (1) receive a selection of one or more particular participants in an event; (2) in response to receiving the selection, retrieve and/or identify (e.g., from one or more remote servers) an existing augmented video feed that corresponds to the selection; and (3) provide the augmented video feed to a computing device associated with the user for display on the computing device”; Note: the custom graphical user interface augmented over a video feed, which is equivalent to the first visual element, includes the segment of the broadcast video related to the user preference/selection of the participant). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chang to use the dynamic video module of the multimedia system to retrieve a segment of broadcast video data associated with the first graphic for the benefit of being able to display the relevant video feed desired by a user, which may help increase user engagement and dwell time. While the interactive game system of Chang has a function of incorporating a custom graphical user interface (graphic) with a video feed, it is silent on how the video feed is retrieved. Instead, the multimedia system of Chang has a dynamic video module to retrieve and/or generate specific video feed. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Chang, to have obtained a collaboration between the interactive game system and the dynamic video module of the multimedia system, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP $2143(A). The prior art included each element recited in claim 4, although not necessarily in a single embodiment, with the only difference being between the claimed element and the prior art being the lack of actual combination of certain elements in a single prior art embodiment, as described above. One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 5, Chang in view of Huke teaches the method of claim 1. Chang further teaches wherein the interactive element represents an advertisement or a market-based prediction offer (Paragraph 0308, 0316, 0477 – “the video player may allow a user to interact with the video, such that the user can request the video player to display information relating to a time and/or location in the video feed, display relevant information relating to the event, switch between video feeds of the event, view advertisements, and the like…the video transformation module 5208 may receive a command to display an advertisement corresponding to a particular frame and location…In response to obtaining the advertisement, the video transformation module 5208 may determine a location at which the advertisement is to be displayed (e.g., in the manner discussed above), and may overlay the advertisement onto the video at the determined location… the custom graphical user interface may include, for example, one or more indications related to: (1) scoring data for the user (e.g., the user's overall, accumulated score for the event); (2) one or more actions performed by one or more participants selected by the user (e.g., in conjunction with and/or substantially immediately after the one or more selected participants preform the one or more actions in the video feed of the sporting event; (3) scoring data for one or more other users (e.g., one or more other users with whom the user is competing in the interactive game; (4) etc.”; Note: the indication in the custom graphical user interface, which is an interactive element, can be an advertisement), and wherein the interactive element is selected based on at least one of the first visual element or user behavioral data (Paragraph 0313, 0416 – “the spatio-temporal index may indicate a respective wall time for each video frame in a video feed (e.g., a real time at which the frame was captured/initially streamed)….the video transformation module 5208 may receive a command to display an advertisement corresponding to a particular frame and location. In response to the command, the video transformation module 5208 determines the advertisement to display from the spatio-temporal index based on the particular frame and location. In embodiments, the video transformation module 5208 may retrieve the advertisement from the multimedia server (or another device). In other embodiments, the advertisement may be transmitted with the video feeds and the spatio-temporal index. In response to obtaining the advertisement, the video transformation module 5208 may determine a location at which the advertisement is to be displayed (e.g., in the manner discussed above), and may overlay the advertisement onto the video at the determined location”; Note: the interactive element (advertisement) is selected based on the spatio-temporal index based on the particular frame and location. The spatio-temporal index represents data from the video feed, which is the first visual element).
Regarding claim 6, Chang in view of Huke teaches the method of claim 1. Chang further teaches wherein the first marker is associated with a period of time of the game (Paragraph 0313 – “spatio-temporal index identifies information associated with particular video frames of a video and/or particular locations depicted in the video frames. In some of these embodiments, the locations may be locations in relation to a playing surface (e.g., at the fifty yard line or at the free throw line) or defined in relation to individual pixels or groups of pixels. It is noted that the pixels may be two-dimensional pixels or three-dimensional pixels (e.g., voxels). The spatio-temporal index may index participants on a playing surface (e.g., players on a basketball court), statistics relating to the participants (e.g., Player A has scored 32 points)…advertisements, score bugs, graphics, and the like. In some embodiments, the spatio-temporal index may index wall times corresponding to various frames. For example, the spatio-temporal index may indicate a respective wall time for each video frame in a video feed (e.g., a real time at which the frame was captured/initially streamed)”; Note: the spatio-temporal index, which is the marker, is associated with a time of the game since it is “temporal”).
Regarding claim 7, Chang in view of Huke teaches the method of claim 1. Chang further teaches wherein the first visual element is generated in near real time (Paragraph 0448, 0515 – “The services and outputs as discussed herein may be employed in different applications with varying time delays relative to the actual occurrence of the sporting event. For example, the actual event may occur at a time T1 and the content feeding or broadcasting may occur at a time T2 with a time delay of T2-T1. The time delay may be small such as of a few seconds so as the content is useful in a live commentary or augmentation of a live video…the system may be configured to display the video feed on a delay that corresponds to the particular processing time. In such embodiments, the system may, for example, be configured to: (1) receive spatiotemporal event data as the sporting event is occurring; (2) process the spatiotemporal event as the spatiotemporal event data is received; (3) display the video feed of the event on the mobile computing device (e.g., or other device) on a time delay that at least generally corresponds to a processing time for processing the spatiotemporal event data to determine scoring data for each particular spatiotemporal event that occurs during the sporting event; and (4) display the custom user interface along with the time delayed video feed such that the system may display user interface features in a manner that the features are displayed at a time that corresponds to an occurrence of the one or more spatiotemporal events during the sporting event”; Note: the video feed with the custom graphical user interface, which is the first visual element, is generated in near real time or in other words, is displayed with a small time delay from the actual occurrence of the event).
Regarding claim 10, Chang in view of Huke teaches the method of claim 1. Chang further teaches outputting, using a second machine learning model, the event data (Paragraph 0067, 0168 – “Event recognition may be based on pattern recognition by machine learning, such as spatiotemporal pattern recognition, and in some cases, may be augmented, confirmed, or aided by human feedback… For example, if spatiotemporal machine learning identifies a segment of a video as showing a pick and roll involving two players, then that video segment may be tagged, so that when that event is found (either by browsing or by filtering to that situation), the video can be displayed. Because the machine understands the precise moment that an event occurs in the video, a user-customizable segment of video can be created”; Note: a video segment of a specific event recognized by a machine learning model is output).
Regarding claim 11, Chang teaches a non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by one or more processors, causes a computing system to perform operations (Paragraph 0487, 0489-0490, 0492-0493 – “FIG. 54 illustrates a diagrammatic representation of a computer architecture 5400 that can be used within the interactive game system 5300…An exemplary computer 5400 includes a processing device 5402…The processing device 5402 may be configured to execute processing logic 5426 for performing various operations and steps discussed herein…The data storage device 5418 may include a non-transitory computer-accessible storage medium 5430 (also known as a non-transitory computer-readable storage medium or a non-transitory computer-readable medium) on which is stored one or more sets of instructions (e.g., software 5422) embodying any one or more of the methodologies or functions described herein… The term “computer-accessible storage medium” should also be understood to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the computer and that cause the computer to perform any one or more of the methodologies of the present invention”) comprising:
receiving, by the computing system (Paragraph 0494 – “an interactive game system 5300 may be implemented in the context of any suitable system (e.g., as a software application running on One or More Mobile Computing Devices 5310, as an overlay to an underlying software application running on the One or More Mobile Computing Devices 5310, as a data processing system utilizing one or more servers to perform particular processing steps, as part of the system 4800 described herein”), data for a game, the data comprising at least tracking data or event data (Paragraph 0500, 0502 – “the system is configured to retrieve a listing of available players from one or more database…the system is configured to determine spatiotemporal event data for at least the selected one or more players during the sporting or other event. Various embodiments of a system configured to retrieve and/or analyze spatiotemporal event data for at least the selected one or more players during the sporting or other event”);
determining, by the computing system, an occurrence of a first trigger event (spatio-temporal event [Wingdings font/0xE0] see quote below) within the game based on the data for the game (Paragraph 0521, 0523 – “a spatiotemporal event may include, for example: (1) a particular play during the sporting event; (2) a particular time period during the sporting event (e.g., a quarter, half, etc.); (3) a particular incident…the system is configured to analyze the spatiotemporal event data to identify one or more spatiotemporal events during the sporting event”; Note: the spatiotemporal event is equivalent to an occurrence of a first trigger event within a game. It is determined based on game event data);
providing, by the computing system, the data for the game and the first trigger event (spatio-temporal event [Wingdings font/0xE0] see quote below) to a system, wherein the system generates a first graphic (custom graphical user interface [Wingdings font/0xE0] see quote below) based on the data for the game and the occurrence of the first trigger event (Paragraph 0497, 0510 – “when executing an Interactive Game Module 5500, the interactive game system 5300 is configured to:…(2) identify events that occur during the sporting event that involve the selected one or more players; (3) determine a score for each event for each of the one or more players involved in the event (e.g., based on one or more scoring criteria)…(5) display a graphical user interface over (e.g., and/or in conjunction with) a video feed of the sporting event (e.g., on a mobile computing device 5310) that includes one or more indications related to, for example: the scoring data for the user, one or more event scores, scoring data for one or more other users…the system is configured to generate one or more custom user interfaces based at least in part on one or more of the one or more user provided inputs, the spatiotemporal event data, the spatiotemporal event score for each of the plurality of spatiotemporal events, and the one or more scoring criteria”; Note: it is implied that the game data and the first trigger event (spatiotemporal event) are provided to the interactive game system since the system uses them to generate a custom user interface, which is the first graphic);
receiving, from the system, the first graphic (custom graphical user interface [Wingdings font/0xE0] see quote below) based on the data for the game and the occurrence of the first trigger event (Paragraph 0510 – “the system is configured to generate one or more custom user interfaces based at least in part on one or more of the one or more user provided inputs, the spatiotemporal event data, the spatiotemporal event score for each of the plurality of spatiotemporal events, and the one or more scoring criteria”; Note: the custom user interface is equivalent to the first graphic, and it is generated based on game data and the spatiotemporal event (occurrence of the first trigger event));
and generating, using a first template of a plurality of templates at the computing system, a first visual element (templated video feed containing the custom graphical user interface [Wingdings font/0xE0] see quote and note below) including the first graphic for presentation within a user interface (Paragraph 0281, 0510 – “a system 4800 (e.g., the system described herein) may output one or more content feeds 4802-1, 4802-2 . . . 4802-N. The content feeds may include video, audio, text, and/or data (e.g., statistics of a game, player names)… the SDK 4804 may include one or more templates that define a manner by which the different content feeds 4802 may be laid out… the interactive game system is configured to generate (e.g., on a computing device associated with each respective user playing the interactive game) a custom graphical user interface over (e.g., in conjunction with) a video feed of the sporting event…the custom graphical user interface may augment an existing video feed of the sporting event”; Note: the custom graphical user interface (first graphic) is augmented over a video feed. Together, they make-up a content feed. A template is used to determine the layout of the content feed and produce a final result, which would be the first visual element),
wherein the first visual element is associated with a first marker (spatio-temporal index [Wingdings font/0xE0] see quote below), and wherein the first marker represents a first recommended position for a first interactive element (augmentations or advertisement [Wingdings font/0xE0] see quote below) to be presented within the user interface (Paragraph 0308, 0313, 0316 – “a wide variety of elements may be indexed temporally (e.g., in relation to individual video frames) and/or spatially (e.g., in relation to pixels, groups of pixels, or “real world” locations depicted in the video frames). Examples of elements that may be indexed include…locations of areas corresponding to the environment (e.g., floor areas, background areas, signage areas) where information, augmentations, graphics, animations, and advertising can be displayed in a frame, indicia of what information, augmentation elements, and the like that are available to augment a video feed in a content channel…spatio-temporal index identifies information associated with particular video frames of a video and/or particular locations depicted in the video frames… the spatio-temporal index may indicate a respective wall time for each video frame in a video feed (e.g., a real time at which the frame was captured/initially streamed)… the video transformation module 5208 determines the advertisement to display from the spatio-temporal index based on the particular frame and location. In embodiments, the video transformation module 5208 may retrieve the advertisement from the multimedia server (or another device). In other embodiments, the advertisement may be transmitted with the video feeds and the spatio-temporal index. In response to obtaining the advertisement, the video transformation module 5208 may determine a location at which the advertisement is to be displayed (e.g., in the manner discussed above), and may overlay the advertisement onto the video at the determined location”; Note: the video feed (visual element) is associated with a spatio-temporal index. The spatio-temporal index is a first marker that represents a first recommended position for a first interactive element (such as an augmentation or advertisement) to be presented within the user interface).
Chang does not teach the “first machine learning model” in the limitations: “providing, by the computing system, the data for the game and the first trigger event to a first machine learning model, wherein the first machine learning model is trained to generate a first graphic based on the data for the game and the occurrence of the first trigger event; receiving, from the first machine learning model, the first graphic based on the data for the game and the occurrence of the first trigger event”. However, Huke teaches providing, by the computing system, the data for the game and the first trigger event to a first machine learning model, wherein the first machine learning model is trained to generate a first graphic based on the data for the game and the occurrence of the first trigger event; and receiving, from the first machine learning model, the first graphic based on the data for the game and the occurrence of the first trigger event (Paragraph 0005 – “the method, system, and apparatus may use the generated probabilities in order to create and display probability graphics on a display device. The graphics may be generated and/or updated using artificial intelligence or machine learning. Display information may be updated in real-time as changes are made in the game (e.g. a player injury, substitutions, changing weather conditions, etc.). In some embodiments the changes may be detected by a detection system on the field”; Note: a machine learning model uses game data from detections and trigger events (changes) to generate or update graphics on display. Thus, it is implied that the game data and event are provided to the machine learning model). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the part of the system of Chang that generates a graphic could have been substituted for the machine-learning model of Huke because both the system part and machine-learning model serve the purpose of generating graphics for display based on game data and an event. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of receiving graphics generated based on game data and an event. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the system part of Chang for the machine-learning model of Huke according to known methods to yield the predictable result of receiving graphics generated based on game data and an event.
Regarding claim 12, Chang in view of Huke teaches the non-transitory computer readable medium of claim 11. Chang further teaches wherein the first trigger event is a pre-determined or dynamically-determined trigger event type (Paragraph 0521, 0523 – “a spatiotemporal event may include, for example: (1) a particular play during the sporting event; (2) a particular time period during the sporting event (e.g., a quarter, half, etc.); (3) a particular incident during the sporting event; (4) a particular action by one or more players during the sporting event (e.g., a pass, an attempted shot… the system is configured to analyze the spatiotemporal event data to identify one or more spatiotemporal events during the sporting event”; Note: the spatiotemporal event is equivalent to the first trigger event, and it is either a pre-determined type (a particular time period) or dynamically-determined (a particular play or action)).
Regarding claim 13, Chang in view of Huke teaches the non-transitory computer readable medium of claim 11. Chang further teaches wherein the first graphic includes one or more of text, a statistic, an image, a segment of broadcast video, or a link to the segment of broadcast video (Fig. 68, Paragraph 0511, 0534 – “the custom graphical user interface may include, for example, one or more indications related to: (1) scoring data for the user (e.g., the user's overall, accumulated score for the event); (2) one or more actions performed by one or more participants selected by the user (e.g., in conjunction with and/or substantially immediately after the one or more selected participants preform the one or more actions in the video feed of the sporting event)… In the user interface shown in FIG. 68, the custom user interface 6800 includes an indicium adjacent (e.g., around) the selected player in addition to an indication of a scoring event (e.g., a spatiotemporal event) associated with the selected player (e.g., +10 points for a rebound)”; Note: the custom user interface, which is equivalent to the first graphic, includes a text related to the scoring, as shown in Fig. 68).
Regarding claim 14, Chang in view of Huke teaches the non-transitory computer readable medium of claim 11. The interactive game system embodiment of Chang does not teach receiving, using the computing system, broadcast video data for the game; and determining, using a second machine learning model, a segment of the broadcast video data associated with the first graphic, wherein the first visual element includes the segment of the broadcast video data or a link to the segment of the broadcast video data. However, a different embodiment of Chang teaches receiving, using the computing system, broadcast video data for the game (Paragraph 0297-0298 – “The multimedia system 5000 may include a media datastore 5010, a communication unit 5030, and a dynamic video module 5020…the media metadata 5018 of a media record may include any suitable information relating to the content feed. For example, the media metadata 5018 may include an identifier of the game to which the content feed corresponds …The dynamic video module 5020 is configured to generate dynamic videos and to deliver dynamic videos to a user device 5080… the dynamic video module 5020 may utilize the media metadata 5018 of the media records 5012 stored in the media datastore 5010 to determine the selection, length, and/or sequence of the constituent media segments”; Note: the dynamic video module receives media records and media metadata, which are broadcast video data for a game); and determining, using a second machine learning model, a segment of the broadcast video data associated with the first graphic (Paragraph 0298-0299, 0510, 0512 – “The dynamic video module 5020 is configured to generate dynamic videos and to deliver dynamic videos to a user device 5080. The dynamic video module 5020 may select the media segments to include in the dynamic video 5002 in any suitable manner. In some embodiments, the dynamic video module 5020 may implement optimization and/or reinforcement learning-based approaches to determine the selection, length, and/or sequence of the constituent media segments… the dynamic video module 5020 is configured to provide personalized dynamic videos 5002. The dynamic video module 5020 may utilize user preferences (either predicted, indicated, or inferred) to customize the dynamic video…the interactive game system is configured to generate (e.g., on a computing device associated with each respective user playing the interactive game) a custom graphical user interface over (e.g., in conjunction with) a video feed of the sporting event (e.g., on the mobile computing device 5310). In various embodiments, the custom graphical user interface may augment an existing video feed of the sporting event….the system may be configured to: (1) receive a selection of one or more particular participants in an event; (2) in response to receiving the selection, retrieve and/or identify (e.g., from one or more remote servers) an existing augmented video feed that corresponds to the selection; and (3) provide the augmented video feed to a computing device associated with the user for display on the computing device”; Note: the dynamic video module, which is equivalent to a second machine learning model, determines segments of the media records associated with user preferences. User preferences include identified video feed of a user-selected participant. The identified video feed of a user-selected participant is part of the custom user interface, which is equivalent to the first graphic), wherein the first visual element includes the segment of the broadcast video data or a link to the segment of the broadcast video data (Paragraph 0298-0299, 0510, 0512 – “The dynamic video module 5020 is configured to generate dynamic videos and to deliver dynamic videos to a user device 5080. The dynamic video module 5020 may select the media segments to include in the dynamic video 5002 in any suitable manner. In some embodiments, the dynamic video module 5020 may implement optimization and/or reinforcement learning-based approaches to determine the selection, length, and/or sequence of the constituent media segments… the dynamic video module 5020 is configured to provide personalized dynamic videos 5002. The dynamic video module 5020 may utilize user preferences (either predicted, indicated, or inferred) to customize the dynamic video…the interactive game system is configured to generate (e.g., on a computing device associated with each respective user playing the interactive game) a custom graphical user interface over (e.g., in conjunction with) a video feed of the sporting event (e.g., on the mobile computing device 5310). In various embodiments, the custom graphical user interface may augment an existing video feed of the sporting event….the system may be configured to: (1) receive a selection of one or more particular participants in an event; (2) in response to receiving the selection, retrieve and/or identify (e.g., from one or more remote servers) an existing augmented video feed that corresponds to the selection; and (3) provide the augmented video feed to a computing device associated with the user for display on the computing device”; Note: the custom graphical user interface augmented over a video feed, which is equivalent to the first visual element, includes the segment of the broadcast video related to the user preference/selection of the participant). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chang to use the dynamic video module of the multimedia system to retrieve a segment of broadcast video data associated with the first graphic for the benefit of being able to display the relevant video feed desired by a user, which may help increase user engagement and dwell time. While the interactive game system of Chang has a function of incorporating a custom graphical user interface (graphic) with a video feed, it is silent on how the video feed is retrieved. Instead, the multimedia system of Chang has a dynamic video module to retrieve and/or generate specific video feed. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Chang, to have obtained a collaboration between the interactive game system and the dynamic video module of the multimedia system, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP $2143(A). The prior art included each element recited in claim 14, although not necessarily in a single embodiment, with the only difference being between the claimed element and the prior art being the lack of actual combination of certain elements in a single prior art embodiment, as described above. One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 15, Chang in view of Huke teaches the non-transitory computer readable medium of claim 11. Chang further teaches wherein the interactive element represents an advertisement or a market-based prediction offer (Paragraph 0308, 0316, 0477 – “the video player may allow a user to interact with the video, such that the user can request the video player to display information relating to a time and/or location in the video feed, display relevant information relating to the event, switch between video feeds of the event, view advertisements, and the like…the video transformation module 5208 may receive a command to display an advertisement corresponding to a particular frame and location…In response to obtaining the advertisement, the video transformation module 5208 may determine a location at which the advertisement is to be displayed (e.g., in the manner discussed above), and may overlay the advertisement onto the video at the determined location… the custom graphical user interface may include, for example, one or more indications related to: (1) scoring data for the user (e.g., the user's overall, accumulated score for the event); (2) one or more actions performed by one or more participants selected by the user (e.g., in conjunction with and/or substantially immediately after the one or more selected participants preform the one or more actions in the video feed of the sporting event; (3) scoring data for one or more other users (e.g., one or more other users with whom the user is competing in the interactive game; (4) etc.”; Note: the indication in the custom graphical user interface, which is an interactive element, can be an advertisement), and wherein the interactive element is selected based on at least one of the first visual element or user behavioral data (Paragraph 0313, 0416 – “the spatio-temporal index may indicate a respective wall time for each video frame in a video feed (e.g., a real time at which the frame was captured/initially streamed)….the video transformation module 5208 may receive a command to display an advertisement corresponding to a particular frame and location. In response to the command, the video transformation module 5208 determines the advertisement to display from the spatio-temporal index based on the particular frame and location. In embodiments, the video transformation module 5208 may retrieve the advertisement from the multimedia server (or another device). In other embodiments, the advertisement may be transmitted with the video feeds and the spatio-temporal index. In response to obtaining the advertisement, the video transformation module 5208 may determine a location at which the advertisement is to be displayed (e.g., in the manner discussed above), and may overlay the advertisement onto the video at the determined location”; Note: the interactive element (advertisement) is selected based on the spatio-temporal index based on the particular frame and location. The spatio-temporal index represents data from the video feed, which is the first visual element).
Regarding claim 16, Chang in view of Huke teaches the non-transitory computer readable medium of claim 11. Chang further teaches wherein the first marker is associated with a period of time of the game (Paragraph 0313 – “spatio-temporal index identifies information associated with particular video frames of a video and/or particular locations depicted in the video frames. In some of these embodiments, the locations may be locations in relation to a playing surface (e.g., at the fifty yard line or at the free throw line) or defined in relation to individual pixels or groups of pixels. It is noted that the pixels may be two-dimensional pixels or three-dimensional pixels (e.g., voxels). The spatio-temporal index may index participants on a playing surface (e.g., players on a basketball court), statistics relating to the participants (e.g., Player A has scored 32 points)…advertisements, score bugs, graphics, and the like. In some embodiments, the spatio-temporal index may index wall times corresponding to various frames. For example, the spatio-temporal index may indicate a respective wall time for each video frame in a video feed (e.g., a real time at which the frame was captured/initially streamed)”; Note: the spatio-temporal index, which is the marker, is associated with a time of the game since it is “temporal”).
Regarding claim 17, Chang in view of Huke teaches the non-transitory computer readable medium of claim 11. Chang further teaches wherein the first visual element is generated in near real time (Paragraph 0448, 0515 – “The services and outputs as discussed herein may be employed in different applications with varying time delays relative to the actual occurrence of the sporting event. For example, the actual event may occur at a time T1 and the content feeding or broadcasting may occur at a time T2 with a time delay of T2-T1. The time delay may be small such as of a few seconds so as the content is useful in a live commentary or augmentation of a live video…the system may be configured to display the video feed on a delay that corresponds to the particular processing time. In such embodiments, the system may, for example, be configured to: (1) receive spatiotemporal event data as the sporting event is occurring; (2) process the spatiotemporal event as the spatiotemporal event data is received; (3) display the video feed of the event on the mobile computing device (e.g., or other device) on a time delay that at least generally corresponds to a processing time for processing the spatiotemporal event data to determine scoring data for each particular spatiotemporal event that occurs during the sporting event; and (4) display the custom user interface along with the time delayed video feed such that the system may display user interface features in a manner that the features are displayed at a time that corresponds to an occurrence of the one or more spatiotemporal events during the sporting event”; Note: the video feed with the custom graphical user interface, which is the first visual element, is generated in near real time or in other words, is displayed with a small time delay from the actual occurrence of the event).
Regarding claim 20, Chang teaches a computing system (Paragraph 0487 – “FIG. 54 illustrates a diagrammatic representation of a computer architecture 5400 that can be used within the interactive game system 5300, for example, as a client computer (e.g., One or More Mobile Computing Devices 5310 shown in FIG. 53), or as a server computer (e.g., One or More Interactive Game Servers 5330, One or More Spatiotemporal Event Servers 5360, etc.) shown in FIG. 53. In particular embodiments, the computer 5400 may be suitable for use as a computer within the context of the interactive game system 5300”; Note: the interactive game system is equivalent to the computing system), comprising:
a processor (Paragraph 0489 – “An exemplary computer 5400 includes a processing device 5402 (e.g., one or more computer processors)”);
and a memory having programming instructions stored thereon, which, when executed by the processor, cause the computing system to perform operations (Paragraph 0492-0493 – “The software 5422 may also reside, completely or at least partially, within the main memory 5404 and/or within the processing device 5402 during execution thereof by the computer 5400—the main memory 5404 and the processing device 5402 also constituting computer-accessible storage media… The term “computer-accessible storage medium” should also be understood to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the computer and that cause the computer to perform any one or more of the methodologies of the present invention”) comprising:
receiving, by a computing system (Paragraph 0494 – “an interactive game system 5300 may be implemented in the context of any suitable system (e.g., as a software application running on One or More Mobile Computing Devices 5310, as an overlay to an underlying software application running on the One or More Mobile Computing Devices 5310, as a data processing system utilizing one or more servers to perform particular processing steps, as part of the system 4800 described herein”), data for a game, the data comprising at least tracking data or event data (Paragraph 0500, 0502 – “the system is configured to retrieve a listing of available players from one or more database…the system is configured to determine spatiotemporal event data for at least the selected one or more players during the sporting or other event. Various embodiments of a system configured to retrieve and/or analyze spatiotemporal event data for at least the selected one or more players during the sporting or other event”);
determining, by the computing system, an occurrence of a trigger event (spatio-temporal event [Wingdings font/0xE0] see quote below) within the game based on the data for the game (Paragraph 0521, 0523 – “a spatiotemporal event may include, for example: (1) a particular play during the sporting event; (2) a particular time period during the sporting event (e.g., a quarter, half, etc.); (3) a particular incident…the system is configured to analyze the spatiotemporal event data to identify one or more spatiotemporal events during the sporting event”; Note: the spatiotemporal event is equivalent to an occurrence of a trigger event within a game. It is determined based on game event data);
providing, by the computing system, the data for the game and the trigger event (spatio-temporal event [Wingdings font/0xE0] see quote below) to a system, wherein the system generates a graphic (custom graphical user interface [Wingdings font/0xE0] see quote below) based on the data for the game and the occurrence of the trigger event (Paragraph 0497, 0510 – “when executing an Interactive Game Module 5500, the interactive game system 5300 is configured to:…(2) identify events that occur during the sporting event that involve the selected one or more players; (3) determine a score for each event for each of the one or more players involved in the event (e.g., based on one or more scoring criteria)…(5) display a graphical user interface over (e.g., and/or in conjunction with) a video feed of the sporting event (e.g., on a mobile computing device 5310) that includes one or more indications related to, for example: the scoring data for the user, one or more event scores, scoring data for one or more other users…the system is configured to generate one or more custom user interfaces based at least in part on one or more of the one or more user provided inputs, the spatiotemporal event data, the spatiotemporal event score for each of the plurality of spatiotemporal events, and the one or more scoring criteria”; Note: it is implied that the game data and the trigger event (spatiotemporal event) are provided to the interactive game system since the system uses them to generate a custom user interface, which is the graphic);
receiving, from the system, the graphic (custom graphical user interface [Wingdings font/0xE0] see quote below) based on the data for the game and the occurrence of the trigger event (Paragraph 0510 – “the system is configured to generate one or more custom user interfaces based at least in part on one or more of the one or more user provided inputs, the spatiotemporal event data, the spatiotemporal event score for each of the plurality of spatiotemporal events, and the one or more scoring criteria”; Note: the custom user interface is equivalent to the graphic, and it is generated based on game data and the spatiotemporal event (occurrence of the trigger event));
and generating, using a template of a plurality of templates at the computing system, a visual element (templated video feed containing the custom graphical user interface [Wingdings font/0xE0] see quote and note below) including the graphic for presentation within a user interface (Paragraph 0281, 0510 – “a system 4800 (e.g., the system described herein) may output one or more content feeds 4802-1, 4802-2 . . . 4802-N. The content feeds may include video, audio, text, and/or data (e.g., statistics of a game, player names)… the SDK 4804 may include one or more templates that define a manner by which the different content feeds 4802 may be laid out… the interactive game system is configured to generate (e.g., on a computing device associated with each respective user playing the interactive game) a custom graphical user interface over (e.g., in conjunction with) a video feed of the sporting event…the custom graphical user interface may augment an existing video feed of the sporting event”; Note: the custom graphical user interface (graphic) is augmented over a video feed. Together, they make-up a content feed. A template is used to determine the layout of the content feed and produce a final result, which would be the visual element),
wherein the visual element is associated with a marker (spatio-temporal index [Wingdings font/0xE0] see quote below), and wherein the marker represents a recommended position for an interactive element (augmentations or advertisement [Wingdings font/0xE0] see quote below) to be presented within the user interface (Paragraph 0308, 0313, 0316 – “a wide variety of elements may be indexed temporally (e.g., in relation to individual video frames) and/or spatially (e.g., in relation to pixels, groups of pixels, or “real world” locations depicted in the video frames). Examples of elements that may be indexed include…locations of areas corresponding to the environment (e.g., floor areas, background areas, signage areas) where information, augmentations, graphics, animations, and advertising can be displayed in a frame, indicia of what information, augmentation elements, and the like that are available to augment a video feed in a content channel…spatio-temporal index identifies information associated with particular video frames of a video and/or particular locations depicted in the video frames… the spatio-temporal index may indicate a respective wall time for each video frame in a video feed (e.g., a real time at which the frame was captured/initially streamed)… the video transformation module 5208 determines the advertisement to display from the spatio-temporal index based on the particular frame and location. In embodiments, the video transformation module 5208 may retrieve the advertisement from the multimedia server (or another device). In other embodiments, the advertisement may be transmitted with the video feeds and the spatio-temporal index. In response to obtaining the advertisement, the video transformation module 5208 may determine a location at which the advertisement is to be displayed (e.g., in the manner discussed above), and may overlay the advertisement onto the video at the determined location”; Note: the video feed (visual element) is associated with a spatio-temporal index. The spatio-temporal index is a marker that represents a recommended position for an interactive element (such as an augmentation or advertisement) to be presented within the user interface).
Chang does not teach the “machine learning model” in the limitations: “providing, by the computing system, the data for the game and the trigger event to a machine learning model, wherein the machine learning model is trained to generate a graphic based on the data for the game and the occurrence of the trigger event; receiving, from the machine learning model, the graphic based on the data for the game and the occurrence of the trigger event”. However, Huke teaches providing, by the computing system, the data for the game and the trigger event to a machine learning model, wherein the machine learning model is trained to generate a graphic based on the data for the game and the occurrence of the trigger event; and receiving, from the machine learning model, the graphic based on the data for the game and the occurrence of the trigger event (Paragraph 0005 – “the method, system, and apparatus may use the generated probabilities in order to create and display probability graphics on a display device. The graphics may be generated and/or updated using artificial intelligence or machine learning. Display information may be updated in real-time as changes are made in the game (e.g. a player injury, substitutions, changing weather conditions, etc.). In some embodiments the changes may be detected by a detection system on the field”; Note: a machine learning model uses game data from detections and trigger events (changes) to generate or update graphics on display. Thus, it is implied that the game data and event are provided to the machine learning model). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the part of the system of Chang that generates a graphic could have been substituted for the machine-learning model of Huke because both the system part and machine-learning model serve the purpose of generating graphics for display based on game data and an event. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of receiving graphics generated based on game data and an event. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the system part of Chang for the machine-learning model of Huke according to known methods to yield the predictable result of receiving graphics generated based on game data and an event.
Claims 8-9 and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Chang in view of Huke and Leventhal et al. (US 20150163562 A1), hereinafter Leventhal.
Regarding claim 8, Chang in view of Huke teaches the method of claim 1. Chang further teaches determining, by the computing system, an occurrence of a second trigger event within the game based on the data for the game (Paragraph 0521, 0523 – “a spatiotemporal event may include, for example: (1) a particular play during the sporting event; (2) a particular time period during the sporting event (e.g., a quarter, half, etc.); (3) a particular incident…the system is configured to analyze the spatiotemporal event data to identify one or more spatiotemporal events during the sporting event”; Note: a second spatiotemporal event of the one or more spatiotemporal events is equivalent to an occurrence of a second trigger event within a game. It is determined based on game event data); providing, by the computing system, the second trigger event to the system (Paragraph 0497, 0510 – “when executing an Interactive Game Module 5500, the interactive game system 5300 is configured to:…(2) identify events that occur during the sporting event that involve the selected one or more players; (3) determine a score for each event for each of the one or more players involved in the event (e.g., based on one or more scoring criteria)…(5) display a graphical user interface over (e.g., and/or in conjunction with) a video feed of the sporting event (e.g., on a mobile computing device 5310) that includes one or more indications related to, for example: the scoring data for the user, one or more event scores, scoring data for one or more other users…the system is configured to generate one or more custom user interfaces based at least in part on one or more of the one or more user provided inputs, the spatiotemporal event data, the spatiotemporal event score for each of the plurality of spatiotemporal events, and the one or more scoring criteria”; Note: it is implied that the trigger event (spatiotemporal event) is provided to the interactive game system since the system uses it to generate a custom user interface. A second spatiotemporal event of the one or more events is equivalent to the second trigger event); receiving, from the system, a second graphic based on the data for the game and the occurrence of the second trigger event (Paragraph 0510 – “the system is configured to generate one or more custom user interfaces based at least in part on one or more of the one or more user provided inputs, the spatiotemporal event data, the spatiotemporal event score for each of the plurality of spatiotemporal events, and the one or more scoring criteria”; Note: a second custom user interface is equivalent to the second graphic, and it is generated based on game data and the spatiotemporal event (second trigger event)); and generating, using a second template of the plurality of templates at the computing system, a second visual element including the second graphic for presentation within the user interface (Paragraph 0281, 0510 – “a system 4800 (e.g., the system described herein) may output one or more content feeds 4802-1, 4802-2 . . . 4802-N. The content feeds may include video, audio, text, and/or data (e.g., statistics of a game, player names)… the SDK 4804 may include one or more templates that define a manner by which the different content feeds 4802 may be laid out… the interactive game system is configured to generate (e.g., on a computing device associated with each respective user playing the interactive game) a custom graphical user interface over (e.g., in conjunction with) a video feed of the sporting event…the custom graphical user interface may augment an existing video feed of the sporting event”; Note: a second custom graphical user interface of the one or more custom graphical user interfaces (second graphic) is augmented over a video feed. Together, they make-up a content feed. A template (second template) is used to determine the layout of the content feed and produce a final result, which would be the second visual element), wherein the second visual element is associated with a second marker, wherein the second marker represents a second recommended position for a second interactive element to be presented within the user interface (Paragraph 0308, 0313, 0316 – “a wide variety of elements may be indexed temporally (e.g., in relation to individual video frames) and/or spatially (e.g., in relation to pixels, groups of pixels, or “real world” locations depicted in the video frames). Examples of elements that may be indexed include…locations of areas corresponding to the environment (e.g., floor areas, background areas, signage areas) where information, augmentations, graphics, animations, and advertising can be displayed in a frame, indicia of what information, augmentation elements, and the like that are available to augment a video feed in a content channel…spatio-temporal index identifies information associated with particular video frames of a video and/or particular locations depicted in the video frames…the video transformation module 5208 determines the advertisement to display from the spatio-temporal index based on the particular frame and location. In embodiments, the video transformation module 5208 may retrieve the advertisement from the multimedia server (or another device). In other embodiments, the advertisement may be transmitted with the video feeds and the spatio-temporal index. In response to obtaining the advertisement, the video transformation module 5208 may determine a location at which the advertisement is to be displayed (e.g., in the manner discussed above), and may overlay the advertisement onto the video at the determined location”; Note: a video feed with the second custom graphical user interface (second visual element) is associated with a spatio-temporal index. The spatio-temporal index of that video feed is a second marker that represents a second recommended position for a second interactive element (the advertisement corresponding to the index) to be presented within the user interface).
Chang does not teach the “first machine learning model” from the limitations: “providing, by the computing system, the second trigger event to the first machine learning model; receiving, from the first machine learning model, a second graphic based on the data for the game and the occurrence of the second trigger event”. However, Huke teaches providing, by the computing system, the trigger event to the first machine learning model; and receiving, from the first machine learning model, a graphic based on the data for the game and the occurrence of the trigger event (Paragraph 0005 – “the method, system, and apparatus may use the generated probabilities in order to create and display probability graphics on a display device. The graphics may be generated and/or updated using artificial intelligence or machine learning. Display information may be updated in real-time as changes are made in the game (e.g. a player injury, substitutions, changing weather conditions, etc.). In some embodiments the changes may be detected by a detection system on the field”; Note: a machine learning model uses game data from detections and trigger events (changes) to generate or update graphics on display. Thus, it is implied that the game data and event are provided to the machine learning model). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the part of the system of Chang that generates a graphic could have been substituted for the machine-learning model of Huke because both the system part and machine-learning model serve the purpose of generating graphics for display based on game data and an event. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of receiving graphics generated based on game data and an event. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the system part of Chang for the machine-learning model of Huke according to known methods to yield the predictable result of receiving graphics generated based on game data and an event.
Furthermore, Chang does not teach wherein the second visual element is configured to be re-positioned in the user interface relative to the first visual element. However, Leventhal teaches wherein the second visual element is configured to be re-positioned in the user interface relative to the first visual element (Fig. 3 and 4, Paragraph 0036, 0039 – “FIG. 3 shows a high level diagram of devices used to view a merged video feed. Video feeds 330 include any of a plurality of video feeds, such as video feeds 102, 104, and 106. Each video feed is of a different live sports game currently being broadcast via television or the internet… This merged video feed 340 comprises videos of games which are actively being played (or recently finished within the amount of time designated as "stale") and which have players on the fantasy sports team of the viewer 342. This viewing device 344 can be a hand-held wireless device…FIG. 4 shows a display device with two fantasy teams being displayed simultaneously, in an embodiment of the disclosed technology. Here, two fantasy teams 412 and 414 are shown along with their scores 420. At the snapshot in time represented in FIG. 4, Fantasy Team 412 is currently showing video feed 102 and Fantasy Team 414 is currently showing video feed 104. At a given time, the same video feeds could be shown for both”; Note: Fig. 3 shows a user interface of a first video feed (first visual element). Fig. 4 shows a user interface of a merged video feed with both the first video feed (first visual element) and second video feed (second visual element). The difference between Fig. 3 and 4 shows how the video feeds are configured to be moved relative to one another so that both are displayed and can be seen on the same user interface). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chang to incorporate the teachings of Leventhal to have the second visual element be configured to be re-positioned in the user interface relative to the first visual element for the benefit of ensuring that both visual elements can be seen by the user and that the second visual element does not block the first visual element. It can also help lead to an arrangement of the elements that is aesthetic and easy to digest.
PNG
media_image2.png
578
387
media_image2.png
Greyscale
Screenshot of Fig. 3 (taken from Leventhal)
PNG
media_image3.png
568
671
media_image3.png
Greyscale
Screenshot of Fig. 4 (taken from Leventhal)
Regarding claim 9, Chang in view of Huke and Leventhal teaches the method of claim 8. Chang further teaches wherein the second marker is associated with a period of time of the game (Paragraph 0313 – “spatio-temporal index identifies information associated with particular video frames of a video and/or particular locations depicted in the video frames. In some of these embodiments, the locations may be locations in relation to a playing surface (e.g., at the fifty yard line or at the free throw line) or defined in relation to individual pixels or groups of pixels. It is noted that the pixels may be two-dimensional pixels or three-dimensional pixels (e.g., voxels). The spatio-temporal index may index participants on a playing surface (e.g., players on a basketball court), statistics relating to the participants (e.g., Player A has scored 32 points)…advertisements, score bugs, graphics, and the like. In some embodiments, the spatio-temporal index may index wall times corresponding to various frames. For example, the spatio-temporal index may indicate a respective wall time for each video frame in a video feed (e.g., a real time at which the frame was captured/initially streamed)”; Note: the spatio-temporal index, which is the marker, is associated with a time of the game since it is “temporal”. A spatio-temporal index of a second advertisement is equivalent to the second marker).
Regarding claim 18, Chang in view of Huke teaches the non-transitory computer readable medium of claim 11. Chang further teaches determining, by the computing system, an occurrence of a second trigger event within the game based on the data for the game (Paragraph 0521, 0523 – “a spatiotemporal event may include, for example: (1) a particular play during the sporting event; (2) a particular time period during the sporting event (e.g., a quarter, half, etc.); (3) a particular incident…the system is configured to analyze the spatiotemporal event data to identify one or more spatiotemporal events during the sporting event”; Note: a second spatiotemporal event of the one or more spatiotemporal events is equivalent to an occurrence of a second trigger event within a game. It is determined based on game event data); providing, by the computing system, the second trigger event to the system (Paragraph 0497, 0510 – “when executing an Interactive Game Module 5500, the interactive game system 5300 is configured to:…(2) identify events that occur during the sporting event that involve the selected one or more players; (3) determine a score for each event for each of the one or more players involved in the event (e.g., based on one or more scoring criteria)…(5) display a graphical user interface over (e.g., and/or in conjunction with) a video feed of the sporting event (e.g., on a mobile computing device 5310) that includes one or more indications related to, for example: the scoring data for the user, one or more event scores, scoring data for one or more other users…the system is configured to generate one or more custom user interfaces based at least in part on one or more of the one or more user provided inputs, the spatiotemporal event data, the spatiotemporal event score for each of the plurality of spatiotemporal events, and the one or more scoring criteria”; Note: it is implied that the trigger event (spatiotemporal event) is provided to the interactive game system since the system uses it to generate a custom user interface. A second spatiotemporal event of the one or more events is equivalent to the second trigger event); receiving, from the system, a second graphic based on the data for the game and the occurrence of the second trigger event (Paragraph 0510 – “the system is configured to generate one or more custom user interfaces based at least in part on one or more of the one or more user provided inputs, the spatiotemporal event data, the spatiotemporal event score for each of the plurality of spatiotemporal events, and the one or more scoring criteria”; Note: a second custom user interface is equivalent to the second graphic, and it is generated based on game data and the spatiotemporal event (second trigger event)); and generating, using a second template of the plurality of templates at the computing system, a second visual element including the second graphic for presentation within the user interface (Paragraph 0281, 0510 – “a system 4800 (e.g., the system described herein) may output one or more content feeds 4802-1, 4802-2 . . . 4802-N. The content feeds may include video, audio, text, and/or data (e.g., statistics of a game, player names)… the SDK 4804 may include one or more templates that define a manner by which the different content feeds 4802 may be laid out… the interactive game system is configured to generate (e.g., on a computing device associated with each respective user playing the interactive game) a custom graphical user interface over (e.g., in conjunction with) a video feed of the sporting event…the custom graphical user interface may augment an existing video feed of the sporting event”; Note: a second custom graphical user interface of the one or more custom graphical user interfaces (second graphic) is augmented over a video feed. Together, they make-up a content feed. A template (second template) is used to determine the layout of the content feed and produce a final result, which would be the second visual element), wherein the second visual element is associated with a second marker, wherein the second marker represents a second recommended position for a second interactive element to be presented within the user interface (Paragraph 0308, 0313, 0316 – “a wide variety of elements may be indexed temporally (e.g., in relation to individual video frames) and/or spatially (e.g., in relation to pixels, groups of pixels, or “real world” locations depicted in the video frames). Examples of elements that may be indexed include…locations of areas corresponding to the environment (e.g., floor areas, background areas, signage areas) where information, augmentations, graphics, animations, and advertising can be displayed in a frame, indicia of what information, augmentation elements, and the like that are available to augment a video feed in a content channel…spatio-temporal index identifies information associated with particular video frames of a video and/or particular locations depicted in the video frames…the video transformation module 5208 determines the advertisement to display from the spatio-temporal index based on the particular frame and location. In embodiments, the video transformation module 5208 may retrieve the advertisement from the multimedia server (or another device). In other embodiments, the advertisement may be transmitted with the video feeds and the spatio-temporal index. In response to obtaining the advertisement, the video transformation module 5208 may determine a location at which the advertisement is to be displayed (e.g., in the manner discussed above), and may overlay the advertisement onto the video at the determined location”; Note: a video feed with the second custom graphical user interface (second visual element) is associated with a spatio-temporal index. The spatio-temporal index of that video feed is a second marker that represents a second recommended position for a second interactive element (the advertisement corresponding to the index) to be presented within the user interface).
Chang does not teach the “first machine learning model” from the limitations: “providing, by the computing system, the second trigger event to the first machine learning model; receiving, from the first machine learning model, a second graphic based on the data for the game and the occurrence of the second trigger event”. However, Huke teaches providing, by the computing system, the trigger event to the first machine learning model; and receiving, from the first machine learning model, a graphic based on the data for the game and the occurrence of the trigger event (Paragraph 0005 – “the method, system, and apparatus may use the generated probabilities in order to create and display probability graphics on a display device. The graphics may be generated and/or updated using artificial intelligence or machine learning. Display information may be updated in real-time as changes are made in the game (e.g. a player injury, substitutions, changing weather conditions, etc.). In some embodiments the changes may be detected by a detection system on the field”; Note: a machine learning model uses game data from detections and trigger events (changes) to generate or update graphics on display. Thus, it is implied that the game data and event are provided to the machine learning model). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the part of the system of Chang that generates a graphic could have been substituted for the machine-learning model of Huke because both the system part and machine-learning model serve the purpose of generating graphics for display based on game data and an event. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of receiving graphics generated based on game data and an event. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the system part of Chang for the machine-learning model of Huke according to known methods to yield the predictable result of receiving graphics generated based on game data and an event.
Furthermore, Chang does not teach wherein the second visual element is configured to be re-positioned in the user interface relative to the first visual element. However, Leventhal teaches wherein the second visual element is configured to be re-positioned in the user interface relative to the first visual element (Fig. 3 and 4, Paragraph 0036, 0039 – “FIG. 3 shows a high level diagram of devices used to view a merged video feed. Video feeds 330 include any of a plurality of video feeds, such as video feeds 102, 104, and 106. Each video feed is of a different live sports game currently being broadcast via television or the internet… This merged video feed 340 comprises videos of games which are actively being played (or recently finished within the amount of time designated as "stale") and which have players on the fantasy sports team of the viewer 342. This viewing device 344 can be a hand-held wireless device…FIG. 4 shows a display device with two fantasy teams being displayed simultaneously, in an embodiment of the disclosed technology. Here, two fantasy teams 412 and 414 are shown along with their scores 420. At the snapshot in time represented in FIG. 4, Fantasy Team 412 is currently showing video feed 102 and Fantasy Team 414 is currently showing video feed 104. At a given time, the same video feeds could be shown for both”; Note: Fig. 3 shows a user interface of a first video feed (first visual element). Fig. 4 shows a user interface of a merged video feed with both the first video feed (first visual element) and second video feed (second visual element). The difference between Fig. 3 and 4 shows how the video feeds are configured to be moved relative to one another so that both are displayed and can be seen on the same user interface). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chang to incorporate the teachings of Leventhal to have the second visual element be configured to be re-positioned in the user interface relative to the first visual element for the benefit of ensuring that both visual elements can be seen by the user and that the second visual element does not block the first visual element. It can also help lead to an arrangement of the elements that is aesthetic and easy to digest.
Regarding claim 19, Chang in view of Huke and Leventhal teaches the non-transitory computer readable medium of claim 18. Chang further teaches wherein the second marker is associated with a period of time of the game (Paragraph 0313 – “spatio-temporal index identifies information associated with particular video frames of a video and/or particular locations depicted in the video frames. In some of these embodiments, the locations may be locations in relation to a playing surface (e.g., at the fifty yard line or at the free throw line) or defined in relation to individual pixels or groups of pixels. It is noted that the pixels may be two-dimensional pixels or three-dimensional pixels (e.g., voxels). The spatio-temporal index may index participants on a playing surface (e.g., players on a basketball court), statistics relating to the participants (e.g., Player A has scored 32 points)…advertisements, score bugs, graphics, and the like. In some embodiments, the spatio-temporal index may index wall times corresponding to various frames. For example, the spatio-temporal index may indicate a respective wall time for each video frame in a video feed (e.g., a real time at which the frame was captured/initially streamed)”; Note: the spatio-temporal index, which is the marker, is associated with a time of the game since it is “temporal”. A spatio-temporal index of a second advertisement is equivalent to the second marker).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Nandy et al. (US 12664778 B1) teaches a method of generating visual overlays on live broadcast videos of an event based on information such as player tracking and identification.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE HAU MA whose telephone number is (571)272-2187. The examiner can normally be reached M-Th 7-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE HAU MA/ Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights