DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 7 and 17 are objected to because of the following informalities:
Claim 7 reads, “at least one of passing and serving” and should read “at least one of passing or serving”
Similar issue in claim 17
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 10 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Antecedent Basis:
Claim 10 recites the limitation "the convolutional path" in line 6. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 7, 11-15, 17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Yan, Cheng, Xin Li, and Guoqiang Li. "A new action recognition framework for video highlights summarization in sporting events." 2021 16th international conference on computer science & education (ICCSE). IEEE, 2021. (hereinafter Yan), and further in view of Y. -C. Huang, I. -N. Liao, C. -H. Chen, T. -U. İk and W. -C. Peng, "TrackNet: A Deep Learning Network for Tracking High-speed and Tiny Objects in Sports Applications," 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Taipei, Taiwan, 2019, pp. 1-8, doi: 10.1109/AVSS.2019.8909871. (hereinafter Huang).
Regarding independent claim 1, Yan discloses A method (abstract, “In this study, we present a high-accuracy framework to automatically clip the sports video stream by using a three-level prediction algorithm based on two classical open-source structures, i.e., YOLO-v3 and OpenPose.”) comprising:
accessing a human classifier configured to generate bounding boxes around humans within the set of video frames during the sports event (page 4, “Its final output is a tensor with dimension N×N×(5×B+C), where N×N, B, C are numbers of grid cell of image, number of bounding boxes and numbers of categories, respectively.” Figure 3, “Different predictions for a player action from (a) YOLO v3 (two results are obtained for one detected player, where a blue box indicates a “playing” prediction and a green box indicates a “non-playing” prediction, and (b) OpenPose (where the tracked skeleton is visualized and whether the player is “playing” or not is also printed).”);
accessing a set of highlight classifiers each configured to identify a corresponding action of a human within the set of video frames of the sports event (Figure 1, see actions as “playing and non-playing;” page 5, “where playing actions include serves, pushes, loops, etc., and non-playing actions include ball-picking, shoes lacing-up, rest, game preparation, etc.”);
capturing a video of the sports event, the video including the set of video frames (Figure 1, “Pipeline for a highlight generation process of a typical sports video:”);
applying the human classifier to generate bounding boxes around humans within the captured video of the sports event (Figure 3, “Different predictions for a player action from (a) YOLO v3 (two results are obtained for one detected player, where a blue box indicates a “playing” prediction and a green box indicates a “non-playing” prediction);
generating a highlight video by combining sets of video frames determined to include humans performing actions corresponding to the set of highlight classifiers (page 3, “As the name implies, a highlights summarization is aggregate that composed of all rally segments (where break (non-playing) segments is removed).”).
Yan fails to explicitly disclose as further recited. However, Huang discloses accessing a ball classifier configured to identify a location within a video frame of a ball and to track the location of the ball as the ball moves within a set of video frames during a sports event (abstract, “In this paper, we develop a deep learning network, called TrackNet, to track the tennis ball from broadcast videos in which the ball images are small, blurry, and sometimes with afterimage tracks or even invisible. The proposed heatmap-based deep learning network is trained to not only recognize the ball image from a single frame but also learn flying patterns from consecutive frames;” page 5, right column, “Once the detection heatmap is generated, the coordinate of the ball can be determined by the following two steps. The first step is to pixel-wisely convert the heatmap into a black-white binary heatmap by the threshold t If a pixel has a value larger than or equal to t, the pixel is set to 255. Otherwise, the pixel is set to 0. Based on the previous discussion regarding the mean radius of a tennis ball, threshold t is set as 128. The second step is to exploit the Hough Gradient Method [2] to find the circle on the black-white binary detection heatmap. If exactly one circle is identified, the centroid of the circle is returned.”);
applying the ball classifier to the captured video of the sports event to determine a movement of the ball within the captured video of the sports event (abstract, “The proposed heatmap-based deep learning network is trained to not only recognize the ball image from a single frame but also learn flying patterns from consecutive frames.” Page 4, left column, ““Trajectory Pattern” indicates the ball movement types and are classified into three categories: flying, hit, and bouncing.”);
identifying, based on the determined movement of the ball, times within the captured video that a change in direction or speed of ball movement exceeds a threshold (page 4, left column, “Trajectory Pattern” indicates the ball movement types and are classified into three categories: flying, hit, and bouncing. They are labeled by 0, 1, and 2, respectively. Figure 4 is an example of striking a ball. The ball is flying at 0021.jpg and 0022.jpg. At 0023.jpg, the ball is labeled as hit. Figure 5 shows a bouncing case. The ball has not reached the ground at 0007.jpg and 0008.jpg. At 0009.jpg, the ball hits the ground and is labeled as bouncing;”);
Yan and Huang in the combination fail to explicitly disclose for each identified time, identifying a set of bounding boxes within a threshold distance of the location of the ball within video frames and within a threshold time of the identified time and applying the set of highlight classifiers to the identified set of bounding boxes to determine if any of the humans within the bounding boxes perform the actions corresponding to the set of highlight classifiers.
However, Yan discloses detection of human bounding boxes at Figure 3, “Different predictions for a player action from (a) YOLO v3 (two results are obtained for one detected player, where a blue box indicates a “playing” prediction and a green box indicates a “non-playing” prediction. Further, Huang discloses the detection of the ball itself (abstract, “The proposed heatmap-based deep learning network is trained to not only recognize the ball image from a single frame but also learn flying patterns from consecutive frames.”). One of ordinary skill in the art before the effective filing date of the claimed invention would easily understand for sports involving a swinging implement and a ball, there must be a threshold distance between a user and the ball to determine if the user caused the change on the ball. For example, if a ball is too far from a user then the user cannot be classified as performing serving. Thus, based on the combination of Yan and Huang, it would have been obvious to a person having ordinary skill in the art before the effective filing date to further develop on the combination to confirm the user in fact was the one to perform the action causing the ball motion result.
Yan is directed toward, “In this study, we present a high-accuracy framework to automatically clip the sports video stream by using a three-level prediction algorithm based on two classical open-source structures, i.e., YOLO-v3 and OpenPose (abstract).” Huang is directed toward, “In this paper, we develop a deep learning network, called TrackNet, to track the tennis ball from broadcast videos in which the ball images are small, blurry, and sometimes with afterimage tracks or even invisible (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Yan and Huang are directed toward similar methods of endeavor of analyzing video data from sporting events. Further, one of ordinary skill in the art before the effective filing date of the claimed invention would easily understand often when viewing sports a user is interested in both the player and the ball. Said differently, often one doesn’t watch sports video to analyze only the player. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Huang in order to ensure both the player and the sporting ball are analyzed to generate a more comprehensive highlight of the sporting event.
Regarding dependent claim 2, the rejection of claim 1 is incorporated herein. Additionally, Huang in the combination further discloses wherein determining the movement of the ball includes recording positions of the ball as two-dimensional coordinates in each video frame and generating a time series of two-dimensional ball positions (abstract, “Ball trajectory data are one of the most fundamental and useful information in the evaluation of players’ performance and analysis of game strategies;” page 1, right column, “TrackNet can take multiple consecutive frames as input;” page 2, left column, “At last, the position of our target object is calculated based on the heatmap generated by the deep learning network;” page 3, left column, “A heatmap which is a scaled and truncated Gaussian distribution centered ate the ball image is then generated by TrackNet to indicate the position of the ball;” page 3, right column, ““X” and “Y” indicate the coordinate of tennis in the pixel coordinate. Due to the high moving speed, tennis images in the broadcast video may be blurry and even have afterimage trace. In such cases, “X” and “Y” are considered as the latest position of the ball's trace. ”).
One of ordinary skill in the art before the effective filing date of the claimed invention would be easily aware tracking of data often occurs in terms of coordinates, so that changes in the position can also be quantified. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Huang in order to ensure changes in the ball movement can be tracked over the time series video data.
Regarding dependent claim 3, the rejection of claim 1 is incorporated herein. Additionally, Yan and Huang in the combination fail to explicitly disclose wherein determining the movement of the ball includes recording positions of the ball as three-dimensional coordinates in each video frame and generating a time series of three-dimensional ball positions.
However, Huang does disclose at page 2, left column, “For example, the Hawk-Eye system [1] has been extensively used in professional competitions to calculate ball trajectories and assist the referee in clarifying controversial calls through 3D visual depictions.” Thus, performing similar analysis of analyzing object motion in 3D data sets has already been introduced into the field of art. Further, one of ordinary skill in the art before the effective filing date of the claimed invention would easily understand though Huang analyzes position data simply in X and Y coordinates (” page 3, right column, ““X” and “Y” indicate the coordinate of tennis in the pixel coordinate. Due to the high moving speed, tennis images in the broadcast video may be blurry and even have afterimage trace. In such cases, “X” and “Y” are considered as the latest position of the ball's trace. ”), adding a third dimension Z is not novel, and is well known to move up in dimensions. Sporting events are well known to be 3D events, in which the ball may move in 3 separate dimensions thus it would be obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to track the ball in three dimensions so that the tracking is accurate and closely models the real life movement.
Regarding dependent claim 4, the rejection of claim 1 is incorporated herein. Additionally, Huang in the combination further discloses wherein the ball classifier is further configured to determine a movement vector of the ball based on changes in positions of the ball between consecutive video frames (page 1, right column, “ we propose to adopt deep learning network to recognize the shape of the ball and learn the trajectory patterns by applying multiple consecutive frames to solve the mentioned issues.” Page 2, right column, “ By exploring the trajectory pattern from consecutive frames, the ball positioning can be effectively improved;” page 3, left column, “A heatmap which is a scaled and truncated Gaussian distribution centered ate the ball image is then generated by TrackNet to indicate the position of the ball. ”).
One of ordinary skill in the art before the effective filing date of the claimed invention would be aware often a change in the motion of an object indicates it has been acted on in some manner meaning that the human movement may also change. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Huang in order to accurately analyze the motion data to generate a quantitative vector to be analyzed in further processing.
Regarding dependent claim 5, the rejection of claim 1 is incorporated herein. Additionally, Yan in the combination further discloses wherein the human classifier is further trained to identify a plurality of joints on a body of a human and determine a pose of the human based on positions of the plurality of joints (page 2, “connecting the dozens of key points of human’s skeleton, human behaviors can be effectively tracked”), and the set of highlight classifiers determines an action performed by a human further based on the pose of the human (abstract, “In this study, we present a high-accuracy framework to automatically clip the sports video stream by using a three-level prediction algorithm based on two classical open-source structures, i.e., YOLO-v3 and OpenPose.;” page 5, “Both YOLO and OpenPose models use the same manually labeled labels (i.e., playing and non-playing), where playing actions include serves, pushes, loops, etc., and non-playing actions include ball-picking, shoes lacing-up, rest, game preparation, etc.”).
Regarding dependent claim 7, the rejection of claim 1 is incorporated herein. Additionally, Yan in the combination further discloses wherein the set of highlight classifiers are trained to identify actions specific to a given sport, and the actions include at least one of passing and serving (abstract, “In this study, we present a high-accuracy framework to automatically clip the sports video stream by using a three-level prediction algorithm based on two classical open-source structures, i.e., YOLO-v3 and OpenPose.;” page 5, “Both YOLO and OpenPose models use the same manually labeled labels (i.e., playing and non-playing), where playing actions include serves, pushes, loops, etc., and non-playing actions include ball-picking, shoes lacing-up, rest, game preparation, etc.”).
Regarding independent claim 11, the rejection of claim 1 applies directly. Additionally, Yan further discloses A non-transitory computer readable medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to perform steps (abstract, “In this study, we present a high-accuracy framework to automatically clip the sports video stream by using a three-level prediction algorithm based on two classical open-source structures, i.e., YOLO-v3 and OpenPose. It is found that by using a modest amount of sports video training data, our methodology can perform sports activity highlights clipping accurately.” The system is implemented on a computer using a program, which must be coded and stored into memory to be executed) comprising:
accessing a human classifier configured to generate bounding boxes around humans within the set of video frames during the sports event (page 4, “Its final output is a tensor with dimension N×N×(5×B+C), where N×N, B, C are numbers of grid cell of image, number of bounding boxes and numbers of categories, respectively.” Figure 3, “Different predictions for a player action from (a) YOLO v3 (two results are obtained for one detected player, where a blue box indicates a “playing” prediction and a green box indicates a “non-playing” prediction, and (b) OpenPose (where the tracked skeleton is visualized and whether the player is “playing” or not is also printed).);
accessing a set of highlight classifiers each configured to identify a corresponding action of a human within the set of video frames of the sports event (Figure 1, see actions as “playing and non-playing;” page 5, “where playing actions include serves, pushes, loops, etc., and non-playing actions include ball-picking, shoes lacing-up, rest, game preparation, etc.”);
capturing a video of the sports event, the video including the set of video frames (Figure 1, “Pipeline for a highlight generation process of a typical sports video:”);
applying the human classifier to generate bounding boxes around humans within the captured video of the sports event (Figure 3, “Different predictions for a player action from (a) YOLO v3 (two results are obtained for one detected player, where a blue box indicates a “playing” prediction and a green box indicates a “non-playing” prediction);
generating a highlight video by combining sets of video frames determined to include humans performing actions corresponding to the set of highlight classifiers (page 3, “As the name implies, a highlights summarization is aggregate that composed of all rally segments (where break (non-playing) segments is removed).”).
Yan fails to explicitly disclose as further recited. However, Huang discloses accessing a ball classifier configured to identify a location within a video frame of a ball and to track the location of the ball as the ball moves within a set of video frames during a sports event (abstract, “In this paper, we develop a deep learning network, called TrackNet, to track the tennis ball from broadcast videos in which the ball images are small, blurry, and sometimes with afterimage tracks or even invisible. The proposed heatmap-based deep learning network is trained to not only recognize the ball image from a single frame but also learn flying patterns from consecutive frames;” page 5, right column, “Once the detection heatmap is generated, the coordinate of the ball can be determined by the following two steps. The first step is to pixel-wisely convert the heatmap into a black-white binary heatmap by the threshold t If a pixel has a value larger than or equal to t, the pixel is set to 255. Otherwise, the pixel is set to 0. Based on the previous discussion regarding the mean radius of a tennis ball, threshold t is set as 128. The second step is to exploit the Hough Gradient Method [2] to find the circle on the black-white binary detection heatmap. If exactly one circle is identified, the centroid of the circle is returned.”);
applying the ball classifier to the captured video of the sports event to determine a movement of the ball within the captured video of the sports event (abstract, “The proposed heatmap-based deep learning network is trained to not only recognize the ball image from a single frame but also learn flying patterns from consecutive frames.” Page 4, left column, ““Trajectory Pattern” indicates the ball movement types and are classified into three categories: flying, hit, and bouncing.”);
identifying, based on the determined movement of the ball, times within the captured video that a change in direction or speed of ball movement exceeds a threshold (page 4, left column, “Trajectory Pattern” indicates the ball movement types and are classified into three categories: flying, hit, and bouncing. They are labeled by 0, 1, and 2, respectively. Figure 4 is an example of striking a ball. The ball is flying at 0021.jpg and 0022.jpg. At 0023.jpg, the ball is labeled as hit. Figure 5 shows a bouncing case. The ball has not reached the ground at 0007.jpg and 0008.jpg. At 0009.jpg, the ball hits the ground and is labeled as bouncing;”).
Yan and Huang in the combination fail to explicitly disclose for each identified time, identifying a set of bounding boxes within a threshold distance of the location of the ball within video frames and within a threshold time of the identified time and applying the set of highlight classifiers to the identified set of bounding boxes to determine if any of the humans within the bounding boxes perform the actions corresponding to the set of highlight classifiers.
However, Yan discloses detection of human bounding boxes at Figure 3, “Different predictions for a player action from (a) YOLO v3 (two results are obtained for one detected player, where a blue box indicates a “playing” prediction and a green box indicates a “non-playing” prediction. Further, Huang discloses the detection of the ball itself (abstract, “The proposed heatmap-based deep learning network is trained to not only recognize the ball image from a single frame but also learn flying patterns from consecutive frames.”). One of ordinary skill in the art before the effective filing date of the claimed invention would easily understand for sports involving a swinging implement and a ball, there must be a threshold distance between a user and the ball to determine if the user caused the change on the ball. For example, if a ball is too far from a user then the user cannot be classified as performing serving. Thus, based on the combination of Yan and Huang, it would have been obvious to a person having ordinary skill in the art before the effective filing date to further develop on the combination to confirm the user in fact was the one to perform the action causing the ball motion result.
Yan is directed toward, “In this study, we present a high-accuracy framework to automatically clip the sports video stream by using a three-level prediction algorithm based on two classical open-source structures, i.e., YOLO-v3 and OpenPose (abstract).” Huang is directed toward, “In this paper, we develop a deep learning network, called TrackNet, to track the tennis ball from broadcast videos in which the ball images are small, blurry, and sometimes with afterimage tracks or even invisible (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Yan and Huang are directed toward similar methods of endeavor of analyzing video data from sporting events. Further, one of ordinary skill in the art before the effective filing date of the claimed invention would easily understand often when viewing sports a user is interested in both the player and the ball. Said differently, often one doesn’t watch sports video to analyze only the player. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Huang in order to ensure both the player and the sporting ball are analyzed to generate a more comprehensive highlight of the sporting event.
Regarding dependent claim 12, the rejection of claim 11 is incorporated herein. Additionally, Huang in the combination further discloses wherein determining the movement of the ball includes recording positions of the ball as two-dimensional coordinates in each video frame and generating a time series of ball positions (abstract, “Ball trajectory data are one of the most fundamental and useful information in the evaluation of players’ performance and analysis of game strategies;” page 1, right column, “TrackNet can take multiple consecutive frames as input;” page 2, left column, “At last, the position of our target object is calculated based on the heatmap generated by the deep learning network;” page 3, left column, “A heatmap which is a scaled and truncated Gaussian distribution centered ate the ball image is then generated by TrackNet to indicate the position of the ball;” page 3, right column, ““X” and “Y” indicate the coordinate of tennis in the pixel coordinate. Due to the high moving speed, tennis images in the broadcast video may be blurry and even have afterimage trace. In such cases, “X” and “Y” are considered as the latest position of the ball's trace. ”).
One of ordinary skill in the art before the effective filing date of the claimed invention would be easily aware tracking of data often occurs in terms of coordinates, so that changes in the position can also be quantified. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Huang in order to ensure changes in the ball movement can be tracked over the time series video data.
Regarding dependent claim 13, the rejection of claim 11 is incorporated herein. Additionally, Yan and Huang in the combination fail to explicitly disclose wherein determining the movement of the ball includes recording positions of the ball as three-dimensional coordinates in each video frame and generating a time series of ball positions.
However, Huang does disclose at page 2, left column, “For example, the Hawk-Eye system [1] has been extensively used in professional competitions to calculate ball trajectories and assist the referee in clarifying controversial calls through 3D visual depictions.” Thus, performing similar analysis of analyzing object motion in 3D data sets has already been introduced into the field of art. Further, one of ordinary skill in the art before the effective filing date of the claimed invention would easily understand though Huang analyzes position data simply in X and Y coordinates (” page 3, right column, ““X” and “Y” indicate the coordinate of tennis in the pixel coordinate. Due to the high moving speed, tennis images in the broadcast video may be blurry and even have afterimage trace. In such cases, “X” and “Y” are considered as the latest position of the ball's trace. ”), adding a third dimension Z is not novel, and is well known to move up in dimensions. Sporting events are well known to be 3D events, in which the ball may move in 3 separate dimensions thus it would be obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to track the ball in three dimensions so that the tracking is accurate and closely models the real life movement.
Regarding dependent claim 14, the rejection of claim 11 is incorporated herein. Additionally, Huang in the combination further discloses wherein the ball classifier is further configured to determine a movement vector of the ball based on changes in positions of the ball between consecutive video frames (page 1, right column, “ we propose to adopt deep learning network to recognize the shape of the ball and learn the trajectory patterns by applying multiple consecutive frames to solve the mentioned issues.” Page 2, right column, “ By exploring the trajectory pattern from consecutive frames, the ball positioning can be effectively improved;” page 3, left column, “A heatmap which is a scaled and truncated Gaussian distribution centered ate the ball image is then generated by TrackNet to indicate the position of the ball. ”).
One of ordinary skill in the art before the effective filing date of the claimed invention would be aware often a change in the motion of an object indicates it has been acted on in some manner meaning that the human movement may also change. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Huang in order to accurately analyze the motion data to generate a quantitative vector to be analyzed in further processing.
Regarding dependent claim 15, the rejection of claim 11 is incorporated herein. Additionally, Yan in the combination further discloses wherein the human classifier is further trained to identify a plurality of joints on a body of a human and determine a pose of the human based on positions of the plurality of joints (page 2, “connecting the dozens of key points of human’s skeleton, human behaviors can be effectively tracked”), and the set of highlight classifiers determines an action performed by a human further based on the pose of the human (abstract, “In this study, we present a high-accuracy framework to automatically clip the sports video stream by using a three-level prediction algorithm based on two classical open-source structures, i.e., YOLO-v3 and OpenPose.;” page 5, “Both YOLO and OpenPose models use the same manually labeled labels (i.e., playing and non-playing), where playing actions include serves, pushes, loops, etc., and non-playing actions include ball-picking, shoes lacing-up, rest, game preparation, etc.”).
Regarding dependent claim 17, the rejection of claim 11 is incorporated herein. Additionally, Yan in the combination further discloses wherein the set of highlight classifiers are trained to identify actions specific to a given sport, and the actions include at least one of passing and serving (abstract, “In this study, we present a high-accuracy framework to automatically clip the sports video stream by using a three-level prediction algorithm based on two classical open-source structures, i.e., YOLO-v3 and OpenPose.;” page 5, “Both YOLO and OpenPose models use the same manually labeled labels (i.e., playing and non-playing), where playing actions include serves, pushes, loops, etc., and non-playing actions include ball-picking, shoes lacing-up, rest, game preparation, etc.”).
Regarding independent claim 20, the rejection of claim 1 applies directly. Additionally, Yan further discloses A computing system (abstract, “In this study, we present a high-accuracy framework to automatically clip the sports video stream by using a three-level prediction algorithm based on two classical open-source structures, i.e., YOLO-v3 and OpenPose. It is found that by using a modest amount of sports video training data, our methodology can perform sports activity highlights clipping accurately.” The system is implemented on a computer using a program, which must be coded and stored into memory to be executed), comprising:
one or more processors (abstract, “In this study, we present a high-accuracy framework to automatically clip the sports video stream by using a three-level prediction algorithm based on two classical open-source structures, i.e., YOLO-v3 and OpenPose. It is found that by using a modest amount of sports video training data, our methodology can perform sports activity highlights clipping accurately.” The system is implemented on a computer using a program, which must be coded and stored into memory to be executed); and
a non-transitory computer readable medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to perform steps (abstract, “In this study, we present a high-accuracy framework to automatically clip the sports video stream by using a three-level prediction algorithm based on two classical open-source structures, i.e., YOLO-v3 and OpenPose. It is found that by using a modest amount of sports video training data, our methodology can perform sports activity highlights clipping accurately.” The system is implemented on a computer using a program, which must be coded and stored into memory to be executed) comprising:
accessing a human classifier configured to generate bounding boxes around humans within the set of video frames during the sports event (page 4, “Its final output is a tensor with dimension N×N×(5×B+C), where N×N, B, C are numbers of grid cell of image, number of bounding boxes and numbers of categories, respectively.” Figure 3, “Different predictions for a player action from (a) YOLO v3 (two results are obtained for one detected player, where a blue box indicates a “playing” prediction and a green box indicates a “non-playing” prediction, and (b) OpenPose (where the tracked skeleton is visualized and whether the player is “playing” or not is also printed).”);
accessing a set of highlight classifiers each configured to identify a corresponding action of a human within the set of video frames of the sports event (Figure 1, see actions as “playing and non-playing;” page 5, “where playing actions include serves, pushes, loops, etc., and non-playing actions include ball-picking, shoes lacing-up, rest, game preparation, etc.”);
capturing a video of the sports event, the video including the set of video frames (Figure 1, “Pipeline for a highlight generation process of a typical sports video:”);
applying the human classifier to generate bounding boxes around humans within the captured video of the sports event (Figure 3, “Different predictions for a player action from (a) YOLO v3 (two results are obtained for one detected player, where a blue box indicates a “playing” prediction and a green box indicates a “non-playing” prediction);
generating a highlight video by combining sets of video frames determined to include humans performing actions corresponding to the set of highlight classifiers (page 3, “As the name implies, a highlights summarization is aggregate that composed of all rally segments (where break (non-playing) segments is removed).”).
Yan fails to explicitly disclose as further recited. However, Huang discloses
accessing a ball classifier configured to identify a location within a video frame of a ball and to track the location of the ball as the ball moves within a set of video frames during a sports event (abstract, “In this paper, we develop a deep learning network, called TrackNet, to track the tennis ball from broadcast videos in which the ball images are small, blurry, and sometimes with afterimage tracks or even invisible. The proposed heatmap-based deep learning network is trained to not only recognize the ball image from a single frame but also learn flying patterns from consecutive frames;” page 5, right column, “Once the detection heatmap is generated, the coordinate of the ball can be determined by the following two steps. The first step is to pixel-wisely convert the heatmap into a black-white binary heatmap by the threshold t If a pixel has a value larger than or equal to t, the pixel is set to 255. Otherwise, the pixel is set to 0. Based on the previous discussion regarding the mean radius of a tennis ball, threshold t is set as 128. The second step is to exploit the Hough Gradient Method [2] to find the circle on the black-white binary detection heatmap. If exactly one circle is identified, the centroid of the circle is returned.”);
applying the ball classifier to the captured video of the sports event to determine a movement of the ball within the captured video of the sports event (abstract, “The proposed heatmap-based deep learning network is trained to not only recognize the ball image from a single frame but also learn flying patterns from consecutive frames.” Page 4, left column, ““Trajectory Pattern” indicates the ball movement types and are classified into three categories: flying, hit, and bouncing.”);
identifying, based on the determined movement of the ball, times within the captured video that a change in direction or speed of ball movement exceeds a threshold (page 4, left column, “Trajectory Pattern” indicates the ball movement types and are classified into three categories: flying, hit, and bouncing. They are labeled by 0, 1, and 2, respectively. Figure 4 is an example of striking a ball. The ball is flying at 0021.jpg and 0022.jpg. At 0023.jpg, the ball is labeled as hit. Figure 5 shows a bouncing case. The ball has not reached the ground at 0007.jpg and 0008.jpg. At 0009.jpg, the ball hits the ground and is labeled as bouncing;”);
Yan and Huang in the combination fail to explicitly disclose for each identified time, identifying a set of bounding boxes within a threshold distance of the location of the ball within video frames and within a threshold time of the identified time and applying the set of highlight classifiers to the identified set of bounding boxes to determine if any of the humans within the bounding boxes perform the actions corresponding to the set of highlight classifiers. However, Yan discloses detection of human bounding boxes at Figure 3, “Different predictions for a player action from (a) YOLO v3 (two results are obtained for one detected player, where a blue box indicates a “playing” prediction and a green box indicates a “non-playing” prediction. Further, Huang discloses the detection of the ball itself (abstract, “The proposed heatmap-based deep learning network is trained to not only recognize the ball image from a single frame but also learn flying patterns from consecutive frames.”). One of ordinary skill in the art before the effective filing date of the claimed invention would easily understand for sports involving a swinging implement and a ball, there must be a threshold distance between a user and the ball to determine if the user caused the change on the ball. For example, if a ball is too far from a user then the user cannot be classified as performing serving. Thus, based on the combination of Yan and Huang, it would have been obvious to a person having ordinary skill in the art before the effective filing date to further develop on the combination to confirm the user in fact was the one to perform the action causing the ball motion result.
Yan is directed toward, “In this study, we present a high-accuracy framework to automatically clip the sports video stream by using a three-level prediction algorithm based on two classical open-source structures, i.e., YOLO-v3 and OpenPose (abstract).” Huang is directed toward, “In this paper, we develop a deep learning network, called TrackNet, to track the tennis ball from broadcast videos in which the ball images are small, blurry, and sometimes with afterimage tracks or even invisible (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Yan and Huang are directed toward similar methods of endeavor of analyzing video data from sporting events. Further, one of ordinary skill in the art before the effective filing date of the claimed invention would easily understand often when viewing sports a user is interested in both the player and the ball. Said differently, often one doesn’t watch sports video to analyze only the player. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Huang in order to ensure both the player and the sporting ball are analyzed to generate a more comprehensive highlight of the sporting event.
Claim(s) 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Yan further in view of Huang as applied to claims 1 and 11 respectively above, and further in view of M. Koshkina and J. H. Elder, "A General Framework for Jersey Number Recognition in Sports Video," 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA, 2024, pp. 3235-3244, doi: 10.1109/CVPRW63382.2024.00329. (hereinafter Koshkina).
Regarding dependent claim 6, the rejection of claim 1 is incorporated herein. Additionally, Yan and Huang in the combination fails to explicitly disclose wherein the human classifier is further trained to differentiate team members based on uniform colors or numbers on uniforms.
However, Koshkina discloses wherein the human classifier is further trained to differentiate team members based on uniform colors or numbers on uniforms (abstract, “we also consider how jersey number recognition at the single-image level can be aggregated across frames to yield tracklet-level jersey number labels. ”).
As noted above, Yan and Huang are directed toward analyzing video data from sporting events. Koshkina is directed toward, “apply scene text recognition models on jersey number data (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date, Yan, Huang and Koshkina are directed toward similar methods of endeavor of analyzing video data from sporting events. Further, one of ordinary skill in the art before the effective filing date of the climae invention would be easily aware some sports are team sports, where different teams are identified by different jersey colors. A user may be interested in which team scored, based on the jersey color. Further, tracking a users jersey number allows for monitoring actions of one specific player throughout the entirety of the video data. This allows one to understand which player specifically performed a specific action in question. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Koshkina in order to allow for a better understanding of which users correlate to specific actions performed.
Regarding dependent claim 16, the rejection of claim 11 is incorporated herein. Additionally, Yan and Huang in the combination fails to explicitly disclose wherein the human classifier is further trained to differentiate team members based on uniform colors or numbers on uniforms.
However, Koshkina discloses wherein the human classifier is further trained to differentiate team members based on uniform colors or numbers on uniforms (abstract, “we also consider how jersey number recognition at the single-image level can be aggregated across frames to yield tracklet-level jersey number labels. ”).
As noted above, Yan and Huang are directed toward analyzing video data from sporting events. Koshkina is directed toward, “apply scene text recognition models on jersey number data (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date, Yan, Huang and Koshkina are directed toward similar methods of endeavor of analyzing video data from sporting events. Further, one of ordinary skill in the art before the effective filing date of the climae invention would be easily aware some sports are team sports, where different teams are identified by different jersey colors. A user may be interested in which team scored, based on the jersey color. Further, tracking a users jersey number allows for monitoring actions of one specific player throughout the entirety of the video data. This allows one to understand which player specifically performed a specific action in question. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Koshkina in order to allow for a better understanding of which users correlate to specific actions performed.
Claim(s) 8, 10 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Yan further in view of Huang as applied to claims 1 and 11 respectively above, and further in view of He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. (hereinafter He).
Regarding dependent claim 8, the rejection of claim 1 is incorporated herein. Additionally, Huang discloses wherein the residual network is trained via a loss function based on per element loss (page 5, right column, “In the training phase, the cross-entropy function is used to calculate the loss function based on P(i,j, k).” page 2, right column, “Then, compared the prediction with the ground truth, a loss function will be evaluated. Finally, the filter coefficients are updated according to the loss by the gradient descent method. The chain rule is adopted to calculate the gradient of the loss function layer by layer. The process will be repeated again and again until the loss falls below an acceptable threshold. Commonly used loss functions include Root Mean Square Error (RMSE) and cross-entropy.”). Yan and Huang in the combination fail to explicitly disclose as further recited.
However, He discloses wherein the set of highlight classifiers is a machine learning model including a residual network (abstract, “Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide com prehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.”)
As noted above, Yan and Huang are directed toward analyzing video data from sporting events. Further, as noted in Huang deep learning networks are used (abstract, “In this paper, we de velop a deep learning network, called TrackNet, to track the tennis ball from broadcast videos in which the ball images”). Thus, the combination of Yan and Huang is further directed toward utilizing neural networks to perform video data analysis. He is directed toward, “a residual learning framework to ease the training of networks that are substantially deeper than those used previously (abstract).” Further, one of ordinary skill in the art before the effective filing date of the claimed invention would be aware when utilizing neural networks there is a desire for efficiency and accuracy. He improves both of these conditions as noted in the abstract, “We provide com prehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.” Thus, being that there is a desire for increased accuracy and efficiency, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the teaching of He.
Regarding dependent claim 9, the rejection of claim 8 is incorporated herein. Additionally, Yan, Huang, and He in the combination fail to explicitly disclose wherein the loss function also includes an exponential term, when an error is smaller than a predetermined threshold, an exponent of the exponential term approaches infinity, causing loss to approach 0.
However, Huang does disclose at page 5, right column, “In the training phase, the cross-entropy function is used to calculate the loss function based on P(i,j, k)” and at page 2, right column, “Then, compared the prediction with the ground truth, a loss function will be evaluated. Finally, the filter coefficients are updated according to the loss by the gradient descent method. The chain rule is adopted to calculate the gradient of the loss function layer by layer. The process will be repeated again and again until the loss falls below an acceptable threshold. Commonly used loss functions include Root Mean Square Error (RMSE) and cross-entropy.” Thus, Huang does disclose the use of a loss function configured based on error between the comparison of the prediction and ground truth. Performing the specific adjustments to the loss function as claimed would be well known to one of ordinary skill in the art before the effective filing date in light of Huang in order to allow the model to focus on the harder examples during training as opposed to the easy examples which provide less value in training.
Regarding dependent claim 10, the rejection of claim 8 is incorporated herein. Additionally, He in the combination further discloses wherein the residual network interactively applies a plurality of residual blocks (Figure 2, the building blocks are then stacked; page 772, left column, “We adopt residual learning to every few stacked layers. Abuilding block is shown in Fig. 2.”);
each residual block including a residual path (Figure 2; straight down path) and an identity path (figure 2, right path; page 771, left column, “formulation of F(x)+x can be realized by feed forward neural networks with “shortcut connections” (Fig. 2). Shortcut connections [2, 33, 48] are those skipping one or more layers. In our case, the shortcut connections simply perform identity mapping, and their outputs are added to the outputs of the stacked layers (Fig. 2). Identity short cut connections add neither extra parameter nor computational complexity.”);
the residual path includes a plurality of convolutional layers configured to output residual feature map (Figure 2; page 772, right column, “We also note that although the above notations are about fully-connected layers for simplicity, they are applicable to convolutional layers. The function F(x,{Wi}) can represent multiple convolutional layers. The element-wise addition is performed on two feature maps, channel by channel.”); and
output of the convolutional path and output of the identity path are combined together to generate output of the residual block (page 771, left column, “Formally, denoting the desired underlying mapping as H(x), we let the stacked nonlinear layers fit another mapping of F(x):=H(x)−x. The orig inal mapping is recast into F(x)+x.”).
As noted in He, the benefit of their model is to increase optimization efficiency and increase accuracy; being that these features of the residual networks are often a goal of general neural network processing, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of He.
Regarding dependent claim 18, the rejection of claim 11 is incorporated herein. Additionally, Huang discloses wherein the residual network is trained via a loss function based on per element loss (page 5, right column, “In the training phase, the cross-entropy function is used to calculate the loss function based on P(i,j, k).” page 2, right column, “Then, compared the prediction with the ground truth, a loss function will be evaluated. Finally, the filter coefficients are updated according to the loss by the gradient descent method. The chain rule is adopted to calculate the gradient of the loss function layer by layer. The process will be repeated again and again until the loss falls below an acceptable threshold. Commonly used loss functions include Root Mean Square Error (RMSE) and cross-entropy.”). Yan and Huang in the combination fail to explicitly disclose as further recited.
However, He discloses wherein the set of highlight classifiers is a machine learning model including a residual network (abstract, “Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide com prehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.”)
As noted above, Yan and Huang are directed toward analyzing video data from sporting events. Further, as noted in Huang deep learning networks are used (abstract, “In this paper, we de velop a deep learning network, called TrackNet, to track the tennis ball from broadcast videos in which the ball images”). Thus, the combination of Yan and Huang is further directed toward utilizing neural networks to perform video data analysis. He is directed toward, “a residual learning framework to ease the training of networks that are substantially deeper than those used previously (abstract).” Further, one of ordinary skill in the art before the effective filing date of the claimed invention would be aware when utilizing neural networks there is a desire for efficiency and accuracy. He improves both of these conditions as noted in the abstract, “We provide com prehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.” Thus, being that there is a desire for increased accuracy and efficiency, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the teaching of He.
Regarding dependent claim 19, the rejection of claim 18 is incorporated herein. Additionally, Yan, Huang, and He in the combination fail to explicitly disclose wherein the loss function also includes an exponential term, when an error is smaller than a predetermined threshold, an exponent of the exponential term approaches infinity, causing loss to approach 0.
However, Huang does disclose at page 5, right column, “In the training phase, the cross-entropy function is used to calculate the loss function based on P(i,j, k)” and at page 2, right column, “Then, compared the prediction with the ground truth, a loss function will be evaluated. Finally, the filter coefficients are updated according to the loss by the gradient descent method. The chain rule is adopted to calculate the gradient of the loss function layer by layer. The process will be repeated again and again until the loss falls below an acceptable threshold. Commonly used loss functions include Root Mean Square Error (RMSE) and cross-entropy.” Thus, Huang does disclose the use of a loss function configured based on error between the comparison of the prediction and ground truth. Performing the specific adjustments to the loss function as claimed would be well known to one of ordinary skill in the art before the effective filing date in light of Huang in order to allow the model to focus on the harder examples during training as opposed to the easy examples which provide less value in training.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
U.S. Publication No. 2016/0292510 to Han et al. discloses, “A solution is provided for detecting in real time video highlights in a sports video at a mobile computing device (abstract)”
U.S. Patent No. 8,923,607 to Kwatra et al. discloses, “A highlight learning technique is provided to detect and identify highlights in sports videos (abstract).”
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Courtney J. Windsor whose telephone number is (571)272-3956. The examiner can normally be reached Monday - Friday 8:00 - 4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/COURTNEY JOAN NELSON/Primary Examiner, Art Unit 2661
Prosecution Insights