Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED OFFICE ACTION
Status of Claims
Claims 25-48 are pending in this Office Action.
Double Patenting
1. The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
2. Claims 25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,42,43,44,45,46 and 47 are rejected on the ground of non-statutory double patenting as being unpatentable over claims 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16, 18,19,20,21,22 and 23 respectively of U.S. Patent No. 12,186,643. Although the claims at issue are not identical, they are not patentably distinct from each other because of the following similarity of the Claim limitations:
Present Application
U.S. Patent No. 12,186,643
As per claim 25, A system for determining a position of a sports ball on a surface of a sports play area, comprising: a camera having a first field of view configured to capture images in a camera coordinate system of a portion of the sports play area, the camera having calibration parameters associated therewith; a storage arrangement including values for the calibration parameters and a three- dimensional (3D) model of the surface of the portion of the sports play area; and a processing arrangement coupled to the camera configured to: detect a pixel location of the sports ball in the images from the camera; determine, based on the values for the calibration parameters, a camera-ball line comprising coordinates of a straight line passing through the camera in a direction of the sports ball; determine, based on the 3D model, an intersection point of the camera-ball line with the 3D model; and output the intersection point as a 3D position of the sports ball.
As per claim 1, A system, comprising: a tracking camera having a first field of view configured to capture images in a camera coordinate system of a sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith; a storage arrangement including values for the intrinsic and extrinsic calibration parameters and a three-dimensional (3D) model of at least part of a sports play area overlapping with the first field of view; and a processing arrangement coupled to the tracking camera configured to: perform a ball detection to detect a pixel location of the sports ball in an image; determine, based on the values for the intrinsic and extrinsic calibration parameters, a camera-ball line comprising 3D coordinates of a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system; determine, based on the 3D model, an intersection point of the camera-ball line with the 3D model; and output the intersection point as a 3D position of the sports ball in the image.
The features of claim 25 of the current application that are not present in claim 1 of the U.S. Patent No. 12,186,643 are :
“determining a position of a sports ball on a surface of a sports play area,”
However, in an analogues art, Johnson et al. (USPUB 20090067670) teaches determining a position of a sports ball on a surface of a sports play area (FIGURE 1 and FIGURE 7 and Paragraphs [0059]- “…estimated trajectory of a ball can differ from the actual trajectory. Lines 1 and 2 show the actual trajectory of a ball prior to and after an impact 3 with a playing surface. Lines 4 and 5 show an estimated trajectory similar to that generated by many commercial systems. Estimated trajectories are often inaccurate, due in part to errors in the estimated impact point of the ball on the playing surface. …”) , Accordingly, the prior art references teach all of the claimed elements ( i.e tracking ball within a surface and 3D imaging and positioning of a ball etc. ) . The combination of the known elements is achieved by the Methods and processes for detecting a mark on a playing surface and for tracking an object mentioned by Johnson et al. within the System And Method For Determining 3D Positional Coordinates Of A Ball taught in claim 1 of U.S. Patent No. 12,186,643.
Therefore, the results would have been predictable to one of ordinary skill in the art.
Based on the above findings, it would have been obvious to one of ordinary skill before the
effective filing date of the invention to add the elements taught in Johnson et al.
to the system taught in claim 1 of U.S. Patent No. 12,186,643 as no more “than the predictable use of prior-art elements according to their established functions.”
As per claim 26, The system of claim 25, wherein the processing arrangement is further configured to: determine whether the sports ball is at rest by comparing the pixel location of the sports ball in successive images; and when the processing arrangement determines that the sports ball is at rest, outputting the intersection point as a 3D rest position of the sports ball in the image.
As per claim 2, The system of claim 1, wherein the processing arrangement is further configured to: determine the sports ball is at rest by comparing the pixel location of the sports ball in successive images; and outputting the intersection point as a 3D rest position of the sports ball in the image.
As per claim 27, The system of claim 25, wherein the captured images are associated with metadata including a time at which the image was captured, an exposure time and a crop region.
As per claim 3, The system of claim 1, wherein the captured images are associated with metadata including a time at which the image was captured, an exposure time and a crop region.
As per claim 28, The system of claim 25, further comprising: a tracking radar having a second field of view at least partially overlapping the first field of view, wherein the processing arrangement is further configured to detect the sports ball in dependence on radar data from the radar to narrow an area searched in the images for the sports ball and to reduce a number of the images searched for the sports ball based on a time for which the radar data indicates the presence of the sports ball in the second field of view.
As per claim 4, The system of claim 1, further comprising: a tracking radar having a second field of view at least partially overlapping the first field of view configured to capture radar data of the sports ball, wherein the processing arrangement is further configured to detect the sports ball in dependence on the radar data to narrow an area searched in the images and a number of images searched in time.
As per claim 29, The system of claim 28, wherein detections of the sports ball using the radar data in combination with the images are used by the processing arrangement to improve an accuracy of 3D position determinations of the sports ball.
As per claim 5, The system of claim 4, wherein detections of the sports ball using the radar data in combination with the images are used by the processing arrangement to improve an accuracy of 3D position determinations of the sports ball.
As per claim 30, The system of claim 25, wherein the processing arrangement is further configured to detect the sports ball in the images using a neural network trained to detect the sports balls.
As per claim 6, The system of claim 1, wherein the processing arrangement is further configured to detect the sports ball using a neural network trained to detect sports balls in images.
As per claim 31, The system of claim 30, wherein the neural network uses information for search regions from previous images to improve an accuracy of detections of the sports ball in current images.
As per claim 7, The system of claim 6, wherein the neural network uses information for search regions from previous images to improve an accuracy of detections of the sports ball in current images.
As per claim 32, The system of claim 27, wherein the processing arrangement is further configured to determine the camera-ball line based on the crop region for the image.
As per claim 8, The system of claim 3, wherein the processing arrangement is further configured to determine the camera-ball line based on the crop region for the image.
As per claim 33, The system of claim 25, wherein the sports play area is a golf course and wherein the 3D model represents a height of the surface for a given position on the golf course.
As per claim 9, The system of claim 1, wherein the sports play area is a golf course and wherein the 3D model of the part of the golf course contains a surface model representing a height of terrain for a given position on the golf course.
As per claim 34, The system of claim 33, wherein the 3D model further represents a terrain type for the given position on the golf course.
As per claim 10, The system of claim 9, wherein the surface model further represents a terrain type for the given position on the golf course.
As per claim 35, The system of claim 34, wherein the processing arrangement is further configured to determine a type of terrain within which the sports ball is detected based on the 3D model.
As per claim 11, The system of claim 10, wherein the processing arrangement is further configured to determine a type of terrain within which the sports ball is detected based on the 3D model.
As per claim 36, The system of claim 33, wherein the 3D model comprises a mesh of triangles or spline surfaces.
As per claim 12, The system of claim 9, wherein the surface model comprises a mesh of triangles or spline surfaces.
As per claim 37, The system of claim 33, wherein the intersection point is determined using an iterative process or using a numerical solver to solve an optimization problem.
As per claim 13, The system of claim 9, wherein the intersection point is determined using an iterative process or using a numerical solver to solve an optimization problem.
As per claim 38, The system of claim 25, wherein the 3D position of the sports ball in the image is output to a database, a 3D graphics rendering engine, or a graphical illustration of the 3D position of the sports ball.
As per claim 14, The system of claim 1, wherein the 3D position of the sports ball in the image is output to a database, a 3D graphics rendering engine, or a graphical illustration of the 3D position of the sports ball.
As per claim 39, The system of claim 25, wherein the 3D model is provided in the 3D coordinate system for the camera-ball line.
As per claim 15, The system of claim 1, wherein the 3D model is provided in the 3D coordinate system for the camera-ball line.
As per claim 40, A method, comprising: detecting a pixel location of a sports ball in an image captured by a camera having a first field of view configured, the camera having calibration parameters associated therewith; storing in a storage arrangement values for the calibration parameters; determining, based on the values for the calibration parameters, a camera-ball line comprising coordinates of a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system; determining, based on a 3D model stored to the storage arrangement, an intersection point of the camera-ball line with the 3D model, wherein the 3D model comprises at least a portion of a surface of a sports play area; and outputting the intersection point as a 3D position of the sports ball.
As per claim 16, A method, comprising: performing a ball detection to detect a pixel location of a sports ball in an image captured by a tracking camera, the tracking camera having a first field of view configured to capture images in a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith, values for the intrinsic and extrinsic calibration parameters stored to a storage arrangement; determining, based on the values for the intrinsic and extrinsic calibration parameters, a camera-ball line comprising three-dimensional (3D) coordinates of a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system; determining, based on a 3D model stored to the storage arrangement, an intersection point of the camera-ball line with the 3D model, wherein the 3D model comprises at least part of a sports play area overlapping with the first field of view; and outputting the intersection point as a 3D position of the sports ball in the image.
As per claim 42, A system, comprising: a camera configured to capture images in an image plane, the camera having calibration parameters associated therewith; a storage arrangement storing values for the calibration parameters; and a processing arrangement coupled to the camera and the storage arrangement, the processing arrangement being configured to: detect a pixel location in the image plane of sports ball in each of a sequence of images; generate a first time-series of the ball detections in the image plane; determine, based on the values for the calibration parameters, a camera-ball line comprising a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system for each of the ball detections in the first time-series; generate a second time-series of an elevation angle of each camera-ball line; and identify, based on minima in the second time-series, bounces of the sports ball captured in the sequence of images.
As per claim 18, A system, comprising: a tracking camera having a first field of view configured to capture images in an image plane of a camera coordinate system of a sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith; a storage arrangement including values for the intrinsic and extrinsic calibration parameters; and a processing arrangement coupled to the tracking camera and the storage arrangement configured to: perform a ball detection to detect a pixel location in the image plane of the sports ball in each of a sequence of images; generate a first time-series of the ball detections in the image plane; determine, based on the values for the intrinsic and extrinsic calibration parameters, a camera-ball line comprising three-dimensional (3D) coordinates of a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system for each of the sports ball detections in the first time-series; generate a second time-series of an elevation angle of each camera-ball line; and identify, based on minima in the second time-series, bounces of the sports ball captured in the sequence of images.
As per claim 43, The system of claim 42, wherein the storage arrangement further stores a three- dimensional (3D) model of a portion of a surface of a sports play area, the processing arrangement being configured to: determine, based on the 3D model, for each identified bounce, an intersection point of the camera-ball line with the 3D model; and output the intersection points as 3D positions of bounces of the sports ball.
As per claim 19, The system of claim 18, wherein the storage arrangement further stores a three-dimensional (3D) model of at least part of a sports play area overlapping with the first field of view, wherein the processing arrangement is further configured to: determine, based on the 3D model, for each identified bounce, an intersection point of the camera-ball line with the 3D model; and output the intersection points as 3D positions of bounces of the sports ball.
As per claim 44, The system of claim 42, further comprising: a radar having a field of view at least partially overlapping with a first field of view of the camera, the radar being configured to capture radar data of the sports ball, wherein the processing arrangement is further configured to identify bounces in dependence on the radar data by determining velocity discontinuities in the radar data.
As per claim 20, The system of claim 18, further comprising: a tracking radar having a second field of view at least partially overlapping the first field of view configured to capture radar data of the sports ball, wherein the processing arrangement is further configured to: identify bounces in dependence on the radar data by determining velocity discontinuities in the radar data.
As per claim 45, The system of claim 42, wherein the processing arrangement is further configured to: distinguish bounces from rolls in a second time series based on a physical model of a bounce; and classify each minima in the second time series that has not been determined to be a bounce as a roll.
As per claim 21, The system of claim 18, wherein the processing arrangement is further configured to: distinguish bounces from rolls in a second time series based on a physical model of a bounce; and classify each minima in the second time series that has not been determined to be a bounce as being a roll.
As per claim 46, The system of claim 42, wherein a 3D position of the sports ball in the image is output to a database, a 3D graphics rendering engine, or a graphical illustration of the 3D position of the sports ball.
As per claim 22, The system of claim 18, wherein a 3D position of the sports ball in the image is output to a database, a 3D graphics rendering engine, or a graphical illustration of the 3D position of the sports ball.
As per claim 47, A method, comprising: detecting a pixel location in an image plane of a sports ball in each of a sequence of images captured by a camera; storing in a storage arrangement values for calibration parameters associated with the camera; generating a first time-series of the ball detections in the image plane; determining, based on the values for the calibration parameters, a camera-ball line comprising a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system for each of the ball detections in the first time-series; generating a second time-series of an elevation angle of each camera-ball line; and identifying, based on minima in the second time-series, bounces of the sports ball captured in the sequence of images.
As per claim 23, A method, comprising: performing a ball detection to detect a pixel location in an image plane of a sports ball in each of a sequence of images captured by a tracking camera, the tracking camera having a first field of view configured to capture images in the image plane of a camera coordinate system of the sports ball bouncing and rolling after an initial trajectory, the tracking camera having intrinsic and extrinsic calibration parameters associated therewith, values for the intrinsic and extrinsic calibration parameters stored to a storage arrangement; generating a first time-series of the ball detections in the image plane; determining, based on the values for the intrinsic and extrinsic calibration parameters, a camera-ball line comprising three-dimensional (3D) coordinates of a straight line passing through the camera in a direction of the sports ball in a 3D coordinate system for each of the ball detections in the first time-series; generating a second time-series of an elevation angle of each camera-ball line; and identifying, based on minima in the second time-series, bounces of the sports ball captured in the sequence of images.
Claims 41 and 48 are rejected as claims 40 and 47 are rejected under double patenting.
It is noted that any citations to specific, pages, columns, lines, or figures in the prior art references and any interpretation of the reference should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. See MPEP 2123.
Examiner’s Notes on Prior Art
3. The Examiner acknowledges the following prior arts below as pertinent to the current applications claim limitations and inventive concept, although the following prior arts shown below were not relied upon to address the limitations within the claim , they are analogous art mentioning the inventive concept key points on (Object/ball tracking and detection, sequence of images, pixel location within images , neural network etc.).
1) Mikel Labayen et al. ,"Accurate ball trajectory tracking and 3D visualization for computer-assisted sports broadcast,"24th August 2013, Multimed Tools Appl (201 ) 73,Page 1820-1835.
2) Anil Kumar et al.,"3D Estimation and Visualization of Motion in a Multicamera Network for Sports," 1st December 2014, 2011 Irish Machine Vision and Image Processing Conference,Pages 15-18.
3) Sébastien Mavromatis et al.,"3D Reconstruction of Soccer Sequences Using Non-calibrated Video Cameras," 22nd May 2013, Machine Vision and Applications (2013) 24,Pages 1561-1567..
4) Tianxiao Zhang,"Efficient Golf Ball Detection and Tracking Based on Convolutional Neural Networks and Kalman Filter," 21st April 2021, Computer Vision and Pattern Recognition, arXiv:2012.09393,Pages 1- 9.
5) Jinchang Ren,"Real-Time Modeling of 3-D Soccer Ball Trajectories From Multiple Fixed Cameras,"19th March,2007, IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 18, NO. 3, MARCH 2008,Pages 350-360.
6) Vock et al. (USPUB 20050012023 )
7) Kiraly et al. (USPUB 20060008116)
8) Johnson et al. (USPUB 20090067670 )
9) House (USPUB 20100020068)
10) Cavallaro et al. (USPUB 20110205077)
11) FEI et al. (USPUB 20160261300 )
12) Raab et al. (USPUB 20170054965 )
13) Gentil (USPUB 20180154232 )
14) Gordon ( USPUB 20200064119 )
15) Adams et al. ( USPUB 20210182596 )
16) TUXEN et al. ( USPUB 20220284628 )
It is noted that any citations to specific, pages, columns, lines, or figures in the prior art references and any interpretation of the reference should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. See MPEP 2123.
Conclusion
4. Any inquiry concerning this communication or earlier communications from the examiner should be directed to OMAR S. ISMAIL whose telephone number is (571)272-9799 and Fax # (571)273-9799. The examiner can normally be reached on M-F: 9:00 AM - 6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http:/ If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, David C. Payne can be reached on (571)272-3024. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free)? If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OMAR S ISMAIL/Primary Examiner, Art Unit 2635
Prosecution Insights