Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-20 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7, 10-14, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Schiffer et al. (US 20250131729 A1 Hereinafter “Schiffer”) in view of Spiegel et al. (US 20200401617 A1 Hereinafter “Spiegel”).
Regrading claim 1, Schiffer teaches a method for training neural networks to identify and geolocate racers, comprising:
obtaining a first dataset comprising images of a race course ([0108]: “To achieve the desired accuracy, a high number of training images is used, which are selected or confirmed manually and are taken on the actual race track 100”. These images make up the first dataset), a second dataset comprising images of racers ([0088]: “In the system, the YOLO real-time object detection system is used for object detection. The detection precision can be improved by using additional training data captured on the relevant race track showing previous racing events and manually tagging vehicles 110 in the training data”. This additional training data acts as the second dataset comprising racers),
generating a second training dataset comprising the second dataset and racer labels identifying each racer in the images of racers ([0088]: “In the system, the YOLO real-time object detection system is used for object detection. The detection precision can be improved by using additional training data captured on the relevant race track showing previous racing events and manually tagging vehicles 110 in the training data”. This additional training data acts as the second training dataset with racer labels);
training a racer identification neural network to identify at least one racer in the images of racers based at least in part on identifying visual appearances of each racer ([0088]: “Vehicles 110 on the race track are also recognized based on artificial intelligence. For example, known object detection algorithm based on publicly available training data may be used to detect vehicles 110. For example, a deep learning network (DLN) or other type of convolutional neural network (CNN) may be used to detect vehicles, people, animals, and other objects on the racing track 100. In the system, the YOLO real-time object detection system is used for object detection. The detection precision can be improved by using additional training data captured on the relevant race track showing previous racing events and manually tagging vehicles 110 in the training data”. For the network to be trained to detect the vehicles, it would have to identify them based on at least the visual appearance of each racer (vehicle)); and
using ([0121-0122]: “The inferencing server 184 further comprises an inferencing unit 190. The inferencing unit 190 is setup to determine the position of the detected vehicles based on GPS position interpolation using known reference points 150 within the section 130. [0122] The inferencing unit 190 is further setup to generate an embedding vector for each detected vehicle 110 using the neural network 160. The determined embedding vector, together with a real-world position of the detected vehicle 110 on the race track 100 and/or an identifier of the camera, which captured the image, is passed to the digital twin server 186 for identification”. The inferencing server uses the detected object’s from the racer identification network and geocoordinates to identify and geolocate positions of the racers).
Schiffer does not expressly disclose obtaining map of the race course with unique geolocations; generating a first training dataset comprising the first dataset and geolocation labels identifying the unique geolocations; training a geolocation identification neural network to identify at least one unique geolocation in the images of the race course and to identify corresponding unique geolocations on the map of the race course, and using that trained geolocation identification neural network to identify and geolocate racers. Schiffer does identify landmarks with GPS coordinates: “In this example, during set-up or training, a GPS or similar position device with a visible marker may be placed at each reference point 150 in turn to determine its exact real-world position. Alternatively, the reference points 150 may coincide with prominent, high contrast features with in the section 130, i.e., edges or corners of segment boundaries, special objects like flagpoles or the like. Such reference points can be easily identified in a video image of the section 130, even if the camera moves, pans, or zooms in and out. Again, the real-world position of such prominent features is measured and stored during set-up or training, and used for triangulation during normal operation of the system.” – but does not give much additional detail.
However, Spiegel teaches obtaining a map of an area in an image ([0143]: “In another example, a set of reference images 20 may be created by sampling images from reference image database 21, which are images at known geolocations of the world, such as panoramic views from positions along streets or other areas featured, for example, by Google™ Maps and Google™ Earth”. The reference images are acquired from known spot on maps, so a map is acquired to obtain the geolocation information), generating a first training dataset comprising a first dataset and geolocation labels identifying the unique geolocations ([0143]: “In another example, a set of reference images 20 may be created by sampling images from reference image database 21, which are images at known geolocations of the world, such as panoramic views from positions along streets or other areas featured, for example, by Google™ Maps and Google™ Earth”. The reference images act as the first dataset are linked unique geolocations, the linking acts as the label identifying the geolocations “In this embodiment map 20 includes an ordered subset of reference images, the sequence of which corresponds to a specific route, each reference image being linked or corresponding to a known geolocation” [0145]), training a geolocation identification neural network to identify at least one unique geolocation in the images and to identify corresponding unique geolocations on the map ([0177]: “A query image 405 may be provided to an integrated neural network 406 for recognition. The integrated neural network 406 may include a feature extractor 407. The query image 405 may be provided to the feature extractor 407 which generates a query feature map provided to a correspondence matcher 408. The correspondence matcher 408 may be implemented within a neural network 406 in order to identify correspondence between a feature map generated by the feature extractor 407 from a query image 405 to a feature map generated from a reference image. The correspondence matcher 408 may output a correspondence map 409. The output correspondence map 409 reflects a confidence level in the query and reference image correspondence which indicates confidence in a location identification of the query image 405 as the location corresponding to the reference feature map 402”. This network is trained to perform this operation of identifying unique geolocations (features in the query image) and corresponding unique geolocations on the map (by matching the features to reference image features) “The system may be connected to a database containing reference image extractions feature maps and a neural network specifically trained to identify correspondence between a query input and a reference entry in a database” [0072]), and geolocate images using the trained geolocation neural network ([0177]: “A query image 405 may be provided to an integrated neural network 406 for recognition. The integrated neural network 406 may include a feature extractor 407. The query image 405 may be provided to the feature extractor 407 which generates a query feature map provided to a correspondence matcher 408. The correspondence matcher 408 may be implemented within a neural network 406 in order to identify correspondence between a feature map generated by the feature extractor 407 from a query image 405 to a feature map generated from a reference image. The correspondence matcher 408 may output a correspondence map 409. The output correspondence map 409 reflects a confidence level in the query and reference image correspondence which indicates confidence in a location identification of the query image 405 as the location corresponding to the reference feature map 402”. The trained network can geolocate the image input in reference to a known image geolocation to geolocate the input image. By using Spiegel’s method on the race track images, the unique geolocations of the images of the race track can be determined alongside the racers inside of the images. Therefore, the combination of Schiffer and Spiegel teaches the limitations not taught by Schiffer alone.).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Schiffer’s vehicle geolocation method to include Spiegel’s geolocation method using query and reference images alongside a trained geolocation model because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, Spiegel’s geolocation method using query and reference images alongside a trained geolocation model permits accurate geolocation of an image using known images of an area. This known benefit in Spiegel is applicable to Schiffer’s vehicle geolocation method as they both share characteristics and capabilities, namely, they are directed to determining the geographical location of objects in images. Schiffer uses GPS sensors with exact coordinates for their points, if Spiegel’s method was used, it would remove the need for the extra GPS sensors in Schiffer for geolocating the vehicle. Therefore, it would have been recognized that modifying Schiffer’s vehicle geolocation method to include Spiegel’s geolocation method using query and reference images alongside a trained geolocation model would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate Spiegel’s geolocation method using query and reference images alongside a trained geolocation model in determining the geographical location of objects in images and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Regrading claim 2, the combination of Schiffer and Spiegel teaches the method of claim 1, in addition, Schiffer further teaches wherein the geolocation identification neural network and the racer identification neural network are trained for a particular race course by inputting only images from the first dataset and the second dataset of the particular race course ([0108]: “To achieve the desired accuracy, a high number of training images is used, which are selected or confirmed manually and are taken on the actual race track 100”. If the training images are of the actual race track then the training data for both the geolocation identification neural network and the racer identification network are of the particular race course. While Spiegel teaches the concept of training the geolocation identification neural network with the acquired images will be from Schiffer).
The rationale for combination for claim 2 is similar to the rationale for combination of claim 1, due to the combination of claim 2 having similar methods of combination (using the network to geolocate images as they do in claim 1) and benefits (provide a method to geolocate images without extra sensors).
Regrading claim 3, the combination of Schiffer and Spiegel teaches the method of claim 1, in addition, Spiegel further teaches wherein the geolocation identification neural network is trained to identify the unique geolocations based on detecting, in the first dataset, a visual appearance of at least one of: starting lights, traffic lights, a turn, a straight, a starting line, a finishing line, a pit stop, a location marker, an incline, a decline, a landmark, or a building (Fig. 4, Spiegel: In the output correspondence map, features of traffic lights and buildings are used to correlate the query image to the reference image to determine unique geolocation of the image. If used on Schiffer’s race track images, it would result in the buildings being detected “An aspect of this process is to use anchors or traps designated in the reference images in a sparse geo-tagged reference database. The trap may be a segment of an image that is easily recognized by image processing techniques” ([0188, Schiffer), these traps used as geolocation spots include buildings “The use of traps may be either for a whole reference image or on areas of interest of the reference image. The traps may represent POIs appearing in an image, for example, billboards, shops, buildings, windows, etc.” ([0180], Schiffer). The list above is recited in the alternative, hence only one of the limitations need met to reach a prima facia case of obviousness).
The rationale for combination for claim 3 is similar to the rationale for combination of claim 1, due to the combination of claim 3 having similar methods of combination (using the correlation of features of street lights and buildings to geolocate images as they do in claim 1) and benefits (provide a method to geolocate images without extra sensors).
Regrading claim 4, the combination of Schiffer and Spiegel teaches the method of claim 1, in addition, Spiegel further teaches wherein the geolocation identification neural network is further configured to output a first confidence value for each identification of a unique geolocation in the first dataset ([0177]: “The output correspondence map 409 reflects a confidence level in the query and reference image correspondence which indicates confidence in a location identification of the query image 405 as the location corresponding to the reference feature map 402”).
The rationale for combination for claim 4 is similar to the rationale for combination of claim 1, due to the combination of claim 4 having similar methods of combination (using the correlation of features of street lights and buildings relative to a confidence to geolocate images as they do in claim 1) and benefits (provide a method to geolocate images without extra sensors).
Regrading claim 5, the combination of Schiffer and Spiegel teaches the method of claim 1, in addition, Schiffer further teaches wherein the racer identification neural network is trained to identify racers in the second dataset based on detecting, in the second dataset, a visual appearance of at least one of: racer’s face, a helmet color, uniform color, vehicle color, vehicle type, or a number assigned to each racer ([0107]: “In the example, object re-identification is implemented using a neural network 160 that has been trained offline, i.e., before the use of the monitoring system in an actual race, using an encoder/decoder model to identify specific instances from a given class of objects, e.g., individual racing cars taking part in a currently running race. Different training sets may be used to train different instances of corresponding neural networks to different classes of objects, for example, Formula 1 cars, normal road cars, motorcycles or the like”. The network is trained to identify a type of car (such as F1 cars), so it is trained on the visual appearance of that vehicle. The list above is recited in the alternative, hence only one of the limitations need met to reach a prima facia case of obviousness).
Regrading claim 7, the combination of Schiffer and Spiegel teaches the method of claim 1, in addition, Schiffer further teaches wherein the trained racer identification neural network is trained to identify the at least one racer in the racing videos based on analyzing historical data corresponding to images of racers from previous races on the same race course ([0088]: “The detection precision can be improved by using additional training data captured on the relevant race track showing previous racing events and manually tagging vehicles 110 in the training data”. The neural network is trained based on historical data (previous racing events) corresponding to images of racers from previous races (manually tagging the vehicles in the previous racing events) on the same course (the relevant race track). The track is relevant because it is the same, as mentioned before “To achieve the desired accuracy, a high number of training images is used, which are selected or confirmed manually and are taken on the actual race track 100” [0108]).
Regarding claim 10, the content of claim 10 is similar to the content of claim 1, with the additional teachings of a memory and processor. Schiffer also discloses this information ([0065]: “an image processing system comprising at least one processor. The processor is configured to segment images of the sequence of images into different areas associated with the race track, detect at least one vehicle in the sequence of images using automatic pattern recognition, in particular object recognition, map the at least one detected vehicle to at least one of the different areas associated with the race track, and, based on a first set of rules, trigger the first warning if the at least one detected vehicle is mapped to a first predefined area of the race track”. To be able to perform these operations, the instructions for the processor must be stored on a memory). Therefore, claim 10 is rejected for the same reasons of obviousness as claim 1, along with the additional teachings above.
Regarding claim 11, the content of claim 11 is similar to the content of claim 2, therefore it is rejected for the same reasons of obviousness as claim 2.
Regarding claim 12, the content of claim 12 is similar to the content of claim 3, therefore it is rejected for the same reasons of obviousness as claim 3.
Regarding claim 13, the content of claim 13 is similar to the content of claim 4, therefore it is rejected for the same reasons of obviousness as claim 4.
Regarding claim 14, the content of claim 14 is similar to the content of claim 5, therefore it is rejected for the same reasons of obviousness as claim 5.
Regarding claim 16, the content of claim 16 is similar to the content of claim 7, therefore it is rejected for the same reasons of obviousness as claim 7.
Regarding claim 17, the content of claim 17 is similar to the content of claim 1, with the additional teachings of a non-transitory computer readable medium. Schiffer also discloses this information ([0065]: “an image processing system comprising at least one processor. The processor is configured to segment images of the sequence of images into different areas associated with the race track, detect at least one vehicle in the sequence of images using automatic pattern recognition, in particular object recognition, map the at least one detected vehicle to at least one of the different areas associated with the race track, and, based on a first set of rules, trigger the first warning if the at least one detected vehicle is mapped to a first predefined area of the race track”. To be able to perform these operations, the instructions for the processor must be stored on a non-transitory computer readable medium). Therefore, claim 10 is rejected for the same reasons of obviousness as claim 1, along with the additional teachings above.
Regarding claim 18, the content of claim 18 is similar to the content of claim 2, therefore it is rejected for the same reasons of obviousness as claim 2.
Regarding claim 19, the content of claim 19 is similar to the content of claim 3, therefore it is rejected for the same reasons of obviousness as claim 3.
Regarding claim 20, the content of claim 20 is similar to the content of claim 4, therefore it is rejected for the same reasons of obviousness as claim 4.
Claims 6 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Schiffer et al. (US 20250131729 A1 Hereinafter “Schiffer”) in view of Spiegel et al. (US 20200401617 A1 Hereinafter “Spiegel”) as evidenced by Redmon et al. (“You Only Look Once: Unified, Real-Time Object Detection” Hereinafter “Redmon”).
Regrading claim 6, the combination of Schiffer and Spiegel teaches the method of claim 1, in addition, Schiffer further teaches method of claim 1, wherein the racer identification neural network is further configured to output a second confidence value for each identification of each racer ([0088]: “Vehicles 110 on the race track are also recognized based on artificial intelligence. For example, known object detection algorithm based on publicly available training data may be used to detect vehicles 110. For example, a deep learning network (DLN) or other type of convolutional neural network (CNN) may be used to detect vehicles, people, animals, and other objects on the racing track 100. In the system, the YOLO real-time object detection system is used for object detection”. The YOLO object detection system outputs the confidence of each object of interest as evidenced by Redmon “Our system divides the input image into an S × S grid.
If the center of an object falls into a grid cell, that grid cell is responsible for detecting that object. Each grid cell predicts B bounding boxes and confidence scores for those boxes. These confidence scores reflect how confident the model is that the box contains an object and
also how accurate it thinks the box is that it predicts (Page 780, section 2)).
Regarding claim 15, the content of claim 15 is similar to the content of claim 6, therefore it is rejected for the same reasons of obviousness as claim 6.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Schiffer et al. (US 20250131729 A1 Hereinafter “Schiffer”) in view of Spiegel et al. (US 20200401617 A1 Hereinafter “Spiegel”) in further view of KANTOROVICH et al. (US 20210001174 A1 Hereinafter “KANTOROVICH”).
Regrading claim 8, the combination of Schiffer and Spiegel teaches the method of claim 1, in addition, Schiffer further teaches further comprising:
Schiffer does not expressly disclose training a telemetry neural network to generate additional telemetry data to supplement obtained telemetry data based at least in part on analyzing obtained telemetry data.
However, KANTOROVICH teaches training a telemetry neural network to generate additional telemetry data to supplement obtained telemetry data based at least in part on analyzing obtained telemetry data ([0200]: “The types of respective actions can be annotated by a human and the data from sensors (wearable IMU/video cameras) around the, annotated actions will be supplied to the supervised machine learning algorithm (see examples above) along with the labels for its training The trained model can be used to predict the action types for each data window around an action which was detected as explained above”. The types of actions are considered additional telemetry data based on the analyzed obtained telemetry data due to the classification being a description of the telemetry data).
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Schiffer’s vehicle geolocation method to include KANTOROVICH’s telemetry neural network and training process because such a modification is the result of applying a known technique to a known device ready for improvement to yield predictable results. More specifically, KANTOROVICH’s telemetry neural network and training process permits using sensor data of athletes to determine action. This known benefit in KANTOROVICH is applicable to Schiffer’s vehicle geolocation method as they both share characteristics and capabilities, namely, they are directed to processing sensor information to obtain data about moving objects in images. Therefore, it would have been recognized that modifying Schiffer’s vehicle geolocation method to include KANTOROVICH’s telemetry neural network and training process model would have yielded predictable results because (i) the level of ordinary skill in the art demonstrated by the references applied shows the ability to incorporate KANTOROVICH’s telemetry neural network and training process in processing sensor information to obtain data about moving objects in images and (ii) the benefits of such a combination would have been recognized by those of ordinary skill in the art.
Allowable Subject Matter
Claim 9 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Howell et al. (US 20170236029 A1) teaches connecting location and image data to locate racer in a race.
Allen et al. (US 20200177969 A1) teaches predictive racing information.
Guerrero del Pozo et al. (US 20250086964 A1) teaches connected detected features in images to geo-locations.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEFANO A DARDANO whose telephone number is (703)756-4543. The examiner can normally be reached Monday - Friday 11:00 - 7:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Greg Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STEFANO ANTHONY DARDANO/ Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights