DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This Office Action is in response to the application filed on February 17th, 2026. Claims 1-20 are presently pending and are presented for examination.
Response to Amendment
In response to Applicant’s amendment filed February 17th 2026, Examiner withdraws the previous, claim 9, 35 U.S.C. 112(b) rejection; and withdraws the previous 35 U.S.C. 102 and 103 prior art rejections.
Response to Arguments
Applicant’s arguments filed February 17th 2026, have been fully considered.
Regarding the arguments provided for the rejections of claims 1-6, 8-12, and 16-20, as put forth on pages 12-13, applicants’ arguments have been fully considered. Applicant argues “Lu fails to disclose all of the elements of the amended claims, as required by 35 U.S.C. 102(a)(1), and provides no apparent reason for modification to include such features…Specifically, regarding calibration Lu states “since the traffic monitoring cameras do not move, we only need to run the calibration procedure once for each camera in the following steps” (Lu, Page 2, right column, bottom paragraph). Lu describes cameras that “are commonly mounted on road infrastructures” (Lu, Page 1, right column, first full paragraph) and states “[s]ince the traffic monitoring cameras do not move, we only need to run the calibration procedure once for each camera” (Lu, Page 2, right column, bottom paragraph). For at least these reasons, the amended independent claim 1 is patentable. Amended claims 17 and 19 are patentable for at least the reasons amended independent claim 1 is patentable. Further, the pending dependent claims are patentable for at least the reasons their respective base claims are patentable”.
As to point (a), Applicant’s arguments have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of US-20220051431 (hereinafter, “Jagadeesan”).
Regarding the arguments provided for the rejections of claims 7 and 13-15, as put forth on pages 13-14, applicant’s arguments have been fully considered. Applicant argues “the applied references, alone or in any combination, fail to disclose or suggest the requirements of the respective independent claims. The dependent claims incorporate the requirements of the respective independent claims, and therefore, these claims are likewise patentable. For at least these reasons, the rejection does not establish a prima facie case for non-patentability of Applicant’s claims 7 and 13-15 under 35 U.S.C. 103 in view of the amended claims.”
As to point (b), see point (a).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-6, 8-12, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over by "CAROM - Vehicle Localization and Traffic Scene Reconstruction from Monocular Cameras on Road Infrastructures" (hereinafter, “Lu”) in view of US-20220051431 (hereinafter, “Jagadeesan”).
Regarding claim 1 Lu discloses a system (see at least [page 1, right column]; “we propose CAROM, a framework that can extract 3D information from the videos, generate a series of structured data records of vehicle states, and reconstruct traffic scenes on a 2D map or a 3D map, as shown in Fig 1.”) comprising:
processing circuitry (while Lu does not explicitly state that processing circuitry is a part of the framework of the system it would be obvious for a system to comprise both a processor and computer readable media); and
non-transitory computer readable media storing instructions that, when executed by the processing circuitry (while Lu does not explicitly state that processing circuitry is a part of the framework of the system it would be obvious for a system to comprise both a processor and computer readable media), configure the processing circuitry to:
obtain as source input, aerial video of a traffic scene including vehicles that traverse the traffic scene (see at least [Page 1, right column]; “The traffic monitoring cameras are commonly mounted on road infrastructures with the advantage of covering a large area. Hence, they can be used to objectively assess the operational safety by calculating a set of safety metrics [1] directly from vehicle movements captured on the videos. Meanwhile, the information of the surrounding traffic scene obtained by these cameras can complement the perception of AVs because the in-vehicle sensors can only reach places in the line-of-sight,” and [Page 5, right column]; “for the second site, we flew a drone and capture videos from 80 meters above the road. We processed the drone videos to obtain the vehicle location and velocity”);
obtain a map image of the traffic scene distinct from any aerial image of the traffic scene within the source input (see at least [Page 2, right column]; “The ground is modeled…as a flat surface corresponding to a 2D satellite image map,” and [Page 5, left column]; “We constructed the 2D map using a satellite image at the place where the camera is mounted. Many online map services (such as Google Maps) offer satellite images”);
determine aerial image reference points of the traffic scene present within the source input which correspond to map image reference points of the traffic scene present within the satellite map image of the traffic scene (see at least [Page 2, right column]; “The calibration procedure constructs two sets of parameters: (1) a camera projection matrix from the world frame to the camera frame, (2) a transformation between the map frame and the world frame. Since the traffic monitoring cameras do not move, we only need to run the calibration procedure once for each camera in the following steps. First, we label a set of at least six-point correspondences on the map and the image, typically using the lane markers and features on the ground.”);
responsive to determination of the aerial image reference points of the traffic scene present within the source input which correspond to map image reference points of the traffic scene present within the map image of the traffic scene, generate calibrated images of the traffic scene from frames of the aerial video of the traffic scene by calibrating the frames of the aerial video with the map image of the traffic scene utilizing the corresponding map image reference points of the traffic scene (see at least [Page 2, right column – page 3, left column]; “The calibration procedure constructs two sets of parameters: (1) a camera projection matrix from the world frame to the camera frame, (2) a transformation between the map frame and the world frame. Since the traffic monitoring cameras do not move, we only need to run the calibration procedure once for each camera in the following steps. First, we label a set of at least six-point correspondences on the map and the image, typically using the lane markers and features on the ground. Second, we create the world frame and compute the transformation between the world frame to the map frame. Usually, the XOY plane of the world frame is the ground plane in the 2D map and the x-axis follows the traffic moving direction. Third, we transform the labeled points on the map to the world frame and compute the camera projection matrix from the point correspondences [34]. Optionally, the calibration of the camera can be automated [16]”)…
…determine multiple unique keypoints on the vehicles that traverse the traffic scene (see at least [Page 2, left column]; “the 3D representation of a vehicle can be a point with an orientation vector, a 3D bounding box, a few key points, a wireframe model, or a parametric 3D shape model”);
track the vehicles that traverse the traffic scene across the frames of the aerial video based on information derived from the multiple unique keypoints (see at least [Page 2, left column]; “Second, in addition to accurate vehicle detection and tracking on the 2D images, robust estimation of vehicle 3D pose and vehicle dimension is required. The 3D representation of a vehicle can be a point with an orientation vector [20], a 3D bounding box [16][21][22], a few key points [23], a wireframe model [24][25], or a parametric 3D shape model [26]. The location and speed of a vehicle are typically determined from three pieces of information: (1) the 2D locations on the images, (2) the 3D poses of the vehicle, and (3) the transformation between image coordinates and the ground coordinates obtained from the camera calibration results. Usually, the vehicle states are also estimated jointly through a filtering process by considering the vehicle kinematics or dynamics [27][28]. The vehicle shape can be reconstructed using stereo cameras [29] or monocular cameras [25][30][31] through a sequence of algorithms for depth estimation, model fitting, and shape optimization. Our paper is based on the existing works for several individual computer vision tasks and we integrated them to a unified framework that extracts the location, speed, and vehicle shape in the 3D space. Further, our tracking results allow the vehicle movements to be replayed on a 2D map or a 3D map so as to support traffic analysis tasks,” the 3D reconstruction with contains key points is used to track the vehicle throughout the scene); and
output vehicle metrics for one or more of the vehicles that traverse the traffic scene (see at least [Page 3, left column]; “For each detected object instance, the system crops a square patch from the image just large enough to contain its 2D bounding box, resizes the cropped patch and runs a classifier to predict its type. The following types are used: {pedestrian, two-wheelers, bus, mini-truck, semi-truck, pickup-truck, convertible, coupe, sedan, all-terrain vehicle, minivan, van, SUV, trailer}.”).
Lu does not disclose wherein generating the calibrated images includes tracking at least a subset of the aerial image reference points across multiple frames of the aerial video and updating a camera pose associated with frames of the aerial video based on the tracked aerial image reference points using a perspective-n-point computation to compensate for camera pose drift.
Jagadeesan, in the same field of endeavor, teaches wherein generating the calibrated images includes tracking at least a subset of the aerial image reference points across multiple frames of the aerial video and updating a camera pose associated with frames of the aerial video based on the tracked aerial image reference points using a perspective-n-point computation to compensate for camera pose drift (see at least [abstract]; “Systems and methods for creating images of an environment includes controlling at least one camera to acquire imaging data from the environment and selecting, from the imaging data, a three-dimensional-two-dimensional correspondence as a control point for use in a perspective-n-point problem to determine a position and orientation of the at least one camera from n known correspondences between three-dimensional object points and their two-dimensional image projections in the environment,” and [0221]; “the camera pose is updated in an iterative process and the reprojection of stored feature points is updated accordingly,” and [0211]; “first module 504 tracks the camera motion between adjacent video frames according to ORB feature matching, which is mainly based on a novel and robust PnP algoritl1m called DynamicR1PPnP. A second module 508 aims to refine the camera motion estimation results at key frames and eliminate the accumulative error, which is based on the minimization of ICP and bundle adjustment (BA) costs.”).
Therefore, it would have been obvious for one of ordinary skill in the art, before the effective filing date of the claimed invention with a reasonable expectation of success to have modified the traffic reconstruction system of Lu with the camera pose updates of Jagadeesan. One of ordinary skill in the art would have been motivated to make this modification for the benefit of creating a more accurate navigation method (see at least Jagadeesan; [0003-0008]).
Regarding claim 2 Lu in view of Jagadeesan renders obvious all of the limitations of claim 1. Additionally, Lu discloses wherein the processing circuitry is further configured to: execute, via the processing circuitry, a CAR-On Map ("CAROM") air framework for vehicle localization and traffic scene reconstruction using the aerial video of the traffic scene (see at least [page 1, right column]; “To address these issues, we propose CAROM, a framework that can extract 3D information from the videos, generate a series of structured data records of vehicle states, and reconstruct traffic scenes on a 2D map or a 3D map, as shown in Fig 1,” and [Page 2, left column]; “We constructed a vehicle tracking, localization, and velocity measurement pipeline using videos taken by monocular road traffic monitoring cameras”).
Regarding claim 3 Lu in view of Jagadeesan renders obvious all of the limitations of claim 1. Additionally, Lu discloses wherein the processing circuitry is further configured to: estimate a vehicle state for each of the vehicles that traverse the traffic scene to establish a current position (see at least [Page 3, right column]; “Additionally, the 3D bounding box is not calculated if certain occlusion conditions are detected using the 2D bounding box overlap and the size of the mask. Finally, the center of this 3D bounding box’s bottom surface is the vehicle’s location on the image. Again, with the transformation T, the location of the vehicle in the world frame is obtained”) and a heading of each of the vehicles within each of the calibrated images (see at least [Page 3, left column]; “Because a vehicle rarely moves backward on the road, the vehicle heading is determined by the line from the center of its 2D bounding box to this vanishing point”).
Regarding claim 4 Lu in view of Jagadeesan renders obvious all of the limitations of claim 1. Additionally, Lu discloses wherein the processing circuitry is further configured to: output the vehicle metrics for at least one of the vehicles that traverse the traffic scene, including one or more of: a vehicle type; a vehicle location; a vehicle speed; a vehicle trajectory; a vehicle traffic violation a vehicle collision incident; a vehicle collision near-incident; a Time-To-Collision (TTC) metric for pairs of adjacent vehicles in a same lane within the traffic scene; or a Post Encroachment Time (PET) metric (see at least [Page 3, left column]; “For each detected object instance, the system crops a square patch from the image just large enough to contain its 2D bounding box, resizes the cropped patch and runs a classifier to predict its type. The following types are used: {pedestrian, two-wheelers, bus, mini-truck, semi-truck, pickup-truck, convertible, coupe, sedan, all-terrain vehicle, minivan, van, SUV, trailer}.” Applicant has listed several vehicle metrics which may be output for the traffic scene; however, Examiner asserts that other metrics may be taught by Lu in addition to identification of vehicle type, however are not required due to Applicant’s recitation of “one or more of”).
Regarding claim 5 Lu in view of Jagadeesan renders obvious all of the limitations of claim 1. Additionally, Lu discloses wherein the processing circuitry is further configured to: calibrate the frames of the aerial video with the map image of the traffic scene utilizing the corresponding map image reference points of the traffic scene by applying a Perspective-n-Points algorithm (PnP algorithm) (Examiner Note: the perspective points algorithm is well-known in the art and appears to band solving defined as taking as an input 3D points and their corresponding 2D coordinates and then preceding to estimate a camera’s pose in a 3D space); to…each of individual frames (see at least [Page 2, left column]; “First, accurate calibration of the cameras is necessary to convert 2D pixels to 3D locations. This can be done manually using labeled point correspondences or automatically using vanishing points calculated from geometric primitives [16][17][18] and objects with known shapes [19]. Typically, the automated calibration algorithm also sets up a 3D world reference frame (not related to any predefined map),” and [Page 2, right column]; “CAROM uses a pinhole camera model and assumes the camera distortion is negligible, as illustrated in Fig. 3. The ground is modeled either as a flat surface corresponding to a 2D satellite image map or a 3D surface with a high-resolution 3D mesh map, as shown in Fig. 4. There are three reference frames: (1) the camera frame in image pixel coordinates, (2) the world frame in metric coordinates, and (3) the map frame in map coordinates. For a 2D map, the origin is the top-left corner, the axes follow east-south directions, and the unit is a map pixel (as in Fig. 3). For a 3D map, the origin can be any point on the ground surface, the axes follow east north- up directions, and the coordinates use the metric unit (as in Fig. 4). The calibration procedure constructs two sets of parameters: (1) a camera projection matrix from the world frame to the camera frame, (2) a transformation between the map frame and the world frame. Since the traffic monitoring cameras do not move, we only need to run the calibration procedure once for each camera in the following steps. First, we label a set of at least six-point correspondences on the map and the image, typically using the lane markers and features on the ground. Second, we create the world frame and compute the transformation between the world frame to the map frame. Usually, the XOY plane of the world frame is the ground plane in the 2D map and the x-axis follows the traffic moving direction,” while not explicitly stated as using the PnP algorithm, the method of Lu as described above appears to complete the same process and therefore corresponds to the PnP algorithm) of the aerial video by movement of a drone over the traffic scene having recorded the aerial video of the traffic scene (see at least [Page 2, left column]; “we evaluate the vehicle localization and velocity measurement performance using both differential GPS and drone videos”).
Lu does not disclose correct for camera pose drift.
Jagadeesan, in the same field of endeavor, teaches correct for camera pose drift (see at least [0221]; “the camera pose is updated in an iterative process and the reprojection of stored feature points is updated accordingly,” and [0211]; “first module 504 tracks the camera motion between adjacent video frames according to ORB feature matching, which is mainly based on a novel and robust PnP algoritl1m called DynamicR1PPnP. A second module 508 aims to refine the camera motion estimation results at key frames and eliminate the accumulative error, which is based on the minimization of ICP and bundle adjustment (BA) costs.”).
Therefore, it would have been obvious for one of ordinary skill in the art, before the effective filing date of the claimed invention with a reasonable expectation of success to have modified the traffic reconstruction system of Lu with the camera pose updates of Jagadeesan. One of ordinary skill in the art would have been motivated to make this modification for the benefit of creating a more accurate navigation method (see at least Jagadeesan; [0003-0008]).
Regarding claim Lu in view of Jagadeesan renders obvious all of the limitations of claim 5. Additionally, Lu discloses wherein the processing circuitry is further configured to:
calibrate the frames of the aerial video with the map image of the traffic scene utilizing the corresponding map image reference points of the traffic scene (see at least [Page 2, left column]; “First, accurate calibration of the cameras is necessary to convert 2D pixels to 3D locations. This can be done manually using labeled point correspondences or automatically using vanishing points calculated from geometric primitives [16][17][18] and objects with known shapes [19]. Typically, the automated calibration algorithm also sets up a 3D world reference frame (not related to any predefined map),” and [Page 2, right column]; “CAROM uses a pinhole camera model and assumes the camera distortion is negligible, as illustrated in Fig. 3. The ground is modeled either as a flat surface corresponding to a 2D satellite image map or a 3D surface with a high-resolution 3D mesh map, as shown in Fig. 4. There are three reference frames: (1) the camera frame in image pixel coordinates, (2) the world frame in metric coordinates, and (3) the map frame in map coordinates. For a 2D map, the origin is the top-left corner, the axes follow east-south directions, and the unit is a map pixel (as in Fig. 3). For a 3D map, the origin can be any point on the ground surface, the axes follow east north- up directions, and the coordinates use the metric unit (as in Fig. 4). The calibration procedure constructs two sets of parameters: (1) a camera projection matrix from the world frame to the camera frame, (2) a transformation between the map frame and the world frame. Since the traffic monitoring cameras do not move, we only need to run the calibration procedure once for each camera in the following steps. First, we label a set of at least six-point correspondences on the map and the image, typically using the lane markers and features on the ground. Second, we create the world frame and compute the transformation between the world frame to the map frame. Usually, the XOY plane of the world frame is the ground plane in the 2D map and the x-axis follows the traffic moving direction,” while not explicitly stated as using the PnP algorithm, the method of Lu as described above appears to complete the same process and therefore corresponds to the PnP algorithm) by processing circuitry further configured to:
compute, via the processing circuitry, a 3D pose of camera (see at least [Page 2, left column]; “First, accurate calibration of the cameras is necessary to convert 2D pixels to 3D locations. This can be done manually using labeled point correspondences or automatically using vanishing points calculated from geometric primitives [16][17][18] and objects with known shapes [19]. Typically, the automated calibration algorithm also sets up a 3D world reference frame (not related to any predefined map). Second, in addition to accurate vehicle detection and tracking on the 2D images, robust estimation of vehicle 3D pose and vehicle dimension is required,” pose of the camera is detected) affixed to the drone concurrent with the recording of the aerial video of the traffic scene by the drone (see at least [Page 2, left column]; “we evaluate the vehicle localization and velocity measurement performance using both differential GPS and drone videos,” the video may be drone video); and
calibrate, via the processing circuitry, the 3D pose of the camera with the corresponding map image reference points of the traffic scene (see at least [Page 2, right column]; “CAROM uses a pinhole camera model and assumes the camera distortion is negligible, as illustrated in Fig. 3. The ground is modeled either as a flat surface corresponding to a 2D satellite image map or a 3D surface with a high-resolution 3D mesh map, as shown in Fig. 4. There are three reference frames: (1) the camera frame in image pixel coordinates, (2) the world frame in metric coordinates, and (3) the map frame in map coordinates. For a 2D map, the origin is the top-left corner, the axes follow east-south directions, and the unit is a map pixel (as in Fig. 3). For a 3D map, the origin can be any point on the ground surface, the axes follow east north- up directions, and the coordinates use the metric unit (as in Fig. 4). The calibration procedure constructs two sets of parameters: (1) a camera projection matrix from the world frame to the camera frame, (2) a transformation between the map frame and the world frame. Since the traffic monitoring cameras do not move, we only need to run the calibration procedure once for each camera in the following steps. First, we label a set of at least six-point correspondences on the map and the image, typically using the lane markers and features on the ground. Second, we create the world frame and compute the transformation between the world frame to the map frame. Usually, the XOY plane of the world frame is the ground plane in the 2D map and the x-axis follows the traffic moving direction,” the map corresponds to Applicant’s satellite map imaging).
Regarding claim 8 Lu in view of Jagadeesan renders obvious all of the limitations of claim 1. Additionally, Lu discloses wherein the processing circuitry is further configured to: obtain the source input having the aerial video of the traffic scene via one or more: a low flying aerial platform; or a drone (see at least [Page 5, right column]; “for the second site, we flew a drone and capture videos from 80 meters above the road. We processed the drone videos to obtain the vehicle location and velocity”).
Regarding claim 9 Lu in view of Jagadeesan renders obvious all of the limitations of claim 1. Additionally, Lu discloses wherein the processing circuitry is further configured to: obtain the map image of the traffic scene via one or more of: a publicly accessible source of satellite imagery containing at least the traffic scene; a subscription-based source of the satellite imagery containing at least the traffic scene; and a publicly accessible Geographic Information System (GIS) source containing at least the traffic scene (see at least [Page 5, left column]; “We constructed the 2D map using a satellite image at the place where the camera is mounted. Many online map services (such as Google Maps) offer satellite images”).
Regarding claim 10 Lu in view of Jagadeesan renders obvious all of the limitations of claim 1. Additionally, Lu discloses wherein the processing circuitry is further configured to: compute a velocity of each of the vehicles in the traffic scene using motion derived from a comparison of keypoints within the frames of the aerial video and a frame rate of the aerial video (see at least [Page 4, left column]; “The system first averages the length of the inlier vectors of the RANSAC results obtained in the previous step and use this length as the distance of vehicle movement from the current image and the previous image. Next, it also obtains the corresponding distance in the world frame using the vehicle’s location, heading, and the transformation T. After that, the system uses the linked list of associated instances to aggregate these distances calculated from previous image pairs in sequence until the total distance exceeds a threshold or up to a certain amount of steps (5 m or 30 steps in our implementation). Finally, the velocity is calculated from this aggregated distance, the number of frame pairs, and the frame interval time,” the vehicle positioning in multiple frames is compared to determine the velocity of the vehicle in the traffic scene, while not explicitly stated as using the keypoints for the comparison it is well-known that the keypoints represent in part positioning of the vehicle).
Regarding claim 11 Lu in view of Jagadeesan renders obvious all of the limitations of claim 1. Additionally, Lu discloses wherein the processing circuitry is further configured to:
execute, via the processing circuitry, a pinhole camera model with no distortion (see at least [Page 2, right column]; “CAROM uses a pinhole camera model and assumes the camera distortion is negligible”);
execute, via the processing circuitry, a flat ground model (see at least [Page 2, right column]; “the ground is modeled either as a flat surface corresponding to a 2D satellite image map or a 3D surface with a high-resolution 3D mesh map”); and
obtain as the source input, the aerial video of the traffic scene including the vehicles that traverse the traffic scene from the pinhole camera model and the flat ground model (see at least Fig. 2; “input video frames,” and [Page 1, right column]; “The traffic monitoring cameras are commonly mounted on road infrastructures with the advantage of covering a large area. Hence, they can be used to objectively assess the operational safety by calculating a set of safety metrics [1] directly from vehicle movements captured on the videos. Meanwhile, the information of the surrounding traffic scene obtained by these cameras can complement the perception of AVs because the in-vehicle sensors can only reach places in the line-of-sight,” since the cameras are located above the AVS they are considered to provide an aerial view).
Regarding claim 12 Lu in view of Jagadeesan renders obvious all of the limitations of claim 1. Additionally, Lu discloses wherein the processing circuitry is further configured to:
execute, via the processing circuitry, a vehicle model fitting algorithm to identify specific vehicle models aggregate, via the vehicle model fitting algorithm, collection of publicly available 3D vehicle models (see at least [Page 4, right column]; “Our objective is to reconstruct the vehicle shape and representing it in a fixed-sized data structure. Here H can be a candidate. However, it usually differs from the actual vehicle shape due to the limited view angles in the voxel carving process and errors in the localization results. To solve this problem, we construct a shape prior model from 80 different 3D CAD vehicle models and fit the model to the reconstructed histogram H by the following procedures.”);
pre-process the aggregated publicly available 3D vehicle models to generate a set of candidate vehicle models having a 1:1 scale corresponding to real-world vehicles (see at least [Page 5, left column]; “The size of the 3D model is scaled to fit the vehicle 3D bounding box”); and
track the vehicles that traverse the traffic scene across the individual frames of the aerial video utilizing the candidate vehicle models (see at least [Page 4, left column]; “The system first averages the length of the inlier vectors of the RANSAC results obtained in the previous step and use this length as the distance of vehicle movement from the current image and the previous image. Next, it also obtains the corresponding distance in the world frame using the vehicle’s location, heading, and the transformation T. After that, the system uses the linked list of associated instances to aggregate these distances calculated from previous image pairs in sequence until the total distance exceeds a threshold or up to a certain number of steps (5 m or 30 steps in our implementation). Finally, the velocity is calculated from this aggregated distance, the number of frame pairs, and the frame interval time,” the vehicle positioning in multiple frames is compared to determine the velocity of the vehicle in the traffic scene, while not explicitly stated as using the keypoints for the comparison it is well-known that the keypoints represent in part positioning of the vehicle).
Regarding claim 16 Lu in view of Jagadeesan renders obvious all of the limitations of claim 1. Additionally, Lu discloses wherein the processing circuitry is further configured to:
subsequent to output of the vehicle metrics for one or more of the vehicles that traverse the traffic scene, output, via the processing circuitry and for display, a simulation of one or more of the vehicles that traverse the traffic scene (see at least [Page 5, left column]; “To replay a traffic scene captured by the cameras, we built two visualizers, one using the 2D map and the other using the 3D map, as shown in Fig. 1. Here, a traffic scene is defined as the collection of the road environment (i.e., the map) and vehicles captured by a specific camera within a certain period (i.e., the tracking results). Both visualizers transform the vehicle states to the map frame using the calibration results and animate the vehicle movement. We use a template 3D mesh model for each vehicle type or the reconstructed vehicle models for the 3D animation. The size of the 3D model is scaled to fit the vehicle 3D bounding box. Besides, during the replay, the user can modify the speed of one specific vehicle, and the visualizer can “re-simulate” this vehicle from the modified states following the recorded trajectory while keeping replaying other vehicles”);
receive as input, modifications to physical properties of the one or more of the vehicles that traverse the traffic scene within the simulation (see at least [Page 5, left column]; “Besides, during the replay, the user can modify the speed of one specific vehicle, and the visualizer can “re-simulate” this vehicle from the modified states following the recorded trajectory while keeping replaying other vehicles”); and
responsive to receipt of the input, the modifications to the physical properties of the one or more of the vehicles that traverse the traffic scene within the simulation, output, via the processing circuitry and for display, a re-simulation of the one or more of the vehicles that traverse the traffic scene using the modifications to the physical properties of the one or more of the vehicles received as input (see at least [Page 5, left column]; “Besides, during the replay, the user can modify the speed of one specific vehicle, and the visualizer can “re-simulate” this vehicle from the modified states following the recorded trajectory while keeping replaying other vehicles”).
Regarding claim 17 Lu discloses a computer-implemented method (see at least [page 1, right column]; “we propose CAROM, a framework that can extract 3D information from the videos, generate a series of structured data records of vehicle states, and reconstruct traffic scenes on a 2D map or a 3D map, as shown in Fig 1”) comprising:
obtaining as source input, aerial video of a traffic scene including vehicles that traverse the traffic scene (see at least [Page 1, right column]; “The traffic monitoring cameras are commonly mounted on road infrastructures with the advantage of covering a large area. Hence, they can be used to objectively assess the operational safety by calculating a set of safety metrics [1] directly from vehicle movements captured on the videos. Meanwhile, the information of the surrounding traffic scene obtained by these cameras can complement the perception of AVs because the in-vehicle sensors can only reach places in the line-of-sight,” and [Page 5, right column]; “for the second site, we flew a drone and capture videos from 80 meters above the road. We processed the drone videos to obtain the vehicle location and velocity”);
obtaining a map image of the traffic scene distinct from any aerial image of the traffic scene within the source input (see at least [Page 2, right column]; “The ground is modeled…as a flat surface corresponding to a 2D satellite image map,” and [Page 5, left column]; “We constructed the 2D map using a satellite image at the place where the camera is mounted. Many online map services (such as Google Maps) offer satellite images”);
determining aerial image reference points of the traffic scene present within the source input which correspond to map image reference points of the traffic scene present within the map image of the traffic scene (see at least [Page 2, right column]; “The calibration procedure constructs two sets of parameters: (1) a camera projection matrix from the world frame to the camera frame, (2) a transformation between the map frame and the world frame. Since the traffic monitoring cameras do not move, we only need to run the calibration procedure once for each camera in the following steps. First, we label a set of at least six-point correspondences on the map and the image, typically using the lane markers and features on the ground.”);
responsive to determining the aerial image reference points of the traffic scene present within the source input which correspond to map image reference points of the traffic scene present within the map image of the traffic scene, generating calibrated images of the traffic scene from frames of the aerial video of the traffic scene by calibrating the frames of the aerial video with the map image of the traffic scene utilizing the corresponding map image reference points of the traffic scene (see at least [Page 2, right column – page 3, left column]; “The calibration procedure constructs two sets of parameters: (1) a camera projection matrix from the world frame to the camera frame, (2) a transformation between the map frame and the world frame. Since the traffic monitoring cameras do not move, we only need to run the calibration procedure once for each camera in the following steps. First, we label a set of at least six-point correspondences on the map and the image, typically using the lane markers and features on the ground. Second, we create the world frame and compute the transformation between the world frame to the map frame. Usually, the XOY plane of the world frame is the ground plane in the 2D map and the x-axis follows the traffic moving direction. Third, we transform the labeled points on the map to the world frame and compute the camera projection matrix from the point correspondences [34]. Optionally, the calibration of the camera can be automated [16]”)…
…determining multiple unique keypoints on the vehicles that traverse the traffic scene (see at least [Page 2, left column]; “the 3D representation of a vehicle can be a point with an orientation vector, a 3D bounding box, a few key points, a wireframe model, or a parametric 3D shape model”);
tracking the vehicles that traverse the traffic scene across the frames of the aerial video based on information derived from the multiple unique keypoints (see at least [Page 2, left column]; “Second, in addition to accurate vehicle detection and tracking on the 2D images, robust estimation of vehicle 3D pose and vehicle dimension is required. The 3D representation of a vehicle can be a point with an orientation vector [20], a 3D bounding box [16][21][22], a few key points [23], a wireframe model [24][25], or a parametric 3D shape model [26]. The location and speed of a vehicle are typically determined from three pieces of information: (1) the 2D locations on the images, (2) the 3D poses of the vehicle, and (3) the transformation between image coordinates and the ground coordinates obtained from the camera calibration results. Usually, the vehicle states are also estimated jointly through a filtering process by considering the vehicle kinematics or dynamics [27][28]. The vehicle shape can be reconstructed using stereo cameras [29] or monocular cameras [25][30][31] through a sequence of algorithms for depth estimation, model fitting, and shape optimization. Our paper is based on the existing works for several individual computer vision tasks and we integrated them to a unified framework that extracts the location, speed, and vehicle shape in the 3D space. Further, our tracking results allow the vehicle movements to be replayed on a 2D map or a 3D map so as to support traffic analysis tasks,” the 3D reconstruction with contains key points is used to track the vehicle throughout the scene); and
outputting vehicle metrics for one or more of the vehicles that traverse the traffic scene (see at least [Page 3, left column]; “For each detected object instance, the system crops a square patch from the image just large enough to contain its 2D bounding box, resizes the cropped patch and runs a classifier to predict its type. The following types are used: {pedestrian, two-wheelers, bus, mini-truck, semi-truck, pickup-truck, convertible, coupe, sedan, all-terrain vehicle, minivan, van, SUV, trailer}.”).
Lu does not disclose wherein generating the calibrated images includes tracking at least a subset of the aerial image reference points across multiple frames of the aerial video and updating a camera pose associated with frames of the aerial video based on the tracked aerial image reference points using a perspective-n-point computation to compensate for camera pose drift.
Jagadeesan, in the same field of endeavor, teaches wherein generating the calibrated images includes tracking at least a subset of the aerial image reference points across multiple frames of the aerial video and updating a camera pose associated with frames of the aerial video based on the tracked aerial image reference points using a perspective-n-point computation to compensate for camera pose drift (see at least [abstract]; “Systems and methods for creating images of an environment includes controlling at least one camera to acquire imaging data from the environment and selecting, from the imaging data, a three-dimensional-two-dimensional correspondence as a control point for use in a perspective-n-point problem to determine a position and orientation of the at least one camera from n known correspondences between three-dimensional object points and their two-dimensional image projections in the environment,” and [0221]; “the camera pose is updated in an iterative process and the reprojection of stored feature points is updated accordingly,” and [0211]; “first module 504 tracks the camera motion between adjacent video frames according to ORB feature matching, which is mainly based on a novel and robust PnP algoritl1m called DynamicR1PPnP. A second module 508 aims to refine the camera motion estimation results at key frames and eliminate the accumulative error, which is based on the minimization of ICP and bundle adjustment (BA) costs.”).
Therefore, it would have been obvious for one of ordinary skill in the art, before the effective filing date of the claimed invention with a reasonable expectation of success to have modified the traffic reconstruction system of Lu with the camera pose updates of Jagadeesan. One of ordinary skill in the art would have been motivated to make this modification for the benefit of creating a more accurate navigation method (see at least Jagadeesan; [0003-0008]).
Regarding claim 18 Lu in view of Jagadeesan renders obvious all of the limitations of claim 17. Additionally, Lu discloses further comprising: executing a CAR-On-Map ("CAROM") air framework for vehicle localization and traffic scene reconstruction using the aerial video of the traffic scene (see at least [page 1, right column]; “To address these issues, we propose CAROM, a framework that can extract 3D information from the videos, generate a series of structured data records of vehicle states, and reconstruct traffic scenes on a 2D map or a 3D map, as shown in Fig 1,” and [Page 2, left column]; “We constructed a vehicle tracking, localization, and velocity measurement pipeline using videos taken by monocular road traffic monitoring cameras”).
Regarding claim 19 Lu discloses computer-readable storage media comprising instructions that, when executed (while Lu does not explicitly state that processing circuitry is a part of the framework of the system it would be obvious for a system to comprise both a processor and computer readable media), configure processing circuitry to:
obtain as source input, aerial video of a traffic scene including vehicles that traverse the traffic scene (see at least [Page 1, right column]; “The traffic monitoring cameras are commonly mounted on road infrastructures with the advantage of covering a large area. Hence, they can be used to objectively assess the operational safety by calculating a set of safety metrics [1] directly from vehicle movements captured on the videos. Meanwhile, the information of the surrounding traffic scene obtained by these cameras can complement the perception of AVs because the in-vehicle sensors can only reach places in the line-of-sight,” and [Page 5, right column]; “for the second site, we flew a drone and capture videos from 80 meters above the road. We processed the drone videos to obtain the vehicle location and velocity”);
obtain a map image of the traffic scene distinct from any aerial image of the traffic scene within the source input (see at least [Page 2, right column]; “The ground is modeled…as a flat surface corresponding to a 2D satellite image map,” and [Page 5, left column]; “We constructed the 2D map using a satellite image at the place where the camera is mounted. Many online map services (such as Google Maps) offer satellite images”);
determine aerial image reference points of the traffic scene present within the source input which correspond to map image reference points of the traffic scene present within the map image of the traffic scene (see at least [Page 2, right column]; “The calibration procedure constructs two sets of parameters: (1) a camera projection matrix from the world frame to the camera frame, (2) a transformation between the map frame and the world frame. Since the traffic monitoring cameras do not move, we only need to run the calibration procedure once for each camera in the following steps. First, we label a set of at least six-point correspondences on the map and the image, typically using the lane markers and features on the ground.”);
responsive to determination of the aerial image reference points of the traffic scene present within the source input which correspond to map image reference points of the traffic scene present within the map image of the traffic scene, generate calibrated images of the traffic scene from frames of the aerial video of the traffic scene by calibrating the frames of the aerial video with the map image of the traffic scene utilizing the corresponding map image reference points of the traffic scene (see at least [Page 2, right column – page 3, left column]; “The calibration procedure constructs two sets of parameters: (1) a camera projection matrix from the world frame to the camera frame, (2) a transformation between the map frame and the world frame. Since the traffic monitoring cameras do not move, we only need to run the calibration procedure once for each camera in the following steps. First, we label a set of at least six-point correspondences on the map and the image, typically using the lane markers and features on the ground. Second, we create the world frame and compute the transformation between the world frame to the map frame. Usually, the XOY plane of the world frame is the ground plane in the 2D map and the x-axis follows the traffic moving direction. Third, we transform the labeled points on the map to the world frame and compute the camera projection matrix from the point correspondences [34]. Optionally, the calibration of the camera can be automated [16]”)…
…determine multiple unique keypoints on the vehicles that traverse the traffic scene (see at least [Page 2, left column]; “the 3D representation of a vehicle can be a point with an orientation vector, a 3D bounding box, a few key points, a wireframe model, or a parametric 3D shape model”);
track the vehicles that traverse the traffic scene across the frames of the aerial video based on information derived from the multiple unique keypoints (see at least [Page 2, left column]; “Second, in addition to accurate vehicle detection and tracking on the 2D images, robust estimation of vehicle 3D pose and vehicle dimension is required. The 3D representation of a vehicle can be a point with an orientation vector [20], a 3D bounding box [16][21][22], a few key points [23], a wireframe model [24][25], or a parametric 3D shape model [26]. The location and speed of a vehicle are typically determined from three pieces of information: (1) the 2D locations on the images, (2) the 3D poses of the vehicle, and (3) the transformation between image coordinates and the ground coordinates obtained from the camera calibration results. Usually, the vehicle states are also estimated jointly through a filtering process by considering the vehicle kinematics or dynamics [27][28]. The vehicle shape can be reconstructed using stereo cameras [29] or monocular cameras [25][30][31] through a sequence of algorithms for depth estimation, model fitting, and shape optimization. Our paper is based on the existing works for several individual computer vision tasks and we integrated them to a unified framework that extracts the location, speed, and vehicle shape in the 3D space. Further, our tracking results allow the vehicle movements to be replayed on a 2D map or a 3D map so as to support traffic analysis tasks,” the 3D reconstruction with contains key points is used to track the vehicle throughout the scene); and
output vehicle metrics for one or more of the vehicles that traverse the traffic scene (see at least [Page 3, left column]; “For each detected object instance, the system crops a square patch from the image just large enough to contain its 2D bounding box, resizes the cropped patch and runs a classifier to predict its type. The following types are used: {pedestrian, two-wheelers, bus, mini-truck, semi-truck, pickup-truck, convertible, coupe, sedan, all-terrain vehicle, minivan, van, SUV, trailer}.”).
Lu does not disclose wherein generating the calibrated images includes tracking at least a subset of the aerial image reference points across multiple frames of the aerial video and updating a camera pose associated with frames of the aerial video based on the tracked aerial image reference points using a perspective-n-point computation to compensate for camera pose drift.
Jagadeesan, in the same field of endeavor, teaches wherein generating the calibrated images includes tracking at least a subset of the aerial image reference points across multiple frames of the aerial video and updating a camera pose associated with frames of the aerial video based on the tracked aerial image reference points using a perspective-n-point computation to compensate for camera pose drift (see at least [abstract]; “Systems and methods for creating images of an environment includes controlling at least one camera to acquire imaging data from the environment and selecting, from the imaging data, a three-dimensional-two-dimensional correspondence as a control point for use in a perspective-n-point problem to determine a position and orientation of the at least one camera from n known correspondences between three-dimensional object points and their two-dimensional image projections in the environment,” and [0221]; “the camera pose is updated in an iterative process and the reprojection of stored feature points is updated accordingly,” and [0211]; “first module 504 tracks the camera motion between adjacent video frames according to ORB feature matching, which is mainly based on a novel and robust PnP algoritl1m called DynamicR1PPnP. A second module 508 aims to refine the camera motion estimation results at key frames and eliminate the accumulative error, which is based on the minimization of ICP and bundle adjustment (BA) costs.”).
Therefore, it would have been obvious for one of ordinary skill in the art, before the effective filing date of the claimed invention with a reasonable expectation of success to have modified the traffic reconstruction system of Lu with the camera pose updates of Jagadeesan. One of ordinary skill in the art would have been motivated to make this modification for the benefit of creating a more accurate navigation method (see at least Jagadeesan; [0003-0008]).
Regarding claim 20 Lu discloses all of the limitations of claim 19. Additionally, Lu discloses further configured to: execute, via the processing circuitry, a CAR-On-Map ("CAROM") air framework for vehicle localization and traffic scene reconstruction using the aerial video of the traffic scene (see at least [page 1, right column]; “To address these issues, we propose CAROM, a framework that can extract 3D information from the videos, generate a series of structured data records of vehicle states, and reconstruct traffic scenes on a 2D map or a 3D map, as shown in Fig 1,” and [Page 2, left column]; “We constructed a vehicle tracking, localization, and velocity measurement pipeline using videos taken by monocular road traffic monitoring cameras”).
Claim(s) 7 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Lu in view of Jagadeesan, as applied to claim 1 above, in view of US-20220366167 (hereinafter, “Desai”).
Regarding claim Lu in view of Jagadeesan renders obvious all of the limitations of claim 1. Lu does not disclose wherein the processing circuitry is further configured to:
execute, via the processing circuitry, a keypoint Region-based Convolutional Neural Network model (keypoint RCNN model) to determine the multiple unique keypoints on the vehicles; and
create, via the keypoint RCNN model, bounding boxes within the calibrated images of the traffic scene encompassing each of the vehicles utilizing the multiple unique keypoints determined for each of the respective vehicles.
Desai, in the same field of endeavor, teaches wherein the processing circuitry is further configured to:
execute, via the processing circuitry, a keypoint Region-based Convolutional Neural Network model (keypoint RCNN model) to determine the multiple unique keypoints on the vehicles (see at least [0040]; “The machine learning model 210 identifies for an input aerial image where each aircraft is located in the image and determines a bounding polygon that encompasses each identified aircraft. The machine learning model 210 may be a neural network, a deep learning model, a convolutional neural network, etc. In some embodiments, the machine learning model 210 may utilize known image processing techniques on areas of the image within bounding polygons to identify a plurality of keypoints. A keypoint will be located at a pixel location of the aircraft on the aerial image and the plurality”); and
create, via the keypoint RCNN model, bounding boxes within the calibrated images of the traffic scene encompassing each of the vehicles utilizing the multiple unique keypoints determined for each of the respective vehicles (see at least [0040]; “The machine learning model 210 identifies for an input aerial image where each aircraft is located in the image and determines a bounding polygon that encompasses each identified aircraft. The machine learning model 210 may be a neural network, a deep learning model, a convolutional neural network, etc. In some embodiments, the machine learning model 210 may utilize known image processing techniques on areas of the image within bounding polygons to identify a plurality of keypoints. A keypoint will be located at a pixel location of the aircraft on the aerial image and the plurality”).
Therefore, it would have been obvious for one of ordinary skill in the art, before the effective filing date of the claimed invention with a reasonable expectation of success to have modified the traffic reconstruction system of Lu as modified by Jagadeesan with the neural network of Desai. One of ordinary skill in the art would have been motivated to make this modification for the benefit of detecting and classifying vehicles accurately and efficiently (see at least Desai; [0002-0007]).
Regarding claim 15 Lu in view of Jagadeesan renders obvious all of the limitations of claim 1. Lu does not disclose wherein the processing circuitry is further configured to:
determine each of the multiple unique keypoints on the vehicles that traverse the traffic scene based on an identifiable vehicle keypoint of each respective one of the vehicles that traverse the traffic scene, wherein each identifiable vehicle keypoint is selected from the group comprising: corner of a vehicle roof top; corner of a vehicle front windshield; corner of a vehicle rear window; center of a vehicle front light; center of a vehicle rear light; center of a vehicle front bumper; center of a vehicle rear bumper; center of a vehicle wheel; corner of a vehicle chassis bottom surface; outermost corner of a vehicle side mirror; corner of a vehicle front door window; wheel-to-ground contact point of a vehicle; and center of a vehicle front brand logo.
Desai, in the same field of endeavor, teaches wherein the processing circuitry is further configured to:
determine each of the multiple unique keypoints on the vehicles that traverse the traffic scene based on an identifiable vehicle keypoint of each respective one of the vehicles that traverse the traffic scene, wherein each identifiable vehicle keypoint is selected from the group comprising: corner of a vehicle roof top; corner of a vehicle front windshield; corner of a vehicle rear window; center of a vehicle front light; center of a vehicle rear light; center of a vehicle front bumper; center of a vehicle rear bumper; center of a vehicle wheel; corner of a vehicle chassis bottom surface; outermost corner of a vehicle side mirror; corner of a vehicle front door window; wheel-to-ground contact point of a vehicle; and center of a vehicle front brand logo (see at least [0040]; “The machine learning model 210 identifies for an input aerial image where each aircraft is located in the image and determines a bounding polygon that encompasses each identified aircraft. The machine learning model 210 may be a neural network, a deep learning model, a convolutional neural network, etc. In some embodiments, the machine learning model 210 may utilize known image processing techniques on areas of the image within bounding polygons to identify a plurality of keypoints. A keypoint will be located at a pixel location of the aircraft on the aerial image and the plurality,” and [0031]; “in another example embodiment (not shown) a system may classify other vehicles captured in an aerial image. For example, a system may classify motor vehicles (e.g., cars, trucks, etc.),” and [0036]; “In some embodiments, the plurality of keypoints may include five keypoints. The keypoints may include a nose keypoint, a center keypoint, a left wing keypoint, a right wing keypoint, and a tail keypoint,” it would be obvious that the keypoints of Desai may be modified to accommodate a ground vehicle rather than an aircraft and the two are obvious variants of one another); and
Therefore, it would have been obvious for one of ordinary skill in the art, before the effective filing date of the claimed invention with a reasonable expectation of success to have modified the traffic reconstruction system of Lu as modified by Jagadeesan with the neural network of Desai. One of ordinary skill in the art would have been motivated to make this modification for the benefit of detecting and classifying vehicles accurately and efficiently (see at least Desai; [0002-0007]).
Claim(s) 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Lu in view of Jagadeesan, as applied to claim 1 above, in view of "Active shape models-their training and application” (hereinafter, “Cootes”).
Regarding claim 13 Lu in view of Jagadeesan renders obvious all of the limitations of claim 12. Additionally, Lu discloses wherein the models are vehicle models (see at least [Page 4, right column]; “we construct a shape prior model from 80 different 3D CAD vehicle models”).
Lu does not disclose wherein the processing circuitry is further configured to: pre-process the aggregated publicly available 3D vehicle models to generate a set of candidate vehicle models by: concatenating (x, y, z) coordinates of all of the multiple unique keypoints on the object onto the set of candidate object models as a shape vector {S1} when corresponding coordinates are available for each respective object within the set of candidate object models.
Cootes, in the same field of endeavor, discloses wherein the processing circuitry is further configured to:
pre-process the aggregated publicly available 3D vehicle models to generate a set of candidate vehicle models by: concatenating (x, y, z) coordinates of all of the multiple unique keypoints (see at least [Page 5, left column]; “Our modeling method works by examining the statistics of the coordinates of the labeled points over the training set. In order to be able to compare equivalent points from different shapes, they must be aligned with respect to a set of axes”) on the object onto the set of candidate object models as a shape vector {S1} when corresponding coordinates are available for each respective object within the set of candidate object models (see at least [Page 6, left column]; “The approach given below attempts to model the shape of this cloud in a high dimensional space, and hence to capture the relationships between the positions of the individual landmark points. We make the assumption that the cloud is approximately ellipsoidal, and proceed to calculate its center (giving a mean shape) and its major axes, which give a way of moving around the cloud. Later we will discuss the implications of this ellipsoid assumption breaking down. Given a set of N aligned shapes, the mean shape, i (the center of the ellipsoidal Allowable Shape Domain), is calculated,” the coordinates of the landmark points, corresponding to Applicant’s keypoints are used to determine the mean shape of the model).
Therefore, it would have been obvious for one of ordinary skill in the art, before the effective filing date of the claimed invention with a reasonable expectation of success to have modified the traffic reconstruction system of Lu as modified by Jagadeesan with the shape modeling of Cootes. One of ordinary skill in the art would have been motivated to make this modification for the benefit of accurately recognizing and locating objects in automated image interpretation (see at least Cootes; [Page 20, left column; conclusions]).
Regarding claim 14 Lu in view of Jagadeesan and Cootes renders obvious all of the limitations of claim 13. Additionally, Lu discloses wherein the processing circuitry is further configured to: create bounding boxes within the calibrated images of the traffic scene encompassing each of the vehicles utilizing the multiple unique keypoints determined for each of the respective (see at least Fig. 6 and [Page 3, right column]; “Finally, using all three vanishing points, the 3D bounding box of a vehicle is computed from the contour of its segmentation mask using the tangent line method [16] (illustrated in Fig. 5),” the vanishing points correspond to applicant’s keypoints)
execute, via processing circuitry, Principal Component Analysis (PCA) on each of the candidate vehicle models as the shape vector {Si} for all candidate vehicle models (see at least [Page 4, right column]; “Step (2): With all 80 model vectors, we run Principal Component Analysis (PCA) to reduce their dimension from n*m to 20. After this step we obtained 20 principal component vectors (denoted as a n*m-by-20 matrix S). The vector set {ui} and matrix S are called the vehicle shape prior, similar to the shape prior models in shape analysis and multiple view reconstruction [40][29][30].”).
Lu does not disclose determine mean shape sm for each of the candidate vehicle models; and
for each respective one of the objects, identify a best fit among the candidate object models based on a comparison of the multiple unique keypoints and the bounding boxes created within the calibrated images using the shape vector {S} and the mean shape sm determined for each of the candidate objects models.
Cootes, in the same field of endeavor, teaches determine mean shape sm for each of the candidate vehicle models (see at least [Page 6, left column]; “The approach given below attempts to model the shape of this cloud in a high dimensional space, and hence to capture the relationships between the positions of the individual landmark points. We make the assumption that the cloud is approximately ellipsoidal, and proceed to calculate its center (giving a mean shape) and its major axes, which give a way of moving around the cloud. Later we will discuss the implications of this ellipsoid assumption breaking down. Given a set of N aligned shapes, the mean shape, i (the center of the ellipsoidal Allowable Shape Domain), is calculated,” the coordinates of the landmark points, corresponding to Applicant’s keypoints are used to determine the mean shape of the model); and
for each respective one of the vehicles that traverse the traffic scene, identify a best fit among the candidate vehicle models based on a comparison of the multiple unique keypoints and the bounding boxes created within the calibrated images using the shape vector {S} and the mean shape sm determined for each of the candidate vehicle models (see at least [Page 1, right column]; “Our technique relies upon each object or image structure being represented by a set of points. The points can represent the boundary, internal features, or even external ones, such as the center of a concave section of boundary. Points are placed in the same way on each of a training set of examples of the object. This is done manually, though tools are available to aid the user. The sets of points are aligned automatically to minimize the variance in distance between equivalent points. By examining the statistics of the positions of the labeled points a "Point Distribution Model" is derived. The model gives the average positions of the points, and has a number of parameters which control the main modes of variation found in the training set. Given such a model and an image containing an example of the object modeled, image interpretation involves choosing values for each of the parameters so as to find the best fit of the model to the image. We describe a technique which allows an initial very rough guess for the best shape, orientation, scale, and position to be refined by comparing the hypothesized model instance with image data, and using differences between model and image to deform the shape”).
Therefore, it would have been obvious for one of ordinary skill in the art, before the effective filing date of the claimed invention with a reasonable expectation of success to have modified the traffic reconstruction system of Lu as modified by Jagadeesan with the shape modeling of Cootes. One of ordinary skill in the art would have been motivated to make this modification for the benefit of accurately recognizing and locating objects in automated image interpretation (see at least Cootes; [Page 20, left column; conclusions]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US-20220215573 teaches a camera pose estimation method wherein camera pose angle is obtained by a solving a Perspective-n-Point algorithm
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ASHLEIGH NICOLE TURNBAUGH whose telephone number is (703)756-1982. The examiner can normally be reached Monday - Friday 9:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hitesh Patel can be reached at (571) 270-5442. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ASHLEIGH NICOLE TURNBAUGH/Examiner, Art Unit 3667
/Hitesh Patel/Supervisory Patent Examiner, Art Unit 3667
4/3/26
Prosecution Insights