Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant has added new claims 21 and 22. Claims 1-22 are currently being considered. Applicant's arguments filed 8/10/2026 have been fully considered but they are not persuasive. Applicant argues that the prior art Bylicka does not teach or suggest refining the first bounding box by minimizing the space between each object and its corresponding bounding box as required by independent claim 1 and new dependent claims 21 and 22. Furthermore, applicant argues that Bylicka’s pruning acts on candidate cross-camera associations, not the boundaries of a detected first bounding box, and that they concern association plausibility, not object-specific box refinement. Applicant argues that Goyal does not teach or suggest the claimed real-world-location projection, association, update, and prediction workflow required in independent claims 1, 12, and 20. Examiner disagrees. Under broadest reasonable interpretation of refining the bounding boxes, Bylicka performs pruning of the bounding boxes that do not meet a bounding box characteristic criteria as shown in Figure 3A. Furthermore, as described in paragraph [0068], those 3D bounding boxes that are pruned have their edge pruned so that the 2D bounding boxes forming that edge, if already established, and 3D bounding box are no longer considered an associated pair of a same subject. Paragraph [0067] discloses that process 400 may include “perform attribute pruning”. This relates to attributes of the subjects that are expected to be bounded by the 3D bounding box, and may be based on general knowledge about the subject. Thus, the size of the 3D bounding box can be limited to the general size, volume, height, and so for the related to a person. This would consider any geometrical tolerances such as camera perspective, distance from the camera, and so forth. This is another example of how Bylicka refines the first bounding box by minimizing the space between each object (in this case, person) and corresponding bounding box, based on the attributes of the person. Goyal also discloses a real-world-location projection, association, update, and prediction workflow in paragraph [0072]. Paragraph [0072] of Goyal discloses controller may be configured to (1) apply an object detection model to the first image frame to predict a location of a first bounding box of an object in the first frame, (2) apply a confidence value to the predicted location of first bounding box, (3) in response to the confidence level exceeding a predetermined threshold, estimate a second bounding box in the second image frame based on the location of the first bounding box and non-zero movement of the vehicle. Thus, the prior arts are still effective in rejecting all claims including new claims as added.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bylicka et. al. (United States Patent Application Publication US 2021/0112238 A1) in view of Goyal et. al. (United States Patent Application Publication US 2024/0185613 A1).
Regarding claim 1, Bylicka et. al. discloses a system for tracking objects comprising: one or more image sensors configured to capture one or more images; a detector processor configured to: detect one or more objects from the images, generate one or more first bounding boxes around the objects, wherein each of the first bounding boxes surrounds each of the objects (Bylicka et. al. Figure 2, Figure 7: Sensor component 706; [0033]: the 3D bounding boxes are formed from projection of 2D bounding boxes from 2D images and into a 3D space or graph to form at least two hyper-planes as sides of the 3D bounding box, and this may be repeated for each pair of objects on different 2D image views); and a tracking processor configured to: refine the first bounding boxes by minimizing space between each objects and the corresponding bounding boxes (Bylicka et. al. [0066]: process may include “prune 3D bounding boxes from graph by removing irrelevant edges”. To reduce computation load (or cost) once the 3D bounding boxes as constructed on the graph or 3D space, the system can check to determine whether or not the 3D bounding boxes meet at least some minimal characteristic conditions or criteria to be an associated subject that should be tracked. The conditions factor distinct features of the 3D bounding boxes and can be easily tailored to specific scenarios or environments. [0067] the size of the 3D bounding box can be limited to the general size, volume, height, and so forth related to a person (the context is an object being tracked) This would consider any geometrical tolerances such as camera perspective, distance from the camera, and so forth.); estimate one or more real-world locations of the objects based on the first bounding boxes (Bylicka et. al. [0066]: the conditions may be related to 3D geometry such as an approximate location and volume of the potential subject in the 3D space or graph).
However, Bylicka et. al. fails to disclose project the real-world locations of the objects into one or more second bounding boxes; receive a new detection of the objects; associate the new detection with the second bounding boxes; update the estimation of the real-world locations of the objects; and predict, based on the updated real-world locations, one or more future real-world locations of the objects.
Goyal teaches project the real-world locations of the objects into one or more second bounding boxes; receive a new detection of the objects; associate the new detection with the second bounding boxes; update the estimation of the real-world locations of the objects; and predict, based on the updated real-world locations, one or more future real-world locations of the objects (Goyal et. al. Abstract, Figure 3, 4, [0011]: processor applies an object detection model image frame to predict a location of a first bounding box of an object in the first image frame; and apply a confidence value to the predicted location of first bounding box. In response to the confidence level exceeding a predetermined threshold, the instructions may direct the processor to estimate a location of a second bounding box of the object in a second image frame based on the location of the first bounding box and non-zero movement of the vehicle and update the object detection model based on the estimated location of the second bounding box in the second image frame. The processor may further predict a location of a third bounding box of the object in a third image frame using the updated object detection model and control an operation of the vehicle based on the predicted location of the third bounding box. See also Figure 5; [0077] i.e., geographic coordinates and/or IMU).
This is important to the claimed invention because being able to predict the location of a moving object based on refined bounding boxes allows systems to make decisions based on information of the location of the object more precisely, especially when the object is in motion. Thus, it would have been obvious to one skilled in the art prior to the effective filing date of the claimed invention to have combined the teachings of Bylicka et. al. and Goyal et. al. so that these features are shown in the solution of the claimed invention.
Regarding claim 12, which recites a method for tracking objects, corresponding to the system of claim 1, which the rejection analysis is incorporated herein.
Regarding claim 20, which recites a non-transitory computer readable medium containing computer executable instructions that corresponds to the system of claim 1, which the rejection analysis is incorporated herein.
Regarding claim 2, Bylicka et. al. and Goyal et. al. disclose the system of claim 1, and Goyal et. al. further discloses wherein one or more objects comprise at least one or more vehicles (Goyal et. al. Figure 5, [0011]: …second bounding box of the object in a second image frame based on the location of the first bounding box and non-zero movement of the vehicle.)
Regarding claim 3 and claim 19, Bylicka et. al. and Goyal et. al. disclose the system of claim 1 and method of claim 12, and Goyal et. al. further discloses wherein the refinement further comprises segmenting the first bounding boxes (Goyal et. al. Figure 4: Deployed segmentation model, [0041]: processor applies the segmentation model to the pixels in image frame to produce or output a first prediction segmentation map including pixel labels. The pixel labels may identify each individual pixel as being part of an object or environment/surroundings to the object. The pixel labels may identify the boundary, shape or edge of the object. [0051]: Segmentation involves pixel-wise labelling of every pixel of an input image.).
Regarding claim 4 and claim 16, Goyal et. al. further discloses the system of claim 3 and method of claim 12, wherein the refinement further comprises segmenting the first bounding boxes by matching the segmentation to a reference image of the one or more objects (Goyal et. al. claim 13: image segmentation system, [0073]: (1) apply a segmentation model to the image to output a first predicted segmentation map including pixel labels, (5) compute a first quantity objectness score for an object in the first predicted segmentation map. [0023]: In some implementations, the bounding box may represent a portion of the image frame that is to be set apart and segmented to determine the shape and perimeter coordinates of the object or obstacle contained within the bounding box.).
Regarding claim 5, Goyal et. al. further discloses the system of claim 4, wherein the reference image comprises at least one selected from the group of a blueprint, model, or computer-aided design (CAD) (Goyal et. al. [0010]: The medium may direct the processor to apply a segmentation model to the image to output a first predicted segmentation map including pixel labels, fuse the first predicted segmentation map and the point cloud and label pixels in the point cloud.).
Regarding claim 6, Bylicka et. al. and Goyal et. al. disclose the system of claim 1, and Goyal et. al. further discloses wherein the one or more image sensors are monocular imaging devices configured for ground-to-ground applications (Goyal et. al. [0070]: Cameras may comprise a monocular/2D camera or may comprise a stereo/3D camera).
Regarding claim 7, Bylicka et. al. and Goyal et. al. disclose the system of claim 1, and Bylicka et. al. further discloses wherein the estimation of the real-world locations of the objects is further based on at least a height of the bounding boxes, a location of the camera, and a pointing angle of the camera (Bylicka et. al. [0033]: the 3D bounding boxes are formed from projection of 2D bounding boxes from 2D images and into a 3D space or graph to form at least two hyper-planes as sides of the 3D bounding box, and this may be repeated for each pair of objects on different 2D image views. [0066]-[0067]: The conditions may be related to 3D geometry such as an approximate location and volume of the potential subject in the 3D space or graph. The size of the 3D bounding box can be limited to the general size, volume, height, and so forth related to a person (object of interest). Figure 6, [0058]: multi-view association graph.).
Regarding claim 8 and claim 17, Bylicka et. al. and Goyal et. al. disclose the system of claim 1 and method of claim 12, and Bylicka et. al. further discloses wherein the projection the real-world locations of the objects into one or more second bounding boxes further comprises: tracking the objects across multiple locations across a predetermined time period, sampling the locations, and projecting the sampled locations into one or more image coordinates (Bylicka et. al. [0057]: “obtain image data of multiple images of the same scene”. This may involve obtaining multiple images of the same scene and at the same time (same or similar time stamp) as mentioned above. [0033]: the 3D bounding boxes are formed from projection of 2D bounding boxes from 2D images and into a 3D space or graph to form at least two hyper-planes as sides of the 3D bounding box, and this may be repeated for each pair of objects on different 2D image views.).
Regarding claim 9 and claim 18, Bylicka et. al. and Goyal et. al. disclose the system of claim 1 and the method of claim 12, and Goyal et. al. further discloses wherein the association of the new detection with the second bounding boxes further comprises associating the new detection with a largest average bounding box overlap among the second bounding boxes (Goyal et. al. [0035]: Several image-based correlation metrics like mutual information, cross-correlation or normalized cross-correlation can be used to optimally find the location of the bounding box within the image. Here, the largest peak indicates largest overlap among the second bounding boxes, which produces the highest cross-correlation value.).
Regarding claim 10, Bylicka et. al. and Goyal et. al. disclose the system of claim 1, and Goyal et. al. further disclose wherein the one or more images sensors comprise a monocular camera and an inertial measurement unit (IMU) (Goyal et. al. Figure 5, [0070]: Cameras may comprise a monocular/2D camera or may comprise a stereo/3D camera. [0067]: Tractor is configured to be steered in an automated fashion by controller from various cameras or sensors provided on tractor and/or according to a predefined steering routine, route or path based upon signals from GPS and/or inertial measurement units.).
Regarding claim 11, Goyal et. al. further discloses the system of claim 10, wherein the monocular camera and IMU are associated with a standalone camera, a drone, or a moving vehicle (Goyal et. al. Figure 5, [0070]: Cameras may comprise a monocular/2D camera or may comprise a stereo/3D camera. [0067]: Tractor is configured to be steered in an automated fashion by controller from various cameras or sensors provided on tractor and/or according to a predefined steering routine, route or path based upon signals from GPS and/or inertial measurement units.).
Regarding claim 13, Bylicka et. al. and Goyal et. al. disclose the method of claim 12, and Goyal et. al. further discloses wherein the updating of the estimation of the real-world location of the objects is achieved through Kalman filtering (Goyal et. al. [0011]: The estimating of the location of the second bounding box in the second image comprises applying a Kalman filter and correlating the estimated location of the second bounding box within a margin around the location of the first bounding box. [0035]: To alleviate the effects of noisy bounding boxes, high confidence bounding boxes are taken from previous frames and correlated within a margin around the estimated bounding boxes from the Kalman Filter. Several image-based correlation metrics like mutual information, cross-correlation or normalized cross-correlation can be used to optimally find the location of the bounding box within the image.).
Regarding claim 14, Bylicka et. al. and Goyal et. al. disclose the method of claim 12, and Goyal et. al. further discloses wherein the predicting of the future location is achieved through at least one selected from the group of a motion model, a Kalman filter, or a particle filter (Goyal et. al. [0035]: To alleviate the effects of noisy bounding boxes, high confidence bounding boxes are taken from previous frames and correlated within a margin around the estimated bounding boxes from the Kalman Filter. Several image-based correlation metrics like mutual information, cross-correlation or normalized cross-correlation can be used to optimally find the location of the bounding box within the image. This step ensures that the tracking capabilities of the Kalman Filter and also the pixel information content in the images are integrated.).
Regarding claim 15, Bylicka et. al. and Goyal et. al. disclose the method of claim 12, and Goyal et. al. further discloses wherein the estimation of the real-world location of the one or more objects is represented as at least one selected from the group of a global positioning system (GPS), a latitude and longitude unit, or a measurement from a position of the image sensors (Goyal et. al. Figure 5, [0070]: Cameras may comprise a monocular/2D camera or may comprise a stereo/3D camera. [0067]: Tractor is configured to be steered in an automated fashion by controller from various cameras or sensors provided on tractor and/or according to a predefined steering routine, route or path based upon signals from GPS and/or inertial measurement units.).
PNG
media_image1.png
998
1048
media_image1.png
Greyscale
Regarding claim 21, Bylicka et. al. and Goyal et. al. disclose the method for tracking objects of claim 12, and Bylicka et. al .further discloses wherein the refining, by the processor, the first bounding boxes comprises minimizing space between each object and the corresponding bounding box (Bylicka et. al. [0067]: that process 400 may include “perform attribute pruning”. This relates to attributes of the subjects that are expected to be bounded by the 3D bounding box, and may be based on general knowledge about the subject. Thus, the size of the 3D bounding box can be limited to the general size, volume, height, and so for the related to a person. This would consider any geometrical tolerances such as camera perspective, distance from the camera, and so forth. This is another example of how Bylicka refines the first bounding box by minimizing the space between each object (in this case, person) and corresponding bounding box, based on the attributes of the person.).
Regarding claim 22, Bylicka et. al. further discloses the non-transitory computer readable medium of claim 20 wherein refining, by the processor, the first bounding boxes comprises minimizing space between each object and the corresponding bounding box (Bylicka et. al. [0067]: that process 400 may include “perform attribute pruning”. This relates to attributes of the subjects that are expected to be bounded by the 3D bounding box, and may be based on general knowledge about the subject. Thus, the size of the 3D bounding box can be limited to the general size, volume, height, and so for the related to a person. This would consider any geometrical tolerances such as camera perspective, distance from the camera, and so forth. This is another example of how Bylicka refines the first bounding box by minimizing the space between each object (in this case, person) and corresponding bounding box, based on the attributes of the person.).
Conclusion
Response to Amendment
Examiner has carefully considered the newly added claims 21-22 and reconsidered the previous claims 1-20. However, after reconsideration, Examiner believes the prior arts of record are still effective in rejecting all claims 1-22.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JESSICA YIFANG LIN whose telephone number is (571)272-6435. The examiner can normally be reached M-F 7:00am-6:15pm, with optional day off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at 571-272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JESSICA YIFANG LIN/Examiner, Art Unit 2668 August 21, 2026
/VU LE/Supervisory Patent Examiner, Art Unit 2668
