CTNF 18/807,806 CTNF 101775 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. DETAILED ACTION The United States Patent & Trademark Office appreciates the application that is submitted by the inventor/assignee. The United States Patent & Trademark Office reviewed the following application and has made the following comments below. Specification 07-29 AIA The disclosure is objected to because of the following informalities: In paragraph [0046], line 5, “in image 322 with identifier ‘car0.86’ ” does not match what is shown in Figure 3. Fig. 3 shows “ car0.96 .” In paragraph [0097], line 4, “ first stage 416 and first stage 416” should read “first stage 416 and second stage 418 .” In paragraph [0105], line 4, “methods can be performed by ___,” appears to be a blank that needs to be filled in. In paragraph [0105], line 8, “the components of the ___” appears to be a blank that needs to be filled in. In paragraph [0116], line 7, “any or all of ___” appears to be a blank that needs to be filled in . Appropriate correction is required. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20 AIA The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action: (a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made. 07-23 AIA The factual inquiries set forth in Graham v. John Deere Co. , 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103(a) are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-21 AIA Claim s 1-4, 6, and 16-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Wuu et al. (U.S. Patent Pub. No. 2023/0177804 A1, hereafter referred to as Wuu) in view of Zhang et al. (NPL “ByteTrack: Multi-object Tracking by Associating Every Detection Box,” 2022, hereafter referred to as Zhang) . Regarding Claim 1, Wuu teaches an apparatus for tracking objects (Paragraph [0010], Fig. 6, Wuu teaches a vehicle computing device which includes a vehicle perception system that tracks objects in the environment of the vehicle and associates tracking data with specific objects in the environment.) , [AltContent: arrow] [AltContent: arrow] PNG media_image1.png 709 1035 media_image1.png Greyscale the apparatus comprising: at least one memory (Paragraph [0066], Fig. 6, Wuu teaches the vehicle computing device (604) can include memory (618).) ; and at least one processor coupled to the at least one memory (Paragraph [0066], Fig. 6, Wuu teaches the vehicle computing device (604) can include processor(s) (616) and memory (618) communicatively coupled with the processor(s) (616).) and configured to: generate features based on a sensor-data frame (Paragraphs [0016], [0023], Wuu teaches the machine-learned model may determine an estimated location at the time t 0 , an estimated orientation of the object, an estimated size of the object, an estimated classification of the object, or the like, as well as any other feature that may be indicated with the object data. The machine-learned model determines the location/features of the object at t 0 based at least in part on stored tracking data associated with the object. The stored tracking data may include multiple frames of sensor data , where each individual frame is associated with a different instant of time.) ; detect an object based on the features (Paragraph [0014], Wuu teaches generating or determining object data associated with one or more objects detected in the environment. The object data may indicate a classification and/or type of object, such as whether the object is a dynamic object that is capable of movement (e.g., a vehicle, motorcycle, bicycle, pedestrian, animal, etc.) and/or a static object (e.g., building, road surface, tree, sign, barrier, curb, parked vehicle, etc.). Additionally, the object data may indicate other information associated with the object, such as velocity of the object at a time point, a confidence associated with the object, etc.) ; generate a bounding box based on the object (Paragraph [0014], Fig. 1, Wuu teaches generating or determining object data associated with one or more objects detected in the environment. The object data may include one or more bounding boxes associated with the respective objects in the environment.) ; PNG media_image2.png 272 458 media_image2.png Greyscale track the bounding box over a plurality of sensor-data frames to generate a tracklet (Paragraphs [0020], [0029], [0042], Wuu teaches an output of tracked object data from the machine-learned model which may indicate, for each object in the environment, object data associated with that object (e.g., bounding box, classification) and a trajectory traversed by that object through the environment. Object movement can be tracked over time.) , wherein the tracklet comprises a respective bounding box for each sensor-data frame of the plurality of sensor-data frames and an identifier (Paragraphs [0014], [0020], [0017], [0029], [0031], Wuu teaches the machine-learned model determines object data associated with the objects. The object data may include one or more bounding boxes associated with the respective objects in the environment. A respective bounding box may be indicative of information associated with a specific object at the first-time instance (t 0 ) in which the sensor data was generated/captured. Multiple frames can be input to the machine-learned model as a collection such that an objects movement can be tracked over time. The machine-learned model outputs object data e.g., bounding boxes. Additionally, the object data may indicate a classification and/or type of the object. Under Broadest Reasonable Interpretation (BRI), the Examiner interprets a classification/type to be an “identifier.”) ; Wuu does not explicitly disclose combin e (ing) the bounding box and a bounding box of the tracklet to generate an output bounding box. Zhang is in the same field of art of tracking objects using bounding boxes and tracklets. Further, Zhang teaches combin e (ing) the bounding box and a bounding box of the tracklet to generate an output bounding box (1 Introduction, 3 BYTE, Zhang teaches matching/associating the high score detection boxes to the tracklets based on motion similarity or appearance similarity. Similarity is computed either by IoU or the Re-ID feature distances between the detection boxes and the predicted box of tracks. The output of each individual frame is the bounding boxes and the identities of the tracks in the current frame.) . Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wuu by matching the bounding boxes with high similarity scores to the tracklets based on similarity that is taught by Zhang, to make the invention that utilizes the similarity with tracklets to strengthen the reliability of detection boxes; thus, one of ordinary skilled in the art would be motivated to combine the references since associating detection boxes with information of the previous frames are leveraged to enhance the video detection performance (Zhang, 2.1 Object Detection in MOT). In addition, similarity with tracklets provides a strong cue to distinguish the objects and the background in low score detection boxes (due to occlusion/motion blur/size changing) (Zhang, 1 Introduction; 3 BYTE). Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention. In regards to Claim 2, Wuu in view of Zhang discloses the apparatus of claim 1, wherein, to combine the bounding box and the bounding box of the tracklet, the at least one processor (Paragraph [0051], Wuu teaches a processor for performing the recited operations.) is configured to process the bounding box and the tracklet using a neural network to generate the output bounding box (Paragraphs [0058-63], [0020], [0045], [0051], Fig. 5, Wuu teaches a machine-learned model (112) which determines object data associated with the object which may include bounding boxes associated with objects in the environment. Next, the machine-learned model determines an estimated location of the object in the environment based on stored tracking data associated with the object. Then, the machine-learned model may associate, as tracked object data, the object data with the tracking data based on the location and the estimated location. Associating of the object data with the stored tracking data may be based on a proximity between the location and estimated location. Finally, the machine-learned model may update the stored tracking data based on the tracked object data and output the tracked object data. Tracked object data received as output from the machine-learned model may indicate, for the object in the environment, object data associated with that object (e.g., a bounding box , classification, confidence, velocity) and a trajectory.) . In regards to Claim 3, Wuu in view of Zhang discloses the apparatus of claim 1, wherein, the at least one processor (Paragraph [0051], Wuu teaches a processor for performing the recited operations.) is configured to combine the bounding box and the bounding box of the tracklet according to a non-max suppression technique (Paragraphs [0045], [0049], Wuu teaches a non-maximal suppression (NMS) component which uses the current track(s) and the score(s), position offset(s), and other data to determine refined current track(s). The refined current track(s) can be used to determine the trajectory level representation(s) and/or additional current track(s) at a future time.) . In regards to Claim 4, Wuu in view of Zhang discloses the apparatus of claim 1, wherein, the at least one processor (Paragraph [0051], Wuu teaches a processor for performing the recited operations.) is configured to combine the bounding box and the bounding box of the tracklet according to an intersection-over-union approach (1 Introduction, 4.1 Setting, Zhang teaches matching the high score detection boxes to the tracklets based on motion similarity or appearance similarity. The similarity can be computed by the intersection-over-union (IoU) of the predicted box and the detection box. For example, if the IoU between the detection box and the tracklet is smaller than 0.2, the matching will be rejected. The Examiner interprets “matching” to be synonymous to “combining” since the claim is silent to the definition of “combining.”) . In regards to Claim 6, Wuu in view of Zhang teaches the apparatus of claim 1, wherein the at least one processor (Paragraph [0051], Wuu teaches a processor for performing the recited operations.) implements a two-stage method to generate the output bounding box (Paragraphs [0025-26], [0020], Fig. 3, Wuu teaches the machine-learned model may include a first portion (e.g., object detection portion) that determines object data associated with the object. The object data may include at least a predicted bounding box associated with the object. The machine-learned model may include a second portion (e.g., object tracking portion) that determines, based at least in part on the object data determined by the first portion, tracking data associated with the movement of the object through the environment. The output may include predicted tracked object data that includes the predicted bounding box and the predicted trajectory.) . PNG media_image3.png 848 1269 media_image3.png Greyscale In regards to Claim 16, Wuu in view of Zhang teaches the apparatus of claim 1, wherein the features are generated by a feature-extractor machine-learning model (Paragraphs [0016], [0034], Wuu teaches the machine-learned model may determine features that may be indicated within the object data. The Examiner interprets any ML model, CNN, etc. to be a feature extractor machine learning model if it is able to extract features from image/sensor data since the claim is silent to the specific type of machine learning model used to extract features.) . In regards to Claim 17, Wuu in view of Zhang teaches the apparatus of claim 1, wherein the objects are detected and the bounding box is generated by an object-detector machine-learning model (Paragraph [0031], Fig. 1, Wuu teaches the detection component (114) of the machine-learned model (112) may output object data (116). The object data (116) may include bounding boxes associated with objects (108) detected in the environment (104) by the detection component, such as bounding box 120(1) associated with the object 108(1) and the bounding box 120(2) associated with the object 108(2).) . PNG media_image4.png 787 522 media_image4.png Greyscale In regards to Claim 18, Wuu in view of Zhang discloses the apparatus of claim 1, wherein the at least one processor (Paragraph [0051], Wuu teaches a processor for performing the recited operations.) is configured to generate the identifier for the object (Paragraph [0020], Wuu teaches receiving from the machine-learned model, an output including the tracked object data. The tracked object data received as output from the machine-learned model may indicate, for each object in the environment, object data associated with that object (e.g., a bounding box, classification, etc.) and a trajectory traversed by that object through the environment. Under BRI, the Examiner interprets a “classification” of the object to be an “identifier” since the claim is silent to the specific definition of “identifier.”) . In regards to Claim 19, Wuu in view of Zhang teaches the apparatus of claim 1, wherein the bounding box is tracked using a Kalman filter or a Bayesian-filtering approach (1 Introduction, Fig. 1, Zhang teaches using a Kalman filter to predict the location of the tracklets in the new frame. The Examiner interprets “or” to mean only one of the two listed approaches/methods is required to meet the claim limitation.) . PNG media_image5.png 711 604 media_image5.png Greyscale In regards to Claim 20, Wuu teaches a method for tracking objects (Paragraph [0058], Fig. 5, Wuu teaches a method associated with a machine-learned model determining tracked object data for use by a vehicle.) , the method comprising: generating features based on a sensor-data frame (Paragraphs [0016], [0023], Wuu teaches the machine-learned model may determine an estimated location at the time t 0 , an estimated orientation of the object, an estimated size of the object, an estimated classification of the object, or the like, as well as any other feature that may be indicated with the object data. The machine-learned model determines the location/features of the object at t 0 based at least in part on stored tracking data associated with the object. The stored tracking data may include multiple frames of sensor data, where each individual frame is associated with a different instant of time.) ; detecting an object based on the features (Paragraph [0014], Wuu teaches generating or determining object data associated with one or more objects detected in the environment. The object data may indicate a classification and/or type of object, such as whether the object is a dynamic object that is capable of movement (e.g., a vehicle, motorcycle, bicycle, pedestrian, animal, etc.) and/or a static object (e.g., building, road surface, tree, sign, barrier, curb, parked vehicle, etc.). Additionally, the object data may indicate other information associated with the object, such as velocity of the object at a time point, a confidence associated with the object, etc.) ; generating a bounding box based on the object (Paragraph [0014], Wuu teaches generating or determining object data associated with one or more objects detected in the environment. The object data may include one or more bounding boxes associated with the respective objects in the environment.) ; tracking the bounding box over a plurality of sensor-data frames to generate a tracklet (Paragraphs [0020], [0029], [0042], Wuu teaches an output of tracked object data from the machine-learned model which may indicate, for each object in the environment, object data associated with that object (e.g., bounding box, classification) and a trajectory traversed by that object through the environment. Object movement can be tracked over time.) , wherein the tracklet comprises a respective bounding box for each sensor-data frame of the plurality of sensor-data frames and an identifier (Paragraphs [0014], [0020], [0017], [0029], [0031], Wuu teaches the machine-learned model determining object data associated with the objects. The object data may include one or more bounding boxes associated with the respective objects in the environment. A respective bounding box may be indicative of information associated with a specific object at the first-time instance (t 0 ) in which the sensor data was generated/captured. Multiple frames can be input to the machine-learned model as a collection such that an objects movement can be tracked over time. The machine-learned model outputs object data e.g., bounding boxes. Additionally, the object data may indicate a classification and/or type of the object. Under Broadest Reasonable Interpretation (BRI), the Examiner interprets a classification/type to be an “identifier.”) . Wuu does not explicitly disclose combining the bounding box and a bounding box of the tracklet to generate an output bounding box. Zhang is in the same field of art of estimating bounding boxes and identities of objects in videos for object detection. Further, Zhang teaches combining the bounding box and a bounding box of the tracklet to generate an output bounding box (1 Introduction, 3 BYTE, Zhang teaches matching the high score detection boxes to the tracklets based on motion similarity or appearance similarity. Similarity is computed either by IoU or the Re-ID feature distances between the detection boxes and the predicted box of tracks. The output of each individual frame is the bounding boxes and the identities of the tracks in the current frame.) . Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wuu by matching/associating the bounding boxes with high scores to the tracklets based on similarity that is taught by Zhang, to make the invention that utilizes the similarity with tracklets to strengthen the reliability of detection boxes; thus, one of ordinary skilled in the art would be motivated to combine the references since associating detection boxes with information of the previous frames are leveraged to enhance the video detection performance (Zhang, 2.1 Object Detection in MOT). In addition, similarity with tracklets provides a strong cue to distinguish the objects and the background in low score detection boxes (due to occlusion/motion blur/size changing) (Zhang, 1 Introduction; 3 BYTE). Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention . 07-21 AIA Claim 5 is rejected under 35 U.S.C. 103(a) as being unpatentable over Wuu et al. (U.S. Patent Pub. No. 2023/0177804 A1, hereafter referred to as Wuu) in view of Zhang et al. (NPL “ByteTrack: Multi-object Tracking by Associating Every Detection Box,” 2022, hereafter referred to as Zhang) in further view of Gao et al. (U.S. Patent Pub No. 20180047193 A1, hereafter referred to as Gao) . Regarding Claim 5, Wuu in view of Zhang teaches the apparatus of claim 1. Wuu in view of Zhang does not explicitly disclose wherein, the at least one processor is configured to combine the bounding box and the bounding box of the tracklet according to a total-area approach. Gao is in the same field of art of performing methods for more accurate object tracking of objects in a scene. Further, Gao teaches wherein, the at least one processor (Paragraph [0009], Gao teaches the processor is configured to and can determine a candidate merged bounding box for a first bounding box.) is configured to combine the bounding box and the bounding box of the tracklet according to a total-area approach (Paragraphs [0106], [0006], Fig. 5, Gao teaches evaluating an overlap ratio to determine whether two bounding boxes should be merged/combined. The overlap ratio can be defined as the total area occupied by both bounding boxes, less the intersecting region, versus the area occupied by the merged bounding boxes.) . PNG media_image6.png 392 370 media_image6.png Greyscale Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wuu in view of Zhang by determining whether or not to combine/match the bounding boxes to the tracklets using a total area approach that is taught by Gao, to make the invention that can more accurately combines bounding boxes and their associated tracklets; thus, one of ordinary skilled in the art would be motivated to combine the references since doing so may lead to more accurate tracking of moving objects in a scene (Gao, Paragraph [0065]). Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention . 07-21 AIA Claim s 7-10 are rejected under 35 U.S.C. 103(a) as being unpatentable over Wuu et al. (U.S. Patent Pub. No. 2023/0177804 A1, hereafter referred to as Wuu) in view of Zhang et al. (NPL “ByteTrack: Multi-object Tracking by Associating Every Detection Box,” 2022, hereafter referred to as Zhang) in further view of Yu et al. (U.S. Patent Pub No. 2023/0326215 A1, hereafter referred to as Yu) . Regarding Claim 7, Wuu in view of Zhang teaches the apparatus of claim 1, Wuu in view of Zhang does not explicitly disclose wherein the at least one processor is configured to, generate, at a transformer machine-learning model, a track using the bounding box as proposal query and the tracklet as track query. Yu is in the same field of art of performing identification and tracking of objects using a multi-stage model where the second stage is a transformer model. Further, Yu teaches wherein the at least one processor is configured to (Paragraphs [0081-82], [0070], Yu teaches a computer device for enabling fast and accurate end-to-end identification and tracking of objects which includes a processing device to perform the method(s).) , generate, at a transformer machine-learning model, a track (Paragraphs [0076-77], [0061], Figs. 2 & 5, Yu teaches the neural network can include one or more classification heads (350-356) that output for one or more objects of the environment, at least one of a type of object, a size of the object, a pose of the object, a velocity of the object, or an acceleration (tracks) of the object. The neural network, set prediction model, (SPM) can have a transformer architecture.) PNG media_image7.png 552 748 media_image7.png Greyscale PNG media_image8.png 536 724 media_image8.png Greyscale using the bounding box as proposal query and the tracklet as track query (Paragraphs [0044-45], Fig. 2, Yu teaches the detector model (210) detects the presence of objects in images and outputs detected objects (212) using any suitable indications (e.g., 2D or 3D bounding boxes) for the depictions of the objects (212) in the input images. Set prediction model (220) processes the detected objects (212) for the plurality of N frames. The SPM also receives seed tracks (222), which can initially be assigned random values or some fixed values. In the course of processing the by SPM (220), m of the M seed tracks, which correspond to the actual tracks, can acquire a status bit SB=1 value and the corresponding tracking information, e.g., bounding boxes, poses, types, velocities, etc.) . Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wuu in view of Zhang by using a transformer model stage to generate tracks for each object by taking as input, detected objects (bounding boxes) and seed tracks that is taught by Yu, to make the invention that enables fast and accurate identification and tracking of objects in a variety of environments; thus, one of ordinary skilled in the art would be motivated to combine the references since transformer models, unlike existing models, rely on performance of various surrogate tasks and further avoids introducing model parameters that cannot be easily determined during training and instead need to be tuned heuristically as part of a complex decision-making process (Yu, Paragraph [0019]). In addition, use of a transformer model (SPM) enables the output detector model to be lightweight and output as little information as the location and size of the bounding boxes for each image frame since the actual correlations of different bounding boxes can be performed by the transformer (SPM) (Yu, Paragraph [0060]). Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention. In regards to Claim 8, Wuu in view of Zhang in further view of Yu discloses the apparatus of claim 7, wherein the at least one processor is configured to (Paragraphs [0081-82], Yu teaches a computer device for enabling fast and accurate end-to-end identification and tracking of objects which includes a processing device to perform the method(s).) : combine a prior track with the tracklet to generate a combined track (Paragraphs [0059], Fig. 4, Yu teaches three frames (402-1, 402-2, and 402-3) having three different time stamps, e.g, t 1 , t 2 , t 3 , as depicted in Fig. 4. Frames 402-n can be obtained through the same sensing modality. Frame 1 (402-1) includes four detected objects, enclosed by a respective bounding box: a pedestrian (403-1), a tree (405-1), a car (407-1) and a truck (409-1). The objects detected in frame 1 (402-1) can be represented by a first set of feature vectors output by detector model (310). For the objects detected in frame 1 (402), the feature tensors of the first set include feature tensors (413-1 … 419-1). Similarly, frames 2 and 3 represent the four detected objects located at the same location or different locations. The feature tensors of the different frames can be joined into a single combined feature tensor (312).) ; PNG media_image9.png 530 757 media_image9.png Greyscale and provide the combined track to the transformer machine-learning model as a track query (Paragraphs [0060-61], Yu teaches providing the single combined feature tensor (312) as input to the SPM (320). SPM can have a transformer architecture. SPM identifies the associations of different bounding boxes/objects in the course of solving the set prediction problem of selecting similar objects from the set of boxes/objects identified by the detector model (310) within all frames 402-n.) . In regards to Claim 9, Wuu in view of Zhang in further view of Yu discloses the apparatus of claim 7, wherein the transformer machine-learning model is trained according to a gradient-boosting technique using losses from training a tracker machine-learning model (Paragraphs [0080], [0066], Yu teaches the processing device can adjust parameters (e.g., weights, biases, etc.) of the neural network in view of the cost (loss) value, e.g., using various techniques of backpropagation that cause cost value of the final candidate sets of tracks to decrease. The elements of the weights matrices of transformer layers can be determined during training of the set prediction model (SPM).) . In regards to Claim 10, Wuu in view of Zhang in further view of Yu discloses the apparatus of claim 9, wherein the tracker machine-learning model tracks the bounding box to generate the tracklet (Paragraph [0032], Wuu teaches a machine-learned model with a tracking component (122). The tracking component (122) may determine, and the machine-learned model may output, tracked object data (124). The tracked-object data may include movement data indicative of object movement. In addition to including the object data (116) (e.g., the bounding boxes 120(1) and 120(2)), the tracked object data may also include trajectories traversed by the objects in the environment (104), such as trajectory 126(1) traversed by object 108(1) and the trajectory 126(2) traversed by the object 108(2).) . 07-21 AIA Claim 11 is rejected under 35 U.S.C. 103(a) as being unpatentable over Wuu et al. (U.S. Patent Pub. No. 2023/0177804 A1, hereafter referred to as Wuu) in view of Zhang et al. (NPL “ByteTrack: Multi-object Tracking by Associating Every Detection Box,” 2022, hereafter referred to as Zhang) in further view of Yu et al. (U.S. Patent Pub No. 2023/0326215 A1, hereafter referred to as Yu) in further view of Pronovost (U.S. Patent No. 12,639,846, hereafter referred to as Pronovost) . Regarding Claim 11, Wuu in view of Zhang in further view of Yu teaches the apparatus of claim 7, wherein: a tracker machine-learning model tracks the bounding box to generate the tracklet (Paragraph [0032], Wuu teaches a machine-learned model with a tracking component (122). The tracking component (122) may determine, and the machine-learned model may output, tracked object data (124). The tracked-object data may include movement data indicative of object movement. In addition to including the object data (116) (e.g., the bounding boxes 120(1) and 120(2)), the tracked object data may also include trajectories traversed by the objects in the environment (104), such as trajectory 126(1) traversed by object 108(1) and the trajectory 126(2) traversed by the object 108(2).) ; training-data samples that result in losses above a loss threshold are identified as the tracker machine-learning model is trained (Paragraph [0026], Wuu teaches the ground truth data and the predicted tracked object data may be compared to determine whether differences exist between the ground truth data and predicted object tracked data. If a difference is determined to meet or exceed a threshold difference, a parameter of the machine-learned model may be altered to minimize the difference. A parameter of one or more portions of the machine-learned model may be altered.) . Wuu in view of Zhang in further view of Yu does not explicitly disclose gradient-descent weights of the training-data samples are increased as the transformer machine-learning model is trained. Pronovost is in the same field of art of performing object detection and using a transformer to generate a predicted trajectory/track. Further, Pronovost teaches gradient-descent weights of the training-data samples are increased as the transformer machine-learning model is trained (Col. 17, lines 39-64, Pronovost teaches altering parameters of the transformer-based machine learning model using gradient descent to reduce the loss such that, if the transformer-based machine-learned model repeated the process on the same input data, the resultant loss would be less than it was on the last run.) . Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wuu in view of Zhang in further view of Yu by altering the parameters/weights of the transformer machine-learned model using a gradient descent technique that is taught by Pronovost, to make the invention that reduces the loss between an output of the transformer-based machine-learned model and the ground truth data to improve the accuracy of the output; thus, one of ordinary skilled in the art would be motivated to combine the references since by altering the parameters using gradient descent technique, the loss can be reduced so that the loss in the next run is smaller than it was on the previous run (Pronovost, Col. 17, lines 47-52) . Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention . 07-21 AIA Claim 12 is rejected under 35 U.S.C. 103(a) as being unpatentable over Wuu et al. (U.S. Patent Pub. No. 2023/0177804 A1, hereafter referred to as Wuu) in view of Zhang et al. (NPL “ByteTrack: Multi-object Tracking by Associating Every Detection Box,” 2022, hereafter referred to as Zhang) in further view of Yu et al. (U.S. Patent Pub No. 2023/0326215 A1, hereafter referred to as Yu) in further view of Papi et al. (U.S. Patent No. 12,416,730, hereafter referred to as Papi) . Regarding Claim 12, Wuu in view of Zhang in further view of Yu teaches the apparatus of claim 7, wherein the at least one processor is configured to (Paragraph [0051], Wuu teaches a processor.) : determine a similarity score based on a comparison between the track and the tracklet (1 Introduction, 2.2 Data Association, Zhang teaches matching the high score detection boxes to the tracklets based on motion similarity or appearance similarity. Similarity between tracklets and detection boxes is computed. The Examiner interprets computing similarity to be synonymous to similarity score.) . Wuu in view of Zhang in further view of Yu does not explicitly disclose determin e (ing) whether to bypass the transformer machine-learning model based on the similarity score. Papi is in the same field of art of performing object detection using a multi-stage model. Further, Papi teaches determin e (ing) whether to bypass the transformer machine-learning model based on the similarity score (Col. 14, lines 17-67, Col. 15, lines 63-67, Papi teaches the results determined by the first pass object detection association component may include a number of subsets that were successfully associated by the first model and/or additional object detections (residual object detections) that were not successfully associated by the first model. The successfully associated object detections may be provided to the first pass track association component to associate the object detections with the tracks of previously detected and tracked objects from the object track data. The inputs to the second pass may exclude the object detections and/or tracks that were associated during the first pass. The Examiner interprets determining “associations” between one or more object detections and one or more previous tracks to be synonymous to “similarity.”) Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wuu in view of Zhang in further view of Yu by providing the residual (non-associated) object detections to the second pass associations component which includes a transformer model that is taught by Papi, to make the invention that determines associations between any of the residual object detections and any residual tracks for which associations could not be determined in the first pass; thus, one of ordinary skilled in the art would be motivated to combine the references since a multi-pass technique for determining associations (similarities) may improve the accuracy and performance of the overall outputs and help to identify the more difficult and less obvious associations among the residual object detections that were not identified during the first pass (Papi, Col. 15, lines 55-67; Col. 16, lines 1-9). Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention . 07-21 AIA Claim 13 is rejected under 35 U.S.C. 103(a) as being unpatentable over Wuu et al. (U.S. Patent Pub. No. 2023/0177804 A1, hereafter referred to as Wuu) in view of Zhang et al. (NPL “ByteTrack: Multi-object Tracking by Associating Every Detection Box,” 2022, hereafter referred to as Zhang) in further view of Yu (U.S. Patent Pub. No. 2023/0326215 A1, hereafter referred to as Yu) in further view of Ye (U.S. Patent Pub. No. 2024/0152734 A1, hereafter referred to as Ye) . Regarding Claim 13, Wuu in view of Zhang in further view of Yu teaches the apparatus of claim 7, wherein the at least one processor is configured to (Paragraph [0051], Wuu teaches a processor.) : determine a similarity score based on a comparison between a prior track and prior tracklet (1 Introduction, 2.2 Data Association, Zhang teaches matching the high score detection boxes to the tracklets based on motion similarity or appearance similarity. Similarity between tracklets and detection boxes is computed. The Examiner interprets a computed similarity score based on IoU, etc. to be a “similarity score.”) . Wuu in view of Zhang in further view of Yu does not explicitly disclose based on the similarity score exceeding a dissimilarity threshold, generate the track at the transformer machine-learning model. Ye is in the same field of art of performing object detection using a machine learning model using a transformer architecture. Further, Ye teaches based on the similarity score exceeding a dissimilarity threshold (Paragraphs [0047], [0051], [0100], Ye teaches determining whether a token is forwarded (or halted) to the subsequent layer depending on a threshold token score. The token score is based on a position of a respective token relative to a foreground object wherein the token score increases when the position of the respective token is closer to the center of the foreground object. Non-halted (Non-excluded/stopped) tokens are provided as input to a subsequent layer during inference of the machine learning model. The Examiner interprets the token score as a “similarity score” since it is a measure of how close the position of the token is to the center of the object, (i.e., how similar their positions are). Additionally, the Examiner interprets “exceeding a dissimilarity threshold” to mean exceeding a minimum threshold of similarity.) , generate the track at the transformer machine-learning model (Abstract, Paragraph [0022], Ye teaches detecting, by the machine learning model, having a transformer architecture, at least one detected object based on the plurality of non-halted tokens. A tracked path of the object can also be output.) . Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wuu in view of Zhang in further view of Yu by passing only tokens that exceed a threshold score to the subsequent layer of the transformer that is taught by Ye, to make the invention that halts object detections that do not meet a threshold similarity score; thus, one of ordinary skilled in the art would be motivated to combine the references since implementing a halting module/threshold condition can reduce superfluous tokens/detections in order to reduce the computational complexity of the transformer’s attention mechanism (Ye, Paragraph [0015]) . Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention . 07-21 AIA Claim s 14 and 15 are rejected under 35 U.S.C. 103(a) as being unpatentable over Wuu et al. (U.S. Patent Pub. No. 2023/0177804 A1, hereafter referred to as Wuu) in view of Zhang et al. (NPL “ByteTrack: Multi-object Tracking by Associating Every Detection Box,” 2022, hereafter referred to as Zhang) in further view of Das et al. (U.S. Patent No. 12,313,727, hereafter referred to as Das) . Regarding Claim 14, Wuu in view of Zhang teaches the apparatus of claim 1, wherein the sensor-data frame comprises an image frame (Paragraph [0023], Wuu teaches the sensor data may be a time-ordered collection of image frames representing sensor data associated with the environment, such that the first frame represents the environment at a first time, a second frame represents the environment at a second time, and so forth.) , wherein the at least one processor is configured to generate sensor features based on sensor data (Paragraphs [0029-30], Wuu teaches the sensor component(s) may capture sensor data associated with the environment surrounding the vehicle. Sensor components may include lidar sensors, radar sensors, location sensors, inertial sensors, cameras, environment sensors, etc. The sensor component(s) may generate sensor data associated with the environment, which may include lidar data, radar data, location data, inertial data, image data, environment sensor data. The sensor data may be sent to one or more computing devices associated with the vehicle. The machine-learned model may be executed using resources (e.g., compute, memory, processing cores, etc.) of the computing devices.) . Wuu in view of Zhang does not explicitly disclose fus e (ing) the sensor features with the features to generate fused features; wherein the object is detected based on the fused features; and wherein the bounding box is generated based on the fused features. Das is in the same field of art of using transformer-based machine learning models for object detection. Further, Das teaches and fus e (ing) the sensor features with the features to generate fused features (Col. 7, lines 24-49, Das teaches the fused feature map may represent image features combined with lidar features or image features combined with radar features.) ; wherein the object is detected based on the fused features (Col. 5, lines 8-24, Das teaches the output layer may produce output inferences indicating where an object is located within the fused features and what class/category the object belongs to. Output inferences may indicate that a vehicle has been detected and a bounding box around the borders of a vehicle.) ; and wherein the bounding box is generated based on the fused features (Col. 9, lines 1-18, Das teaches the 3D bounding boxes can be determined when the fused features are in top-down view.) . Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Wuu in view of Zhang by fusing the data captured by various sensors and detecting the object based on fused sensor information that is taught by Das, to make the invention that detects objects using combined sensor data from different sensors; thus, one of ordinary skilled in the art would be motivated to combine the references since combined processing of data from different sensor modalities may result in more accurate detection of objects, e.g., lower false positive rate, compared to processing the data independently (Das, Col. 1, lines 23-34). Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention. In regards to Claim 15, Wuu in view of Zhang in further view of Das discloses the apparatus of claim 14, wherein the sensor data comprises at least one of: a radio detection and ranging (RADAR) frame; or a light detection and ranging (LIDAR) frame (Paragraph [0029], Wuu teaches collecting sensor data associated with the environment which may include lidar data, radar data. Sensor data may be in the form of multiple snapshots (e.g., frames) that can be input into the machine-learned model.) . Pertinent Prior Art 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Huang et al. (U.S. Patent No. 11,188,740) teaches comparing the value a threshold confidence score to determine whether to process the image using an additional pass of the cascaded CNN. Nguyen et al. (U.S. Patent Pub. No. 2024/0144489 A1) teaches a method for multi-object tracking using a Transformer architecture. Yang et al. (NPL “Adaptively Bypassing Vision Transformer Blocks for Efficient Visual Tracking,” July 2024) teaches an adaptive computation framework that adaptively bypasses transformer blocks for efficient visual tracking . Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to SYDNEY L BLACKSTEN whose telephone number is (571)272-7120. The examiner can normally be reached 8:30am-4:30pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached at 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /SYDNEY L BLACKSTEN/Examiner, Art Unit 2674 /ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674 Application/Control Number: 18/807,806 Page 2 Art Unit: 2674 Application/Control Number: 18/807,806 Page 3 Art Unit: 2674 Application/Control Number: 18/807,806 Page 4 Art Unit: 2674 Application/Control Number: 18/807,806 Page 5 Art Unit: 2674 Application/Control Number: 18/807,806 Page 6 Art Unit: 2674 Application/Control Number: 18/807,806 Page 7 Art Unit: 2674 Application/Control Number: 18/807,806 Page 8 Art Unit: 2674 Application/Control Number: 18/807,806 Page 9 Art Unit: 2674 Application/Control Number: 18/807,806 Page 11 Art Unit: 2674 Application/Control Number: 18/807,806 Page 12 Art Unit: 2674 Application/Control Number: 18/807,806 Page 13 Art Unit: 2674 Application/Control Number: 18/807,806 Page 14 Art Unit: 2674 Application/Control Number: 18/807,806 Page 15 Art Unit: 2674 Application/Control Number: 18/807,806 Page 16 Art Unit: 2674 Application/Control Number: 18/807,806 Page 17 Art Unit: 2674 Application/Control Number: 18/807,806 Page 18 Art Unit: 2674 Application/Control Number: 18/807,806 Page 19 Art Unit: 2674 Application/Control Number: 18/807,806 Page 20 Art Unit: 2674 Application/Control Number: 18/807,806 Page 21 Art Unit: 2674 Application/Control Number: 18/807,806 Page 22 Art Unit: 2674 Application/Control Number: 18/807,806 Page 23 Art Unit: 2674 Application/Control Number: 18/807,806 Page 24 Art Unit: 2674 Application/Control Number: 18/807,806 Page 25 Art Unit: 2674 Application/Control Number: 18/807,806 Page 26 Art Unit: 2674 Application/Control Number: 18/807,806 Page 27 Art Unit: 2674
Prosecution Insights