DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
The present application claims benefit of provisional application 63/595,948 filed on 11/03/2023 and provisional application 63/548,537 filed on 11/14/2023.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3-4, 10, 12-13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Nguyen et al. (US 20240144489 A1) in view of Zhang et al. (“MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries”; copy provided by Examiner).
Regarding claims 1, 10, and 19
Nguyen teaches: [claim 1: A computer-implemented method for tracking multiple objects with multiple cameras using a transformer-based model (MCTR)],
[claim 10: A system for tracking multiple objects with multiple cameras using a transformer-based model (MCTR) comprising: a
one or more processor devices ]
[claim 19: A ]
(Nguyen discloses a transformer-based multi-object tracking method/system implemented on computing hardware executing executable programs (¶ [0001] “method applies Deep Learning techniques to detect and track multiple objects simultaneously in a video or camera stream”; [0015] “FIG. 1 is a schematic diagram that describes the general logical components and operation of the proposed multi-object tracking system on streaming video”; See FIG. 1, including multiple cameras (e.g. Camera 1, Camera 2, Camera N-1, and Camera N), and hardware components in ¶ [0023], found below).
PNG
media_image1.png
225
421
media_image1.png
Greyscale
Nguyen further teaches the system includes Joint Detection Association Transformer, Transformer Decoder, Feature relation Transformer, and a Differentiable Matching layer for performing transformer-based multiple object tracking (¶ [0010]; ¶ [0037]).);
processing track embeddings and detection embeddings of video feeds obtained from (Nguyen teaches “multi-object tracking from video footage” (¶ [0009]) and extracting temporal features from consecutive video frames (¶¶ [0028]-[0033]). Nguyen further teaches “The object tracking module performs object detection and association based on the input feature of the frame of interest, combined with the current tracking state” (¶ [0035]). Nguyen additionally teaches “the track queries set consists of
N
t
r
a
c
k
vectors, each representing a track” and a Feature Relation Transformer that “takes two sets of embedding vectors as input: (1) the set of hidden states of object queries
S
o
b
s
and (2) the set of track queries's embeddings
S
t
r
a
c
k
s
(¶ [0038]; ¶ [0040]). The hidden states of the object queries (
S
o
b
s
)
reasonably corresponds to the claimed detection embeddings because they are one of the two sets of embedding vectors processed by the Feature Relation Transformer and are used to “predict object's properties including center coordinate, width, height, class (e.g., person, bicycle, motorcycle, etc.), and confidence” (see ¶¶ [0039]-[0040]). Nguyen further teaches that a Temporal Aggregation Module outputs “the updated feature embedding vector
f
e
a
t
u
r
e
t
+
1
” which “acts as track query for the Transformer Decoder… in the next timestep” (¶ [0044]). Accordingly, Nguyen teaches extracting features from video frames, processing embedding vectors corresponding to tracked objects (
S
t
r
a
c
k
s
)
together with embedding vectors corresponding to object queries (
S
o
b
s
), and updating the track embeddings, reasonably corresponding to the claimed processing of track embeddings and detection embeddings of video feeds to generate updated track embeddings. While Nguyen teaches a system with multiple cameras (FIG. 1), Nguyen fails to explicitly disclose performing such processing using video feeds obtained from multiple cameras.);
associating the updated track embeddings with the detection embeddings to generate track-detection associations (TDA) for each (Nguyen teaches “a soft assignment matrix where each row represents a track query” and each element is “a probability score” of association (¶ [0040]). Nguyen further teaches using the soft assignment matrix to associate tracks with detected objects (see ¶ [0041] and ¶ [0043]). The soft assignment matrix reasonably corresponds to the claimed TDA because both represent probabilistic associations between track representations and detected objects. Nguyen does not explicitly disclose generating a separate TDA for each camera view.);
calculating a differentiable loss from the TDA by combining a detection loss, [and] a track loss (Nguyen teaches “(i) object detection loss” and “(ii) soft assignment matrix loss,” wherein soft assignment matrix loss “is Focal loss on the output soft assignment matrix” ¶ [0045]. Nguyen further teaches that the soft assignment matrix is trained against a ground-truth assignment matrix specifying which existing track is linked to which detected object (¶ [0040]). Accordingly, the focal loss measures errors in linking previously tracked objects to the correct current detections and reasonably corresponds to the claimed track loss.);
and training the MCTR using the differentiable loss and contiguous video segments sampled from a training dataset to track multiple objects (Nguyen teaches “the JDAT model is trained based on the tracking results on a sequence of Z consecutive frames…These loss values are accumulated across timesteps in a video…. The entire model is trained with supervised learning signal on video datasets” (¶ [0045).).
Although Nguyen does not expressly describe a memory device operatively coupled to the processor (claim 10) or expressly identify a non-transitory computer-readable storage medium (claim 19), Nguyen does teach executable programs deployed on GPU or TPU equipped computer hardware (¶ [0023]). Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to store those executable programs and associated model data in processor-accessible memory and on a non-transitory computer-readable storage medium operatively coupled to the disclosed processor hardware because such arrangements are conventional implementation of executable computer programs and doing so would predictably permit the processor to retrieve and execute the disclosed tracking instructions.
Nguyen does not explicitly disclose: processing track embeddings and detection embeddings of video feeds obtained from multiple cameras; generating track-detection associations (TDA) for each camera view; including in the differentiable loss an auxiliary track loss that provides additional supervision for previously tracked objects; training the transformer-based model (MCTR) to track multiple objects using multiple cameras.
In a related art, Zhang teaches: processing track embeddings and detection embeddings of video feeds obtained from multiple cameras to generate updated track embeddings (Zhang teaches “modeling 3D dynamic objects in complex scenes across multiple cameras” (Abstract). Specifically, Zhang teaches “3D track query, which directly models the 3D states and appearance features of an object track over time and across cameras. At each frame, a 3D track query sample features from all visible cameras” (p. 4537 left column lines 2-7). Zhang further teaches that its “transformer decoder head takes track queries and attends them with multi-camera image features” after which its extracted query features are used to decode object candidates (p. 4539 left column first sentence of 3.3; also see FIG’s. 1-2, including FIG. 2’s accompanying description “Track queries attend with multi-camera features to decode object candidates in the current frame” and the updated queries proceed to following frames “to track the same objects” (p. 4539).; p. 4537 left column second summary bullet point “3D track query samples feature from all visible cameras and update the track frame-by-frame end-to-end.”). Accordingly, Zhang teaches processing information representing previous tracked objects together with image information obtained from multiple camera views to update the representation of the tacked objects, reasonably corresponding to processing track embeddings and detection embeddings obtained from multiple cameras to generate updated track embeddings.);
associating the updated track embeddings with the detection embeddings to generate track-detection associations (TDA) [across multiple camera views and successive frames] (Zhang teaches an “end-to-end Multi-camera 3D tracking framework…It handles multi-camera 3D detection, and cross-camera, cross-frame objects association end-to-end fashion” (FIG. 1 description). Zhang further teaches “3D track query samples feature from all visible cameras” that updates “the track frame-by-frame end-to-end” (p. 4537 left column second summary bullet point). Accordingly, Zhang teaches associating previously tracked objects with current object detections using information obtained from multiple camera views across successive frames, reasonably corresponding to associating updated tracked-object representations with current object detections across multiple camera views and successive frames.);
calculating a differentiable loss (Zhang teaches that its “query-based object tracking loss assigns different regression targets for two different types of queries, newborn queries, and old queries” (p. 4538 right column second full paragraph). Zhang further teaches that “Newborn quires are responsible for detecting newly appeared objects in the current frame,” whereas “Old queries are active queries from previous frames which successfully detected or tracked objects,” and old queries “are responsible for tracking previously appeared objects in the current frame,” and applies focal classification loss and
L
1
regression loss to the query outputs (p. 4538 right column 6 lines from the bottom through p. 4539 left column line 4 and Equations (1) and (2).). Accordingly, Zhang applies separate differential loss supervision to previously established tracks, in addition to the supervision used for newly detected objects, reasonably corresponding to the claimed auxiliary track loss.);
training the MCTR using the differentiable loss and (Zhang teaches that its training loss is expressed as
PNG
media_image2.png
83
256
media_image2.png
Greyscale
where each paired box loss includes focal classification loss and
L
1
(see p. 4538-4539 Equations (1) and (2) and their accompanying descriptions). Zhang further teaches “We use nuScenes [6] dataset for all of our experiments. It consists of 1000 real-world sequences, 700 sequences for training… Each frame includes images from six cameras with a full 360-degree field of view” (p. 4540 right column section 4.1) and states “[we] train our tracker with three frames video clips for 72 epochs” (p. 4541 left column section 4.3, subsection “Training details”). Accordingly, Zhang teaches training the transformer-based tracking model using the disclosed differentiable training loss on multi-frame video clips from a multi-camera training dataset model, reasonably corresponding to the claimed training limitation.).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the tracking model taught by Nguyen to update its tracked object representations using the multi-camera image features taught by Zhang, thereby enabling the tracked object representations to be updated using image information obtained from multiple camera views and predictably improving tracking across camera views.
It further would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Zhang’s additional loss supervision for previously established tracks (corresponding to claimed auxiliary track loss) into Nguyen’s differentiable training loss in order to improve temporal tracking consistency.
Additionally, although Zhang teaches performing object association in a multi-camera environment, neither Zhang nor Nguyen explicitly disclose generating a separate track-detection association for each camera view. However, after modifying Nguyen’s tracking model to update its tracked-object representations using the multiple camera image information taught by Zhang (and discussed above), it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to perform Nguyen’s disclosed track-to-detection association process for the detections corresponding to each respective camera view because Nguyen teaches generating object detections and their corresponding assignment matrix for a current frame (Nguyen ¶ [0041]). Applying the same known association process to each camera view would have predictably generated a respective track-detection association for each camera view and frame, thereby enabling detections observed from each viewpoint to be linked to the updated tracked-object representations and improving the robustness and accuracy of tracking objects in a multi-camera scene, particularly when an object is more clearly visible or detectable in one view than another.
Nguyen and Zhang are analogous art because both are directed to end-to-end multi-object tracking using object track representations, differentiable training, and object associations across successive frames.
Regarding claims 3 and 12,
Nguyen and Zhang teach the computer-implemented method of claim 1 and the system of claim 10.
Nguyen further teaches: wherein processing the track embeddings further comprises aggregating observations from a frame into a global track embeddings through self-attention and feed-forward layers (Nguyen teaches “One problem to be solved is the need to update the information/state of the tracks over multiple timesteps… To solve this, we propose a Temporal Aggregation Module (TAM), with the aim of updating each track's feature over time as soon as new observations are available…TAM is a simple module consisting of a self-attention layer, combined with feed-forward layers (FFN). The input of this module is a sequence of vectors representing each valid track-object pair… Each track is represented by… a vector
f
e
a
t
u
r
e
t
encode embedding… [and] the hidden vector
h
i
d
d
e
n
t
corresponding to the observation feature obtained from the Feature Relation Transformer model…. These vectors are then passed through the TAM model, the output is …the updated feature embedding vector
f
e
a
t
u
r
e
t
+
1
… The sum of these two vectors…acts as track query for the Transformer Decoder…in the next timestep” (¶ [0044]). Accordingly, Nguyen teaches processing track embeddings by updating an existing track embedding with new observation features using a Temporal Aggregation Module comprising self-attention and feedforward layers to generate an updated track embedding for use in subsequent timestamps, reasonably corresponding to the global track embedding because the updated track embedding maintains information about the tracked object by incorporating new observations over time.).
Regarding claims 4 and 13,
Nguyen and Zhang teach the computer-implemented method of claim 3 and the system of claim 12.
Nguyen further teaches: wherein aggregating the observations further comprises performing cross attention between the track embeddings and the detection embeddings (Nguyen teaches “We propose to use a Feature Relation Transformer model based on self-attention and cross-attention layers. This model takes two sets of embedding vectors as input: (1) the set of hidden states of object queries
S
o
b
s
and (2) the set of track queries's embeddings
S
t
r
a
c
k
s
from the previous timestep…
S
o
b
s
and
S
t
r
a
c
k
s
are passed through the Feature Relation Transformer model, which includes self-attention layers to model interactions within each sets, and cross-attention layers to model interactions across the two sets.” (¶ [0040]). Refer back to Nguyen’s teachings found in claim 1, 10 and 19’s 103 rejection for further explanation of Nguyen’s teachings of
S
t
r
a
c
k
s
and
S
o
b
s
corresponding to track embeddings and detection embeddings respectively. Accordingly, Nguyen teaches passing track embeddings (
S
t
r
a
c
k
s
) and detection embeddings (
S
o
b
s
)
through the Feature Relation Transformer model, including cross-attention layers that model interactions across two sets to generate updated track embeddings.).
Claims 2, 11, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Nguyen et al. (US 20240144489 A1) in view of Zhang et al. (“MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries”; copy provided by Examiner), and in further view of Niculescu-Mizil et al. (US 20180374569 A1).
Regarding claims 2, 11, and 20,
Nguyen and Zhang teach the computer-implemented method of claim 1, the system of claim 10, and the non-transitory computer program product of claim 19, including a trained MCTR.
Nguyen and Zhang fail to explicitly disclose: detecting anomalies from monitored entities within a specified location
In a related art, Niculescu-Mizil teaches: detecting anomalies from monitored entities within a specified location (Niculescu-Mizil teaches using at least one or more cameras (including multiple cameras) to obtain images for detecting anomalies from monitored entities within a specified location and providing the detected anomalies to assist a decision-making process (e.g. discarding items having anomalies, halting production when an anomaly is detected, notifying operator of item having been tagged an anomaly, or otherwise responding to the detected anomaly.) (See excerpt of Niculescu-Mizil, paragraphs [0032]-[0033]) below)
PNG
media_image3.png
619
422
media_image3.png
Greyscale
Niculescu-Mizil does not explicitly disclose performing the anomaly detection using the trained multi-camera tracking transformer (MCTR) recited in the claim.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combined system taught by Nguyen and Zhang to utilize the trained MCTR to detect anomalies from monitored entities and provide the detected anomalies to assist a decision-making process, as taught by Niculescu-Mizil. Applying the known anomaly detection techniques of Niculescu-Mizil to the tracked entity outputs of the trained MCTR of the system taught by Nguyen and Zhang would have predictably improved the usefulness of the tracking system by enabling abnormal conditions associated with monitored entities to be identified and support downstream decision-making in inspection, monitoring, or automated response systems. All three inventions are analogous art because they relate to computer vision systems that analyze images from multiple cameras to monitor objects or entities and produce information to be used in downstream operations.
Claims 5-6 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Nguyen et al. (US 20240144489 A1) in view of Zhang et al. (“MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries”; copy provided by Examiner), and in further view of Sun et al. (“Multiple Traffic Target Tracking with Spatial-Temporal Affinity Network”; copy provided by Examiner).
Regarding claims 5 and 14,
Nguyen and Zhang teach the computer-implemented method of claim 1 and the system of claim 10, including detection embeddings, track embeddings, and processing detection and track embeddings, updating track embeddings, and associating the updated track embeddings.
Nguyen further teaches: that the detection embeddings
(
S
o
b
s
) and track embeddings (
S
t
r
a
c
k
s
)
are input to the Feature Relation Transformer, which outputs “two matrices
H
o
b
s
and
H
t
r
a
c
k
s
” representing transformed detection and track representations, respectively (¶ [0040]; also see Nguyen’s teachings of detection embeddings
(
S
o
b
s
) and track embeddings (
S
t
r
a
c
k
s
)
found in claim 1, 10 and 19’s 103 rejection). These transformed representations reasonably correspond to the claimed detection matrices and track matrices.
Accordingly, Nguyen, as modified by Zhang teaches detection embeddings, track embeddings, and obtaining detection matrices and track matrices, but fail to explicitly disclose: performing linear transformation on the detection embeddings and track embeddings using respective multi-layer perceptrons (MLP) to obtain detection matrices and track matrices.
In a related art, Sun teaches: performing linear transformation (Sun teaches a proposed network wherein “the spatial transformer encoder module is used to encode the detections at the image level. The temporal transformer encoder module is used to extract the discriminative information from the tracklets of the tracked objects at the tracklet level” (p. 4 left column, first paragraph of section 3.1), and “the two-stage transformer encoder framework is used to encode the feature maps of the detections and the tracklets. In the first stage, the spatial-transformer encoder module is used to encode the spatial image information… In the second stage the temporal transformer encoder module is used to encode the history trajectory information…” (p. 4 left column left column first paragraph of section 3.2). Sun further teaches “First, the spatial patches…are embedded through a linear embedding layer…” (p. 4 right column lines 3-4), and an MLP for processing respective detection and track representations (see p. 2 Figure 1, including a MLP for respective detection and tracked branches). Thus, Sun teaches performing a linear transformation of the detection and track representations using respective transformer encoder branches including a linear embedding layer, and MLP processing prior to association.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to apply Sun’s teachings of performing linear transformation using respective multi-layer perceptrons (MLP) to Nguyen and Zhang’s processing of detection embeddings and track embeddings within Nguyen’s Feature Relation Transformer in order to perform a linear transformation of the detection and track embeddings to obtain corresponding detection and track matrices for use in the subsequent association process because Sun teaches that its transformer-based tracking method achieves “run-time performance…above the average of all the listed state-of-the-art trackers” (Sun p. 10 right column section 4.5). Applying Sun’s known transformer processing to Nguyen’s transformed detection and track representations would have predictably generated corresponding detection matrices and track matrices for use in the subsequent association process while incorporating the favorable run-time performance taught by Sun.
Although Sun’s tracking model is directed at tracking traffic targets, Nguyen, Zhang, and Sun are analogous art because each relates to computer vision systems that process images for monitoring or tracking objects in dynamic scenes.
Regarding claims 6 and 15,
Nguyen, Zhang, and Sun teach the computer-implemented method of claim 5 and the system of claim 14, including associating the updated track embeddings.
Nguyen further teaches: applying row-wise softmax operation to a dot-product of the detection matrices and track matrices (Nguyen teaches that the dot product of the detection matrix and track matrix is represented by similarity matrix
M
s
i
m
, and that a dual-softmax performs a softmax operation on the input matrix along the rows before also performing column-wise softmax (¶ [0040] “The dot product of these two matrices represents the feature similarity between each pair of track query and object query, denotes as a matrix
M
s
i
m
=
H
t
r
a
c
k
s
H
o
b
j
s
… For dual-softmax, the algorithm performs a softmax operation on the input matrix along the rows… and along the columns”), thereby teaching the claimed application of a row-wise softmax operation to a dot-product of the detection matrices and track matrices.).
Claims 7, 9, 16, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Nguyen et al. (US 20240144489 A1) in view of Zhang et al. (“MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries”; copy provided by Examiner), and in further view of Carion et al (“End-to-end Object Detection with Transformers”; reference was provided in a presentation-style format by Applicant in IDS; alternative report-style version of reference relied upon and quoted below is provided by Examiner).
Regarding claims 7 and 16,
Nguyen and Zhang teach the computer-implemented method of claim 1 and the system of claim 10, including calculating a differentiable loss.
Nguyen and Zhang fail to explicitly disclose: wherein calculating the differentiable loss further comprises computing the detection loss by finding a bipartite assignment of detections to ground truth.
In a related art, Carion teaches: computing the detection loss by finding a bipartite assignment of detections to ground truth (Carion teaches computing a loss by first finding an optimal bipartite assignment between predicted object detections and ground truth objects, and then computing the object-specific losses for the matched pairs, corresponding to the claimed computing the detection loss by finding a bipartite assignment of detections to ground truth (See Abstract “The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bi partite matching, and a transformer encoder-decoder architecture.”; p. 2 second paragraph “DEtection TRansformer (DETR, see Figure 1) predicts all objects at once, and is trained end-to-end with a set loss function which performs bipartite matching between predicted and ground-truth objects.”; Carion teaches “Our loss produces an optimal bipartite matching between predicted and ground truth objects, and then optimize object-specific (bounding box) losses…. To find a bipartite matching between these two sets” then defines the Hungarian assignment (see equation (1) on p. 5) and further teaches “This optimal assignment is computed efficiently with the Hungarian algorithm” and “The second step is to compute the loss function, the Hungarian loss for all pairs matched in the previous step” (p. 5 ¶¶ [0001]-[0005] of 3.1); see FIG. 1 and accompanying description).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Nguyen and Zhang’s calculation of the differentiable loss by incorporating Carion’s known DETR model of computing a set loss function using bipartite matching between predicted and ground-truth objects in order to compute the claimed detection loss by finding a bipartite assignment of detections to ground truth because Carion teaches that its set-based loss uses bipartite matching to produce unique assignments between predicted detections and ground truth objects (Carion Abstract), thereby avoiding duplicate assignments during the computation of the detection loss. Carion, Nguyen, and Zhang are analogous art because they each relate to transformer-based computer vision systems that process images to detect and track objects.
Regarding claims 9 and 18,
Nguyen and Zhang teach the computer-implemented method of claim 1 and the system of claim 10, including an overall multi-camera tracking framework that includes calculating a differentiable loss, an auxiliary track loss, tracks, views, and relationships between tracks and views.
Nguyen and Zhang fail to explicitly disclose: wherein calculating the differentiable loss further comprises computing the auxiliary track loss as a sum of all detections associated with a ground truth annotation by Hungarian matching for a view and an intersection over union loss for bounding boxes for respective tracks and views and bounding boxes for a ground truth annotation for all tracks, views, and ground truth annotations.
In a related art, Carion teaches: truth annotation by Hungarian matching (Carion teaches “Our loss produces an optimal bipartite matching between predicted and ground truth objects” and “the Hungarian loss for all pairs matched in the previous step,” where “optimal assignment is computed efficiently with the Hungarian algorithm” and the loss is computed as a sum over the matched prediction/ground truth pairs (see section 3.1 found on p. 5 through top of p. 6, including “
L
m
a
t
c
h
(
Y
i
,
Y
^
σ
(
i
)
)
is a pair-wise matching cost between ground truth
Y
i
and a prediction with index σ(i). This optimal assignment is computed efficiently with the Hungarian algorithm”)
an intersection over union loss (Carion teaches that after the Hungarian assignment, the bounding-box loss is computed as a linear combination including the generalized intersection over union (IoU) loss (see p. 5 second paragraph titled “Bounding box loss”)
bounding boxes for a ground truth annotation (Carion teaches a ground truth set in which each ground truth object includes a ground truth box
b
i
and computes
L
b
o
x
(
b
i
,
b
^
σ
(
i
)
)
(box loss) between the ground truth box and the predicted box (see section 3.1 found from p. 5 through p. 6)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Nguyen and Zhang’s calculation of the auxiliary track loss used in the differentiable loss by incorporating Carion’s teachings of computing the Hungarian loss using Hungarian matching and a generalized intersection over union loss (IoU) for the matched bounding boxes in order to uniquely associate detections with corresponding ground-truth annotations when computing the auxiliary track loss because Carion teaches that its matching loss uniquely assigns a prediction to a ground truth object before computing the Hungarian loss over the matched prediction/ground truth pairs (see Carion p. 4 introduction to section 3 at bottom of page), thereby avoiding duplicate assignments during the computation of the auxiliary track loss.
Allowable Subject Matter
Claims 8 and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form, including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAMUEL DAVID BAYNES whose telephone number is (571)272-0607. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen R Koziol can be reached at (408)918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SDB/
Samuel D. Baynes
Examiner, Art Unit 2665
/Stephen R Koziol/Supervisory Patent Examiner, Art Unit 2665
Prosecution Insights