DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s response, filed 12 March 2026, to the last office action has been entered and made of record.
In response to the cancellation of claims 3-5, they are acknowledged and made of record.
In response to the amendments to the specification and claims, they are acknowledged, supported by the original disclosure, and no new matter is added.
In response to the amendments to the specification, the amended language has overcome the objections to the specification of the previous Office action, and the respective objections have been withdrawn.
In response to the amendments to the claims, specifically addressing the objection to the claim 5 of the previous Office action, the cancellation of claim 5 has obviated the respective objection, and the objection has been withdrawn.
In response to the amendments to the claims, specifically addressing the rejections of the claims under 35 U.S.C. § 112(b), of the previous Office action, the amended language has partially overcome the respective rejections, and the respective rejections have been withdrawn. Please see below corresponding response to arguments and updated
Amendments to the independent claim 1 has necessitated an updated ground of rejection over the applied prior art. Please see below for the updated interpretations and rejections.
Response to Arguments
Applicant's arguments filed 12 March 2026 have been fully considered but they are not persuasive.
In response to Applicant’s arguments on p. 11-12 of Applicant’s reply, that independent claim 1 has been amended to eliminate the ambiguity between the recitation of “going to S4, and performing a fusion feature extraction” and “S4, extracting spatio-temporal trajectory fusion features”, the Examiner respectfully disagrees and notes that the at issue claim language continues to be recited in the claim.
Applicant further cites specification paragraphs [0009] and [0011] as support to indicate a clear distinction between trajectory level fusion and feature extraction. Specification paragraphs [0009] and [0011] recites:
“[0009] S2, judging whether retrieval videos in the retrieval data set query include occluded images, inputting sequences of the occluded images into the trajectory prediction model for a future trajectory prediction, and obtaining a prediction set query_pred including a predicted trajectory; and going directly to S4, and performing a fusion feature extraction but not the trajectory prediction directly for sequences of images without occlusion in S4;”
“[0011] S4, extracting spatio-temporal trajectory fusion features including apparent visual information and motion trajectory information by using a video re-identification model for the query_TP, performing a feature distance measure and candidate video ranking, and obtaining final re-identification performance evaluation indexes mAP and Rank-k, where mAP represents a mean average precision, Rank-k indicates a possibility of a cumulative match characteristic (CMC) curve matching correctly in the first k videos in the ranked gallery, and the CMC curve reflects cumulative match characteristics of a retrieval precision of an algorithm; and using a Rank-1 result as a video re-identification result.”
Specification paragraphs [0009] and [0011] merely recite the at issue claim language, and does not provide additional clarity in determining whether “fusion feature extraction” and “extracting spatio-temporal trajectory fusion features” in “S4” is intended to be the same process or separate processes.
Upon further review of the specification and drawings, Fig. 1 and [0028] appears to depict that “fusion feature extraction” is performed directly for query image sequences without occlusion and without performing “trajectory prediction”; whereas “trajectory prediction” is further performed for images sequences with occlusion, a new fused video sequence “query_TP” is obtained from fusing obtained prediction trajectory “query_pred” with gallery candidate videos in a time domain and a space domain, and performing “fusion feature extraction” for the fused video sequence “query_TP”.
Thus, for the purposes of further treating the Application on the merits, the Examiner assumes the broadest reasonable interpretation of the claims in light of the specification, where both refer to the same process for extracting “spatio-temporal trajectory fusion features”.
In response to Applicant’s arguments on p. 11-12 of Applicant’s reply, that amended independent claim 1 has been amended to eliminate the insufficient antecedent basis issue for the recited “wherein in the S3, in the spatio-temporal trajectory fusion features”, the Examiner respectfully disagrees and notes that the at issue claim language continues to be recited in the claim.
Amended claim 1 presently recites the limitation, “wherein in the S3, in the spatio-temporal trajectory fusion features”. While claim 1 before recites, “S3, fusing the obtained prediction set query_pred with candidate videos in the candidate data set gallery, and obtaining a new fused video set query_TP”, and subsequent to the at issue limitation, recites “S4, extracting spatio-temporal trajectory fusion features”, amended claim 1 is noted to recite that “the spatio-temporal trajectory fusion features” are extracted in the “S4” claim limitations. Thus, “the spatio-temporal trajectory fusion features” does not have proper antecedent basis in the context of the “S3” claim limitations.
In response to Applicant’s arguments on p. 15-26 of Applicant’s reply, that the combination of previously cited prior art of Chen, Li, Shen, Gupta, and Hou fail to provide suggested teachings to the amended claim subject matter of independent claim 1, the Examiner respectfully disagrees.
Examiner notes the claims are treated with their broadest reasonable interpretations consistent with the specification. See MPEP 2111. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Furthermore, the test for obviousness is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ871 (CCPA 1981).
In regards to Applicant’s arguments on p. 19-21, addressing the claim limitations of “adding a number of time frames and space coordinate information to each person on the MARS_traj”, the Examiner notes that the combined teachings of Chen and Li are relied upon to suggest the broadest reasonable interpretation of the claimed subject matter.
Chen is relied upon to teach a method of performing video-based person re-identification based on a network model for learning spatial-temporal fusion features to associate image sequences of a person with positive gallery image sequences of the same person which uses video datasets including different identities and corresponding image sequences or tracklets (see Chen sect. 3.1. Evaluation Datasets and sect. 3.2. Experimental Setup).
Li is relied upon to further teach a known technique of further annotating a dataset for attribute based and image-based person retrieval, where each human image of the dataset is further annotated with different attribute annotations such as time interval of occurrences, scene ID, bounding box locations of body/head-shoulder/upper-body/lower-body/accessories and occlusions (see Li Sect. III. The RAP Dataset, Table III, and Fig. 4).
The combined teachings of Chen and Li suggests to one of ordinary skill in the art that the video datasets for video-based person re-identification used further include additional attribute annotations such as time interval of occurrences, scene ID, bounding box locations of body/head-shoulder/upper-body/lower-body/accessories and occlusions. Here, the additional attribute annotations corresponding to time interval of occurrences, scene ID, and bounding box locations of detected body and accessors and occlusions provides for the broadest reasonable interpretation for “a number of time frames and space coordinate information to each person”.
Thus, the combined teachings of the cited prior art provides for the broadest reasonable interpretation of “adding a number of time frames and space coordinate information to each person on the MARS_traj”.
In regards to Applicant’s arguments on p. 21-22, addressing the claim limitations of “judging whether retrieval videos in the retrieval data set query comprise occluded images”, the Examiner notes that the combined teachings of Chen and Li, with Shen are relied upon to suggest the broadest reasonable interpretation of the claimed subject matter.
Chen is noted to further teach that the datasets include occluded image sequences (see Chen sect. 3.1. Evaluation Datasets).
Shen is relied upon to teach a known technique for occluded pedestrian re-identification where an occlusion dataset for learning occluded images is obtained, occluded image are used as a query and the full-body images as the gallery to obtain the network model, and the gallery dataset is input into the trained model and save pedestrian image features to obtain a pedestrian image feature database and a query pedestrian image is input to obtain pedestrian features, retrieve image features, calculate similarity, and select the photo with the highest similarity and obtain the pedestrian ID of the query image (see Shen [n0020]-[n0027]).
The combined teachings of Chen, Li, and Shen suggests to one of ordinary skill in the art that the occluded image features can be learned to perform re-identification of an occluded pedestrian used as a query image and associate with a corresponding non occluded pedestrian ID, providing for the broadest reasonable interpretation of judging whether retrieval videos in the retrieval data set query comprise occluded images.
Thus, the combined teachings of the cited prior art provides for the broadest reasonable interpretation of “judging whether retrieval videos in the retrieval data set query comprise occluded images”.
In regards to Applicant’s arguments on p. 22-24, addressing the claim limitations of “inputting sequences of the occluded images into a trajectory prediction model for a future trajectory prediction, and obtaining a prediction set query_pred comprising a predicted trajectory”, the Examiner notes that the combined teachings of Chen, Li, Shen, with Gupta are relied upon to suggest the broadest reasonable interpretation of the claimed subject matter.
The combined teachings of Chen, Li , and Shen suggest a base method of performing video-based person re-identification based on a network model for learning spatial-temporal fusion features to associate image sequences of a person with positive gallery image sequences of the same person which uses video datasets including different identities and corresponding image sequences or tracklets of occluded image sequences, and further learns occluded image features to perform re-identification of an occluded pedestrian used as a query image and associate with a corresponding non occluded pedestrian ID (see Chen sect. 2. Our Method, sect. 3.1. Evaluation Datasets; see Shen [n0020]-[n0027]).
Gupta teaches a known technique train generative adversarial network (GAN) with given history of human motion paths to generate a variety of socially-acceptable trajectories, where the generator of the GAN takes as input past trajectories for people of in a scene and the outputs predicted trajectories that are trained to be socially acceptable (see Gupta sect. 3.1. Problem Definition, sect. 3.3. Socially-Aware GAN, and Fig. 2).
The combined teachings of Chen, Li, Shen, and Gupta suggests to one of ordinary skill in the art that identified corresponding image sequences for an occluded pedestrian can be used to predict socially acceptable and realistic trajectories, providing for the broadest reasonable interpretation of inputting sequences of the occluded images into a trajectory prediction model for a future trajectory prediction, and obtaining a prediction set query_pred comprising a predicted trajectory.
Thus, the combined teachings of the cited prior art provides for the broadest reasonable interpretation of “inputting sequences of the occluded images into a trajectory prediction model for a future trajectory prediction, and obtaining a prediction set query_pred comprising a predicted trajectory”.
In regards to Applicant’s arguments on p. 24, addressing the claim limitations of calculating a temporal fusion loss and space fusion loss and calculating a limited fusion loss, the Examiner notes that the further teachings of Chen are relied upon to suggest the broadest reasonable interpretation of the claimed subject matter.
Chen is noted to further teach determining temporal and spatial features, and the temporal feature is fused with spatial features to calculate a final feature representation, where loss functions are determined using the final features representation for an image sequence pair, and that loss function maximizes the distance between negative pair and minimizes relative distance between positive pairs (see Chen sect. 2.4. RNN Layer, Eq. (5), sect. 2.5. Spatial-Temporal Feature Fusion, and sect. 2.6. Multi-Loss Layer, Eq. (7) and Eq. (9)).
Thus, the combined teachings of the cited prior art, notably the further teachings of Chen provides for the broadest reasonable interpretation for the claimed subject matter regarding calculating a temporal fusion loss and space fusion loss and calculating a limited fusion loss.
In regards to Applicant’s arguments on p. 24-26, addressing the claim limitations of the step “S4” of “performing a fusion feature extraction but not the trajectory prediction directly for sequences of images without occlusion in the S4” and “extracting spatio-temporal trajectory fusion features comprising apparent visual information and motion trajectory information by using a video re-identification model for the new fused video set query_TP”, the Examiner notes that the combined teachings of Chen, Li, Shen, and Gupta, with Hou are relied upon to suggest the broadest reasonable interpretation of the claimed subject matter.
Chen, Li , Shen, and Gupta are relied upon to suggests a method of performing video-based person re-identification based on a network model for learning spatial-temporal fusion features to associate image sequences of a person with positive gallery image sequences of the same person which uses video datasets including different identities and corresponding image sequences or tracklets of occluded image sequences, and further learns occluded image features to perform re-identification of an occluded pedestrian used as a query image and associate with a corresponding non occluded pedestrian ID, and predicts socially acceptable and realistic trajectories using identified corresponding image sequences for an occluded pedestrian (see Chen sect. 2. Our Method, sect. 3.1. Evaluation Datasets; see Shen [n0020]-[n0027]; see Gupta sect. 3.1. Problem Definition, sect. 3.3. Socially-Aware GAN, and Fig. 2).
Chen is noted to further teach determining temporal and spatial features, and the temporal feature is fused with spatial features to calculate a final feature representation, where loss functions are determined using the final features representation for an image sequence pair, and that loss function maximizes the distance between negative pair and minimizes relative distance between positive pairs (see Chen sect. 2.4. RNN Layer, Eq. (5), sect. 2.5. Spatial-Temporal Feature Fusion, and sect. 2.6. Multi-Loss Layer, Eq. (7) and Eq. (9)).
Hou is further relied upon to teach a known technique for a TCLNet that extracts complementary features of consecutive video frames for video person re-identification, where a video consisting of consecutive frames is used to extract segment features, and applies a temporal average pooling that aggregates the set of segment features to generate video features and a final video representation can be obtained by concatenating the feature vectors, where the video feature is extracted using the trained TCLNet for retrieval under cosine distance, and that the performance of the method is evaluated based on mAP and top-1 accuracy metrics (see Hou sect. 3.3 Overall Architecture and Fig. 4, sect. 4.1 Dataset and Settings, sect. 4.2 Comparison with State-of-the-Art Methods and Table 1).
The combined teachings of Chen, Li, Shen, Gupta, and Hou suggests to one of ordinary skill in the art that segment features representing consecutive frames of a video can be further extracted and used to perform video person re-identification, where the video frames used for the extraction correspond to the occluded and non-occluded image sequences of a person, which have extracted temporal and spatial features fused the features into a final feature representation.
Thus, the combined teachings of the cited prior art provides for the broadest reasonable interpretation for the claim limitations of step “S4” of “performing a fusion feature extraction but not the trajectory prediction directly for sequences of images without occlusion in the S4” and “extracting spatio-temporal trajectory fusion features comprising apparent visual information and motion trajectory information by using a video re-identification model for the new fused video set query_TP”.
Claim Objections
Claims 1 and 6 are objected to because of the following informalities:
Amended claim 1 recites, in the limitation beginning with, “wherein in the S3, in the spatio-temporal trajectory fusion features…”, the amended feature:
“wherein
PNG
media_image1.png
43
146
media_image1.png
Greyscale
represents Euclidean distances between the coordinates corresponding to predicted trajectory sequences and candidate sequences in the gallery”. The recited formula is illegible due to the degraded resolution quality.
For the purposes of further treating the Application on the merits, the Examiner assumes the following is intended, “wherein
l
j
=
∑
i
=
1
n
p
i
n
, (n=9-j),
p
i
represents Euclidean distances between the coordinates corresponding to predicted trajectory sequences and candidate sequences in the gallery”.
Amended claim 1 recites, in the limitation beginning with, “wherein in the S3, after the temporal fusion loss and the space fusion loss are obtained…”, the amended feature, “where in N2 is a total number of video sequences…”, where a typographical error is assumed to exist, and “wherein N2 is a total number of video sequences” is assumed to be intended.
Amended claim 6 recites, “wherein in the S4, the new fused video set query_TP extracted after the fusion of temporal trajectory fusion and the space trajectory fusion and the candidate set gallery are sent to a temporal complementary learning network (TCLNet)”, where the Examiner assumes a typographical error exists, and that “wherein in the S4, the new fused video set query_TP, extracted after the fusion of temporal trajectory fusion and the space trajectory fusion, and the candidate set gallery are sent to a temporal complementary learning network (TCLNet)” is intended for grammatical clarity and the intended interpretation of the claim subject matter.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-2 and 6 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Amended claim 1 recites the limitations, “going to S4, and performing a fusion feature extraction but not the trajectory prediction directly for sequences of images without occlusion in the S4” and “S4, extracting spatio-temporal trajectory fusion features comprising apparent visual information and motion trajectory information by using a video re-identification model for the new fused video set query_TP”.
As “a fusion feature extraction” is recited to be performed for sequences of images without occlusion in the S4 step, and then the S4 step further recites “extracting spatio-temporal trajectory fusion features”, the claim does not clearly indicate whether the “fusion feature extraction” and “extracting spatio-temporal trajectory fusion features” are referring to a process for extracting “fusion features” and a separate process for extracting “spatio-temporal trajectory fusion features”, or both are the same process for extracting “spatio-temporal trajectory fusion features”.
Thus, claim 1 fails to particularly point out and distinctly claim the subject matter which the inventor regards as the invention.
For the purposes of further treating the Application on the merits, the Examiner assumes the broadest reasonable interpretation of the claims in light of the specification, where both refer to the same process for extracting “spatio-temporal trajectory fusion features”.
Amended claim 1 further recite the limitation, “wherein in the S3, in the spatio-temporal trajectory fusion features”. There is insufficient antecedent basis for this limitation in the claim. While claim 1 recites before, “S3, fusing the obtained prediction set query_pred with candidate videos in the candidate data set gallery, and obtaining a new fused video set query_TP”, and subsequently recites “S4, extracting spatio-temporal trajectory fusion features”, claim 1 is noted to recite that “the spatio-temporal trajectory fusion features” are extracted in the “S4” claim limitations and does not have proper antecedent basis in the context of the “S3” claim limitations.
For the purposes of further treating the Application on the merits, the Examiner assumes claim 1 is intended to refer to the “fusing the obtained prediction set query_pred with candidate videos in the candidate data set gallery”, and “wherein in the S3, in the fusing of the obtained prediction set query_pred with the candidate videos in the candidate data set gallery…” is intended to be recited.
Claims 2 and 6 are dependent claims of independent claim 1 and incorporate the at issue claim limitations by dependency, and are similarly rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-2 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (“Deep Spatial-Temporal Fusion Network for Video-Based Person Re-Identification”) in view of Li et al. (“A Richly Annotated Pedestrian Dataset for Person Retrieval in Real Surveillance Scenarios”), herein Li, Shen et al. (CN 112801051), herein Shen, Gupta et al. (“Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks”), herein Gupta, and Hou et al. (“Temporal Complementary Learning for Video Person Re-identification”), herein Hou.
Regarding claim 1, Chen discloses a video personnel re-identification method based on trajectory fusion in a complex underground space, comprising following steps:
S1, establishing a trajectory fusion data set MARS_traj, comprising personnel identity data and video sequences (see Chen sect. 3.1. Evaluation Datasets, where video datasets for video-based person re-identification are used, including different identities and corresponding image sequences or tracklets); wherein test sets in the MARS_traj comprise a retrieval data set query and a candidate data set gallery (see Chen sect. 3.2. Experimental Setup, where during testing, all frames of a certain person are chosen and divided into 16 image frame to extract fusion features with the pre-trained model and extract final average feature to compute the matching rate against the features of gallery sequences);
S3, fusing the obtained query_pred with candidate videos in the candidate data set gallery, and obtaining a new fused video set query_TP (see Chen sect. 2. Our Method, where the spatial and temporal features of a person are learned and used to train a CNN and RNN network model to pull instances of the same person closer and instances belonging to different persons farther; see Chen Fig. 2 and sect. 2.2. CNN layer, where the input are image sequences of instances of a person; see Chen sect. 2.6 Multi-Loss Layer, where fused feature representations are used to compute loss functions which maximizes the distance between negative pair and minimizes relative distance between positive pairs; thus suggesting associating input image sequence instances of a person with positive gallery image sequences of the same person);
wherein in the S3, in the spatio-temporal trajectory fusion features, a temporal trajectory fusion is to calculate a temporal fusion loss in a time domain considering a continuity of a predicted trajectory and a favourable historical trajectory, as shown in formula (1):
l
t
t
e
m
=
max
ϕ
Δ
t
-
T
,
0
(1),
wherein
Δ
t
is a frame difference between a last frame of the video sequences in the query and a first frame of video sequences in the gallery, and a frame constant threshold T and a large constant
ϕ
determine a temporal continuity of the frame difference
Δ
t
between the query and the gallery (see Chen sect. 2.4. RNN Layer, Eq. (5), sect. 2.5. Spatial-Temporal Feature Fusion, and sect. 2.6. Multi-Loss Layer, and Eq. (7) for condition i
≠
j, where a temporal feature is determined, and the temporal feature is fused with spatial features to calculate a final feature representation, and loss functions are determined using the final features representation for an image sequence pair, and that loss function maximizes the distance between negative pair and minimizes relative distance between positive pairs);
wherein in the S3, in the spatio-temporal trajectory fusion features, a space trajectory fusion is to calculate a space fusion loss
l
t
s
p
a
considering a dislocation of the predicted trajectory and the frames of the candidate videos in the gallery:
l
t
s
p
a
=
min
l
j
,
∀
j
∈
1,2
,
…
,
N
,
N
=
2,3
,
…
,
7
,
(2),
wherein
l
j
=
∑
i
=
1
n
p
i
n
, (n=9-j),
p
i
represents Euclidean distances between the coordinates corresponding to predicted trajectory sequences and candidate sequences in the gallery; and N represents an allowable deviation range of the predicted trajectory from candidate video frames (see Chen sect. 2.3. Temporal pooling, Eq. (2), sect. 2.5. Spatial-Temporal Feature Fusion, and sect. 2.6. Multi-Loss Layer, Eq. (7) for condition i=j, where a spatial feature is determined, and the spatial feature is fused with temporal features to calculate a final feature representation, and loss functions are determined using the final features representation for an image sequence pair, and that loss function maximizes the distance between negative pair and minimizes relative distance between positive pairs);
wherein in the S3, after the temporal fusion loss and the space fusion loss are obtained, a limited fusion loss
l
i
j
in the time domain and a space domain of the jth video in the gallery and the ith video in the query_pred is calculated according to formula (3):
l
i
j
=
min
l
t
t
e
m
+
l
t
s
p
a
,
∀
j
∈
1,2
,
…
,
N
2
(3),
where in N2 is a total number of video sequences in the gallery, and a minimum j value that minimizes
l
i
j
is obtained according to the formula (3), so that the jth video in the gallery is sent to the query_TP set for a subsequent extraction of the spatio-temporal trajectory fusion features (see Chen sect. 2.5. Spatial-Temporal Feature Fusion, and sect. 2.6. Multi-Loss Layer and Eq. (9), where the spatial feature is fused with temporal features to calculate a final feature representation, and a combined loss function is determined using the final features representation for an image sequence pair, and that loss function maximizes the distance between negative pair and minimizes relative distance between positive pairs).
Chen does not explicitly disclose adding a number of time frames and space coordinate information to each person on the MARS_traj.
Li teaches in a related and pertinent richly annotated pedestrian (RAP) dataset which serves as a unified benchmark for both attribute based and image-based person retrieval in real surveillance scenarios (see Li Abstract), where each human image of the dataset is further annotated with different attribute annotations such as time interval of occurrences, scene ID, bounding box locations of body/head-shoulder/upper-body/lower-body/accessories and occlusions (see Li Sect. III. The RAP Dataset, Table III, and Fig. 4).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Li to the teachings of Chen, such that the video datasets for video-based person re-identification used further include additional attribute annotations such as time interval of occurrences, scene ID, bounding box locations of body/head-shoulder/upper-body/lower-body/accessories and occlusions.
This modification is rationalized as an application of a known technique to a known method ready for improvement to yield predictable results.
In this instance, Chen disclose a base method of performing video-based person re-identification based on a network model for learning spatial-temporal fusion features to associate image sequences of a person with positive gallery image sequences of the same person which uses video datasets including different identities and corresponding image sequences or tracklets.
Li teaches a known technique of further annotating a dataset for attribute based and image-based person retrieval, where each human image of the dataset is further annotated with different attribute annotations such as time interval of occurrences, scene ID, bounding box locations of body/head-shoulder/upper-body/lower-body/accessories and occlusions.
One of ordinary skill in the art would have recognized that by applying Li's technique would allow for the method of Chen to further use video datasets with additional attribute annotations such as time interval of occurrences, scene ID, bounding box locations of body/head-shoulder/upper-body/lower-body/accessories and occlusions, predictably leading to an improved video dataset with a richer annotation data for performing video-based person re-identification.
While Chen teaches that the datasets include occluded image sequences (see Chen sect. 3.1. Evaluation Datasets);
Chen and Li do not explicitly disclose S2, judging whether retrieval videos in the retrieval data set query comprise occluded images.
Shen teaches in a related and pertinent occluded pedestrian re-identification method based on multi-task learning (see Shen Abstract), where a random occlusion module to automatically generate partially occluded images and obtain an occlusion dataset for learning occluded images, occluded image are used as a query and the full-body images as the gallery to obtain the network model, and the gallery dataset is input into the trained model and save pedestrian image features to obtain a pedestrian image feature database and a query pedestrian image is input to obtain pedestrian features, retrieve image features, calculate similarity, and select the photo with the highest similarity and obtain the pedestrian ID of the query image (see Shen [n0020]-[n0027]).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Shen to the teachings of Chen and Li, such that the occluded image features can be learned to perform re-identification of an occluded pedestrian used as a query image and associate with a corresponding non occluded pedestrian ID.
This modification is rationalized as an application of a known technique to a known method ready for improvement to yield predictable results.
In this instance, Chen and Li disclose a base method of performing video-based person re-identification based on a network model for learning spatial-temporal fusion features to associate image sequences of a person with positive gallery image sequences of the same person which uses video datasets including different identities and corresponding image sequences or tracklets which include occluded image sequences.
Shen teaches a known technique for occluded pedestrian re-identification where an occlusion dataset for learning occluded images is obtained, occluded image are used as a query and the full-body images as the gallery to obtain the network model, and the gallery dataset is input into the trained model and save pedestrian image features to obtain a pedestrian image feature database and a query pedestrian image is input to obtain pedestrian features, retrieve image features, calculate similarity, and select the photo with the highest similarity and obtain the pedestrian ID of the query image.
One of ordinary skill in the art would have recognized that by applying Shen’s technique would allow for the method of Chen and Li to further learn occluded image features to perform re-identification of an occluded pedestrian used as a query image and associate with a corresponding non occluded pedestrian ID, providing for the broadest reasonable interpretation of judging whether retrieval videos in the retrieval data set query comprise occluded images, and predictably leading to an improved video-based person re-identification method, where occluded pedestrian images can be associated with corresponding non occluded pedestrian ID.
Chen, Li, and Shen do not explicitly disclose inputting sequences of the occluded images into a trajectory prediction model for a future trajectory prediction, and obtaining a prediction set query_pred comprising a predicted trajectory.
Gupta teaches in a related and pertinent method to train generative adversarial network (GAN) with given history of human motion paths to generate a variety of socially-acceptable trajectories (see Gupta Abstract and sect. 1. Introduction), where the generator of the GAN takes as input past trajectories for people of in a scene and the outputs predicted trajectories, where the encoder of the generator of the GAN learns the state of a person and stores their history of motion and a decoder generates the future trajectory conditioned on pooled hidden states of all the people present in the scene, and that a discriminator further classifies the input and predicted trajectories as socially acceptable or fake (see Gupta sect. 3.1. Problem Definition, sect. 3.3. Socially-Aware GAN, and Fig. 2).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Gupta to the teachings of Chen, Li, and Shen, such that identified corresponding image sequences for an occluded pedestrian can be used to predict socially acceptable and realistic trajectories.
This modification is rationalized as an application of a known technique to a known method ready for improvement to yield predictable results.
In this instance, Chen, Li , and Shen disclose a base method of performing video-based person re-identification based on a network model for learning spatial-temporal fusion features to associate image sequences of a person with positive gallery image sequences of the same person which uses video datasets including different identities and corresponding image sequences or tracklets of occluded image sequences, and further learns occluded image features to perform re-identification of an occluded pedestrian used as a query image and associate with a corresponding non occluded pedestrian ID.
Gupta teaches a known technique train generative adversarial network (GAN) with given history of human motion paths to generate a variety of socially-acceptable trajectories, where the generator of the GAN takes as input past trajectories for people of in a scene and the outputs predicted trajectories that are trained to be socially acceptable.
One of ordinary skill in the art would have recognized that by applying Gupta’s technique would allow for the method of Chen, Li, and Shen to predict socially acceptable and realistic trajectories using identified corresponding image sequences for an occluded pedestrian, and predictably leading to an improved video-based person re-identification method, where socially acceptable and realistic trajectories are predicted using occluded pedestrian image sequences.
While Chen teaches that the performance of the method is evaluating based on CMC rank curves and computes Rank R results including Rank 1 results (see Chen sect. 3.3. Evaluation and Results, Table 1, 2, 3, Fig. 4, and Fig. 5); Chen, Li, and Shen do not explicitly disclose
and going to S4, and performing a fusion feature extraction but not the trajectory prediction directly for sequences of images without occlusion in the S4;
and S4, extracting spatio-temporal trajectory fusion features comprising apparent visual information and motion trajectory information by using a video re-identification model for the new fused video set query_TP, performing a feature distance measure and candidate video ranking, and obtaining final re-identification performance evaluation indexes mAP and Rank-k, wherein mAP represents a mean average precision, Rank-k indicates a possibility of a cumulative match characteristic (CMC) curve matching correctly in first k videos in a ranked gallery, and the CMC curve reflects cumulative match characteristics of a retrieval precision of an algorithm; and using a Rank-1 result as a video re-identification result.
Hou teaches in a related and pertinent Temporal Complementary Learning Network (TCLNet) that extracts complementary features of consecutive video frames for video person re-identification (see Hou Abstract), where the TCLNet accepts a video consisting of consecutive frames to extract segment features, and applies a temporal average pooling that aggregates the set of segment features to generate video features and a final video representation can be obtained by concatenating the feature vectors (see Hou sect. 3.3 Overall Architecture and Fig. 4), where the video feature is extracted using the trained TCLNet for retrieval under cosine distance (see Hou sect. 4.1 Dataset and Settings), and that the performance of the method is evaluated based on mAP and top-1 accuracy (see Hou sect. 4.2 Comparison with State-of-the-Art Methods and Table 1).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Hou to the teachings of Chen, Li, Shen and Gupta, such that segment features representing consecutive frames of a video can be further extracted and used to perform video person re-identification, where the video frames used for the extraction correspond to the occluded and non-occluded image sequences of a person, which have extracted temporal and spatial features fused the features into a final feature representation.
This modification is rationalized as an application of a known technique to a known method ready for improvement to yield predictable results.
In this instance, Chen, Li , Shen, and Gupta disclose a base method of performing video-based person re-identification based on a network model for learning spatial-temporal fusion features to associate image sequences of a person with positive gallery image sequences of the same person which uses video datasets including different identities and corresponding image sequences or tracklets of occluded image sequences, and further learns occluded image features to perform re-identification of an occluded pedestrian used as a query image and associate with a corresponding non occluded pedestrian ID, and predicts socially acceptable and realistic trajectories using identified corresponding image sequences for an occluded pedestrian.
Hou teaches a known technique for a TCLNet that extracts complementary features of consecutive video frames for video person re-identification, where a video consisting of consecutive frames is used to extract segment features, and applies a temporal average pooling that aggregates the set of segment features to generate video features and a final video representation can be obtained by concatenating the feature vectors, where the video feature is extracted using the trained TCLNet for retrieval under cosine distance, and that the performance of the method is evaluated based on mAP and top-1 accuracy metrics.
One of ordinary skill in the art would have recognized that by applying Hou’s techniques would allow for the method of Chen, Li, Shen, and Gupta to further extract segment features representing consecutive frames of a video and be used to perform video person re-identification where the video frames used for the extraction correspond to the occluded and non-occluded image sequences of a person, which have extracted temporal and spatial features fused the features into a final feature representation, predictably leading to an improved video-based person re-identification method, where additional spatial-temporal information is used from the extracted video segment features to perform video-based person re-identification.
Regarding claim 2, please see the above rejection of claim 1. Chen, Li, Shen, Gupta, and Hou disclose the video personnel re-identification method based on the trajectory fusion in the complex underground space according to claim 1,
wherein in the S2, the future trajectory prediction is based on the favourable historical trajectory, and is realized by a Social GAN model and belongs to historical trajectory coordinates of known personnel, and predicted trajectory coordinates are obtained (see Gupta sect. 3.3. Socially-Aware GAN and Fig. 2, where the generator of the GAN takes as input past trajectories for people of in a scene and the outputs predicted trajectories, where the encoder of the generator of the GAN learns the state of a person and stores their history of motion and a decoder generates the future trajectory conditioned on pooled hidden states of all the people present in the scene, and that a discriminator further classifies the input and predicted trajectories as socially acceptable or fake).
Regarding claim 6, please see the above rejection of claim 1. Chen, Li, Shen, Gupta, and Hou disclose the video personnel re-identification method based on the trajectory fusion in the complex underground space according to claim 1, wherein in the S4, the new fused video set query_TP extracted after the temporal trajectory fusion and the space trajectory fusion and the candidate set gallery are sent to a temporal complementary learning network (TCLNet), and finally, group features are aggregated by temporal average pooling to obtain a final fused video feature vector (see Hou sect. 3.3 Overall Architecture and Fig. 4, where the Temporal Complementary Learning Network (TCLNet) accepts a video consisting of consecutive frames to extract segment features, and applies a temporal average pooling that aggregates the set of segment features to generate video features and a final video representation can be obtained by concatenating the feature vectors);
the TCLNet takes a ResNet-50 network as a backbone network, wherein a temporal saliency boosting (TSB) module and a temporal saliency erasing (TSE) module are inserted (see Hou sect. 3.3 Overall Architecture and Fig. 4, where the architecture of TCLNet integrates TSE and TSB modules with a ResNet-50 network); and
for a T-frame continuous video, the backbone network with the TSB inserted extracts the features from each frame, and the features are labelled as
F
=
F
1
,
F
2
,
…
,
F
T
, and then the features are equally divided into k groups; each group comprises N continuous frame features
C
k
=
F
k
-
1
N
+
1
,
…
,
F
k
N
, and each group is input into the TSE, and complementary features are extracted by formula (4):
c
k
=
T
S
E
F
k
-
1
N
+
1
,
…
,
F
k
N
=
T
S
E
(
C
k
)
(4);
(see Hou sect. 3.3 Overall Architecture, where given a video consisting of T consecutive frames , the backbone network with TSB inserted extracts features for each frame and the features are equally divided into segments, and each segment contains N consecutive feature maps, and each segment is fed into TSE to extract complementary features for the segment frames; see also Hou Eq. (7)) the distance measure between a video feature vector A (x1, y1) in the new fused video set query_TP and the video feature vector B (x2, y2) in the candidate set gallery is calculated by a cosine similarity, as shown in formula (5):
cos
θ
=
x
1
x
2
+
y
1
y
2
x
1
2
+
y
1
2
x
2
2
+
y
2
2
(5);
and the videos in the gallery are ranked according to the distance measure (see Hou sect. 4.1 Dataset and Settings, where the video feature is extracted using the trained TCLNet for retrieval under cosine distance), and the re-identification evaluation indexes mAP and Rank-k are calculated according to a ranking result, and the Rank-1 result is taken as the video re-identification result (see Chen sect. 3.3. Evaluation and Results, Table 1, 2, 3, Fig. 4, and Fig. 5, where the performance of the method is evaluated based on CMC rank curves and computes Rank R results including Rank 1 results; and see Hou sect. 4.2 Comparison with State-of-the-Art Methods and Table 1, where the performance of the method is evaluated based on mAP and top-1 accuracy).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIMOTHY WING HO CHOI whose telephone number is (571)270-3814. The examiner can normally be reached 9:00 AM to 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VINCENT RUDOLPH can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIMOTHY CHOI/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights