Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on September 1, 2026 has been entered.
Specification
The disclosure is objected to because of the following informalities:
In [0018] line 8, “encode” should read “encoder”.
Appropriate correction is required.
Claim Objections
Claim 1 is objected to because of the following informalities:
Claim 1 as recited has a spelling error as “encode” should read “encoder”. For the record, the examiner recommends claim 1 to be rewritten as follow, and interpretation will be as such until clarification is made of record or applicant accepts this proposal and makes changes accordingly.
1. (Currently Amended) A system for detecting people, comprising: an electronic sensor, wherein the electronic sensor captures a video of people around a vehicle to generate a plurality of consecutive frames, and wherein the plurality of consecutive frames comprises an original frame and one or more new frames; a person detector and tracker, comprising a boundary box generator that detects an enclosed boundary box within the original frame and one or more new frames, and a confidence pre- processing procedure for predicting locations of each person identified using a mathematical model; wherein the confidence pre-processing procedure includes a tracker match comparator which predicts at least one tracker based on the original frame and one or more new frames, compares the boundary box and the at least one tracker between each frame, and identifies a match between the boundary box and the at least one tracker in the original and each of the one or more new frames; a tracker optimizer for optimizing the at least one tracker into an updated tracker based on detection by the tracker match comparator, wherein the tracker optimizer validates a matched tracker after persistence for at least a first threshold number (M) of consecutive frames, and retains an unmatched tracker for up to a second threshold number (K) of consecutive frames before discarding the unmatched tracker; and a pose estimator, comprising a backbone network for constructing a feature map of pose related information from the updated tracker and the boundary box using deep learning approaches, and a keypoint encoder and decoder for encoding at least one keypoint location into a 2-D representation and decoding maximum values of the 2-D representation using statistical methods; and wherein the tracker optimizer evaluates matched trackers and unmatched trackers to determine valid detections, converts the trackers determined to be valid detections into updated trackers, and provides the updated trackers to the pose estimator for pose estimation.
Response to Arguments
Applicant's arguments filed September 1, 2026 have been fully considered but they are not persuasive.
Regarding claims 1 and 16,
(1) applicant states that “none of the cited references Lee, Zheng, Mandal, or Xiao alone or in combination with other references teach this limitation. As such, Lee in view of Zheng, Mandal, and Xiao does not teach each and every limitation of independent claim 1” and “The cited references alone and in combination do not teach this limitation. As such, Lee in view of Zheng, Mandal, and Xiao does not teach each and every limitation of independent claim 16.”. Examiner disagrees with this statement. In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, it would have been prima facie obvious to one of ordinary skill in the art to have modified Lee to incorporate the teachings of Zheng for the tracker optimizer to validate a matched tracker after persistence for at least a first threshold number (M) of consecutive frames in order to improve tracking accuracy by determining valid detections. It would have been prima facie obvious to one of ordinary skill in the art to incorporate the teachings of Mandal to retain an unmatched tracker for up to a threshold number of consecutive frames before discarding the unmatched tracker in order to avoid nonnegligible true object missing and to determine valid detections. It would have been prima facie obvious to one of ordinary skill in the art to incorporate the teachings of Xiao to utilize a pose estimator in order to perform simultaneous pose detection and tracking.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-8 and 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Lee (US Patent Publication No.: US 2023/0351610 A1), hereinafter Lee, in view of Zheng (PCT Patent Pub. No.: WO 2020001302 A1), hereinafter Zheng, further in view of Mandal (ByteTrack: Multi-Object Tracking by Associating Every Detection Box, Qure.ai Tech Blog, August 8, 2022, https://blog.qure.ai/notes/byte-track-multi-object-tracking), hereinafter Mandal, further in view of Xiao (Simple Baselines for Human Pose Estimation and Tracking, ECCV 2018), hereinafter Xiao.
Regarding claim 1, Lee teaches a system for detecting people, comprising: an electronic sensor, wherein the electronic sensor captures a video (The camera 1130 may capture a photo and/or a video. [0106]) of people around a vehicle (For example, the electronic device 1100 may be implemented as at least a part of a mobile device such as a mobile phone, a smart phone, a PDA, a netbook, a tablet computer or a laptop computer, a wearable device such as a smart watch, a smart band or smart glasses, a computing device such as a desktop or a server, a home appliance such as a television, a smart television or a refrigerator, a security device such as a door lock, or a vehicle such as an autonomous vehicle or a smart vehicle. [0104]) to generate a plurality of consecutive frames (In another general aspect, an electronic device includes: a camera configured to generate an input image comprising a plurality of image frames. [0024]), and wherein the plurality of consecutive frames comprises an original frame (In response to a target object being determined in a first image frame, a target box 202 corresponding to the target object may be designated (e.g., generated or determined). [0055]) and one or more new frames (Referring to FIG. 3, a plurality of image frames of a search image 300 may be divided into a plurality of sequence groups including a first sequence group 301 and a second sequence group 302. [0058]
PNG
media_image1.png
814
690
media_image1.png
Greyscale
); a person detector and tracker (
PNG
media_image2.png
572
822
media_image2.png
Greyscale
), comprising a boundary box generator (The first tracking result may include first bounding boxes of the first image frames according to the forward object tracking performed on the first image frames. [0008]) that detects an enclosed boundary box within the original frame (In response to a target object being determined in a first image frame, a target box 202 corresponding to the target object may be designated (e.g., generated or determined). [0055]) and one or more new frames (The object tracking apparatus may determine a target box 204 corresponding to the target object in the search region 203 (i.e., one or more new frames) through the regression analysis and generate a tracking result based on box information of the target box 204. [0057]), and a confidence pre-processing procedure for predicting locations of each person identified (determining a confidence of the template candidate using a result of comparing a first tracking result determined by the forward object tracking performed on the first image frames and a second tracking result determined by the backward object tracking performed on the first image frames. [0005]) using a mathematical model (The object tracking apparatus 100 may generate the tracking result 103 using an object tracking model 110. The object tracking model 110 may be or include an artificial intelligence model trained or learned based on machine learning. [0046]); wherein the confidence pre-processing procedure includes a tracker match comparator (The object tracking apparatus may calculate (e.g., determine) a similarity by comparing the template feature map 211 and the search feature map 212. [0056]) which predicts at least one tracker (Referring to FIG. 1, an object tracking apparatus 100 may output a tracking result 103 based on a template image 101 and a search image 102. [0044]
PNG
media_image3.png
768
612
media_image3.png
Greyscale
) based on the original frame and one or more new frames (For example, the template image 101 (i.e., the original frame) may correspond to one of a plurality of image frames included in an input video file, and the search image 102 (i.e., one or more new frames) may correspond to one or more of the image frames after ( e.g., temporally subsequent to) the image frame corresponding to the template image 101. [0045]), compares the boundary box and the at least one tracker between each frame (The object tracking apparatus may determine a target box 204 corresponding to the target object in the search region 203 through the regression analysis and generate a tracking result based on box information of the target box 204. [0057]), and identifies a match between the boundary box and the at least one tracker in the original and each of the one or more new frames (In response to a target object being determined in a first image frame, a target box 202 corresponding to the target object may be designated (e.g., generated or determined). [0055]. The object tracking apparatus may determine a target box 204 corresponding to the target object in the search region 203 through the regression analysis and generate a tracking result based on box information of the target box 204. [0057]
PNG
media_image4.png
542
862
media_image4.png
Greyscale
); a tracker optimizer for optimizing the at least one tracker into an updated tracker (For example, a difference between an appearance of a target object appearing in first image frames in the first sequence group 301 and an appearance of a target object appearing in second image frames of the second sequence group 302 may occur, and a first template 311 for object tracking in the first sequence group 301 may be updated to a second template 321 for object tracking in the second sequence group 302. [0059]) based on detection by the tracker match comparator (For example, the object tracking apparatus may determine the confidence of the template candidate 412 based on a degree of overlap between at least some of corresponding pairs obtained from the first tracking result F, and the second tracking result B. [0067]); wherein the tracker optimizer evaluates matched trackers to determine valid detections (For example, the object tracking apparatus may determine the confidence of the template candidate 412 based on a degree of overlap between at least some of corresponding pairs obtained from the first tracking result F, and the second tracking result B. [0067]), converts the trackers determined to be valid detections into updated trackers (For example, a difference between an appearance of a target object appearing in first image frames in the first sequence group 301 and an appearance of a target object appearing in second image frames of the second sequence group 302 may occur, and a first template 311 for object tracking in the first sequence group 301 may be updated to a second template 321 for object tracking in the second sequence group 302. [0059]).
Lee does not teach the following limitations as further recited, but Zheng further teaches wherein the tracker optimizer validates a matched tracker after persistence for at least a first threshold number (M) of consecutive frames (Optionally, the analyzing the validity of the identification number based on data in the consecutive M frames of images includes: Acquiring data of the identification number in consecutive M frames of images, and determining whether M is within a specified numerical range; wherein the data of the identification number in consecutive M frames of images includes: the identification number of consecutive M frames The coordinates and / or timestamp of the central pixel of the portrait in each frame of the image; If M is greater than a preset minimum value Mmin and less than a preset maximum value Mmax, the identification number is determined to be a valid number, and it is retained in the number statistics list. Page 3 4th paragraph).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Lee to incorporate the teachings of Zheng for the tracker optimizer to validate a matched tracker after persistence for at least a first threshold number (M) of consecutive frames in order to improve tracking accuracy by counting a valid tracker.
The combination of Lee and Zheng does not teach the following limitations as further recited, but Mandal further teaches retains an unmatched tracker (For the unmatched tracks T_re-remain after the second association, we put them into T_lost. Page 8 2nd paragraph) for up to a second threshold number (K) of consecutive frames before discarding the unmatched tracker (For each track in T_lost , only when it exists for more than a certain number of frames (in paper, this value is 30 frames), we delete it from the tracks T. Page 8 2nd paragraph); wherein the tracker optimizer evaluates unmatched trackers (For the unmatched tracks T_re-remain after the second association, we put them into T_lost. Page 8 2nd paragraph) to determine valid detections (For each track in T_lost , only when it exists for more than a certain number of frames (in paper, this value is 30 frames), we delete it from the tracks T. Page 8 2nd paragraph).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Mandal to retain an unmatched tracker for up to a threshold number of consecutive frames before discarding the unmatched tracker in order to avoid nonnegligible true object missing and to determine valid detections.
The combination of Lee, Zheng and Mandal does not teach the following limitations as further recited, but Xiao further teaches a pose estimator, comprising a backbone network (Our pose estimation is based on a few deconvolutional layers added on a backbone network, ResNet in this work. Page 2 2nd paragraph) for constructing a feature map (Our pose estimation is based on a few deconvolutional layers added on a backbone network, ResNet in this work. It is probably the simplest way to estimate heat maps from deep and low resolution feature maps. Page 2 2nd paragraph) of pose related information (Fig. 2 (d) Pose to be tracked
PNG
media_image5.png
695
1018
media_image5.png
Greyscale
) from the updated tracker (Fig. 2 (e) Shifted tracked pose) and the boundary box (For the processing frame in videos, the boxes from a human detector and boxes generated by propagating joints from previous frames using optical flow are unified using a bounding box Non-Maximum Suppression (NMS) operation. Page 6 last paragraph) using deep learning approaches (Our pose estimation is based on a few deconvolutional layers added on a backbone network, ResNet in this work. Page 2 2nd paragraph), and a keypoint encoder and decoder for encoding at least one keypoint location into a 2-D representation (The annotations include 15 body keypoints location, a unique person id and a head bounding box for each person instance. Page 10 3rd paragraph) and decoding maximum values of the 2-D representation using statistical methods (The targeted heatmap
PNG
media_image6.png
38
44
media_image6.png
Greyscale
for joint k is generated by applying a 2D gaussian (i.e., statistical methods) centered (i.e., maximum values) on the kth joint’s ground truth location. Page 2 7th paragraph); provides the updated trackers (Fig. 2 (e) Shifted tracked pose
PNG
media_image7.png
594
866
media_image7.png
Greyscale
) to the pose estimator for pose estimation (Simple Baselines for Human Pose Estimation and Tracking. Title).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Xiao to utilize a pose estimator in order to perform simultaneous pose detection and tracking.
Regarding claim 16, Lee in the combination teaches a method for detecting people, comprising: capturing a video of the one or more people (
PNG
media_image2.png
572
822
media_image2.png
Greyscale
) to generate a plurality of consecutive frames (In another general aspect, an electronic device includes: a camera configured to generate an input image comprising a plurality of image frames. [0024]), wherein the consecutive frames include an original frame (In response to a target object being determined in a first image frame, a target box 202 corresponding to the target object may be designated (e.g., generated or determined). [0055]) and one or more new frames (Referring to FIG. 3, a plurality of image frames of a search image 300 may be divided into a plurality of sequence groups including a first sequence group 301 and a second sequence group 302. [0058]
PNG
media_image1.png
814
690
media_image1.png
Greyscale
); generating a boundary box around each of the one or more detected people within the original frame and one or more new frames (
PNG
media_image2.png
572
822
media_image2.png
Greyscale
); performing a confidence pre-processing procedure (determining a confidence of the template candidate using a result of comparing a first tracking result determined by the forward object tracking performed on the first image frames and a second tracking result determined by the backward object tracking performed on the first image frames. [0005]) by predicting at least one tracker (Referring to FIG. 1, an object tracking apparatus 100 may output a tracking result 103 based on a template image 101 and a search image 102. [0044]
PNG
media_image3.png
768
612
media_image3.png
Greyscale
) based on the original frame and one or more new frames (For example, the template image 101 (i.e., the original frame) may correspond to one of a plurality of image frames included in an input video file, and the search image 102 (i.e., one or more new frames) may correspond to one or more of the image frames after ( e.g., temporally subsequent to) the image frame corresponding to the template image 101. [0045]), comparing the boundary box and the at least one tracker between each frame (The object tracking apparatus may determine a target box 204 corresponding to the target object in the search region 203 through the regression analysis and generate a tracking result based on box information of the target box 204. [0057]), and identifying a match between the boundary box and the at least one tracker in the original and each of the one or more new frames (In response to a target object being determined in a first image frame, a target box 202 corresponding to the target object may be designated (e.g., generated or determined). [0055]. The object tracking apparatus may determine a target box 204 corresponding to the target object in the search region 203 through the regression analysis and generate a tracking result based on box information of the target box 204. [0057]
PNG
media_image4.png
542
862
media_image4.png
Greyscale
); wherein the one or more matched trackers are evaluated to determine valid detections (For example, the object tracking apparatus may determine the confidence of the template candidate 412 based on a degree of overlap between at least some of corresponding pairs obtained from the first tracking result F, and the second tracking result B. [0067]), valid detections are converted into updated trackers (For example, a difference between an appearance of a target object appearing in first image frames in the first sequence group 301 and an appearance of a target object appearing in second image frames of the second sequence group 302 may occur, and a first template 311 for object tracking in the first sequence group 301 may be updated to a second template 321 for object tracking in the second sequence group 302. [0059]).
Zheng in the combination further teaches validating a matched tracker after persistence for at least a first threshold number (M) of consecutive frames (Optionally, the analyzing the validity of the identification number based on data in the consecutive M frames of images includes: Acquiring data of the identification number in consecutive M frames of images, and determining whether M is within a specified numerical range; wherein the data of the identification number in consecutive M frames of images includes: the identification number of consecutive M frames The coordinates and / or timestamp of the central pixel of the portrait in each frame of the image; If M is greater than a preset minimum value Mmin and less than a preset maximum value Mmax, the identification number is determined to be a valid number, and it is retained in the number statistics list. Page 3 4th paragraph).
Mandal in the combination further teaches retaining an unmatched tracker (For the unmatched tracks T_re-remain after the second association, we put them into T_lost. Page 8 2nd paragraph) for up to a second threshold number (K) of consecutive frames prior to discarding the unmatched tracker (For each track in T_lost , only when it exists for more than a certain number of frames (in paper, this value is 30 frames), we delete it from the tracks T. Page 8 2nd paragraph); wherein the one or more unmatched trackers are evaluated (For the unmatched tracks T_re-remain after the second association, we put them into T_lost. Page 8 2nd paragraph) to determine valid detections (For each track in T_lost , only when it exists for more than a certain number of frames (in paper, this value is 30 frames), we delete it from the tracks T. Page 8 2nd paragraph).
Xiao in the combination further teaches performing a pose estimation (Simple Baselines for Human Pose Estimation and Tracking. Title) on the boundary boxes based on the confidence pre-processing (Similar as in [11], we first drop low-confidence detections, which tends to decrease the mAP metric but increase the MOTA tracking metric. Also, since the tracking metric MOT penalizes false positives equally regardless of the scores, we drop low confidence joints first to generate the result as in [11]. We choose the boxes and joints drop threshold in a data-driven manner on validation set, 0.5 and 0.4 respectively. Page 11 4th paragraph); wherein updated trackers (Fig. 2 (e) Shifted tracked pose
PNG
media_image7.png
594
866
media_image7.png
Greyscale
) are provided to the pose estimator for pose estimation (Simple Baselines for Human Pose Estimation and Tracking. Title).
Claims 3-8 and 15, unamended and are rejected based on the combination of Lee, in view of Zheng, further in view of Mandal, and further in view of Xiao. The grounds of rejection established in the last Office Action is fully incorporated herein.
Method claim 17 is drawn to the method of using the corresponding apparatus claimed in claims 4 and 5. Therefore method claim 17 corresponds to apparatus claims 4 and 5 and is rejected for the same reasons of obviousness as used above.
Method claim 18 is drawn to the method of using the corresponding apparatus claimed in claim 6. Therefore method claim 17 corresponds to apparatus claim 6 and is rejected for the same reasons of obviousness as used above.
Method claim 19 is drawn to the method of using the corresponding apparatus claimed in claims 7 and 8. Therefore method claim 17 corresponds to apparatus claims 7 and 8 and is rejected for the same reasons of obviousness as used above.
Claims 2 and 20, unamended and are rejected based on the combination of Lee (US Patent Publication No.: US 2023/0351610 A1), hereinafter Lee, in view of Zheng (PCT Patent Pub. No.: WO 2020001302 A1), hereinafter Zheng, further in view of Mandal (ByteTrack: Multi-Object Tracking by Associating Every Detection Box, Qure.ai Tech Blog, August 8, 2022, https://blog.qure.ai/notes/byte-track-multi-object-tracking), hereinafter Mandal, further in view of Xiao (Simple Baselines for Human Pose Estimation and Tracking, ECCV 2018), hereinafter Xiao, and further in view of Jun (Monocular Human Depth Estimation via Pose Estimation, IEEE Access (Volume: 9), date of publication November 8, 2021), hereinafter Jun. The ground of rejection established in the last Office Action is fully incorporated herein.
Claims 9-11, unamended and are rejected based on the combination of Lee (US Patent Publication No.: US 2023/0351610 A1), hereinafter Lee, in view of Zheng (PCT Patent Pub. No.: WO 2020001302 A1), hereinafter Zheng, further in view of Mandal (ByteTrack: Multi-Object Tracking by Associating Every Detection Box, Qure.ai Tech Blog, August 8, 2022, https://blog.qure.ai/notes/byte-track-multi-object-tracking), hereinafter Mandal, further in view of Xiao (Simple Baselines for Human Pose Estimation and Tracking, ECCV 2018), hereinafter Xiao, and further in view of Luffca (SORT: Simple Online and Realtime Tracking, https://www.luffca.com/2023/04/multiple-object-tracking-sort/, 2023-04-08), hereinafter Luffca. The ground of rejection established in the last Office Action is fully incorporated herein.
Claims 12-14, unamended and are rejected based on the combination of Lee (US Patent Publication No.: US 2023/0351610 A1), hereinafter Lee, in view of Zheng (PCT Patent Pub. No.: WO 2020001302 A1), hereinafter Zheng, further in view of Mandal (ByteTrack: Multi-Object Tracking by Associating Every Detection Box, Qure.ai Tech Blog, August 8, 2022, https://blog.qure.ai/notes/byte-track-multi-object-tracking), hereinafter Mandal, further in view of Xiao (Simple Baselines for Human Pose Estimation and Tracking, ECCV 2018), hereinafter Xiao, and further in view of Sun (Deep High-Resolution Representation Learning for Human Pose Estimation, Proceedings of the IEEE/CVF conference, 2019), hereinafter Sun. The ground of rejection established in the last Office Action is fully incorporated herein.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEI ZHAO whose telephone number is (703)756-1922. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VU LE can be reached at (571)272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LEI ZHAO/Examiner, Art Unit 2668
/VU LE/Supervisory Patent Examiner, Art Unit 2668
