Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 10 objected to because of the following informalities:
“The method of claim 9, wherein the CNN uses no more then three stabilized frames to detect the one or more objects.” Should read “The method of claim 9, wherein the CNN uses no more than three stabilized frames to detect the one or more objects.”
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3-7, 11, 17-18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Chen (United States Patent Application Publication US 2021/0366126 A1) in view of Schlattmann et. al. (United States Patent Application Publication US 2017/0111585 A1).
Regarding claim 1, Chen discloses a method for tracking objects in a video comprising: receiving, by a processor, one or more frames of a video; dividing, by the processor, each frame of a video into non-overlapping sub-images; extracting, by the processor, keypoints from each sub-image (Chen: [0113]: The sample annotation information corresponding to the video frame sample may be sample key point position information for annotating key points in the video frame sample, and sample region position information for annotating a target object in the video frame sample. The key points in the video frame sample are key points used for determining the target object, and the quantity of the key points may be, for example, 3 or 4.); estimating, by the processor, an affine transformation by analyzing the resulting keypoint matches (Chen: [0119]: The computer device may determine a region box that includes the target object based on the key points in each video frame sample, and expand the region box by a range to obtain an ROI for each video frame sample. [0121]: Further, the computer device can calculate a reflection similarity according to a size of each video frame sample, key point position information, the size of the template, and the template key point position information to obtain a transformation matrix. The transformation matrix includes affine transformation information.[0089]: The revising historical affine transformation information is adjusting historical affine transformation parameters to obtain updated affine transformation parameters, which is based on the feature map; [0116]: Obtaining the standard affine transformation information corresponding to the video frame sample by calculation according to the sample key point position information and the template key point position information.); and align, by the processor, the prior frame to the current frame using the affine transformation (Chen abstract, [0007]: The method includes obtaining a current frame and historical affine transformation information of a previous video frame in a video frame sequence; perform affine transformation on the current frame according to the historical affine transformation information to obtain a candidate region image corresponding to the current frame; performing feature extraction on the candidate region image to obtain a feature map corresponding to the candidate region image; performing semantic segmentation based on the feature map to obtain a segmentation result corresponding to a target in the current frame; and revising the historical affine transformation information according to the feature map to obtain updated affine transformation information, and using the updated affine transformation information as historical affine transformation information corresponding to a subsequent video frame in the video frame sequence).
However, Chen fails to disclose wherein the keypoint matches occur only between keypoints within the same sub-image of a prior frame and a current frame; finding, by the processor, keypoint matches between the keypoints and one or more predetermined features.
Schlattmann et. al. teaches wherein the keypoint matches occur only between keypoints within the same sub-image of a prior frame and a current frame; finding, by the processor, keypoint matches between the keypoints and one or more predetermined features (Schlattmann et. al. [0038]: The salient feature points may be matched between the frames of the sequence. For example, point 110 will be matched with the point representing the front left corner of car 108 in further frames.). This is important to the claimed invention because it serves as the mechanism for which objects are matched between moving video frames. The matching algorithm the Schlattmann et. al. teaches allows objects to be accurately tracked between frames. Thus, it would have been obvious to one skilled in the art prior to the effective filing date of the claimed invention to have combined the teachings of Chen and Schlattmann et. al. so that these features are seen in the solution of the claimed invention.
Regarding claim 3, Chen and Schlattmann et. al. disclose the method of claim 1. Schlattmann et. al. further discloses wherein the keypoints are detected by a keypoint detection algorithm (Schlattmann et. al. [0035]: On step 204, salient feature points are obtained for the frame. The term salient feature point refers to an outstanding or noticeable point in the frame, such as a corner of an object, an edge, or the like. The points may be identified, in a non-limiting example, by applying the Shi-Thomasi corner detector algorithm, or by any other corner detection image processing algorithm).
This is important to the claimed invention because the keypoint is a specific point on the object of interest, as shown in Figure 4A of the specification of the current application. This is important to the claimed invention because the keypoint must be relatively noticeable within the frame of the video that is captured. Thus, it would have been obvious to one skilled in the art prior to the effective filing date of the claimed invention to have combined the teachings of Chen and Schlattmann et. al. to use an algorithm for keypoint detection for greater accuracy and precision.
Regarding claim 4, Chen and Schlattmann et. al. disclose the method of claim 1, and Schlattmann et. al. further disclose wherein the keypoints represented one or more distinctive features within the sub-image (Schlattmann et. al. [0035]: On step 204, salient feature points are obtained for the frame. The term salient feature point refers to an outstanding or noticeable point in the frame, such as a corner of an object, an edge, or the like.).
Regarding claim 5, Chen and Schlattmann et. al. disclose the method of claim 1, and Schlattmann et. al. further disclose wherein the method further comprises: generating, by the processor upon extracting the keypoints, one or more keypoint descriptors by extracting raw pixel information within a patch surrounding each keypoint (Schlattmann et. al. [0025]-[0026]: The system and method receive a sequence of images, and start by identifying or selecting a group of salient feature points (also referred to as “salient points”, “feature points” or “points”) within the image sequence. [0037]: The predetermined threshold may relate to an absolute number of points per frame or to a number of points relative to the number of points, or pixels, in the frame. Additional points are added for providing better coverage of the frame. Fig. 1).
Regarding claim 6, Chen and Schlattmann et. al. disclose the method of claim 1, and Schlattmann et. al. further discloses wherein the affine transformation compensates for a camera motion between the prior frame and the current frame (Schlattmann et. al. [0014]: dropping salient feature points associated with objects moving in shaking movements; computing a transformation between pairs of consecutive frames from amongst the frames, based upon a multiplicity of non-dropped salient feature points).
Regarding claim 7, Schlattmann et. al. discloses the method of claim 6, wherein the alignment of the prior frame with the current frame stabilizes the video by compensating for camera motion between the current frame and prior frame (Schlattmann et. al. [0014]: determining a center position for the frames based upon the multiplicity of transformations; determining a stabilizing transformation from a current frame to the center position; and applying the stabilizing transformation to the current frame to obtain a stabilized frame.).
Regarding claim 11, which recites a system and processor configured to: receiving one or more frames of a video, which corresponds to the method of claim 1, which the rejection analysis is incorporated herein. Schlattmann et. al. further discloses a video stabilization system used herein (Schlattmann et. al. Abstract: a method, system, and computer program product for stabilizing frames).
Regarding claim 17, Chen and Schlattmann et. al. disclose the system of claim 11, and Schlattmann et. al. further disclose wherein the processor is further configured to store the one or more frames in a database (Schlattmann et. al. [0081]: Processor may comprise communication with image source component for communicating with an image source, such as a storage device storing images, a capture device, or the like. In some embodiments, the frames may be stored on storage device.).
Regarding claim 18, Schlattmann et. al. discloses the system of claim 17, wherein the processor is further configured to retrieve a predetermined amount of frames from the data storage unit for processing (Schlattmann et. al. [0081]: Processor may comprise communication with image source component for communicating with an image source, such as a storage device storing images, a capture device, or the like. In some embodiments, the frames may be stored on storage device.).
Regarding claim 20, which is a non-transitory computer readable medium containing computer executable instructions that, when executed by a wearable device comprising a processor, configure the computer hardware arrangement to perform procedures comprising: the method of claim 1, which the rejection analysis is incorporated herein. Chen discloses an apparatus that can be configured to obtain a video frame sample.
Claim(s) 2 is rejected under 35 U.S.C. 103 as being unpatentable over Chen (United States Patent Application Publication US 2021/0366126 A1) in view of Schlattmann et. al. (United States Patent Application Publication US 2017/0111585 A1) as applied to claim 1 above, and further in view of Grundmann et. al. (United States Patent Application Publication US 2017/0195575 A1).
Regarding claim 2, Chen and Schlattmann et. al. disclose the method of claim 1. However, Chen and Schlattmann et. al. fail to disclose wherein the sub-images are non-overlapping and arranged in a grid.
Grundmann et. al. teaches wherein the sub-images are non-overlapping and arranged in a grid (Grundmann et. al. [0045]: Further, in generating the tracked features the tracking module may divide the frame into multiple layers of grides having different sizes (e.g. 4x4 or 16 grids total, 8x8 grids, and 16x16 grides. The gradient threshold for what may be considered a tracked feature may set on a per-grid basis to normalize the number of tracked features generated per cell of the grid.).
This is important to the claimed invention because it helps to identify the key features based on the grid location. Thus, it would have been obvious to one skilled in the art prior to the effective filing date of the claimed invention to have combined the teachings of Chen, Schlattmann et. al. and Grundmann et. al. so that the grid is part of the solution of the claimed invention.
Claim(s) 8-10, 12-16 are rejected under 35 U.S.C. 103 as being unpatentable over Chen (United States Patent Application Publication US 2021/0366126 A1) in view of Schlattmann et. al. (United States Patent Application Publication US 2017/0111585 A1) as applied to claim 1 above, and further in view of Khadloya et. al. (United State Patent Application Publication US 2022/0301275 A1).
Regarding claim 8, Chen and Schlattmann et. al. disclose the method of claim 1. However, Chen and Schlattmann et. al. fail to disclose wherein the method further comprises: detecting, by the processor, one or more objects present in one or more stabilized frames via a trained objection detection algorithm
Khadloya et. al. teaches wherein the method further comprises: detecting, by the processor, one or more objects present in one or more stabilized frames via a trained objection detection algorithm (Khadloya et. al. [0082]-[0085]: The matching algorithm is designed in the following way: for each object on the previous frame the new object position on the current frame is calculated. Given the predicted position, a candidate short-list of detected objects which have sufficient overlap with the predicted position are generated. From there, compute visual similarities and reject all candidates which have a visual similarity below a threshold.).
This is important to the claimed invention because the object is larger than the keypoint, and thus the technique for detecting objects is distinct from the keypoint detection algorithm. This involves the object of interest for each frame of the video. Thus, it would have been obvious for one skilled in the art prior to the effective filing date of the claimed invention to have combined the teachings of Chen, Schlattmann et. al., and Khadloya et. al. so that the algorithm for object detection is included in the solution of the claimed invention.
Regarding claim 9, Khadloya et. al. discloses the method of claim 8, wherein the objection detection algorithm is a convolutional neural network (CNN) (Khadloya et. al. [0085]: For the MD-to-MO cases, the matching algorithm is designed in the following way: for each object on the previous frame the new object position on the current frame is calculated. Depending on the hardware it’s accomplished by velocity vector, cam/median shift, optical flow or CNN-based predictions).
Regarding claim 10, Khadloya et. al. discloses the method of claim 9, wherein the CNN uses no more than three stabilized frames to detect the one or more objects (Khadloya et. al. [0087]: Fig. 9-10, the reference object detection 902 is achieved by performing semantic segmentation of the reference object using any kind of object detection CNN, wherein said CNN is trained on a dataset of images with object annotated using at least one of the bounding box or pixel-wise mask).
Regarding claim 12, Chen and Schlattmann et. al. disclose the system of claim 11. However, Schlattmann et. al. and Chen fail to disclose wherein the processor is further configured to detect one or more objects present in one or more stabilized frames via a trained objection detection algorithm.
Khadloya et. al. teaches wherein the processor is further configured to detect one or more objects present in one or more stabilized frames via a trained objection detection algorithm (Khadloya et. al. [0082]-[0085]: The matching algorithm is designed in the following way: for each object on the previous frame the new object position on the current frame is calculated. Given the predicted position, a candidate short-list of detected objects which have sufficient overlap with the predicted position are generated. From there, compute visual similarities and reject all candidates which have a visual similarity below a threshold.).
This is important to the claimed invention because the object is larger than the keypoint, and thus the technique for detecting objects is distinct from the keypoint detection algorithm. This involves the object of interest for each frame of the video. Thus, it would have been obvious for one skilled in the art prior to the effective filing date of the claimed invention to have combined the teachings of Chen, Schlattmann et. al., and Khadloya et. al. so that the algorithm for object detection is included in the solution of the claimed invention.
Regarding claim 13, Chen, Schlattmann et.al. and Khadloya et. al. disclose the system of claim 12, and Khadloya et. al. further discloses wherein the processor is further configured to generate one or more bounding boxes around the one or more objects (Khadloya et. al. [0052]: The tracking unit is configured to track the one or more candidate objects using one of the following techniques: 1) using the bounding box estimation data of the second (current) frame and the first (previous) frame and formulating a correspondence based on a cost matrix, wherein the bounding box data is computed by the object detection system 302 a feature/model-based tracking).
Regarding claim 14, Chen, Schlattmann et. al., and Khadloya et. al. disclose the system of claim 13, and Khadloya et. al. further discloses wherein the objects include at least one selected from the group of a human being, vehicle, or machinery (Khadloya et. al. [0002]: An object-to-be tracked could range from a person, vehicle, animal, building, an article to any other similar object).
Regarding claim 15, Chen, Schlattmann et. al., and Khadloya et. al. disclose the system of claim 14, and Khadloya et. al. further discloses wherein the objection detection algorithm is a convolutional neural network (CNN) (Khadloya et. al. [0085]: For the MD-to-MO cases, the matching algorithm is designed in the following way: for each object on the previous frame the new object position on the current frame is calculated. Depending on the hardware it’s accomplished by velocity vector, cam/median shift, optical flow or CNN-based predictions).
Regarding claim 16, Chen, Schlattmann et. al., and Khadloya et. al. disclose the system of claim 15, and Khadloya et. al. further discloses wherein the CNN uses no more than three stabilized frames to detect the one or more objects (Khadloya et. al. [0087]: Fig. 9-10, the reference object detection 902 is achieved by performing semantic segmentation of the reference object using any kind of object detection CNN, wherein said CNN is trained on a dataset of images with object annotated using at least one of the bounding box or pixel-wise mask).
Claim(s) 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chen (United States Patent Application Publication Application US 2021/0366126 A1) in view of Schlattmann et. al. (United States Patent Application Publication US 2017/0111585 A1) as applied to claim 11 above, and further in view of Karpenko (United States Patent Application Publication US 2017/0180647 A1).
Regarding claim 19, Chen and Schlattmann et. al. disclose the system of claim 11. However, Schlattmann et. al. and Chen fail to disclose wherein the processor is configured receive camera movement data from an inertial measurement unit (IMU).
Karpenko teaches wherein the processor is configured receive camera movement data from an inertial measurement unit (IMU) (Karpenko [0040]: Furthermore, inexpensive MEMS gyroscopes may be implemented to measure camera motion directly. Inertial measurement units (IMUs) may be used for image de-blurring and for aiding a KLT feature tracker.). This is important to the claimed invention because the videos captured are obtained while moving. Thus, it would have been obvious to one skilled in the art prior to the effective filing date of the claimed invention to have combined the teachings of Schlattmann et. al., Chen, and Karpenko so that the IMU unit is included in the solution of the claimed invention.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JESSICA YIFANG LIN whose telephone number is (571)272-6435. The examiner can normally be reached M-F 7:00am-6:15pm, with optional day off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at 571-272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JESSICA YIFANG LIN/Examiner, Art Unit 2668 June 12, 2026
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights