DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The preliminary amendments filed on 12/26/2024 has been entered.
Claim 9 was amended.
Claims 15-20 from Claims Filed 12/05/2024 were cancelled. Newly added 15 and 16 in claims Filed 12/26/2024 replaces the claims 15 and 16 from 12/05/2024.
Claims 21-24 were newly added.
Claims 1-16, 21-24 remain pending in the application.
Claim Objections
Claims 4, 12, 24 objected to because of the following informalities: typos. The claims state “pointes”, which are most likely meant to be read as “points”. Appropriate correction is required.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, 6, 11, 14, 23 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 3, 6, 11, 14, 23 recites the limitation "the second feature group" in, as an example, second limitation paragraph for claim 3. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 2, 9, 10, 21, 22 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ray (“Multi-Mask Based Stabilization of Turbulence Degraded Videos Containing Moving Objects”, 2019).
Regarding claims 1, 9, 21, Ray teaches A method for tracking a moving object (Ray, Abstract: “This mask is employed to properly detect the moving objects”. Moving object detection is being interpreted as involving “tracking a moving object”) in a sequence of image frames (Ray, Abstract: “videos”), comprising:
obtaining a first filtered image (Ray, see nearest image below, “difference image“ is being interpreted as involving “a first filtered image”) by filtering out the moving object (Ray, see nearest image below, “the pixels having moving objects have higher intensity differences than the other parts of frames”) in the i-th image frame (Ray, see nearest image below, “input frame” is being interpreted as involving the i-th image frame) among the sequence of image frames (Ray, see nearest image below, “videos”), wherein the moving object (Ray, see nearest image below, “the pixels having moving objects have higher intensity differences than the other parts of frames”) is an object in the i-th image frame that has a positional change relative to the object in the (i-1)-th image frame among the sequence of image frames (Ray, see nearest image below, “a mask is generated using the difference image of each frame with respect to the background frame.” “Difference image” is being interpreted as involving an object in the i-th image frame that has a positional change relative to the object in the (i-1)-th image frame. “Difference image” is also being interpreted as involving “sequence of image frames”), and i is an integer larger than 1 (Ray, pg 64, column 2, last two paragraphs:
PNG
media_image1.png
668
716
media_image1.png
Greyscale
. “videos” is being interpreted as involving “i is an integer larger than 1”);
determining, based on the first filtered image, a pixel mapping relationship between the (i- 1)-th image frame and the i-th image frame (Ray, see nearest image above, “the pixels having moving objects have higher intensity differences than the other parts of frames”; “difference image”, which is being interpreted as involving “a pixel mapping relationship” between the (i- 1)-th image frame and the i-th image frame”); and
tracking, according to the pixel mapping relationship, the moving object (Ray, Abstract: “This mask is employed to properly detect the moving objects”. “Mask” is being interpreted as involving “the pixel mapping relationship”. Moving object detection is being interpreted as involving tracking the moving object).
Regarding claim 2, Ray teaches The method of claim 1, obtaining the first filtered image by filtering out the moving object in the i-th image frame comprises:
obtaining a first detection frame (Ray, Section II, ¶1: “In the first step, a mask is generated using the difference image of each frame with respect to the background frame”. “a mask is generated using the difference image” is being interpreted as involving “a first detection frame”) by performing an object detection on the moving object (Ray, Section II, ¶1: “In the first step, a mask is generated using the difference image of each frame with respect to the background frame”. “a mask is generated using the difference image” is being interpreted as involving “object detection on the moving object” ) in the (i-1)-th image frame (Ray, Section II, ¶1: “In the first step, a mask is generated using the difference image of each frame with respect to the background frame”. The frame involved with creating the “background frame” is being interpreted as the “(i-1)-th image frame”);
obtaining a second detection frame (Ray, Section II, ¶1: “In the first step, a mask is generated using the difference image of each frame with respect to the background frame”. “a mask is generated using the difference image” is being interpreted as involving “a second detection frame”) by performing the object detection (Ray, Section II, ¶1: “In the first step, a mask is generated using the difference image of each frame with respect to the background frame”. “a mask is generated using the difference image” is being interpreted as involving “object detection on the moving object” ) on the moving object in the i-th image frame (Ray, Section II, ¶1: “For each input frame”, which is being interpreted as involving “the i-th image frame” that has a moving object); and
obtaining the first filtered image (Ray, Section II, ¶1: “In the first step, a mask is generated using the difference image of each frame with respect to the background frame”. “Difference image” is being interpreted as involving “the first filtered image”) by filtering out (Ray, Section II, ¶1: “a mask”, which is being interpreted as involving “filtering out”) the moving object (Ray, pg 64, column 2, paragraph before Section II, “the pixels having moving objects have higher intensity differences than the other parts of frames”. ) in the i-th image frame (Ray, Section II, ¶1: “for each input frame”, which is being interpreted as involving the i-th image frame) based on the first detection frame and the second detection frame (Ray, Section II, ¶1, “difference image” is being interpreted as involving first and second detection frame).
Claims 10, 22 are rejected using the same rationale as applied to claim 2 discussed above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3, 4, 11, 12, 23, 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ray, in view of Kalal (“Tracking-Learning-Detection”, 2012).
Regarding claim 3, Ray teaches The method of claim 2, obtaining the first filtered image by filtering out the moving object in the i-th image frame based on the first detection frame and the second detection frame comprises:
obtaining first feature groups (Ray, see nearest image below, “generate a mask” is being interpreted as involving “first feature groups”) each including a first feature point (Ray, see nearest image below, “high-level threshold” is being interpreted as involving the “first feature point”) and a second feature point (Ray, see nearest image below, “low-level” threshold is being interpreted as involving “the second feature point”), wherein the first feature point is one feature point obtained (Ray, see nearest image below, “high-level threshold” is being interpreted as involving the “first feature point”) by performing a feature extraction processing (Ray, see nearest image below, “difference image” is being interpreted as involving “feature extraction processing”) on the (i-1)-th image frame (Ray, see nearest image below, “background image”), and the second feature point is one feature point obtained (Ray, see nearest image below, “low-level” threshold is being interpreted as involving “the second feature point”) by performing the feature extraction processing on the i-th image frame (Ray, pg 65, column 1, line before equation 1 to paragraph afterwards, reproduced below:
PNG
media_image2.png
376
706
media_image2.png
Greyscale
); and
However, Ray does not appear to explicitly teach taking each of the first feature groups as the second feature group, in response to the first feature point in the first feature group being not within the first detection frame and the second feature point in the first feature group being not within the second detection frame, wherein the first filtered image is composed of the second feature points in the second feature group.
Pertaining to the same field of endeavor, Kalal teaches
taking each of the first feature groups (Kalal, see nearest image below and Section 7, the features, from example feature space X, of the labeled set is being interpreted as involving first feature groups) as the second feature group (Kalal, see nearest image below and Section 7, “unlabeled set”, or when the object changes appearance or moves out of camera view, is being interpreted as involving second feature group.), in response to the first feature point in the first feature group being not within the first detection frame (Kalal, see nearest image below and Section 7, when the object goes out of camera view for at least two frames, the first feature point in the first feature group from feature space X are not detected) and the second feature point in the first feature group being not within the second detection frame (Kalal, see nearest image below and Section 7, when the object goes out of camera view for at least two frames, the second feature point in the second feature group from feature space X are not detected), wherein the first filtered image (Kalal, pg 1416, Section 5.3.3: “After filtering the patches by the variance filter and the ensemble classifier”, which is being interpreted as involving the first filtered image) is composed of the second feature points in the second feature group (Kalal, pg 1412, Section 4.1, ¶1-2, reproduced below:
PNG
media_image3.png
674
710
media_image3.png
Greyscale
; pg 1419, Section 7, ¶1, “In this paper, we studied the problem of tracking of an unknown object in a video stream, where the object changes appearance frequently moves in and out of the camera view.” The unlabeled set or when the object appears within camera view again are being interpreted as examples of second feature points, which would be used to learn or detect the object, in the second feature group, or from the unlabeled set [or when the object appears again]).
Ray and Kalal are considered to be analogous art because they are directed to moving object detection. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for video stabilization and object detection (as taught by Ray) to include “taking each of the first feature groups as the second feature group, in response to the first feature point in the first feature group being not within the first detection frame and the second feature point in the first feature group being not within the second detection frame, wherein the first filtered image is composed of the second feature points in the second feature group” (as taught by Kalal) because the combination provides an improvement to object tracking performance for objects moving in and out of camera view (Kalal, Abstract, Section 7).
Regarding claim 4, Ray teaches The method of claim 3, obtaining the first feature groups comprises:
obtaining the first feature points (Kalal, see nearest image below, feature space is being interpreted as involving first feature points) and first descriptors (Kalal, see nearest image below, feature space is being interpreted as involving first descriptors) corresponding to the first feature pointes (Kalal, see nearest image below, feature space is being interpreted as involving first feature points), by performing the feature extraction processing on the (i-1)-th image frame (Kalal, pg 1412, Section 4.1, ¶1-2, reproduced below:
PNG
media_image3.png
674
710
media_image3.png
Greyscale
; pg 1419, Section 7, ¶1, “In this paper, we studied the problem of tracking of an unknown object in a video stream, where the object changes appearance frequently moves in and out of the camera view.” “Feature space” shows feature extraction occurs. The labeled set is being interpreted as involving “(i-1)-th image frame”);
obtaining the second feature points (Kalal, see nearest image above, feature space is being interpreted as involving second feature points; a non-limiting example could be other frames in the labeled set or the unlabeled set) obtained by performing the feature extraction processing (Kalal, see nearest image above, feature space shows that “feature extraction” occurs) on the i-th image frame (Kalal, see nearest image above, a non-limiting example could be other frames in the labeled set or the unlabeled set for the i-th image frame) and a second descriptor corresponding to each of the second feature points (Kalal, see nearest image above, feature space for other image frames in the labeled set or the unlabeled set are being interpreted as involving “a second descriptor corresponding to each of the second feature points”); and
obtaining the first feature groups by matching each of the first feature points and each of the second feature points (Kalal, see nearest image above, “The task of P-N learning is to learn a classifier…from labeled set… and bootstrap its performance by the unlabeled set”. “Bootstrap” is being interpreted as involving “matching”. “Labeled set” is being interpreted as involving “first feature groups” and “first feature points”. Unlabeled set is being interpreted as involving second feature points, as a non-limiting example) based on the corresponding first descriptor and the corresponding second descriptor (Kalal, see nearest image above, “feature space” is being interpreted as involving first and second descriptors).
Claims 11, 23 is rejected using the same rationale as applied to claim 3 discussed above.
Claims 12, 24 is rejected using the same rationale as applied to claim 4 discussed above.
Claim(s) 5, 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ray, as modified by Kalal, in view of Tsagkatakis (“Online Distance Metric Learning for Object Tracking”, 2011).
Regarding claim 5, Ray teaches The method of claim 4, obtaining the first feature groups by matching each of the first feature points and each of the second feature points based on the corresponding first descriptor and the corresponding second descriptor comprises:
However, Ray and Kalal do not appear to explicitly teach calculating feature distance and finding minimum feature distance. Though both do teach feature extraction. Kalal teaches replacement through updating the model.
Pertaining to the same field of endeavor, Tsagkatakis teaches
calculating a feature distance (Tsagkatakis, pg 1815, Section B, ¶2: “Second, in addition to modeling the appearance of the object and the background, we also obtain the distances between all examples from the same class and the distances between all the examples from different classes.” Which is being interpreted to involve features) between the first descriptor of each of the first feature points and the second descriptor of each of the second feature points (Tsagkatakis, pg 1815, Section B, ¶2: “Second, in addition to modeling the appearance of the object and the background, we also obtain the distances between all examples from the same class and the distances between all the examples from different classes.” As a non-limiting example, feature distance between same classes would involve first and second feature descriptors and points as; image A would have first feature point and descriptor, image B would have first feature point and descriptor, then the feature distance between could be calculated);
generating a third feature group (Tsagkatakis, pg 1815, Section B, ¶2: “We denote… u… the minimum allowable distance”. “u” is being interpreted as a non-limiting example of a third feature group) with the second feature point (Tsagkatakis, pg 1815, Section B, ¶2: “Second, in addition to modeling the appearance of the object and the background, we also obtain the distances between all examples from the same class and the distances between all the examples from different classes.” The distance between features would be captured and would involve the second feature point) and the first feature point corresponding to the minimum feature distance (Tsagkatakis, pg 1815, Section B, ¶2: “We denote… u… the minimum allowable distance”) among the calculated feature distances (Tsagkatakis, pg 1815, Section B, ¶2: “Second, in addition to modeling the appearance of the object and the background, we also obtain the distances between all examples from the same class and the distances between all the examples from different classes.” Which shows non-limiting examples of calculated feature distances); and
taking the third feature group as the first feature group (Tsagkatakis , pg 1816, Section C, ¶2: “If this assertion holds, then the template element that achieves the smallest distance (most similar) to the object’s current appearance is replaced with the new appearance.” “The smallest distance” is being interpreted as the third feature group. The replacement is being interpreted as taking the third feature group as the first feature group), in response to the feature distance corresponding to the third feature group being larger than a preset threshold (Tsagkatakis , pg 1816, Section C, ¶2: “A new appearance is considered a candidate for the template library if S_t > S_t−1”. The greater than symbol is being interpreted as “larger than a preset threshold”).
Ray, Kalal, and Tsagkatakis are considered to be analogous art because they are directed to object detection. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for video stabilization and object detection with feature extraction and replacement (as taught by Ray, modified by Kalal) to include calculating feature distance and finding minimum feature distance (as taught by Tsagkatakis) because the combination provides an improvement to tracking performance (Tsagkatakis, Abstract). Further, PHOSITA would understand tracking does not happen magically, features of the object are required to track the object. Additionally, feature distance is common knowledge, used in such object detection methods as template matching to measure the distance between candidate and template in the known library.
Claim 13 is rejected using the same rationale as applied to claim 5 discussed above.
Claim(s) 6, 7, 14, 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ray, as modified by Kalal, in view of Liu (“Geometric affine transformation estimation via correlation filter for visual tracking”, 2016).
Regarding claim 6, Ray teaches The method of claim 3, determining, based on the first filtered image, the pixel mapping relationship between the (i-1)-th image frame and the i-th image frame comprises:
… based on the second feature group (Kalal, see claim 3 image above and Section 7, “unlabeled set”, or when the object changes appearance or moves out of camera view, is being interpreted as involving second feature group) corresponding to the first filtered image (Kalal, pg 1416, Section 5.3.3: “After filtering the patches by the variance filter and the ensemble classifier”, which is being interpreted as involving the first filtered image); and
However, Ray and Kalal do not appear to explicitly teach overdetermined equations. Though, PHOSITA image transformation would be required to stabilize an image. Further, overdetermined system of equations is common mathematical knowledge.
Pertaining to the same field of endeavor, Liu teaches
constructing an overdetermined system (Liu, pg 112, column 2, last paragraph: “to solve a linear overdetermined equation system in order to estimate the affine matrix G”) …
calculating a transformation matrix by solving the overdetermined system (Liu, pg 112, column 2, last paragraph: “to solve a linear overdetermined equation system in order to estimate the affine matrix G”), wherein the transformation matrix (Liu, pg 112, column 2, last paragraph: “to solve a linear overdetermined equation system in order to estimate the affine matrix G”) is for representing the pixel mapping relationship between the (i-1)-th image frame and the i-th image frame (Liu, pg 112, Section 3.2, ¶1, “After stable patches obtained, how to estimate the object's affine transformation matrix is a deliberate problem”. “Transformation” shows there is a pixel mapping relationship to allow for the transformation to occur. “Stable patches” are being interpreted as involving “(i-1)-th image frame and the i-th image frame”, as a non-limiting example).
Ray, Kalal, and Liu are considered to be analogous art because they are directed to moving object detection. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for video stabilization and object detection with feature extraction and replacement (as taught by Ray, modified by Kalal) to include overdetermined system of equations (as taught by Liu) because the combination provides an improvement to tracking performance (Liu, Abstract).
Regarding claim 7, Ray teaches The method of claim 6, tracking, according to the pixel mapping relationship, the moving object comprises:
obtaining a first transformed image (Liu, pg 111, Figure 1, which shows “Affine matrix” leads to the “first transformed image” on the right ) by transforming the i-th image frame (Liu, pg 111, Figure 1, which shows the image before “Affine matrix” being interpreted as i-th image frame) based on the transformation matrix (Liu, pg 111, Figure 1, “affine matrix” box is being interpreted as “transformation matrix”);
obtaining a third detection frame (Liu, pg 111, Figure 1, which shows “Affine matrix” leads to the “first transformed image” on the right. The bounding box shows “the third detection frame”) by transforming the second detection frame (Liu, Figure 1, “stable patches” are being interpreted as involving “second detection frame” as the detected faces are used) based on the transformation matrix (Liu, pg 111, Figure 1, “Affine Matrix” is being interpreted as involving “transformation matrix”); and
tracking the moving object based on the first transformed image (Liu, pg 111, Figure 1, which shows “Affine matrix” leads to the “first transformed image” on the right, which is eventually passed to the KCF tracking for tracking) and the third detection frame (Liu, pg 111, Figure 1, which shows “Affine matrix” leads to the “first transformed image” on the right. The bounding box shows “the third detection frame”).
Claim 14 is rejected using the same rationale as applied to claim 6 discussed above.
Claim 15 is rejected using the same rationale as applied to claim 7 discussed above.
Claim(s) 8, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ray, in view of Wang (“Real-time Tracking with Stabilized Frame”, 2020).
Regarding claim 8, Ray teaches The method of claim 2, obtaining the first detection frame by performing the object detection on the moving object in the (i-1)-th image frame comprises:
However, Ray does not appear to explicitly teach trained detection model.
Pertaining to the same field of endeavor, Wang teaches
obtaining a trained detection model for detecting the moving objects in an image (Wang, pg 4433, Figure 2, reproduced below:
PNG
media_image4.png
353
1149
media_image4.png
Greyscale
PNG
media_image5.png
148
1282
media_image5.png
Greyscale
. “SiamMask” is being interpreted as involving “a trained detection model”. The “Predicted output” shows “detecting the moving objects in an image”; further shown by the mask and bounding box); and
obtaining the first detection frame by inputting the (i-1)-th image frame into the detection model (Wang, see nearest image above, “Predicted Output” and “Inverse Transform” show that a “first detection frame” is created. The “Raw image” and “transformed storage” are being interpreted as involving “the (i-1)-th image frame” being input into the SiamMask detection model).
Ray and Wang are considered to be analogous art because they are directed to video stabilization and object detection. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method and system for video stabilization and object detection (as taught by Ray) to include a trained detection model (as taught by Wang) because the combination provides an improvement to tracking performance (Wang, Abstract).
Claim 16 is rejected using the same rationale as applied to claim 8 discussed above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Runz et al ("MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects”, 2018) discloses object tracking with Visual SLAM (the camera moves).
Rahmaniar et al (“Real-Time Detection and Recognition of Multiple Moving Objects for Aerial Surveillance”, 2019) discloses object detection with jittery camera (the drone camera moves).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHNNY B DUONG whose telephone number is (571)272-1358. The examiner can normally be reached Monday - Thursday 10a-9p (ET).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.B.D./Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
