DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on June 12, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Tao, Li, Xueting Wang, and Toshihiko Yamasaki. "Self-supervised video representation learning using inter-intra contrastive framework." Proceedings of the 28th ACM international conference on multimedia. 2020. (hereinafter Tao), and further in view of U.S. Patent No. 11,526,996 to Rubinstein et al. (hereinafter Rubinstein).
Regarding independent claim 1, Tao discloses A video representation self-supervised contrastive learning method (abstract, “We propose a self-supervised method to learn feature representations from videos. A standard approach in traditional self-supervised methods uses positive-negative data pairs to train with contrastive learning strategy. In such a case, different modalities of the same video are treated as positives and video clips from a different video are treated as negatives. Because the spatio-temporal information is important for video representation, we extend the negative sam ples by introducing intra-negative samples, which are transformed from the same anchor video by breaking temporal relations in video clips. With the proposed Inter-Intra Contrastive (IIC) frame work, we can train spatio-temporal convolutional networks to learn video representations.”), comprising:
Tao fails to explicitly disclose as further recited. However, Rubinstein discloses calculating, according to optical flow information corresponding to each video frame of a video clip, a motion amplitude map corresponding to each video frame of the video clip (column 1, line 6, “It is advantageous in a variety of applications to detect and/or magnify motion that is present in a video. For example, it could be beneficial to magnify motion in a video of a sleeping child in order to verify that the child is breathing and/or to detect a rate of breathing of the child. In order to magnify and/or detect motion in a video, a variety of techniques can be applied. For example, an optical flow map could be determined by comparing different images of the video. Such an optical flow map could then be used to magnify motion within the video, e.g., by distorting the image frames of the video to enhance motion represented in the optical flow map.;” column 13, line 59, “FIG. 4B shows a “heat map” of motion magnitude within such generated image phase information. Thus, the “heat map” shows increased motion magnitude at the frequency of the cardiovascular pulse at the area, within the video stream, that corresponds to the neck of the person depicted.”);
determining, according to motion amplitude maps corresponding to the video frames of the video clip, motion information corresponding to the video clip (column 13, line 59, “FIG. 4B shows a “heat map” of motion magnitude within such generated image phase information. Thus, the “heat map” shows increased motion magnitude at the frequency of the cardiovascular pulse at the area, within the video stream, that corresponds to the neck of the person depicted.”).
Tao is directed toward, “a self-supervised method to learn feature representations from videos (abstract).” Rubinstein is directed toward, “Example embodiments allow for fast, efficient motion-magnification of video streams by decomposing image frames of the video stream into local phase information at multiple spatial scales and/or orientations (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Tao and Rubinstein are directed toward similar methods of endeavor of video processing. Further, Rubinstein allows for the analysis of “motion-magnified video stream (abstract).” One of ordinary skill in the art before the effective filing date of the claimed invention would be easily aware motion detection can be more difficult at very small levels; thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Rubinstein in order to detect motion on much smaller levels than standard video processing methods to output accurate motion details.
Tao and Rubinstein in the combination fail to explicitly disclose performing, according to a sequence of video clips and the motion information corresponding to each video clip, video representation self-supervised contrastive learning as claimed. However, Tao does disclose methods of utilizing self-supervised contrasted learning in the abstract, “Because the spatio-temporal information is important for video representation, we extend the negative samples by introducing intra-negative samples, which are transformed from the same anchor video by breaking temporal relations in video clips. With the proposed Inter-Intra Contrastive (IIC) frame work, we can train spatio-temporal convolutional networks to learn video representations (abstract)” and further on page 2200, right column, “In this paper, we proposed IIC,a self-supervised method for video representation learning, to learn rich temporal features from videos. We utilized the advantages of intra- and inter-sample learning and trained a spatio-temporal convolution neural network(3D CNN) with intra-negative samples in contrastive Multiview coding.” Thus, as seen from above, Tao does disclose the utilization of self-supervised contrastive learning for analysis of video features. Additionally, Rubinstein discloses analyzing motion data through motion heat maps to specifically target smaller movement data (abstract, “generate a motion-magnified video stream;” column 13, line 59, “FIG. 4B shows a “heat map” of motion magnitude within such generated image phase information. Thus, the “heat map” shows increased motion magnitude at the frequency of the cardiovascular pulse at the area, within the video stream, that corresponds to the neck of the person depicted.”). Tao doesn’t disclose the use of determining magnitudes of movement, however, one of ordinary skill in the art would recognize the amount of motion as opposed to just motion being present and acknowledge would understand finer detail can be provided. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Tao and Rubinstein in order to ensure one can analyze small motion features accurately while using self-supervised contrastive learning.
Regarding dependent claim 18, the rejection of claim 1 is incorporated herein. Additionally, Tao in the combination further discloses further comprising: processing a video to be processed according to a learned video representation model to obtain a corresponding video feature (page 2195, left column, “For video representation, many supervised methods have been proposed. Temporal Segment Networks (TSN) [40] split one video into several segments and sampled one frame from each segment as the input data of a 2D CNN. In addition to a single 2D CNN for RGB data, two-stream ConvNets [4, 5, 32] have been used with an additional optical flow stream. Recently, spatio-temporal convolution (3D-CNN) was applied to video recognition task.”).
Regarding independent claim 19, the rejection of claim 1 applies directly. Additionally, Tao discloses A video representation self-supervised contrastive learning apparatus (abstract, “We propose a self-supervised method to learn feature representations from videos. A standard approach in traditional self-supervised methods uses positive-negative data pairs to train with contrastive learning strategy. In such a case, different modalities of the same video are treated as positives and video clips from a different video are treated as negatives. Because the spatio-temporal information is important for video representation, we extend the negative samples by introducing intra-negative samples, which are transformed from the same anchor video by breaking temporal relations in video clips. With the proposed Inter-Intra Contrastive (IIC) frame work, we can train spatio-temporal convolutional networks to learn video representations;” the apparatus is read as the processor implementing the method), comprising:
a memory (Figure 2, “Memory bank”); and
a processor coupled to the memory, the processor being configured to perform, based on instructions stored in the memory, the video representation self-supervised contrastive learning method (abstract, “We propose a self-supervised method to learn feature representations from videos.” Machine learning is well known to be executed on a computer using a written program).
Tao fails to explicitly disclose as further recited. However, Rubinstein discloses calculating, according to optical flow information corresponding to each video frame of a video clip, a motion amplitude map corresponding to each video frame of the video clip (column 1, line 6, “It is advantageous in a variety of applications to detect and/or magnify motion that is present in a video. For example, it could be beneficial to magnify motion in a video of a sleeping child in order to verify that the child is breathing and/or to detect a rate of breathing of the child. In order to magnify and/or detect motion in a video, a variety of techniques can be applied. For example, an optical flow map could be determined by comparing different images of the video. Such an optical flow map could then be used to magnify motion within the video, e.g., by distorting the image frames of the video to enhance motion represented in the optical flow map.;” column 13, line 59, “FIG. 4B shows a “heat map” of motion magnitude within such generated image phase information. Thus, the “heat map” shows increased motion magnitude at the frequency of the cardiovascular pulse at the area, within the video stream, that corresponds to the neck of the person depicted.”);
determining, according to motion amplitude maps corresponding to the video frames of the video clip, motion information corresponding to the video clip (column 13, line 59, “FIG. 4B shows a “heat map” of motion magnitude within such generated image phase information. Thus, the “heat map” shows increased motion magnitude at the frequency of the cardiovascular pulse at the area, within the video stream, that corresponds to the neck of the person depicted.”).
Tao is directed toward, “a self-supervised method to learn feature representations from videos (abstract).” Rubinstein is directed toward, “Example embodiments allow for fast, efficient motion-magnification of video streams by decomposing image frames of the video stream into local phase information at multiple spatial scales and/or orientations (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Tao and Rubinstein are directed toward similar methods of endeavor of video processing. Further, Rubinstein allows for the analysis of “motion-magnified video stream (abstract).” One of ordinary skill in the art before the effective filing date of the claimed invention would be easily aware motion detection can be more difficult at very small levels; thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Rubinstein in order to detect motion on much smaller levels than standard video processing methods to output accurate motion details.
Tao and Rubinstein in the combination fail to explicitly disclose performing, according to a sequence of video clips and the motion information corresponding to each video clip, video representation self-supervised contrastive learning as claimed. However, Tao does disclose methods of utilizing self-supervised contrasted learning in the abstract, “Because the spatio-temporal information is important for video representation, we extend the negative samples by introducing intra-negative samples, which are transformed from the same anchor video by breaking temporal relations in video clips. With the proposed Inter-Intra Contrastive (IIC) frame work, we can train spatio-temporal convolutional networks to learn video representations (abstract)” and further on page 2200, right column, “In this paper, we proposed IIC,a self-supervised method for video representation learning, to learn rich temporal features from videos. We utilized the advantages of intra- and inter-sample learning and trained a spatio-temporal convolution neural network(3D CNN) with intra-negative samples in contrastive Multiview coding.” Thus, as seen from above, Tao does disclose the utilization of self-supervised contrastive learning for analysis of video features. Additionally, Rubinstein discloses analyzing motion data through motion heat maps to specifically target smaller movement data (abstract, “generate a motion-magnified video stream;” column 13, line 59, “FIG. 4B shows a “heat map” of motion magnitude within such generated image phase information. Thus, the “heat map” shows increased motion magnitude at the frequency of the cardiovascular pulse at the area, within the video stream, that corresponds to the neck of the person depicted.”). Tao doesn’t disclose the use of determining magnitudes of movement, however, one of ordinary skill in the art would recognize the amount of motion as opposed to just motion being present and acknowledge would understand finer detail can be provided. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Tao and Rubinstein in order to ensure one can analyze small motion features accurately while using self-supervised contrastive learning.
Regarding independent claim 20, the rejection of claim 1 applies directly. Additionally, Tao discloses a non-transitory computer-readable storage medium stored a computer program which, when executed by a processor, implements the steps of the video representation self-supervised contrastive learning method (abstract, “We propose a self-supervised method to learn feature representations from videos. A standard approach in traditional self-supervised methods uses positive-negative data pairs to train with contrastive learning strategy. In such a case, different modalities of the same video are treated as positives and video clips from a different video are treated as negatives. Because the spatio-temporal information is important for video representation, we extend the negative samples by introducing intra-negative samples, which are transformed from the same anchor video by breaking temporal relations in video clips. With the proposed Inter-Intra Contrastive (IIC) frame work, we can train spatio-temporal convolutional networks to learn video representations;” a processor is well known in the art to implement learning methods as in the described method):
Tao fails to explicitly disclose as further recited. However, Rubinstein discloses calculating, according to optical flow information corresponding to each video frame of a video clip, a motion amplitude map corresponding to each video frame of the video clip (column 1, line 6, “It is advantageous in a variety of applications to detect and/or magnify motion that is present in a video. For example, it could be beneficial to magnify motion in a video of a sleeping child in order to verify that the child is breathing and/or to detect a rate of breathing of the child. In order to magnify and/or detect motion in a video, a variety of techniques can be applied. For example, an optical flow map could be determined by comparing different images of the video. Such an optical flow map could then be used to magnify motion within the video, e.g., by distorting the image frames of the video to enhance motion represented in the optical flow map.;” column 13, line 59, “FIG. 4B shows a “heat map” of motion magnitude within such generated image phase information. Thus, the “heat map” shows increased motion magnitude at the frequency of the cardiovascular pulse at the area, within the video stream, that corresponds to the neck of the person depicted.”);
determining, according to motion amplitude maps corresponding to the video frames of the video clip, motion information corresponding to the video clip (column 13, line 59, “FIG. 4B shows a “heat map” of motion magnitude within such generated image phase information. Thus, the “heat map” shows increased motion magnitude at the frequency of the cardiovascular pulse at the area, within the video stream, that corresponds to the neck of the person depicted.”)
Tao is directed toward, “a self-supervised method to learn feature representations from videos (abstract).” Rubinstein is directed toward, “Example embodiments allow for fast, efficient motion-magnification of video streams by decomposing image frames of the video stream into local phase information at multiple spatial scales and/or orientations (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Tao and Rubinstein are directed toward similar methods of endeavor of video processing. Further, Rubinstein allows for the analysis of “motion-magnified video stream (abstract).” One of ordinary skill in the art before the effective filing date of the claimed invention would be easily aware motion detection can be more difficult at very small levels; thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Rubinstein in order to detect motion on much smaller levels than standard video processing methods to output accurate motion details.
Tao and Rubinstein in the combination fail to explicitly disclose performing, according to a sequence of video clips and the motion information corresponding to each video clip, video representation self-supervised contrastive learning as claimed. However, Tao does disclose methods of utilizing self-supervised contrasted learning in the abstract, “Because the spatio-temporal information is important for video representation, we extend the negative samples by introducing intra-negative samples, which are transformed from the same anchor video by breaking temporal relations in video clips. With the proposed Inter-Intra Contrastive (IIC) frame work, we can train spatio-temporal convolutional networks to learn video representations (abstract)” and further on page 2200, right column, “In this paper, we proposed IIC,a self-supervised method for video representation learning, to learn rich temporal features from videos. We utilized the advantages of intra- and inter-sample learning and trained a spatio-temporal convolution neural network(3D CNN) with intra-negative samples in contrastive Multiview coding.” Thus, as seen from above, Tao does disclose the utilization of self-supervised contrastive learning for analysis of video features. Additionally, Rubinstein discloses analyzing motion data through motion heat maps to specifically target smaller movement data (abstract, “generate a motion-magnified video stream;” column 13, line 59, “FIG. 4B shows a “heat map” of motion magnitude within such generated image phase information. Thus, the “heat map” shows increased motion magnitude at the frequency of the cardiovascular pulse at the area, within the video stream, that corresponds to the neck of the person depicted.”). Tao doesn’t disclose the use of determining magnitudes of movement, however, one of ordinary skill in the art would recognize the amount of motion as opposed to just motion being present and acknowledge would understand finer detail can be provided. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Tao and Rubinstein in order to ensure one can analyze small motion features accurately while using self-supervised contrastive learning.
Claim(s) 5 is rejected under 35 U.S.C. 103 as being unpatentable over Tao further in view of Rubinstein as applied to claim 1 above, and further in view of WO 2016/081778 A1 to Bentley et al. (hereinafter Bentley).
Regarding dependent claim 5, the rejection of claim 1 is incorporated herein. Additionally, Rubenstein in the combination discloses wherein the motion information corresponding to the video clip comprises one or more of a spatiotemporal motion map, a spatial motion map, and a temporal motion map corresponding to the video clip, wherein:
determining the spatiotemporal motion map corresponding to the video clip comprises:(column 13, line 59, “FIG. 4B shows a “heat map” of motion magnitude within such generated image phase information. Thus, the “heat map” shows increased motion magnitude at the frequency of the cardiovascular pulse at the area, within the video stream, that corresponds to the neck of the person depicted. Such magnitude information could be determined based on a moving average of the local phase magnitude or according to some other method.”)
Tao and Rubinstein in the combination fail to explicitly discloses as further recited. However, Bentley discloses wherein the motion information corresponding to the video clip comprises one or more of a spatiotemporal motion map, a spatial motion map, and a temporal motion map corresponding to the video clip, wherein:
determining the spatiotemporal motion map corresponding to the video clip comprises: superposing, in a temporal dimension, the motion amplitude maps for the video frames of the video clip to form the spatiotemporal motion map for the video clip (paragraph 0070, “ overlays of data or graphics on the video or on selected frames showing the value of metrics from the motion analysis;” had the data been motion amplitude maps of Rubinstein, Bentley would overlay this data on the video data in a temporal dimension);
determining the spatial motion map corresponding to the video clip comprises: pooling, along the temporal dimension, the spatiotemporal motion map for the video clip to obtain the spatial motion map for the video clip; or
determining the temporal motion map corresponding to the video clip comprises: pooling, along a spatial dimension, the spatiotemporal motion map for the video clip to obtain the temporal motion map for the video clip.
As noted above, Tao and Rubinstein are directed toward similar methods of endeavor of video processing. Further, Bentley is directed toward “Enables intelligent synchronization and transfer of generally concise event videos synchronized with motion data (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Tao, Rubinstein and Bentley are directed toward similar methods of endeavor of motion analysis in video data. Further, one of ordinary skill in the art before the effective filing date of the claimed invention would be easily aware motion data is most useful when analyzed in conjunction with the initial video data; said differently knowing a high level of motion occurred at some point in a video clip is not as relevant as knowing where in the video clip, and what was occurring at that exact time. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date to incorporate the teaching of Bentley in order to ensure a user can get contextual video motion data from the output, as opposed to just raw motion data.
Allowable Subject Matter
Claims 2-4 and 6-17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claims 2-4:
The following is a statement of reasons for the indication of allowable subject matter: the closest prior arts of record teach methods of analyzing video data including optical flow information and motion information. However, none of them alone or in any combination teaches aggregating amplitudes of gradient fields in two directions to generate a motion amplitude map per video frame.
The closest prior art being previously cited Rubinstein discloses analysis of motion in video data, and detecting small scale motion by aggregating data through the video (abstract, Figure 4b). However, there is no generation of a motion map according to gradient fields.
Thus, Rubinstein fails to disclose aggregating amplitudes of gradient fields in two directions to generate a motion amplitude map per video frame
Claims 6-17
The following is a statement of reasons for the indication of allowable subject matter: the closest prior arts of record teach methods of analyzing video data including optical flow information and motion information. However, none of them alone or in any combination teaches performing data augmentation of the video clip and performing self-supervised contrastive learning on the augmented video data in combination with a contrastive loss and/or a motion alignment loss.
The closest prior art being previously cited Tao discloses in algorithm 1, generation of a contrastive loss (see Algorithm 1, line 7). However, there is no data augmentation of the video clip data utilized to generate that contrastive loss.
Thus, Tao fails to disclose performing data augmentation of the video clip and performing self-supervised contrastive learning on the augmented video data in combination with a contrastive loss and/or a motion alignment loss.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
U.S. Publication No. 2018/0053300 to Podilchuk et al. discloses, “utilizing optical flow to generate tracking or mapping information between the sequential image frames; and generating a persistence value as a number of the sequential image frames that the region of interest appears or can be correlated between certain ones of the sequential image frames using the tracking or mapping information (paragraph 0015).”
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Courtney J. Windsor whose telephone number is (571)272-3956. The examiner can normally be reached Monday - Friday 8:00 - 4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/COURTNEY JOAN NELSON/Primary Examiner, Art Unit 2661
Prosecution Insights