DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see Remarks page 1, filed 01/15/2026, with respect to the objections of claims 8 and 19 have been fully considered and are persuasive. The objections of claims 8 and 19 have been withdrawn.
Applicant’s arguments, see Remarks page 1, filed 01/15/2026, with respect to the rejection of claim 20 under 35 U.S.C. 112(b) have been fully considered and are persuasive. The rejection of claim 20 has been withdrawn.
Applicant's arguments, see Remarks pages 1-5, filed 01/15/2026, with respect to the rejections of claims 1-20 under 35 U.S.C. 103 have been fully considered but they are not persuasive.
On pages 1-2 of Remarks, Applicant argues:
PNG
media_image1.png
1041
668
media_image1.png
Greyscale
Examiner respectfully disagrees.
PNG
media_image2.png
294
1466
media_image2.png
Greyscale
MPEP 2123(II) discloses:
Section 5.2. Frame Differencing Experiments of Ellenfeld discloses “Equations 1 and 2 explain how frame differencing is done throughout the scope of this paper. In contrast to other three-frame differencing approaches [63] we do not use past and future frames to calculate the differences. Instead, we use only past frames as we want to enable inference of live sources (e.g., streams from surveillance cameras) without adding a constant delay.” Thus, although Ellenfeld discloses a multi-modal motion segmentation system which processes a previous frames and a current frame, as opposed to a previous and subsequent image, in order to avoid the addition of a delay, the mere disclosure of Ellenfeld’s reasoning for using a previous frames and a current frame does not constitute teaching away. Instead, Ellenfeld’s disclosure serves as an alternative for the purposes of motion segmentation, wherein as MPEP 2123(II) discloses “’[t]he prior art's mere disclosure of more than one alternative does not constitute a teaching away from any of these alternatives because such disclosure does not criticize, discredit, or otherwise discourage the solution claimed....’”.
In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, the Abstract of Xu discloses “On the basis of the traditional three frame difference method, the "XOR" and "OR" operations are used instead of the "AND" operation to extract the foreground to solve the hole problem in the extraction of the foreground. But it will cause problems such as elongated target, blurred outline and noise…We consider combining two foregrounds by using the "AND" operation to obtain a comprehensive foreground image. The experimental results show that the algorithm can extract moving objects quickly, accurately and completely.”
It would have been obvious to substitute the three-frame differencing method disclosed by Ellenfeld with the three-frame difference method taught by Xu, because Xu’s disclosed three-frame difference method serves to resolve problems caused by traditional differencing methods, thus serving to provide fast and accurate frame differencing results.
Therefore, the rejection of claim 1 under 35 U.S.C. 103 is maintained.
On page 3 of Remarks, Applicant argues:
PNG
media_image3.png
681
680
media_image3.png
Greyscale
Examiner respectfully disagrees.
In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Section 1. Introduction of Ellenfeld discloses “the current image contributes with appearance information to distinguish between relevant and irrelevant motion and frame differencing captures the temporal information, which is the scene’s motion independent of the camera motion. We fuse this information in the DCNN to receive an effective and efficient approach for robust motion segmentation.”. Wherein the DCNN processes an input frame image, for the purposes of object detection for extracting “relevant motion”, and a difference frame for extracting motion based on differences between frames. The input images are fused by the DCNN for motion segmentation, wherein moving objects are identified.
Section 3 Faster R-CNN of Ren discloses “Our object detection system, called Faster R-CNN, is
composed of two modules. The first module is a deep fully convolutional network that proposes regions, and the second module is the Fast R-CNN detector [2] that uses the proposed regions. The entire system is a single, unified network for object detection (Figure 2). Using the recently popular terminology of neural networks with ‘attention’ [31] mechanisms, the RPN module tells the Fast R-CNN module where to look. In Section 3.1 we introduce the designs and properties of the network for region proposal. In Section 3.2 we develop algorithms for training both modules with features shared.” Wherein Ren discloses a two-stage region-based deep computational neural network, which is configured to first propose object regions with a region proposal network, then classify the proposed regions with a Fast R-CNN module.
Thus, it would have been obvious for one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to substitute the DCNN model for motion segmentation disclosed by Ellenfeld in view of Xu with the Faster R-CNN model disclosed by Ren trained for extracting region proposals based on the resulting images disclosed by Ellenfeld in view of Xu. Thus, allowing for the two-stage region-based deep computational neural network to identify moving objects only in portions of the image frame that are highlighted in the resulting image.
Therefore, the rejection of claim 7 under 35 U.S.C. 103 is maintained.
On Pages 4-5 of Remarks, Applicant argues:
PNG
media_image4.png
726
674
media_image4.png
Greyscale
Examiner respectfully disagrees.
Section 3.3 of Cui discloses “The categories to determine kv need to be predefined during the training process and are not reconfigurable at inference time. In other words, a well-trained model is not adaptive to multiple configurations. Take
Θ
= 3, which represents objects with slow, medium, and fast motion speeds, as an example. Given a frame with the motion IoU of sm = 0.75, it is always categorized as medium motion speeds, which requires [2/3k] frames for aggregation…
To solve the issues mentioned above, we extend the vanilla dynamic aggregation module to a deformable version, which is more effective and reconfigurable. Instead of classifying the input frame Ii into
Θ
categories, we use a function
σ
to project the s
∈
[0, 1] in the range of 0 and 1, where s is a score which takes both the motion IoU sm
∈
[0, 1] and size ss
∈
[0, 1] of objects in the current frame Ii into account…
In real-world applications, when there are enough computational resources like applications on servers, we can choose
σ
, which casts s
∈
[0, 1] in the range of 0 and 1. When there are not enough resources, like applications on cellphones or servers where partial machines are under maintenance, we can reload the configure file where a new
σ
projects s in a new range (e.g. [0, 0.5]) to use fewer frames for aggregation without the need of training a new model.” Wherein Cui discloses a method for performing object detection in a video based on a selection of a number of frames to process, which include a center frame with previous frames and subsequent frames. Wherein the number of frames selected is determined based on an object’s classified speed and the computational resources available. Thus, the selection of the number of video frames based on an object’s classified speed and the computational resources available constitute an accounting for an expected object speed, as well as an inherent camera frame-rate latency, since the processed video frames are captured based on a camera’s frame rate.
In addition, Section 4.3 Model Analysis of Cui discloses “We conduct experiments with deformable feature aggregation on the choices of sampling function as Table 3. “Nearest" and “Furthest" represent choosing the closest and furthest kd frames for aggregation, while “Bin" means binning the k frames into kd buckets and sample 1 frame from each bucket”. Wherein based on the Sampling Function for choosing the frames, the distance between the center frame and the adjacent frames differ. In addition, even if the nearest method is selected, the distance between a preceding frame, such as frame 8 in Table 2, and a center frame differ based on the number of video frames selected.
Therefore, Cui discloses the claim 12 limitations “adjusting x so as to account for an inherent frame-rate latency of a camera that recorded the video and an expected speed of a given moving object.”
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 10-11, and 14-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ellenfeld et al. (Deep Fusion of Appearance and Frame Differencing for Motion Segmentation) hereinafter referenced as Ellenfeld, in view of Xu et al. (Moving Object Detection Based on Improved Three Frame Difference and Background Subtraction) hereinafter referenced as Xu.
Regarding claim 1, Ellenfeld discloses: A method for detecting objects in a video (Ellenfeld: Abstract) comprising: selecting an initial image frame m from the video, wherein the initial image frame m is preceded by a previous image frame m-p, where p represents a number of frames; creating a second difference image by subtracting the initial image frame m from the previous image frame m-p (Ellenfeld: 5.2. Frame Differencing Experiments: “Difference images are created by either two-frame or three-frame differencing [42, 50]. While two-frame differencing is evaluated for temporal offsets ΔF of one, five, and ten frames between the current and the reference frame, three-frame differencing is investigated using two different fusion strategies, minimum and sum (see Eqs. 1 and 2), and a ΔF of one, two, and five frames between each frame starting from current going back.”; Wherein the current frame is subtracted from a previous frame.);
concatenating the second difference image and the initial image frame m to create a concatenated input (Ellenfeld: 4. Proposed Approach “Frame differencing outputs a difference image that we feed into the DCNN together with the current image.”; Wherein the inputting of the difference and initial image frames together into the DCNN constitutes a concatenated input.);
feeding the concatenated input into an object detection network; and wherein the object detection network is configured to leverage motion-cue side information contained in the second difference image and the initial image frame m to facilitate rapid detection of moving objects in the video (Ellenfeld: 1. Introduction: “we propose a Deep Convolutional Neural Network (DCNN) for multi-modal motion segmentation: the current image contributes with appearance information to distinguish between relevant and irrelevant motion and frame differencing captures the temporal information, which is the scene’s motion independent of the camera motion. We fuse this information in the DCNN to receive an effective and efficient approach for robust motion segmentation.”).
Ellenfeld does not disclose expressly: wherein the initial image frame m is followed by a subsequent image frame m+n, where n represents a number of frames; creating a first difference image by subtracting the subsequent image frame m+n from the initial image frame m; combining the first and second difference images to create a resulting image; and concatenating the resulting image and the initial image frame m to create a concatenated input.
Thus, Ellenfeld does not disclose expressly: wherein the object detection network leverages previous and subsequent motion-cue side information contained in the resulting image and the initial image frame m to facilitate rapid detection of moving objects in the video.
Xu discloses: wherein the initial image frame m is followed by a subsequent image frame m+n, where n represents a number of frames; creating a first difference image by subtracting the subsequent image frame m+n from the initial image frame m (Xu: II. THREE FRAME DIFFERENCE METHOD: “Three frame difference is to do two differential operations, respectively denoted by ft-1(x,y), ft(x,y), ft+1(x,y), find the corresponding difference Dt(x,y) and Dt+1(x,y)… The expressions are given as
PNG
media_image5.png
108
642
media_image5.png
Greyscale
”; Wherein Dt+1(x,y) is calculated based on the difference between the current image and the subsequent frame.);
combining the first and second difference images to create a resulting image (Xu: II. THREE FRAME DIFFERENCE METHOD: “Dt+1(x,y), Dt(x,y), and Yt(x,y) are obtained by the traditional three frame difference. 2) For Yt(x,y) and Dt+1(x,y) do "OR" operation and get the image Mt1(x,y)…3) Dt+1(x,y) and Dt(x,y)are expanded respectively to obtain Pt+1(x,y) and Pt(x,y), and then do "OR" operation to obtain the image Mt2(x,y)… For Mt1(x,y) and Mt2(x,y), do "OR" operation and get the image M1(x,y).”; Wherein the difference images Dt and Dt+1 are combined into a resulting images Mt1 and Mt2, which are combined into a final image M1.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the frame differencing method disclosed by Ellenfeld with the three-frame difference method taught by Xu, such that the second difference image of Ellenfeld is replaced by the combined final image of Xu. The suggestion/motivation for doing so would have been “On the basis of the traditional three frame difference method, the "XOR" and "OR" operations are used instead of the "AND" operation to extract the foreground to solve the hole problem in the extraction of the foreground.” (Xu: Abstract). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ellenfeld with Xu to obtain the invention as specified in claim 1.
Regarding claim 2, Ellenfeld in view of Xu discloses: The object detection method of claim 1, wherein the combining step comprises using a bitwise OR gate to create the resulting image (Xu: II. THREE FRAME DIFFERENCE METHOD: “3) Dt+1(x,y) and Dt(x,y) are expanded respectively to obtain Pt+1(x,y) and Pt(x,y), and then do "OR" operation to obtain the image Mt2(x,y)”; Wherein for the creation of resulting image Mt2, the bitwise OR operator is used between the two difference images.).
Regarding claim 3, Ellenfeld in view of Xu discloses: The object detection method of claim 1, wherein the combining step comprises using a bitwise AND gate to create the resulting image (Xu: II. THREE FRAME DIFFERENCE METHOD: “After the binarization of Dt+1(x,y) and Dt(x,y), the "AND" operation is done, moving target foreground image is obtained as
PNG
media_image6.png
98
628
media_image6.png
Greyscale
…2) For Yt(x,y) and Dt+1(x,y) do "OR" operation and get the image Mt1(x,y)”; Wherein for the creation of resulting image Mt1, Yt is created by using the bitwise AND between the two difference images.).
Regarding claim 4, Ellenfeld in view of Xu discloses: The object detection method of claim 2, further comprising: using a bitwise AND gate to combine the first and second difference images to create a second resulting image in such a way that the second resulting image only retains pixels from objects that have moved in the initial image frame with respect to the previous image frame (Xu: II. THREE FRAME DIFFERENCE METHOD: “After the binarization of Dt+1(x,y) and Dt(x,y), the "AND" operation is done, moving target foreground image is obtained as
PNG
media_image6.png
98
628
media_image6.png
Greyscale
…2) For Yt(x,y) and Dt+1(x,y) do "OR" operation and get the image Mt1(x,y) which is shown as
PNG
media_image7.png
82
368
media_image7.png
Greyscale
”; Wherein for the creation of resulting image Mt1, Yt is created by using the bitwise AND between the two difference images.
Also, wherein the creation of the Mt1 image based on Yt and difference image Dt+1 constitutes the retaining of only pixels from objects that have moved in the initial image with respect to, or in relation to, the previous image frame); and wherein the concatenating step comprises concatenating both the resulting image and the second resulting image with the initial image frame m to create the concatenated input (Xu: II. THREE FRAME DIFFERENCE METHOD: “For Mt1(x,y) and Mt2(x,y), do "OR" operation and get the image M1(x,y) which is shown as
PNG
media_image8.png
68
679
media_image8.png
Greyscale
.”; Wherein resulting images Mt1 and Mt2 are concatenated into M1, which is input into the DCNN with the initial image frame as disclosed by Ellenfeld.).
Regarding claim 5, Ellenfeld in view of Xu discloses: The object detection method of claim 1, wherein the object detection network is a deep neural network (Ellenfeld: Abstract: “Motion segmentation is a technique to detect and localize class-agnostic motion in videos. This motion is assumed to be relative to a stationary background and usually originates from objects such as vehicles or humans…we propose a Deep Convolutional Neural Network (DCNN) for multimodal motion segmentation”).
Regarding claim 10, Ellenfeld discloses: A method for detecting objects in a video (Ellenfeld: Abstract) comprising: parsing the video into frames; selecting one in every predetermined quantity of frames as a key frame of interest m (Ellenfeld: 5.2. Frame Differencing Experiments: “Difference images are created by either two-frame or three-frame differencing [42, 50]…three-frame differencing is investigated using two different fusion strategies, minimum and sum (see Eqs. 1 and 2), and a ∆F of one, two, and five frames between each frame starting from current going back…Referring to the F1-score, our DCNN performs best if the difference image is calculated using three-frame differencing with ∆F of 5 frames distance between each considered frame”; Wherein during calculation, the key and reference frames are determined based on a ∆F distance between each frame.);
calculating the absolute difference between a given key frame of interest mt at time t and a preceding frame mt-x according to abs(mt-mt-x) to create a first difference image, where x is a predetermined amount of time; calculating the absolute difference between the given key frame of interest mt and a second preceding frame mt-2x according to abs(mt-mt-2x) to create a second difference image; performing a concatenation function on the first and second difference images to create a first resulting image (Ellenfeld: 5.2. Frame Differencing Experiments: “Difference images are created by either two-frame or three-frame differencing [42, 50]…three-frame differencing is investigated using two different fusion strategies, minimum and sum (see Eqs. 1 and 2), and a ∆F of one, two, and five frames between each frame starting from current going back…
PNG
media_image9.png
195
571
media_image9.png
Greyscale
Equations 1 and 2 explain how frame differencing is done throughout the scope of this paper…Referring to the F1-score, our DCNN performs best if the difference image is calculated using three-frame differencing with ∆F of 5 frames distance between each considered frame”; Wherein the frame differencing is determined based a concatenation of the absolute differences It - It-1 and It - It-2, wherein each frame is a ∆F distance from another frame.);
feeding the first resulting image and the given key frame of interest mt into a deep neural network configured to detect moving objects in the video (Ellenfeld: 4. Proposed Approach “Frame differencing outputs a difference image that we feed into the DCNN together with the current image.”; Wherein the inputting of the difference and initial image frames together into the DCNN constitutes a concatenated input.).
Ellenfeld does not disclose expressly: calculating the absolute difference between the given key frame of interest mt and a subsequent frame mt+x according to abs(mt-mt+x) to create a second difference image; performing a bitwise OR function on the first and second difference images to create a first resulting image.
Xu discloses: calculating the absolute difference between the given key frame of interest mt and a subsequent frame mt+x according to abs(mt-mt+x) to create a second difference image (Xu: II. THREE FRAME DIFFERENCE METHOD: “Three frame difference is to do two differential operations, respectively denoted by ft-1(x,y), ft(x,y), ft+1(x,y), find the corresponding difference Dt(x,y) and Dt+1(x,y)… The expressions are given as
PNG
media_image10.png
48
286
media_image10.png
Greyscale
”; Wherein Dt+1(x,y) is calculated based on the difference between the current image and the subsequent frame.); performing a bitwise OR function on the first and second difference images to create a first resulting image (Xu: II. THREE FRAME DIFFERENCE METHOD: “3) Dt+1(x,y) and Dt(x,y)are expanded respectively to obtain Pt+1(x,y) and Pt(x,y), and then do "OR" operation to obtain the image Mt2(x,y)… For Mt1(x,y) and Mt2(x,y), do "OR" operation and get the image M1(x,y).”; Wherein for the creation of a resulting image Mt2, the bitwise OR operator is used between the two difference images.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the frame differencing method disclosed by Ellenfeld with the three-frame difference method taught by Xu, such that the first resulting image of Ellenfeld is replaced by the combined final image of Xu. The suggestion/motivation for doing so would have been “On the basis of the traditional three frame difference method, the "XOR" and "OR" operations are used instead of the "AND" operation to extract the foreground to solve the hole problem in the extraction of the foreground.” (Xu: Abstract). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ellenfeld with Xu to obtain the invention as specified in claim 10.
Regarding claim 11, Ellenfeld in view of Xu discloses: The method of claim 10, further comprising performing a bitwise AND function on the first and second difference images to create a second resulting image in such a way that the second resulting image only retains pixels from objects that have moved in the given key frame of interest with respect to the previous image frame (Xu: II. THREE FRAME DIFFERENCE METHOD: “After the binarization of Dt+1(x,y) and Dt(x,y), the "AND" operation is done, moving target foreground image is obtained as
PNG
media_image6.png
98
628
media_image6.png
Greyscale
…2) For Yt(x,y) and Dt+1(x,y) do "OR" operation and get the image Mt1(x,y) which is shown as
PNG
media_image7.png
82
368
media_image7.png
Greyscale
”; Wherein for the creation of resulting image Mt1, Yt is created by using the bitwise AND between the two difference images.
Also, wherein the creation of the Mt1 image based on Yt and difference image Dt+1 constitutes the retaining of only pixels from objects that have moved in the initial image with respect to, or in relation to, the previous image frame); and wherein the feeding step further comprises feeding the second resulting image into the deep neural network configured to detect moving objects in the video (Xu: II. THREE FRAME DIFFERENCE METHOD: “For Mt1(x,y) and Mt2(x,y), do "OR" operation and get the image M1(x,y) which is shown as
PNG
media_image8.png
68
679
media_image8.png
Greyscale
.”; Wherein resulting images Mt1 and Mt2 are concatenated into M1, which is input into the DCNN with the initial image frame as disclosed by Ellenfeld, which constitutes the feeding of the second resulting image into the deep neural network.).
Regarding claim 14, Ellenfeld discloses: A method for detecting objects in a video taken by a camera (Ellenfeld: Abstract) comprising: extracting motion information from the video by subtracting a frame of interest from its associated adjacent frames to create two initial difference images; creating a resulting image from the two initial difference images, wherein the resulting image retains past motion information relative to the frame of interest (Ellenfeld: 5.2. Frame Differencing Experiments: “Difference images are created by either two-frame or three-frame differencing [42, 50]…three-frame differencing is investigated using two different fusion strategies, minimum and sum (see Eqs. 1 and 2), and a ∆F of one, two, and five frames between each frame starting from current going back…
PNG
media_image9.png
195
571
media_image9.png
Greyscale
Equations 1 and 2 explain how frame differencing is done throughout the scope of this paper…Referring to the F1-score, our DCNN performs best if the difference image is calculated using three-frame differencing with ∆F of 5 frames distance between each considered frame”; Wherein a resulting frame difference result is determined based on a concatenation of the absolute differences It - It-1 and It - It-2, wherein each frame is a ∆F distance from another frame and It is a frame of interest.);
concatenating the resulting image onto the frame of interest; and feeding the concatenated images into a deep neural network configured to identify moving objects in the frame of interest (Ellenfeld: 4. Proposed Approach “Frame differencing outputs a difference image that we feed into the DCNN together with the current image.”; Wherein the inputting of the difference and initial image frames together into the DCNN constitutes a concatenating the resulting image onto the frame of interest.).
Ellenfeld does not disclose expressly: using one or more of a bitwise-AND function and a bitwise-OR function to create a resulting image from the two initial difference images, wherein the resulting image retains past and future motion information relative to the frame of interest.
Xu discloses: using one or more of a bitwise-AND function and a bitwise-OR function to create a resulting image from the two initial difference images(Xu: II. THREE FRAME DIFFERENCE METHOD: “After the binarization of Dt+1(x,y) and Dt(x,y), the "AND" operation is done, moving target foreground image is obtained as
PNG
media_image6.png
98
628
media_image6.png
Greyscale
…2) For Yt(x,y) and Dt+1(x,y) do "OR" operation and get the image Mt1(x,y)…3) Dt+1(x,y) and Dt(x,y)are expanded respectively to obtain Pt+1(x,y) and Pt(x,y), and then do "OR" operation to obtain the image Mt2(x,y)… For Mt1(x,y) and Mt2(x,y), do "OR" operation and get the image M1(x,y).”; Wherein the difference images Dt and Dt+1 are combined into a resulting images Mt1 and Mt2 using “AND” and “OR” operations, respectively. The Mt1 and Mt2 images are then combined into a final image M1.), wherein the resulting image retains past and future motion information relative to the frame of interest (Xu: II. THREE FRAME DIFFERENCE METHOD: “Three frame difference is to do two differential operations, respectively denoted by ft-1(x,y), ft(x,y), ft+1(x,y), find the corresponding difference Dt(x,y) and Dt+1(x,y)… The expressions are given as
PNG
media_image10.png
48
286
media_image10.png
Greyscale
”; Wherein Dt+1 and Dt retain the past and future motion information relative to the current frame, respectively.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the frame differencing method disclosed by Ellenfeld with the three-frame difference method taught by Xu, such that the resulting image of Ellenfeld is replaced by the combined final image of Xu. The suggestion/motivation for doing so would have been “On the basis of the traditional three frame difference method, the "XOR" and "OR" operations are used instead of the "AND" operation to extract the foreground to solve the hole problem in the extraction of the foreground.” (Xu: Abstract). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ellenfeld with Xu to obtain the invention as specified in claim 14.
Regarding claim 15, Ellenfeld in view of Xu discloses: The method of claim 14, wherein if the camera is stationary relative to its surroundings, the bitwise-AND function is used to create the resulting image (Xu: II. THREE FRAME DIFFERENCE METHOD: “After the binarization of Dt+1(x,y) and Dt(x,y), the "AND" operation is done, moving target foreground image is obtained as
PNG
media_image11.png
35
227
media_image11.png
Greyscale
…2) For Yt(x,y) and Dt+1(x,y) do "OR" operation and get the image Mt1(x,y)”; Wherein for the creation of resulting image Mt1, Yt is created by using the bitwise AND between the two difference images, whether or not the camera is stationary.).
Regarding claim 16, Ellenfeld in view of Xu discloses: The method of claim 15, wherein if the camera was moving relative to its surroundings when the video was taken, the bitwise-OR function is used to create the resulting image (Xu: II. THREE FRAME DIFFERENCE METHOD: “3) Dt+1(x,y) and Dt(x,y) are expanded respectively to obtain Pt+1(x,y) and Pt(x,y), and then do "OR" operation to obtain the image Mt2(x,y)”; Wherein for the creation of resulting image Mt2, the bitwise OR operator is used between the two difference images, whether or not the camera was moving).
Regarding claim 17, Ellenfeld in view of Xu discloses: The method of claim 14, wherein both the bitwise-AND and the bitwise-OR functions are used to create two resulting images that are both concatenated onto the frame of interest prior to being fed into the deep neural network (Xu: II. THREE FRAME DIFFERENCE METHOD: “After the binarization of Dt+1(x,y) and Dt(x,y), the "AND" operation is done, moving target foreground image is obtained as
PNG
media_image6.png
98
628
media_image6.png
Greyscale
…2) For Yt(x,y) and Dt+1(x,y) do "OR" operation and get the image Mt1(x,y)…3) Dt+1(x,y) and Dt(x,y)are expanded respectively to obtain Pt+1(x,y) and Pt(x,y), and then do "OR" operation to obtain the image Mt2(x,y)… For Mt1(x,y) and Mt2(x,y), do "OR" operation and get the image M1(x,y).”; Wherein the difference images Dt and Dt+1 are combined into a resulting images Mt1 and Mt2 using “AND” and “OR” operations, respectively. The Mt1 and Mt2 images are then concatenated into a final image M1, which is concatenated with the initial image frame into the DCNN as taught by Ellenfeld).
Claim(s) 6-7 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ellenfeld in view of Xu, and further in view of Ren et al. (Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks) hereinafter referenced as Ren.
Regarding claim 6, Ellenfeld in view of Xu discloses: The object detection method of claim 5.
Ellenfeld in view of Xu does not disclose expressly: wherein the deep neural network is a two-stage region-based computational neural network.
Ren discloses: a two-stage region-based computational neural network (Ren: 3 FASTER R-CNN: “Our object detection system, called Faster R-CNN, is composed of two modules. The first module is a deep fully convolutional network that proposes regions, and the second module is the Fast R-CNN detector [2] that uses the proposed regions. The entire system is a single, unified network for object detection (Figure 2).”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the DCNN model for motion segmentation disclosed by Ellenfeld in view of Xu with the Faster R-CNN model disclosed by Ren. The suggestion/motivation for doing so would have been “By sharing convolutional features with the down-stream detection network, the region proposal step is nearly cost-free…The learned RPN also improves region proposal quality and thus the overall object detection accuracy” (Ren: 5 CONCLUSION). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ellenfeld in view of Xu with Ren to obtain the invention as specified in claim 6.
Regarding claim 7, Ellenfeld in view of Xu discloses: The object detection method of claim 5, wherein the deep neural network is configured to perform motion segmentation by fusing appearance information extracted from the current image, used to distinguish between relevant and irrelevant motion, with the temporal information extracted from the resulting images (Ellenfeld: 1. Introduction: “the current image contributes with appearance information to distinguish between relevant and irrelevant motion and frame differencing captures the temporal information, which is the scene’s motion independent of the camera motion. We fuse this information in the DCNN to receive an effective and efficient approach for robust motion segmentation.”; Wherein the appearance and temporal information is fused for the extraction of moving objects.).
Ellenfeld in view of Xu does not disclose expressly: wherein the deep neural network is configured to perform repeated convolutions to identify moving objects only in portions of the initial image frame m that are highlighted in the resulting image.
Thus, Ellenfeld in view of Xu does not disclose expressly: wherein the deep neural network is configured to perform repeated convolutions to identify objects only in portions of the initial image frame that are highlighted in the resulting image.
Ren discloses: a deep neural network configured to perform repeated convolutions to identify objects only in portions of the initial image frame that are highlighted in the image, and extracted by a region proposal network (Ren: Abstract: “we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with “attention” mechanisms, the RPN component tells the unified network where to look.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the DCNN model for motion segmentation disclosed by Ellenfeld in view of Xu with the Faster R-CNN model disclosed by Ren trained for extracting region proposals based on the resulting images disclosed by Ellenfeld in view of Xu. The suggestion/motivation for doing so would have been “By sharing convolutional features with the down-stream detection network, the region proposal step is nearly cost-free…The learned RPN also improves region proposal quality and thus the overall object detection accuracy” (Ren: 5 CONCLUSION). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ellenfeld in view of Xu with Ren to obtain the invention as specified in claim 7.
Regarding claim 18, Ellenfeld in view of Xu discloses: The method of claim 14.
Ellenfeld in view of Xu does not disclose expressly: wherein the deep neural network is a Faster R-CNN.
Ren discloses: a Faster R-CNN deep neural network (Ren: 3 FASTER R-CNN: “Our object detection system, called Faster R-CNN, is composed of two modules. The first module is a deep fully convolutional network that proposes regions, and the second module is the Fast R-CNN detector [2] that uses the proposed regions. The entire system is a single, unified network for object detection (Figure 2).”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the DCNN model for motion segmentation disclosed by Ellenfeld in view of Xu with the Faster R-CNN model disclosed by Ren. The suggestion/motivation for doing so would have been “By sharing convolutional features with the down-stream detection network, the region proposal step is nearly cost-free…The learned RPN also improves region proposal quality and thus the overall object detection accuracy” (Ren: 5 CONCLUSION). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ellenfeld in view of Xu with Ren to obtain the invention as specified in claim 18.
Claim(s) 8-9 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ellenfeld in view of Xu, and further in view of Din et al. (Abandoned Object Detection using Frame Differencing and Background Subtraction) hereinafter referenced as Din.
Regarding claim 8, Ellenfeld in view of Xu discloses: The object detection method of claim 4.
Ellenfeld in view of Xu does not disclose expressly: further comprising applying a Gaussian filter to the initial image frame m, the previous image frame m-p, and the subsequent image frame m+n prior to the creating steps if the video exceeds a noise level threshold.
Thus, Ellenfeld in view of Xu does not discloses expressly: the applying of a Gaussian filter to processed images prior to the creation of difference images, if the video exceeds a noise level threshold.
Din discloses: the application of a Gaussian filter to video frame images, prior to the application of image processing steps, if the video exceeds a noise level threshold (Din: III. PROPOSED METHODOLOGY: “The preprocessing is a necessary and standard step used generally in every machine vision algorithms. This will help to remove the noise in video frames to avoid false detection and improve the performance of the algorithm…we convert the image into grayscale followed by the Gaussian filtering to smooth the image and suppress the noise.”; Wherein the application of Gaussian filter as a pre-processing step constitutes the exceeding of a minimum noise level threshold.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify implement the Gaussian filter preprocessing step disclosed by Din prior processing the video frames disclosed by Ellen in view of Xu. The suggestion/motivation for doing so would have been “This will help to remove the noise in video frames to avoid false detection and improve the performance of the algorithm…we convert the image into grayscale followed by the Gaussian filtering to smooth the image and suppress the noise.” (Din: III. PROPOSED METHODOLOGY). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ellenfeld in view of Xu with Din to obtain the invention as specified in claim 8.
Regarding claim 9, Ellenfeld in view of Xu and Din discloses: The object detection method of claim 8, further comprising correcting for video camera motion by using motion information metadata (Ellenfeld: 4.1. Frame Differencing: “We first perform image registration and alignment. Local image features are detected [48] and tracked across the image sequence using sparse optical flow [30]. This approach provides good accuracy without sacrificing too much speed. An affine transformation matrix (also known as homography) is estimated and used to align the previous images to the current one. This step is optional and suitable only, if the camera is moving.”; Wherein prior to the frame differencing, the images are aligned in order to correct camera motion).
Regarding claim 19, Ellenfeld in view of Xu discloses: The method of claim 15, wherein morphological operations are applied to the difference images prior to being input into the DCNN with the current image (Ellenfeld: 4.1. Frame Differencing: “we apply morphological operations directly to the gray-valued difference image. We use morphological opening to remove noise followed by morphological closing to fill holes in the difference image…This image is then fed into the DCNN together with the current image”).
Ellenfeld in view of Xu does not disclose expressly: further comprising applying a Gaussian filter to the frame of interest and to the two initial difference images prior to creating the resulting image if the video exceeds a noise level threshold.
Thus, Ellenfeld in view of Xu does not disclose expressly: applying a Gaussian filter to the frame of interest and to the two initial difference images prior to being input together into the DCNN.
Din discloses: applying a Gaussian filter to input video frames prior to being processed by machine learning techniques, if the video exceeds a noise level threshold (Din: III. PROPOSED METHODOLOGY: “The preprocessing is a necessary and standard step used generally in every machine vision algorithms. This will help to remove the noise in video frames to avoid false detection and improve the performance of the algorithm…we convert the image into grayscale followed by the Gaussian filtering to smooth the image and suppress the noise.”; Wherein the application of Gaussian filter as a pre-processing step constitutes the exceeding of a minimum noise level threshold.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the morphological opening operation for removing noise in the difference images disclosed by Ellenfeld in view of Xu with the Gaussian filter process applied to images prior to being processed by machine-learning techniques disclosed by Din. The suggestion/motivation for doing so would have been “This will help to remove the noise in video frames to avoid false detection and improve the performance of the algorithm…we convert the image into grayscale followed by the Gaussian filtering to smooth the image and suppress the noise.” (Din: III. PROPOSED METHODOLOGY). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ellenfeld in view of Xu with Din to obtain the invention as specified in claim 19.
Claim(s) 12-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ellenfeld in view of Xu, and further in view of Cui (DFA: Dynamic Feature Aggregation for Efficient Video Object Detection).
Regarding claim 12, Ellenfeld in view of Xu discloses: The method of claim 11,
Ellenfeld in view of Xu does not disclose expressly: further comprising adjusting x so as to account for an inherent frame-rate latency of a camera that recorded the video and an expected speed of a given moving object.
Cui discloses: further comprising adjusting x so as to account for an inherent frame-rate latency of a camera that recorded the video and an expected speed of a given moving object (Cui: 3 Methodology: “The key idea of our method is to replace the fixed number of frames with a dynamic size in the current feature aggregation-based video object detectors. Therefore, instead of using fixed frames, our model can adaptively choose the frames for aggregation according to the inputs, as shown in Figure 2.”;
3.3 Deformable Dynamic Aggregation: “we use a function σ to project the s ∈ [0, 1] in the range of 0 and 1, where s is a score which takes both the motion IoU sm ∈ [0, 1] and size ss ∈ [0, 1] of objects in the current frame Ii into account…when there are enough computational resources like applications on servers, we can choose σ, which casts s ∈ [0, 1] in the range of 0 and 1. When there are not enough resources, like applications on cellphones or servers where partial machines are under maintenance, we can reload the configure file where a new σ projects s in a new range (e.g. [0, 0.5]) to use fewer frames for aggregation without the need of training a new model.”;
4.3 Model Analysis: “We conduct experiments with deformable feature aggregation on the choices of sampling function as Table 3. “Nearest” and “Furthest” represent choosing the closest and furthest kd frames for aggregation…From Table 2, “Nearest” sampling has the best performance compared with the other methods”;
Wherein the adjustment of the number of frames based on computation resources and the motion of objects in the frame Ii constitutes the accounting for a camera’s frame-rate latency and object’s expected speed.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the model for adaptively selecting the number of frames processed disclosed by Cui into the motion segmentation process disclosed by Ellenfeld in view of Xu. The suggestion/motivation for doing so would have been “the performance and inference speed are heavily influenced by the number of frames used for aggregation. In this paper, we aim to perform dynamic aggregation to the current methods to balance the performance and inference speed.” (Cui: 5 Conclusion). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ellenfeld in view of Xu with Cui to obtain the invention as specified in claim 12.
Regarding claim 13, Ellenfeld in view of Xu and Cui discloses: The method of claim 12, further comprising incorporating ego-motion to compensate for camera motion by matching pixels between the given key frame of interest and the preceding and subsequent frames to a fiduciary point so as to ignore motion noise generated by the camera’s motion (Ellenfeld: 4.1. Frame Differencing: “We first perform image registration and alignment. Local image features are detected [48] and tracked across the image sequence using sparse optical flow [30]. This approach provides good accuracy without sacrificing too much speed. An affine transformation matrix (also known as homography) is estimated and used to align the previous images to the current one. This step is optional and suitable only, if the camera is moving.”; Wherein the alignment of the image frames prior to frame differencing, constitutes the matching of pixels between frames to a point).
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ellenfeld in view of Xu and Din, and further in view of Lei et al. (US2023156179A1), hereinafter referenced as Lei.
Regarding claim 20, Ellenfeld in view of Xu and Din discloses: The method of claim 19.
Ellenfeld in view of Xu and Din does not disclose expressly: wherein the camera has a frame rate of n frames per second, the frame of interest occurs at a time t, and the associated adjacent frames are frames that occur respectively at t-x and t+x, wherein x is at least 1/nth of a second thereby accounting for an inherent frame-rate latency of the camera.
Thus, Ellenfeld in view of Xu and Din does not disclose expressly: wherein the camera has a frame rate of n frames per second and there is at least a 1/n second delay between the frame of interest and its adjacent frames, in order to account for an inherent frame-rate latency of the camera.
Lei discloses: a monitoring system configured for receiving video feeds from cameras, and determining whether frame anomalies, such as missed, delayed, or stale frames were identified (Lei: 0003). Wherein for a camera has a frame rate of 30 frames per second and there is at least a 1/30 second delay between each frame received, in order to account for an inherent frame-rate latency of the camera (Lei: 0030: “As shown in scenario (A), in a normal state, the monitoring system 124 receives all the frames from camera 106 A sequentially at 30 frames per second, and the time delta between two consecutive sequence numbers frames is 33 milliseconds for each video (e.g., a delay of one frame in 30 fps video corresponds to 1/30th of a second or to 33.3 milliseconds of latency). ”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the algorithms for monitoring the delay between video frames taught by Lei for the monitoring of the video streams disclosed by Ellenfeld in view of Xu and Din. The suggestion/motivation for doing so would have been “Such a system may allow for the detection of network anomalies such as missing frames, delayed frames, or stale frames in the video feed. A remote driver may be alerted with network anomalies and may take action accordingly. ” (Lei: 0014; Wherein frame delays, causing inaccuracies in data processed, may detected and acted upon). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Ellenfeld in view of Xu and Din with Lei to obtain the invention as specified in claim 20.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672