DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 are rejected under 35 U.S.C. 103 as being unpatentable Chen et al (US 2019/0130580) in view of Bjerge et al (An automated light trap to monitor moths (Lepidoptera) using computer vision based tracking and deep learning) .
As to claim 1,Chen et al teaches a method comprising:
removing a first background element of one or more background elements of a first image (Blob detection can utilize background subtraction to determine a background portion of a scene and a foreground portion of scene. Blobs can then be detected based on the foreground portion of the scene. Blob bounding regions (e.g., bounding boxes or other bounding region) can be associated with the blobs, in which case a blob and a blob bounding region can be used interchangeably. A blob bounding region is a shape surrounding a blob, and can be used to represent the blob, paragraph [005], the blob detection system 104 can perform background subtraction for a frame, and can then detect foreground pixels in the frame. Foreground blobs are generated from the foreground pixels using morphology operations and spatial analysis(paragraph [0142])); generating a second image based on the removing of the first background element of the one or more background elements ( A first bounding box from the blob bounding boxes 1324 and a second bounding box from the detector bounding boxes 1323 can be determined to be associated with one another when a large portion of the area of the first or second bounding box belongs to an overlapping area of the first and second bounding boxes. The overlapping area can include an intersecting region, which includes a region that includes the overlapping portion of the first bounding box and the second bounding box, paragraph [0221])
remove a first background element of the one or more background elements( The process 1800 further includes determining the category of the object is not a category of interest, and removing the bounding region from the first set of one or more bounding regions in response to determining the category of the object is not a category of interest, paragraph [0270])
detecting one or more objects included in the second image (The process 1800 further includes determining that, during object tracking for the video frame, a tracker associated with the bounding region will be output in response to determining the confidence value associated with the bounding region is above the confidence threshold, paragraph[0273])).
generating, based on the second image, a spatial partition model associated with understanding a z-index and at least a partially obscured area while preserving relative sizes of the one or more objects (([0221] "The overlapping area can include an intersecting region, which includes a region that includes the overlapping portion of the first bounding box and the second bounding box", [0226-0228] etc., concerning explicitly disclosed overlap, and Chen further suggests an appearance similarity should either ofat least object size/dimensions, and/or object class, be considered a characteristic of appearance'). Chen further discloses in e.g. [0228] that a threshold for percentage of overlap may be scaled based on situation specific confidences, expected object sizes, etc..). Chen teaches Fig. 13 1325-1326, [0220] "For a current key frame, a final set of bounding boxes 1326 can be determined using detector bounding boxes (or "high confidence bounding boxes")produced by the deep learning system 1208 and foreground bounding boxes produced by the blob detection system 1204 for the current key frame. For example, the foreground bounding boxes 1224 generated by the blob detection system
1204 and the detector bounding boxes 1323 generated by the deep learning system 1208
(in some cases, filtered from the pre-processing module) are output to the bounding box
aggregation engine 1325. The bounding box aggregation engine 1325 can aggregate the
blob bounding boxes 1323 and the blob bounding boxes 1324 (which can include lists of
bounding boxes BB Detector and BBBg Sub, respectively) to produce the final set of
bounding boxes 1326 for a current key frame. The final set of bounding boxes 1326 for a
key frame can include a final list of bounding boxes, which can be denoted as BB Final. In
some examples, a status can also be determined for each of the bounding boxes in the
final set of bounding boxes 1326. Each of the bounding boxes in the final set 1326 can
represent a blob detected for the video frame", [0221] "A first bounding box from the
blob bounding boxes 1324 and a second bounding box from the detector bounding
boxes 1323 can be determined to be associated with one another when a large portion of
the area of the first or second bounding box belongs to an overlapping area of the first
and second bounding boxes. The overlapping area can include an intersecting region,
which includes a region that includes the overlapping portion of the first bounding box
and the second bounding box", [0228]). While Chen teaches the limitations above, Chen fails to teach “ determining, based on the spatial partition model, one or more foreground elements associated with the second image that obscures a capture area in the second image; removing a first foreground element of the one or more foreground elements; and displaying a third image with the first foreground element of the one or more foreground elements removed. “
However, Bjerge et al teaches computer vision algorithm referred to as Moth Classification and
Counting (MCC), based on deep learning analysis of the captured images, tracked and counted the
number of insects and identified moth species. Observations over 48 nights resulted in the capture of
more than 250,000 images with an average of 5675 images per night. A customized convolutional
neural network was trained on 2000 labeled images of live moths represented by eight different
classes, achieving a high validation F1-score of 0.93. The algorithm measured an average classification
and tracking F1-score of 0.71 and a tracking detection rate of 0.79. Overall, the proposed computer
vision system and algorithm showed promising results as a low-cost solution for non-destructive
and automatic monitoring of moths ( abstract) .Bjerge et al teaches a grayscaled foreground image was made by subtracting a fixed background image of the white sheet without insects. The methods of using a global threshold or regions of adaptive threshold segmentation were investigated to perform segmentation of a black and white image. The Otsu [28] threshold algorithm turned out to be the best choice. A black and white image was made using Otsu threshold on the foreground image, followed
by a morphological open and close operation to filter small noisy blobs and closing of blobs. Finally, the contour of blobs was found, and the bounding box of insect regions was estimated. In rare cases, when two or more individuals were too close together, the estimation of the position based on bounding boxes was unsuccessful. A result of segmentation and bounding box estimation is shown in Figure 4. ( page 5 and 6) . It would have been obvious to one skilled in the art before the effective filing date to modify the system of Chen in order to improve insects’ population monitoring systems and to accelerate the study of moth populations and minimize the impact on their populations. Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
As to claim 2, Bjerge et al teaches the method of claim 1, further comprising: detecting one or more objects included in the third image; filtering the detected one or more objects based on a predefined acceptance criteria associated with one or more classes generating count statistics associated with the detected one or more objects in the one or more classes; and transmitting the count statistics ((Bjerge Table 4, 6, Figs. 8-9, etc., in view of section(s) 3.2 and 3.3 in particular "The seasonal dynamics of each of the eight species of moths detected by the MCC algorithm showed clear similarities with peak abundance during the last days of August 2019 (Figure 8). However, Autograph a gamma observations were restricted to mostly a single night, while other species were frequent during several consecutive nights. Some species including the Hoplodrina complex exhibited a second smaller peak
in abundance around 10 September2019. This variation was mirrored to some extent in the weather patterns from the same period (Figure 9). The statistical models of the relationships with weather patterns revealed that abundance of all species was positively related to night temperatures, while three species also responded to wind speed, and another three species responded to both air humidity and wind speed (Table 6)").
As to claim 3, Chen et al teaches the method of claim 1, wherein the first image is captured by a sensor (IP camera device or other suitable device used to capture video sequences of a scene, paragraph[202]).
As to claim 4, Chen et al teaches the method of claim 1, wherein the z-index is associated with a two-layer z-index ( paragraph [0148][335]).
As to claim 5, Chen et al teaches the method of claim 1, wherein the z-index is associated with a full z-index( paragraph [0148][335])..
As to claim 7, Bjerge et al teaches the method of claim 1, wherein the one or more foreground elements comprise a strut, a condition associated with lighting, or dust ("Our novel image processing pipeline incorporates the temporal dimension of image sequences", Fig. 2 at page 4 of 18 "Figure 2. A picture of3840X 2160 pixels taken by the light trap of the light table and nine resting moths", Fig. 3 Input
Image, page 5 Section 2.2 "The MCC algorithm was composed by a number of sequential steps, where each image in the recording was analyzed as illustrated in Figure 3".
As to claim 7, Chen et al teaches the method of claim 1, wherein the one or more background elements comprise one or more grid lines or a texture of the capture area These bounding boxes are weighted by the predicted probabilities. For example, as shown in FIG. 9A, the YOLO detector divides up the image into a grid of 13-by-13 cells, paragraph [0200]).
As to claim 8, Chen et al teaches the method of claim 1, wherein the generating the second image based on the removing of the first background element of the one or more background elements is further based on whether a color of the first background element is approximately a same color of the one or more objects in the one or more classes (The background subtraction engine 312 can generate a foreground mask with foreground pixels based on the result of background subtraction. For example, the foreground mask can include a binary image containing the pixels making up the foreground objects (e.g., moving objects) in a scene and the pixels of the background. In some examples, the background of the foreground mask (background pixels) can be a solid color, such as a solid white background, a solid black background, or other solid color. In such examples, the foreground pixels of the foreground mask can be a different color than that used for the background pixels, such as a solid black color, a solid white color, or other solid color. In one illustrative example, the background pixels can be black (e.g., pixel color value 0 in 8-bit grayscale or other suitable value) and the foreground pixels can be white (e.g., pixel color value 255 in 8-bit grayscale or other suitable value). In another illustrative example, the background pixels can be white and the foreground pixels can be black, paragraph [0149-0150]).
As to claim 9, Chen et al teaches the method of claim 1, wherein the one or more background elements comprises one or more portions attached to the capture area ( When blobs (making up at least portions of objects) are detected from an input video frame, blob trackers from the previous video frame need to be associated to the blobs in the input video frame according to a cost calculation. The blob trackers can be updated based on the associated foreground blobs. In some instances, the steps in object tracking can be conducted in a series manner., paragraph [0148-0150][0165]).
The limitation of claims 10-20 has been addressed above.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NANCY BITAR whose telephone number is (571)270-1041. The examiner can normally be reached Mon-Friday from 8:00 am to 5:00 p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ms. Jennifer Mehmood can be reached at 571-272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
NANCY . BITAR
Examiner
Art Unit 2664
/NANCY BITAR/Primary Examiner, Art Unit 2664
Prosecution Insights