DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-10 are pending for examination in the Application No. 18/899,097 filed September 27th, 2024.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed as foreign Patent Application No. DE102023126724.8, filed on September 29th, 2023.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on September 27th, 2024, is in compliance with the provisions of 37 CFR 1.97. Accordingly, the IDS is being considered and attached by the examiner.
Response to Amendment
Applicant’s amendments filed November 26th, 2024, to the Specification has been entered.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 6-7, and 10 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ngo et al. (Ngo; “A Room Monitoring System Using Deep learning and Perspective Correction Techniques,” 2020; provided by Applicant in the IDS filed September 27th, 2024).
Regarding claim 1, Ngo discloses a computer-implemented method, comprising:
obtaining an image depicting a plurality of instances of an object (section 3.3.2. on pg. 6, recite(s)
[3.3.2. Object Detection and a Person Class Filter] “This section presents the object detection and the person class filter on each tile image and the full-frame one. We apply the Mask R-CNN scheme to detect objects on each tile image as well as the full-frame one. …”
, where the “full-frame one” is an image depicting a plurality of instances of an object (e.g., “objects”));
breaking down the image into multiple tiles, wherein pairs of neighboring tiles of the multiple tiles comprise respective overlap regions (reference signs “
W
t
i
l
e
” are tiles and “
d
i
n
t
e
r
s
e
c
t
” are overlap regions as depicted in Fig. 4 on pg. 7:
PNG
media_image1.png
364
455
media_image1.png
Greyscale
);
processing the multiple tiles using an instance segmentation algorithm, to obtain respective instance boundaries for the plurality of instances of the object (2nd para. of section 3.1 on pg. 5 and Fig. 6 on pg. 8:
[2nd para. of section 3.1] “… After initial detection using the Mask R-CNN, the bounding boxes for each tile image and the full-size images are collected as the initial results. …”
PNG
media_image2.png
381
661
media_image2.png
Greyscale
, where “Mask R-CNN” is an instance segmentation algorithm and “bounding boxes” are instance boundaries); and
for each of the pairs of neighboring tiles and for each instance boundary truncated by a border of a tile of the respective pair: selectively merging the respective instance boundary truncated by the border of the tile with a further instance boundary positioned in a further tile of the respective pair depending on an overlap degree between the respective instance boundary truncated by the border of the tile and the further instance boundary (Fig. 6 on pg. 8—see preceding citation immediately above—, where the 2nd para. of section 3.3.3 on pg. 8 further recite(s):
[2nd para. of section 3.3.3] “Figure 6 shows an instance of an object residing on the border. The IoU and BTA values must be considered to merge the neighboring bounding boxes.…”
, where an “Object residing on the border of two tiles” is an instance boundary truncated by a border of a tile of a pair of neighboring tiles; and merging the “bounding boxes of each tile image” is selectively merging the respective instance boundary truncated by the border of the tile with a further instance boundary positioned in a further tile of the respective pair depending on an overlap degree (e.g., “IoU”—i.e. Intersection Over Union)).
Regarding claim 2, Ngo discloses the computer-implemented method of claim 1, wherein said selectively merging comprises:
upon determining that the overlap degree is above a first predefined threshold: merging the respective instance boundary truncated by the border of the tile and the further instance boundary (step 5 detailing Algorithm 1 on pg. 9, recite(s)
[step 5 detailing Algorithm 1] “5. Lines (13–17) show that if IoU value of the coupled neighboring bounding boxes is greater than the IoU threshold, these two coupled neighboring bounding boxes can be merged to create a new bounding box.”
, where the “IoU threshold” is a first predefined threshold).
Regarding claim 6, Ngo discloses the computer-implemented method of claim 1, further comprising:
determining, based on the instance boundaries for the plurality of instances of the object, the pairs of neighboring tiles in which at least one instance boundary is truncated by the border of a tile of the respective pair (Fig. 6 on pg. 8 and detailed in the 2nd para. of section 3.3.3 on pg. 8—see citation claim 1 limitation “for each pairs of neighboring tiles…” above—, where determining “neighboring bounding boxes” for an “object residing on the border of two tiles” is determining the pairs of neighboring tiles).
Regarding claim 7, Ngo discloses the computer-implemented method of claim 1, wherein the overlap degree is determined within the respective overlap region of the tile and the further tile of the respective pair (the “IoU” of an “Object residing on the border of two tiles” depicted in Fig. 6 on pg. 8 and detailed in the 2nd para. of section 3.3.3 on pg. 8—see citation claim 1 limitation “for each pairs of neighboring tiles…” above).
Regarding claim 10, the claim recites similar limitations to claim 1 but in the form of a computing device, the device comprising a processor and a memory. Ngo discloses said computing device… comprising a processor and a memory (the 1st para. of section 4 on pg. 11, recite(s)
[1st para. of section 4] “…The system is implemented on a computer with an Intel Core i7 CPU 3.70 GHz, 64-bit Windows operating system, 32 GB RAM, and NVIDIA TITAN V GPU. The program is developed using Anaconda3 v5.2.0 as the integrated development environment and Python v3.6.5 as the programming language.”
, where the “computer” is a computing device comprising at least a processor (e.g., “GPU”) and memory (e.g., “RAM”)). Therefore, claim 10 is rejected for similar rationale and reasoning as claim 1 (see the analysis for claim 1 above).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 3 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Ngo as applied to claim 1 above, and further in view of Brouard et al. (Brouard; US 2020/0242357 A1).
Regarding claim 3, Ngo discloses the computer-implemented method of claim 1, wherein Brouard teaches in the same field of endeavor of instance segmentation said selectively merging comprises:
upon determining that the overlap degree is below a first predefined threshold and is above a second predefined threshold: selecting either the respective instance boundary truncated by the border of the tile or the further instance boundary, and removing the selected one of the of the respective instance boundary truncated by the border of the tile and the further instance boundary (para(s). [0036] and [0068], recite(s)
[0036] “…An implementation of the invention avoid these shortcomings by dividing large input images into image blocks. In addition, the auxiliary/supplemental data may also be divided into blocks. As such, multiple instances of the same CNN model and the same post-model filter may be deployed independently at a block level into distributed and parallel computing platforms/machines (e.g., distributed physical machines and/or virtual machines). In some implementations, the image blocks may be constructed such that neighboring blocks overlap with one another to some extent. …”
[0068] “As another example illustrated by process 516 of FIG. 5, swimming pools labeled in the labeled aerial block may be removed if they are detected to overlap with a building more than a predetermined percentage threshold. In particular, the CNN model may falsely identify solar panels installed on roofs as swimming pools, as shown by 620 of FIG. 6 (where solar panels 624 on the roof of building 622 may be recognized and segmented by the CNN model as a swimming pool). As such, the process 516 facilitates improving labeling accuracy by removing such false positives. A filter criterion of overlapping percentage may be pre-established. In one implementation, a minimum overlapping percentage threshold may be predetermined for such filtering. For example, a labeled swimming pool is removed as false positive only when it overlaps with a building by at least, e.g., 70%. The percentage threshold may not be need to be 100% or close to 100%. A percentage threshold lower than 100% may provide some tolerance to CNN model inaccuracy. In particular, a set of solar panels installed on the roof of a residential building may be recognized as swimming pool by the CNN model. The segmentation by the CNN model may not be completely accurate, leading to a detected overlapping between the falsely identified swimming pool and the residential building to be less than 100%. Using a percentage threshold of 100% or close to 100%, such solar panel falsely identified as swimming pools may not be considered false positive and the process 516 may fail to remove such false swimming pool identification. Likewise, the percentage threshold may be predetermined to a value that is not too small. Otherwise, a real swimming pool by a residential building may be removed inadvertently due to segmentation inaccuracy of the CNN model. In some implementations, the minimum percentage threshold value for the filtering and removal process 516 may be set at a value between 30% and 70%.”
, where “more than a predetermined [minimum overlapping] percentage threshold” is an overlap degree above a second predefined threshold and a “percentage threshold lower than 100%” is an overlap degree below a first predefined threshold; where a “block may be removed if they are detected to overlap… more than a percentage threshold” is removing a selected one of the respective instance boundary (e.g., an image “block”) truncated (i.e., “overlapping”) by the border of the tile and the further instance boundary (e.g., “neighboring blocks overlap”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Ngo to incorporate selectively merging a selected instance boundary by removing the selected instance boundary based on an overlap degree below a first predefined threshold and above a second predefined threshold to prevent false instance boundary segmentations and inadvertently removing an instance boundary from segmentation as taught by Brouard above.
Regarding claim 9, Ngo discloses the computer-implemented method of claim 1, wherein Brouard teaches in the same field of endeavor of instance segmentation the image is a microscopy image or a satellite image (para(s). [0003], recite(s)
[0003] “…The trained final CNN models are augmented with post-model filters to enhance prediction accuracy by removing false positive object recognition and segmentation. The system and method provide improved accuracy to predict object labels to append to unlabeled image blocks in digital images. The system may be useful for enhancing, e.g., a digital landmark registry, by appending identifying labels on new infrastructure improvements recognized in aerial or satellite land images.”
, where an input image for instance segmentation is at least an “aerial or satellite land image[s]”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to try substituting the input image for instance segmentation in the system of Ngo as at least a satellite image because Brouard also discloses performing instance segmentation on images to yield the predictable result of segmenting instances by processing multiple image tiles comprising at least truncated instance boundaries between neighboring image tiles (Brouard; para. [0056], recite(s)
[0056] “The post-model filtering process flow and data pipeline 211 may further include process 260 for aggregating the filtered labels of various aerial blocks and for removing duplicate labels of landmarks in the overlap regions. The output of the aggregation and deduplication process 260 is final labeled orthophotos 206. An exemplary implementation of the deduplication process 260 for landmarks across block boundaries is illustrated in process 900 of FIG. 9. Specifically, the aerial blocks labeled and filtered by landmark recognition and segmentation process flow and data pipeline 210 and post-model filtering process flow and data pipeline 211 may be scanned for duplicates at boundaries by process 260 sequentially. For example, aerial blocks 902 and 904 illustrate blocks that have been sequentially scanned (labels across block boundaries have been deduplicated) by process 260 up to a current aerial block 906. The process 260 then proceed to scan the current block 906 by detecting duplicate landmarks across boundaries between block 906 and its neighboring blocks 904. In some implementations, duplication labels across block boundaries may be identified by evaluating overlap between recognized boundary boxes of landmarks in overlap regions of the aerial blocks. If two boundary boxes overlap at all or overlap more than a predetermined absolute or relative (e.g., percentage) threshold level in an overlap region of the aerial blocks, they may be merged as one boundary box in the final labeled orthophotos 206. The scanning process in 900 progresses until all aerial blocks are processed. The aerial blocks may be scanned in any order such that all the overlapping regions between the aerial blocks are processed.”
).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Ngo in view of Brouard as applied to claim 3 above, and further in view of Lee et al. (Lee; “Multi-Class Object Detection Using Adaptive Non-Maximum Suppression in Dense Images,” 2022).
Regarding claim 4, Ngo in view of Brouard discloses the computer-implemented method of claim 3, wherein Lee teaches in the same field of endeavor of selectively merging in instance segmentation said selectively merging comprises:
determining a first confidence value associated with the respective instance boundary truncated by the border of the tile and determining a second confidence value associated with the further instance boundary (the 2nd to last para. of col. 1 of pg. 3 and sections 3.2 and 3.3 on pg. 5, recite(s)
[the 2nd to last para. of col. 1 of pg. 3] “However, in the NMS algorithm, when multiple objects are closely overlapping, the object detection performance is likely to be degraded by removing the correctly predicted bounding box by setting the probability score
s
i
=
0
as shown in (1). To improve the algorithm of NMS, a Soft-Non-Maximum Suppression (Soft-NMS) algorithm has been proposed. In Soft-NMS, when multiple objects are closely overlapping, the probability score is lowered but not zero, so that the bounding box and the probability score maintained for the further process.”
[3.2 on pg. 5] “The bounding box is sorted in descending order based on the probability score. Starting with the bounding box with the highest probability score, the
i
o
u
(
·
)
of the sorted bounding box is calculated with respect to the adjacent bounding box. …where
A
represents a bounding box closer to the ground truth, and
B
represents an adjacent bounding box of
A
. The probability score
s
i
of the
B
bounding box is updated according to the conditions of the
i
o
u
(
A
,
B
)
and
N
t
. …”
[3.3 on pg. 5] “
C
S
represents the Confidence Score of each bounding box of n number of extracted bounding boxes. The calculation of our Confidence Score threshold
C
S
t
is relatively straightforward as defined in Eq. (7) and (8). Among the detected bounding boxes when an object is detected has a probability score
n
'
>
0.1
that bounding box is used. …”
, where the “probability score” and/or “Confidence Score” for “each bounding box” is determining at least a first confidence value associated with a respective instance boundary (e.g., bounding box “
A
”) and a second confidence value associated with the further instance boundary (e.g., “adjacent bounding box” “
B
”),
wherein said selecting of either the respective instance boundary truncated by the border of the tile or the further instance boundary is based on a comparison of the first confidence value and the second confidence value (section 3.4 on pg. 5, recite(s)
[3.4 on pg. 5] “If
C
S
corresponding to each bounding box is more significant than
C
S
t
, it is recognized as an object, and the bounding box is maintained. If it is smaller than
C
S
t
, it is treated as an overlapped bounding box and then deleted. The remained bounding box is displayed as an object box.”
, where “maintain[ing]” and/or “delet[ing]” one of the adjacent “bounding box[es]” is selecting either instance boundary based on a comparison of the confidence values (e.g., “smaller” and/or “more significant”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Ngo in view of Brouard to incorporate selecting either the respective instance boundary truncated by the border of the tile or the further instance boundary based on a comparison of a determined first confidence value associated with the respective instance boundary and second confidence value associated with the further instance boundary to minimize the false removal of positive instance boundaries while removing false positive instance boundaries as taught by Lee (section 3 on pg. 3, recite(s)
[section 3 on pg. 3] “Detecting closely overlapped objects in an image is a difficult problem. Therefore, setting
N
t
to the appropriate value is key to obtaining high accuracy results. When com paring results from out experiments we found that
N
t
=
0.50
performs poorly because of the removal of positive bounding boxes on a closely overlapped image or a small image.
We present an adaptive version of the Soft-NMS algorithm (e.i., Adaptive-NMS) to find the optimum Intersection over Union (IoU) and Confidence Score (CS) thresholds that can minimize the false removal of positive bounding boxes. …”
).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Ngo as applied to claim 1 above, and further in view of Audebert et al. (Audebert; “Segment-before-Detect: Vehicle Detection and Classification through Semantic Segmentation of Aerial Images,” 2017; provided by Applicant in the IDS filed September 27th, 2024) .
Regarding claim 5, Ngo discloses the computer-implemented method of claim 1, wherein Audebert teaches in the same field of endeavor of instance segmentation further comprising:
upon determining that one or more artifacts are produced after said merging of the respective instance boundary truncated by the border of the tile and the further instance boundary: removing the respective merged instance boundary truncated by the border of tiles (section 3.2 on pgs. 4-5, recite(s)
[section 3.2] “Assuming that we will work on VHR aerial images on which a human observer can distinguish cars, the semantic maps predicted by SegNet should be accurate enough to avoid the merging of neighboring cars into a single blob. If this hypothesis is verified, finding vehicle instances in the pixel-level mask is only a matter of extracting connected components. Then, it is possible to regress the bounding box of the vehicle under the mask.
However, predictions from SegNet can be noisy, as CNN tends to have blurred transitions between classes [34]. Therefore, to alleviate perturbations in the predictions coming out of the network, we first operate morphological opening with a small radius to erode the vehicle mask. Second, we eliminate the objects smaller than a threshold to remove potential false positives due to segmentation artifacts such as vents on roofs or clutter on the street that might have been misclassified as vehicles. Despite its simplicity, this morphological opening, combined with the connected component extraction, is enough to perform efficient vehicle detection.”
, where “remov[ing] objects smaller than a threshold” after “merging” of instance boundaries is removing a respective instance boundary upon determining that one or more artifacts are produced after said merging of the instance boundaries (e.g., “remove potential false positives due to segmentation artifacts”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Ngo to incorporate removing the respective merged instance boundary truncated by the border of tiles upon determining that one or more artifacts are produced after said merging of the respective instance boundary truncated by the border of the tile and the further instance boundary to remove potential false positive instance boundaries due to the instance segmentation algorithm as taught by Audebert above.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Ngo as applied to claim 1 above, and further in view of Lee et al. (Lee; “Multi-Class Object Detection Using Adaptive Non-Maximum Suppression in Dense Images,” 2022).
Regarding claim 8, Ngo discloses the computer-implemented method of claim 1, wherein Lee teaches in the same field of endeavor of instance segmentation further comprising:
for each instance boundary truncated by the border of the tile of the respective pair: determining a confidence value (the 2nd to last para. of col. 1 of pg. 3 and sections 3.2 and 3.3 on pg. 5—see citation in claim 4 limitation “determining a first confidence value…” above—, where determining a “probability score” and/or “Confidence score” of each bounding box for a “bounding box” and its “adjacent bounding box” is determining a confidence value for each instance boundary truncated (e.g., “overlapping”) by the border of the tile of the respective pair);
for each of the pairs of neighboring tiles: selecting at least one instance boundary with the highest confidence value (section 3.4 on pg. 5—see citation in claim 4 limitation “wherein said sleeting of either…” above—, where “maintain[ing]” the bounding box that is “more significant
C
S
t
[Confidence Score threshold]” is selecting at least one instance boundary with the highest (e.g., “more significant”) confidence value);
wherein said selectively merging is further based on the selected at least one instance boundary (section 3.4 on pg. 5—see citation in claim 4 limitation “wherein said sleeting of either…” above—, where “maintain[ing]” the “more significant” instance boundary and “delet[ing]” the “smaller” confidence score instance boundary is selectively merging based on the selected at least one instance boundary (e.g., the “bounding box” with the “more significant
C
S
t
”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Ngo to incorporate selecting at least one instance boundary with the highest confidence value for each pairs of neighboring tiles and selectively merging based on the selected at least one instance boundary to minimize the false removal of positive instance boundaries while removing false positive instance boundaries as taught by Lee (section 3 on pg. 3—see citation in the motivation to combine Lee in claim 4 above).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JULIA Z YAO whose telephone number is (571)272-2870. The examiner can normally be reached Monday - Friday (8:30AM - 5PM).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571)270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.Z.Y./Examiner, Art Unit 2666
/MING Y HON/Primary Examiner, Art Unit 2666
Prosecution Insights