DETAILED ACTION
Allowable Subject Matter
Claims 5-9 are objected to as being dependent upon a rejected base claim but would be allowable if the 112 rejection is overcome, and rewritten in independent form including all of the limitations of the base claim and any intervening claims.
NOTE: Claims are 101 eligible because, although they recite mathematical processing through convolution, the claim as a whole is directed to a specific technological implementation for processing image information using artificial neural networks rather than to the mathematical concept itself. The claim requires calculating window set information based on selected feature map, thereby integrating the mathematical operation into a practical application that improves how the neural network analyzes image and select image features.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-7, 10-12 and 14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. In this case, the claims state numerical values such as 1-2, 1-2, 3-1, 3-2 which renders the claim indefinite because it’s unclear what they are or what they represent. For examination purposes, the examiner interpreted the numbers as adjectives: for example, 1-1 feature map as first feature map; 1-2 feature map as being second feature map. etc…
Also “selected final feature map” should have the same terminology as the statement it is referring to above (“determining one of the 3-1 feature map and the 3-2 feature map”), or else, it raises a lack of antecedent basis issue.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 4, 12 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Mantey et al. (Pub. No. US 2021/0110564) in view of Wu et al. (Pub. No. US 2021/0224581) further in view of Liang et al. (Patent No. US 11,423,644).
Regarding claim 14, Mantey teaches device for automatically calculating window set information (window polygon) using artificial neural networks (image segmentation model) [Para. 49 “Accordingly, the image segmentation tool predicts the structural layout elements of the building as output. Examples of the structural layout elements are doorways, windows, stairs, garage doors, and building extents.” and Para. 43 “For example, there is a window polygon, a doorway polygon, a stairs polygon, a garage door polygon, a building extent polygon, etc.” comprising: a pre-trained (para. 34-35) artificial neural network module calculating window set information having a first input information comprising image information (camera view images) as input information (image to be segmented) [Para. 49; In this case, the claim is broadly interpreted to mean “pretrained neural network module receiving an input image”].
However, Montey doesn’t explicitly teach a window set information extracting window set characteristics from the image information as output information; and a feature extractor for extracting a 1-1 feature map and a 1-2 feature map having different scale values for the image information, in the artificial neural network module calculating window set information,
Wu teaches a window set information extracting window set characteristics (feature map) from the image information as output information [para. 34]; and a feature extractor (backbone network) for extracting a 1-1 feature map (high level feature maps) and a 1-2 feature map (low level feature maps) having different scale values (lower resolutions and higher resolutions) for the image information, in the artificial neural network module calculating window set information [Para. 34]; wherein the artificial neural network module calculating window set information is generating a 3-1 feature map and a 3-2 feature map (final feature of each feature layer) by applying convolution (3x3 convolution) to the 1-1 feature map and the 1-2 feature map, respectively [Para. 48; because convolution is separately performed on each feature layer, two differently scaled feature maps produce corresponding convolution final features].
It would have been obvious to one of ordinary skill in the art before the effective filing ate to modify Mantey’s image segmentation model by incorporating Wu’s teaching of extracting feature maps having different resolutions and applying convolution to the respective feature maps to obtain corresponding final features. This modification improves Mantey’s system by permitting image features corresponding to the objects at different scales to be represented and processed at appropriate resolution. Thereby improving object detection accuracy across variations in object size and image scale.
Mantey in view Wu doesn’t explicitly teach determining one of the 3-1 feature map and the 3-2 feature map as a final feature map based on a preset criteria.
However, Liang teaches determining one of the 3-1 feature map and the 3-2 feature map (most suitable feature map level) as a final feature map based on a preset criteria (predetermined criteria) [Col. 12 lines 1-5].
It would have been obvious to one of ordinary skill in the art before the effective filing ate to modify Mantey’s image segmentation model, modified by Wu, by incorporating Liang’s teaching of selecting a most suitable feature map level based on predetermined criteria. This modification improves Montey’s system by selecting a feature representation appropriate to the scale of the object while reducing unnecessary resampling computation, thereby proving more efficient scale-appropriate processing of detected building features such as windows.
Mantey teaches then calculating (predict value) the window set information [Para. 43 The class prediction value for each individual polygon indicates the type of feature (e.g., windows, doorways, stairs etc.) identified by the polygon. For example, there is a window polygon, a doorway polygon, a stairs polygon, a garage door polygon, a building extent polygon, etc].
However, Mantey in view of Wu doesn’t explicitly teach then calculating (predict value) the window set information based on the selected final feature map.
However, Liang teaches determined fined (most suitable feature map level) final feature map [Col. 12 lines 1-5].
It would have been obvious to one of ordinary skill in the art before the effective filing ate to modify Mantey’s image segmentation model, modified by Wu, by incorporating Liang’s teaching of selecting a most suitable feature map level based on predetermined criteria. This modification improves Montey’s system by selecting a feature representation appropriate to the scale of the object while reducing unnecessary resampling computation, thereby proving more efficient scale-appropriate processing of detected building features such as windows.
Mantey also teaches wherein the artificial neural network module calculating window set information is performing training using a labeling information (ground truth information) labeled for the window set information (window polygon) included in the image information [Para. 43 “the segmented features (in the form of polygons drawn around features) are provided as ground truth information (e.g., as input) to train the image segmentation model. The segmented features shown in FIG. 2 correspond to the above-mentioned five features/physical layout elements of the building. During the training phase, ground truth information is fed to the model so that the model can recognize features in new images during a subsequent prediction stage. As a result, the trained model can indicate features of the building by drawing polygons around the features.”].
Claims 1, 3, and 12 are rejected for the same reason as claim 14 above.
Regarding Claim 4 Mantey in view of Wu further in view of Liang teaches the device for automatically calculating window set information using artificial neural networks according to The device for automatically calculating window set information using artificial neural networks according to wherein the feature extractor is extracting the 1-2 feature map having a different scale value with the scale value of the 1-lfeature map from the first pre-trained artificial neural network [see claim 14 rejection].
Claims 2 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Mantey et al. (Pub. No. US 2021/0110564) in view of Wu et al. (Pub. No. US 2021/0224581) further in view of Liang et al. (Patent No. US 11,423,644) further in view of Hao et al. (Pub. No. US 20210209368).
Regarding claim 2 and 11, Mantey teaches the device for automatically calculating (estimate value) window set information using artificial neural networks according to The device for automatically calculating window set information using artificial neural networks according to wherein the artificial neural network module calculating window set information is calculating information for the 3-1feature map and the 3-2 feature map and then determining the feature map having a higher value as the final feature map according to the preset criteria [Para. 43 The class prediction value for each individual polygon indicates the type of feature (e.g., windows, doorways, stairs etc.) identified by the polygon. For example, there is a window polygon, a doorway polygon, a stairs polygon, a garage door polygon, a building extent polygon, etc].
However, Montey in view of Wu further in view of Liang everything, as stated above, except determining the feature map having a higher confidence value as the final feature map according to the preset criteria.
Hao teaches determining the feature map having a higher confidence value as the final feature map according to the preset criteria [Para. 22 “For example, a feature map for a frame may include a confidence feature map (or confidence information) and a localization feature map (or localization information). The confidence feature map indicates what is detected (e.g., a class: person, building, car, background etc.) and the confidence in the detection (e.g., a score ranging from 0 to 1)..”; para. 24 “In other embodiments, the object detection results are generated by comparing shared object patches and retaining the prediction result having the higher confidence”; Para. 38 “object detection localization and confidence scoring for shared object patches between first and second feature maps are retained based on higher confidence scoring for each of the shared object patches to generate a bidirectional enhanced feature map for the current frame”].
It would have been obvious to one of ordinary skill in the art before the effective filing ate to modify Mantey’s image segmentation model, modified by Wu and Liang, by incorporating Hao’s teaching of calculating confidence information for the candidate feature maps and selecting the higher confidence result is selected for final window processing. This modification improves Mantey’s modified system by causing the window calculation to use the feature representation having the greater detection confidence, thereby improving the reliability of the resulting window detection.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Mantey et al. (Pub. No. US 2021/0110564) in view of Wu et al. (Pub. No. US 2021/0224581) further in view of Liang et al. (Patent No. US 11,423,644).
Claim 10 teaches extracting 1-1 and 1-2 feature maps discloses the claimed invention except extracting additional 1-3 feature map. It would have been obvious to one having ordinary skill in the art at the time the invention was made to apply the same extracting method to generate different feature maps multiple times, since it has been held that mere duplication of the essential working part of a device involves only routine skill in the art. St. Regis Paper Co. v. Bemis Co., 193 USPQ 8; In re Harza, 274 F.2d 669, 124 USPQ 378 (CCPA 1960).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Mantey et al. (Pub. No. US 2021/0110564) in view of Wu et al. (Pub. No. US 2021/0224581) further in view of Liang et al. (Patent No. US 11,423,644) further in view of Du et al. (Pub. No. US 2020/0348132).
Regarding claim 15, Mantey in view of Wu further in view of Liang don’t explicitly teach the claim limitations.
However, Du teaches wherein the artificial neural network module calculating window set information is performing training (trained by using) by using a second input information including the labeling information (ground truth images) and the first input information (training images) together as input information [Para. 215].
It would have been obvious to one of ordinary skill in the art before the effective filing ate to modify Mantey’s image segmentation model, modified by Wu and Liang, by incorporating Du’s teaching of training a CNN-based AI engine by inputting training images (first input information) together with corresponding ground truth image (labeling information), thereby improving learned feature detection parameters and subsequent detection of labeled image objects such as windows.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOLOMON G BEZUAYEHU whose telephone number is (571)270-7452. The examiner can normally be reached on Monday-Friday 10 AM-7 PM..
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-0101 (IN USA OR CANADA) or 571-272-1000.
/SOLOMON G BEZUAYEHU/ Primary Examiner, Art Unit 2666
