DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Applicant’s response to the Non-final Office Action dated 04/22/2026, filed with the office on 07/21/2026, has been entered and made of record.
Status of Claims
Claims 1-15 are pending. Claim 1 is amended. Claim 15 is new.
Response to Arguments
Applicant’s amendment of independent Claim 1, which has altered the scope of the claims of the instant application, has necessitated the new ground(s) of rejection presented in this office action with respect to claims of the instant application. Accordingly, in response to Applicant’s arguments that are merely directed to the amended portion of the claims, new analyses have been presented below, which make Applicant’s arguments moot.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7 and 9-14 are rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US 2020/0160124 A1), in view of Price et al. (US 2017/0140236 A1) and in further view of Sriram et al. (US 2020/0151489 A1).
Regarding claim 1, Fu teaches, A method comprising: extracting (Fu, ¶0113: “method in the subject matter described herein, comprising: extracting”) key local landmarks from an input image, (Fu, ¶0033: “extract one or more local features 223, and each local feature 223 may indicate partial feature information indicative of the image”) and combining all or some of the plurality of local feature representations with global feature representations (Fu, ¶0037: “global feature information of the image 170, and thus may be combined with the local feature 223”) produced at a deepest layer of the deep CNN model to create combined feature representations. (Fu, ¶0037: “output of the FC layer 217 is concatenated to the output of the FC layer 227 of the classification sub-network 226 of the learning network 220 and is input to a further Softmax layer (not shown) to determine the category of the object in the image 170. The Softmax layer is also referred to as a fully-connected fusion layer”). However, Fu does not explicitly teach, each key local landmark representing a single pixel in the original image that, when considered together, are spatially indicative of a shape of an object in the input image; mapping the pixel locations of the key local landmarks into a feature map of an intermediate convolutional layer of a deep CNN model at positions in the intermediate convolutional layer corresponding to the pixel locations of the key local landmarks from the input image; extracting a plurality of local feature representations from the feature map of the intermediate convolutional layer of the deep CNN model containing the mapped key local landmarks, each local feature representation representing a single location in the intermediate convolution layer to which a key local landmark was mapped
In an analogous field of endeavor, Price teaches, each key local landmark representing a single pixel in the original image (Price, ¶0085: “identify pixels belonging to a target object and select pixels belonging to the target object”) that, when considered together, are spatially indicative of a shape of an object in the input image; (Price, ¶0085: “identify an object boundary for the target object (e.g., by identifying pixels along the edge of the target object”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fu using the teachings of Price to introduce identifying a boundary of an object by identifying key pixels. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of classifying an object using the boundary shape of the object. Therefore, it would have been obvious to combine the analogous arts Fu and Price to obtain the above-described limitations in claim 1. However, the combination of Fu and Price does not explicitly teach mapping the pixel locations of the key local landmarks into a feature map of an intermediate convolutional layer of a deep CNN model at positions in the intermediate convolutional layer corresponding to the pixel locations of the key local landmarks from the input image; extracting a plurality of local feature representations from the feature map of the intermediate convolutional layer of the deep CNN model containing the mapped key local landmarks, each local feature representation representing a single location in the intermediate convolution layer to which a key local landmark was mapped;
In an analogous field of endeavor, Sriram teaches, mapping the pixel locations of the key local landmarks (Sriram, ¶0031: “the bounding shapes may be computed by the object detector 106 as pixel locations”) into a feature map of an intermediate convolutional layer of a deep CNN model at positions in the intermediate convolutional layer (Sriram, ¶0082: “CNN 112D may more accurately account for the portions of the feature map computed by the feature extractor 504 that correspond to the person region 518, the object region 520, and the union region 522”) corresponding to the pixel locations of the key local landmarks (Sriram, ¶0072: “object coordinates (e.g., coordinates corresponding to the object region, such as the bounding shape delineating the object region”) from the input image; (Sriram, ¶0033: “For a given object detected within an instance of the sensor data 102, pixel coordinate(s) corresponding to the object may be used”) extracting a plurality of local feature representations from the feature map of the intermediate convolutional layer of the deep CNN model containing the mapped key local landmarks, (Sriram, ¶0082: “pooling layers may reduce the size of the feature vectors or the feature map by some factor (e.g., a factor of 2, 3, etc.), to account for the potential of small movements in the position of features in the input image”) each local feature representation representing a single location (Sriram, ¶0037: “include pixel locations corresponding to vertices and/or other points along the bounding shapes”) in the intermediate convolution layer to which a key local landmark was mapped (Sriram, ¶0082: “pooling layers may be used to perform a pooling operation (e.g., a filter) on the feature vectors or feature maps”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fu in view of Price using the teachings of Sriram to introduce mapping landmark pixel locations representing the boundary in a feature map. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of classifying an object using CNN based on the features in the feature map. Therefore, it would have been obvious to combine the analogous arts Fu, Price and Sriram to obtain the invention in claim 1.
Regarding claim 2, Fu in view of Price and in further view of Sriram teaches, The method of claim 1, further comprising: sending the combined feature representations to a classifier to be used (Fu, ¶0037: “FC layer 217 is concatenated to the output of the FC layer 227… input to a further Softmax layer”) to classify the object in the input image from which the key local landmarks were extracted. (Fu, ¶0037: “input to a further Softmax layer (not shown) to determine the category of the object in the image”).
Regarding claim 3, Fu in view of Price and in further view of Sriram teaches, The method of claim 1, further comprising: selecting a subset of the local feature representations (Fu, ¶0084: “second attention region is comprised in the first attention region and comprises a discriminative sub-portion of the object in the image”) to be combined with the global feature representations. (Fu, ¶0046: “the local feature 333 may be combined with the global feature 213 and/or local feature 223 to determine the category of the object in the image 170”).
Regarding claim 4, Fu in view of Price and in further view of Sriram teaches, The method of claim 3, wherein the subset of local feature representations is selected based on a weighting scheme wherein a predetermined number of higher-weighted local feature representations are selected. (Sriram, ¶0039: “generate a plurality of confidences associated with each object/person pair, and the highest confidence 116 may be used”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fu, in view of Price and in further view of Sriram, using the additional teachings of Sriram to introduce confidence scoring. A person skilled in the art would be motivated to combine the known elements, as described above, and achieve the predictable result of selecting the objects with highest confidence. Therefore, it would have been obvious to combine the analogous arts Fu, Price and Sriram to obtain the invention in claim 4.
Regarding claim 5, Fu in view of Price and in further view of Sriram teaches, The method of claim 4, wherein the weighting scheme is a learned weighting scheme. (Fu, ¶0024: “accurate discriminative portion localization can promote learning fine-grained features, which in turn can further help to accurately localize the discriminative portions”).
Regarding claim 6, Fu in view of Price and in further view of Sriram teaches, The method of claim 5, wherein the learned weighting scheme assigns weights depending on the ability of the local feature representations to discriminate (Sriram, ¶0033: “a confidence 116 may be determined for each object”) between objects in the input image belonging to different subclasses. (Fu, ¶0021: “for different species of birds, the differences may lie in the colors and/or patterns of their necks, backs or tails, the shapes and/or colors of their beaks or claws, or the like. Such a portion that is applicable to determine a specific category of an object may be referred to as a discriminative portion of the object”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fu, in view of Price and in further view of Sriram, using the additional teachings of Sriram to introduce assigning confidence based on accuracy of detection. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of only selecting the feature representations with higher detection accuracy. Therefore, it would have been obvious to combine the analogous arts Fu, Price and Sriram to obtain the invention in claim 6.
Regarding claim 7, Fu in view of Price and in further view of Sriram teaches, The method of claim 4, wherein the predetermined number is a learned number. (Sriram, ¶0048: “the parameters may be learned by the machine learning model(s) 112 during training”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fu, in view of Price and in further view of Sriram, using the additional teachings of Sriram, to introduce a learned parameter. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of learning to optimize a selection parameter during training. Therefore, it would have been obvious to combine the analogous arts Fu, Price and Sriramto obtain the invention in claim 7.
Regarding claim 9, Fu in view of Price and in further view of Sriram teaches, The method of claim 3, wherein the subset of local feature representations is selected based on explicit knowledge of a domain of objects depicted in the input image. (Fu, ¶0052: “if the learning network structure in FIG. 2 or 3 is to be trained as being capable of recognizing a plurality of species of birds, the training images may include images of different species of birds”).
Regarding claim 10, Fu in view of Price and in further view of Sriram teaches, The method of claim 1, wherein the local feature representations are combined with the global feature representations by concatenation. (Fu, ¶0037: “FC layer 217 is concatenated to the output of the FC layer 227”).
Regarding claim 11, Fu in view of Price and in further view of Sriram teaches, The method of claim 1, wherein the key local landmarks in the input image are mapped to the feature map (Fu, ¶0035: "after a third convolutional layer of the deep CNN model. (Li, ¶0091: “feature map is extracted from the fifth convolutional layer of the learning network 510 even though the feature map from any other layer may be used”). The proposed combination as well as the motivation for combining Fu, Price and Sriram references presented in the rejection of claim 1, apply to claim 11 and are incorporated herein by reference. Thus, the method recited in claim 11 is met by Fu, Price and Sriram.
Regarding claim 12, Fu in view of Price and in further view of Sriram teaches, The method of claim 1, wherein extracting key local landmarks from an input image comprises exposing the input image to a CNN model trained with a dataset comprising images with annotated landmarks. (Fu, ¶0022: “determining specific regions in the image by known bounding boxes or region annotations in a supervised fashion; then, extracting features from each of the regions and recognizing a specific category of the object based on these features”).
Regarding claim 13, Fu in view of Price and in further view of Sriram teaches, A system, comprising: a processor; and memory, storing software that, when executed by the processor, performs the method of claim 1. (Fu, ¶0105: “The device comprises: a processing unit; and a memory coupled to the processing unit and having instructions stored thereon. The instructions, when executed by the processing unit, cause the device to perform acts”).
Regarding claim 14, Fu in view of Price and in further view of Sriram teaches, A system, comprising: a processor; and memory, storing software that, when executed by the processor, performs the method of claim 4. (Fu, ¶0105: “The device comprises: a processing unit; and a memory coupled to the processing unit and having instructions stored thereon. The instructions, when executed by the processing unit, cause the device to perform acts”).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US 2020/0160124 A1), in view of Price et al. (US 2017/0140236 A1), in further view of Sriram et al. (US 2020/0151489 A1) and still in further view of Iwamoto et al. (US 2014/0328543 A1).
Regarding claim 8, Fu in view of Price and in further view of Sriram teaches, The method of claim 7, (Fu, ¶0021: “for different species of birds, the differences may lie in the colors and/or patterns of their necks, backs or tails, the shapes and/or colors of their beaks or claws, or the like. Such a portion that is applicable to determine a specific category of an object may be referred to as a discriminative portion of the object”). However, the combination of Fu, Price and Sriram does not explicitly teach, wherein the predetermined number is learned based on an optimal number of local feature presentations needed.
In an analogous field of endeavor, Iwamoto teaches, wherein the predetermined number is learned based on an optimal number of local feature presentations needed (Iwamoto, ¶0100: “the selection number determining unit 50 may be configured so as to determine the number of feature points and the number of dimensions so that at least one of the number of feature points and the number of dimensions is reduced”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fu in view of Price and in further view of Sriram using the teachings of Iwamoto, to introduce selecting an optimal number of feature points. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of increasing the speed of operation. Therefore, it would have been obvious to combine the analogous arts Fu, Price, Sriram and Iwamoto to obtain the invention in claim 8.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US 2020/0160124 A1), in view of Price et al. (US 2017/0140236 A1), in further view of Sriram et al. (US 2020/0151489 A1) and still in further view of Liao et al. (US 2021/0201071 A1).
Regarding claim 15, Fu in view of Price and in further view of Sriram teaches, The method of claim 1. However, the combination of Fu, Price and Sriram does not explicitly teach wherein the feature map of the intermediate convolutional layer is the same resolution as the input image.
In an analogous field of endeavor, Liao teaches, wherein the feature map of the intermediate convolutional layer is the same resolution as the input image. (Liao, ¶0047: “feature maps obtained from different convolutional layers may be of different sizes due to downsampling, and may be converted to have the same size by upsampling or downsampling the feature maps”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fu in view of Price and in further view of Sriram using the teachings of Liao to introduce up sampling a feature map to preserve the resolution. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of performing segmentation and object detection tasks based on the feature map. Therefore, it would have been obvious to combine the analogous arts Fu, Price, Sriram and Liao to obtain the invention in claim 15.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MEHRAZUL ISLAM whose telephone number is (571)270-0489. The examiner can normally be reached Monday-Friday: 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Saini Amandeep can be reached at (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MEHRAZUL ISLAM/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
