DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/07/2026 has been entered.
Response to Amendment
5. In light of Applicant’s amendment of the claims, the claim objections with respect to claims 4-12 are withdrawn.
Status of Claims
Claims 1-14 are pending. Claims 1, 2 and 4-14 are amended.
Response to Arguments
Applicant’s amendment of independent Claim 1, which has altered the scope of the claims of the instant application, has necessitated the new ground(s) of rejection presented in this office action with respect to claims of the instant application. Accordingly, in response to Applicant’s arguments that are merely directed to the amended portion of the claims, new analyses have been presented below, which make Applicant’s arguments moot.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-7 and 9-14 are rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US 2020/0160124 A1), in view of Sriram et al. (US 2020/0151489 A1) and in further view of Liao et al. (US 2021/0201071 A1).
Regarding claim 1, Fu teaches, A method comprising: extracting (Fu, ¶0113: “method in the subject matter described herein, comprising: extracting”) key local landmarks from an input image; the key local landmarks representing (Fu, ¶0033: “sub-network 222 performs the feature extraction at a finer scale than the full-image scale, the extracted feature 223 may be referred to as a local feature”) that, when considered together, are indicative of a structure of an object in the input image; (Fu, ¶0022: “extracting features from each of the regions and recognizing a specific category of the object based on these features”) global feature representations (Fu, ¶0037: “global feature information of the image 170, and thus may be combined with the local feature 223”) produced by the deep CNN model (Fu, ¶0034: “sub-network 222 may be comprised of a CNN network”) to create combined feature representations. (Fu, ¶0046: “the local feature 333 may be combined with the global feature 213”). However, Fu does not explicitly teach, mapping the pixel locations of the key local landmarks into a feature map of an intermediate convolutional layer of a deep CNN model at positions in the intermediate convolutional layer corresponding to the pixel locations of the key local landmarks from the input image; extracting local feature representations of the key landmarks from the feature map of the intermediate convolutional layer of the deep CNN model containing the mapped key local landmarks.
In an analogous field of endeavor, Sriram teaches, mapping the pixel locations of the key local landmarks (Sriram, ¶0031: “the bounding shapes may be computed by the object detector 106 as pixel locations”) into a feature map of an intermediate convolutional layer of a deep CNN model at positions in the intermediate convolutional layer (Sriram, ¶0082: “CNN 112D may more accurately account for the portions of the feature map computed by the feature extractor 504 that correspond to the person region 518, the object region 520, and the union region 522”) corresponding to the pixel locations of the key local landmarks (Sriram, ¶0072: “object coordinates (e.g., coordinates corresponding to the object region, such as the bounding shape delineating the object region”) from the input image; (Sriram, ¶0033: “For a given object detected within an instance of the sensor data 102, pixel coordinate(s) corresponding to the object may be used”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fu using the teachings of Sriram to introduce mapping landmark pixel locations representing the boundary/structure of an object in a CNN based feature map. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of classification of an object using key pixel coordinates representing the object. Therefore, it would have been obvious to combine the analogous arts Fu and Sriram to obtain the above-described limitations in claim 1. However, the combination of Fu and Sriram does not explicitly teach, extracting local feature representations of the key landmarks from the feature map of the intermediate convolutional layer of the deep CNN model containing the mapped key local landmarks.
In another analogous field of endeavor, Liao teaches, extracting local feature representations of the key landmarks from the feature map of the intermediate convolutional layer of the deep CNN model (Liao, ¶0091: “feature map is extracted from the fifth convolutional layer of the learning network 510 even though the feature map from any other layer may be used”) containing the mapped key local landmarks. (Liao, ¶0032: “objects contained in the image such as persons, animals, natural features, buildings and the like”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fu in view of Sriram using the teachings of Liao to introduce extracting features from an intermediate convolutional layer. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of obtaining multiscale representation of an object for improved object classification. Therefore, it would have been obvious to combine the analogous arts Fu, Sriram and Liao to obtain the invention in claim 1.
Regarding claim 2, Fu in view of Sriram and in further view of Liao teaches, The method of claim 1, further comprising: sending the combined feature representations to a classifier to be used (Fu, ¶0037: “FC layer 217 is concatenated to the output of the FC layer 227… input to a further Softmax layer”) to classify the object in the input image from which the key local landmarks were extracted. (Fu, ¶0037: “input to a further Softmax layer (not shown) to determine the category of the object in the image”).
Regarding claim 3, Fu in view of Sriram and in further view of Liao teaches, The method of claim 1, further comprising: selecting a subset of the local feature representations (Fu, ¶0084: “second attention region is comprised in the first attention region and comprises a discriminative sub-portion of the object in the image”) to be combined with the global feature representations. (Fu, ¶0046: “the local feature 333 may be combined with the global feature 213 and/or local feature 223 to determine the category of the object in the image 170”).
Regarding claim 4, Fu in view of Sriram and in further view of Liao teaches, The method of claim 3, wherein the subset of local feature representations is selected based on a weighting scheme wherein a predetermined number of higher-weighted local feature representations are selected. (Sriram, ¶0039: “generate a plurality of confidences associated with each object/person pair, and the highest confidence 116 may be used”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fu, in view of Sriram, in further view of Liao, using the additional teachings of Sriram to introduce confidence scoring. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of selecting the objects with highest confidence. Therefore, it would have been obvious to combine the analogous arts Fu, Sriram and Liao to obtain the invention in claim 4.
Regarding claim 5, Fu in view of Sriram and in further view of Liao teaches, The method of claim 4, wherein the weighting scheme is a learned weighting scheme. (Fu, ¶0024: “accurate discriminative portion localization can promote learning fine-grained features, which in turn can further help to accurately localize the discriminative portions”).
Regarding claim 6, Fu in view of Sriram and in further view of Liao teaches, The method of claim 5, wherein the learned weighting scheme assigns weights depending on the ability of the local feature representations to discriminate (Sriram, ¶0033: “a confidence 116 may be determined for each object”) between objects in the input image belonging to different subclasses. (Fu, ¶0021: “for different species of birds, the differences may lie in the colors and/or patterns of their necks, backs or tails, the shapes and/or colors of their beaks or claws, or the like. Such a portion that is applicable to determine a specific category of an object may be referred to as a discriminative portion of the object”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fu, in view of Sriram, in further view of Liao, using the additional teachings of Sriram to introduce assigning confidence based on accuracy of detection. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of only selecting the feature representations with higher detection accuracy. Therefore, it would have been obvious to combine the analogous arts Fu, Sriram and Liao to obtain the invention in claim 6.
Regarding claim 7, Fu in view of Sriram and in further view of Liao teaches, The method of claim 4, wherein the predetermined number is a learned number. (Sriram, ¶0048: “the parameters may be learned by the machine learning model(s) 112 during training”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fu, in view of Sriram, and in further view of Liao, using the additional teachings of Sriram, to introduce a learned parameter. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of learning to optimize a selection parameter during training. Therefore, it would have been obvious to combine the analogous arts Fu, Sriram and Liao to obtain the invention in claim 7.
Regarding claim 9, Fu in view of Sriram and in further view of Liao teaches, The method of claim 3, wherein the subset of local feature representations is selected based on explicit knowledge of a domain of objects depicted in the input image. (Fu, ¶0052: “if the learning network structure in FIG. 2 or 3 is to be trained as being capable of recognizing a plurality of species of birds, the training images may include images of different species of birds”).
Regarding claim 10, Fu in view of Sriram and in further view of Liao teaches, The method of claim 1, wherein the local feature representations are combined with the global feature representations by concatenation. (Fu, ¶0037: “FC layer 217 is concatenated to the output of the FC layer 227”).
Regarding claim 11, Fu in view of Sriram and in further view of Liao teaches, The method of claim 1, wherein the key local landmarks in the input image are mapped to the feature map (Fu, ¶0035: "after a third convolutional layer of the deep CNN model. (Li, ¶0091: “feature map is extracted from the fifth convolutional layer of the learning network 510 even though the feature map from any other layer may be used”). The proposed combination as well as the motivation for combining Fu, Sriram and Liao references presented in the rejection of claim 1, apply to claim 11 and are incorporated herein by reference. Thus, the method recited in claim 11 is met by Fu, Sriram and Liao.
Regarding claim 12, Fu in view of Sriram and in further view of Liao teaches, The method of claim 1, wherein extracting key local landmarks from an input image comprises exposing the input image to a CNN model trained with a dataset comprising images with annotated landmarks. (Fu, ¶0022: “determining specific regions in the image by known bounding boxes or region annotations in a supervised fashion; then, extracting features from each of the regions and recognizing a specific category of the object based on these features”).
Regarding claim 13, Fu in view of Sriram and in further view of Liao teaches, A system, comprising: a processor; and memory, storing software that, when executed by the processor, performs the method of claim 1. (Fu, ¶0105: “The device comprises: a processing unit; and a memory coupled to the processing unit and having instructions stored thereon. The instructions, when executed by the processing unit, cause the device to perform acts”).
Regarding claim 14, Fu in view of Sriram and in further view of Liao teaches, A system, comprising: a processor; and memory, storing software that, when executed by the processor, performs the method of claim 4. (Fu, ¶0105: “The device comprises: a processing unit; and a memory coupled to the processing unit and having instructions stored thereon. The instructions, when executed by the processing unit, cause the device to perform acts”).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US 2020/0160124 A1), in view of Sriram et al. (US 2020/0151489 A1), in further view of Liao et al. (US 2021/0201071 A1) and still in further view of Iwamoto et al. (US 2014/0328543 A1).
Regarding claim 8, Fu in view of Sriram, in further view of Liao teaches, The method of claim 7, (Fu, ¶0021: “for different species of birds, the differences may lie in the colors and/or patterns of their necks, backs or tails, the shapes and/or colors of their beaks or claws, or the like. Such a portion that is applicable to determine a specific category of an object may be referred to as a discriminative portion of the object”). However, the combination of Fu, Sriram and Liao does not explicitly teach, wherein the predetermined number is learned based on an optimal number of local feature presentations needed.
In an analogous field of endeavor, Iwamoto teaches, wherein the predetermined number is learned based on an optimal number of local feature presentations needed (Iwamoto, ¶0100: “the selection number determining unit 50 may be configured so as to determine the number of feature points and the number of dimensions so that at least one of the number of feature points and the number of dimensions is reduced”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fu, in view of Li and in further view of Liao using the teachings of Iwamoto, to introduce selecting an optimal number of feature points. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of increasing the speed of operation. Therefore, it would have been obvious to combine the analogous arts Fu, Sriram, Liao and Iwamoto to obtain the invention in claim 8.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MEHRAZUL ISLAM whose telephone number is (571)270-0489. The examiner can normally be reached Monday-Friday: 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Saini Amandeep can be reached at (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MEHRAZUL ISLAM/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights