DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/23/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The drawings are objected to because Figure 11 shows multiple elements labeled as ‘gg’, where the elements appear to be referring to different things. Paragraph 23 of the Specification describes ‘ggm’, which seems to suggest that there are multiple ‘gg’ images. Paragraph 21 of the Specification further specifies “ggm (m = 1, ..., M). Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-5 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “relatively high possibility” in claims 1-5 is a relative term which renders the claim indefinite. The term “relatively high possibility” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. Accordingly, the limitations including this relative term are rendered indefinite. “Relatively high possibility” is thus being interpreted as any probability.
Claim 1 recites two instances of retrieving the same images, rendering it unclear at which point in the process the images are retrieved. The claim recites, “when K (K is an integer equal to or more than one) gallery images having a relatively high possibility of including a subject included in the query image are retrieved from the plurality of the gallery images”, and in the next step, “retrieve the K gallery images from the plurality of the gallery images on a basis of the feature vector of the acquired query image and the feature vector of each of the gallery images”. This is being interpreted as the second instance is defining how the retrieval in the first instance is performed, rather than a second discrete retrieval step.
Claims 2-4 are rejected as dependent on claim 1.
Claim 5 is rejected for analogous reasons to the above rejection of claim 1.
Claim 3 recites the limitation “give the query image to the second learning model and acquire reliability of the group". There is insufficient antecedent basis for this limitation in the claim. Claim 3 describes that learning images are grouped by reliability (“each of the learning images is given and the reliability for a group including each of the learning images is given”). However, the groups introduced in the latter instance appear to only exist for learning images, making it unclear what “the group” is in reference to with respect to the query image. Accordingly, “the group” is being interpreted as a new element (i.e. a second group).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 and 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Doumbouya (US20190130202A1) in view of Li (Difficulty Guided Image Retrieval Using Linear Multiple Feature Embedding).
Regarding claim 1, Doumbouya teaches “An image retrieving device comprising: processing circuitry” (Doumbouya, Figure 1.)
“configured to give a query image that is an image to be identified to a first learning model, acquire a feature vector of the query image from the first learning model, give each of a plurality of gallery images that are images to be identified to the first learning model, and acquire a feature vector of each of the gallery images from the first learning model;” (Doumbouya, Paragraphs 102 and 104, “Before proceeding with the method shown in FIG. 14, the server system 108 determines feature vectors of the query image and the gallery image. As described above, a feature vector is an n-dimensional vector of numerical features (numbers) that represent an image of an object that can be processed by computers. By comparing the feature vector of one image of one object with the feature vector of another image, a computer implementable process may determine whether the one image and the other image are images of the same object. The image signatures (or feature vectors, or embedding, or representation, etc.) are multi-dimensional vectors calculated by (for example convolutional) neural networks.”; “In this example implementation, the server system 108 uses a learning machine to process the query image and the gallery image to generate the feature vectors or signatures of the images of the persons-of-interest captured in the video. The learning machine is for example a neural network such as a convolutional neural network (CNN) running on a graphics processing unit (GPU) or vision processing unit (VPU). The CNN may be trained using training datasets containing millions of pairs of similar and dissimilar images. The CNN, for example, is a Siamese network architecture trained with a contrastive loss function to train the neural networks. An example of a Siamese network may be described in Bromley, Jane, et al. “Signature verification using a “Siamese” time delay neural network.” International Journal of Pattern Recognition and Artificial Intelligence 7.04 (1993): 669-688.” Note that there are a plurality of gallery images (see abstract, Paragraph 6).
While Doumbouya discloses retrieval of gallery images having a relatively high possibility of including a subject included in the query image from the plurality of gallery images, (Doumbouya, Paragraph 121,“At block 1450, the server system 108 repeats blocks 1410, 1420 and 1440 for each other gallery image identified in the recorded video (or portion of recorded video which the user wishes to have analyzed), thereby obtaining a fused similarity score for each gallery image identified in the recorded video/portion of recorded video. At block 1460, the server system 108 ranks the gallery images according to their respective fused similarity scores, and at block 1470 instructs display of the ranked gallery images on the display 126 of the computer terminal 104. The ranked gallery images correspond to the image search results 406 in FIG. 4.” Note that images with high similarity score are more likely to include the same subject.) Doumbouya does not expressly disclose “give the query image to a second learning model, and acquire, from the second learning model, reliability of retrieval” when the similar gallery images to the query image are retrieved, and “specify the reliability of retrieval from the acquired reliability.”
Li teaches “give the query image to a second learning model, and acquire, from the second learning model, reliability of retrieval” when similar image search results are received, and “specify the reliability of retrieval from the acquired reliability.” (Li, Section III, Paragraphs 1-3, “Fig. 2 illustrates the framework of proposed QDE integrated image retrieval system, which contains a linear multiple feature embedding module and a difficulty model learning module in the offline part, an image query difficulty estimation module and difficulty guided retrieval applications module in the online part. In the offline part, a linear transformation is derived from LME, by mining a joint representation from multiple image features. Firstly, the images in a specified feature space are correlated by their visual similarities into a graph, which models the locality structure of the images in the corresponding feature space. Based on the Laplacian embedding, we linearly combine different graphs together, and compute the optimal linear transformation by jointly utilizing information from different spaces (see Section V). In the offline part, we also learn a difficulty model with the low embeddings of labeled queries and their initial search results. In the query difficulty estimation module, the high-dimensional features of the image query and its initial retrieval results are firstly mapped into the subspace learned by LME. Since it is a linear transformation, the computation cost is low enough to meet the online requirement. To calculate the difficulty, we mine the correlations among the query image and its top returned images, as well as the database images, in the embedded feature space. These correlations are constructed as difficulty features and query difficulty is estimated with difficulty model trained in the offline part (see Section IV).” Note that the higher difficulty, the lower the reliability of the results. Further note that estimating the query difficulty (search reliability) amounts to both acquiring and specifying of the reliability.)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to use the learning model for acquiring and specifying reliability of retrieval based on query and result images, taught by Li, using the query image and the K gallery images retrieved by Doumbouya.
The motivation for doing so is outlined by Li (Li, Section I Paragraph 4, “QDE is important and valuable for CBIR from three perspectives. A retrieval system could adaptively invoke appropriate retrieval strategies according to queries’ difficulties. Users may benefit from QDE by refining query images according to QDE scores to restate their search intention. Finally, QDE could be a useful clue for the collection selection and the results merge in federated image search.” Note that CBIR is ‘content based image retrieval’, and QDE is ‘query difficulty estimation’.) Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Doumbouya with the above teachings of Li to fully disclose, “give the query image to a second learning model, and acquire, from the second learning model, reliability of retrieval when K (K is an integer equal to or more than one) gallery images having a relatively high possibility of including a subject included in the query image are retrieved from the plurality of the gallery images;” and “and specify the reliability of retrieval from the acquired reliability.”
Doumbouya in view of Li further disclose “retrieve the K gallery images from the plurality of the gallery images on a basis of the feature vector of the acquired query image and the feature vector of each of the gallery images;” (Doumbouya, Paragraphs 92 and 121, “Similarity calculation can be just an extension of the above. Specifically, by calculating the Euclidean distance between two feature vectors of two images captured by one or more of the cameras 169, a computer implementable process can determine a similarity score to indicate how similar the two images may be, as described in further detail below.”; “At block 1450, the server system 108 repeats blocks 1410, 1420 and 1440 for each other gallery image identified in the recorded video (or portion of recorded video which the user wishes to have analyzed), thereby obtaining a fused similarity score for each gallery image identified in the recorded video/portion of recorded video. At block 1460, the server system 108 ranks the gallery images according to their respective fused similarity scores, and at block 1470 instructs display of the ranked gallery images on the display 126 of the computer terminal 104. The ranked gallery images correspond to the image search results 406 in FIG. 4.” Therefore, the feature vectors are used to compute the similarity score, which determines the retrieval of the similar gallery images, which are then displayed as shown in Figure 4.)
Regarding claim 5, claim 5 recites a method with steps corresponding to the elements of the system recited in Claim 1. Therefore, the recited steps of this claim are mapped to the analogous elements in the corresponding system claim. Additionally, the rationale and motivation to combine the Doumbouya and Li references, presented in rejection of Claim 1, apply to this claim.
Allowable Subject Matter
Claims 2-4 are rejected under 35 USC 112(b) and objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and amended to overcome 35 USC 112(b) rejections.
The following is a statement of reasons for the indication of allowable subject matter: With respect to claims 2-4, in addition to other limitations in the claims the Prior Art of Record fails to teach, disclose or render obvious the applicant' s invention as claimed, in particular:
Claim 2 recites: “The image retrieving device according to claim 1, wherein the second learning model is a learning model in which each of learning images that are a plurality of images for learning included in a learning image group is sequentially given as a reference image, and learning of the reliability is performed when the reliability of retrieval at a time when K learning images having a relatively high possibility of including a subject included in the reference image are retrieved from among learning images other than the reference image included in the learning image group is given as teacher data.”
Claim 3 recites: “The image retrieving device according to claim 1, wherein learning images, which are a plurality of images for learning, are grouped by the reliability, the second learning model is a learning model in which learning of the reliability is performed when each of the learning images is given and the reliability for a group including each of the learning images is given as teacher data, the processing circuitry is further configured to give the query image to the second learning model and acquire reliability of the group as the reliability of retrieval when K gallery images having a relatively high possibility of including a subject included in the query image are retrieved from the second learning model, and specify the reliability of retrieval from the acquired reliability of the group.”
Claim 4 recites: “The image retrieving device according to claim 1, wherein each of learning images that are a plurality of images for learning included in a learning image group is sequentially set as a reference image, a degree of similarity between each reference image and each learning image other than the reference image included in the learning image group is represented by a distance between a position of the reference image in an image space and a position of each of the learning images in the image space, and each of the learning images is classified into any one of a plurality of distance classes by a distance to the reference image, the second learning model is a learning model in which learning of the reliability is performed when each of the reference images is given and the reliability for a plurality of distance classes is given as teacher data, the processing circuitry is further configured to give the query image to the second learning model and acquire reliability for a plurality of distance classes as the reliability of retrieval when K gallery images having a relatively high possibility of including a subject included in the query image are retrieved from the second learning model, and acquire reliability of a distance class including K gallery images retrieved from among the acquired reliability of the plurality of distance classes and specifies the reliability of the retrieval from the reliability acquired for the distance classes.”
Doumbouya teaches matching of a query image to gallery images based on matching feature vectors extracted from the images. Li teaches a method of analyzing the difficulty of a particular image query search based on the search results and a trained model. Tian (Exploration of Image Search Results Quality Assessment) discloses strategies for assessing the quality of retrieved search results in response to a query based on a preference learning model. Geng (Content-Aware Ranking for Visual Search) teaches a ranking strategy for image search results in which both textual and visual features are used in the ranking. Alcock (US 20200082212 A1) teaches a method for improving speed of similarity searches that includes calculating a Euclidian distance between feature vectors of a query image and images of a database. Dhua (US 10176198 B1) teaches a method for identifying visual similarities that includes extracting and comparing feature vectors of query images and images in a dataset, wherein confidence scores are generated that indicate match likelihood. However, none of these references disclose the bolded limitations above. The closest reference is Li, which discloses the ‘second machine learning model’ of claims 1 and 5. However, the model training of Li is expressly described as differing significantly from the training methods implemented in claims 2-4.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AARON JOSEPH SORRIN whose telephone number is (703)756-1565. The examiner can normally be reached Monday - Friday 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON JOSEPH SORRIN/
Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672