DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested:
SYSTEMS AND METHODS FOR IMAGE SEARCH COMPRISING GENERATING A QUERY OUTPUT BASED ON FIRST CANDIDATE IMAGES BASED ON A QUERY VECTOR BASED ON A QUERY IMAGE AND SECOND CANDIDATE IMAGES BASED ON RECOGNIZED TEXT STRINGS ON THE QUERY IMAGE.
Election/Restrictions
Applicant’s election without traverse of Invention I, Species I in the reply filed on 7/27/2026 is acknowledged.
Claims 10-20 are withdrawn from further consideration pursuant to 37 CFR 1.142(b) as being drawn to a nonelected Inventions II and III, there being no allowable generic or linking claim. Election was made without traverse in the reply filed on 7/27/2026.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1 and 4-8 is/are rejected under 35 U.S.C. 102(a1 and/or a2) as being anticipated by Koyama (US Pub 2010/0191769).
Re claim 1: (Original) A method for image search, the method comprising:
receiving a query image (e.g. the invention discloses an input of an image, which is taught in ¶ [19].);
[0019] By reference to FIG. 1, a structure of an image search apparatus according to the exemplary embodiment will be described. The image search apparatus searches and retrieves, from among registered images which are registered in an existing image database or image log storage device (not shown), a registered image having a high degree of similarity with respect to an input document image 100. The input document image 100 may be, for example, image data obtained by reading a paper document by a scanner, or a bitmap image obtained by converting an image file created by any of various types of application programs. Upon receiving the input document image 100, the image search apparatus supplies the input image document 100 to each of a photograph image search section 10 and a text search section 20.
recognizing one or more text strings on the query image (e.g. the system discloses recognizing the text within the image input, which is taught in ¶ [25]-[29].);
[0025] The text search section 20 will next be described. The text search section 20 is a section for searching for a similar registered image using the feature of a text character string existing in the input document image 100, and is typically implemented by execution of a computer program. The text search section 20 includes a character recognition section 22, a word extraction section 24, and a search processing section 26.
[0026] The character recognition section 22 recognizes characters contained in the input document image 100 by means of an OCR (optical character recognition) algorithm or an OCR circuit. Here, conventional OCR algorithms or OCR circuits may be used.
[0027] The word extraction section 24 performs known natural language analysis such as morphologic analysis with respect to a character string which is output from the character recognition section 22, to thereby extract words appearing in the input document image 100. Thus, the word extraction section 24 outputs data concerning a set of words contained in the input document image 100, for example. Preferably, the data concerning the set of words include, for each word, information concerning the number of times the word appears within the input document image 100. Further, words to be extracted may be limited to specific word classes (for example, to nouns only).
[0028] The search processing section 26 searches through a text DB 40 using the set of appearing words obtained by the word extraction section 24 and calculates the similarity score of each registered image with respect to the input document image 100.
[0029] Here, a list of document IDs of registered images in each of which a certain word appears is registered in the text DB 40, with each word being used as an index. The text DB 40 may be created as follows. Specifically, at the time of registering an image of each registered image in the image database, the log, or the like, character recognition and word extraction are performed concerning each registered image, and the document ID of the registered image is registered in a column of the text DB 40 having a word which is extracted as an index. Here, in view that a single word is highly likely to appear a plurality of times in a registered image, preferably, in addition to simple registration of the document ID in association with each index word, the number of times the word appears in the registered image may be also registered.
generating a query vector based on the query image (e.g. the system discloses extracting image feature amount of the input image. Figure 2 shows an example of a list of numbers representing a two-dimensional distribution of edge amount as an image feature amount. This list of numbers that uses numerical values to describe a feature of an image is considered as an image feature vector, which is described in ¶ [21] and [22].);
[0021] An image feature amount extraction section 12 of the photograph image search section 10 extracts an image feature amount of the input document image 100. As the image feature amount, a two-dimensional distribution of edge amounts may be used, for example. More specifically, as shown in FIG. 2, an image 200 of a predetermined size is divided into a predetermined number (e.g., 8.times.8, 16.times.16, 32.times.32) of blocks 210 of a predetermined size. An edge extraction filter is then applied to the image 200 to calculate an edge amount for each block 210, and a combination; i.e. a distribution of the edge amount of each blocks 210, is obtained as the image feature amount. This method which uses the edge amount distribution is suitable for a photograph image captured by a digital camera or the like. Alternatively, it is also possible to obtain an average color, rather than the edge amount, for each block and use a combination (or distribution) of the average color of each block as the image feature amount. In addition, it is further possible to binarize an image and obtain a distribution of a ratio of black pixels in each block 210 as the image feature amount. As such, various types of image feature amounts and corresponding scoring methods have been conventionally proposed, and the photograph image search section 10 may use any of these image feature amounts and the corresponding scoring methods. Further, distribution of contour line features or frequency features obtained for each block, may be used as the image feature amount for matching. In addition, any combination of two or more image feature amounts described above may be used for image matching.
[0022] Here, the above-described image feature amounts for use in the photograph image search section 10 have a comparatively simple algorithm and may therefore be calculated at high speed even with software processing. Further, calculation of such an image feature amount and matching processing using the image feature amount may also be implemented by a hardware circuit and is therefore suitable for high-speed processing.
conducting a first image search based at least in part on the query vector to generate one or more first candidate images (e.g. the system discloses performing an image search based on the two dimensional distribution of edge amounts in order to generate candidate images that are similar to the input image, which is shown in figure 7, step 10 and taught in ¶ [23], [24], [77] and [78].);
[0023] A feature amount matching section 14 calculates a degree of similarity between the image feature amount (or a combination of multiple image feature amounts) of the input document image 100 which is obtained by the image feature amount extraction section 12 and the image feature amount (or a combination of multiple image feature amounts) of each registered image stored in the image database or image log (both not shown). The image feature amount of each registered image has been calculated using the same algorithm as used in the image feature extraction section 12 and is then registered in a feature amount DB (database) 30 at the time of registering each image in the image database or the image log. Specifically, the image feature amount (such as a distribution of edge amounts described above) of each registered image is registered in the feature amount DB 30 in association with a document ID (identification information) of the registered image. The feature amount matching section 14 calculates, for each registered image, a similarity score indicative of the level of similarity between the image feature amount of the registered image and the image feature amount of the input document image 100. The similarity score may be calculated by means of a conventionally known method.
[0024] As described above, the photograph image search section 10 calculates and outputs, for each registered image, a similarity score (i.e. a score based on the image feature amount) of the registered image with respect to the input document image 100. This similarity score is represented as a first score 110 in FIG. 1. As such, the photograph image search section 10 outputs, for each registered image, a pair consisting of the document ID of the registered image and the first score 110, for example, which is then supplied to a candidate extraction section 50.
[0077] Referring to FIG. 7, a further example of the image search apparatus will be described. In FIG. 7, elements which are identical with or similar to those shown in FIG. 1 are denoted by the same reference numerals, and their descriptions will not be repeated.
[0078] The apparatus of this modified example adopts a structure in which the photograph image search section 10 provided at the first stage, the text search section 20 provided at the second stage, and the document image search section 52 provided at the third stage are arranged in series. With this structure, the photograph image search section 10 first performs search to extract first candidates which are similar to the input document image 100 from among all the registered images, and then the text search section 20 checks each of the first candidate registered images against the input document image 100 using text search and extracts second candidates having high scores from among the first candidates.
[0079] Here, with regard to the search performed by the text search section 20, it is also advantageous to combine the similarity scores obtained by the text search with the similarity scores based on the images features which are received from the previous stage and narrow the search for obtaining the second candidate registered images on the basis of the resulting combined scores, rather than narrowing the search on the sole basis of the similarity scores obtained by the text search. With this structure, when the input document image 100 contains a large amount of text, registered images which are more similar to the input document image can be preferentially extracted due to the search performed by the text search section, and even when the input document image 100 contains a smaller amount of text, a reduction in the search precision can be suppressed by adding the similarity scores obtained by the search based on the image features. Then, the document image search section 52 finally performs matching based on a projection waveform between each of the second candidate registered images and the input document image 100, thereby providing the final search result.
conducting a second image search based at least in part on the one or more recognized text strings to generate one or more second candidate images (e.g. the text strings are used to check initial candidates from the first list of candidates in order to generate second candidate images, which is taught in ¶ [78] above.); and
generating a query output based on the one or more first candidate images and the one or more second candidate images (e.g. a list of documents are output to show the most relevant search results based on the text candidates and the image candidates. This is taught in ¶ [78], [79] above, [61] and [62].);
[0060] The document image search section 52 obtains a similarity score based on a projection waveform for each candidate as described above, and outputs a list listing a pair consisting of the document ID of each candidate and the corresponding similarity score, in descending order of similarity score. Here, the list to be output may include only a predetermined number of documents with high-ranked similarity scores or only the documents whose similarity scores are equal to or greater than a predetermined value.
[0061] Further, it is also possible to add the combined score obtained in the candidate extraction section 50 to the similarity score based on a projection waveform which is obtained by the document image search section 52 to thereby obtain a second combined score, rank each candidate in accordance with the second combined score, and create a list of the candidates according to this ranking. The second combined score can be calculated by performing the above-described normalization with respect to the similarity score based on a projection waveform and then combining the value of normalized result and the combined score obtained from the candidate extraction section 50 as described above.
[0062] A search result output section 60 receives the list thus created from the document image search section 52, sorts the listed registered images in descending order of combined score value, and outputs them as search results. In this manner, it is possible to provide a search result 150 in which the registered images are arranged in descending order of similarity with respect to the input image a.
wherein the method is performed using one or more processors (e.g. the invention can be performed with a CPU and memory, which is taught in ¶ [82].).
[0082] The embodiment and the examples of the present invention have been described. The image search apparatuses described above are typically implemented by executing, in a general-purpose computer, a program in which the function or the processing content (which will be described in detail below) of each section of the computer system is described. Such a computer has a circuit structure in which a CPU (central processing unit) 80, a memory (primary storage) 82, various I/O (input/output) interfaces 84, or the like are connected via a bus 86. Further, a hard disk drive 88 and a disk drive 90 for reading portable, non-volatile storage media of various standards such as CDs, DVDs, or flash memories, are connected, via the I/O interface 84, for example, to the bus 86. Such a drive 88 or 90 functions as an external storage device with respect to the memory. Specifically, a program in which the processing content of the embodiment is described is stored, via a storage medium such as a CD, a DVD, or the like, or via the network, in a fixed storage device such as the hard disk drive 88, and then installed in the computer system. The program stored in the fixed storage device is then read out and stored in the memory and is further executed by the CPU, thereby achieving the processing of the embodiment.
Re claim 4: (Original) Koyama discloses the method of claim 1, wherein the recognizing one or more text strings on the query image comprises recognizing the one or more text strings using an optical character recognition (OCR) algorithm (e.g. an OCR algorithm is used to recognize text from an input image, which is taught in ¶ [26] above,).
Re claim 5: (Original) Koyama discloses the method of claim 1, wherein the query vector represents one or more characteristics of the query image (e.g. the image feature amount represents an edge characteristic of the input image, which is taught in ¶ [21] and [22] above.).
Re claim 6: (Original) The method of claim 1, wherein the conducting a first image search based at least in part on the query vector comprises:
using the query vector to select one or more candidate vectors from a search index, wherein the search index includes a plurality of candidate vectors, each candidate vector of the plurality of candidate vectors corresponding to a candidate image (e.g. the system determines if an image feature amount is similar to an image within a database. The determination of similarity is represented by a calculated similarity score to indicate the level of similarity between the image feature amount of the registered image in the database and the image input, which is taught in ¶ [23] above.);
wherein the one or more candidate vectors are selected from the plurality of candidate vectors based on a vector distance between a corresponding candidate vector and the query vector (e.g. the system selects certain images within the database based on how similar the input document is to the stored document in the database reflected as a similarity score. Based on the score, it is selected if the score is above a specific threshold or meets a certain criteria, which is taught in ¶ [52]-[55].).
[0052] Here, the similarity scores to be combined may be obtained based on aspects having low relative correlation with each other. For example, when the text-based score described above is used together with a score obtained based on the feature of the edge amount distribution or the average color distribution, which provides a high level of identification with respect to photographs, it is possible to perform search with high precision with respect to both text documents and photographs.
[0053] In the above example, the photograph image search section 10 and the text search section 20 supply the similarity scores concerning all the registered images to the candidate extraction section 50. Alternatively, it is also possible to supply to the candidate extraction section 50 only the similarity scores concerning registered images which are equal to or greater than a predetermined value. In this case, the candidate extraction section 50 may set the similarity scores concerning registered images which the candidate extraction section 50 has not received to a predetermined value such as "0," and may then perform the above-described score combination processing.
[0054] In this manner, the candidate extraction section 50 calculates, for each registered image, a combined score indicative of a degree of similarity of the registered image with respect to an input image a (the input document image 100).
[0055] The candidate extraction section 50 then sequentially selects, as a candidate for search to be performed in the subsequent document image search section 52, a registered image in descending order of the combined score thus obtained. Stated differently, the candidate extraction section 50 performs the first narrowing of candidates among the group of registered images stored in the document image database or the document image log storage device (not shown). When candidates are extracted, a predetermined number of registered images having combined scores of higher rank may be sequentially extracted, or registered images having higher-ranked combined scores corresponding to a predetermined ratio with respect to all the registered images may be extracted. It is also possible to extract a registered image having a combined score which is equal to or greater than a threshold value. Of course, the above-described methods of extracting candidates are merely examples, and other conditions may be used for sequentially extracting a registered image in descending order of the combined score. The candidate extraction section 50 provides the document ID of each candidate thus extracted to the document image search section 52. Here, in order for the document image search section 52 to perform search in consideration of the combined score, the candidate extraction section 50 provides both the document ID and the combined score of each candidate to the document image search section 52.
Re claim 7: (Original) Koyama discloses the method of claim 6, wherein the one or more candidate vectors are selected based on a threshold (e.g. the system discloses calculating a score from the image feature amounts that are compared to a predetermined value, which is taught in ¶ [60].).
[0060] The document image search section 52 obtains a similarity score based on a projection waveform for each candidate as described above, and outputs a list listing a pair consisting of the document ID of each candidate and the corresponding similarity score, in descending order of similarity score. Here, the list to be output may include only a predetermined number of documents with high-ranked similarity scores or only the documents whose similarity scores are equal to or greater than a predetermined value.
Re claim 8: (Original) Koyama discloses the method of claim 6, further comprising: ranking the plurality of candidate vectors based on a plurality of vector distances, each vector distance of the plurality of vector distances corresponding to a respective candidate vector and the query vector (e.g. the system discloses ranking the distances between the registered images within the database and the input image evaluated through the use of ranking the similarity scores representative of the distance. This is taught in ¶ [55] above.);
wherein the one or more candidate vectors are selected from the plurality of candidate vectors based at least in part on the ranking (e.g. the higher ranked candidates are selected for further comparison based on their similarity scores, which is taught in ¶ [60] and [61] above.).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2 and 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Koyama in view of Tu (US Pub 2017/0293611).
Re claim 2: (Original) Koyama discloses the method of claim 1, wherein the conducting a second image search based at least in part on the one or more recognized text strings comprises conducting the second image search based at least in part on the one or more text strings (e.g. the text strings are used to check initial candidates from the first list of candidates in order to generate second candidate images, which is taught in ¶ [78] above.).
However, Koyama fails to specifically teach the features of wherein the recognizing one or more text strings on the query image comprises translating the one or more recognized text strings from a first language to a second language different from the first language, wherein the conducting a second image search based at least in part on the one or more recognized text strings comprises conducting the second image search based at least in part on the one or more translated text strings in the second language.
However, this is well known in the art as evidenced by Tu. Similar to the primary reference, Tu discloses a translation model for language translation (same field of endeavor or reasonably pertinent to the problem).
Tu discloses wherein the recognizing one or more text strings on the query image comprises translating the one or more recognized text strings from a first language to a second language different from the first language, wherein the conducting a second image search based at least in part on the one or more recognized text strings comprises conducting the second image search based at least in part on the one or more translated text strings in the second language (e.g. the system discloses acquiring text strings from pictures that are recognized. The text is recognized using an OCR performed. The text recognized is obtained to be translated from one language to a target language by an object alignment model based on scores, which is taught in ¶ [157] and [160]. This aspect, combined with the primary reference, would allow for first translation and then the comparison by the first reference to determine further candidates.).
[0157] In step S1, the terminal apparatus acquires pictures of an object, or transfers the pictures of the object to a translation system in the server, and performs recognition on the object first. With reference to features required by an object recognition model, an image feature is extracted from an image, and optical character recognition (OCR) is performed, a text feature is extracted, and the object recognition is performed according to the object recognition model to obtain a source object to be translated and text in the image. The source objects are translated, by an object alignment model, into target object candidates according to scores, and the source texts in the image are translated into a target language by a text translation model and a language model.
[0160] In the second embodiment of the present disclosure, the object recognition model, the object alignment model and the object knowledge graph database are established by collecting a large amount of multi-modal data of the source object and the target object, including pictures and texts, extracting a text feature and an image feature, establishing an object type discrimination model and an object recognition model in each type by supervised learning, calculating a score of text description similarity between the source object and the target object according to an existing translation model (object alignment model) between a source language and a target language, extracting text features and image features, and obtaining an object alignment model by training with supervised learning. Entities and relation attributes of related objects are extracted from the collected large-scale multi-modal data related to the objects, and are expressed by three categories <entity, entity, relation, and an object look-up index is established according to a preset object type, to form object knowledge graph data. The “relation” has a specific corresponding meaning for different objects. For example, for “the color of the object A is blue”, <A, blue, color> can be established for the categories. In the present disclosure, the “relation” attribute of different objects will be described in detail in derivative translation. In derivative translation, “relation” can be expressed as “key information” of an object, i.e., the attribute information of a preset attribute. For example, the key information of drugs includes effects, side effects, dosage, contraindicated crowds, and manner of accessing, the key information of foods includes ingredients, allergens, energy, suitable consumers, and flavor, wherein the key information of packaged foods includes place of origin, production date, and shelf life, and the key information of unpackaged dishes (meals) and menus also includes flavor, scent, temperature, color, eating and cooking methods, and the key information of cosmetics includes ingredients, place of origin, effects, suitable crowds, shelf life, and usage.
Therefore, in view of Tu, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the recognizing one or more text strings on the query image comprises translating the one or more recognized text strings from a first language to a second language different from the first language, wherein the conducting a second image search based at least in part on the one or more recognized text strings comprises conducting the second image search based at least in part on the one or more translated text strings in the second language, incorporated in the device of Koyama, in order to provide a model to translate text from one language into another using a to identify text candidates, which increases the range of machine translation subjects (as stated in Tu ¶ [174]]).
Re claim 3: (Original) However, Koyama fails to specifically teach the features of the method of claim 2, wherein the translating the one or more recognized text strings from a first language to a second language comprises translating the one or more recognized text strings using a machine learning model.
However, this is well known in the art as evidenced by Tu. Similar to the primary reference, Tu discloses a translation model for language translation (same field of endeavor or reasonably pertinent to the problem).
Tu discloses wherein the translating the one or more recognized text strings from a first language to a second language comprises translating the one or more recognized text strings using a machine learning model (e.g. text recognized is obtained to be translated from one language to a target language by an object alignment model based on scores, which is taught in ¶ [157] and [160] above.).
Therefore, in view of Tu, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the translating the one or more recognized text strings from a first language to a second language comprises translating the one or more recognized text strings using a machine learning model, incorporated in the device of Koyama, in order to provide a model to translate text from one language into another using a to identify text candidates, which increases the range of machine translation subjects (as stated in Tu ¶ [174]]).
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Koyama in view of Lin (US Pub 2018/0267996).
Re claim 9: (Original) However, Koyama fails to specifically teach the features of the method of claim 1, further comprising: generating an explanation of the first image search, wherein the explanation includes a heatmap representing one or more focus aspects.
However, this is well known in the art as evidenced by Lin. Similar to the primary reference, Lin discloses generating a heatmap based on an image search (same field of endeavor or reasonably pertinent to the problem).
Lin discloses further comprising: generating an explanation of the first image search, wherein the explanation includes a heatmap representing one or more focus aspects (e.g. the invention discloses the generation of a heat map that is visually indicative of a relevance of particular regions of the subject image with respect to certain keyword tags. The keyword tags can be listed along with the heat map as an explanation of the heatmap relevant area, which is taught in ¶ [17], [25], [26], [33] and [47.).
[0017] Embodiments of the present invention relate to, among other things, a framework for associating dense images (i.e., images having a high resolution or otherwise having an abundance of associated information) with topics that are indicative of the subject matter of the images utilizing embedding learning. The framework is trained utilizing multiple images, each image having associated keyword tags. The tag information associated with each image is utilized to generate a weighted word vector (or “soft topic feature vector”) for each image (that is, a feature vector representing a dominant concept of the subject image derived from the tags) by calculating a weighted average of word vector representations generated for the tags associated with the image. Visual information associated with each image is divided into a number of regions. For each region of each image, one or more visual features are computed, an image feature vector is generated from the visual features, and an image embedding vector is generated from the image feature vector. The image feature vector for each region of a given image and weighted word vector for the image are aligned (i.e., made as close as possible to one another) into a common embedding space and a heat map is computed for the image. The heat map is visually indicative of the relevance of particular regions of the subject image with respect to certain keyword tags. The heat map includes multiple regions that correspond to the regions of the image and at least one region of the heat map corresponds to each of the tags. Once trained, the framework described herein can be utilized to automatically tag additional input images and rank the relevance of images and/or image regions with respect to queried keywords based upon associated heat maps.
[0025] Various embodiments of the present invention address the technical challenge of appropriately identifying and presenting the most relevant images available in response to an input search query by providing an approach that facilitates associating dense images with topics that are indicative of the subject matter of the images utilizing embedding learning. During a training phase, multiple dense images are received, each image having associated visual characteristics and keyword tags. In some embodiments, the keyword tags are user-provided. As to the visual characteristics associated with the images, each dense image includes a number of regions. Each region encompasses less than the entire image and a sum of the regions encompasses the entire image. For each region of each image, one or more visual features are computed, the visual features are utilized to create an image feature vector for each image. On the keyword side, during the training phase, the keyword tags are utilized to create an image-specific soft topic feature vector (or weighted word vector). More particularly, a word vector representation is generated for each keyword tag associated with an image and a weighted average of the generated word vector representations is calculated to generate a soft topic feature (or weighted word) vector for the image, i.e., a feature vector that is representative of a dominant concept among the keyword tags. The image embedding for each image region and the soft topic feature vector are aligned (that is, made as close as possible to one another) into a common embedding space and a heat map is computed for the image. The heat map is visually indicative of the relevance of particular regions of the subject image with respect to certain keyword tags. The heat map includes regions that correspond to the regions of the image and at least one region of the heat map corresponds to each of the tags.
[0026] Once the framework is trained, an auto-tagging functionality of the framework can be utilized to predict a list of keyword tags and an associated heat map for an input image. Similarly, and by way of reverse application, heat maps generated by the image auto-tagging system can be utilized to build a search engine for identifying images relevant to an input search query, be it a text-based or image-based query.
[0033] The image embedding system 104 also includes an aligning component 116. The aligning component 116 is configured to align (that is, render as close as possible to one another) an image feature vector and a soft topic feature vector to compute a heat map corresponding to the image. In accordance with embodiments of the present invention, the image feature vector is associated with a region of an image that comprises less than the entire image. The soft topic feature vector is image-specific. The heat map is visually indicative of the relevance of particular regions of the subject image with respect to certain keyword tags and includes a plurality of regions that correspond to a plurality of regions of the image. At least one region of the heat map corresponds to each of the keyword tags.
[0047] When associating topics and tagging dense images (that is, images having a high resolution or otherwise having an abundance of associated information), tags and their relevance, as indicated by an image-specific heat map, is performed for each region of a subject image. Dense tagging attempts to predict tags and their relevancy score for each region of an image. As shown FIG. 5, the embedding learning framework 500 (previously illustrated as network 200 in FIG. 2) is transformed to a fully-convolution network by viewing the fully connected weights as 1×1 convolution kernels and the batch normalization layers as spatial batch normalization layers. “1×1 convolution kernels” are filters used in the convolution layer. The filter size is 1×1×C where C represents the number of channels. “Batch normalization layers” are a kind of feature normalization layer that makes network training faster and more stable. Convolution kernels and batch normalization layers are known to those having ordinary skill in the art and, accordingly, are not further described herein. After this transformation, the output is a set of heat maps 510, each heat map corresponding to a different keyword tag.
Therefore, in view of Lin, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of further comprising: generating an explanation of the first image search, wherein the explanation includes a heatmap representing one or more focus aspects, incorporated in the device of Koyama, in order to generate a heat map to visually show that relevant area of the image searched, which automatically and more easily shows and determines the relevant areas of the image (as stated in Lin ¶ [03] and [04]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Li discloses a model that can evaluate an image or text area
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHAD S DICKERSON whose telephone number is (571)270-1351. The examiner can normally be reached Monday-Friday 10AM-6PM EST..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abderrahim Merouan can be reached at 571-270-5254. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHAD DICKERSON/ Primary Examiner, Art Unit 2683
