DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-11 are rejected under 35 U.S.C. 103 as being unpatentable over Rodríguez-Serrano et al., US 2017/0083792 A1 (hereinafter “Rodríguez-Serrano” – as cited in the IDS filed 27 December 2024) in view of YAN et al., US 2021/0042607 A1 (hereinafter “Yan”).
Claim 1: Rodríguez-Serrano teaches an image retrieval system comprising: a processing portion, wherein the processing portion comprises: a neural network; and wherein the neural network comprises: a pooling layer (Rodríguez-Serrano, [0026] note an image representation is based on a real-valued tensor, such as a vector, containing the activations output from a layer of a model derived from part of a trained deep convolutional neural network (CNN), such as a max-pooling layer),
wherein image data and a plurality of pieces of database image data are input to the processing portion (Rodríguez-Serrano, [0045] note At S110, the query image 12 is input to the neural network model 42 at the output of the selected layer of the model is used to generate a representation 46 of the query image, in a similar manner to the annotated images),
wherein the processing portion is configured to compare the image data with the plurality of pieces of database image data (Rodríguez-Serrano, [0042] note At S104, a set 40 of annotated images 38 is provided. Each annotated image 38 is annotated with a bounding box which identifies a location of an object of interest, [0043] note At S106, the neural network model 42 is used to generate a representation 48 of each of the annotated images 38, [0046] note At S112, a similarity is computed between the query image representation 46 and the dataset image representations 48),
wherein the processing portion is configured to extract database image data from the plurality of pieces of database image data that comprise an area or a plurality of areas with a high degree of correspondence to the image data as extracted image data (Rodríguez-Serrano, [0046] note At S112, a similarity is computed between the query image representation 46 and the dataset image representations 48 (optionally projected into the new feature space) to identify a subset 56 of one or more similar annotated image(s), i.e., those which have the most similar representations (in the new feature space)), and
wherein, after acquiring the extracted image data, the processing portion is configured to extract data from the extracted image data that comprise the area or the plurality of areas with a high degree of correspondence to the image data as partial image data (Rodríguez-Serrano, [0047] note At S114, the bounding box annotations of the annotated image(s) in the set 56 is/are used to compute a bounding box annotation 34 for the query image, [0048] note At S116, information 36 may be extracted from the region of the query image 12 contained within the bounding box 34, [0049] note At S118, the annotated query image 12 and/or information 36 extracted therefrom, is output).
Rodríguez-Serrano does not explicitly teach wherein, after acquiring the partial image data, the image data and the partial image data are input to a first layer of the neural network, wherein the pooling layer is configured to output a first output value corresponding to the image data, wherein the pooling layer is configured to output a second output value corresponding to the partial image data, wherein the processing portion is configured to compare the first output value with the second output value, and wherein the processing portion is configured to calculate the degree of similarity between the partial image data and the image data.
However, Yan teaches this (Yan, [0022] note a convolutional neural net (CNN) that includes convolutional and pooling layers for handling image recognition applications, [Fig. 5], [0036] note FIG. 5 illustrates CNN 500 that may be employed during a testing phase or during run-time operation… It is contemplated that RGB network 510 and normalization network 530 may include different variation of convolutional layers, max pooling layers, flattening layers, dropout layers and dense layers depending on a given application, [0037] note RGB network 510 may provide output vector data to an RGB encoded feature 550 and normalization network 530 may provide output vector data to one or more normal encoded features 560, 570, [0038] note CNN 500 may then use the output vector data provided by RGB encoded feature 550 and the one or more normal encoded features 560, 570 to compute a set of distance vectors 580 from the feature vectors of the normalized images 540 and the feature vectors encoded within RGB image 520. CNN 500 may then select the distance vector that has the smallest distance in the semantic space, [Fig. 6], [0040] note cropping algorithm to generate a first bounding box 610, second bounding box 620, third bounding box 630, and fourth bounding box 640, [0043] note run-time operation may crop a captured image (e.g., RGB image 113 received by client system 110) into different parts. CNN may then be operable to compare the parts in the image with. the corresponding areas in two or more two normal map images).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the similarity based detection of prominent objects of Rodríguez-Serrano with the CNN that includes convolutional and pooling layers for handling image recognition of Yan according to known methods (i.e. detecting prominent objects using a CNN). Motivation for doing so is that this may improve performance of the CNN (Yan, [0022]).
Claim 2: Rodríguez-Serrano teaches an image retrieval system comprising: a processing portion, wherein the processing portion comprises: a neural network; and a transistor, wherein the neural network comprises: a pooling layer (Rodríguez-Serrano, [0026] note an image representation is based on a real-valued tensor, such as a vector, containing the activations output from a layer of a model derived from part of a trained deep convolutional neural network (CNN), such as a max-pooling layer),
wherein image data and a plurality of pieces of database image data are input to the processing portion (Rodríguez-Serrano, [0045] note At S110, the query image 12 is input to the neural network model 42 at the output of the selected layer of the model is used to generate a representation 46 of the query image, in a similar manner to the annotated images),
wherein the processing portion is configured to compare a plurality of first pixel data of the image data with a plurality of second pixel data of the plurality of pieces of database image data (Rodríguez-Serrano, [0042] note At S104, a set 40 of annotated images 38 is provided. Each annotated image 38 is annotated with a bounding box which identifies a location of an object of interest, [0043] note At S106, the neural network model 42 is used to generate a representation 48 of each of the annotated images 38, [0046] note At S112, a similarity is computed between the query image representation 46 and the dataset image representations 48, [0047] note image data of the image, which is input to the CNN model, may include colorant values for each of the pixels in the image),
wherein the plurality of first pixel data corresponds to a plurality of pixels of the image data (Rodríguez-Serrano, 0047] note image data of the image, which is input to the CNN model, may include colorant values for each of the pixels in the image),
wherein the processing portion is configured to extract database image data from the plurality of pieces of database image data that comprise pixel data with a high degree of correspondence to the plurality of first pixel data of the image data as extracted image data (Rodríguez-Serrano, [0046] note At S112, a similarity is computed between the query image representation 46 and the dataset image representations 48 (optionally projected into the new feature space) to identify a subset 56 of one or more similar annotated image(s), i.e., those which have the most similar representations (in the new feature space)), and
wherein, after acquiring the extracted image data, the processing portion is configured to extract data from the extracted image data that comprise the pixel data with a high degree of correspondence to the plurality of first pixel data of the image data as partial image data (Rodríguez-Serrano, [0047] note At S114, the bounding box annotations of the annotated image(s) in the set 56 is/are used to compute a bounding box annotation 34 for the query image, [0048] note At S116, information 36 may be extracted from the region of the query image 12 contained within the bounding box 34, [0049] note At S118, the annotated query image 12 and/or information 36 extracted therefrom, is output).
Rodríguez-Serrano does not explicitly teach wherein, after acquiring the partial image data, the image data and the partial image data are input to a first layer of the neural network, wherein the pooling layer is configured to output a first output value corresponding to the image data, wherein the pooling layer is configured to output a second output value corresponding to the partial image data, wherein the processing portion is configured to compare the first output value with the second output value, and wherein the processing portion is configured to calculate the degree of similarity between the partial image data and the image data.
However, Yan teaches this (Yan, [0022] note a convolutional neural net (CNN) that includes convolutional and pooling layers for handling image recognition applications, [Fig. 5], [0036] note FIG. 5 illustrates CNN 500 that may be employed during a testing phase or during run-time operation… It is contemplated that RGB network 510 and normalization network 530 may include different variation of convolutional layers, max pooling layers, flattening layers, dropout layers and dense layers depending on a given application, [0037] note RGB network 510 may provide output vector data to an RGB encoded feature 550 and normalization network 530 may provide output vector data to one or more normal encoded features 560, 570, [0038] note CNN 500 may then use the output vector data provided by RGB encoded feature 550 and the one or more normal encoded features 560, 570 to compute a set of distance vectors 580 from the feature vectors of the normalized images 540 and the feature vectors encoded within RGB image 520. CNN 500 may then select the distance vector that has the smallest distance in the semantic space, [Fig. 6], [0040] note cropping algorithm to generate a first bounding box 610, second bounding box 620, third bounding box 630, and fourth bounding box 640, [0043] note run-time operation may crop a captured image (e.g., RGB image 113 received by client system 110) into different parts. CNN may then be operable to compare the parts in the image with. the corresponding areas in two or more two normal map images).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the similarity based detection of prominent objects of Rodríguez-Serrano with the CNN that includes convolutional and pooling layers for handling image recognition of Yan according to known methods (i.e. detecting prominent objects using a CNN). Motivation for doing so is that this may improve performance of the CNN (Yan, [0022]).
Claim 3: Rodríguez-Serrano and Yan teach the image retrieval system according to claim 2, wherein each of the plurality of first pixel data corresponds to a luminance value (Rodríguez-Serrano, [0057] note image data of the image, which is input to the CNN model, may include colorant values for each of the pixels in the image, such as grayscale values, for each of a set of color separations, such as L*a*b* or RGB, or be expressed in another other color space in which different colors can be represented. In general, “grayscale” refers to the optical density value of any single color channel, however expressed (L*a*b*, RGB, YCbCr, etc.). The exemplary embodiment may also be used for black and white (monochrome) images).
Claim 4: Rodríguez-Serrano and Yan teach the image retrieval system according to claim 1, wherein the number of pieces of pixel data included in the image data is less than or equal to the number of pieces of pixel data included in the plurality of pieces of database image data (Rodríguez-Serrano, [0044] note At S108, a query image 12 is received, and may be preprocessed, e.g., by reducing the pixel dimensions to the same as those of the annotated images 38).
Claim 5: Rodríguez-Serrano and Yan teach the image retrieval system according to claim 2, wherein the number of pieces of pixel data included in the image data is less than or equal to the number of pieces of pixel data included in the plurality of pieces of database image data (Rodríguez-Serrano, [0044] note At S108, a query image 12 is received, and may be preprocessed, e.g., by reducing the pixel dimensions to the same as those of the annotated images 38).
Claim 6: Rodríguez-Serrano and Yan teach the image retrieval system according to claim 1, wherein the processing portion is configured to compare the image data with the plurality of pieces of database image data by area-based matching (Rodríguez-Serrano, [0042] note At S104, a set 40 of annotated images 38 is provided. Each annotated image 38 is annotated with a bounding box which identifies a location of an object of interest, [0043] note At S106, the neural network model 42 is used to generate a representation 48 of each of the annotated images 38, [0046] note At S112, a similarity is computed between the query image representation 46 and the dataset image representations 48).
Claim 7: Rodríguez-Serrano and Yan teach the image retrieval system according to claim 2, wherein the processing portion is configured to compare the plurality of first pixel data of the image data with the plurality of second pixel data of the plurality of pieces of database image data by area-based matching (Rodríguez-Serrano, [0042] note At S104, a set 40 of annotated images 38 is provided. Each annotated image 38 is annotated with a bounding box which identifies a location of an object of interest, [0043] note At S106, the neural network model 42 is used to generate a representation 48 of each of the annotated images 38, [0046] note At S112, a similarity is computed between the query image representation 46 and the dataset image representations 48).
Claim 8: Rodríguez-Serrano teaches an image retrieval method comprising the steps of:
comparing image data with a plurality of pieces of database image data (Rodríguez-Serrano, [0042] note At S104, a set 40 of annotated images 38 is provided. Each annotated image 38 is annotated with a bounding box which identifies a location of an object of interest, [0043] note At S106, the neural network model 42 is used to generate a representation 48 of each of the annotated images 38, [0046] note At S112, a similarity is computed between the query image representation 46 and the dataset image representations 48);
extracting database image data from the plurality of pieces of database image data that comprise an area or a plurality of areas with a high degree of correspondence to the image data as extracted image data (Rodríguez-Serrano, [0046] note At S112, a similarity is computed between the query image representation 46 and the dataset image representations 48 (optionally projected into the new feature space) to identify a subset 56 of one or more similar annotated image(s), i.e., those which have the most similar representations (in the new feature space));
extracting data from the extracted image data that comprise the area or the plurality of areas with a high degree of correspondence to the image data as partial image data (Rodríguez-Serrano, [0047] note At S114, the bounding box annotations of the annotated image(s) in the set 56 is/are used to compute a bounding box annotation 34 for the query image, [0048] note At S116, information 36 may be extracted from the region of the query image 12 contained within the bounding box 34, [0049] note At S118, the annotated query image 12 and/or information 36 extracted therefrom, is output);
inputting the image data to a neural network comprising a convolutional layer and a pooling layer (Rodríguez-Serrano, [0026] note an image representation is based on a real-valued tensor, such as a vector, containing the activations output from a layer of a model derived from part of a trained deep convolutional neural network (CNN), such as a max-pooling layer, [0045] note At S110, the query image 12 is input to the neural network model 42 at the output of the selected layer of the model is used to generate a representation 46 of the query image, in a similar manner to the annotated images);
obtaining a first output value output from the pooling layer, the first output value corresponding to the image data (Rodríguez-Serrano, [0047] note At S114, the bounding box annotations of the annotated image(s) in the set 56 is/are used to compute a bounding box annotation 34 for the query image, [0048] note At S116, information 36 may be extracted from the region of the query image 12 contained within the bounding box 34, [0049] note At S118, the annotated query image 12 and/or information 36 extracted therefrom, is output).
Rodríguez-Serrano does not explicitly teach inputting the partial image data to the neural network; obtaining a second output value output from the pooling layer, the second output value corresponding to the partial image data; comparing the first output value with the second output value; and calculating the degree of similarity between the first output value and the second output value.
However, Yan teaches this (Yan, [0022] note a convolutional neural net (CNN) that includes convolutional and pooling layers for handling image recognition applications, [Fig. 5], [0036] note FIG. 5 illustrates CNN 500 that may be employed during a testing phase or during run-time operation… It is contemplated that RGB network 510 and normalization network 530 may include different variation of convolutional layers, max pooling layers, flattening layers, dropout layers and dense layers depending on a given application, [0037] note RGB network 510 may provide output vector data to an RGB encoded feature 550 and normalization network 530 may provide output vector data to one or more normal encoded features 560, 570, [0038] note CNN 500 may then use the output vector data provided by RGB encoded feature 550 and the one or more normal encoded features 560, 570 to compute a set of distance vectors 580 from the feature vectors of the normalized images 540 and the feature vectors encoded within RGB image 520. CNN 500 may then select the distance vector that has the smallest distance in the semantic space, [Fig. 6], [0040] note cropping algorithm to generate a first bounding box 610, second bounding box 620, third bounding box 630, and fourth bounding box 640, [0043] note run-time operation may crop a captured image (e.g., RGB image 113 received by client system 110) into different parts. CNN may then be operable to compare the parts in the image with. the corresponding areas in two or more two normal map images).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the similarity based detection of prominent objects of Rodríguez-Serrano with the CNN that includes convolutional and pooling layers for handling image recognition of Yan according to known methods (i.e. detecting prominent objects using a CNN). Motivation for doing so is that this may improve performance of the CNN (Yan, [0022]).
Claim 9: Rodríguez-Serrano and Yan teach the image retrieval method according to claim 8, wherein the image data is compared with the plurality of pieces of database image data by area-based matching (Rodríguez-Serrano, [0042] note At S104, a set 40 of annotated images 38 is provided. Each annotated image 38 is annotated with a bounding box which identifies a location of an object of interest, [0043] note At S106, the neural network model 42 is used to generate a representation 48 of each of the annotated images 38, [0046] note At S112, a similarity is computed between the query image representation 46 and the dataset image representations 48).
Claim 10: Rodríguez-Serrano and Yan teach the image retrieval method according to claim 9, wherein an input data value input into the convolutional layer corresponds to a gray level represented by pixel data, wherein, during the area-based matching, a plurality of first pixel data of the image data is compared with a plurality of second pixel data of the plurality of pieces of database image data, wherein the plurality of first pixel data corresponds to a plurality of pixels of the image data, wherein each of the plurality of first pixel data corresponds to a luminance value, and wherein the luminance value represents a plurality of gray levels (Rodríguez-Serrano, [0042] note At S104, a set 40 of annotated images 38 is provided. Each annotated image 38 is annotated with a bounding box which identifies a location of an object of interest, [0043] note At S106, the neural network model 42 is used to generate a representation 48 of each of the annotated images 38, [0046] note At S112, a similarity is computed between the query image representation 46 and the dataset image representations 48, [0057] note image data of the image, which is input to the CNN model, may include colorant values for each of the pixels in the image, such as grayscale values, for each of a set of color separations, such as L*a*b* or RGB, or be expressed in another other color space in which different colors can be represented. In general, “grayscale” refers to the optical density value of any single color channel, however expressed (L*a*b*, RGB, YCbCr, etc.). The exemplary embodiment may also be used for black and white (monochrome) images).
Claim 11: Rodríguez-Serrano and Yan teach the image retrieval method according to claim 8, wherein the image data comprises a plurality of pieces of pixel data, and wherein a plurality of pieces of image data that differ in the number of pieces of the pixel data to be provided are generated on the basis of the image data, and then the image data is compared with the plurality of pieces of database image data (Rodríguez-Serrano, [0057] note image data of the image, which is input to the CNN model, may include colorant values for each of the pixels in the image, [0044] note At S108, a query image 12 is received, and may be preprocessed, e.g., by reducing the pixel dimensions to the same as those of the annotated images 38).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Ishikake, US 20210/174518 A1 – A convolutional neural network (CNN), which is suitable for image recognition processing. A CNN includes convolutional layers that perform a convolution operation and pooling layers. Each convolutional layer performs a filter process. Each pooling layer performs a pooling operation for reducing the size vertically and horizontally. An output layer of the CNN is, for example, well-known softmax layer.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Giuseppi Giuliani whose telephone number is (571)270-7128. The examiner can normally be reached Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kavita Stanley can be reached at (571)272-8352. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GIUSEPPI GIULIANI/Primary Examiner, Art Unit 2153
Prosecution Insights