DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In response to Applicant’s claims filed on November 25, 2025, claims 1-20 are now pending for examination in the application.
Response to Arguments
101 rejection arguments and Examiner’s reply
Remarks, page 9 include:
PNG
media_image1.png
280
650
media_image1.png
Greyscale
Examiner’s reply:
Examiner respectfully disagrees. The alleged improvement in the claims is directed to improving the abstract idea (a better way of matching similar images). The claims do not make a computer itself function better.
Remarks, page 11 include:
PNG
media_image2.png
82
640
media_image2.png
Greyscale
Examiner’s reply:
The claims do not recite the specific training techniques described in the specification such as the data sampler, mini batch balancing, selective loss calculation, etc. Instead, the claims broadly recite a multi task classification model trained using loss functions and data sets. See MPEP 2106.04(d)(1) which includes, with emphasis, “Second, if the specification sets forth an improvement in technology, the claim must be evaluated to ensure that the claim itself reflects the disclosed improvement. That is, the claim includes the components or steps of the invention that provide the improvement described in the specification.“
Remarks, page 11 also include:
PNG
media_image3.png
214
639
media_image3.png
Greyscale
Examiner’s reply:
The claim does not recite disclosed components/steps of a data sampler, disjoint dataset sampling, balanced mini-batches, dataset specific selective loss computation. Claim 1 is directed to a results-oriented description (model trained with separate loss functions and separate datasets) without details of a technical implementation.
Remarks, page 11 also include:
PNG
media_image4.png
273
641
media_image4.png
Greyscale
Examiner’s reply:
Receiving data and displaying results are conventional computer functions that merely link the abstract idea to a generic computing environment. These I/O steps do not integrate an abstract idea into a practical application. See MPEP 2106.05(g) Insignificant Extra-Solution Activity.
Remarks, page 12-13 include:
PNG
media_image5.png
81
636
media_image5.png
Greyscale
PNG
media_image6.png
49
636
media_image6.png
Greyscale
Examiner’s reply:
Although mathematical concepts may be eligible when additional elements integrate the exception into a practical application, the current claims do not include any additional elements that provide such an integration. The claims broadly apply mathematical models to image search, which is inefficient under MPEP 2106.05(a).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-patentable subject matter. The claims are directed to an abstract idea without significantly more.
Claim 1-20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The judicial exception is not integrated into a practical application. The claims do not include additional elements that are sufficient to amount to significantly more than judicial exception. The eligibility analysis in support of these findings is provided below, on Claim Rejections - 35 USC 101 accordance with the "2019 Revised Patent Subject Matter Eligibility Guidance" (published on 1/7/2019 in Fed, Register, Vol. 84, No. 4 at pgs. 50-57, hereinafter referred to as the "2019 PEG").
Step 1. in accordance with Step 1 of the eligibility inquiry (as explained in MPEP 2106), it is first noted the claim(s) 1-20 are directed to one of the eligible categories of subject matter and therefore satisfies Step 1.
Step 2A. In accordance with Step 2A, prong one of the 2019 PEG, it is noted that the independent claims recite an abstract idea falling within the Mathematical Concepts & Mental Processes enumerated groupings of abstract ideas set forth in the 2019 PEG. Examiner is of the position that independent claims 1, 13, and 20 are directed towards the Mental Process Grouping of Abstract Ideas.
Independent claims 1, 13 and 20 recites the following limitations directed towards a Mathematical Concepts & Mental Processes:
identifying, using the first embedding and from among a plurality of embeddings corresponding to images in the dataset of images, a set of images that are similar to the first input image (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search for other images); and
Step 2A. In accordance with Step 2A, prong two of the 2019 PEG, the judicial exception is not integrated into a practical application because of the recitation in claim(s) 1, 13, and 20:
receiving, from a client device, a query requesting to search a dataset of images using a first input image, wherein the query includes first image data for the first input image (recites insignificant extra solution activity that amounts to mere data gathering);
inputting the first image data into a multi-task classification model trained to identify one or more images from the dataset of images in response to image data for a particular image (recites insignificant extra solution activity of data inputting),
wherein the multi-task classification model is a neural network that is trained using a plurality of classification heads corresponding to a plurality of classification tasks (recites insignificant extra solution activity of model training),
wherein the multi-task classification model is trained using separate loss functions for each respective classification task (recites insignificant extra solution activity of model training), and
wherein the plurality of classification tasks include one or more classification tasks based on separate training datasets (recites insignificant extra solution activity of model training);
obtaining, as output from a layer of the neural network preceding the plurality of classification heads used for training and in response to the first image data, a first embedding for the first image data (recites insignificant extra solution activity that amounts to mere data gathering);
providing, in response to the received query and for display on a device, the set of images (recites insignificant extra solution activity for providing data).
Step 2B. Similar to the analysis under 2A Prong Two, the claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Because the additional elements of the independent claims amount to insignificant extra solution activity and/or mere instructions, the additional elements do not add significantly more to the judicial exception such that the independent claims as a whole would be patent eligible.
Therefore, independent claims 1, 13, and 20 are rejected under 35 U.S.C. 101.
With respect to claim(s) 2 and 14:
Step 2A, prong one of the 2019 PEG:
generating, using the multi-task classification model, the plurality of embeddings for the images in the dataset of images (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 3 and 15:
Step 2A, prong one of the 2019 PEG:
computing, using a nearest neighbor algorithm, a respective distance metric between the first embedding and each embedding in the plurality of embeddings (The limitation recites a mathematical concept);
identifying a set of embeddings from among the plurality of embeddings for which the corresponding distance metric satisfies a predetermined threshold (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images); and
determining the set of images corresponding to the identified set of embeddings (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 4 and 16:
Step 2A, prong one of the 2019 PEG:
identifying the set of embeddings with top-N distance metrics from among the computed distance metrics, wherein N is the predetermined threshold and N is an integer greater than 1, or
identifying the set of embeddings with respective distance metrics that each meet or exceed the predetermined threshold (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 5 and 17:
Step 2A, prong one of the 2019 PEG:
a fine-grained taxonomy classification (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images);
a top-level taxonomy classification (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images); and
a primary color classification (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 6 and 18:
Step 2A, prong one of the 2019 PEG:
wherein the low-resolution image-based fine-grained taxonomy classification is fine-grained taxonomy classification based on user-uploaded images as opposed to professional images uploaded by a platform (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 7 and 19:
Step 2A, prong one of the 2019 PEG:
identifying, for each classification head and its corresponding classification task, a respective training dataset of training images and a corresponding set of labels, wherein the training datasets are separate and each training dataset is used for training a particular classification head (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images);
sampling, from each training dataset, a number of training images and corresponding labels to form a training mini-batch (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images);
generating, using the multi-task classification model, an embedding for each training image in the mini-batch of training images (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search for other images);
computing, for each classification head in the plurality of classification heads, a loss value that is based on a comparison of output of the classification head for training images obtained from the training dataset corresponding to the classification head (The limitation recites a mathematical concept); and
optimizing, each classification head in the plurality of classification heads, using the respective computed loss values for the classification head (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search for other images).
Step 2A Prong Two Analysis:
providing the training mini-batch as input to the multi-task classification model (recites insignificant extra solution activity for classifying images and visual querying).
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 8:
Step 2A, prong one of the 2019 PEG:
sampling an equal number, for each classification head, of training images and corresponding labels from each training dataset (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 9:
Step 2A, prong one of the 2019 PEG:
wherein the loss value is computed using a sparse categorical cross entropy function (The limitation recites a mathematical concept).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 10:
Step 2A, prong one of the 2019 PEG:
an EfficientNet or ResNext-based neural network The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images;
a separate classification head for each classification task (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images); and
a softmax activation for each respective classification head (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search and classify images).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 11:
Step 2A, prong one of the 2019 PEG:
identifying an index value of a given training image and label (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search for other images);
determining the index value is a valid index value for a first classification head (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search for other images); and
in response to determining the index value is a valid index value for the first classification head, optimizing the first classification head using a loss value computed using the given training image and label (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind by using an image to search for other images).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
With respect to claim(s) 12:
Step 2A, prong one of the 2019 PEG:
determining the index value is included in a set of valid index values for the first classification head (The limitation recites a mental process of observation and/or evaluation capable of being performed by the human mind of determining a value).
Step 2A Prong Two Analysis:
This judicial exception is not integrated into a practical application because there are no additional elements to provide practical application.
Step 2B Analysis:
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The claim is not patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Turkelson et al. (US Pub No. 20210004589) in view of Perera et al. (US Pub. No. 20220067449).
With respect to claim 1, Turkelson et al. teaches a method, comprising:
receiving, from a client device, a query requesting to search a dataset of images using a first input image, wherein the query includes first image data for the first input image (Paragraph 67 discloses a visual search may be executed on client-side (e.g., on a mobile computing device) and Paragraph 116 discloses visual search using the image of the object as an input query image);
inputting the first image data into a multi-task classification model trained to identify one or more images from the dataset of images in response to image data for a particular image (Paragraph 158 discloses receive a query image, pass the image to a deep neural network that extracts deep features, before computing distances to all images in the index and presenting a nearest neighbor as a search result),
wherein the multi-task classification model is a neural network that is trained using a plurality of classification heads corresponding to a plurality of classification tasks (Paragraph 174 discloses the object recognition model may be a deep learning model, such as, and without limitation, a convolutional neural network (CNN), a region-based CNN (R-CNN), a Fast R-CNN, a Masked R-CNN, Single Shot Multibox (SSD), and a You-Only-Look-Once (YOLO) model),
obtaining, as output from a layer of the neural network preceding the plurality of classification heads used for training and in response to the first image data, a first embedding for the first image data (Paragraph 147 discloses produce an embedding for each image from the training data set and each newly received image);
identifying, using the first embedding and from among a plurality of embeddings corresponding to images in the dataset of images, a set of images that are similar to the first input image (Paragraph 170 discloses identify the object and retrieve information regarding the identified object); and
providing, in response to the received query and for display on a device, the set of images (Paragraph 116 discloses provide information regarding one or more results of the visual search). Turkelson et al. does not disclose wherein the multi-task classification model is trained using separate loss functions for each respective classification task.
However, Perera et al. teaches wherein the multi-task classification model is trained using separate loss functions for each respective classification task (Paragraph 76 discloses the generative-discriminative image-classification system 206 in determines a self-supervision loss and a classification loss utilizing the loss functions 624),
wherein the plurality of classification tasks include one or more classification tasks based on separate training datasets (Paragraph 93 discloses classification improvement over conventional systems with respect to the corresponding test datasets for out-of-distribution image samples);
Therefore, it would have been obvious before the effective filing date of invention was made to a person having ordinary skill in the art to modify Turkelson et al. with Perera et al. This would have image classification and searching. See Perera et al. Paragraph(s) 2-3.
The Turkelson et al. reference as modified by Perera et al. teaches all the limitations of claim 1. With respect to claim 2, Turkelson et al. teaches the method of claim 1, further comprising:
generating, using the multi-task classification model, the plurality of embeddings for the images in the dataset of images (Paragraph 147 discloses produce an embedding for each image from the training data set and each newly received image).
The Turkelson et al. reference as modified by Perera et al. teaches all the limitations of claim 2. With respect to claim 3, Turkelson et al. teaches the method of claim 2, wherein identifying the set of images that are similar to the first input image, comprises:
computing, using a nearest neighbor algorithm, a respective distance metric between the first embedding and each embedding in the plurality of embeddings (Paragraph 158 discloses determine the nearest neighbor, computing its distance in vector space);
identifying a set of embeddings from among the plurality of embeddings for which the corresponding distance metric satisfies a predetermined threshold (Paragraph 151 discloses Using distance metrics to analyze similarity in feature vectors (e.g., Cosine distance, Euclidean distance, Manhattan distance, Minkowski distance, Mahalanobis distance), the feature vector closest to the submitted feature vector may be identified, and the object corresponding to that feature vector may be determined to be a “matching” and Paragraph 246 discloses a threshold number of criteria); and
determining the set of images corresponding to the identified set of embeddings (Paragraph 240 discloses extract the number of images included within each indexed training data set).
The Turkelson et al. reference as modified by Perera et al. teaches all the limitations of claim 3. With respect to claim 4, Turkelson et al. teaches method of claim 3, wherein identifying the set of embeddings from among the plurality of embeddings for which the corresponding distance metric satisfied the predetermined threshold, comprises one of:
identifying the set of embeddings with top-N distance metrics from among the computed distance metrics, wherein N is the predetermined threshold and N is an integer greater than 1, or
identifying the set of embeddings with respective distance metrics that each meet or exceed the predetermined threshold (Paragraph 100 discloses a feature vector describing an image depicting an object from the training data set) may be compared to a threshold distance. If the distance is less than or equal to the threshold distance, then the two images may be classified as being similar, classified as depicting a same or similar object, or both).
The Turkelson et al. reference as modified by Perera et al. teaches all the limitations of claim 1. With respect to claim 5, Turkelson et al. teaches the method of claim 1, wherein the plurality of classification tasks comprise a low-resolution image-based fine-grained taxonomy classification and at least one or more of:
a fine-grained taxonomy classification (Paragraph 47 discloses the object recognition model may be trained on a training data set in which both objects depicted are labeled and scenes are labeled according to the context (e.g., scene) ontology or taxonomy, such that the object recognition model is responsive to both pixel values and context classifications when recognizing objects);
a top-level taxonomy classification (Paragraph 47 discloses the object recognition model may be trained on a training data set in which both objects depicted are labeled and scenes are labeled according to the context (e.g., scene) ontology or taxonomy, such that the object recognition model is responsive to both pixel values and context classifications when recognizing objects); and
a primary color classification (Paragraph 47 discloses the object recognition model may be trained on a training data set in which both objects depicted are labeled and scenes are labeled according to the context (e.g., scene) ontology or taxonomy, such that the object recognition model is responsive to both pixel values and context classifications when recognizing objects).
The Turkelson et al. reference as modified by Perera et al. teaches all the limitations of claim 5. With respect to claim 6, Turkelson et al. teaches the method of claim 5, wherein the low-resolution image-based fine-grained taxonomy classification is fine-grained taxonomy classification based on user-uploaded images as opposed to professional images uploaded by a platform (Paragraph 169 discloses a batch identification number indicating the batch of images that were uploaded, temporally (e.g., with a timestamp indicating a time that an image was (i) obtained by computer system 102B, (ii) captured by an image capturing device, (iii) provided to image database 132B, and the like), geographically (e.g., with geographic metadata indicating a location of where the object was located), as well as based on labels assigned to each image which indicate an identifier for an object depicted within the image).
The Turkelson et al. reference as modified by Perera et al. teaches all the limitations of claim 5. With respect to claim 7, Turkelson et al. teaches the method of claim 5, further comprising training the multi-task classification model, comprising:
identifying, for each classification head and its corresponding classification task, a respective training dataset of training images and a corresponding set of labels, wherein the training datasets are separate and each training dataset is used for training a particular classification head (Paragraph 172 discloses the input images, the features extracted from each of the input images, an identifier labeling each of the input image, or any other aspect capable of being used to describe each input image, or a combination thereof, may be stored in training data database 134B as a training data set used to train a computer-vision object recognition model);
sampling, from each training dataset, a number of training images and corresponding labels to form a training mini-batch (Paragraph 158 discloses importing a batch of catalog product images, which may be passed to a deep neural network that extracts deep features for each image, which may be used to create and store an index);
providing the training mini-batch as input to the multi-task classification model (Paragraph 46 discloses The context classification model may receive an image and output a context classification vector indicative of a confidence that the image depicts a particular context);
generating, using the multi-task classification model, an embedding for each training image in the mini-batch of training images (Paragraph 46 discloses The context classification model may receive an image and output a context classification vector indicative of a confidence that the image depicts a particular context);
computing, for each classification head in the plurality of classification heads, a loss value that is based on a comparison of output of the classification head for training images obtained from the training dataset corresponding to the classification head (Paragraph 181 discloses object recognition model may be adjusted based on changes to an optimization of a loss function for the model); and
optimizing, each classification head in the plurality of classification heads, using the respective computed loss values for the classification head (Paragraph 181 discloses object recognition model may be adjusted based on changes to an optimization of a loss function for the model).
The Turkelson et al. reference as modified by Perera et al. teaches all the limitations of claim 7. With respect to claim 8, Turkelson et al. teaches the method of claim 7, wherein sampling, from each training dataset, a number of training images and corresponding labels to form a training mini-batch comprises: sampling an equal number, for each classification head, of training images and corresponding labels from each training dataset (Paragraph 232 discloses the sampling rate for the video may be adjusted to increase or decrease a number of images obtained from the captured video. For example, the computer vision system may sub-sample the video at 60 frames per second (fps), 100 fps, or 200 fps. Quality checks with respect to the blurriness or recognizability of the particular item within each photo may be performed and, if the quality check satisfies quality criteria, the images (or features extracted therefrom) may be added to a database as being associated with that particular object, an identifier for the object from an object ontology, or both).
The Turkelson et al. reference as modified by Perera et al. teaches all the limitations of claim 7. With respect to claim 9, Turkelson et al. teaches the method of claim 7, wherein the loss value is computed using a sparse categorical cross entropy function (Paragraph 146 discloses Some embodiments accommodate sparse training sets by implementing continual learning (or other forms of incremental learning) in a discriminative computer-vision model for object-detection).
The Turkelson et al. reference as modified by Perera et al. teaches all the limitations of claim 7. With respect to claim 10, Turkelson et al. teaches the method of claim 7, wherein the multi-task classification model comprises:
an EfficientNet or ResNext-based neural network (Paragraph 164 discloses neural networks include, but are not limited to MobileNet V1, MobileNet V2, MobileNet V3, ResNet, NASNet, EfficientNet, and others);
a separate classification head for each classification task (Paragraph 164 discloses context classification models, object recognition models, or other models, may be generated using a neural network architecture that runs efficiently on mobile computing devices (e.g., smart phones, tablet computing devices, etc.)); and
a softmax activation for each respective classification head (Paragraph 75 includes the CNN may include, in addition to the plurality of convolutional layers, a number of batch normalization layers, a number of ReLU layers, a number of max-pooling layers, one or more fully-connected layers, and one or more Softmax layers).
The Turkelson et al. reference as modified by Perera et al. teaches all the limitations of claim 7. With respect to claim 11, Turkelson et al. teaches the method of claim 7, wherein optimizing, each classification head in the plurality of classification heads, using the respective computed loss values for the classification head comprises:
identifying an index value of a given training image and label (Paragraph 158 discloses enhance a training set for a visual search process that includes the following operations: 1) importing a batch of catalog product images, which may be passed to a deep neural network that extracts deep features for each image, which may be used to create and store an index);
determining the index value is a valid index value for a first classification head (Paragraph 159 discloses Based on the distance (e.g., if the distance is less than 0.05 on a scale of 0-1), embodiments may designate the search was successful with a value indicating relatively high confidence, and embodiments may add the query image to the product catalog as ground truth to the index); and
in response to determining the index value is a valid index value for the first classification head, optimizing the first classification head using a loss value computed using the given training image and label (Paragraph 181 discloses the weights and biases of the object recognition model may be adjusted based on changes to an optimization of a loss function for the model as a result of the newly added subset of features).
The Turkelson et al. reference as modified by Perera et al. teaches all the limitations of claim 11. With respect to claim 12, Turkelson et al. teaches the method of claim 11, wherein determining the index value is a valid index value for the first classification head comprises:
determining the index value is included in a set of valid index values for the first classification head (Paragraph 158 discloses create and store an index; later, at run time, 2) receive a query image, pass the image to a deep neural network that extracts deep features, before computing distances to all images in the index and presenting a nearest neighbor as a search result).
With respect to claim 13, teaches a system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving, from a client device, a query requesting to search a dataset of images using a first input image, wherein the query includes first image data for the first input image (Paragraph 67 discloses a visual search may be executed on client-side (e.g., on a mobile computing device) and Paragraph 116 discloses visual search using the image of the object as an input query image);
inputting the first image data into a multi-task classification model trained to identify one or more images from the dataset of images in response to image data for a particular image (Paragraph 158 discloses receive a query image, pass the image to a deep neural network that extracts deep features, before computing distances to all images in the index and presenting a nearest neighbor as a search result),
wherein the multi-task classification model is a neural network that is trained using a plurality of classification heads corresponding to a plurality of classification tasks (Paragraph 174 discloses the object recognition model may be a deep learning model, such as, and without limitation, a convolutional neural network (CNN), a region-based CNN (R-CNN), a Fast R-CNN, a Masked R-CNN, Single Shot Multibox (SSD), and a You-Only-Look-Once (YOLO) model),
obtaining, as output from a layer of the neural network preceding the plurality of classification heads used for training and in response to the first image data, a first embedding for the first image data (Paragraph 147 discloses produce an embedding for each image from the training data set and each newly received image);
identifying, using the first embedding and from among a plurality of embeddings corresponding to images in the dataset of images, a set of images that are similar to the first input image (Paragraph 170 discloses identify the object and retrieve information regarding the identified object); and
providing, in response to the received query and for display on a device, the set of images (Paragraph 116 discloses provide information regarding one or more results of the visual search). Turkelson et al. does not disclose wherein the multi-task classification model is trained using separate loss functions for each respective classification task.
However, Perera et al. teaches wherein the multi-task classification model is trained using separate loss functions for each respective classification task (Paragraph 76 discloses the generative-discriminative image-classification system 206 in determines a self-supervision loss and a classification loss utilizing the loss functions 624),
wherein the plurality of classification tasks include one or more classification tasks based on separate training datasets (Paragraph 93 discloses classification improvement over conventional systems with respect to the corresponding test datasets for out-of-distribution image samples);
Therefore, it would have been obvious before the effective filing date of invention was made to a person having ordinary skill in the art to modify Turkelson et al. with Perera et al. This would have image classification and searching. See Perera et al. Paragraph(s) 2-3.
With respect to claim 14, it is rejected on grounds corresponding to above rejected claim 2, because claim 14 is substantially equivalent to claim 2.
With respect to claim 15, it is rejected on grounds corresponding to above rejected claim 3, because claim 15 is substantially equivalent to claim 3.
With respect to claim 16, it is rejected on grounds corresponding to above rejected claim 4, because claim 16 is substantially equivalent to claim 4.
With respect to claim 17, it is rejected on grounds corresponding to above rejected claim 5, because claim 17 is substantially equivalent to claim 5.
With respect to claim 18, it is rejected on grounds corresponding to above rejected claim 6, because claim 18 is substantially equivalent to claim 6.
With respect to claim 19, it is rejected on grounds corresponding to above rejected claim 7, because claim 19 is substantially equivalent to claim 7.
With respect to claim 20, Turkelson et al. teaches one or more computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
receiving, from a client device, a query requesting to search a dataset of images using a first input image, wherein the query includes first image data for the first input image (Paragraph 67 discloses a visual search may be executed on client-side (e.g., on a mobile computing device) and Paragraph 116 discloses visual search using the image of the object as an input query image);
inputting the first image data into a multi-task classification model trained to identify one or more images from the dataset of images in response to image data for a particular image (Paragraph 158 discloses receive a query image, pass the image to a deep neural network that extracts deep features, before computing distances to all images in the index and presenting a nearest neighbor as a search result),
wherein the multi-task classification model is a neural network that is trained using a plurality of classification heads corresponding to a plurality of classification tasks (Paragraph 174 discloses the object recognition model may be a deep learning model, such as, and without limitation, a convolutional neural network (CNN), a region-based CNN (R-CNN), a Fast R-CNN, a Masked R-CNN, Single Shot Multibox (SSD), and a You-Only-Look-Once (YOLO) model),
obtaining, as output from a layer of the neural network preceding the plurality of classification heads used for training and in response to the first image data, a first embedding for the first image data (Paragraph 147 discloses produce an embedding for each image from the training data set and each newly received image);
identifying, using the first embedding and from among a plurality of embeddings corresponding to images in the dataset of images, a set of images that are similar to the first input image (Paragraph 170 discloses identify the object and retrieve information regarding the identified object); and
providing, in response to the received query and for display on a device, the set of images (Paragraph 116 discloses provide information regarding one or more results of the visual search). Turkelson et al. does not disclose wherein the multi-task classification model is trained using separate loss functions for each respective classification task.
However, Perera et al. teaches wherein the multi-task classification model is trained using separate loss functions for each respective classification task (Paragraph 76 discloses the generative-discriminative image-classification system 206 in determines a self-supervision loss and a classification loss utilizing the loss functions 624),
wherein the plurality of classification tasks include one or more classification tasks based on separate training datasets (Paragraph 93 discloses classification improvement over conventional systems with respect to the corresponding test datasets for out-of-distribution image samples);
Therefore, it would have been obvious before the effective filing date of invention was made to a person having ordinary skill in the art to modify Turkelson et al. with Perera et al. This would have image classification and searching. See Perera et al. Paragraph(s) 2-3.
Relevant Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US PG-PUB 20190043201 is directed to ANALYTIC IMAGE FORMAT FOR VISUAL COMPUTING: [0232] To illustrate, image classification is a common visual data operation that uses a neural network to identify the contents of an image. For example, in machine learning, a convolutional neural network (CNN) is a type of feed-forward artificial neural network where the input is generally assumed to be an image. CNNs are commonly used for image classification, where the goal is to determine the contents of an image with some level of confidence. For example, a CNN is first trained for a specific classification task using a set of images whose object classes or features have been labeled, and the CNN can then be used to determine the probability of whether other images contain the respective object classes.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS E ALLEN whose telephone number is (571)270-3562. The examiner can normally be reached Monday through Thursday 830-630.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Boris Gorney can be reached at (571) 270-5626. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.E.A/Examiner, Art Unit 2154
/SYED H HASAN/Primary Examiner, Art Unit 2154