DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (“IDS”) filed on 01/23/2025 was reviewed and the listed references were noted.
Drawings
The 14-page drawings have been considered and placed on record in the file.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-8, 14-15, and 18-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more, and the claimed invention is directed to non-statutory subject matter as follows. The independent claims recite a memorability prediction system comprising three machine learning models receiving an image, partitioning the image, generating a first value based on the image, identifying a relationship between a sub-image and the received image, generating intermediation information based on the relationship, and generating a final value based on the first value and the intermediate information.
Step 1:
With regard to Step 1, the instant claims are directed to a method, which is among the statutory categories of invention.
Step 2A – Prong 1:
With regard to Step 2A – Prong 1, for example in Claim 1, the limitations of "generating, by the MPS, a first value based at least in part on the received image; identifying, by the MPS, a relationship between a first sub-image of the plurality of sub-images and the received image; generating, by the MPS, intermediate information based at least in part on the identified relationship between the first sub-image and the received image; and generating, by the MPS, a final value based at least in part on the first value and the intermediate information”, as drafted only involves mental processes, such as the identification of a relationship between a sub-image and the received image. That is, nothing in the above-described claim elements preclude the steps from practically being performed in the mind or on a piece of paper. If a claim limitation, under its broadest reasonably interpretation covers performance of the limitation in the mind or through mathematical calculations, but for the recitation of a generic apparatus components, such as a processor, then it falls within the "mental processes", which include concepts performed in the human mind, including an observation, evaluation, judgement, opinion, or mathematical calculations groupings of the abstract idea. Accordingly, the claim recites an abstract idea.
Step 2A – Prong 2:
The 2019 PEG defines the phrase “integration into a practical application” to require an additional element or a combination of additional elements in the claim to apply, rely on, or use the judicial exception. In the instant case, the additional elements in the claims do not apply, rely on, or use the judicial exception.
This judicial exception is not integrated into a practical application because the claim only recites the following additional step "A method, comprising: receiving, by a memorability prediction system (MPS), an image file corresponding to an image, the MPS comprising a first machine learning (ML) model, a second ML model, and a third ML model; partitioning, by the MPS, the received image into a plurality of sub-images”, i.e., insignificant extra-solution activity. The other additional recited element in certain other claims is just a processor and a computer-readable storage medium, which are generic computer components. Accordingly, these additional elements do not integrate the abstract idea into a practical application because it is a field-of-use limitation that does not impose any meaningful limits on practicing the abstract idea. Therefore, the claim as a whole, recites an abstract idea.
Step 2B:
Because the claim fails under Step 2A, the claims are further evaluated under Step 2B. The claim herein does not include additional steps that are sufficient to amount to significantly more than the judicial exception because as discussed above with respect to integration of the abstract idea into practical application, the additional elements/steps amount to no more than insignificant extra-solution activities. Mere instructions to apply an exception using generic apparatus component, such as a processor, cannot provide an inventive concept. The claim is not patent eligible. It should be noted that a similar analysis may be performed with respect to independent Claims 14 and 18.
Further, with regard to dependent Claims 2-8, 15, and 19 viewed individually, these additional steps are under their broadest reasonable interpretation, cover performance of the limitation in the mind and do not provide meaningful limitations to transform the abstract idea into a patent eligible application of the abstract idea such that the claims limitations amount to significantly more than the abstract idea itself. For example, the first and second models comprising vision transformers and the third model is a convolution residual network as recited in Claim 2 or passing the relationship from the first to the second model as recited in Claim 5 are only examples of routine and conventional image processing steps or steps that could be completed within the human mind and do not amount to significantly more to consider as inventive steps. Accordingly, Claims 1-8, 14-15, and 18-19 are rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3-6, and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Basavaraju et al. ("Multiple instance learning based deep CNN for image memorability prediction") in view of Dubey ("What makes an object memorable?") and Fajtl et al. (“Amnet: Memorability estimation with attention”).
Regarding Claim 1, Basavaraju teaches "A method, comprising: receiving, by a memorability prediction system (MPS), an image file corresponding to an image, the MPS comprising a first machine learning (ML) model, a second ML model, and a third ML model"; (Basavaraju, Figure 2 and Section 3.1, teaches a deep learning-based prediction model to predict image memorability scores wherein the proposed model contains a VGGMemNet applied to an input image and two MCDRMemNets in which one is applied to an input image and one is applied to salient image patches, i.e., memorability prediction system which receives an image file and comprises three machine learning models being the one VGGMemNet and the two MCDRMemNets);
"partitioning, by the MPS, the received image into a plurality of sub-images"; (Basavaraju, Section 3.2 and Figure 4, teaches local patches are extracted from top four salient regions, i.e., partition the received image into a plurality of sub-images being the local patches);
"generating, by the MPS, a first value based at least in part on the received image"; (Basavaraju, Figure 3 and Section 3.1, teaches the VGGMemNet model predicts memorability score Yo for the input image, i.e., memorability prediction system generates a first value based on the received image);
"
"
"and generating, by the MPS, a final value based at least in part on the first value and the intermediate information"; (Basavaraju, Figure 2 and Sections 3.2-3.3, teaches obtaining the final predicted memorability score for the given input image by ensembling the networks wherein the outputs of the MCDRNets on the input image and the image patches is a linear combination of inputs with equal weights in which a weighted average is computed for its memorability score Ye averaged with Yo to compute the final memorability Y, i.e., memorability prediction system generates a final memorability value based at least in part on the first memorability value of the first model and the intermediate information being the weighted memorability between image patches and the whole image).
However, Basavaraju does not explicitly teach "identifying, by the MPS, a relationship between a first sub-image of the plurality of sub-images and the received image; generating, by the MPS, intermediate information based at least in part on the identified relationship between the first sub-image and the received image”.
In an analogous field of endeavor, Dubey teaches "identifying, by the MPS, a relationship between a first sub-image of the plurality of sub-images and the received image"; (Dubey, Sections 2, 3.4, and 4, teaches the objects inside images are defined as image segments wherein the correlation between the scores of the most memorable object in each image and the memorability score of each image is computed wherein the model utilizes a conv-net with a trained support vector regressor to map deep features to memorability scores, i.e., identify by the memorability prediction system a relationship being the correlation between the sub-image being the segment and the received image).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Basavaraju by including the identification of a relationship between a sub-image and the received image taught by Dubey. One of ordinary skill in the art would be motivated to combine the references since it allows for high predictive ability sensitive to segmentation (Dubey, Section 4, teaches the motivation of combination to be that the conv-net model has high predictive ability which is sensitive to the segmentations).
However, the combination of references of Basavaraju in view of Dubey does not explicitly teach "generating, by the MPS, intermediate information based at least in part on the identified relationship between the first sub-image and the received image".
In an analogous field of endeavor, Fajtl teaches "generating, by the MPS, intermediate information based at least in part on the identified relationship between the first sub-image and the received image"; (Fajtl, Figures 1 and 2 and Sections 3.2-3.3, teaches the CNN extracts image features as a tensor with dimensions having 14 x 14 locations represented as a vector X wherein the soft attention network model produces a probability weight for every information element in which the attention probabilities are conditioned on the entire image feature vector and previous LSTM hidden state, i.e., memorability prediction system generating intermediate information being a relative weight of the sub-image being the probability weight of the spatial feature location based on the relationship between the sub-image and the received image being the weight calculated for every information element being every location conditioned on the entire image).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Basavaraju and Dubey wherein the identified relationship is a memorability relationship between the sub-image and the received image by including the generation of intermediation information based on the identified relationship taught by Fajtl. One of ordinary skill in the art would be motivated to combine the references since it improve memorability (Fajtl, Section 1, teaches the motivation of combination to be to improve memorability of specific parts of an interface).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Regarding Claim 3, the combination of references of Basavaraju in view of Dubey and Fajtl teaches "The method of claim 1, wherein the first value is a standalone memorability of the received image generated by the first ML model"; (Basavaraju, Figure 3 and Section 3.1, teaches the VGGMemNet model predicts memorability score Y0 for the input image, i.e., memorability prediction system generates a first value being a standalone memorability based on the received image generated by the first model).
Regarding Claim 4, the combination of references of Basavaraju in view of Dubey and Fajtl teaches "The method of claim 1, wherein the relationship between a first sub-image of the plurality of sub-images and the received image is information indicating the relationship between their estimated memorability, and wherein the relationship is generated by the first ML model"; (Dubey, Sections 2, 3.4, and 4, teaches the objects inside images are defined as image segments wherein the correlation between the scores of the most memorable object in each image and the memorability score of each image is computed wherein the model utilizes a conv-net with a trained support vector regressor to map deep features to memorability scores, i.e., relationship between the sub-image and the received image being the correlation between the object segment and the image is information indicating the relationship between the estimated memorability of the segment and the image wherein relationship is generated with help from the machine learning model).
The proposed combination as well as the motivation for combining the Basavaraju, Dubey, and Fajtl references presented in the rejection of Claim 1, applies to claim 4. Thus, the method recited in claim 4 is met by Basavaraju in view of Dubey and Fajtl.
Regarding Claim 5, the combination of references of Basavaraju in view of Dubey and Fajtl teaches "The method of claim 4, further comprising passing the relationship from the first ML model to the second ML model"; (Basavaraju, Sections 3.2-3.3, teaches the last fully-connected layer output from the upper branch provides a single global representation and the last FC layer output of the lower branch provides four local representations in which the representation of the entire bag is produced by aggregating the global and local representations which is also then passed forward to the other model for memorability prediction, i.e., relationship of the upper branch being its global representation is passed to the second model being the lower branch for aggregation with the local representations).
Regarding Claim 6, the combination of references of Basavaraju in view of Dubey and Fajtl teaches "The method of claim 1, wherein the intermediate information, generated by the second ML model, comprises a relative weight of the first sub-image of the plurality of sub-images"; (Fajtl, Figures 1 and 2 and Sections 3.2-3.3, teaches the CNN extracts image features as a tensor with dimensions having 14 x 14 locations represented as a vector X wherein the soft attention network model produces a probability weight for every information element in which the attention probabilities are conditioned on the entire image feature vector and previous LSTM hidden state, i.e., intermediate information is a relative weight of the sub-image generated by a second model).
The proposed combination as well as the motivation for combining the Basavaraju, Dubey, and Fajtl references presented in the rejection of Claim 1, applies to claim 6. Thus, the method recited in claim 6 is met by Basavaraju in view of Dubey and Fajtl.
Regarding Claim 9, the combination of references of Basavaraju in view of Dubey and Fajtl teaches "The method of claim 1, further comprising training the MPS using a plurality of training datapoints, wherein each training datapoint in the plurality of training datapoints comprises a training image and ground truth information, and ground truth information comprises a target memorability of the training image"; (Basavaraju, Sections 4.2-4.4, teaches the three copies of the models are trained on the LaMem dataset which contains 60,000 image sand divided into 5 sets for cross-validation which contains 45,000 training samples, 10,000 testing samples, and 3,741 validation samples wherein performance is represented by means of rank correlation between ground truth and predicted memorability scores of the image, i.e., training using a plurality of data comprising a training image and ground truth information comprising target memorability of the image being the ground truth memorability scores).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Basavaraju in view of Dubey, Fajtl, and Jingmin et al. ("Insect recognition based on complementary features from multiple views").
Regarding Claim 2, the combination of references of Basavaraju in view of Dubey and Fajtl does not explicitly teach "The method of claim 1, wherein the first ML model is a first vision transformer, the second ML model is a second vision transformer, and the third ML model is a convolution residual network".
In an analogous field of endeavor Jingmin teaches "The method of claim 1, wherein the first ML model is a first vision transformer, the second ML model is a second vision transformer, and the third ML model is a convolution residual network"; (Jingmin, Figure 1, teaches inputting an image to one CNN-based backbone being ResNet152 and two attention-based backbones being ViT and Swin-T, i.e., two models are a first and second vision transformer and the third is a convolution residual network).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Basavaraju, Dubey, and Fajtl by including the models including a first and second vision transformer and a convolution residual network taught by Jingmin. One of ordinary skill in the art would be motivated to combine the references since it increases performance and robustness (Jingmin, Abstract teaches the motivation of combination to be to increase classification performance and demonstrate good robustness).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Basavaraju in view of Dubey, Fajtl, and Khosla et al. ("Understanding and predicting image memorability at a large scale").
Regarding Claim 7, the combination of references of Basavaraju in view of Dubey and Fajtl does not explicitly teach "The method of claim 1, further comprising generating values of a memorability map per pixel by the third ML model".
In an analogous field of endeavor, Khosla teaches "The method of claim 1, further comprising generating values of a memorability map per pixel by the third ML model"; (Khosla, Section 5, teaches applying a fully-convolutional network to images of arbitrary sizes to generate different sized memorability maps and averaging the outputs to generate the final memorability map, i.e., third model generates values of a memorability map per pixel).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Basavaraju, Dubey, and Fajtl by including the generation of values for a memorability map by a third model taught by Khosla. One of ordinary skill in the art would be motivated to combine the references since it captures cognitively salient regions (Khosla, Section 5, teaches the motivation of combination to be to capture cognitively salient regions with meaningful objects).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Basavaraju in view of Dubey, Fajtl, Khosla, and Zaffar et al. (“Memorable maps: A framework for re-defining places in visual place recognition”).
Regarding Claim 8, the combination of references of Basavaraju in view of Dubey, Fajtl, and Khosla teaches "
"and wherein the final value is a memorability score of the received image"; (Basavaraju, Figure 2 and Sections 3.2-3.3, teaches obtaining the final predicted memorability score for the given input image by ensembling the networks wherein the outputs of the MCDRNets on the input image and the image patches is a linear combination of inputs with equal weights in which a weighted average is computed for its memorability score Ye averaged with Yo to compute the final memorability Y, i.e., final value is a memorability score of the image).
However, the combination of references of Basavaraju in view of Dubey, Fajtl, and Khosla does not explicitly teach "The method of claim 7, wherein the final value is further based at least in part on the values of memorability map per pixel".
In an analogous field of endeavor, Zaffar teaches "The method of claim 7, wherein the final value is further based at least in part on the values of memorability map per pixel"; (Zaffar, Section III-D and Section III-E, teaches computing the memorability score of an image as the average value of the memorability map, i.e., final value being the memorability score of the image based on the memorability map values).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Basavaraju, Dubey, Fajtl, and Khosla by including the final value being based in part on memorability map values taught by Zaffar. One of ordinary skill in the art would be motivated to combine the references since it provides a significant performance boost (Zaffar, Abstract, teaches the motivation of combination to be to provide a significant performance boost to visual place recognition).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Claims 10-13 are rejected under 35 U.S.C. 103 as being unpatentable over Basavaraju in view of Dubey, Fajtl, Zaffar, and Andoni et al. (US 20190228312 A1).
Regarding Claim 10, the combination of references of Basavaraju in view of Dubey, Fajtl, and Zaffar teaches "The method of claim 9, wherein training the MPS further comprises, for at least a first training datapoint in the plurality of training datapoints: partitioning the training image into a first training sub-image and a second training sub-image"; (Basavaraju, Sections 4.2-4.3, teaches the training process of the MCDRNet for the upper and lower branch including loading the branches with pre-trained weights of EmoMemNet wherein MCDRNet is fine-tuned under MIL framework and is fed with one global patch and four local patches extracted from top four salient regions from the original image, i.e., partition training image into at least a first and second training sub-image being the training process including the extraction of training local patches);
"training the first ML model to predict a standalone memorability based in part on the training image"; (Basavaraju, Figure 3 and Section 3.1 and Section 4.2, teaches the VGGMemNet model predicts memorability score Yo for the input image, i.e., memorability prediction system generates a first value based on the received image);
"training a second ML model to predict a first relative memorability based in part on the first training sub-image and a second relative memorability based in part on the second training sub-images"; (Dubey, Section 4, teaches a trained support vector regressor using 6-fold cross-validation on the original object segments to map deep features to memorability scores and using that model to predict memorability scores for the top K = 20 object segments obtained using the MCG algorithm, i.e., a second ML model predicting relative memorability based on the each training sub-image being the predicted memorability score for each object segment);
"training a third ML model to predict values of memorability map per pixel based in part on the training image"; (Zaffar, Section III-A, teaches a fine-tuned Hybrid-CNN which splits the re-scaled image into non-overlapping crops and sequentially feed them as inputs to the CNN to output a memorability matrix which comprises a memorability of each cropped image in which the matrix is rescaled with bilinear interpolation to create a memorability map, i.e., a third model predicts values of the memorability map per pixel based on the training image).
The proposed combination as well as the motivation for combining the Basavaraju, Dubey, Fajtl, and Zaffar references presented in the rejection of Claims 1 and 8, applies to claim 10.
However, the combination of references of Basavaraju in view of Dubey, Fajtl, and Zaffar does not explicitly teach "generating an aggregated loss based in part on a first loss associated with the first ML model, a second loss associated with the second ML model, and a third loss associated with the third ML model; and minimizing the aggregated loss using a loss minimization technique wherein the minimizing comprises updating one or more trainable parameters associated with the first ML model, the second ML model, and the third ML model”.
In an analogous field of endeavor, Andoni teaches "generating an aggregated loss based in part on a first loss associated with the first ML model, a second loss associated with the second ML model, and a third loss associated with the third ML model"; (Andoni, FIG 1A and Paras. 21-22, 24, and 29-33, teaches an aggregate loss L of equation 8 is based on reconstruction loss as well as first and second KL divergence loss wherein the first neural network outputs the cluster probability vector, the second neural network outputs reconstruction and variance, and the third neural network outputs latent space distributions, i.e., generate an aggregated loss based on three losses associated with the three machine learning models);
"and minimizing the aggregated loss using a loss minimization technique wherein the minimizing comprises updating one or more trainable parameters associated with the first ML model, the second ML model, and the third ML model"; (Andoni, FIG. 1A and Para.33, teaches the calculator/detector may initiate adjustment to the first, second, and third neural based on the aggregate loss wherein ink weights, bias functions, bias values, etc. may be modified via backpropagation to minimize the aggregate loss L using stochastic gradient descent, i.e., minimize aggregated loss by updating trainable parameters associated with the three models).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Basavaraju in view of Dubey, Fajtl, and Zaffar by including the generation of an aggregated loss and the minimization of the aggregated loss taught by Andoni. One of ordinary skill in the art would be motivated to combine the references since it improves performance (Andoni, Para. 33, teaches the motivation of combination to be to minimize the loss of the network which improves performance).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Regarding Claim 11, the combination of references of Basavaraju in view of Dubey, Fajtl, Zaffar, and Andoni teaches "The method of claim 10, wherein training the first ML model further comprises computing the first loss based at least in part on the predicted standalone memorability of the training image and the ground truth information"; (Basavaraju, Section 4.2, teaches the l2 loss function used to train the models which is evaluated as the summation of the difference between the predicted and the ground truth memorability scores of the training images, i.e., model comprises computing the loss based on the predicted memorability and the ground truth).
Regarding Claim 12, the combination of references of Basavaraju in view of Dubey, Fajtl, Zaffar, and Andoni teaches "The method of claim 10, wherein training the second ML model further comprises: calculating a sum of the first predicted relative memorability and the second predicted relative memorability; and computing the second loss based at least in part on the sum and the ground truth information"; (Basavaraju, Section 3.2 and 4.2, teaches the predicted memorability score of the entire bag is defined using an aggregate function used to combine the global and local representations wherein a loss is computed based on the difference between the predicted and ground truth memorability scores of the image, i.e., calculate a sum of predicted relative memorability being the aggregate function of the local representations and computing a loss based on the sum and the ground truth).
Regarding Claim 13, the combination of references of Basavaraju in view of Dubey, Fajtl, Zaffar, and Andoni teaches "The method of claim 10, wherein training the third ML model further comprises: calculating a sum of the predicted values of the memorability map per pixel"; (Zaffar, Section III-A and Section III-D, teaches a fine-tuned Hybrid-CNN which splits the re-scaled image into non-overlapping crops and sequentially feed them as inputs to the CNN to output a memorability matrix which comprises a memorability of each cropped image in which the matrix is rescaled with bilinear interpolation to create a memorability map wherein the memorability score of an image is computed as the average value of the memorability map, i.e., calculate sum of predicted values of the memorability map per pixel via the average memorability map calculation);
"and computing the third loss based at least in part on the sum and the ground truth information"; (Basavaraju, Section 3.2 and 4.2, teaches the predicted memorability score of the entire bag is defined using an aggregate function used to combine the global and local representations wherein a loss is computed based on the difference between the predicted and ground truth memorability scores of the image, i.e., compute a loss based in part on the sum and ground truth).
The proposed combination as well as the motivation for combining the Basavaraju, Dubey, Fajtl, Zaffar, and Andoni references presented in the rejection of Claim 10, applies to claim 13. Thus, the method recited in claim 13 is met by Basavaraju in view of Dubey, Fajtl, Zaffar, and Andoni.
Claims 14, 16, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Basavaraju in view of Dubey, Fajtl, and Pan et al. (US 20230196710 A1).
Regarding Claim 14, the combination of references of Basavaraju in view of Dubey and Fajtl teaches "
"receiving, by a memorability prediction system (MPS), an image file corresponding to an image, the MPS comprising a first machine learning (ML) model, a second ML model, and a third ML model"; (Basavaraju, Figure 2 and Section 3.1, teaches a deep learning-based prediction model to predict image memorability scores wherein the proposed model contains a VGGMemNet applied to an input image and two MCDRMemNets in which one is applied to an input image and one is applied to salient image patches, i.e., memorability prediction system which receives an image file and comprises three machine learning models being the one VGGMemNet and the two MCDRMemNets);
"partitioning, by the MPS, the received image into a plurality of sub-images"; (Basavaraju, Section 3.2 and Figure 4, teaches local patches are extracted from top four salient regions, i.e., partition the received image into a plurality of sub-images being the local patches);
"generating, by the MPS, a first value based at least in part on the received image"; (Basavaraju, Figure 3 and Section 3.1, teaches the VGGMemNet model predicts memorability score Yo for the input image, i.e., memorability prediction system generates a first value based on the received image);
"identifying, by the MPS, a relationship between a first sub-image of the plurality of sub-images and the received image by the first ML model"; (Dubey, Sections 2, 3.4, and 4, teaches the objects inside images are defined as image segments wherein the correlation between the scores of the most memorable object in each image and the memorability score of each image is computed wherein the model utilizes a conv-net with a trained support vector regressor to map deep features to memorability scores, i.e., identify by the memorability prediction system a relationship being the correlation between the sub-image being the segment and the received image);
"passing, by the MPS, the relationship from the first ML model to the second ML model"; (Basavaraju, Sections 3.2-3.3, teaches the last fully-connected layer output from the upper branch provides a single global representation and the last FC layer output of the lower branch provides four local representations in which the representation of the entire bag is produced by aggregating the global and local representations which is also then passed forward to the other model for memorability prediction, i.e., relationship of the upper branch being its global representation is passed to the second model being the lower branch for aggregation with the local representations);
"generating, by the MPS, intermediate information based at least in part on the identified relationship between the first sub-image and the received image"; (Fajtl, Figures 1 and 2 and Sections 3.2-3.3, teaches the CNN extracts image features as a tensor with dimensions having 14 x 14 locations represented as a vector X wherein the soft attention network model produces a probability weight for every information element in which the attention probabilities are conditioned on the entire image feature vector and previous LSTM hidden state, i.e., memorability prediction system generating intermediate information being a relative weight of the sub-image being the probability weight of the spatial feature location based on the relationship between the sub-image and the received image being the weight calculated for every information element being every location conditioned on the entire image);
"and generating, by the MPS, a final value based at least in part on the first value, and the intermediate information"; (Basavaraju, Figure 2 and Sections 3.2-3.3, teaches obtaining the final predicted memorability score for the given input image by ensembling the networks wherein the outputs of the MCDRNets on the input image and the image patches is a linear combination of inputs with equal weights in which a weighted average is computed for its memorability score Ye averaged with Yo to compute the final memorability Y, i.e., memorability prediction system generates a final memorability value based at least in part on the first memorability value of the first model and the intermediate information being the weighted memorability between image patches and the whole image).
The proposed combination as well as the motivation for combining the Basavaraju, Dubey, and Fajtl references presented in the rejection of Claim 1, applies to claim 14.
However, the combination of references of Basavaraju in view of Dubey and Fajtl does not explicitly teach "A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a computing system, cause the one or more processors to perform operations comprising:".
In an analogous field of endeavor, Pan teaches "A non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a computing system, cause the one or more processors to perform operations comprising:"; (Pan, Paras. 6, 42, and 87, teaches a computer readable storage medium wherein a system includes a processor and memory device coupled with the processor to perform operations).
It would have been obvious to one having ordinary skill in the art before the effective filing date to modify the invention of Basavaraju, Dubey, and Fajtl by including the non-transitory computer-readable medium storing instructions for execution taught by Pan. One of ordinary skill in the art would be motivated to combine the references since allows the processor to be configured for fine-tuning (Pan, Para. 6, teaches the motivation of combination to be to fine-tune the neural network).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date.
Regarding Claim 16, the combination of references of Basavaraju in view of Dubey, Fajtl, and Pan teaches "The non-transitory computer-readable medium of claim 14, further comprising training the MPS using a plurality of training datapoints, wherein each training datapoint in the plurality of training datapoints comprises a training image and ground truth information, and ground truth information comprises a target memorability of the training image"; (Basavaraju, Sections 4.2-4.4, teaches the three copies of the models are trained on the LaMem dataset which contains 60,000 image sand divided into 5 sets for cross-validation which contains 45,000 training samples, 10,000 testing samples, and 3,741 validation samples wherein performance is represented by means of rank correlation between ground truth and predicted memorability scores of the image, i.e., training using a plurality of data comprising a training image and ground truth information comprising target memorability of the image being the ground truth memorability scores).
Regarding Claim 18, the combination of references of Basavaraju in view of Dubey, Fajtl, and Pan teaches "A computing system, comprising: one or more processors; and one or more non-transitory computer readable media storing computer-executable instructions that, when executed by the one or more processors of the computing system, cause the computing system to:"; (Pan, Paras. 6, 42, and 87, teaches a computer readable storage medium wherein a system includes a processor and memory device coupled with the processor to perform operations);
"receive, by a memorability prediction system (MPS) of the computing system, an image file corresponding to an image, the MPS comprising a first machine learning (ML) model, a second ML model, and a third ML model"; (Basavaraju, Figure 2 and Section 3.1, teaches a deep learning-based prediction model to predict image memorability scores wherein the proposed model contains a VGGMemNet applied to an input image and two MCDRMemNets in which one is applied to an input image and one is applied to salient image patches, i.e., memorability prediction system which receives an image file and comprises three machine learning models being the one VGGMemNet and the two MCDRMemNets);
"partition, by the computing system, the received image into a plurality of sub-images"; (Basavaraju, Section 3.2 and Figure 4, teaches local patches are extracted from top four salient regions, i.e., partition the received image into a plurality of sub-images being the local patches);
"generate, by the computing system, a first value based at least in part on the received image"; (Basavaraju, Figure 3 and Section 3.1, teaches the VGGMemNet model predicts memorability score Yo for the input image, i.e., memorability prediction system generates a first value based on the received image);
"identify, by the computing system, a relationship between a first sub-image of the plurality of sub-images and the received image by the first ML model"; (Dubey, Sections 2, 3.4, and 4, teaches the objects inside images are defined as image segments wherein the correlation between the scores of the most memorable object in each image and the memorability score of each image is computed wherein the model utilizes a conv-net with a trained support vector regressor to map deep features to memorability scores, i.e., identify by the memorability prediction system a relationship being the correlation between the sub-image being the segment and the received image);
"pass, by the computing system, the relationship from the first ML model to the second ML model"; (Basavaraju, Sections 3.2-3.3, teaches the last fully-connected layer output from the upper branch provides a single global representation and the last FC layer output of the lower branch provides four local representations in which the representation of the entire bag is produced by aggregating the global and local representations which is also then passed forward to the other model for memorability prediction, i.e., relationship of the upper branch being its global representation is passed to the second model being the lower branch for aggregation with the local representations);
"generate, by the computing system, intermediate information based at least in part on the identified relationship between the first sub-image and the received image"; (Fajtl, Figures 1 and 2 and Sections 3.2-3.3, teaches the CNN extracts image features as a tensor with dimensions having 14 x 14 locations represented as a vector X wherein the soft attention network model produces a probability weight for every information element in which the attention probabilities are conditioned on the entire image feature vector and previous LSTM hidden state, i.e., memorability prediction system generating intermediate information being a relative weight of the sub-image being the probability weight of the spatial feature location based on the relationship between the sub-image and the received image being the weight calculated for every information element being every location conditioned on the entire image);
"and generate, by the computing system, a final value based at least in part on the first value, and the intermediate information"; (Basavaraju, Figure 2 and Sections 3.2-3.3, teaches obtaining the final predicted memorability score for the given input image by ensembling the networks wherein the outputs of the MCDRNets on the input image and the image patches is a linear combination of inputs with equal weights in which a weighted average is computed for its memorability score Ye averaged with Yo to compute the final memorability Y, i.e., memorability prediction system generates a final memorability value based at least in part on the first memorability value of the first model and the intermediate information being the weighted memorability between image patches and the whole image).
The proposed combination as well as the motivation for combining the Basavaraju, Dubey, Fajtl, and Pan references presented in the rejection of Claims 1 and 14, applies to claim 18. Thus, the system recited in claim 18 is met by Basavaraju in view of Dubey, Fajtl, and Pan.
Claims 15 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Basavaraju in view of Dubey, Fajtl, Pan, Khosla, and Zaffar.
Regarding Claim 15, the combination of references of Basavaraju in view of Dubey, Fajtl, Pan, Khosla, and Zaffar teaches "The non-transitory computer-readable medium of claim 14, further comprising: generating values of memorability map per pixel by the third ML model"; (Khosla, Section 5, teaches applying a fully-convolutional network to images of arbitrary sizes to generate different sized memorability maps and averaging the outputs to generate the final memorability map, i.e., third model generates values of a memorability map per pixel);
"generating the final value based at least in part on the first value, the intermediate information, (Basavaraju, Figure 2 and Sections 3.2-3.3, teaches obtaining the final predicted memorability score for the given input image by ensembling the networks wherein the outputs of the MCDRNets on the input image and the image patches is a linear combination of inputs with equal weights in which a weighted average is computed for its memorability score Ye averaged with Yo to compute the final memorability Y, i.e., memorability prediction system generates a final memorability value based at least in part on the first memorability value of the first model and the intermediate information being the weighted memorability between image patches and the whole image);
"generating the final value based at least in part on (Zaffar, Section III-D and Section III-E, teaches computing the memorability score of an image as the average value of the memorability map, i.e., final value being the memorability score of the image based on the memorability map values);
"wherein the relationship between a first sub-image of the plurality of sub-images and the received image is information indicating the relationship between their estimated memorability"; (Dubey, Sections 2, 3.4, and 4, teaches the objects inside images are defined as image segments wherein the correlation between the scores of the most memorable object in each image and the memorability score of each image is computed wherein the model utilizes a conv-net with a trained support vector regressor to map deep features to memorability scores, i.e., relationship between the sub-image and the received image being the correlation between the object segment and the image is information indicating the relationship between the estimated memorability of the segment and the image wherein relationship is generated with help from the machine learning model);
"wherein the intermediate information, generated by the second ML model, comprises a relative weight of the first sub-image of the plurality of sub-images"; (Fajtl, Figures 1 and 2 and Sections 3.2-3.3, teaches the CNN extracts image features as a tensor with dimensions having 14 x 14 locations represented as a vector X wherein the soft attention network model produces a probability weight for every information element in which the attention probabilities are conditioned on the entire image feature vector and previous LSTM hidden state, i.e., intermediate information is a relative weight of the sub-image generated by a second model);
"and wherein the final value is a memorability score of the received image"; (Basavaraju, Figure 2 and Sections 3.2-3.3, teaches obtaining the final predicted memorability score for the given input image by ensembling the networks wherein the outputs of the MCDRNets on the input image and the image patches is a linear combination of inputs with equal weights in which a weighted average is computed for its memorability score Ye averaged with Yo to compute the final memorability Y, i.e., final value is a memorability score of the image).
The proposed combination as well as the motivation for combining the Basavaraju, Dubey, Fajtl, Pan, Khosla, and Zaffar references presented in the rejection of Claims 1, 7, 8, and 14 applies to claim 15. Thus, the system recited in claim 15 is met by Basavaraju in view of Dubey, Fajtl, Pan, Khosla, and Zaffar.
Regarding Claim 19, the combination of references of Basavaraju in view of Dubey, Fajtl, Pan, Khosla, and Zaffar teaches "The computing system of claim 18, wherein the system is further caused to: generate values of memorability map per pixel by the third ML model"; (Khosla, Section 5, teaches applying a fully-convolutional network to images of arbitrary sizes to generate different sized memorability maps and averaging the outputs to generate the final memorability map, i.e., third model generates values of a memorability map per pixel);
"generate the final value based at least in part on the first value, the intermediate information, (Basavaraju, Figure 2 and Sections 3.2-3.3, teaches obtaining the final predicted memorability score for the given input image by ensembling the networks wherein the outputs of the MCDRNets on the input image and the image patches is a linear combination of inputs with equal weights in which a weighted average is computed for its memorability score Ye averaged with Yo to compute the final memorability Y, i.e., memorability prediction system generates a final memorability value based at least in part on the first memorability value of the first model and the intermediate information being the weighted memorability between image patches and the whole image);
"generate the final value based at least in part on the (Zaffar, Section III-D and Section III-E, teaches computing the memorability score of an image as the average value of the memorability map, i.e., final value being the memorability score of the image based on the memorability map values);
"wherein the relationship between a first sub-image of the plurality of sub-images and the received image is information indicating the relationship between their estimated memorability"; (Dubey, Sections 2, 3.4, and 4, teaches the objects inside images are defined as image segments wherein the correlation between the scores of the most memorable object in each image and the memorability score of each image is computed wherein the model utilizes a conv-net with a trained support vector regressor to map deep features to memorability scores, i.e., relationship between the sub-image and the received image being the correlation between the object segment and the image is information indicating the relationship between the estimated memorability of the segment and the image wherein relationship is generated with help from the machine learning model);
"wherein the intermediate information, generated by the second ML model, comprises a relative weight of the first sub-image of the plurality of sub-images"; (Fajtl, Figures 1 and 2 and Sections 3.2-3.3, teaches the CNN extracts image features as a tensor with dimensions having 14 x 14 locations represented as a vector X wherein the soft attention network model produces a probability weight for every information element in which the attention probabilities are conditioned on the entire image feature vector and previous LSTM hidden state, i.e., intermediate information is a relative weight of the sub-image generated by a second model);
"and wherein the final value is a memorability score of the received image"; (Basavaraju, Figure 2 and Sections 3.2-3.3, teaches obtaining the final predicted memorability score for the given input image by ensembling the networks wherein the outputs of the MCDRNets on the input image and the image patches is a linear combination of inputs with equal weights in which a weighted average is computed for its memorability score Ye averaged with Yo to compute the final memorability Y, i.e., final value is a memorability score of the image).
The proposed combination as well as the motivation for combining the Basavaraju, Dubey, Fajtl, Pan, Khosla, and Zaffar references presented in the rejection of Claims 1, 7, 8, and 14 applies to claim 19. Thus, the system recited in claim 19 is met by Basavaraju in view of Dubey, Fajtl, Pan, Khosla, and Zaffar.
Claims 17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Basavaraju in view of Dubey, Fajtl, Pan, Zaffar, and Andoni.
Regarding Claim 17, the combination of references of Basavaraju in view of Dubey, Fajtl, Pan, Zaffar, and Andoni teaches "The non-transitory computer-readable medium of claim 16, wherein training the MPS further comprises, for at least a first training datapoint in the plurality of training datapoints: partitioning the training image into a first training sub-image and a second training sub-image"; (Basavaraju, Sections 4.2-4.3, teaches the training process of the MCDRNet for the upper and lower branch including loading the branches with pre-trained weights of EmoMemNet wherein MCDRNet is fine-tuned under MIL framework and is fed with one global patch and four local patches extracted from top four salient regions from the original image, i.e., partition training image into at least a first and second training sub-image being the training process including the extraction of training local patches);
"training the first ML model to predict a standalone memorability based in part on the training image"; (Basavaraju, Figure 3 and Section 3.1 and Section 4.2, teaches the VGGMemNet model predicts memorability score Yo for the input image, i.e., memorability prediction system generates a first value based on the received image);
"training a second ML model to predict a first relative memorability based in part on the first training sub-image and a second relative memorability based in part on the second training sub-images"; (Dubey, Section 4, teaches a trained support vector regressor using 6-fold cross-validation on the original object segments to map deep features to memorability scores and using that model to predict memorability scores for the top K = 20 object segments obtained using the MCG algorithm, i.e., a second ML model predicting relative memorability based on the each training sub-image being the predicted memorability score for each object segment);
"training a third ML model to predict values of memorability map per pixel based in part on the training image"; (Zaffar, Section III-A, teaches a fine-tuned Hybrid-CNN which splits the re-scaled image into non-overlapping crops and sequentially feed them as inputs to the CNN to output a memorability matrix which comprises a memorability of each cropped image in which the matrix is rescaled with bilinear interpolation to create a memorability map, i.e., a third model predicts values of the memorability map per pixel based on the training image);
"generating an aggregated loss based in part on a first loss associated with the first ML model, a second loss associated with the second ML model, and a third loss associated with the third ML model"; (Andoni, FIG 1A and Paras. 21-22, 24, and 29-33, teaches an aggregate loss L of equation 8 is based on reconstruction loss as well as first and second KL divergence loss wherein the first neural network outputs the cluster probability vector, the second neural network outputs reconstruction and variance, and the third neural network outputs latent space distributions, i.e., generate an aggregated loss based on three losses associated with the three machine learning models);
"and minimizing the aggregated loss using a loss minimization technique wherein the minimizing comprises updating one or more trainable parameters associated with the first ML model, the second ML model, and the third ML model"; (Andoni, FIG. 1A and Para.33, teaches the calculator/detector may initiate adjustment to the first, second, and third neural based on the aggregate loss wherein ink weights, bias functions, bias values, etc. may be modified via backpropagation to minimize the aggregate loss L using stochastic gradient descent, i.e., minimize aggregated loss by updating trainable parameters associated with the three models).
The proposed combination as well as the motivation for combining the Basavaraju, Dubey, Fajtl, Pan, Zaffar, and Andoni references presented in the rejection of Claims 1, 8, 10, and 14 applies to claim 19. Thus, the system recited in claim 19 is met by Basavaraju in view of Dubey, Fajtl, Pan, Zaffar, and Andoni.
Regarding Claim 20, the combination of references of Basavaraju in view of Dubey, Fajtl, Pan, Zaffar, and Andoni teaches "The computing system of claim 18, wherein the system is further caused to: train the MPS using a plurality of training datapoints, wherein each training datapoint in the plurality of training datapoints comprises a training image and ground truth information, and ground truth information comprises a target memorability of the training image"; (Basavaraju, Sections 4.2-4.4, teaches the three copies of the models are trained on the LaMem dataset which contains 60,000 image sand divided into 5 sets for cross-validation which contains 45,000 training samples, 10,000 testing samples, and 3,741 validation samples wherein performance is represented by means of rank correlation between ground truth and predicted memorability scores of the image, i.e., training using a plurality of data comprising a training image and ground truth information comprising target memorability of the image being the ground truth memorability scores);
"wherein training the MPS further comprises, for at least a first training datapoint in the plurality of training datapoints: partitioning the training image into a first training sub-image and a second training sub-image"; (Basavaraju, Sections 4.2-4.3, teaches the training process of the MCDRNet for the upper and lower branch including loading the branches with pre-trained weights of EmoMemNet wherein MCDRNet is fine-tuned under MIL framework and is fed with one global patch and four local patches extracted from top four salient regions from the original image, i.e., partition training image into at least a first and second training sub-image being the training process including the extraction of training local patches);
"training the first ML model to predict a standalone memorability based in part on the training image"; (Basavaraju, Figure 3 and Section 3.1 and Section 4.2, teaches the VGGMemNet model predicts memorability score Yo for the input image, i.e., memorability prediction system generates a first value based on the received image);
"training a second ML model to predict a first relative memorability based in part on the first training sub-image and a second relative memorability based in part on the second training sub-images"; (Dubey, Section 4, teaches a trained support vector regressor using 6-fold cross-validation on the original object segments to map deep features to memorability scores and using that model to predict memorability scores for the top K = 20 object segments obtained using the MCG algorithm, i.e., a second ML model predicting relative memorability based on the each training sub-image being the predicted memorability score for each object segment);
"training a third ML model to predict values of memorability map per pixel based in part on the training image"; (Zaffar, Section III-A, teaches a fine-tuned Hybrid-CNN which splits the re-scaled image into non-overlapping crops and sequentially feed them as inputs to the CNN to output a memorability matrix which comprises a memorability of each cropped image in which the matrix is rescaled with bilinear interpolation to create a memorability map, i.e., a third model predicts values of the memorability map per pixel based on the training image);
"generating an aggregated loss based in part on a first loss associated with the first ML model, a second loss associated with the second ML model, and a third loss associated with the third ML model"; (Andoni, FIG 1A and Paras. 21-22, 24, and 29-33, teaches an aggregate loss L of equation 8 is based on reconstruction loss as well as first and second KL divergence loss wherein the first neural network outputs the cluster probability vector, the second neural network outputs reconstruction and variance, and the third neural network outputs latent space distributions, i.e., generate an aggregated loss based on three losses associated with the three machine learning models);
"and minimizing the aggregated loss using a loss minimization technique wherein the minimizing comprises updating one or more trainable parameters associated with the first ML model, the second ML model, and the third ML model"; (Andoni, FIG. 1A and Para.33, teaches the calculator/detector may initiate adjustment to the first, second, and third neural based on the aggregate loss wherein ink weights, bias functions, bias values, etc. may be modified via backpropagation to minimize the aggregate loss L using stochastic gradient descent, i.e., minimize aggregated loss by updating trainable parameters associated with the three models).
The proposed combination as well as the motivation for combining the Basavaraju, Dubey, Fajtl, Pan, Zaffar, and Andoni references presented in the rejection of Claims 1, 8, 10, and 14 applies to claim 20. Thus, the system recited in claim 20 is met by Basavaraju in view of Dubey, Fajtl, Pan, Zaffar, and Andoni.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW STEVEN BUDISALICH whose telephone number is (703)756-5568. The examiner can normally be reached Monday - Friday 8:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW S BUDISALICH/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights