DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea (mental process of clustering images) without significantly more.
Claim 1 recite(s):
“…comparing…each image in a dataset of highly-similar images to every other image in the dataset of highly-similar images to generate a comparison score for each image-image pair;”; Which can be reasonably interpreted as a human observer viewing and mentally comparing each pair of images to mentally generate a score.
“…clustering the images in the dataset of highly-similar images into a plurality of image clusters based upon the comparison scores”; Which can be reasonably interpreted as a human observer mentally, or with a pencil and paper, clustering the images based on the generated scores.
“…selecting one of the plurality of image clusters as representing an original image”; Which can be reasonably interpreted as a human observer mentally, or with a pencil and paper, selecting a cluster as representing an original image.
“…performing a data processing operation on the dataset of highly-similar images based upon the selecting”; Which can be reasonably interpreted as a human observer mentally, or with a pencil and paper, performing a data processing operation on the dataset based on the selection, such as the deletion/removal of all images not assigned to the selected cluster.
This judicial exception is not integrated into a practical application because of additional elements:
“…a computer hardware system including a comparison engine”; are generically recited computer elements that do not add a meaningful limitation to the abstract idea because they amount to simply implementing the abstract idea on a computer and pertain to a generically recited computer hardware system and a generically recited comparison engine.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because of additional elements:
“…a computer hardware system including a comparison engine”; are well-understood, routine, and conventional computer elements that do not add a meaningful limitation to the abstract idea because they amount to simply implementing the abstract idea on a computer and pertain to a well-understood, routine, and conventional computer hardware system and a well-understood, routine, and conventional comparison engine.
Depending claims 2-8 do not remedy these deficiencies.
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an additional element of a machine learning engine for performing the cluster selection of claim 1. The machine learning engine is recited generically such that it does not provide any meaningful limitations on performing the abstract idea. This claim is not patent eligible.
Claims 3-5 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an additional element of a graphical user interface for performing the cluster selection of claim 1 and displaying the image clusters. The graphical user interface is recited generically such that it does not provide any meaningful limitations on performing the abstract idea. These claims are not patent eligible.
Claim 6-8 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of deleting and tagging, with a link to an associated original image, the dataset images not corresponding to a selected cluster. This is a mental process. The claim is not patent eligible.
As per claim(s) 9-16, arguments made in rejecting claim(s) 1-8 are analogous, respectively. Note that these claims recite additional elements: “a hardware processor including a comparison engine,” which are generically recited and well-understood, routine, and conventional.
As per claim(s) 17-20, arguments made in rejecting claim(s) 1, 3, and 7-8 are analogous, respectively. Note that these claims recite additional elements: “computer program product,” and “a computer readable storage medium having stored therein program code,” which are generically recited and well-understood, routine, and conventional.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-3, 7, 9-11, 15, and 17-19 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Singh et al. (US-20240185079-A1) hereinafter referenced as Singh.
Regarding claim 1, Singh discloses: A computer-implemented method within a computer hardware system including a comparison engine, comprising: comparing, using the comparison engine, each image in a dataset of highly-similar images (Singh: 0052-0053: “FIG. 4 shows flowchart 400 depicting a second method according to an embodiment of the present invention. This method will now be discussed, over the course of the following paragraphs and may be implemented on computing environment 100 ( FIG. 1 ) by a program such as resource evaluation program 300. Processing begins at step 402, where image sets are prepared for evaluation by a trained model for clustering images by similarity to existing images.”; Wherein the clustering of images based on similarity discloses a dataset of highly-similar image.) to every other image in the dataset of highly-similar images to generate a comparison score for each image-image pair (Singh: 0053-0054: “Processing begins at step 402, where image sets are prepared for evaluation by a trained model for clustering images by similarity to existing images… Processing proceeds to step 404 , where the trained model evaluates the image sets in view of a clusters pipeline establishing clusters of resources of various origination including authorized origins. As discussed in more detail herein, the evaluation process may include image embedding of the image sets and evaluation of the embeddings with respect to clusters having pre-defined similarity thresholds for associating a new image with a particular cluster. The thresholds may be based on a Euclidian distance from a cluster centroid, or medoid for siamese network models.”; Wherein the evaluation of each target image’s embedding with respect to each clusters’ similarity threshold constitutes the generation of a comparison score for each image-image pair. );
clustering the images in the dataset of highly-similar images into a plurality of image clusters based upon the comparison scores (Singh: 0054: “As discussed in more detail herein, the evaluation process may include image embedding of the image sets and evaluation of the embeddings with respect to clusters having pre-defined similarity thresholds for associating a new image with a particular cluster. The thresholds may be based on a Euclidian distance from a cluster centroid, or medoid for siamese network models.”);
selecting one of the plurality of image clusters as representing an original image (Singh: 0053-0055: “The image pairs may be grouped as authentic/authentic, unauthorized/unauthorized, and/or authentic/unauthorized. For the triplets, the information may be in the form of “anchor, positive, and negative images…Processing proceeds to decision step 406 , where it is determined whether a target image represents a resource of authorized origin. For resources of authentic origin, processing follows the “YES” branch to step 408 , where an expert review approves the determination and provides feedback (step 410 ) to the trained model for ongoing training refinement, reinforcement learning.”; Wherein the image clusters an image is assigned to are classified as either authentic or not.); and
performing a data processing operation on the dataset of highly-similar images based upon the selecting (Singh: 0054: “Processing proceeds to step 404 , where the trained model evaluates the image sets in view of a clusters pipeline establishing clusters of resources of various origination including authorized origins. As discussed in more detail herein, the evaluation process may include image embedding of the image sets and evaluation of the embeddings with respect to clusters having pre-defined similarity thresholds for associating a new image with a particular cluster.”;
0057: “Processing ends at step 416 , where an expert evaluator reviews any new clusters created at steps 412 or 414 prior to recording the clusters to the clusters pipeline, or knowledge base.”; Wherein processed images are assigned to their determined clusters, which further increases the cluster pipeline used by the trained model.).
Regarding claim 2, Singh discloses: The method of claim 1, wherein the selecting is performed by a machine learning engine (Singh: 0054-0055: “Processing proceeds to step 404, where the trained model evaluates the image sets in view of a clusters pipeline establishing clusters of resources of various origination including authorized origins…Processing proceeds to decision step 406 , where it is determined whether a target image represents a resource of authorized origin. For resources of authentic origin, processing follows the “YES” branch to step 408 , where an expert review approves the determination and provides feedback (step 410 ) to the trained model for ongoing training refinement, reinforcement learning.”; Wherein the trained machine learning model determines whether images are assigned to clusters classified as authentic.).
Regarding claim 3, Singh discloses: The method of claim 1, wherein the selecting includes providing a graphical user interface configured to: visually display a plurality of images respectively representing each of the plurality of image clusters; and receive a selection indicating the one of the plurality of image clusters as representing the original image (Singh: 0055-0057: “For resources of authentic origin, processing follows the “YES” branch to step 408 , where an expert review approves the determination and provides feedback (step 410 ) to the trained model for ongoing training refinement, reinforcement learning...Processing ends at step 416 , where an expert evaluator reviews any new clusters created at steps 412 or 414 prior to recording the clusters to the clusters pipeline, or knowledge base.”;
0124: “The receiving human feedback on the identification done by the computer program, comprises: receiving input from user to confirm the output of the computer program; sending verified inputs to the system for adjusting, weights and parameters, of the model; and improving output accuracy of the system with continuous feedback.”; Wherein the computer inputted human feedback/expert reviews done based on output clustering determinations implies the use of a GUI.).
Regarding claim 7, Singh discloses: The method of claim 1, wherein the data processing operation includes tagging images in the dataset of highly-similar images not corresponding to the selected one of the plurality of image clusters as representing the original image (Singh: 0054: “As discussed in more detail herein, the evaluation process may include image embedding of the image sets and evaluation of the embeddings with respect to clusters having pre-defined similarity thresholds for associating a new image with a particular cluster.”; 0057: “where an expert evaluator reviews any new clusters created at steps 412 or 414 prior to recording the clusters to the clusters pipeline, or knowledge base”; Wherein the recording of image cluster assignments constitutes tagging images.).
As per claim(s) 9, arguments made in rejecting claim(s) 1 are analogous. In addition, Figure 1 and 0017 of Singh disclose the limitation “A computer hardware system, comprising: a hardware processor including a comparison engine.”
As per claim(s) 10, arguments made in rejecting claim(s) 2 are analogous.
As per claim(s) 11, arguments made in rejecting claim(s) 3 are analogous.
As per claim(s) 15, arguments made in rejecting claim(s) 7 are analogous.
As per claim(s) 17, arguments made in rejecting claim(s) 1 are analogous. In addition, 0016 of Singh discloses the limitation “A computer program product, comprising: a computer readable storage medium having stored therein program code, the program code, which when executed by the computer hardware system including a comparison engine, cause the computer hardware system to perform.”
As per claim(s) 18, arguments made in rejecting claim(s) 3 are analogous.
As per claim(s) 19, arguments made in rejecting claim(s) 7 are analogous.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4-5 and 12-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Singh, and further in view of Jing et al. (Google Image Swirl: A Large-Scale Content-Based Image Browsing System) hereinafter referenced as Jing.
Regarding claim 4, Singh discloses: The method of claim 3.
Singh does not disclose expressly: wherein the graphical user interface is further configured to present the plurality of images as a radial cluster.
Jing discloses: a system for processing a plurality of images by extracting features from each image, wherein the images are hierarchically clustered based on their features (Jing: 1. EXEMPLAR TREE: “We begin organizing up to 1000 results of an image search query by building a pairwise similarity matrix among these images. The similarity computation is based on a combination of image features such as color, texture, local features, face signatures, and the metadata associated with the images. We then perform clustering to partition the search results into hierarchical clusters, each associated with a representative, or exemplar, image.”). Wherein the system comprises a graphical user interface configured to present the plurality of images as a radial cluster (Jing: Figure 2; 2. USER INTERFACE: “After hierarchical clustering has been performed, the results of an image search query are organized in the structure of a tree. A number of options exist for how to present such a tree to the user…In this demonstration, we used radial layout in which each layer of the tree is arranged radially around its parent, see Figure 1. When the user selects a branch of the tree to explore, it is separated from the parent and expanded, while the parent is shrunk to make space, see Figures 2.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the clustering method implemented by the trained model disclosed by Singh with a hierarchical clustering method, which displays clustering results using a dynamic web-based user interface, taught by Jing. The suggestion/motivation for doing so would have been “When the user selects a branch of the tree to explore, it is separated from the parent and expanded, while the parent is shrunk to make space, see Figures 2. The rearrangement is animated to allow the user to follow the change without getting lost” (Jing: 2. USER INTERFACE). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Singh with Jing to obtain the invention as specified in claim 4.
Regarding claim 5, Singh discloses: The method of claim 3.
Singh does not disclose expressly: wherein the graphical user interface is further configured to display one or more differences between respective images associated with a pair of selected clusters.
Jing discloses: a system for processing a plurality of images by extracting features from each image, wherein the images are hierarchically clustered based on their features (Jing: 1. EXEMPLAR TREE: “We begin organizing up to 1000 results of an image search query by building a pairwise similarity matrix among these images. The similarity computation is based on a combination of image features such as color, texture, local features, face signatures, and the metadata associated with the images. We then perform clustering to partition the search results into hierarchical clusters, each associated with a representative, or exemplar, image.”). Wherein the system comprises a graphical user interface configured to display one or more differences between respective images associated with a pair of selected clusters (Jing: Abstract: “A dynamic web-based user interface allows the user to navigate this hierarchy, allowing fast and interactive browsing. The exemplars of each cluster provide a comprehensive visual overview of the query results, and allow the user to quickly navigate to the images of interest.”;
1. EXEMPLAR TREE: “We then perform clustering to partition the search results into hierarchical clusters, each associated with a representative, or exemplar, image.”; Wherein the exemplars of each cluster allows for a user to navigate the tree by comparing the displayed exemplar images and their feature differences.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the clustering method implemented by the trained model disclosed by Singh with a hierarchical clustering method, which displays clustering results using a dynamic web-based user interface, taught by Jing. The suggestion/motivation for doing so would have been “When the user selects a branch of the tree to explore, it is separated from the parent and expanded, while the parent is shrunk to make space, see Figures 2. The rearrangement is animated to allow the user to follow the change without getting lost” (Jing: 2. USER INTERFACE). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Singh with Jing to obtain the invention as specified in claim 5.
As per claim(s) 12, arguments made in rejecting claim(s) 4 are analogous.
As per claim(s) 13, arguments made in rejecting claim(s) 5 are analogous.
Claim(s) 6 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Singh, and further in view of Woo (KR-102354826-B1).
Regarding claim 6, Singh discloses: The method of claim 1.
Singh does not disclose expressly: wherein the data processing operation includes deleting images in the dataset of highly-similar images not corresponding to the selected one of the plurality of image clusters as representing the original image.
Woo discloses: a method for searching and displaying registered dental clinical images (Woo: 0072: “Referring to FIGS. 1 and 8, the processing procedure for searching and displaying the registered dental clinical image is described as follows.”). Wherein a database is queried based on search keywords in order to retrieved the dental clinical images (Woo: 0074: “In steps S210 and S215, the cloud storage device (150) searches for previously stored dental clinical images in response to the search keyword, and searches the clinical photo management DB (50) as illustrated in FIG. 4 using the search keyword, and transmits the result to the clinical image search device (170).”). The retrieved dental clinical images are then analyzed in order to determine whether the images have been forged or altered (Woo: 0077: “and in steps S225 and S230, the blockchain server (160) verifies whether the specific dental clinical image is forged or altered and then transmits the result.”), wherein if a retrieved dental clinical image has been determined to have been forged or altered, the image is deleted and replaced by an image corresponding to the original (Woo: 0081-0082: “the image correction device (120) responds by deleting the falsified/modified dental clinical image and regenerating the dental clinical image using the corresponding dental original photograph. [And, in step S260, the image correction device (120) re-registers the regenerated image in the cloud storage device (150),”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique of deleting and replacing stored falsified/modified images taught by Woo by deleting the stored non-training data corresponding to non-authentic images disclosed by Singh in order to retrieve images assigned as authentic. The suggestion/motivation for doing so would have been “By doing so, the present invention…can improve the reliability of dental clinical images by deleting and regenerating them using a normal method if a registered dental clinical image is falsified/modified” (Woo: 0084). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Singh with Woo to obtain the invention as specified in claim 6.
As per claim(s) 14, arguments made in rejecting claim(s) 6 are analogous.
Claim(s) 8, 16, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Singh, and further in view of Bharati et al. (Transformation-Aware Embeddings for Image Provenance) hereinafter referenced as Bharati.
Regarding claim 8, Singh discloses: The method of claim 7.
Singh does not disclose expressly: wherein the tagging includes adding a link to the original image within metadata associated with each of the images in the dataset of highly-similar images not corresponding to the selected one of the plurality of image clusters as representing the original image.
Bharati discloses: adding a link to the original image within metadata associated with each of the images in the dataset not classified as an original image (Bharati: Fig. 3: “Proposed deep learning framework to compute provenance chains. Given, as Input, a set of near-duplicate images where each one is obtained from a transformation on top of the other, the proposed approach learns an image embedding space where less transformed (positive) images lie closer to the original source (anchor) than more transformed versions (weak positive). Unrelated images (negative) must lie in the space farther to the anchor than the positive and weak positive ones. Function d(., .) expresses the distance between two images in the learned space. We use a novel Edit Sequence Loss (ESL) in the learning process. The desired Output is an ordered set that represents the process of transforming one image into the other.”; Wherein the ordered set, which is associated to all transformed images, corresponds to metadata linking the original image to the transformed images.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify the image based clustering system taught by Singh by integrating the deep learning framework taught by Bharati in order to link non-authentic images to the authentic images they are associated to. The suggestion/motivation for doing so would have been “Given multiple variants for any given image, an important question is: can we find the original? Not necessarily just for maliciously manipulated images, this question is valid for a number of circumstances where image manipulation is present. We propose a framework that helps to determine the sequence of image transformations to trace the provenance of the content.” (Bharati: Fig. 1.; Wherein the determination of the image provenance allows for greater understanding of the composition of each image and allows for a tracing of an image based on its provenance relationships). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Singh with Bharati to obtain the invention as specified in claim 8.
As per claim(s) 16, arguments made in rejecting claim(s) 8 are analogous.
As per claim(s) 20, arguments made in rejecting claim(s) 8 are analogous.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672