DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 13-17, 19-22, 24, 25, 1-5, 7-10, and 12 (1-5, 7-10, 12-17, 19-22, 24, and 25) are rejected under 35 U.S.C. 103 as being unpatentable over Zheng et al., “EXIF as Language: Learning Cross-Modal Associations Between Images and Camera Metadata” (Zheng), and further in view of Ghosh et al., US 2020/0402223 A1 (Ghosh).
Regarding claim 13, Zheng teaches a machine learning (trained model) (p. 6947; Section 3.1., 1st paragraph) method for image splice detection and localization (localizing and detecting spliced image regions) (p. 6945; Abstract and p. 6946; left column, 3rd paragraph), comprising:
processing an image (processing an input image) (p. 6948; Figure 3) using a patch partitioning algorithm (patch encoder) (p. 6948; Figure 3) to generate a plurality of image patches (generating a plurality of patches from the input image; which is obviously done by a type of algorithm) (p. 6947-6948; Figure 3 and Section 3.3.);
processing the plurality of image patches into a plurality of feature embeddings in a high-dimensional feature space (processing the plurality of image patches into patch embeddings projected into multi-modal embedding space) (p. 6948; Figure 3 and Section 4.1.); and
processing the plurality of feature embeddings using a learning model (learned features through the model) (p. 6946; left column, 2nd paragraph) to generate an output indicative of whether the image has been spliced or manipulated (processing the plurality of patch embeddings to generate an output showing if there is a detected splice) (p. 6947-6948; Figure 3 and Section 3.3.).
Zheng teaches a model for localizing spliced image regions by clustering the visual embeddings for all of the patches within an image (p. 6945; Abstract). However, Zheng does not explicitly teach using a “deep machine learning” model.
Ghosh teaches a system for improved localization of image forgery (Abstract); wherein the system can determine a splicing manipulation localization by a trained neural network (Abstract); wherein the input is high-dimensional, such as an image ([0025]); and in particular the system trains a deep neural network (machine learning system) ([0034]) used for localizing splicing manipulations ([0033]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zheng to include deep machine learning model practices since it can improve the localization of image forgery while improving an ability of computer systems to more efficiently process data (Ghosh; [0004]).
Regarding claim 14, Zheng teaches wherein the output comprises a graphical indication of a component of the image that has been spliced or manipulated (wherein the output comprises a similarity map showing an indication of a detected splice) (p. 6947-6948; Figure 3 and Section 3.3.).
Regarding claim 15, Zheng teaches wherein the patch partitioning algorithm (patch encoder) (p. 6948; Figure 3) processes all patches from the image (for each image patch; and by taking dot product for every pair of patches) (p. 6947-6948; Figure 3 and Section 3.3.).
Regarding claim 16, Zheng teaches wherein the patches comprise non-overlapping patches of k x k dimensions (wherein the patches are shown non-overlapping and as squares; i.e. k x k dimensions) (p. 6946; Figure 2 and p. 6948; Figure 3).
Regarding claim 17, Zheng teaches wherein the patch partitioning algorithm processes selected patches from the image from which one or more camera features can be derived (wherein from the image patches camera properties can be predicted) (p. 6947-6948; Table 1, and Sections 3.2. and 3.3.).
Regarding claim 19, Zheng teaches wherein the plurality of feature embeddings indicate camera patterns present in the plurality of patches (wherein the patch embeddings indicates patterns from the camera properties present in the patches) (p. 6947-6948; Figure 3 and Sections 3.1. and 3.3.).
Regarding claim 20, Zheng teaches further comprising executing a permutation equivariate shared processing backbone module (using backbone architectures; i.e. ResNet-50) (p. 6949; left column, 1st paragraph).
Regarding claim 21, Zheng teaches further comprising executing a set-level classifier (detection classification) (detecting inconsistent embeddings within an image) (p. 6945; Figure 1 and p.6946; left column, 3rd paragraph) module on output of the shared processing backbone module (using backbone architectures; i.e. ResNet-50) (p. 6949; left column, 1st paragraph).
Regarding claim 22, Zheng teaches further comprising executing a point-level classifier (localization classification) (zero shot splice localization) (p.6948; Figure 3 and left column, 1st paragraph) on the output of the shared processing backbone module (using backbone architectures; i.e. ResNet-50) (p. 6949; left column, 1st paragraph) in parallel (wherein the detection and localization are done in parallel) (p. 6945-6946; Figure 1 and Figure 2) with the set-level classifier module (detection classification) (detecting inconsistent embeddings within an image) (p. 6945; Figure 1 and p.6946; left column, 3rd paragraph).
Regarding claim 24, Ghosh teaches further comprising executing a regularization technique to learn robust representations (using regularization for improved learning and training for representations) ([0022-0024]).
Regarding claim 25, Zheng teaches further comprising executing a transformer model for capturing long-range dependencies and contextual information (using image patches and EXIF metadata; wherein the metadata is converted into text and the processes the result with a transformer) (p. 6946; Figure 1 and Abstract).
Regarding claim 1, see the rejection made to claim 13, as well as prior art Ghosh for a system (computer system 302) (Fig. 11; [0057]) comprising a memory (storage device 304) (Fig. 11; [0057]) storing an image (wherein an image can be stored) ([0020] and [0057-0058]); and a processor (central processing unit 312) (Fig. 11; [0057]) in communication (communication bus 310) (Fig. 11; [0057]) with the memory (the processing unit in communication with the storage device) (Fig. 11; [0057-0058]), for they teach all the limitations within this claim.
Regarding claim 2, see the rejection made to claim 14, as well as prior art Ghosh for a system (computer system 302) (Fig. 11; [0057]) comprising a memory (storage device 304) (Fig. 11; [0057]) storing an image (wherein an image can be stored) ([0020] and [0057-0058]); and a processor (central processing unit 312) (Fig. 11; [0057]) in communication (communication bus 310) (Fig. 11; [0057]) with the memory (the processing unit in communication with the storage device) (Fig. 11; [0057-0058]), for they teach all the limitations within this claim.
Regarding claim 3, see the rejection made to claim 15, as well as prior art Ghosh for a system (computer system 302) (Fig. 11; [0057]) comprising a memory (storage device 304) (Fig. 11; [0057]) storing an image (wherein an image can be stored) ([0020] and [0057-0058]); and a processor (central processing unit 312) (Fig. 11; [0057]) in communication (communication bus 310) (Fig. 11; [0057]) with the memory (the processing unit in communication with the storage device) (Fig. 11; [0057-0058]), for they teach all the limitations within this claim.
Regarding claim 4, see the rejection made to claim 16, as well as prior art Ghosh for a system (computer system 302) (Fig. 11; [0057]) comprising a memory (storage device 304) (Fig. 11; [0057]) storing an image (wherein an image can be stored) ([0020] and [0057-0058]); and a processor (central processing unit 312) (Fig. 11; [0057]) in communication (communication bus 310) (Fig. 11; [0057]) with the memory (the processing unit in communication with the storage device) (Fig. 11; [0057-0058]), for they teach all the limitations within this claim.
Regarding claim 5, see the rejection made to claim 17, as well as prior art Ghosh for a system (computer system 302) (Fig. 11; [0057]) comprising a memory (storage device 304) (Fig. 11; [0057]) storing an image (wherein an image can be stored) ([0020] and [0057-0058]); and a processor (central processing unit 312) (Fig. 11; [0057]) in communication (communication bus 310) (Fig. 11; [0057]) with the memory (the processing unit in communication with the storage device) (Fig. 11; [0057-0058]), for they teach all the limitations within this claim.
Regarding claim 7, see the rejection made to claim 19, as well as prior art Ghosh for a system (computer system 302) (Fig. 11; [0057]) comprising a memory (storage device 304) (Fig. 11; [0057]) storing an image (wherein an image can be stored) ([0020] and [0057-0058]); and a processor (central processing unit 312) (Fig. 11; [0057]) in communication (communication bus 310) (Fig. 11; [0057]) with the memory (the processing unit in communication with the storage device) (Fig. 11; [0057-0058]), for they teach all the limitations within this claim.
Regarding claim 8, see the rejection made to claim 20, as well as prior art Ghosh for a system (computer system 302) (Fig. 11; [0057]) comprising a memory (storage device 304) (Fig. 11; [0057]) storing an image (wherein an image can be stored) ([0020] and [0057-0058]); and a processor (central processing unit 312) (Fig. 11; [0057]) in communication (communication bus 310) (Fig. 11; [0057]) with the memory (the processing unit in communication with the storage device) (Fig. 11; [0057-0058]), for they teach all the limitations within this claim.
Regarding claim 9, see the rejection made to claim 21, as well as prior art Ghosh for a system (computer system 302) (Fig. 11; [0057]) comprising a memory (storage device 304) (Fig. 11; [0057]) storing an image (wherein an image can be stored) ([0020] and [0057-0058]); and a processor (central processing unit 312) (Fig. 11; [0057]) in communication (communication bus 310) (Fig. 11; [0057]) with the memory (the processing unit in communication with the storage device) (Fig. 11; [0057-0058]), for they teach all the limitations within this claim.
Regarding claim 10, see the rejection made to claim 22, as well as prior art Ghosh for a system (computer system 302) (Fig. 11; [0057]) comprising a memory (storage device 304) (Fig. 11; [0057]) storing an image (wherein an image can be stored) ([0020] and [0057-0058]); and a processor (central processing unit 312) (Fig. 11; [0057]) in communication (communication bus 310) (Fig. 11; [0057]) with the memory (the processing unit in communication with the storage device) (Fig. 11; [0057-0058]), for they teach all the limitations within this claim.
Regarding claim 12, see the rejection made to claim 24, as well as prior art Ghosh for a system (computer system 302) (Fig. 11; [0057]) comprising a memory (storage device 304) (Fig. 11; [0057]) storing an image (wherein an image can be stored) ([0020] and [0057-0058]); and a processor (central processing unit 312) (Fig. 11; [0057]) in communication (communication bus 310) (Fig. 11; [0057]) with the memory (the processing unit in communication with the storage device) (Fig. 11; [0057-0058]), for they teach all the limitations within this claim.
Claim(s) 18 and 6 (6 and 18) are rejected under 35 U.S.C. 103 as being unpatentable over Zheng et al., “EXIF as Language: Learning Cross-Modal Associations Between Images and Camera Metadata” (Zheng), Ghosh et al., US 2020/0402223 A1 (Ghosh), and further in view of US 11,875,597 B2 (Gupta).
Regarding claim 18, Zheng teaches wherein the patch partitioning algorithm evaluates an exposure of each patch (wherein each patch has camera properties including exposure time and program) (p. 6947; Table 1). Ghosh teaches using high-pass filters ([0021]). However, neither explicitly teaches wherein the method “filters out underexposed or overexposed patches”.
Gupta teaches identifying an object within an image (Abstract); and wherein by applying normalization, unintended effects like over-exposition can be filtered out throughout the processing of the image (col. 7, lines 37-39).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of prior arts to include filtering out overexposed (over-exposition) in images since it allows for an increased efficiency of identifying the object in the image (Gupta; col. 7, lines 39-41).
Regarding claim 6, see the rejection made to claim 18, as well as prior art Ghosh for a system (computer system 302) (Fig. 11; [0057]) comprising a memory (storage device 304) (Fig. 11; [0057]) storing an image (wherein an image can be stored) ([0020] and [0057-0058]); and a processor (central processing unit 312) (Fig. 11; [0057]) in communication (communication bus 310) (Fig. 11; [0057]) with the memory (the processing unit in communication with the storage device) (Fig. 11; [0057-0058]), for they teach all the limitations within this claim.
Claim(s) 23 and 11 (11 and 23) are rejected under 35 U.S.C. 103 as being unpatentable over Zheng et al., “EXIF as Language: Learning Cross-Modal Associations Between Images and Camera Metadata” (Zheng), Ghosh et al., US 2020/0402223 A1 (Ghosh), and further in view of Wang et al., “Image Splicing Tamper Detection Based on Deep Learning and Attention Mechanism” (Wang).
Regarding claim 23, Zheng teaches localizing spliced image regions (p. 6945; Abstract). Ghosh teaches determining a splicing manipulation localization by a trained neural network (Abstract); wherein it is of importance to focus on relevant information so the predictive model isn’t hindered ([0025]); and wherein constriction of mutual information allows the neural network to ignore irrelevant semantic content contained in any image patch and focus its capacity on learning useful features required to classify a source camera model ([0035]).
However, neither explicitly state “further comprising executing an attention mechanism for selectively focusing on relevant features or parts of the image”.
Wang teaches an algorithm for image tampering detection (p. 271; Section D.); including to locate and detect spliced and tampered images (p. 271; Section D.); and wherein further comprising executing an attention mechanism (attention mechanism) (p. 268; left column, 1st paragraph) for selectively focusing on relevant features or parts of the image (using the attention mechanism to pay more attention to the prominent areas in the image when extracting the tampered image features) (p. 268; left column, 1st paragraph).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of prior arts to include an attention mechanism since it allows for the system to achieve better detection results (Wang; p. 268, left column, 1st paragraph); and such that the model can capture tamper traces and extract more critical tamper features, thus improving the accuracy of real images and tamper image classification (Wang; p. 271, Section D.)
Regarding claim 11, see the rejection made to claim 23, as well as prior art Ghosh for a system (computer system 302) (Fig. 11; [0057]) comprising a memory (storage device 304) (Fig. 11; [0057]) storing an image (wherein an image can be stored) ([0020] and [0057-0058]); and a processor (central processing unit 312) (Fig. 11; [0057]) in communication (communication bus 310) (Fig. 11; [0057]) with the memory (the processing unit in communication with the storage device) (Fig. 11; [0057-0058]), for they teach all the limitations within this claim.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL J VANCHY JR whose telephone number is (571)270-1193. The examiner can normally be reached Monday - Friday 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL J VANCHY JR/Primary Examiner, Art Unit 2666 Michael.Vanchy@uspto.gov
Prosecution Insights