DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 3-10, and 12-19 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by He et al. (US 20240096072 A1).
Regarding claims 1, 10, and 19, He discloses an apparatus (figs. 1 and 10, a computer vision, [0056]) and a method (fig. 9) for processing images comprising:
a storage device (1004 and 1006 of fig. 10, [0060]-[0063]); and
one or more processors (1002 of fig. 10, [0060]-[0063]) coupled to the storage device and configured to execute instructions on the storage device such that when executed, cause the apparatus (fig. 1) to:
receive an input latent image comprising latent image patches containing latent image data (104, 106a-106n, and 108a-108n of fig.1, [0026] a first subset of patches 106a, 106b, . . . , 106n (individually and/or collectively herein referred to as 106) may be selected to be visible (also interchangeably herein referred to as visible patches 106 or unmasked patches 106) and a second subset of patches 108a, 108b, . . . , 108n (individually and/or collectively herein referred to as 108) may be selected to be masked (also interchangeably herein referred to as masked patches 108) during the pre-training; 910 and 920 of fig. 9, [0051] the computing system (e.g., computing system 1000) may divide the image into a set of patches. For instance, the image (e.g., image 102) may be split or divided into a grid of non-overlapping patches, such as, for example, grid or patches 104 as shown in FIG. 1. The patches 106 and 108 in the input image 104 are treated as the latent image patches containing latent image data in the input latent image);
select a subset of the latent image patches (106a-106n of fig. 1, 930 of fig. 9, [0026], [0026], and [0051] the computing system (e.g., computing system 1000) may select a first subset of these patches to be visible (e.g., visible patches 106) and a second subset of these patches to be masked (e.g., masked patches 108) during the pre-training process);
apply the latent image patches to an input of a first encoder in the apparatus (110 of fig. 1, 930 of fig. 9, [0031], [0032], and [0035] the operations performed by the transformer encoder 110);
receive conditioning side information ([0031] the useful information; [0032], [0037], and [0053] positional information and positional embeddings)
encode, by the first encoder, the subset of latent image patches based on the conditioning side information to generate encoded latent image patches ([0023] and [0036] full set of encoded patches output by encoder);
combine the encoded latent image patches with a plurality of mask tokens ([0036] and [0037] For instance, after encoding, the list of mask tokens 114 (e.g., vectors or latent representations corresponding to masked patches 108) may be appended to the list of encoded patches 112 to form a full list or set 115, which may be unshuffled (e.g., inverting the random shuffle operation that was earlier performed during masking) to align all tokens with their targets; responsive to forming the full set of tokens consisting of (1) encoded visible patches (e.g., latent representations 112 corresponding to visible patches 106) and (2) mask tokens corresponding to the masked patches 108, the full set of tokens may be provided as input to the decoder 116, [0052] and [0053]);
apply the combined encoded latent image patches and the plurality of mask tokens to an input of a decoder in the apparatus (114 and 116 of fig.1, [0036] For instance, after encoding, the list of mask tokens 114 (e.g., vectors or latent representations corresponding to masked patches 108) may be appended to the list of encoded patches 112 to form a full list or set 115, which may be unshuffled (e.g., inverting the random shuffle operation that was earlier performed during masking) to align all tokens with their targets; [0037] the full set of tokens may be provided as input to the decoder 116; );
decode, by the decoder, the combined encoded latent image patches and the plurality of mask tokens based on the conditioning side information to generate a reconstructed latent feature map (116 of fig.1, [0039] an example architecture of a transformer decoder 116, in accordance with particular embodiments. The transformer decoder 116 begins with receiving output embedding 422 representing the full set of tokens, which may include (1) encoded visible patches (e.g., latent representations 112 corresponding to visible patches 106) and (2) mask tokens corresponding to the masked patches 108; 950 of fig. 9, [0054]); and
rearrange the reconstructed latent feature map to produce an output latent image ([0007] the decoder of the first ML model may process the full set or list of tokens including (1) the first latent representations corresponding to the first subset of patches (e.g., visible patches) and (2) mask tokens corresponding to the second subset of patches (e.g., masked patches) to generate reconstructed patches corresponding to the masked patches. The reconstructed patches generated by the decoder may include predicted pixel values for each masked patch. In particular embodiments, the reconstructed patches and the first subset of patches (e.g., visible patches) may be used to generate a reconstructed image; [0054] In particular embodiments, the reconstructed patches and the first subset of patches (e.g., visible patches 106) may be used to generate a reconstructed image, such as image 120 as shown in FIG. 1).
Regarding claims 3 and 12, He teaches the method of claim 1 and the apparatus of claim 10, wherein the input latent image comprises N latent image patches arranged in a two-dimensional (2D) array (104, 106, and 108 of fig. 1, [0025]).
Regarding claims 4 and 13, He teaches the method of claim 1 and the apparatus of claim 12, He further teaches wherein the 2D array is an MxM array (106 and 106 of fig. 1, [0026]).
Regarding claims 5 and 14, He teaches the method of claim 1 and the apparatus of claim 13, He further teaches wherein the apparatus selects the subset of the latent image patches by masking out, by the first encoder, the plurality of the latent image patches in the MxM array ([0023], [0028], and [0035].
Regarding claims 6 and 15, He teaches the method of claim 1 and the apparatus of claim 10, He further teaches wherein the apparatus applies the latent image patches to the input of the first encoder by applying unmasked latent image patches to the input of the first encoder ([0026] unmasked patches).
Regarding claims 7 and 16, He teaches the method of claim 1 and the apparatus of claim 10, He further teaches wherein the conditioning side information comprises semantic information ([0031] and [0037] positional embeddings.
Regarding claims 8 and 17, He teaches the method of claim 1 and the apparatus of claim 10, He further teaches wherein the conditioning side information comprises at least one of text data, image data, or a semantic map ([0031] useful information, [0032] the learned information, to map all input sequences into an abstract continuous representation that holds the learned information for that entire sequence).
Regarding claims 9 and 18, He teaches the method of claim 1 and the apparatus of claim 10, He further teaches wherein the conditioning side information comprises at least one of representation data, confidence ratio data, or an anchor mask, or other information that represents a semantic information of the input latent image ([0031] vector representation or features, [0032] to map all input sequences into an abstract continuous representation that holds the learned information for that entire sequence).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2, 11, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over He et al. (US 20240096072 A1) in view of Galpin et al. (US 20260075202 A1).
Regarding claims 2, 11, and 20, He teaches the method of claim 1 and the apparatus of claim 10, wherein the input latent image comprises a latent image tensor received from a second encoder.
Galpin teaches wherein the input latent image comprises a latent image tensor received from a second encoder ([0033] and [0038] an input such as a tensor representing a group of images, a tensor representing a part (crop) of a group of images, is compressed by a deep encoder to produce main latents of the image that comprise the image tensor; [0063] a method for entropy encoding/decoding a latent tensor that improves the compression performance of existing auto-encoders without requiring retraining their parameters).
Taking the teachings of He and Galpin as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the second encoder of Galpin into the apparatus of He for encoding or decoding a latent tensor that improves the compression performance of existing auto-encoders without requiring retraining their parameters.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Chen et al. (US 20230351558 A1) discl0ses a computerized method for generating an inpainted image from a masked image is described. An image including a masked region and an unmasked region is received, and the received image is divided into a plurality of patches including masked patches, wherein the masked patch includes at least a portion of the masked region of the image. The plurality of patches is encoded into a plurality of feature vectors, wherein each patch is encoded to a feature vector.
Ding et al. (US 20230336738 A1) discloses a first neural network in the multi-rate compression domain computer vision task decoder converts the value of the parameter to a tensor. The tensor is input to one or more layers in a second neural network in the multi-rate compression domain computer vision task decoder. The second neural network generates the computer vision task result according to the compressed image and the tensor.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TUNG T VO whose telephone number is (571)272-7340. The examiner can normally be reached Monday-Friday 6:30 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian Pendleton can be reached at 571-272-7527. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
TUNG T. VO
Primary Examiner
Art Unit 2425
/TUNG T VO/Primary Examiner, Art Unit 2425
Prosecution Insights