DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of Invention I, Species I.B (claims 4-9 and 15) in the reply filed on April 20th 2026 is acknowledged. The application has pending claims 1-21 (withdrawn claims 3, 10, 13, 16 and 18-21 are withdrawn from further consideration).
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. The title of the application is "IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM." The title is objected to because it includes the word "PROGRAM" but there are no claims directed to a computer program or non-transitory computer-readable medium currently pending in the application.
The following title is suggested: IMAGE PROCESSING APPARATUS AND IMAGE PROCESSING METHOD" (MPEP 606.01).
The abstract of the disclosure is objected to because it begins with the conjunctive word "Therefore," and uses phrasing "an image processing apparatus according to the present disclosure is --" that reads like a concluding paragraph of a specification rather than a standalone narrative summary of the technical disclosure. A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Objections
Claim 8 is objected to because of the following informalities: claim 8 recites “at each of times”, this phrase is grammatically improper and should be amended to “at each time” or other clear supported wording. Appropriate correction is required.
Claim 14 is objected to because of the following informalities: claim 14 recites that “the image feature amount is a vector and used as an initial input for a neural network”, the phrase should be corrected to recite that “the image feature amount is a vector and is used as an initial input for a neural network”. Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
The term "image understanding unit" in Claims 1-2 and 4-5.
The term "text understanding unit" in Claim 1-2, 5 and 9.
The term "feature amount mixing unit" in Claims 1-2.
The term "text generation unit" in Claims 4-5 and 8.
The term "parameter update unit" in Claims 4, 6-7.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112(a)
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1 and 11 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention.
Claims 1 and 11 requires projecting both the image feature amount and text feature amount "onto the same vector space". The specification's Formulas 1–3 [0058] describe one specific implementation, projecting the text feature amount via MLP onto the image feature space, but [0055] also states: "there are cases where one feature amount is projected to the vector space of the other feature amount, and where one feature amount is projected to a third vector space different from each other". This acknowledgment of a third vector space option in the specification is different than the claim's "same vector space" requirement, but more critically, the specification provides minimal written description support for what constitutes "the same vector space" across different neural network architectures. If applicant claims any shared vector space implementation, the written description is insufficient for the full claimed scope.
The claims broadly cover any implementation that projects image and text feature amounts onto a shared vector space. The specification enables only one specific architecture (MLP projection of text features onto image feature space, Formulas 1–3). The full scope of claim 1 encompasses vastly different architectural approaches (e.g., cross-attention mechanisms, joint embedding spaces, contrastive pre-training approaches like CLIP) that are not described or enabled in the specification. This breadth-enablement gap supports a § 112(a) enablement rejection under Amgen v. Sanofi (2023) principles, the claims are broader than the disclosed embodiments.
Claim 9 is rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 9 depends from Claim 4. Claim 9 recites that "the text understanding unit extracts the text feature amount by performing weighted pooling using a probability that indicates a likelihood of a predetermined-order token related to the generated text data.". But claim 4 is directed to a “text generation unit” and a “parameter update unit”, claim 4 does not require that the attached text data is missing. According to the Specifications’ [0102] and [Fig. 7], the “weighted pooling using a probability pj that indicates a likelihood of the predetermined-order (j-th) token tj related to the generated text data” is specifically and uniquely executed by the text understanding unit during the inference phase when attached text data is not attached (Fig. 7, Step S133-NO to S135). If attached text data is attached, the probability pj = 1 for all tokens, and the understanding unit does not use the generated text data for pooling. Because Claim 9 depends from Claim 4 (which implies the training phase where the attached text data exists to update the parameter), claiming the weighted pooling of generated text data without the condition that attached text data is missing (as recited in Claim 5) constitute claiming subject matter broader than what the inventor possessed in the Specification. Claim 9 should depend from Claim 5 as written.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2 and 5 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 2 recites the limitation "the neural network" in claim. There is insufficient antecedent basis for this limitation in the claim. The claim recites that the image understanding unit, the text understanding unit, and the feature amount mixing unit are “each configured by a neural network”, but subsequently recites that these units perform processing on the basis of model parameters of “the neural network”. Because the claim first recites a neural network for each of multiple units, the phrase “the neural network” lacks clear antecedent basis and renders unclear whether the claim requires one shared neural network or respective multi neural networks.
Where applicant acts as his or her own lexicographer to specifically define a term of a claim contrary to its ordinary meaning, the written description must clearly redefine the claim term and set forth the uncommon definition so as to put one reasonably skilled in the art on notice that the applicant intended to so redefine that claim term. Process Control Corp. v. HydReclaim Corp., 190 F.3d 1350, 1357, 52 USPQ2d 1029, 1033 (Fed. Cir. 1999). The term “each configured by a neural network” in claim 2 is used by the claim to mean "implemented by", "comprising" or "including", while the accepted meaning is “defined by a neural network”. The term is indefinite because the specification does not clearly redefine the term.
Regarding claim 5, the claim recites “when the attached text data is not attached to the image data”. This phrase is internally inconsistent because “attached text data” is already recited as text data attached to the image data. The scope of the condition in claim is unclear because it is unclear whether the claim refers to absence of text data, text data that exists but is not attached, or some other condition.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1–2, 4–9, 11, 14–15, and 17 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to a judicial exception without significantly more.
This rejection has been made in accordance with the current USPTO subject matter eligibility framework, including MPEP §§ 2103–2106.07, the 2019 Revised Patent Subject Matter Eligibility Guidance, the October 2019 Patent Eligibility Guidance Update, the 2024 Guidance Update on Patent Subject Matter Eligibility, Including on Artificial Intelligence, the July 2024 AI Subject Matter Eligibility Examples, the August 4, 2025 USPTO memorandum titled “Reminders on evaluating subject matter eligibility of claims under 35 U.S.C. § 101,” and the USPTO’s guidance concerning Ex parte Desjardins, Appeal No. 2024-000567. The claims have been evaluated under the broadest reasonable interpretation, and the claims have been considered as a whole.
Step 1: Statutory category
Independent claim 1 is directed to an image processing apparatus and therefore falls within the statutory category of a machine. Independent claim 11 is directed to an image processing method and therefore falls within the statutory category of a process. Accordingly, the analysis proceeds to Step 2A.
Step 2A, Prong One (Judicial exception)
Independent claim 1 recites: vectorizing an image pattern of image data to extract an image feature amount; vectorizing a text pattern of attached text data to extract a text feature amount; and generating a mixed feature amount by projecting the image feature amount and the text feature amount onto the same vector space and mixing the image feature amount and the text feature amount. These limitations recite an abstract idea, namely collecting multimodal information, converting information into numerical representations, performing mathematical projections in a vector space, and calculating a mathematically combined feature quantity.
The claim recites information collection, data translation, mathematical calculation, and combination of numerical vectors. The “image data”, “attached text data”, “image feature amount”, “text feature amount”, “vector space”, and “mixed feature amount” are used as mathematical constructs and information items in a data-analysis and classification process. The claim does NOT recite an improvement to the way images are captured, digitized, encoded, compressed, stored, or transmitted. Rather, the claim uses generic computer units to obtain image and text data, mathematically convert them into vectors, and project/ mix them into a shared space.
The claim is similar in character to claims that courts have found abstract where the focus is collecting information, analyzing the information mathematically, and presenting or acting on the results of the analysis. In Electric Power Group, LLC v. Alstom S.A., the Federal Circuit recognized claims directed to collecting and analyzing information as abstract. The present claims similarly collect image and text information, mathematically analyze the data to convert it into vectors, and combine the vectors to produce a classification feature. Furthermore, observing an image, reading an attached text description, and combining those semantic meanings to draw a conclusion about the image's content are mental processes historically performed by humans.
Claims 2, 14, and 17 further limit the apparatus and method to units configured by a "neural network" having an "image similarity parameter" and a "text generation probability parameter", using the vector as an initial input for a neural network, and using a "language model". These limitations recite the use of mathematical models and statistical algorithms as tools for performing the abstract image-and-text translation task. The claims do not recite any specific, unconventional hardware architecture or technical improvement to machine-learning hardware technology itself.
Claims 4, 6, and 7 recite updating the model parameters based on a loss function value between generated text and attached text, specifically using a negative log-likelihood function, or updating parameters such that features of the same class approach each other and features of different classes move away from each other (contrastive loss). These limitations recite pure mathematical concepts, specifically the optimization of mathematical equations (loss functions) to adjust numerical weights.
Claims 5 and 15 recite extracting the text feature amount using generated text data generated from the image feature amount when attached text is missing. This limitation recites predictive data analysis and the logical generation of information based on a mathematical model.
Claims 8 and 9 recite generating text data by random sampling from high-order tokens, and extracting the text feature amount by performing weighted pooling using a probability that indicates a likelihood of a predetermined-order token. These limitations recite explicit mathematical formulas and statistical analysis, specifically random sampling algorithms and probability-weighted averaging calculations.
Independent claim 11 recites substantially the same abstract idea in method form using generic computer processes. Merely implementing the same mathematical vectorization and data-mixing process using a generic method does not avoid the judicial exception.
The claims are also consistent with the reasoning of AI Visualize, Inc. v. Nuance Communications, Inc., where the Federal Circuit looked to the character of the claims as a whole and affirmed ineligibility where the claims were directed to obtaining, manipulating, and using information at a high level of generality rather than to a specific improvement in computer functionality. Here, the character of the elected claims as a whole is image and text data collection, mathematical vector extraction, spatial projection calculations, and mathematical loss-function optimization, not an improvement to camera technology, computer memory, network operation, or machine-learning computational architecture itself.
Accordingly, claims 1–2, 4–9, 11, 14–15, and 17 recite an abstract idea under Step 2A, Prong One.
Step 2A, Prong Two (Practical Application)
The additional elements, considered individually and in combination, do not integrate the abstract idea into a practical application.
The recited "neural network", "language model", "image data", "text data", "feature amount" and "parameter" amount to data objects, mathematical models, and generic computer implementation of the abstract feature extraction and calculation concept.
The claims do not recite a particular improvement to computer or imaging technology. They do not improve how a digital image is physically captured, sensed, or stored. The claims merely require obtaining image and text data that has already been captured. The claims also do not recite a particular improvement to graphical processing hardware. Rather, the claims use standard digital images and text as input data for mathematical projection and algorithmic vector mixing.
The claims further do not recite a particular improvement to artificial-intelligence or machine-learning technology. The claims do not train a model in a novel way that improves the computer's operation, update model parameters to reduce processing load, modify model architecture to save hardware resources, reduce model storage, preserve prior model knowledge, or improve inference speed by a specific claimed technical mechanism. The neural networks, cross-entropy loss functions, contrastive loss functions, and probability-weighted pooling are recited functionally as mathematical tools for identifying relationships between images and text.
This analysis is consistent with the USPTO’s 2024 AI subject matter eligibility guidance and AI examples, which emphasize that AI-related claims may be eligible when they recite a specific technological improvement or otherwise integrate a judicial exception into a practical application. The present claims do NOT recite such a specific technological improvement. Instead, the claims use generic computer operations to collect multimodal information, mathematically vectorize the data, compare vectors, and optimize statistical weights.
This case is distinguishable from Ex parte Desjardins. In Desjardins, the claims were found to reflect an improvement in machine-learning technology itself, including training a machine-learning model on a series of tasks while preserving prior knowledge and reducing complexity/storage burdens. Here, the claims do NOT recite a particular parameter-update mechanism, memory-saving arrangement, or data structure that improves the physical or computational operation of a machine-learning model. The claimed neural networks and loss functions merely automate the mathematical task of mapping images and text to a shared space. Nor does limiting the abstract idea to the environment of image classification make the claims eligible. In Recentive Analytics, Inc. v. Fox Corp., the Federal Circuit rejected the argument that applying machine learning to a new field of use was sufficient for eligibility where the claims did not recite a technical improvement to the machine-learning process itself. Similarly here, applying neural networks and mathematical vector mixing to image/text feature extraction is a field-of-use limitation, not an integration of the abstract idea into a practical application.
Accordingly, the claims do not integrate the judicial exception into a practical application under Step 2A, Prong Two.
Step 2B: (Inventive Concept)
The additional elements, considered both individually and as an ordered combination, do not amount to significantly more than the abstract idea.
The claims use generic computer components to perform ordinary computer functions, including obtaining image and text data, vectorizing data patterns, projecting vectors, mixing arrays, updating parameters, calculating statistical loss, and performing mathematical weighted pooling. These are conventional data-processing and mathematical operations performed using generic computer technology.
The ordered combination also does not provide an inventive concept. The ordered combination follows the abstract idea itself: obtain an image and text, extract a vector for each, project them into the same space, mix them, and update the mathematical parameters based on a loss calculation. This is no more than the abstract mathematical idea implemented on generic computer components.
Dependent claims 2, 4–9, 14, 15, and 17 recite additional steps of configuring units as neural networks, calculating negative log-likelihood, updating parameters based on distance between classes, generating text randomly or via sampling, performing weighted pooling, and using language models. These limitations merely specify the generic machine-learning tools, known statistical probability formulas, and mathematical gathering mechanisms used in the abstract evaluation and do not add significantly more.
The independent apparatus and method claims recite generic counterparts using basic computing logic to perform substantially the same operations. The recitation of generic neural network frameworks and vector calculations does not transform the abstract idea into patent-eligible subject matter.
Accordingly, claims 1–2, 4–9, 11, 14–15, and 17 are directed to a judicial exception without significantly more and are therefore rejected under 35 U.S.C. § 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1–2, 11, 14 and 17 are rejected under 35 U.S.C. §103 as being unpatentable over Zhang (Zhang et al, US 2020/0356821 A1, 2020).
Regarding claim 1, Zhang teaches an image processing apparatus for extracting a feature amount of image data ( [0002], [0005-0008], [Fig. 3]: Zhang teaches a method, terminal, computer readable storage medium, and device for image classification, including determining an image feature vector, determining a text feature vector, and determining an image-text feature vector. ), the image processing apparatus comprising:
an image understanding unit that vectorizes an image pattern of the image data to extract an image feature amount;
( [0017], [0017-0019], [0029-0031], [0067], [Fig. 1 - Step 101], [Fig. 3]: Zhang discloses a determining module 301, and its corresponding processing component, configured to determine an image feature vector of an image based on a convolutional neural network. Zhang further teaches that the image feature vector can be determined based on a processing result of a convolution layer or pooling layer in the convolutional neural network, and that the image feature vector contains multiple points, each point corresponding to a feature map and a weight value. )
a text understanding unit that vectorizes a text pattern of attached text data attached to the image data to extract a text feature amount; and
( [0018], [0020-0021], [0047-0053], [0068], [0071-0076], [Fig. 1 - Step 102], [Fig. 3]: Zhang discloses a vector generation module 302, comprising word segmentation submodule 3021, position determining submodule 3022, index value generation submodule 3023, first invoking submodule 3024, and second invoking submodule 3025, configured to determine a text feature vector based on the textual information and an embedded network. Specifically, Zhang discloses performing word segmentation on the textual information attached to the image, determining a description vector corresponding to each segmented word based on an index value and the embedded network, and determining a text feature vector by weighting and averaging description vectors corresponding to the multiple segmented words in the same dimensions, thereby vectorizing a text pattern of the attached text data to extract a text feature amount. )
a feature amount mixing unit that generates a mixed feature amount as the feature amount by projecting the image feature amount extracted by the image understanding unit and the text feature amount extracted by the text understanding unit onto the same vector space and mixing the image feature amount and the text feature amount.
( [0023-0024], [0054-0056], [0069], [0077]-[0079], [Fig. 1 - Step 103], [Fig. 3]: Zhang discloses a joining module 303 comprising a mapping submodule 3031 and a joining submodule 3032. The mapping submodule 3031 is configured to determine an image-text feature vector by joining the image feature vector with the text feature vector. Zhang further teaches determining a mapped text feature vector and a mapped image feature vector by mapping the text feature vector and the image feature vector in the same dimensions. Zhang explains that, because the image feature vector output by the convolutional neural network and the text feature vector output by the recurrent neural network are not in the same space and have different dimensions, spatial mapping is performed on the two feature vectors so that they are mapped to the same space and have the same dimensions. Zhang then generates an image-text feature vector by joining the mapped text feature vector with the mapped image feature vector dimensionally. )
Although different embodiments of Zhang have been referred to, it would have been exceedingly obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhang by combining Zhang’s similar embodiments in order to not limit the embodiments to themselves but include other evident combinations and extensions thereof (see Zhang’s [0018-0019]: “In some embodiments--”).
Regarding claim 2, Zhang teaches the image processing apparatus according to claim 1,
wherein the image understanding unit, the text understanding unit, and the feature amount mixing unit are each configured by a neural network, and
( [0017–0022], [0025–0026], [0054–0056], [0065], [0067–0070], [0078], [0102-0113], [0124], [Fig. 1-3]: Zhang discloses that the determining module 301 ["image understanding unit"] is configured by a convolutional neural network (CNN) for determining the image feature vector; the vector generation module 302 ["text understanding unit"] is configured by an embedded network, a trained word embedding model, for determining the text feature vector by weighting and averaging description vectors corresponding to segmented words; and the joining module 303 and classification module 304 ["feature amount mixing unit"] is configured by a deep neural network for spatially mapping the image and text feature vectors and generating classification result vectors. )
the image understanding unit, the text understanding unit, and the feature amount mixing unit perform processing on the basis of model parameters of the neural network.
( [0019], [0021], [0047–0055], [0059–0064], [0085–0088], [0102-0113], [0119-0124], [Fig. 1-3]: Zhang discloses that the convolutional neural network of the determining module 301 ["image understanding unit"] determines the image feature vector based on processing through convolution layers and pooling layers, each defined by learned weight parameters; the embedded network of the vector generation module 302 ["text understanding unit"] determines description vectors corresponding to each segmented word based on index values input to the embedded network, wherein the embedded network is trained on textual information corresponding to sample images to produce the text feature set and associated description vectors, which are the model parameters of the embedded network; and the deep neural network of the joining module 303 and classification module 304 ["feature amount mixing unit"] performs spatial mapping via a full connection layer, defined by learned weight matrix parameters, and determines classification result vectors based on the image feature vector, text feature vector, and image-text feature vector, wherein the weights
W
text
,
W
image
, and
W
text-image
used to compute the target result vector
P
=
W
text
P
text
+
W
image
P
image
+
W
text-image
P
text-image
are model parameters of the deep neural network. )
Regarding claim 11. The rationale provided for claim 1 is incorporated herein. In addition, the method for image processing apparatus of claim 1 corresponds to the method of claim 11, and performs the steps disclosed herein. Therefore, the claim is rejected.
Regarding claim 14, Zhang teaches the image processing method of claim 11, wherein the image feature amount is a vector and used as an initial input for a neural network.
( [0017-0019], [0025-0027], [0058-0060], [0065], [0102-0105], [0119-0122], [0124], [Fig. 1, Steps 104]: Zhang teaches that the image feature amount is an image feature vector determined based on a convolutional neural network [a vector comprising multiple points each corresponding to a feature map and a weight value]. Zhang further teaches that a category of the image is determined based on a result of a deep neural network, where the result is determined based on the image feature vector, the text feature vector, and the image-text feature vector. Zhang also teaches determining a first classification result vector corresponding to the image feature vector based on the deep neural network. )
Regarding claim 17, Zhang teaches the image processing method of claim 11, wherein the text data is converted into vectors using a language model.
( [0020-0021], [0047-0053], [0055], [0065], [0124]: Zhang teaches determining a text feature vector based on textual information and an embedded network. Zhang further teaches processing the textual information by performing word segmentation to obtain multiple segmented words, generating index values for the segmented words based on their positions in a text feature set, inputting the index values into the embedded network, determining description vectors corresponding to the segmented words using the embedded network, and determining a text feature vector by weighting and averaging the description vectors in the same dimensions. Zhang also teaches that the text feature vector is output by a recurrent neural network and that the embedded network is used as the master network for text feature extraction. )
Claims 4–6 and 15 are rejected under 35 U.S.C. §103 as being unpatentable over Zhang in view of Tsimpoukelli (Tsimpoukelli et al. “Multimodal Few-Shot Learning with Frozen Language Models.” ArXiv:2106.13884 [Cs], 3 July 2021.).
Regarding claim 4, Zhang teaches the image processing apparatus according to claim 2, further comprising:
Zhang teaches the base multimodal framework: image feature vector [Fig. 2 - Step 201], text feature vector [Fig. 2 - Step 202-205], mapping them into the same space [Fig. 2 - Step 206], and joining them into an image-text feature vector [Fig. 2 - Step 207-208]; but Zhang does not expressly teach where Tsimpoukelli teaches:
a text generation unit that generates generated text data by projecting the image feature amount extracted by the image understanding unit onto a vector space of the attached text data; and
( [Sec. 1, 3.1-3.3], [Figs. 2–3]: Tsimpoukelli teaches Frozen, which uses aligned image-caption data to train a vision encoder to represent each image as a sequence of continuous embeddings such that a pretrained language model prompted with this visual prefix generates an appropriate caption. Tsimpoukelli further teaches that Frozen includes a neural network trained to encode images into the word embedding space of a large pretrained language model such that the language model generates captions for those images. Tsimpoukelli teaches that the vision encoder takes a raw image and emits a continuous sequence consumed by the transformer, and that the visual prefix is formed by linearly mapping the vision encoder output into embeddings having the same dimensionality as token embeddings; thereby teaching projecting an image feature amount into a language/ text-token embedding vector space and generating text/ caption data from the projected image feature amount. )
a parameter update unit that updates a text generation probability parameter included in the model parameters on the basis of the attached text data and the generated text data generated by the text generation unit.
( [Sec. 3.2], [Figs. 2]: Tsimpoukelli teaches a pre-trained autoregressive language model that parameterizes a probability distribution over text, where text is decomposed into tokens, token embeddings are input to a transformer, and the transformer output parameterizes a categorical distribution over the vocabulary. Tsimpoukelli further teaches that a vision encoder maps an image into a visual prefix having the same dimensionality as token embeddings, so that the language model generates caption text conditioned on the image. During training, Tsimpoukelli updates the vision-encoder/ projection parameters using paired image-caption data by maximizing the likelihood of caption text given the image. Although the language-model parameters are frozen, the updated vision-encoder/ projection parameters determine the visual prefix supplied to the language model and thereby affect the conditional probability distribution over generated caption tokens. Therefore, Tsimpoukelli teaches updating model parameters that control or affect text-generation probabilities based on paired image/ caption text and the model’s generated/ predicted token distribution. )
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Zhang’s multimodal image, text feature-vector classification apparatus to include Tsimpoukelli’s image-conditioned text-generation technique, because Zhang teaches improving image classification by considering textual information associated with an image, and Tsimpoukelli teaches a known technique for mapping image features into a language-model embedding space so that a language model can generate caption [text data from the image]. A person of ordinary skill in the art would have been motivated to do so to provide additional or supplemental textual feature information for Zhang’s multimodal image/ text classification pipeline, thereby improving the availability and usefulness of text features used in Zhang’s same-space image/ text feature fusion, with the predictable result of generating image-conditioned text usable with image features.
Regarding claim 5, Zhang teaches the image processing apparatus according to claim 1, further comprising:
Zhang does recognize that sample images may or may not have textual information, and when a sample image has no textual information, Zhang’s description set is simply null ( [0036-0040], [0082-0084], [016-0118] ). That means Zhang handles missing text by leaving the description/ text set empty, not by generating replacement text where Tsimpoukelli fills this by teaching:
a text generation unit that generates generated text data by projecting the image feature amount extracted by the image understanding unit onto a vector space of the attached text data; and
( [Sec. 1, 3.1-3.3], [Figs. 2–3]: Tsimpoukelli discloses a visual prefix formed by linearly mapping the vision encoder's output image feature vector to the dimensionality D of the language model's token embedding space, reshaping it as a sequence of n embeddings each with dimensionality D, and presenting this visual prefix to the frozen autoregressive transformer as a sequence of continuous embeddings in the token embedding space, thereby projecting the image feature amount onto the vector space of the text data. The language model then generates text autoregressively conditioned on this visual prefix, producing generated text data from the image feature amount without requiring any attached text input. )
wherein, when the attached text data is not attached to the image data, the text understanding unit extracts the text feature amount by using the generated text data generated by the text generation unit as the attached text data.
( [Sec. 1, 3.1-3.3], [Figs. 1–3]: Tsimpoukelli teaches that, once trained, Frozen generates appropriate caption/ text output from image embeddings using a pretrained language model, and can process sequences of interleaved image and text embeddings. Tsimpoukelli therefore teaches generating caption/text data from image data itself, without requiring pre-existing attached text for the target image. In view of Zhang’s text-feature extraction pipeline, it would have been obvious to use Tsimpoukelli’s generated caption/ text as substitute textual information for Zhang’s text feature extraction when Zhang’s image has no attached textual information. )
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Zhang’s system to use Tsimpoukelli’s generated caption/ text as textual input when an image lacks attached textual information, because Zhang expressly recognizes that an image may lack textual information and leaves the corresponding description set null, while Zhang’s classification improvement depends on using textual information together with image information. Tsimpoukelli provides a known solution to that missing-text problem by generating caption/ text data from the image through an image-conditioned language model. A person of ordinary skill in the art would have been motivated to use Tsimpoukelli’s generated text in Zhang’s text-understanding pipeline to avoid a null/ empty text feature set and preserve Zhang’s intended multimodal image/text classification benefits, with the predictable result of generating substitute textual information from the image and using that text to extract a text feature vector for Zhang’s image/text feature fusion.
Regarding claim 6, Zhang [as modified by Tsimpoukelli] teaches the image processing apparatus according to claim 4, wherein the parameter update unit updates the text generation probability parameter on the basis of a loss based on the image feature amount and the attached text data and a loss based on the image feature amount and the generated text data.
( [Sec. 3.1-3.2], [Figs. 2]: Tsimpoukelli teaches updating vision encoder/ projection parameters that affect the conditional text-generation probabilities by maximizing the conditional likelihood
log
p
y
∣
x
=
∑
l
log
p
y
l
∣
x
y
1
.
.
.
y
l
-
1
, where the visual prefix
i
1
.
.
.
i
n
derived from the image feature amount is presented to the frozen transformer alongside the ground-truth caption tokens
y
1
.
.
.
y
L
, and gradients are back-propagated through the frozen transformer to the vision encoder parameters. This constitutes a loss based on the image feature amount and the attached text data, the negative log-likelihood of the ground-truth attached caption
y
conditioned on the visual prefix produced from the image feature amount
x
; thereby teaching updating model parameters that affect text-generation probability based on a loss/ negative log-likelihood determined from the image feature/ visual prefix and the attached caption text, and based on the model’s generated/ predicted token probabilities for the generated caption text. )
Regarding claim 15, Zhang teaches the image processing method of claim 11, however Zhang does not teach where Tsimpoukelli teaches:
wherein the image feature amount is projected onto a vector space of the text data and generates text data for a query.
( [Sec. 1, 3.1-3.3], [Figs. 2–3]: Tsimpoukelli teaches Frozen, which trains a vision encoder to represent each image as a sequence of continuous embeddings such that a pretrained language model prompted with the visual prefix generates appropriate caption/ text output. Tsimpoukelli teaches that the vision encoder takes a raw image and emits a continuous sequence to be consumed by the transformer language model. Tsimpoukelli further teaches forming a visual prefix by linearly mapping the vision encoder output to the dimensionality of the language model’s token embeddings and reshaping the result as a sequence of embeddings, each having the same dimensionality as a token embedding; thereby projecting the image feature amount onto the vector space of the text/n token embeddings. Tsimpoukelli further teaches that, at inference time, a language model conditioned upon an arbitrary text prompt or prefix generates text sequences autoregressively, and that images may be included in the prompt by placing image embeddings next to text embeddings. Tsimpoukelli expressly teaches use of this interface for visual question answering and outside-knowledge question answering. Thus, Tsimpoukelli teaches generating text data for a query based on an image-derived visual prefix in the text/ token embedding space. )
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Zhang’s image processing method to include Tsimpoukelli’s image-conditioned language-model generation technique because Zhang teaches a multimodal image/ text feature-processing method in which image information and text information are mapped and used together, while Tsimpoukelli teaches a known technique for mapping image-derived features into a language-model token-embedding space and generating text output in response to image/ text prompts or queries. A person of ordinary skill in the art would have been motivated to incorporate Tsimpoukelli’s query-responsive text generation into Zhang’s multimodal image/text processing method to provide generated textual information corresponding to image content and usable in Zhang’s text-feature processing/ fusion pipeline, with the predictable result of generating query-responsive text from image features projected into a language/ text vector space.
Claim 7 is rejected under 35 U.S.C. §103 as being unpatentable over Zhang [as modified by Tsimpoukelli] in view of Khosla (Khosla et al, “Supervised Contrastive Learning.” ArXiv.org, 2020 ).
Regarding claim 7, Zhang [as modified by Tsimpoukelli] teaches the image processing apparatus according to claim 4,
Zhang [as modified by Tsimpoukelli] teaches updating parameters affecting text-generation probability, but does not expressly teach that the update is performed based on same-class and different-class relationships between attached-text and generated-text feature amounts where Khosla teaches:
wherein the parameter update unit updates the text generation probability parameter such that a text feature amount of the attached text data and a text feature amount of the generated text data for image data of the same class approach each other, and updates the text generation probability parameter such that a text feature amount of the attached text data and a text feature amount of the generated text data for image data of different classes move away from each other.
( [Sec. 1, 3], [Figs. 2]: Khosla teaches supervised contrastive learning in which label information is used so that clusters of points belonging to the same class are pulled together in embedding space while clusters of samples from different classes are pushed apart. Khosla teaches that normalized embeddings from the same class are pulled closer together than embeddings from different classes. Khosla further teaches that, for each anchor, positives are samples having the same class label as the anchor, while negatives are samples from the remainder of the batch having different class labels. Khosla’s supervised contrastive loss includes the same-class positives in the numerator and includes positives and negatives in the denominator, thereby training the model to increase similarity between same-class embeddings and decrease relative similarity between different-class embeddings. Khosla also defines negatives N(i) as samples whose labels differ from the anchor and teaches that the loss gradient includes negative-sample terms, such that hard negatives have larger gradient contributions during training. Therefore, Khosla teaches updating model parameters using a supervised contrastive loss so that same-class feature representations approach each other and different-class feature representations move away from each other. )
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the parameter update unit of Zhang [as modified by Tsimpoukelli] to incorporate the contrastive loss function taught by Khosla. The motivation for doing so would be to improve the clustering robustness and discriminative power of the shared image-text vector space. By ensuring that the model parameters are updated such that feature amounts from the same class approach each other while those from different classes move away from each other, the system forces the neural network to learn class-specific semantic representations, thereby increasing classification accuracy and stability beyond what standard cross-entropy loss can achieve alone.
Claim 8 is rejected under 35 U.S.C. §103 as being unpatentable over Zhang [as modified by Tsimpoukelli] in view of Fan (Fan et al, “Hierarchical Neural Story Generation.” ArXiv.org, 2018).
Regarding claim 8, Zhang [as modified by Tsimpoukelli] teaches the image processing apparatus according to claim 4,
Zhang [as modified by Tsimpoukelli] teaches generating text data from an image-derived feature amount using a language model, but does not expressly teach controlling the text-generation process where Fan teaches:
wherein the text generation unit uses both generation of the generated text data by random sampling from a predetermined number of high-order tokens at each of times and generation of the generated text data at a normal time.
( [Sec. 5.4]: Fan teaches generating text using a top-k random sampling scheme. At each timestep, the model generates a probability for each word in the vocabulary as the likely next word and randomly samples from k = 10 most likely candidates from this distribution. Fan further teaches that subsequent timesteps generate words based on the previously selected words. Therefore, Fan teaches randomly sampling one token from a predetermined number of high-order/ high-probability candidate tokens at each generation timestep. Fan also discusses normal generation alternatives, including beam search and unrestricted random sampling, and explains that top-k sampling is used to reduce low-probability samples while avoiding repetitive text from beam search. )
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the text generation unit of Zhang [as modified by Tsimpoukelli] to incorporate the generation techniques taught by Fan. The motivation for doing so would be to overcome the well-known limitations of purely deterministic decoding, which frequently results in generic, repetitive, or logically trapped text sequences. Zhang [as modified by Tsimpoukelli] supplies ordinary/ greedy language-model generation, and Fan supplies top-k random sampling as an alternative/ additional decoding mode. By enabling the system to alternate or selectively use random sampling from a predetermined number of high-order tokens (Top-k sampling) alongside standard generation, the system maintains contextual accuracy while significantly increasing the diversity and natural fluency of the generated text data.
Claim 9 is rejected under 35 U.S.C. §103 as being unpatentable over Zhang [as modified by Tsimpoukelli] in view of Mihaylova (Mihaylova et al, “Scheduled Sampling for Transformers.” ArXiv.org, 2019).
Regarding claim 9, Zhang [as modified by Tsimpoukelli] teaches the image processing apparatus according to claim 4,
Zhang [as modified by Tsimpoukelli] teaches extracting a text feature amount and generating text tokens based on a parameterized probability distribution, but does not expressly teach where Mihaylova teaches:
wherein the text understanding unit extracts the text feature amount by performing weighted pooling using a probability that indicates a likelihood of a predetermined-order token related to the generated text data.
( [Sec. 3], [Figs. 1]: Mihaylova teaches a Transformer scheduled-sampling architecture in which a first decoder pass produces model predictions, including scores for vocabulary words at each position, and a second decoder pass conditions on a mixture of the gold target sequence and the model predictions. Mihaylova further teaches several ways of forming a vector from the model predictions, including mixing top-k embeddings by using the weighted average of embeddings of the top-5 scored vocabulary words. Mihaylova also teaches forming a vector as a sum of embeddings of vocabulary words weighted by a softmax of the prediction scores, i.e., using weights based on the model’s predicted likelihoods/scores for candidate tokens. Therefore, Mihaylova teaches extracting a text/ vector representation by weighted pooling of token embeddings using probabilities/ scores indicating the likelihood of predetermined high-order/ top-k tokens related to generated text data. )
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the text feature extraction unit of Zhang [as modified by Tsimpoukelli] to incorporate the continuous approximation and weighted probability pooling taught by Mihaylova. The motivation for doing so would be to allow the model to remain fully differentiable during the text generation and extraction process. By extracting the text feature amount via a weighted sum (pooling) of token embeddings based on their predicted likelihoods (probabilities), the system avoids the non-differentiable operation of hard discrete token selection (argmax). This continuous approximation passes the model's exact uncertainty and probability distribution forward into the text feature amount, significantly improving gradient flow and the stability of the neural network during training.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEN KUDO whose telephone number is (571)272-4498. The examiner can normally be reached M-F 8am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
KEN KUDO
Examiner
Art Unit 2671
/KEN KUDO/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights