DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This communication is in response to the Application Filed on 12/02/2024
Claims 1–18 are pending in this application.
Drawings
The drawing(s) filed on 12/02/2024 are accepted by the Examiner.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/02/2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1–3, 6–8, 10, 12–14, 17 and 18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The limitations, under their broadest reasonable interpretation, cover mental process (concept performed in a human mind, including as observation, evaluation, judgment, opinion, organizing human activity and mathematical concepts and calculations). The independent claim(s) 1, 8 and 12 recite(s) a device, a device and a method. This judicial exception is not integrated into a practical application because the steps do not add meaningful limitations to be considered specifically applied to a particular technological problem to be solved .The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the steps of the claimed invention can be done mentally and no additional features in the claims would preclude them from being performed as such except for the generic computer elements at high level of generality (i.e., processor, memory).
According to the USPTO guidelines, a claim is directed to non-statutory subject matter if:
STEP 1: the claim does not fall within one of the four statutory categories of invention (process, machine, manufacture or composition of matter), or
STEP 2: the claim recites a judicial exception, e.g. an abstract idea, without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis:
STEP 2A (PRONG 1): Does the claim recite an abstract idea, law of nature, or natural phenomenon?
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
Using the two-step inquiry, it is clear that the independent claims 1, 8 and 12 are directed to an abstract idea as shown below:
STEP 1: Do the claims fall within one of the statutory categories? YES. Independent claims 1, 8 and 12 are directed to a device, a device and a method respectively.
STEP 2A (PRONG 1): Is the claim directed to a law of nature, a natural phenomenon or an abstract idea? YES, the claims are directed toward a mental process (i.e. abstract idea).
With regard to STEP 2A (PRONG 1), the guidelines provide three groupings of subject matter that are considered abstract ideas:
Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations;
Certain methods of organizing human activity – fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions); and
Mental processes – concepts that are practicably performed in the human mind (including an observation, evaluation, judgment, opinion).
Independent claims 1, 8 and 12 comprise a mental process that can be practicably performed in the human mind (or generic computers or components configured to perform the method) and, therefore, an abstract idea.
Regarding independent claim(s) 1: the limitations recite:
A device for extracting a multimodal object on the basis of composite embedding, the device comprising (Generic Computer Component):
a memory configured to store computer-readable instructions (Generic Computer Component); and
at least one processor configured to execute the instructions (Generic Computer Component),
wherein the at least one processor executes the instructions to (Generic Computer Component):
extract training natural language text and training images from a training data storage (mental process including observation and evaluation, and can be done mentally in the human mind);
generate image composite embeddings including embeddings of the training images and key objects included in the training images (mathematical relationships, mathematical calculations);
generate natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text (mathematical relationships, mathematical calculations); and
measure multimodal similarities between the image composite embeddings and the natural language composite embeddings (mental process including observation and evaluation, and can be done mentally in the human mind).
Regarding independent claim(s) 8: the limitations recite:
A device for extracting a multimodal object on the basis of composite embedding, the device comprising (Generic Computer Component):
a memory configured to store computer-readable instructions (Generic Computer Component); and
at least one processor configured to execute the instructions (Generic Computer Component),
wherein the at least one processor executes the instructions to (Generic Computer Component):
generate, for each of training images stored in a training data storage, image composite embeddings including image embeddings for all the training images and key object embeddings which are embeddings of key objects included in all the training images (mathematical relationships, mathematical calculations) and stores the image composite embeddings in an image composite embedding storage (Generic Computer Component);
extract training natural language text and one or more training images from the training data storage (mental process including observation and evaluation, and can be done mentally in the human mind);
generate natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text (mathematical relationships, mathematical calculations);
extract image composite embeddings matching the one or more training images from the image composite embedding storage (mental process including observation and evaluation, and can be done mentally in the human mind); and
measure multimodal similarities between the image composite embeddings matching the one or more training images and the natural language composite embeddings (mental process including observation and evaluation, and can be done mentally in the human mind).
Regarding independent claim(s) 12: the limitations recite:
A method of extracting a multimodal object on the basis of composite embedding, the method comprising (Generic Computer Component):
extracting, by a device for extracting a multimodal object on the basis of composite embedding (mental process including observation and evaluation, and can be done mentally in the human mind) which includes a memory for storing computer-readable instructions (Generic Computer Component) and at least one processor for executing the instructions (Generic Computer Component), training natural language text and training images from a training data storage;
generating, by the device, image composite embeddings including embeddings of the training images and key objects included in the training images (mathematical relationships, mathematical calculations);
generating, by the device, natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text (mathematical relationships, mathematical calculations); and
measuring, by the device, multimodal similarities between the image composite embeddings and the natural language composite embeddings (mental process including observation and evaluation, and can be done mentally in the human mind).
These limitations, as drafted, is a simple process that, under their broadest reasonable interpretation, covers performance of the limitations in the mind or by a human. The Examiner notes that under MPEP 2106.04(a)(2)(III), the courts consider a mental process (thinking) that “can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 ("‘[M]ental processes[] and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978) (same).
As such, a person could mentally pick an image to extract visual text and objects then represent the text and objects using math and compare the formula or representation to see how close they are. The mere nominal recitation that the various steps are being executed by a device, a processor and memory does not take the limitations out of the mental process grouping. Thus, the claims recite a mental process.
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application? NO, the claims do not recite additional elements that integrate the judicial exception into a practical application.
With regard to STEP 2A (prong 2), whether the claim recites additional elements that integrate the judicial exception into a practical application, the guidelines provide the following exemplary considerations that are indicative that an additional element (or combination of elements) may have integrated the judicial exception into a practical application:
an additional element reflects an improvement in the functioning of a computer, or an improvement to other technology or technical field;
an additional element that applies or uses a judicial exception to affect a particular treatment or prophylaxis for a disease or medical condition;
an additional element implements a judicial exception with, or uses a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim;
an additional element effects a transformation or reduction of a particular article to a different state or thing; and
an additional element applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception.
While the guidelines further state that the exemplary considerations are not an exhaustive list and that there may be other examples of integrating the exception into a practical application, the guidelines also list examples in which a judicial exception has not been integrated into a practical application:
an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea;
an additional element adds insignificant extra-solution activity to the judicial exception; and
an additional element does no more than generally link the use of a judicial exception to a particular technological environment or field of use.
Independent claims 1, 8 and 12 do not recite any of the exemplary considerations that are indicative of an abstract idea having been integrated into a practical application. Independent claims 1, 8 and 12 discloses generic computer components, for example, a device, memory, processor, which are generic computer components and/or insignificant pre/post-solution extra activity that do not add a meaningful limitation to the abstract idea because they amount to simply implementing the abstract idea in a method.
These limitations are recited at a high level of generality (i.e. as a general action or change being taken based on the results of the acquiring step) and amounts to mere post solution actions, which is a form of insignificant extra-solution activity. Further, the claims are claimed generically and are operating in their ordinary capacity such that they do not use the judicial exception in a manner that imposes a meaningful limit on the judicial exception. Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No, the claims do not recite additional elements that amount to significantly more than the judicial exception.
With regard to STEP 2B, whether the claims recite additional elements that provide significantly more than the recited judicial exception, the guidelines specify that the pre-guideline procedure is still in effect. Specifically, that examiners should continue to consider whether an additional element or combination of elements:
adds a specific limitation or combination of limitations that are not well-understood, routine, conventional activity in the field, which is indicative that an inventive concept may be present; or
simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, which is indicative that an inventive concept may not be present.
Independent claim(s) 1, 8 and 12 do not recite any additional elements that are not well-understood, routine or conventional. The use of a generic computer elements are routine, well-understood and conventional process that is performed by computers.
Thus, since independent claims 1, 8 and 12 are: (a) directed toward an abstract idea, (b) do not recite additional elements that integrate the judicial exception into a practical application, and (c) do not recite additional elements that amount to significantly more than the judicial exception, it is clear that independent claims 1, 8 and 12 are not eligible subject matter under 35 U.S.C 101.
Regarding claim 2, 3, 6, 7, 10, 13, 14, 17 and 18: the additional limitations do not integrate the mental process into practical application or add significantly more to the mental process. The limitation(s) are mental processes that include observation and evaluation.
Regarding claim 4, 5, 9, 11, 15 and 16: the additional limitations do integrate the mental process into practical application or add significantly more to the mental process.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claim(s) 1, 2, 4–13 and 15–18 are rejected under 35 U.S.C. 103 as being unpatentable over Pham et al. (US 12400432 B2, hereafter, "Pham") in view of Yu et al. (US 20240290081 A1, hereafter, "Yu") further in view of Yuan et al. (See NPL attached, "Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval", hereafter, "Yuan").
Regarding claim 1, Pham discloses a device for extracting a multimodal object on the basis of composite embedding (See Pham, [Abstract], Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using memory-optimized contrastive learning to train image encoder and text encoder neural networks), the device comprising:
a memory configured to store computer-readable instructions (See Pham, [Col. 13, ln. 12–17], Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data); and
at least one processor configured to execute the instructions, wherein the at least one processor executes the instructions (See Pham, [Col. 12, ln. 20–24], The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers) to:
extract training natural language text and training images from a training data storage (See Pham, [Col. 6, ln. 3–5], The system obtains a batch of training pairs (step 202). Each training pair including an input image and an input text segment);
[generate image composite embeddings including embeddings of the training images and key objects included in the training images;
generate natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text]; and
measure multimodal similarities between the image composite embeddings and the natural language composite embeddings (See Pham, [Col. 6, ln. 59–66], The system then generates, for each training pair in the batch and using the respective image embeddings and the respective text embeddings for the plurality of chunks stored in the memory of the set of one or more computing devices, a respective similarity between the image embedding of the input image in the training pair and the respective text embeddings of the input text segments in all of the training pairs in the batch (step 212)).
However, Pham fail(s) to teach generate image composite embeddings including embeddings of the training images and key objects included in the training images; generate natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text.
Yu, working in the same field of endeavor, teaches: generate image composite embeddings including embeddings of the training images and key objects included in the training images (See Yu, ¶ [0029], The visual features of each RoI are then encoded through a feature extractor to generate the feature map embeddings 118. The RoI bounding boxes 116, the feature map embeddings 118, and the object category embeddings 120 are combined with (e.g., . . . added to) a vision segment embedding 115 associated with the vision modality to form the vision embeddings 110. Note: Examiner is interpreting the composite embeddings as the vision embeddings and the object embedding represent the key objects and the feature map embeddings that represent the image).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Pham’s reference to generate image composite embeddings including embeddings of the training images and key objects included in the training images based on the method of Yu’s reference. The suggestion/motivation would have been to improve the tasks of image classification and text and image searching (See Yu, [Col. 1, ln. 54–58]).
However, Pham and Yu fail(s) to teach generate natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text.
Yuan, working in the same field of endeavor, teaches: generate natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text (See Yuan, [Pg. 5, Col. 1, 3) Keywords Embedding, ln. 45–48], Considering that keywords information is a supplement to sentence information, we make the keyword embedding matrix and sentence embedding matrix share the parameter matrix
W
e
. Note: Examiner is interpreting the shared parameter matrix as the composite embedding).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Pham’s reference generate natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text based on the method of Yuan’s reference. The suggestion/motivation would have been to improve accuracy of image retrieval using text and image (See Yuan, [Pg. 11, Table II]).
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Yu and Yuan with Pham to obtain the invention as specified in claim 1.
Regarding claim 2, Pham discloses the device of claim 1, wherein the at least one processor measures errors of the multimodal similarities on the basis of ground truth information extracted from the training data storage (See Pham, [Col. 4, ln. 38–42], Based on the embeddings for the images and the text segments in the pairs in the mini-batch, an NxN similarity matrix A is computed, where
A
i
j
is a value that represents how similar the image embedding of xi is to the text embedding. [Col. 4, ln. 46–52], The system 100 can then train the neural network 110, the neural network 120, or both using gradients of a contrastive loss computed using the matrix A. For example, the contrastive loss can be the cross-entropy loss on the rows and columns of A, where the diagonal entries are treated as correct classes while other entries are treated as incorrect classes).
Regarding claim 4, Pham in view of Yu further in view of Yuan teaches the device of claim 1, [wherein the at least one processor generates image embeddings for the training images using an image embedding model, extracts the key objects from the training images using a key object extraction model, generates key object embeddings which are the embeddings for the key objects using a key object embedding model, and generates the image composite embeddings on the basis of the image embeddings and the key object embeddings].
However, Pham fail(s) to teach wherein the at least one processor generates image embeddings for the training images using an image embedding model, extracts the key objects from the training images using a key object extraction model, generates key object embeddings which are the embeddings for the key objects using a key object embedding model, and generates the image composite embeddings on the basis of the image embeddings and the key object embeddings.
Yu, working in the same field of endeavor, teaches: wherein the at least one processor generates image embeddings for the training images using an image embedding model (See Yu, ¶ [0063], the vision embeddings include RoI bounding boxes (e.g., RoI bounding boxes 116), feature map embeddings (e.g., feature map embeddings 118), and object category embeddings (e.g., object category embeddings 120) which are generated using operations and/or processes (e.g., an object detection model that detects and/or identifies objects in portions of an image, a model that classifies those detected objects into categories, and/or a component that identifies the boundaries of bounding boxes in the image)), extracts the key objects from the training images using a key object extraction model, generates key object embeddings which are the embeddings for the key objects using a key object embedding model, and generates the image composite embeddings on the basis of the image embeddings and the key object embeddings (See Yu, ¶ [0029], The visual features of each RoI are then encoded through a feature extractor to generate the feature map embeddings 118. The RoI bounding boxes 116, the feature map embeddings 118, and the object category embeddings 120 are combined with (e.g., . . . added to) a vision segment embedding 115 associated with the vision modality to form the vision embeddings 110. Note: Examiner is interpreting the composite embeddings as the vision embeddings and the object embedding represent the key objects and the feature map embeddings that represent the image).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Yu’s reference to wherein the at least one processor generates image embeddings for the training images using an image embedding model, extracts the key objects from the training images using a key object extraction model, generates key object embeddings which are the embeddings for the key objects using a key object embedding model, and generates the image composite embeddings on the basis of the image embeddings and the key object embeddings based on the method of Yu’s reference. The suggestion/motivation would have been to improve the tasks of image classification and text and image searching (See Yu, [Col. 1, ln. 54–58]).
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Yu with Pham and Yuan to obtain the invention as specified in claim 4.
Regarding claim 5, Pham in view of Yu further in view of Yuan teaches the device of claim 1, [wherein the at least one processor generates natural language embeddings which are the embeddings of the training natural language text using a natural language embedding model, extracts the key words of the training natural language text using a key word extraction model, generates key word embeddings which are the embeddings of the key words using the natural language embedding model, and generates the natural language composite embeddings on the basis of the natural language embeddings and the key word embeddings].
However, Pham and Yu fail(s) to teach wherein the at least one processor generates natural language embeddings which are the embeddings of the training natural language text using a natural language embedding model, extracts the key words of the training natural language text using a key word extraction model, generates key word embeddings which are the embeddings of the key words using the natural language embedding model, and generates the natural language composite embeddings on the basis of the natural language embeddings and the key word embeddings.
Yuan, working in the same field of endeavor, teaches: wherein the at least one processor generates natural language embeddings which are the embeddings of the training natural language text using a natural language embedding model, extracts the key words of the training natural language text using a key word extraction model, generates key word embeddings which are the embeddings of the key words using the natural language embedding model, and generates the natural language composite embeddings on the basis of the natural language embeddings and the key word embeddings (See Yuan, [Pg. 5, Col. 1, 3) Keywords Embedding, ln. 45-48], Considering that keywords information is a supplement to sentence information, we make the keyword embedding matrix and sentence embedding matrix share the parameter matrix
W
e
. Note: Examiner is interpreting the shared parameter matrix as the composite embedding).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Pham’s reference to wherein the at least one processor generates natural language embeddings which are the embeddings of the training natural language text using a natural language embedding model, extracts the key words of the training natural language text using a key word extraction model, generates key word embeddings which are the embeddings of the key words using the natural language embedding model, and generates the natural language composite embeddings on the basis of the natural language embeddings and the key word embeddings based on the method of Yuan’s reference. The suggestion/motivation would have been to improve accuracy of image retrieval using text and image (See Yuan, [Pg. 11, Table II]).
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Yuan with Pham and Yu to obtain the invention as specified in claim 5.
Regarding claim 6, Pham discloses the device of claim 4, wherein the at least one processor updates parameters of the image embedding model and the key object embedding model on the basis of errors of the multimodal similarities (See Pham, [Col. 4, ln. 46-48], The system 100 can then train the neural network 110, the neural network 120, or both using gradients of a contrastive loss computed using the matrix A. See also [FIG. 1], 110 IMAGE ENCODER NEURAL NETWORK, 120 TEXT ENCODER NEURAL NETWORK).
Regarding claim 7, Pham discloses the device of claim 5, wherein the at least one processor updates parameters of the natural language embedding model on the basis of errors of the multimodal similarities (See Pham, [Col. 4, ln. 46-48], The system 100 can then train the neural network 110, the neural network 120, or both using gradients of a contrastive loss computed using the matrix A. See also [FIG. 1], 110 IMAGE ENCODER NEURAL NETWORK, 120 TEXT ENCODER NEURAL NETWORK).
Regarding claim 8, Pham discloses a device for extracting a multimodal object on the basis of composite embedding (See Pham, [Abstract], Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using memory-optimized contrastive learning to train image encoder and text encoder neural networks), the device comprising:
a memory configured to store computer-readable instructions (See Pham, [Col. 13, ln. 12–17], Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data); and
at least one processor configured to execute the instructions (See Pham, [Col. 12, ln. 20–24], The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers),
wherein the at least one processor executes the instructions to (See Pham, [Col. 12, ln. 20–24], The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers):
[generate, for each of training images stored in a training data storage, image composite embeddings including image embeddings for all the training images and key object embeddings which are embeddings of key objects included in all the training images] and stores the image composite embeddings in an image composite embedding storage (See Pham, [Col. 6, ln. 47–49], The system then stores, in memory of the set of one or more computing devices, the respective image embeddings and the respective text embeddings (step 210));
extract training natural language text and one or more training images from the training data storage (See Pham, [Col. 6, ln. 3–5], The system obtains a batch of training pairs (step 202). Each training pair including an input image and an input text segment);
[generate natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text];
extract image composite embeddings matching the one or more training images from the image composite embedding storage (See Pham, [Col. 6, ln. 59–62], The system then generates, for each training pair in the batch and using the respective image embeddings and the respective text embeddings for the plurality of chunks stored in the memory of the set of one or more computing devices. Note: the method is extracting the embeddings stored in memory to compare them); and
measure multimodal similarities between the image composite embeddings matching the one or more training images and the natural language composite embeddings (See Pham, [Col. 6, ln. 59–66], The system then generates, for each training pair in the batch and using the respective image embeddings and the respective text embeddings for the plurality of chunks stored in the memory of the set of one or more computing devices, a respective similarity between the image embedding of the input image in the training pair and the respective text embeddings of the input text segments in all of the training pairs in the batch (step 212)).
However, Pham fail(s) to teach generate, for each of training images stored in a training data storage, image composite embeddings including image embeddings for all the training images and key object embeddings which are embeddings of key objects included in all the training images; generate natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text.
Yu, working in the same field of endeavor, teaches: generate, for each of training images stored in a training data storage, image composite embeddings including image embeddings for all the training images and key object embeddings which are embeddings of key objects included in all the training images (See Yu, ¶ [0029], The visual features of each RoI are then encoded through a feature extractor to generate the feature map embeddings 118. The RoI bounding boxes 116, the feature map embeddings 118, and the object category embeddings 120 are combined with (e.g., . . . added to) a vision segment embedding 115 associated with the vision modality to form the vision embeddings 110. Note: Examiner is the composite embeddings as the vision embeddings and the object embedding represent the key objects and the feature map embeddings that represent the image).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Pham’s reference generate, for each of training images stored in a training data storage, image composite embeddings including image embeddings for all the training images and key object embeddings which are embeddings of key objects included in all the training images based on the method of Yu’s reference. The suggestion/motivation would have been to improve the tasks of image classification and text and image searching (See Yu, [Col. 1, ln. 54–58]).
However, Pham and Yu fail(s) to teach generate natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text.
Yuan, working in the same field of endeavor, teaches: generate natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text (See Yuan, [Pg. 5, Col. 1, 3) Keywords Embedding, ln. 45–48], Considering that keywords information is a supplement to sentence information, we make the keyword embedding matrix and sentence embedding matrix share the parameter matrix
W
e
. Note: Examiner is interpreting the shared parameter matrix as the composite embedding).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Pham’s reference generate natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text based on the method of Yuan’s reference. The suggestion/motivation would have been to improve accuracy of image retrieval using text and image (See Yuan, [Pg. 11, Table II]).
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Yu and Yuan with Pham to obtain the invention as specified in claim 8.
Regarding claim 9, claim 9 is rejected the same as claim 4 and the arguments similar to that presented above for claim 4 are equally applicable to the claim 9, and all of the other limitations similar to claim 4 are not repeated herein, but incorporated by reference.
Regarding claim 10, Pham in view of Yu further in view of Yuan teaches the device of claim 9, [wherein the image embedding model and the key object embedding model are pretrained models].
However, Pham fail(s) to teach wherein the image embedding model and the key object embedding model are pretrained models.
Yu, working in the same field of endeavor, teaches: wherein the image embedding model and the key object embedding model are pretrained models (See Yu, ¶ [0063], the vision embeddings include RoI bounding boxes (e.g., RoI bounding boxes 116), feature map embeddings (e.g., feature map embeddings 118), and object category embeddings (e.g., object category embeddings 120) which are generated using operations and/or processes (e.g., an object detection model that detects and/or identifies objects in portions of an image, a model that classifies those detected objects into categories, and/or a component that identifies the boundaries of bounding boxes in the image)).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Yu’s reference wherein the image embedding model and the key object embedding model are pretrained models based on the method of Yu’s reference. The suggestion/motivation would have been to improve the tasks of image classification and text and image searching (See Yu, [Col. 1, ln. 54–58]).
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Yu with Pham and Yuan to obtain the invention as specified in claim 10.
Regarding claim 11, Pham in view of Yu further in view of Yuan teaches the device of claim 10, [wherein the at least one processor generates natural language embeddings which are the embeddings of the training natural language text using a natural language embedding model, extracts the key words of the training natural language text using a key word extraction model, generates key word embeddings which are the embeddings of the key words using the natural language embedding model, generates the natural language composite embeddings on the basis of the natural language embeddings and the key word embeddings], and updates parameters of the natural language embedding model on the basis of errors of the multimodal similarities (See Pham, [Col. 4, ln. 46-48], The system 100 can then train the neural network 110, the neural network 120, or both using gradients of a contrastive loss computed using the matrix A. See also [FIG. 1], 110 IMAGE ENCODER NEURAL NETWORK, 120 TEXT ENCODER NEURAL NETWORK).
However, Pham and Yuan fail(s) to teach wherein the at least one processor generates natural language embeddings which are the embeddings of the training natural language text using a natural language embedding model, extracts the key words of the training natural language text using a key word extraction model, generates key word embeddings which are the embeddings of the key words using the natural language embedding model, generates the natural language composite embeddings on the basis of the natural language embeddings and the key word embeddings.
Yuan, working in the same field of endeavor, teaches: wherein the at least one processor generates natural language embeddings which are the embeddings of the training natural language text using a natural language embedding model, extracts the key words of the training natural language text using a key word extraction model, generates key word embeddings which are the embeddings of the key words using the natural language embedding model, generates the natural language composite embeddings on the basis of the natural language embeddings and the key word embeddings (See Yuan, [Pg. 5, Col. 1, 3) Keywords Embedding, ln. 45–48], Considering that keywords information is a supplement to sentence information, we make the keyword embedding matrix and sentence embedding matrix share the parameter matrix
W
e
. Note: Examiner is interpreting the shared parameter matrix as the composite embedding).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Pham’s reference to wherein the at least one processor generates natural language embeddings which are the embeddings of the training natural language text using a natural language embedding model, extracts the key words of the training natural language text using a key word extraction model, generates key word embeddings which are the embeddings of the key words using the natural language embedding model, generates the natural language composite embeddings on the basis of the natural language embeddings and the key word embeddings based on the method of Yuan’s reference. The suggestion/motivation would have been to improve accuracy of image retrieval using text and image (See Yuan, [Pg. 11, Table II]).
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Yuan with Pham and Yu to obtain the invention as specified in claim 11.
Regarding claim 12, claim 12 is rejected the same as claim 1 and the arguments similar to that presented above for claim 1 are equally applicable to the claim 12, and all of the other limitations similar to claim 1 are not repeated herein, but incorporated by reference.
Regarding claim 13, claim 13 is rejected the same as claim 2 and the arguments similar to that presented above for claim 2 are equally applicable to the claim 13, and all of the other limitations similar to claim 2 are not repeated herein, but incorporated by reference.
Regarding claim 15, claim 15 is rejected the same as claim 4 and the arguments similar to that presented above for claim 4 are equally applicable to the claim 15, and all of the other limitations similar to claim 4 are not repeated herein, but incorporated by reference.
Regarding claim 16, claim 16 is rejected the same as claim 5 and the arguments similar to that presented above for claim 5 are equally applicable to the claim 16, and all of the other limitations similar to claim 5 are not repeated herein, but incorporated by reference.
Regarding claim 17, claim 17 is rejected the same as claim 6 and the arguments similar to that presented above for claim 6 are equally applicable to the claim 17, and all of the other limitations similar to claim 6 are not repeated herein, but incorporated by reference.
Regarding claim 18, claim 18 is rejected the same as claim 7 and the arguments similar to that presented above for claim 7 are equally applicable to the claim 18, and all of the other limitations similar to claim 7 are not repeated herein, but incorporated by reference.
Claim(s) 3 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Pham et al. (US 12400432 B2, hereafter, "Pham") in view of Yu et al. (US 20240290081 A1, hereafter, "Yu") further in view of Yuan et al. (See NPL attached, "Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval", hereafter, "Yuan") and further in view of Obinata et al. (US 20240212355 A1, hereafter, "Obinata").
Regarding claim 3, Pham in view of Yu further in view of Yuan teaches the device of claim 1, [wherein the at least one processor extracts a positive image related to the training natural language text from the training data storage, extracts one or more negative images unrelated to the training natural language text from the training data storage, and configures the training images including the positive image and the negative images].
However, Pham, Yu and Yuan fail(s) to teach generate natural language composite embeddings including embeddings of the training natural language text and key words included in the training natural language text.
Obinata, working in the same field of endeavor, teaches: wherein the at least one processor extracts a positive image related to the training natural language text from the training data storage, extracts one or more negative images unrelated to the training natural language text from the training data storage, and configures the training images including the positive image and the negative images (See Obinata, ¶ [0095], That is, since one positive example and N−1 negative examples are set for each piece of the training data, N positive examples and N.sup.2−N negative examples are generated in the entire mini-batch. For example, in the example of the similarity matrix M1, N elements of diagonal components for which black and white inversion display is performed are used as positive examples, and N.sup.2−N elements for which white background display is performed are used as negative examples).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Pham’s reference wherein the at least one processor extracts a positive image related to the training natural language text from the training data storage, extracts one or more negative images unrelated to the training natural language text from the training data storage, and configures the training images including the positive image and the negative images based on the method of Obinata’s reference. The suggestion/motivation would have been to provide a more robust system against varying types of data (See Obinata, ¶ [0093, 0094]).
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Obinata with Pham, Yu and Yuan to obtain the invention as specified in claim 3.
Regarding claim 14, claim 14 is rejected the same as claim 3 and the arguments similar to that presented above for claim 3 are equally applicable to the claim 14, and all of the other limitations similar to claim 3 are not repeated herein, but incorporated by reference.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Marri et al. (US 20240378863 A1) teaches systems and methods for image tagging are provided. One aspect of the systems and methods includes encoding an image and a tag of the image using a multimodal encoder to obtain an image embedding and a text embedding, respectively. Another aspect of the systems and methods includes generating training data for a machine learning model by filtering a plurality of image-tag pairs based on a similarity between the image embedding and the text embedding. Another aspect of the systems and methods includes training the machine learning model using the training data.
Xu et al. (US 20230368509 A1) teaches one example method includes processing items that each have an associated image and a textual description. A first image feature vector is generated by processing a first image using a first machine learning model. A first textual feature vector is generated by processing a first textual description using a second machine learning model. The first image feature vector and the first textual feature vector are combined to generate a first combined feature vector for a first item. Similarity lists of similar items are generated for the first item based on similarities between the first image feature vector, the first text feature vector, the first combined feature vector and respective corresponding vectors of other items. The similarity lists for the first item are combined to generate a combined similarity list for the first item.
Wang et al. (US 20210150255 A1) teaches a bidirectional image-text retrieval method based on a multi-view joint embedding space includes: performing retrieval with reference to a semantic association relationship at a global level and a local level, obtaining the semantic association relationship at the global level and the local level in a frame-sentence view and a region-phrase view, and obtaining semantic association information in a global level subspace of frame and sentence in the frame-sentence view, obtaining semantic association information in a local level subspace of region and phrase in the region-phrase view, processing data by a dual-branch neural network in the two views to obtain an isomorphic feature and embedding the same in a common space, and using a constraint condition to reserve an original semantic relationship of the data during training, and merging the two semantic association relationships using multi-view merging and sorting to obtain a more accurate semantic similarity between data.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DION J SATCHER whose telephone number is (703)756-5849. The examiner can normally be reached Monday - Thursday 5:30 am - 2:30 pm, Friday 5:30 am - 9:30 am PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DION J SATCHER/Patent Examiner, Art Unit 2676
/SHEFALI D GORADIA/Primary Patent Examiner, Art Unit 2676
Prosecution Insights