DETAILED ACTION
Notice of Pre-AIA or AIA Status
Claims 1-10 are pending in this application. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-10 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claimed invention is directed to a judicial exception (i.e., abstract idea – an idea to itself/mathematical concept) without significantly more.
(1) Are the claims directed to a process, machine, manufacture or composition of matter;
(2A) Prong One: Are the claims directed to a judicially recognized exception, i.e., a law of nature, a natural phenomenon, or an abstract idea;
Prong Two: If the claims are directed to a judicial exception under Prong One, then is the judicial exception integrated into a practical application;
(2B) If the claims are directed to a judicial exception and do not integrate the judicial exception, do the claims provide an inventive concept.
(Step 1) In the context of the flowchart in MPEP § 2106, subsection III, Step (1): Are the claims directed to a process, machine, manufacture or composition of matter? YES: the instant claims recite a device, a method and a non-transitory computer readable medium.
(Step 2A) In the context of the flowchart in MPEP § 2106, subsection III, Step 2A Prong Two determines whether: Is the claim directed to a law of nature, a natural phenomenon, or an abstract idea? YES.
When viewed under the broadest most reasonable interpretation, the instant claims are directed to a Judicial Exception – an abstract idea belonging to the group of mathematical concept or mental process. As a whole, the claimed features can be interpreted as an overall mathematical process, as they are a natural language representation of an overall mathematical algorithm, which qualifies as a judicial exception.
Claim 9 is exemplary of the independent claims 1, 9 and 10, and is presented below. Specifically, the claim recites:
-; 9. An image selection method executed by a computer, comprising:
-; 9. acquiring text information specifying an image to be acquired from an image group;
-; 9. calculating a score which represents a degree of match between each image of the image group and the text information; and
-; 9. selecting images from the image group by sampling based on a distribution of the score.
When viewed under the broadest most reasonable interpretation, the instant claims are directed to a Judicial Exception – an abstract idea belonging to the group of mathematical concept or mental process.As a whole, the claimed features can be interpreted as a mental process that uses mathematical elements, and are a natural language representation of an overall mathematical algorithm, which qualifies as a judicial exception. The step of “acquiring text information specifying an image” is considered to be extra-solution activity. The steps for “calculating a score which represents a degree of match between each image of the image group and the text information” is considered to be judicially recited mathematical concept/algorithm. The step of “selecting images from the image group by sampling based on a distribution of the score” is merely an extra-solution step of ““selecting” to a meaningful dataset based on a mathematical comparison. There is nothing in the claim that requires more than an operation that a human, armed with the appropriate apparatus executing a mathematical algorithm (in this case “values”) can perform.
These features, in combination, are applied to an overall data stream, and the nature of the term "data stream" or "bitstream" can be applied to any input that can be computationally processed. As the overall features of the claims can be interpred to be a mathematical algorithm, it does meet the requirements of Step2A for a judicial exception.
Since the claim as a whole does not integrate the exception into a practical application, in which case the claim is directed to the judicial exception (Step 2A: YES), it requires further analysis under Step 2B (where it may still be eligible if it amounts to an inventive concept).
(Step 2B) In the context of the flowchart in MPEP 2106, subsection III, Step 2B determines whether: Does the claim recited additional elements that amount to significantly more than the judicial exception? NO.
The instant claims do not apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception of “provide/receive first/second fractional flow reserve value” and therefore does not integrate the judicial exception into a practical application.In accordance with the MPEP § 2106.04(d) Integration of a Judicial Exception Into A Practical Application [R-07.2022],
Similarly, in a growing body of decisions, the Federal Circuit has distinguished between claims that are ‘‘directed to’’ a judicial exception (which require further analysis to determine their eligibility) and those that are not (which are therefore patent eligible), e.g., claims that improve the functioning of a computer or other technology or technological field. See Diamond v. Diehr, 450 U.S. 175, 209 USPQ 1 (1981); Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972). See, e.g., MPEP § 2106.06(b) (summarizing Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 118 USPQ2d 1684 (Fed. Cir. 2016), McRO, Inc. v. Bandai Namco Games Am. Inc., 837 F.3d 1299, 120 USPQ2d 1091 (Fed. Cir. 2016), and other cases that were eligible as improvements to technology or computer functionality instead of being directed to abstract ideas)”
“Accordingly, after determining that a claim recites a judicial exception in Step 2A Prong One, examiners should evaluate whether the claim as a whole integrates the recited judicial exception into a practical application of the exception in Step 2A Prong Two. A claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception.” See MPEP § 2106.05 for discussion of Step 2B.
With respect to the claimed limitations, the following features are directed to a mathematical process that do not integrate the judicial exception into a practical application, as there is no “meaningful limit”.
In particular, the claim includes additional elements as follows and includes using a processing apparatus to perform the following:
a. -; 9. An image selection method executed by a computer, comprising:
b. -; 9. acquiring text information specifying an image to be acquired from an image group;
c. -; 9. calculating a score which represents a degree of match between each image of the image group and the text information; and
d. -; 9. selecting images from the image group by sampling based on a distribution of the score.
Step a. is a preamble and constitutes intended use. Step b. use an apparatus to “acquire” “text”/”image” at a high level of generality such that said “text”/”image” can be used in the operation of the recited judicial exception (the mental step of “acquire”). Supplying “text”/”image” does not provide for “integration” of the abstract idea into a practical application, and the terms are used at a level of generality that does not preclude the use of a general purpose computer for the overall implementation. In fact, there are no limits on the apparatus, which is recited at a high level of generality and thus said apparatus does nothing more than perform generic computing functions of “acquire” in the claim.
Step c. is a mathematical calculation that can be done mentally; there are no specifics about what they are doing in ‘calculating’ thus one can merely ‘look’ at the data and make a ‘determination’ as to whether a “score” represents a degree of match. This is considered a mathematical calculation step. One can manually calculate the ‘score’ and or use a mathematical formula for the calculation. Step d. of “selecting images” is an extra-solution step as there is no qualification on the metrics being assessed for “degree of math” or “sampling based on a distribution of the score”. At best, these appear to be mathematical operations, and the step of “selecting” does not make the claim as a whole patent eligible because the claim as a whole of judicial exception does not integrate into a practical application.
With regard to (2B), the pending claims show that the mental process allows the “technology” (whether it is computer technology or any other technology) to do something that it previously was not able to do, and does not show what is more than a routine in the art presented in the claims, i.e., the additional elements are nothing more than routine and well-known steps. There is no improvement to technology here.
The claimed features, at best would invoke the analysis of the features under MPEP § 2106.05(a) as to whether the features qualify as improvements to the functioning of a computer or to any other technology or technical field. Even when considering the relevant consideration for evaluating whether the additional elements amount to an inventive concept, the limitations do not recite any elements that can be considered to qualify as “significantly more” when recited in a claim with judicial exception. At best, the claimed limitations only recite features that “apply” the judicial exception, and add “insignificant extra-solution activity to the judicial exception”. Dependent claims 2-8 are rejected for the same reasons; claims are directed to a judicial exception and do not integrate the judicial exception.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 4, and 6-10 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Yu et al. (US PGPub 20240290081, originally filed on February 28, 2023).
Consider Claims 1, 9 and 10
Yu teaches:
1. An image selection device comprising: at least one memory configured to store instructions; and at least one processor configured to execute the instructions to: / 9. An image selection method executed by a computer, comprising: / 10. A non-transitory computer readable storage medium storing a program executed by a computer, the program causing the computer to: (Yu: abstract, A computerized method trains and uses a multimodal fusion transformer (MFT) model for content moderation. Language modality data and vision modality data associated with a multimodal media source is received. Language embeddings are generated from the language modality data and vision embeddings are generated from the vision modality data. Both kinds of embeddings are generated using operations and/or processes that are specific to the associated modalities. The language embeddings and vision embeddings are combined into combined embeddings and the MFT model is used with those combined embeddings to generate a language semantic output token, a vision semantic output token, and a combined semantic output token. Contrastive loss data is generated using the three semantic output tokens and the MFT model is adjusted using that contrastive loss data. After the MFT model is trained sufficiently, it is configured to perform content moderation operations using semantic output tokens. [0022] FIG. 1 is a block diagram illustrating an example system 100 configured for training a multimodal fusion transformer (MFT) 126 using contrastive loss data 134. In some examples, language modality data 104 and vision modality data 106 are extracted from a multimodal media source 102. Language embeddings 108 are generated from the language modality data 104 and vision embeddings 110 are generated from the vision modality data 106. The language embeddings 108 and the vision embeddings 110 are combined into combined embeddings 124 and provided to the MFT 126 as input. The MFT 126 is transformer model that is configured to generate a language semantic output token 128, a combined semantic output token 130 and a vision semantic output token 132. In other examples, the MFT 126 is configured to generate a combined semantic output token 130 using a combination of modality embeddings based on input modality data of a plurality of modalities and to generate modality-specific semantic output tokens (e.g., the language semantic output token 128 and the vision semantic output token 132) using modality embeddings based on input modality data of specific modalities. The tokens 128, 130, and 132 are used to generate contrastive loss data 134. Parameters and/or other aspects of the MFT 126 are adjusted based on the contrastive loss data 134 to train the MFT 126 to produce semantic output that is focused on the semantic meaning of the combination of the language modality and the vision modality of a multimodal media source, rather than semantic meanings of the language modality and/or vision modality alone.)
1. acquire text information specifying an image to be acquired from an image group; / 9. acquiring text information specifying an image to be acquired from an image group; / 10. acquire text information specifying an image to be acquired from an image group; (Yu: [0027] In some examples, the system 100 is configured to generate the language embeddings 108 from the language modality data 104 using trained language model(s) or parts thereof (e.g., an embedding layer of a transformer). For instance, in an example, the generation of the language embeddings 108 includes tokenizing the language modality data 104 into a set of language tokens and then mapping those language tokens to text token embeddings (e.g., text token embedding vectors as generated by embedding layers of a transformer). The language embeddings include text token embeddings 112 and position embeddings 114. The text token embeddings 112 include tokens of semantic data corresponding to specific words, phrases, symbols, or the like in the language modality data 104. The position embeddings 114 include position data that represents the positions of each corresponding text token embeddings 112 relative to each other (e.g., the first token has a position embedding 114 indicating that it is first, before the second token).)
1. calculate a score which represents a degree of match between each image of the image group and the text information; and /9. calculating a score which represents a degree of match between each image of the image group and the text information; / 10. calculate a score which represents a degree of match between each image of the image group and the text information; and (Yu: [0046] Additionally, or alternatively, the vision embeddings 210 include RoI bounding boxes 216, feature map embeddings 218, object category embeddings 220, and/or other vision embeddings (e.g., vision segment embedding 115). In some examples, a portion of embedding data associated with one or more of the RoI bounding boxes 216 are masked using RoI masks 244. For instance, in an example, an RoI mask 244 is applied to feature map embeddings 218 and object category embeddings 220 associated with one RoI bounding box of the RoI bounding boxes 216. The output of the MFT 226 is then used with the visual information of the masked embeddings to generate the masked RoI loss 250. The masked RoI loss 250 is used to train the MFT 226 to better predict visual information in a bounding box based on corresponding language tokens and/or other associated information. [0047] For instance, in an example, the masked RoI loss Lroi-cis is represented by the equation 4 below, where fVL is the MFT 226 output and cv denotes the object category tokens or scores detected by the object detection model.
PNG
media_image1.png
44
314
media_image1.png
Greyscale
)
1. select images from the image group by sampling based on a distribution of the score. /9.and selecting images from the image group by sampling based on a distribution of the score. / 10. select images from the image group by sampling based on a distribution of the score.(Yu: [0029] In an embodiment, the language modality data 104 is tokenized through a tokenizer and then provided to a token embedding layer to generate the text token embeddings 112, which are associated with corresponding position embeddings 114 and a language segment embedding 111 associated with the language modality. The vision modality data 106 is analyzed using an object detection model (e.g., . . . the R-CNN described above) that detects objects present in the image. A RoI bounding box 116 is defined surrounding the entire image without an associated object category and, for each detected object, a category label and RoI bounding box 116 is generated. The category labels are processed using the token embedding layer used to generate the text token embeddings 112, such that the object category embeddings 120 are semantically comparable to the text token embeddings 112. The bounding boxes of the detected objects are included with the RoI bounding boxes 116 in the vision embeddings 110. The visual features of each RoI are then encoded through a feature extractor to generate the feature map embeddings 118. The RoI bounding boxes 116, the feature map embeddings 118, and the object category embeddings 120 are combined with (e.g., . . . added to) a vision segment embedding 115 associated with the vision modality to form the vision embeddings 110. The concatenated pair of the language embeddings 108 and the vision embeddings 110 form the combined embeddings 124, which are provided to the MFT 126 as described herein. [0030] Segment embeddings, such as the language segment embedding 111 and the vision segment embedding 115, are embedding vectors that are specific to modalities. The segment embeddings are used to label associated embeddings to indicate a modality type with which the labeled embeddings are associated.)
Consider Claim 2.
Yu teaches:
2. The image selection device according to claim 1, wherein the image group is a sequence of images constituting a video, and wherein the at least one processor is configured to, upon determining that a time interval between two images selected by the sampling is shorter than a predetermined interval, re-perform the sampling of at least one of the two images. (Yu: [0019] In previous VL works, the model is first pretrained on a large scale VL dataset, and then finetuned for downstream tasks. In both pretrain and finetune stages, all the data used contains both image and text modalities. In the disclosure, this constraint is relaxed by mixing a multimodal dataset with additional unimodal CM datasets in pretraining. This is asymmetry in data as either modality may be missing in the data, so that the conventional multimodality (e.g, . . . each sample contains both modalities) setup becomes mixed-modality (e.g., . . . a mix of multimodality and unimodality, where each sample may contain both modalities or each unimodality). By including unimodal CM dataset in pretraining, the disclosed model learns the domain-specific knowledge which helps the model adapt to the downstream tasks. Hence, the downstream CM task performance is improved. [0075] At 504, language embeddings are generated using the language modality data and, at 506, vision embeddings are generated using the vision modality data. In some examples, 504 and 506 are performed in substantially the same way as 404 and 406 of method 400 as described above. For instance, in some examples, the language embeddings include text token embeddings and positional embeddings while the vision embeddings include RoI bounding boxes, feature map embeddings and object category token embeddings. Further, it should be understood that the generation of the language embeddings and the vision embeddings are performed by separate components at substantially the same time or in parallel without departing from the description. [0076] Further, in some examples, the method 500 includes generating a language classification token, a vision classification token, and a combined classification token as described above with respect to method 400. The classification tokens are then included as part of the combined embeddings for use by the MFT model as input.)
Consider Claim 4.
Yu teaches:
4. The image selection device according to claim 1, wherein the image group is a sequence of images constituting a video, andwherein the at least one processor is configured to calculate the score of each image of the sequence, based on the each image and a predetermined number of images adjacent to the each image in the sequence. (Yu: [0068] As shown by FIG. 6, image analyzer 298 comprises video module 340. Image analyzer 298 further comprises fill detection modules 342, 344 and grain quality module 346, 348 for image sources 294, 296, respectively. Video module 340 comprises program logic for receiving image signals from cameras 312, 322 and outputting such images to display 257 described above. [0029] In an embodiment, the language modality data 104 is tokenized through a tokenizer and then provided to a token embedding layer to generate the text token embeddings 112, which are associated with corresponding position embeddings 114 and a language segment embedding 111 associated with the language modality. The vision modality data 106 is analyzed using an object detection model (e.g., . . . the R-CNN described above) that detects objects present in the image. A RoI bounding box 116 is defined surrounding the entire image without an associated object category and, for each detected object, a category label and RoI bounding box 116 is generated. The category labels are processed using the token embedding layer used to generate the text token embeddings 112, such that the object category embeddings 120 are semantically comparable to the text token embeddings 112. The bounding boxes of the detected objects are included with the RoI bounding boxes 116 in the vision embeddings 110. The visual features of each RoI are then encoded through a feature extractor to generate the feature map embeddings 118. The RoI bounding boxes 116, the feature map embeddings 118, and the object category embeddings 120 are combined with (e.g., . . . added to) a vision segment embedding 115 associated with the vision modality to form the vision embeddings 110. The concatenated pair of the language embeddings 108 and the vision embeddings 110 form the combined embeddings 124, which are provided to the MFT 126 as described herein. [0030] Segment embeddings, such as the language segment embedding 111 and the vision segment embedding 115, are embedding vectors that are specific to modalities. The segment embeddings are used to label associated embeddings to indicate a modality type with which the labeled embeddings are associated.)
Consider Claim 6.
Yu teaches:
6. The image selection device according to claim 1, wherein the at least one processor is configured to detect a region of an object from each image of the image group, and calculate the score based on the region of the object and the text information. (Yu: [0046] Additionally, or alternatively, the vision embeddings 210 include RoI bounding boxes 216, feature map embeddings 218, object category embeddings 220, and/or other vision embeddings (e.g., vision segment embedding 115). In some examples, a portion of embedding data associated with one or more of the RoI bounding boxes 216 are masked using RoI masks 244. For instance, in an example, an RoI mask 244 is applied to feature map embeddings 218 and object category embeddings 220 associated with one RoI bounding box of the RoI bounding boxes 216. The output of the MFT 226 is then used with the visual information of the masked embeddings to generate the masked RoI loss 250. The masked RoI loss 250 is used to train the MFT 226 to better predict visual information in a bounding box based on corresponding language tokens and/or other associated information. [0047] For instance, in an example, the masked RoI loss Lroi-cis is represented by the equation 4 below, where fVL is the MFT 226 output and cv denotes the object category tokens or scores detected by the object detection model.
PNG
media_image1.png
44
314
media_image1.png
Greyscale
[0048] Further, in some examples, the MFT 226 is used to generate domain-specific classification loss 252 that is then used to train the MFT 226 to more accurately generate semantic output based on input associated with a particular domain. When training the MFT 226 for domain-specific performance, the training data pairs 236 include or consist entirely of domain-specific training data pairs 236. For instance, in an example where the MFT 226 is being trained to process media sources for content moderation, the training data pairs 236 include instances of media that would be considered for content moderation and indications of whether those instances of media are candidates for moderation based on their content. By iteratively training the MFT 226 using large quantities of domain-specific training data, the MFT 226 becomes more accurate at determining semantic meaning from input in that domain.)
Consider Claim 7.
Yu teaches:
7. The image selection device according to claim 1, wherein the at least one processor is configured to extract language features which are features of the text information, extract image features which are features of each image of the image group, and calculate the score of the each image of the image group, based on the image features of the each image of the image group and the language features. (Yu: [0022] FIG. 1 is a block diagram illustrating an example system 100 configured for training a multimodal fusion transformer (MFT) 126 using contrastive loss data 134. In some examples, language modality data 104 and vision modality data 106 are extracted from a multimodal media source 102. Language embeddings 108 are generated from the language modality data 104 and vision embeddings 110 are generated from the vision modality data 106. The language embeddings 108 and the vision embeddings 110 are combined into combined embeddings 124 and provided to the MFT 126 as input. The MFT 126 is transformer model that is configured to generate a language semantic output token 128, a combined semantic output token 130 and a vision semantic output token 132. In other examples, the MFT 126 is configured to generate a combined semantic output token 130 using a combination of modality embeddings based on input modality data of a plurality of modalities and to generate modality-specific semantic output tokens (e.g., the language semantic output token 128 and the vision semantic output token 132) using modality embeddings based on input modality data of specific modalities. The tokens 128, 130, and 132 are used to generate contrastive loss data 134. Parameters and/or other aspects of the MFT 126 are adjusted based on the contrastive loss data 134 to train the MFT 126 to produce semantic output that is focused on the semantic meaning of the combination of the language modality and the vision modality of a multimodal media source, rather than semantic meanings of the language modality and/or vision modality alone. [0029] In an embodiment, the language modality data 104 is tokenized through a tokenizer and then provided to a token embedding layer to generate the text token embeddings 112, which are associated with corresponding position embeddings 114 and a language segment embedding 111 associated with the language modality. The vision modality data 106 is analyzed using an object detection model (e.g., . . . the R-CNN described above) that detects objects present in the image. A RoI bounding box 116 is defined surrounding the entire image without an associated object category and, for each detected object, a category label and RoI bounding box 116 is generated. The category labels are processed using the token embedding layer used to generate the text token embeddings 112, such that the object category embeddings 120 are semantically comparable to the text token embeddings 112. The bounding boxes of the detected objects are included with the RoI bounding boxes 116 in the vision embeddings 110. The visual features of each RoI are then encoded through a feature extractor to generate the feature map embeddings 118. The RoI bounding boxes 116, the feature map embeddings 118, and the object category embeddings 120 are combined with (e.g., . . . added to) a vision segment embedding 115 associated with the vision modality to form the vision embeddings 110. The concatenated pair of the language embeddings 108 and the vision embeddings 110 form the combined embeddings 124, which are provided to the MFT 126 as described herein. [0030] Segment embeddings, such as the language segment embedding 111 and the vision segment embedding 115, are embedding vectors that are specific to modalities. The segment embeddings are used to label associated embeddings to indicate a modality type with which the labeled embeddings are associated.)
Consider Claim 8.
Yu teaches:
8. The image selection device according to claim 1, wherein the at least one processor is configured to perform, using the selected images, machine learning of a machine learning model used in calculation of the score, and select the images again from the image group by sampling based on the distribution of the score which is calculated based on the machine learning model after the machine learning. (Yu: [0046] Additionally, or alternatively, the vision embeddings 210 include RoI bounding boxes 216, feature map embeddings 218, object category embeddings 220, and/or other vision embeddings (e.g., vision segment embedding 115). In some examples, a portion of embedding data associated with one or more of the RoI bounding boxes 216 are masked using RoI masks 244. For instance, in an example, an RoI mask 244 is applied to feature map embeddings 218 and object category embeddings 220 associated with one RoI bounding box of the RoI bounding boxes 216. The output of the MFT 226 is then used with the visual information of the masked embeddings to generate the masked RoI loss 250. The masked RoI loss 250 is used to train the MFT 226 to better predict visual information in a bounding box based on corresponding language tokens and/or other associated information. [0047] For instance, in an example, the masked RoI loss Lroi-cis is represented by the equation 4 below, where fVL is the MFT 226 output and cv denotes the object category tokens or scores detected by the object detection model.
PNG
media_image1.png
44
314
media_image1.png
Greyscale
[0048] Further, in some examples, the MFT 226 is used to generate domain-specific classification loss 252 that is then used to train the MFT 226 to more accurately generate semantic output based on input associated with a particular domain. When training the MFT 226 for domain-specific performance, the training data pairs 236 include or consist entirely of domain-specific training data pairs 236. For instance, in an example where the MFT 226 is being trained to process media sources for content moderation, the training data pairs 236 include instances of media that would be considered for content moderation and indications of whether those instances of media are candidates for moderation based on their content. By iteratively training the MFT 226 using large quantities of domain-specific training data, the MFT 226 becomes more accurate at determining semantic meaning from input in that domain.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1, 3, 5, 9 and 10 are further rejected under 35 U.S.C. 103 as being unpatentable over Yu et al. (US PGPub 20240290081, originally filed on February 28, 2023), in view of Sikka et al. (US PGPub US 2019/0325342 A1, filed on April 12, 2019 hereby referred to as “Sikka”).
Consider Claims 1, 3, 9 and 10
Yu teaches:
1. An image selection device comprising: at least one memory configured to store instructions; and at least one processor configured to execute the instructions to: / 9. An image selection method executed by a computer, comprising: / 10. A non-transitory computer readable storage medium storing a program executed by a computer, the program causing the computer to: (Yu: abstract, A computerized method trains and uses a multimodal fusion transformer (MFT) model for content moderation. Language modality data and vision modality data associated with a multimodal media source is received. Language embeddings are generated from the language modality data and vision embeddings are generated from the vision modality data. Both kinds of embeddings are generated using operations and/or processes that are specific to the associated modalities. The language embeddings and vision embeddings are combined into combined embeddings and the MFT model is used with those combined embeddings to generate a language semantic output token, a vision semantic output token, and a combined semantic output token. Contrastive loss data is generated using the three semantic output tokens and the MFT model is adjusted using that contrastive loss data. After the MFT model is trained sufficiently, it is configured to perform content moderation operations using semantic output tokens. [0022] FIG. 1 is a block diagram illustrating an example system 100 configured for training a multimodal fusion transformer (MFT) 126 using contrastive loss data 134. In some examples, language modality data 104 and vision modality data 106 are extracted from a multimodal media source 102. Language embeddings 108 are generated from the language modality data 104 and vision embeddings 110 are generated from the vision modality data 106. The language embeddings 108 and the vision embeddings 110 are combined into combined embeddings 124 and provided to the MFT 126 as input. The MFT 126 is transformer model that is configured to generate a language semantic output token 128, a combined semantic output token 130 and a vision semantic output token 132. In other examples, the MFT 126 is configured to generate a combined semantic output token 130 using a combination of modality embeddings based on input modality data of a plurality of modalities and to generate modality-specific semantic output tokens (e.g., the language semantic output token 128 and the vision semantic output token 132) using modality embeddings based on input modality data of specific modalities. The tokens 128, 130, and 132 are used to generate contrastive loss data 134. Parameters and/or other aspects of the MFT 126 are adjusted based on the contrastive loss data 134 to train the MFT 126 to produce semantic output that is focused on the semantic meaning of the combination of the language modality and the vision modality of a multimodal media source, rather than semantic meanings of the language modality and/or vision modality alone.)
1. acquire text information specifying an image to be acquired from an image group; / 9. acquiring text information specifying an image to be acquired from an image group; / 10. acquire text information specifying an image to be acquired from an image group; (Yu: [0027] In some examples, the system 100 is configured to generate the language embeddings 108 from the language modality data 104 using trained language model(s) or parts thereof (e.g., an embedding layer of a transformer). For instance, in an example, the generation of the language embeddings 108 includes tokenizing the language modality data 104 into a set of language tokens and then mapping those language tokens to text token embeddings (e.g., text token embedding vectors as generated by embedding layers of a transformer). The language embeddings include text token embeddings 112 and position embeddings 114. The text token embeddings 112 include tokens of semantic data corresponding to specific words, phrases, symbols, or the like in the language modality data 104. The position embeddings 114 include position data that represents the positions of each corresponding text token embeddings 112 relative to each other (e.g., the first token has a position embedding 114 indicating that it is first, before the second token).)
1. calculate a score which represents a degree of match between each image of the image group and the text information; and /9. calculating a score which represents a degree of match between each image of the image group and the text information; / 10. calculate a score which represents a degree of match between each image of the image group and the text information; and (Yu: [0046] Additionally, or alternatively, the vision embeddings 210 include RoI bounding boxes 216, feature map embeddings 218, object category embeddings 220, and/or other vision embeddings (e.g., vision segment embedding 115). In some examples, a portion of embedding data associated with one or more of the RoI bounding boxes 216 are masked using RoI masks 244. For instance, in an example, an RoI mask 244 is applied to feature map embeddings 218 and object category embeddings 220 associated with one RoI bounding box of the RoI bounding boxes 216. The output of the MFT 226 is then used with the visual information of the masked embeddings to generate the masked RoI loss 250. The masked RoI loss 250 is used to train the MFT 226 to better predict visual information in a bounding box based on corresponding language tokens and/or other associated information. [0047] For instance, in an example, the masked RoI loss Lroi-cis is represented by the equation 4 below, where fVL is the MFT 226 output and cv denotes the object category tokens or scores detected by the object detection model.
PNG
media_image1.png
44
314
media_image1.png
Greyscale
)
1. select images from the image group by sampling based on a distribution of the score. /9.and selecting images from the image group by sampling based on a distribution of the score. / 10. select images from the image group by sampling based on a distribution of the score.(Yu: [0029] In an embodiment, the language modality data 104 is tokenized through a tokenizer and then provided to a token embedding layer to generate the text token embeddings 112, which are associated with corresponding position embeddings 114 and a language segment embedding 111 associated with the language modality. The vision modality data 106 is analyzed using an object detection model (e.g., . . . the R-CNN described above) that detects objects present in the image. A RoI bounding box 116 is defined surrounding the entire image without an associated object category and, for each detected object, a category label and RoI bounding box 116 is generated. The category labels are processed using the token embedding layer used to generate the text token embeddings 112, such that the object category embeddings 120 are semantically comparable to the text token embeddings 112. The bounding boxes of the detected objects are included with the RoI bounding boxes 116 in the vision embeddings 110. The visual features of each RoI are then encoded through a feature extractor to generate the feature map embeddings 118. The RoI bounding boxes 116, the feature map embeddings 118, and the object category embeddings 120 are combined with (e.g., . . . added to) a vision segment embedding 115 associated with the vision modality to form the vision embeddings 110. The concatenated pair of the language embeddings 108 and the vision embeddings 110 form the combined embeddings 124, which are provided to the MFT 126 as described herein. [0030] Segment embeddings, such as the language segment embedding 111 and the vision segment embedding 115, are embedding vectors that are specific to modalities. The segment embeddings are used to label associated embeddings to indicate a modality type with which the labeled embeddings are associated.)
Even if Yu does not teach limitations from Claim 3 for: “perform clustering of the image group”
Sikka teaches:
1. An image selection device comprising: at least one memory configured to store instructions; and at least one processor configured to execute the instructions to: / 9. An image selection method executed by a computer, comprising: / 10. A non-transitory computer readable storage medium storing a program executed by a computer, the program causing the computer to: (Sikka: abstract, Embedding multimodal content in a common geometric space includes for each of a plurality of content of the multimodal content, creating a respective, first modality feature vector representative of content of the multimodal content having a first modality using a first machine learning model; for each of a plurality of content of the multimodal content, creating a respective, second modality feature vector representative of content of the multimodal content having a second modality using a second machine learning model; and semantically embedding the respective, first modality feature vectors and the respective, second modality feature vectors in a common geometric space that provides logarithm-like warping of distance space in the geometric space to capture hierarchical relationships between seemingly disparate, embedded modality feature vectors of content in the geometric space; wherein embedded modality feature vectors that are related, across modalities, are closer together in the geometric space than unrelated modality feature vectors.)
1. acquire text information specifying an image to be acquired from an image group; / 9. acquiring text information specifying an image to be acquired from an image group; / 10. acquire text information specifying an image to be acquired from an image group; (Sikka: [0032] Thus, the inventors have discovered a way to better understand, exploit, and organize multimodal content in order to extract semantics and to make high-level sense of the multimodal content. This includes extracting short and long-term events from the multimodal content even from unorganized, but structured content, such as social media content. The embodiments of the present principles described herein allow data in multiple modalities to provide rich information that allows for interpolation of information by providing complementary information, to fill gaps in individual modalities, and to provide performance benefits. The multimodal embeddings provided by the present principles exploit correlations in different modalities and provide explicit vectorial representations for data points without relying on fixed architectures such as late fusion techniques. [0036] FIG. 3 is a method 300 for embedding multimodal content in a common non-Euclidean geometric space according to some embodiments of the present principles. For the sake of brevity, the following examples are based on content with two modalities—text and images. However, the concepts may be expanded to include any number of modalities including, for example, video, audio, etc. In block 302, a first machine learning model is trained using content relating to a first modality. In some embodiments, the first modality may be, for example, text. In general, the model performance improves as the data set size used for training increases. In block 304, a first modality feature vector is created using the first machine learning model from an input that has multimodal content. The first modality feature vector represents a first modality feature of the multimodal content. For example, if the multimodal content is an image with a caption, the first modality feature vector may represent a first modality (text) of the multimodal content (caption portion). In some embodiments, a pre-existing single modality model such as, for example, word2vec (Euclidean space) may be used to provide the first feature vector for text modalities. However, the inventors have found that performance may be increased by retraining the word2vec with vectors from a non-Euclidean space. [0037])
1. calculate a score which represents a degree of match between each image of the image group and the text information; and /9. calculating a score which represents a degree of match between each image of the image group and the text information; / 10. calculate a score which represents a degree of match between each image of the image group and the text information; and (Sikka: [0039] There are no limits to the number of modalities of the content nor number of machine learning models that may be used. Similarly, there are no limits on the numbers of feature vectors included in the embedded multimodal feature vector (e.g., may include associated data from more than two modality sources). In some embodiments, additional information may be infused into the multimodal feature vector during embedding processes and/or may be included as an attribute to the embedded multimodal feature vector. Because the content of the first and second modalities are embedded together, the non-Euclidean common geometric space is an embedding space that represents meaning multiplication of the content from the different modalities while preserving hierarchies of the multimodal content. Meaning multiplication denotes that, for example, when someone posts a meme on social media, the meaning is a combination of the image and its caption—the combination yielding meaning greater than the meaning of the image alone or the meaning of the caption alone. Meaning multiplication is discussed further in the examples that follow.)
1. select images from the image group by sampling based on a distribution of the score. /9.and selecting images from the image group by sampling based on a distribution of the score. / 10. select images from the image group by sampling based on a distribution of the score. (Sikka: [0033] As illustrated in a view 100A of FIG. 1A, the classes in Euclidean space are more tightly packed and are scattered throughout the embedding. As shown in a view 100B of FIG. 1B, using a non-Euclidean space such as, for example, a Poincaré space allows distinct classes to form in broader categories such as “plants” and “land animals.” The hierarchical structure of the data is retained in the structure of the embedding. Thus, the inventors have found that there is a need to devise embedding methods that retain such semantic hierarchies.
[0034] Non-Euclidean embeddings such as, for example, hyperbolic embeddings provide a way to capture distances that grow exponentially through a logarithm-like warping of distance space. As shown in a view 200 of FIG. 2, hyperbolic spaces have tree-like properties that naturally indicate hierarchies and learning can be easily integrated into gradient based optimization. Thus, the top of the hierarchy, such as plant in plant-flowers-roses, is brought much closer to the bottom of the hierarchy. The hyperbolic embeddings provide a way to capture real-world semantic hierarchies while retaining all of the advantages of unsupervised continuous representation that Euclidean embeddings provide. Hyperbolic embeddings do capture real-world semantic hierarchies, even when realized through a simple approximation that is not completely hyperbolic and match or exceed the state of the art in retrieval accuracy. [0035] The hierarchies provided by non-Euclidean embeddings may be used to contract and/or expand a class (e.g., user community) as well. This is particularly useful during clustering (see below for clustering processes). The hierarchy may be traversed upwards or downwards (e.g., from a community of users to a single user or from a community of users to a select group of users, etc.). Similarly, the hierarchy may be leveraged to move from a single user or select group of users to find a more general group of users or a community of users. In some instances, attributes may be passed up or down a hierarchy. For example, a user living in a particular state such as, for example, Maine, also lives in the United States. Other attributes may only pass in one direction as, for example, some users may live on the coast of Maine while other users in Maine may not. Other attributes may be metric in nature and others may be continuously variable such as, for example, incomes.)
3. The image selection device according to claim 1, wherein the at least one processor is configured to perform clustering of the image group and perform the sampling for each cluster generated by the clustering. (Sikka: [0045] The inventors have found that by using a model that jointly learns agent and content embedding, additional information can be extracted with regard to the original poster of the content and/or other agents who appear nearby in the embedding space. The model may also be adjusted such that agents are clustered based on their posted and/or associated content. The image-text joint embedding framework is leveraged to create content to user embeddings. Each training example is a tuple of an image and a list of users who posted the image. The objective is to minimize distance between the image embedding and the user embeddings by altering the ranking loss algorithm. As shown in a view 900 of FIG. 9, the standard loss 902 is extended by adding a ranking loss term 904 for the cluster center vectors 906 along with a clustering loss 908.[ 0053]
In addition to the approach outlined above, some embodiments have a modification which allows learning of a clustering of users in addition to the user embeddings inside the same network. Learning the clusters jointly allows for a better and automatic sharing of information about similar images between user embeddings than what is available explicitly in the dataset. To this end, an additional matrix wc∈Figure US20190325342A1-20191024-P00006 K×D is maintained, where C is the number of clusters, a hyperparameter. Each row of wC, represented by ci,l=1, 2, . . . , K, is the vector representing the cluster center of the lth cluster. Let the clusters of users be denoted by Cl, l=1, 2, . . . , K. In order to learn the cluster centers, the loss function proposed previously is further modified. Two other terms are added into the loss function (Eq. 6) given by:
PNG
media_image2.png
77
642
media_image2.png
Greyscale
[0054] The first term a, Lrank-clusters(.) is the cluster-center analogue of the Lrank-users. In effect, it tries to push the image features closer to the right clusters and farther away from the wrong clusters. The second term LK-means is similar to the K-means loss function which is used to ensure that the cluster centers are indeed representative of the users assigned to that cluster. Since nearest neighbor computation is not a differentiable operation, the cluster assignments cannot be found inside the network. As an approximation, the cluster assignments are found only once every 1000 iterations and optimize the user embeddings and the cluster centers with the fixed cluster assignments.)
It would have been obvious before the effective filing date of the claimed invention was made to one of ordinary skill in the art to modify a method and system for training and using a model for content moderation using a multi-modal fusion transformer as disclosed by Yu with Sikka’s multi-modal content embedment process as suggested by Sikka. The determination of obviousness is predicated upon the following findings: Both references are in the same field of endeavor for multi-modal content representation, and one skilled in the art would have been motivated to modify Yu in this manner in order to enhance the efficiency and accuracy of the overall machine learning process using Sikka’s embedments. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Yu, while the teaching of Sikka continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of improving the overall multi-modal content representation of Yu with the mathematical model and embedment architecture of Sikka. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claim 3.
The combination of Yu and Sikka teaches:
3. The image selection device according to claim 1, wherein the at least one processor is configured to perform clustering of the image group and perform the sampling for each cluster generated by the clustering. (Sikka: [0045] The inventors have found that by using a model that jointly learns agent and content embedding, additional information can be extracted with regard to the original poster of the content and/or other agents who appear nearby in the embedding space. The model may also be adjusted such that agents are clustered based on their posted and/or associated content. The image-text joint embedding framework is leveraged to create content to user embeddings. Each training example is a tuple of an image and a list of users who posted the image. The objective is to minimize distance between the image embedding and the user embeddings by altering the ranking loss algorithm. As shown in a view 900 of FIG. 9, the standard loss 902 is extended by adding a ranking loss term 904 for the cluster center vectors 906 along with a clustering loss 908.[ 0053]
In addition to the approach outlined above, some embodiments have a modification which allows learning of a clustering of users in addition to the user embeddings inside the same network. Learning the clusters jointly allows for a better and automatic sharing of information about similar images between user embeddings than what is available explicitly in the dataset. To this end, an additional matrix wc∈Figure US20190325342A1-20191024-P00006 K×D is maintained, where C is the number of clusters, a hyperparameter. Each row of wC, represented by ci,l=1, 2, . . . , K, is the vector representing the cluster center of the lth cluster. Let the clusters of users be denoted by Cl, l=1, 2, . . . , K. In order to learn the cluster centers, the loss function proposed previously is further modified. Two other terms are added into the loss function (Eq. 6) given by:
PNG
media_image2.png
77
642
media_image2.png
Greyscale
[0054] The first term a, Lrank-clusters(.) is the cluster-center analogue of the Lrank-users. In effect, it tries to push the image features closer to the right clusters and farther away from the wrong clusters. The second term LK-means is similar to the K-means loss function which is used to ensure that the cluster centers are indeed representative of the users assigned to that cluster. Since nearest neighbor computation is not a differentiable operation, the cluster assignments cannot be found inside the network. As an approximation, the cluster assignments are found only once every 1000 iterations and optimize the user embeddings and the cluster centers with the fixed cluster assignments.)
Consider Claim 5.
Claim 5 is rejected for the same reasons as Claims 1 and 3 as presented above.
The combination of Yu and Sikka teaches:
5. The image selection device according to claim 1, wherein the at least one processor is configured to acquire plural pieces of the text information indicating plural categories, and for each image of the image group, normalize the scores among the plural categories. (Sikka: 0040]
In this manner, words and images are transformed into vectors that are embedded into a common non-Euclidean space. Distance between vectors is small when vectors are semantically similar. Hierarchies are determined by the normalization of the embedded vectors. As illustrated in a view 400A of FIG. 4A, Euclidean space does not inherently preserve hierarchies as the content is spread across a single plane. However, as shown in a view 400B of FIG. 4B, non-Euclidean space inherently preserves the natural hierarchies of the content. In one example of an embodiment of the present principles, Microsoft's COCO (Common Objects in Context) database (see, T. Lin, C. L. Zitnick, and P. Doll, “Microsoft COCO: Common Objects in Context,” pp. 1-15) was used for training the model. A contrastive loss function with Riemannian SGD (Stochastic Gradient Descent) was used for the embedding. A Lorentzian space model or Poincaré n-ball space model was used as the common non-Euclidean embedding space. A deep learning convolutional neural network (CNN) 504 with fully connected (FC) and linear layers 506 was used to process the images 502. A bidirectional long short term memory (Bi-LSTM) network 512 was used to process the descriptive text 510 of the image. The results from both processes were then mapped to the non-Euclidean common space 508 as shown in a view 500 of FIG. 5. [0056] Once the embeddings for each user is obtained, user retrieval accuracy can be measured as follows. Compute NU×NU distance matrix using each user Uj for which the closest M closest users can be found. As discussed before, the knowledge of the category to which each user belongs to is known. Thus, the top-M accuracy based on the ground-truth categories can be computed. Normalized discounted cumulative gain (NDCG) is a widely used metric in information retrieval that takes into account the position of the retrieved elements. The formula for calculating NDCG at k i.e., with k retrieved elements (Eq. 7) is given by:
PNG
media_image3.png
207
541
media_image3.png
Greyscale
)
Conclusion
The prior art made of record in form PTO-892 and not relied upon is considered pertinent to applicant's disclosure.
PNG
media_image4.png
144
914
media_image4.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
/Tahmina Ansari/
September 30, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
Prosecution Insights