DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities: on paragraph [00110], “For brevity, descriptions of these components has not been repeated with regard to each figure” should read “For brevity, descriptions of these components have not been repeated with regard to each figure.”
Appropriate correction is required.
Claim Objections
Claims 9, 19 are objected to because of the following informalities:
Regarding claim 9, “The system of claim 8, wherein the domain adaptation process is DAPL" should read “The system of claim 8, wherein the domain adaptation process is Domain Adaptation via Prompt Learning (DAPL).”
Regarding claim 19, “The system of claim 18, wherein the domain adaptation process is DAPL" should read “The system of claim 18, wherein the domain adaptation process is Domain Adaptation via Prompt Learning (DAPL).”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6-9 and 16-19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 6 recites the limitations:
"obtaining a set of known classes corresponding to the known domains"
"and the contrastive learning model is trained with a contrastive objective configured to align corresponding images and text representations of the known classes and the known domains in a shared feature space”
There is insufficient antecedent basis for these limitations in the claim. It is unclear whether “the known domains” is referring to “the known domain statistics” from claim 1. For the purposes of examination, the Examiner has interpreted these instances and all subsequent instances in the dependent claims as the known domain statistics from claim 1.
Claim(s) 7-9 are rejected for at least the same reasons as claim 6 since they depend on claim 6.
Claim(s) 16 recites substantially similar limitations to claim(s) 6, and is/are therefore rejected under the same analysis. Claim(s) 17-19 are rejected for at least the same reasons as claim 16 since they depend on claim 16.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1-20 are rejected under 35 U.S.C. 101 because the claims are directed towards an abstract idea without significantly more.
Regarding Claim 1:
Subject Matter Eligibility Analysis Step 1:
Claim 1 recites a system and is thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 1 recites:
comparing statistics of data samples collected from an edge device against a plurality of known domain statistics to detect a new domain (This limitation is a mental process as it encompasses a human mentally comparing statistics and is thus an evaluation.)
using descriptions generated for the collected data samples to determine a pseudo label associated with the new domain (This limitation is a mental process as it encompasses a human mentally determine a label and is thus an evaluation.)
wherein the pseudo label is generated (This limitation is a mental process as it encompasses a human mentally generate a label and is thus an evaluation.)
and applying a domain adaptation process using prompt learning based on the new domain and on the associated pseudo label to generate new prompts usable with a ML multimodal model for the new domain (This limitation is a mental process as it encompasses a human mentally generating prompts and is thus an evaluation.)
Therefore, claim 1 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 1 further recites additional elements of:
A system comprising: a memory comprising instructions; and a processor communicatively coupled to the memory and configured to execute the instructions, the instructions comprising (This element does not integrate the abstract idea into a practical application because it recites generic computing components on which to perform the abstract idea (see MPEP 2106.05(f)).)
using unsupervised machine learning (ML) (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
and to update the known domain statistics to include statistics of the new domain (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
wherein the multimodal model is trained on text similarity (This element does not integrate the abstract idea into a practical application because it recites the insignificant extra-solution activity of training a model (see MPEP 2106.05(g)).)
Therefore, claim 1 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 1 do not provide significantly more than the abstract
idea itself, taken alone and in combination because
A system comprising: a memory comprising instructions; and a processor communicatively coupled to the memory and configured to execute the instructions, the instructions comprising uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
using unsupervised machine learning (ML) uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
and to update the known domain statistics to include statistics of the new domain uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
wherein the multimodal model is trained on text similarity is the well understood, routine, and conventional activity of iteratively training a model (US 2021/0125108 A1, Metzler et al. page 11, paragraph 0052, “a classifier is trained using a conventional iterative machine learning training process that determines weights for each result list position”).
Therefore, claim 1 is subject-matter ineligible.
Regarding Claim 2:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 2 recites:
The system of claim 1, wherein the descriptions for the collected data samples are generated (This limitation is a mental process as it encompasses a human mentally generating descriptions and is thus an evaluation.)
Therefore, claim 2 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 2 further recites additional elements of:
by applying a plurality of ML image-to-text models to the collected data samples (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
Therefore, claim 2 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 2 do not provide significantly more than the abstract
idea itself, taken alone and in combination because
by applying a plurality of ML image-to-text models to the collected data samples uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 2 is subject-matter ineligible.
Regarding Claim 3:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 3 recites:
and selecting a label for a defined domain that is identified as predominant, to be the associated pseudo label (This limitation is a mental process as it encompasses a human mentally selecting a label and is thus an evaluation.)
wherein the predominant domain is identified based on a frequency of domain occurrence in the generated descriptions (This limitation is a mental process as it encompasses a human mentally identifying a domain and is thus an evaluation.)
Therefore, claim 3 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 3 further recites additional elements of:
wherein using descriptions generated for the collected data samples further comprises: obtaining a general large language model (LLM) and a text prompt configured to analyze image descriptions (This element does not integrate the abstract idea into a practical application because it recites the insignificant extra-solution activity of receiving data (see MPEP 2106.05(g)).)
applying the LLM and the text prompt to the descriptions generated by the image-to-text models for the collected data samples to define domains for each image (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
Therefore, claim 3 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 3 do not provide significantly more than the abstract
idea itself, taken alone and in combination because
wherein using descriptions generated for the collected data samples further comprises: obtaining a general large language model (LLM) and a text prompt configured to analyze image descriptions is the well understood, routine, and conventional activity of "transmitting or receiving data over a network" (see MPEP 2106.05(d)(II); OIP Techs., Inc., V. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
applying the LLM and the text prompt to the descriptions generated by the image-to-text models for the collected data samples to define domains for each image uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 3 is subject-matter ineligible.
Regarding Claim 4:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 4 recites the same abstract idea as claim 1. Therefore, claim 4 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 4 further recites additional elements of:
wherein the domain adaptation process is unsupervised (This element does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception (see MPEP 2106.05(h)).)
Therefore, claim 4 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 4 do not provide significantly more than the abstract
idea itself, taken alone and in combination because
wherein the domain adaptation process is unsupervised specifies a technological environment to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(h)).
Therefore, claim 4 is subject-matter ineligible.
Regarding Claim 5:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 5 recites the same abstract idea as claim 4. Therefore, claim 5 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 5 further recites additional elements of:
wherein the multimodal model trained on text similarity is a ML contrastive learning model (This element does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception (see MPEP 2106.05(h)).)
Therefore, claim 5 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 5 do not provide significantly more than the abstract
idea itself, taken alone and in combination because
wherein the multimodal model trained on text similarity is a ML contrastive learning model specifies a technological environment to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(h)).
Therefore, claim 5 is subject-matter ineligible.
Regarding Claim 6:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 6 recites:
and wherein the new prompts for the new domain are generated (This limitation is a mental process as it encompasses a human mentally generating prompts and is thus an evaluation.)
Therefore, claim 6 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 6 further recites additional elements of:
wherein the domain adaptation process further comprises: obtaining a set of known classes corresponding to the known domains (This element does not integrate the abstract idea into a practical application because it recites the insignificant extra-solution activity of receiving data (see MPEP 2106.05(g)).)
using the contrastive learning model (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
and the contrastive learning model is trained with a contrastive objective configured to align corresponding images and text representations of the known classes and the known domains in a shared feature space (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
Therefore, claim 6 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 6 do not provide significantly more than the abstract
idea itself, taken alone and in combination because
wherein the domain adaptation process further comprises: obtaining a set of known classes corresponding to the known domains is the well understood, routine, and conventional activity of "transmitting or receiving data over a network" (see MPEP 2106.05(d)(II); OIP Techs., Inc., V. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network)).
using the contrastive learning model uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
and the contrastive learning model is trained with a contrastive objective configured to align corresponding images and text representations of the known classes and the known domains in a shared feature space uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
Therefore, claim 6 is subject-matter ineligible.
Regarding Claim 7:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 7 recites:
wherein the contrastive objective is further configured to maximize a similarity measure between a given image and a particular corresponding text representation as a positive pair, and to minimize the similarity measure between the given image and other text representations that are determined to be irrelevant as negative pairs (This limitation is a mental process as it encompasses a human mentally maximizing and minimizing a similarity measure and is thus an evaluation.)
Therefore, claim 7 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 7 does not further recite any additional elements. Therefore, claim 7 is not
integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 7 does not provide significantly more than
the abstract idea itself, taken alone and in combination. Therefore, claim 7 is subject-matter
ineligible.
Regarding Claim 8:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 8 recites:
wherein the similarity measure is a cosine similarity (This limitation is a mental process as it is further modifying the similarity measure defined in claim 7.)
Therefore, claim 8 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 8 does not further recite any additional elements. Therefore, claim 8 is not
integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
Since there are no additional elements, claim 8 does not provide significantly more than
the abstract idea itself, taken alone and in combination. Therefore, claim 8 is subject-matter
ineligible.
Regarding Claim 9:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 9 recites the same abstract idea as claim 8. Therefore, claim 9 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 9 further recites additional elements of:
wherein the domain adaptation process is DAPL (This element does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception (see MPEP 2106.05(h)).)
Therefore, claim 9 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 9 do not provide significantly more than the abstract
idea itself, taken alone and in combination because
wherein the domain adaptation process is DAPL specifies a technological environment to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(h)).
Therefore, claim 9 is subject-matter ineligible.
Regarding Claim 10:
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 10 recites the same abstract idea as claim 1. Therefore, claim 10 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 10 further recites additional elements of:
wherein the edge device is a camera in a connected car (This element does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception (see MPEP 2106.05(h)).)
and the data samples include image data from the camera (This element does not integrate the abstract idea into a practical application because it recites a technological environment in which to apply a judicial exception (see MPEP 2106.05(h)).)
Therefore, claim 10 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 10 do not provide significantly more than the abstract
idea itself, taken alone and in combination because
wherein the edge device is a camera in a connected car specifies a technological environment to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(h)).
and the data samples include image data from the camera specifies a technological environment to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(h)).
Therefore, claim 10 is subject-matter ineligible.
Regarding Claim 11:
Subject Matter Eligibility Analysis Step 1:
Claim 11 recites a method and is thus a process, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 11 recites:
A method comprising: comparing statistics of data samples collected from an edge device against a plurality of known domain statistics to detect a new domain (This limitation is a mental process as it encompasses a human mentally comparing statistics and is thus an evaluation.)
using descriptions generated for the collected data samples to determine a pseudo label associated with the new domain (This limitation is a mental process as it encompasses a human mentally determining a label and is thus an evaluation.)
wherein the pseudo label is generated (This limitation is a mental process as it encompasses a human mentally generating a label and is thus an evaluation.)
and applying a domain adaptation process using prompt learning based on the new domain and on the associated pseudo label to generate new prompts usable with a ML multimodal model for the new domain (This limitation is a mental process as it encompasses a human mentally generating new prompts and is thus an evaluation.)
Therefore, claim 11 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 11 further recites additional elements of:
using unsupervised ML (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
and to update the known domain statistics to include statistics of the new domain (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
wherein the multimodal model is trained on text similarity (This element does not integrate the abstract idea into a practical application because it recites the insignificant extra-solution activity of training a model (see MPEP 2106.05(g)).)
Therefore, claim 11 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 11 do not provide significantly more than the abstract
idea itself, taken alone and in combination because
using unsupervised ML uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
and to update the known domain statistics to include statistics of the new domain uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
wherein the multimodal model is trained on text similarity is the well understood, routine, and conventional activity of iteratively training a model (US 2021/0125108 A1, Metzler et al. page 11, paragraph 0052, “a classifier is trained using a conventional iterative machine learning training process that determines weights for each result list position”).
Therefore, claim 11 is subject-matter ineligible.
Claim(s) 12 recites substantially similar limitations to claim(s) 2, and is/are therefore rejected under the same analysis.
Claim(s) 13 recites substantially similar limitations to claim(s) 3, and is/are therefore rejected under the same analysis.
Claim(s) 14 recites substantially similar limitations to claim(s) 4, and is/are therefore rejected under the same analysis.
Claim(s) 15 recites substantially similar limitations to claim(s) 5, and is/are therefore rejected under the same analysis.
Claim(s) 16 recites substantially similar limitations to claim(s) 6, and is/are therefore rejected under the same analysis.
Claim(s) 17 recites substantially similar limitations to claim(s) 7, and is/are therefore rejected under the same analysis.
Claim(s) 18 recites substantially similar limitations to claim(s) 8, and is/are therefore rejected under the same analysis.
Claim(s) 19 recites substantially similar limitations to claim(s) 9, and is/are therefore rejected under the same analysis.
Regarding Claim 20:
Subject Matter Eligibility Analysis Step 1:
Claim 20 recites a storage medium and is thus a machine, one of the four statutory categories of patentable subject matter.
Subject Matter Eligibility Analysis Step 2A Prong 1:
Claim 20 recites:
comparing statistics of data samples collected from an edge device against a plurality of known domain statistics to detect a new domain (This limitation is a mental process as it encompasses a human mentally comparing statistics and is thus an evaluation.)
using descriptions generated for the collected data samples to determine a pseudo label associated with the new domain (This limitation is a mental process as it encompasses a human mentally determining a label and is thus an evaluation.)
wherein the pseudo label is generated (This limitation is a mental process as it encompasses a human mentally generating a label and is thus an evaluation.)
and applying a domain adaptation process using prompt learning based on the new domain and on the associated pseudo label to generate new prompts usable with a ML multimodal model for the new domain (This limitation is a mental process as it encompasses a human mentally generating prompts and is thus an evaluation.)
Therefore, claim 20 recites an abstract idea.
Subject Matter Eligibility Analysis Step 2A Prong 2:
Claim 20 further recites additional elements of:
A non-transitory processor-readable storage medium having stored thereon program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps (This element does not integrate the abstract idea into a practical application because it recites generic computing components on which to perform the abstract idea (see MPEP 2106.05(f)).)
using unsupervised ML (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
and to update the known domain statistics to include statistics of the new domain (This element does not integrate the abstract idea into a practical application because it amounts to mere “apply it on a computer” (see MPEP 2106.05(f)).)
wherein the multimodal model is trained on text similarity (This element does not integrate the abstract idea into a practical application because it recites the insignificant extra-solution activity of training a model (see MPEP 2106.05(g)).)
Therefore, claim 20 is not integrated into a practical application.
Subject Matter Eligibility Analysis Step 2B:
The additional elements of claim 20 do not provide significantly more than the abstract
idea itself, taken alone and in combination because
A non-transitory processor-readable storage medium having stored thereon program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
using unsupervised ML uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
and to update the known domain statistics to include statistics of the new domain uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f)).
wherein the multimodal model is trained on text similarity is the well understood, routine, and conventional activity of iteratively training a model (US 2021/0125108 A1, Metzler et al. page 11, paragraph 0052, “a classifier is trained using a conventional iterative machine learning training process that determines weights for each result list position”).
Therefore, claim 20 is subject-matter ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1, 4-9, 11, 14-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ge et al. (“Domain Adaptation via Prompt Learning”) in view of da Silva et al. (US 20240078409 A1).
Regarding claim 1, Ge teaches
comparing statistics of data samples…against a plurality of known domain statistics to detect a new domain (Ge, page 3, “Given a set of labeled source images Ds = {(xs i, ys i )}Ns i=1 and a set of unlabeled target images Du = {(xu i )}Nu i=1, we adopt a model trained from a source domain to a target domain. Here, Ns and Nu denote the scale of source domain dataset Ds and target domain dataset Du, respectively.” Examiner notes the data samples are the unlabeled target images, the known domain statistics are the labeled source images, and the new domain is the target domain.)
using descriptions generated for the collected data samples to determine a pseudo label associated with the new domain (Ge, page 3, “We adopt CLIP [42] as our backbone.” Ge, page 3, “CLIP [42] is trained with image-text pairs in a contrastive manner. Each input text describes a category in the format of “a photo of a [CLASS]” ([CLASS] is the class token). A positive pair is an image xi with its corresponding text ti describing the category of xi.” Ge, page 5, “To further exploit the unlabeled data, We generate pseudo labels on the target domain.” Examiner notes the descriptions generated for the collected data samples associated with the new domain are the corresponding texts of each image when training the model, which are therefore used to generate the pseudo labels for the unlabeled data associated with the new domain.)
wherein the pseudo label is generated using unsupervised machine learning (ML); and applying a domain adaptation process using prompt learning based on the new domain and on the associated pseudo label to generate new prompts usable with a ML multimodal model for the new domain (Ge, page 2, “We propose Domain Adaptation via Prompt Learning (DAPL) for unsupervised domain adaptation.” Ge, page 3, “Figure 2. Example prompt structure. Our proposed prompt consists of three parts: (a) Domain-specific prompt; (b) Domain-agnostic prompt; (c) Class label. The first two parts are continuous and learned from data. The words shown here are for illustrative purposes.” Ge, page 3, Fig. 2. Examiner notes the method DAPrompt is unsupervised, and generates pseudo labels for the target domain using CLIP, a machine learning model. Examiner further notes prompts are generated for the target domain, and prompts are also based on domain specific features, according to the prompt section of Fig. 2.)
wherein the multimodal model is trained on text similarity (Ge, page 3, “CLIP [42] is trained with image-text pairs in a contrastive manner. Each input text describes a category in the format of “a photo of a [CLASS]” ([CLASS] is the class token). A positive pair is an image xi with its corresponding text ti describing the category of xi. A negative pair is an image xi with an irrelevant description tj,j= i in the mini-batch. The training objective is to maximize the cosine similarity of positive pairs and minimize the cosine similarity of negative pairs. The contrastive learning objective aligns the image and text representation in the same feature space.” Ge, page 4, Fig. 3.)
Ge does not, but da Silva teaches
A system comprising: a memory comprising instructions; and a processor communicatively coupled to the memory and configured to execute the instructions, the instructions comprising (da Silva, paragraph 0065, “The embodiments disclosed herein may include the use of a special purpose or general-purpose computer including various computer hardware or software modules, as discussed in greater detail below. A computer may include a processor and computer storage media carrying instructions that, when executed by the processor and/or caused to be executed by the processor, perform any one or more of the methods disclosed herein, or any part(s) of any method disclosed.”)
Ge teaches a method for unsupervised domain adaptation using prompt learning using a specialized machine learning model. da Silva teaches a computer medium with instructions to run a method for adversarial domain adaption using a specialized machine learning model. Ge and da Silva both teach domain adaption on unlabeled data using a machine learning model and thus are considered analogous to the claimed invention. It would have been obvious to one with ordinary skill in the art to modify Ge to implement the aforementioned method as instructions stored in memory to be executed by a processor like da Silva as this would be applying a known technique to a known device to yield the predictable result of running software instructions on a computer medium successfully (see MPEP 2143(d)).
da Silva further teaches
comparing statistics of data samples collected from an edge device (da Silva, paragraph 0017, “In general, some embodiments employ a mechanism to perform a federated domain adaptation inside an entity domain, such as a customer warehouse for example, that may take advantage of the amount of available data that has been collected at an edge node defined by an edge device such as a forklift or AMR (autonomous mobile robot).”)
and to update the known domain statistics to include statistics of the new domain (da Silva, paragraph 0043, “After the verification, if the divergence between the original dataset and the collected dataset is higher than a threshold th, then it may be concluded that the respective domains which with those datasets are associated have diverged, that is, changed, and the ML model adaptation procedure may be started to accommodate the change in the domain. Embodiments may also update the current d.sub.0 (initial dataset) with samples from the new domain d.sub.c, for domain verification. The model adaptation procedure may run with the original dataset, as can be seen in the next section.” Examiner notes the known domain statistics is the initial dataset.)
Ge and da Silva both teach both teach domain adaption on unlabeled data using a machine learning model and thus are analogous to the claimed invention. It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the application to have modified Ge to use data collected from an edge device and update the current domain with data from the new domain like da Silva. Doing so would have been advantageous because “keeping event detection models updated using only the unlabeled data collected in the edge device operating domain” (da Silva, paragraph 0027) can be performed “without looking at or exposing that unlabeled data, that is, keeping the data private since companies do not want to share their data with other companies” (da Silva, paragraph 0027).
Regarding claim 4, Ge in view of da Silva teaches the system of claim 1. Ge further teaches
wherein the domain adaptation process is unsupervised (Ge, page 2, “We propose Domain Adaptation via Prompt Learning (DAPL) for unsupervised domain adaptation.”)
Regarding claim 5, Ge in view of da Silva teaches the system of claim 4. Ge further teaches
wherein the multimodal model trained on text similarity is a ML contrastive learning model (Ge, page 2, “We adopt Contrastive Language Image Pre training (CLIP) [42] as our backbone to facilitate prompt learning and contrastive learning.”)
Regarding claim 6, Ge in view of da Silva teaches the system of claim 5. Ge further teaches
wherein the domain adaptation process further comprises: obtaining a set of known classes corresponding to the known domains (Ge, page 3, “Given a set of labeled source images Ds = {(xs i, ys i )}Ns i=1 and a set of unlabeled target images Du = {(xu i )}Nu i=1, we adopt a model trained from a source domain to a target domain.” Ge, page 2, “The prompt consists of three parts: domain-agnostic context, domain specific context, and class label (token). Each image corresponds to a ground truth class through the class label of prompt. For example, an image that shows “an art work of a dog” could correspond to the prompt “An image of a painting Dog”. The domain-agnostic context represents general task information and is shared among all images. The domain-specific context represents domain information and is shared in each domain. The class label distinguishes different categories.” Ge, page 3, Fig. 2. Examiner notes the known domains are the labeled source images and the known classes are the class labels corresponding to each of the labeled source images.)
and wherein the new prompts for the new domain are generated using the contrastive learning model (Ge, page 1, “We introduce the prompt tuning framework for domain adaptation. Top: conventional domain adaptation methods aim to remove domain-specific information via domain alignment or adversarial loss. This could lead to distorted feature representation when the manifold structures under lying the data distributions are complex [3]. Bottom: Our method preserves domain information and tunes a prompt for each domain. Our model learns with a contrastive objective.” Examiner notes the method used for creating prompts involves a model that learns with a contrastive objective, making the model a contrastive learning model.)
and the contrastive learning model is trained with a contrastive objective configured to align corresponding images and text representations of the known classes and the known domains in a shared feature space (Ge, page 1, “our method preserves domain information and tunes a set of prompt for each domain. Our model learns with a contrastive objective.” Ge, page 3, “CLIP [42] is trained with image-text pairs in a contrastive manner. Each input text describes a category in the format of “a photo of a [CLASS]” ([CLASS] is the class token).” Ge, page 3, “The contrastive learning objective aligns the image and text representation in the same feature space.”)
Regarding claim 7, Ge in view of da Silva teaches the system of claim 6. Ge further teaches
wherein the contrastive objective is further configured to maximize a similarity measure between a given image and a particular corresponding text representation as a positive pair, and to minimize the similarity measure between the given image and other text representations that are determined to be irrelevant as negative pairs (Ge, page 3, “CLIP [42] is trained with image-text pairs in a contrastive manner. Each input text describes a category in the format of “a photo of a [CLASS]” ([CLASS] is the class token). A positive pair is an image xi with its corresponding text ti describing the category of xi. A negative pair is an image xi with an irrelevant description tj,j= i in the mini-batch. The training objective is to maximize the cosine similarity of positive pairs and minimize the cosine similarity of negative pairs. The contrastive learning objective aligns the image and text representation in the same feature space.”)
Regarding claim 8, Ge in view of da Silva teaches the system of claim 7. Ge further teaches
wherein the similarity measure is a cosine similarity (Ge, page 3, “The training objective is to maximize the cosine similarity of positive pairs and minimize the cosine similarity of negative pairs.”)
Regarding claim 9, Ge in view of da Silva teaches the system of claim 8. Ge further teaches
wherein the domain adaptation process is DAPL (Ge, page 2, “we propose domain adaptation via prompt learning (DAPL).”)
Regarding claim 11, Ge teaches
A method comprising: comparing statistics of data samples…against a plurality of known domain statistics to detect a new domain (Ge, page 3, “Given a set of labeled source images Ds = {(xs i, ys i )}Ns i=1 and a set of unlabeled target images Du = {(xu i )}Nu i=1, we adopt a model trained from a source domain to a target domain. Here, Ns and Nu denote the scale of source domain dataset Ds and target domain dataset Du, respectively.” Examiner notes the data samples are the unlabeled target images, the known domain statistics are the labeled source images, and the new domain is the target domain.)
using descriptions generated for the collected data samples to determine a pseudo label associated with the new domain (Ge, page 3, “We adopt CLIP [42] as our backbone.” Ge, page 3, “CLIP [42] is trained with image-text pairs in a contrastive manner. Each input text describes a category in the format of “a photo of a [CLASS]” ([CLASS] is the class token). A positive pair is an image xi with its corresponding text ti describing the category of xi.” Ge, page 5, “To further exploit the unlabeled data, We generate pseudo labels on the target domain.” Examiner notes the descriptions generated for the collected data samples associated with the new domain are the corresponding texts of each image when training the model, which are therefore used to generate the pseudo labels for the unlabeled data associated with the new domain.)
wherein the pseudo label is generated using unsupervised machine learning (ML); and applying a domain adaptation process using prompt learning based on the new domain and on the associated pseudo label to generate new prompts usable with a ML multimodal model for the new domain (Ge, page 2, “We propose Domain Adaptation via Prompt Learning (DAPL) for unsupervised domain adaptation.” Ge, page 3, “Figure 2. Example prompt structure. Our proposed prompt consists of three parts: (a) Domain-specific prompt; (b) Domain-agnostic prompt; (c) Class label. The first two parts are continuous and learned from data. The words shown here are for illustrative purposes.” Ge, page 3, Fig. 2. Examiner notes the method DAPrompt is unsupervised, and generates pseudo labels for the target domain using CLIP, a machine learning model. Examiner further notes prompts are generated for the target domain, and prompts are also based on domain specific features, according to the prompt section of Fig. 2.)
wherein the multimodal model is trained on text similarity (Ge, page 3, “CLIP [42] is trained with image-text pairs in a contrastive manner. Each input text describes a category in the format of “a photo of a [CLASS]” ([CLASS] is the class token). A positive pair is an image xi with its corresponding text ti describing the category of xi. A negative pair is an image xi with an irrelevant description tj,j= i in the mini-batch. The training objective is to maximize the cosine similarity of positive pairs and minimize the cosine similarity of negative pairs. The contrastive learning objective aligns the image and text representation in the same feature space.” Ge, page 4, Fig. 3.)
Ge does not, but da Silva teaches
comparing statistics of data samples collected from an edge device (da Silva, paragraph 0017, “In general, some embodiments employ a mechanism to perform a federated domain adaptation inside an entity domain, such as a customer warehouse for example, that may take advantage of the amount of available data that has been collected at an edge node defined by an edge device such as a forklift or AMR (autonomous mobile robot).”)
and to update the known domain statistics to include statistics of the new domain (da Silva, paragraph 0043, “After the verification, if the divergence between the original dataset and the collected dataset is higher than a threshold th, then it may be concluded that the respective domains which with those datasets are associated have diverged, that is, changed, and the ML model adaptation procedure may be started to accommodate the change in the domain. Embodiments may also update the current d.sub.0 (initial dataset) with samples from the new domain d.sub.c, for domain verification. The model adaptation procedure may run with the original dataset, as can be seen in the next section.” Examiner notes the known domain statistics is the initial dataset.)
Ge and da Silva both teach both teach domain adaption on unlabeled data using a machine learning model and thus are analogous to the claimed invention. It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the application to have modified Ge to use data collected from an edge device and update the current domain with data from the new domain like da Silva. Doing so would have been advantageous because “keeping event detection models updated using only the unlabeled data collected in the edge device operating domain” (da Silva, paragraph 0027) can be performed “without looking at or exposing that unlabeled data, that is, keeping the data private since companies do not want to share their data with other companies” (da Silva, paragraph 0027).
Regarding claim 14, Ge in view of da Silva teaches the method of claim 11. Ge further teaches
wherein the domain adaptation process is unsupervised (Ge, page 2, “We propose Domain Adaptation via Prompt Learning (DAPL) for unsupervised domain adaptation.”)
Regarding claim 15, Ge in view of da Silva teaches the method of claim 14. Ge further teaches
wherein the multimodal model trained on text similarity is a ML contrastive learning model (Ge, page 2, “We adopt Contrastive Language Image Pre training (CLIP) [42] as our backbone to facilitate prompt learning and contrastive learning.”)
Regarding claim 16, Ge in view of da Silva teaches the method of claim 15. Ge further teaches
wherein the domain adaptation process further comprises: obtaining a set of known classes corresponding to the known domains (Ge, page 3, “Given a set of labeled source images Ds = {(xs i, ys i )}Ns i=1 and a set of unlabeled target images Du = {(xu i )}Nu i=1, we adopt a model trained from a source domain to a target domain.” Ge, page 2, “The prompt consists of three parts: domain-agnostic context, domain specific context, and class label (token). Each image corresponds to a ground truth class through the class label of prompt. For example, an image that shows “an art work of a dog” could correspond to the prompt “An image of a painting Dog”. The domain-agnostic context represents general task information and is shared among all images. The domain-specific context represents domain information and is shared in each domain. The class label distinguishes different categories.” Ge, page 3, Fig. 2. Examiner notes the known domains are the labeled source images and the known classes are the class labels corresponding to each of the labeled source images.)
and wherein the new prompts for the new domain are generated using the contrastive learning model (Ge, page 1, “We introduce the prompt tuning framework for domain adaptation. Top: conventional domain adaptation methods aim to remove domain-specific information via domain alignment or adversarial loss. This could lead to distorted feature representation when the manifold structures under lying the data distributions are complex [3]. Bottom: Our method preserves domain information and tunes a prompt for each domain. Our model learns with a contrastive objective.” Examiner notes the method used for creating prompts involves a model that learns with a contrastive objective, making the model a contrastive learning model.)
and the contrastive learning model is trained with a contrastive objective configured to align corresponding images and text representations of the known classes and the known domains in a shared feature space (Ge, page 1, “our method preserves domain information and tunes a set of prompt for each domain. Our model learns with a contrastive objective.” Ge, page 3, “CLIP [42] is trained with image-text pairs in a contrastive manner. Each input text describes a category in the format of “a photo of a [CLASS]” ([CLASS] is the class token).” Ge, page 3, “The contrastive learning objective aligns the image and text representation in the same feature space.”)
Regarding claim 17, Ge in view of da Silva teaches the method of claim 16. Ge further teaches
wherein the contrastive objective is further configured to maximize a similarity measure between a given image and a particular corresponding text representation as a positive pair, and to minimize the similarity measure between the given image and other text representations that are determined to be irrelevant as negative pairs (Ge, page 3, “CLIP [42] is trained with image-text pairs in a contrastive manner. Each input text describes a category in the format of “a photo of a [CLASS]” ([CLASS] is the class token). A positive pair is an image xi with its corresponding text ti describing the category of xi. A negative pair is an image xi with an irrelevant description tj,j= i in the mini-batch. The training objective is to maximize the cosine similarity of positive pairs and minimize the cosine similarity of negative pairs. The contrastive learning objective aligns the image and text representation in the same feature space.”)
Regarding claim 18, Ge in view of da Silva teaches the method of claim 17. Ge further teaches
wherein the similarity measure is a cosine similarity (Ge, page 3, “The training objective is to maximize the cosine similarity of positive pairs and minimize the cosine similarity of negative pairs.”)
Regarding claim 19, Ge in view of da Silva teaches the method of claim 18. Ge further teaches
wherein the domain adaptation process is DAPL (Ge, page 2, “we propose domain adaptation via prompt learning (DAPL).”)
Regarding claim 20, Ge teaches
comparing statistics of data samples…against a plurality of known domain statistics to detect a new domain (Ge, page 3, “Given a set of labeled source images Ds = {(xs i, ys i )}Ns i=1 and a set of unlabeled target images Du = {(xu i )}Nu i=1, we adopt a model trained from a source domain to a target domain. Here, Ns and Nu denote the scale of source domain dataset Ds and target domain dataset Du, respectively.” Examiner notes the data samples are the unlabeled target images, the known domain statistics are the labeled source images, and the new domain is the target domain.)
using descriptions generated for the collected data samples to determine a pseudo label associated with the new domain (Ge, page 3, “We adopt CLIP [42] as our backbone.” Ge, page 3, “CLIP [42] is trained with image-text pairs in a contrastive manner. Each input text describes a category in the format of “a photo of a [CLASS]” ([CLASS] is the class token). A positive pair is an image xi with its corresponding text ti describing the category of xi.” Ge, page 5, “To further exploit the unlabeled data, We generate pseudo labels on the target domain.” Examiner notes the descriptions generated for the collected data samples associated with the new domain are the corresponding texts of each image when training the model, which are therefore used to generate the pseudo labels for the unlabeled data associated with the new domain.)
wherein the pseudo label is generated using unsupervised machine learning (ML); and applying a domain adaptation process using prompt learning based on the new domain and on the associated pseudo label to generate new prompts usable with a ML multimodal model for the new domain (Ge, page 2, “We propose Domain Adaptation via Prompt Learning (DAPL) for unsupervised domain adaptation.” Ge, page 3, “Figure 2. Example prompt structure. Our proposed prompt consists of three parts: (a) Domain-specific prompt; (b) Domain-agnostic prompt; (c) Class label. The first two parts are continuous and learned from data. The words shown here are for illustrative purposes.” Ge, page 3, Fig. 2. Examiner notes the method DAPrompt is unsupervised, and generates pseudo labels for the target domain using CLIP, a machine learning model. Examiner further notes prompts are generated for the target domain, and prompts are also based on domain specific features, according to the prompt section of Fig. 2.)
wherein the multimodal model is trained on text similarity (Ge, page 3, “CLIP [42] is trained with image-text pairs in a contrastive manner. Each input text describes a category in the format of “a photo of a [CLASS]” ([CLASS] is the class token). A positive pair is an image xi with its corresponding text ti describing the category of xi. A negative pair is an image xi with an irrelevant description tj,j= i in the mini-batch. The training objective is to maximize the cosine similarity of positive pairs and minimize the cosine similarity of negative pairs. The contrastive learning objective aligns the image and text representation in the same feature space.” Ge, page 4, Fig. 3.)
Ge does not, but da Silva teaches
A non-transitory processor-readable storage medium having stored thereon program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps (da Silva, paragraph 0065, “The embodiments disclosed herein may include the use of a special purpose or general-purpose computer including various computer hardware or software modules, as discussed in greater detail below. A computer may include a processor and computer storage media carrying instructions that, when executed by the processor and/or caused to be executed by the processor, perform any one or more of the methods disclosed herein, or any part(s) of any method disclosed.”)
Ge teaches a method for unsupervised domain adaptation using prompt learning using a specialized machine learning model. da Silva teaches a computer medium with instructions to run a method for adversarial domain adaption using a specialized machine learning model. Ge and da Silva both teach domain adaption on unlabeled data using a machine learning model and thus are considered analogous to the claimed invention. It would have been obvious to one with ordinary skill in the art to modify Ge to implement the aforementioned method as instructions stored in memory to be executed by a processor like da Silva as this would be applying a known technique to a known device to yield the predictable result of running software instructions on a computer medium successfully (see MPEP 2143(d)).
da Silva further teaches
comparing statistics of data samples collected from an edge device (da Silva, paragraph 0017, “In general, some embodiments employ a mechanism to perform a federated domain adaptation inside an entity domain, such as a customer warehouse for example, that may take advantage of the amount of available data that has been collected at an edge node defined by an edge device such as a forklift or AMR (autonomous mobile robot).”)
and to update the known domain statistics to include statistics of the new domain (da Silva, paragraph 0043, “After the verification, if the divergence between the original dataset and the collected dataset is higher than a threshold th, then it may be concluded that the respective domains which with those datasets are associated have diverged, that is, changed, and the ML model adaptation procedure may be started to accommodate the change in the domain. Embodiments may also update the current d.sub.0 (initial dataset) with samples from the new domain d.sub.c, for domain verification. The model adaptation procedure may run with the original dataset, as can be seen in the next section.” Examiner notes the known domain statistics is the initial dataset.)
Ge and da Silva both teach both teach domain adaption on unlabeled data using a machine learning model and thus are analogous to the claimed invention. It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the application to have modified Ge to use data collected from an edge device and update the current domain with data from the new domain like da Silva. Doing so would have been advantageous because “keeping event detection models updated using only the unlabeled data collected in the edge device operating domain” (da Silva, paragraph 0027) can be performed “without looking at or exposing that unlabeled data, that is, keeping the data private since companies do not want to share their data with other companies” (da Silva, paragraph 0027).
Claim(s) 2-3, 12-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ge in view of da Silva and further in view of Menon et al. (“Visual Classification via Description from Large Language Models”).
Regarding claim 2, Ge in view of da Silva teach the system of claim 1.
Ge in view of da Silva does not, but Menon teaches
wherein the descriptions for the collected data samples are generated by applying a plurality of ML image-to-text models to the collected data samples (Menon, page 2, “We instead mine large language models to automatically build descriptors, and perform recognition by comparing to the category descriptors.” Menon, page 3, “We use vision-language models to visually ground the natural language descriptors generated by the large language models” Examiner notes the descriptions are the descriptors.)
Ge, da Silva, and Menon all teach classifying unlabeled data using a specialized machine learning model and thus are analogous to the claimed invention. It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the application to have modified Ge and da Silva to use a plurality of ML image-to-text models to generate descriptions for the data like Menon. Doing so would have been advantageous because the method “improves accuracy across datasets and distribution shifts, achieving a ∼ 4-5% increase on top-1 ImageNet accuracy” (Menon, page 2).
Regarding claim 3, Ge in view of da Silva in further view of Genon teaches the system of claim 2. Genon further teaches
wherein using descriptions generated for the collected data samples further comprises: obtaining a general large language model (LLM) and a text prompt configured to analyze image descriptions (Menon, page 3, “We propose to automatically construct this set by prompting a large language model, such as GPT-3, to describe the visual features that distinguish that object category in a photograph. We prompt the language model with the input: Q: What are useful features for distinguishing a {category name} in a photo? A: There are several useful visual features to tell there is a {category name} in a photo: where {category name} is substituted for a given c.”)
applying the LLM and the text prompt to the descriptions generated by the image-to-text models for the collected data samples to define domains for each image (Menon, page 3, “We propose to automatically construct this set by prompting a large language model, such as GPT-3, to describe the visual features that distinguish that object category in a photograph. We prompt the language model with the input: Q: What are useful features for distinguishing a {category name} in a photo? A: There are several useful visual features to tell there is a {category name} in a photo: where {category name} is substituted for a given c.” Menon, page 5, “In contrast, our approach acquires visual descriptions from a large language model, which allows it to build new classifiers for categories that φ has not encountered yet.” Menon, page 6, “Even though these categories are relatively new, GPT3 is able to build descriptors for them because enough people on the Internet have visually described them. By combining these descriptors together, the model can recognize the new category.” Examiner notes that GPT3 is the LLM which uses the given text prompt, the descriptions are the descriptors, the domains are the categories, and the LLM can define new categories based on the descriptors.)
and selecting a label for a defined domain that is identified as predominant, to be the associated pseudo label, wherein the predominant domain is identified based on a frequency of domain occurrence in the generated descriptions (“Menon, page 3, “we estimate a score for category c through the additive decomposition:
PNG
media_image1.png
45
203
media_image1.png
Greyscale
where D(c) is the set of descriptors for the category c and φ(d,x) is the log probability that descriptor d pertains to the image x. Our approach will represent the descriptors d also through a natural language sentence; we explain how to obtain these in the next section. This model s(c,x) will output a high score when the dictionary for the category D(c) contains many descriptors that highly match the observed image x. Figure 2 illustrates this approach to classification. We use addition so that some descriptors can be missing in the image, and normalize by the number of descriptors for the class to allow different classes to have different numbers of descriptors. Since the descriptors are both additive and expressed in natural language, the model is naturally interpretable. To understand why the model predicts category c, one can simply read which descriptors have a high score.” Examiner notes the label for a defined domain is the predicted category c, and the descriptions are the descriptors D(c).)
Ge, da Silva, and Menon all teach classifying unlabeled data using a specialized machine learning model and thus are analogous to the claimed invention. It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the application to have modified Ge and da Silva to use generated descriptions to generate a label for the new domain for each image using a large language model and text prompts like Menon. Doing so would have been advantageous because “by basing decisions on these descriptors, we can provide additional cues that encourage using the features we want to be used” (Menon, page 1). Furthermore, doing so would “provide an alternative to the current zero-shot classification paradigm with vision-language models, comparing to class descriptors obtained from a large language model instead of just the class directly. This requires no additional training, and does not require substantial computational over head during inference. By construction, this provides some level of inherent interpretability; we can know an image was labeled a tiger because the model saw its stripes rather than its tail. Rather than compromising performance metrics, our approach improves accuracy across datasets and distribution shifts, achieving a ∼ 4-5% increase on top-1 ImageNet accuracy” (Menon, page 2).
Regarding claim 12, Ge in view of da Silva teach the method of claim 11.
Ge in view of da Silva does not, but Menon teaches
wherein the descriptions for the collected data samples are generated by applying a plurality of ML image-to-text models to the collected data samples (Menon, page 2, “We instead mine large language models to automatically build descriptors, and perform recognition by comparing to the category descriptors.” Menon, page 3, “We use vision-language models to visually ground the natural language descriptors generated by the large language models” Examiner notes the descriptions are the descriptors.)
Ge, da Silva, and Menon all teach classifying unlabeled data using a specialized machine learning model and thus are analogous to the claimed invention. It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the application to have modified Ge and da Silva to use a plurality of ML image-to-text models to generate descriptions for the data like Menon. Doing so would have been advantageous because the method “improves accuracy across datasets and distribution shifts, achieving a ∼ 4-5% increase on top-1 ImageNet accuracy” (Menon, page 2).
Regarding claim 13, Ge in view of da Silva in further view of Genon teaches the method of claim 12. Genon further teaches
wherein using descriptions generated for the collected data samples further comprises: obtaining a general large language model (LLM) and a text prompt configured to analyze image descriptions (Menon, page 3, “We propose to automatically construct this set by prompting a large language model, such as GPT-3, to describe the visual features that distinguish that object category in a photograph. We prompt the language model with the input: Q: What are useful features for distinguishing a {category name} in a photo? A: There are several useful visual features to tell there is a {category name} in a photo: where {category name} is substituted for a given c.”)
applying the LLM and the text prompt to the descriptions generated by the image-to-text models for the collected data samples to define domains for each image (Menon, page 3, “We propose to automatically construct this set by prompting a large language model, such as GPT-3, to describe the visual features that distinguish that object category in a photograph. We prompt the language model with the input: Q: What are useful features for distinguishing a {category name} in a photo? A: There are several useful visual features to tell there is a {category name} in a photo: where {category name} is substituted for a given c.” Menon, page 5, “In contrast, our approach acquires visual descriptions from a large language model, which allows it to build new classifiers for categories that φ has not encountered yet.” Menon, page 6, “Even though these categories are relatively new, GPT3 is able to build descriptors for them because enough people on the Internet have visually described them. By combining these descriptors together, the model can recognize the new category.” Examiner notes that GPT3 is the LLM which uses the given text prompt, the descriptions are the descriptors, the domains are the categories, and the LLM can define new categories based on the descriptors.)
and selecting a label for a defined domain that is identified as predominant, to be the associated pseudo label, wherein the predominant domain is identified based on a frequency of domain occurrence in the generated descriptions (“Menon, page 3, “we estimate a score for category c through the additive decomposition:
PNG
media_image1.png
45
203
media_image1.png
Greyscale
where D(c) is the set of descriptors for the category c and φ(d,x) is the log probability that descriptor d pertains to the image x. Our approach will represent the descriptors d also through a natural language sentence; we explain how to obtain these in the next section. This model s(c,x) will output a high score when the dictionary for the category D(c) contains many descriptors that highly match the observed image x. Figure 2 illustrates this approach to classification. We use addition so that some descriptors can be missing in the image, and normalize by the number of descriptors for the class to allow different classes to have different numbers of descriptors. Since the descriptors are both additive and expressed in natural language, the model is naturally interpretable. To understand why the model predicts category c, one can simply read which descriptors have a high score.” Examiner notes the label for a defined domain is the predicted category c, and the descriptions are the descriptors D(c).)
Ge, da Silva, and Menon all teach classifying unlabeled data using a specialized machine learning model and thus are analogous to the claimed invention. It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the application to have modified Ge and da Silva to use generated descriptions to generate a label for the new domain for each image using a large language model and text prompts like Menon. Doing so would have been advantageous because “by basing decisions on these descriptors, we can provide additional cues that encourage using the features we want to be used” (Menon, page 1). Furthermore, doing so would “provide an alternative to the current zero-shot classification paradigm with vision-language models, comparing to class descriptors obtained from a large language model instead of just the class directly. This requires no additional training, and does not require substantial computational over head during inference. By construction, this provides some level of inherent interpretability; we can know an image was labeled a tiger because the model saw its stripes rather than its tail. Rather than compromising performance metrics, our approach improves accuracy across datasets and distribution shifts, achieving a ∼ 4-5% increase on top-1 ImageNet accuracy” (Menon, page 2).
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ge in view of da Silva and further in view of Kumar et al. (US-20230316715-A1).
Regarding claim 10, Ge in view of da Silva teaches the system of claim 1. Ge in view of da Silva do not, but Kumar teaches
wherein the edge device is a camera in a connected car, and the data samples include image data from the camera (Kumar, paragraph 0049, “Although vehicles 102 are described as legacy vehicles retrofitted with autonomous piloting technology, it should be understood that vehicles 102 can be originally manufactured autonomous vehicles, vehicles equipped with advanced driver-assistance systems (ADAS), vehicles outfitted with dashcams or other systems/sensors, and so on. The data received from vehicles 102 can be any data collected by vehicles 102 and utilized for any purpose (e.g., park assist, lane assist, auto start/stop, etc.).” Kumar, Fig. 1.)
Ge, da Silva, and Kumar all teach classifying unlabeled data using a specialized machine learning model and thus are analogous to the claimed invention. It would have been obvious to one having ordinary skill in the art prior to the effective filing date of the application to have modified Ge and da Silva to have the unlabeled data be image data collected from a camera attached to a car. Doing so would have been advantageous because “detecting unseen objects (i.e. objects that were not used to train a neural network) has been an enormous challenge and a problem of significant relevance in the field of computer vision, especially in detecting rare objects. The implications of solving this problem are not just limited to real time perception modules, but also for offline or off-board perception applications such as automated tagging, data curation, etc.” (Kumar, paragraph 0003). Furthermore, there are many uses for such classifications of car camera image data, “including, but not limited to, actuarial calculation, machine learning research, autonomous vehicle simulations, etc.” (Kumar, paragraph 0050).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Badong et al. (CN117218415A) teaches an unsupervised domain adaptation method that uses prompting to classify features from unlabeled input images.
Marrero et al. (US20220108134) teaches unsupervised domain adaptation, wherein features are extracted and separate classifiers are used to classify the data and the domain.
Bianco et al. (“Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusion”) teaches a method combining image descriptions from a plurality of image-to-text models using a large language model to label the image.
Lei et al. (“Prompt Learning in Computer Vision: A Survey”) discloses methods of prompt learning in image classification machine learning.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Zared O. Cohen whose telephone number is (571)270-0531. The examiner can normally be reached M-Th, 8am to 5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Z.O.C./Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights