DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are presented for examination.
Claim Objections
Claims 3, 8, 10, 12, 14, 15, 16, and 17 are objected to because of the following informalities:
Claim 3: “the generating the one or more consensus labels” should read “the determining the one or more consensus labels”
Claim 8: “one or more third labels is” should read “one or more third labels are”
Claim 10: “the generation the” should read “the generation of the”
Claim 12: “a second label from the one or more second labels” should read “a second label from the one or more first labels”
Claim 14: “determining” should read “determine”; “the one or more values of the one or more metrics” should read “the one or more values for the one or more metrics”
Claim 15: “determining” should read “determine”; “the one or more values of the one or more metrics” should read “the one or more values for the one or more metrics”
Claim 16: “one or one or more machine learning models” should read “one or more machine learning models”; “determining” should read “determine”; “associated a second” should read “associated with a second”
Claim 17 is objected to due to dependency of objected-to claim 16
Appropriate correction is required.
Specification
The disclosure is objected to because of the following informalities:
[0001]: “not fully understand” should read “not fully understanding”
[0034]: “that that” should read “that”; “includes” should read “include”
[0036]: “than accuracy requirement score” should read “than the accuracy requirement score”
[0039]: "With reference to FIG. 1A" should read "With reference to FIG. 1"
[0045]: "illustrates the first sensor representation 202(1) is being labeled" should read "illustrates the first sensor representation 202(1) as being labeled"
[0046]: "client devises" should read "client devices"; "sensos" should read "sensor"
[0055]: "202(3) a user 118(2)" should read "202(3) to a user 118(1)"; "in contrast the" should read "in contrast to the"
[0056]: “202(4) a user 118(2)” should read “202(4) to a user 118(2)”
[0064]: "accurately all" should read "accurately label all"
[0071]: "a second examples" should read "a second example"; "that that" should read "that"
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “a system for performing one or more digital twin operations”; “a system for performing light transport simulation”; “a system for performing collaborative content creation for 3D assets”; “a system for performing one or more deep learning operations”; “a system for performing one or more generative Al operations”; “a system for performing operations using one or more large language models (LLMs)”; “a system for performing operations using one or more visual language models (VLMs)”; “a system for performing one or more conversational Al operations”; “a system for generating synthetic data”; “a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content” in claims 18 and 20.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 18 and 20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 18 and 20 recite the limitations “a system for performing one or more digital twin operations”; “a system for performing light transport simulation”; “a system for performing collaborative content creation for 3D assets”; “a system for performing one or more deep learning operations”; “a system for performing one or more generative Al operations”; “a system for performing operations using one or more large language models (LLMs)”; “a system for performing operations using one or more visual language models (VLMs)”; “a system for performing one or more conversational Al operations”; “a system for generating synthetic data”; and “a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content” which invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed functions and to clearly link the structure, material, or acts to the functions. Therefore, the written description is inadequate to show that the inventor had possession of the claimed invention at the time of filing. See rejections under 35 U.S.C. 112(b) below for further analysis.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 18 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim limitations “a system for performing one or more digital twin operations”; “a system for performing light transport simulation”; “a system for performing collaborative content creation for 3D assets”; “a system for performing one or more deep learning operations”; “a system for performing one or more generative Al operations”; “a system for performing operations using one or more large language models (LLMs)”; “a system for performing operations using one or more visual language models (VLMs)”; “a system for performing one or more conversational Al operations”; “a system for generating synthetic data”; and “a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content” invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. Paragraph [0038] of the specification merely repeats the claim language verbatim, and does not disclose and clearly link the corresponding structures and algorithms for performing the entire claimed functions. Therefore, the claims are indefinite and are rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent
Subject Matter Eligibility Guidance (“2019 PEG”).
Claim 1
Step 1: The claim recites a method, and therefore is directed to the statutory category of processes.
Step 2A Prong 1: The claim recites, inter alia:
“generating, based at least on one or more first user inputs associated with the one or more automatically generated labels, one or more first human labels for the one or more first sensor representations”; This limitation encompasses mentally generating one or more first human labels for the one or more first sensor representations based at least on one or more first user inputs associated with one or more automatically generated labels.
“generating, based at least on one or more second user inputs, one or more second human labels for one or more second sensor representations”; This limitation encompasses mentally generating one or more second human labels for one or more second sensor representations based at least on one or more second user inputs.
“determining one or more consensus labels based at least on the one or more second human labels”; This limitation encompasses mentally determining one or more consensus labels based at least on the one or more second human labels.
“determining, based at least on the one or more first human labels and the one or more consensus labels, one or more values for one or more metrics associated with the one or more first human labels”; This limitation encompasses mentally determining one or more values for one or more metrics associated with the one or more first human labels based at least on the one or more first human labels and the one or more consensus labels.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “obtaining one or more automatically generated labels for one or more first sensor representations, the one or more automatically generated labels being determined using one or more machine learning models,” however this limitation amounts to the insignificant extra-solution activity of mere data gathering (MPEP 2106.05(g)).
Step 2B: The claim does not contain significantly more than the judicial exception. The obtaining limitation, in addition to reciting insignificant extra-solution activity, is also directed to the well-understood, routine, and conventional activity of receiving or transmitting data over a network (MPEP 2106.05(d)(II)(i) OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)). As an ordered whole, the claim is directed to a mentally performable process of generating one or more first human labels, generating one or more second human labels, determining one or more consensus labels, and determining one or more values for one or more metrics associated with the one or more first human labels. Nothing in the claim provides significantly more than this. As such, the claim is not patent eligible.
Claim 2
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
“comparing the one or more first human labels to the one or more consensus labels”; This limitation encompasses mentally comparing the one or more first human labels to the one or more consensus labels.
“determining, based at least on the comparing, one or more errors associated with the one or more first human labels”; This limitation encompasses mentally determining one or more errors associated with the one or more first human labels.
“determining, based at least on the one or more errors, the one or more values for the one or more metrics associated with the one or more first human labels”; This limitation encompasses mentally determining the one or more values for the one or more metrics associated with the one or more first human labels based at least on the one or more errors.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 1.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 1.
Claim 3
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
“generating the one or more consensus labels based at least on one or more of: computing one or more averages associated with the one or more second human labels; computing one or more modes associated with the one or more second human labels; computing one or more medians associated with the one or more second human labels; computing one or more amounts of overlap associated with the one or more second human labels”; This limitation encompasses mentally generating the one or more consensus labels by computing one or more averages, modes, medians, and/or amounts of overlap associated with the one or more second human labels.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 1.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 1.
Claim 4
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
“determining, based at least on the one or more values for the one or more metrics, a first score associated with an accuracy of training data that includes at least a portion of the one or more first sensor representations”; This limitation encompasses mentally determining a first score associated with an accuracy of training data that includes at least a portion of the one or more first sensor representations.
“determining a second score associated with one or more product requirements corresponding to the training data”; This limitation encompasses mentally determining a second score associated with one or more product requirements corresponding to the training data.
“determining, based at least on the first score and the second score, whether the training data satisfies the one or more product requirements”; This limitation encompasses mentally determining whether the training data satisfies the one or more product requirements based at least on the first score and the second score.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 1.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 1.
Claim 5
Step 1: A process, as above.
Step 2A Prong 1: The claim recites, inter alia:
“determining, based at least on the one or more values for the one or more metrics, at least one of: a first evaluation associated with a user that provided at least a portion of the one or more first user inputs; or a second evaluation associated with a group of users that provided the one or more first user inputs”; This limitation encompasses mentally determining at least one of a first evaluation associated with a user or a second evaluation associated with a group of users.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 1.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 1.
Claim 6
Step 1: The claim recites a system comprising one or more processors, and therefore is directed to the statutory category of machines.
Step 2A Prong 1: The claim recites, inter alia:
“generate first data representative of one or more first labels associated with one or more first sensor representations”; This limitation encompasses mentally generating first data representative of one or more first labels associated with one or more first sensor representations.
“generate, based at least on one or more user inputs, second data representative of one or more second labels associated with one or more second sensor representations that correspond to the one or more first sensor representations”; This limitation encompasses mentally generating second data representative of one or more second labels associated with one or more second sensor representations.
“determine, based at least on the one or more second labels, one or more values for one or more metrics associated with the one or more first labels”; This limitation encompasses mentally determining one or more values for one or more metrics associated with the one or more first labels based at least on the one or more second labels.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “A system comprising: one or more processors to: [perform the judicial exception],” however this limitation amounts to mere instructions to apply a judicial exception using a generic computer (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above. As an ordered whole, the claim is directed to a mentally performable process of generating first data representative of one or more first labels associated with one or more first sensor representations, generating, based at least on one or more user inputs, second data representative of one or more second labels associated with one or more second sensor representations that correspond to the one or more first sensor representations, and determining, based at least on the one or more second labels, one or more values for one or more metrics associated with the one or more first labels. Nothing in the claim provides significantly more than this. As such, the claim is not patent eligible.
Claim 7
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites the same judicial exception as claim 6.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “obtain third data representative of one or more third labels associated with the one or more first sensor representations, the one or more third labels being determined using one or more machine learning models; and receive one or more second inputs associated with the one or more third labels, wherein the generation of the first data is based at least on the one or more second inputs,” however, these limitations amount to the insignificant extra-solution activity of mere data gathering (MPEP 2106.05(g)).
Step 2B: The claim does not contain significantly more than the judicial exception. The obtain data and receive inputs limitations, in addition to reciting insignificant extra-solution activity, are also directed to the well-understood, routine, and conventional activity of receiving or transmitting data over a network (MPEP 2106.05(d)(II)(i) OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
Claim 8
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites the same judicial exception as claim 6.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “wherein the one or more second inputs indicate one or more of: that a first label of the one or more third labels is accurate; an update to a second label of the one or more third labels; that a third label of the one or more third labels needs to be removed; or that the one or more third labels is missing a fourth label,” however, this merely further limits the “receive one or more second inputs” limitation of claim 7, and still amounts to the insignificant extra-solution activity of mere data gathering (MPEP 2106.05(g)).
Step 2B: The claim does not contain significantly more than the judicial exception. The receive inputs limitation, in addition to reciting insignificant extra-solution activity, is also directed to the well-understood, routine, and conventional activity of receiving or transmitting data over a network (MPEP 2106.05(d)(II)(i) OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015)).
Claim 9
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites, inter alia:
“generate one or more third labels based at least on the one or more second labels, wherein the determination of the one or more values for the one or more metrics is based at least on the one or more third labels”; This limitation encompasses mentally generating one or more third labels based at least on the one or more second labels, and mentally determining the one or more values for the one or more metrics based at least on the one or more third labels.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 6.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 6.
Claim 10
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites, inter alia:
“generating the one or more third labels based at least on one or more of: computing one or more averages associated with the one or more second labels; computing one or more modes associated with the one or more second labels; computing one or more medians associated with the one or more second labels; or computing one or more amounts of overlap associated with the one or more second labels”; This limitation encompasses mentally generating the one or more third labels by computing one or more averages, modes, medians, and/or amounts of overlap associated with the one or more second labels.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 6.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 6.
Claim 11
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites, inter alia:
“determining, based at least on the one or more second labels, one or more errors associated with the one or more first labels”; This limitation encompasses mentally determining one or more errors associated with the one or more first labels based at least on the one or more second labels.
“determining, based at least on the one or more errors, the one or more values for the one or more metrics associated with the one or more first labels”; This limitation encompasses mentally determining the one or more values for the one or more metrics associated with the one or more first labels based at least on the one or more errors.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 6.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 6.
Claim 12
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites, inter alia:
“wherein the one or more errors are associated with at least one of: a first object represented by the one or more first sensor representations not including a first label from the one or more first labels; a second object represented by the one or more first sensor representations including a second label from the one or more second labels that is inaccurate; or a third object represented by the one or more first sensor representations not including a third label from the one or more first labels”; This limitation merely further limits the determining one or more errors limitation of claim 11, which is still mentally performable when the one or more errors are associated with at least one of the recited options, as one can mentally determine if a label is not included or inaccurate.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 6.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 6.
Claim 13
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites, inter alia:
“generate the one or more second sensor representations based at least on replicating the one or more first sensor representations”; This limitation encompasses mentally generating the one or more second sensor representations by replicating the one or more first sensor representations, such as by replicating the one or more first sensor representations using pen and paper.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 6.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 6.
Claim 14
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites, inter alia:
“determine that at least a portion of the one or more values for the one or more metrics are associated with a user”; This limitation encompasses mentally determining that at least a portion of the one or more values for the one or more metrics are associated with a user.
“determining, based at least on the portion of the one or more values of the one or more metrics, a score indicating an accuracy associated with the user”; This limitation encompasses mentally determining a score indicating an accuracy associated with the user based at least on the portion of the one or more values of the one or more metrics.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 6.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 6.
Claim 15
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites, inter alia:
“determine that at least a portion of the one or more values for the one or more metrics are associated with a group of users”; This limitation encompasses mentally determining that at least a portion of the one or more values for the one or more metrics are associated with a group of users.
“determining, based at least on the portion of the one or more values of the one or more metrics, a score indicating an accuracy associated with the user”; This limitation encompasses mentally determining a score indicating an accuracy associated with the group of users based at least on the portion of the one or more values of the one or more metrics.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 6.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 6.
Claim 16
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites, inter alia:
“determine, based at least on the one or more values for the one or more metrics, a first score associated with a first accuracy of training data that includes at least a portion of the one or more first sensor representations”; This limitation encompasses mentally determining a first score associated with a first accuracy of training data based at least on the one or more values for the one or more metrics.
“determine a second score associated a second accuracy corresponding to one or one or more machine learning models”; This limitation encompasses mentally determining a second score associated with a second accuracy corresponding to one or more machine learning models.
“determining, based at least on the first score and the second score, whether the training data satisfies the second accuracy corresponding to the one or more machine learning models”; This limitation encompasses mentally determining whether the training data satisfies the second accuracy based at least on the first score and the second score.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 6.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 6.
Claim 17
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites, inter alia:
“determining, based at least on one or more product requirements corresponding to the one or more machine learning models, one or more second values associated with the second accuracy”; This limitation encompasses mentally determining one or more second values associated with the second accuracy based at least on one or more product requirements corresponding to the one or more machine learning models.
“determining the second score based at least on the one or more second values”; This limitation encompasses mentally determining the second score based at least on the one or more second values.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. No further additional elements are recited, see analysis of claim 6.
Step 2B: The claim does not contain significantly more than the judicial exception. No further additional elements are recited, see analysis of claim 6.
Claim 18
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites the same judicial exception as claim 6.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative Al operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing one or more conversational Al operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources,” however, this limitation amounts to mere instructions to apply a judicial exception on a generic computer programmed with a generic class of computer algorithms (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above.
Claim 19
Step 1: The claim recites one or more processors, and therefore is directed to the statutory category of machines.
Step 2A Prong 1: The claim recites, inter alia:
“generate first data representing an evaluation associated with one or more first labels corresponding to one or more first sensor representations, wherein the first data is generated based at least on comparing the one or more first labels to one or more second labels corresponding to one or more second sensor representations as determined using one or more user inputs”; This limitation encompasses mentally generating first data representing an evaluation associated with one or more first labels by mentally comparing the one or more first labels to one or more second labels.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “One or more processors comprising: processing circuitry to… [perform the judicial exception],” however this limitation amounts to mere instructions to apply a judicial exception on a generic computer (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above. As an ordered whole, the claim is directed to a mentally performable process of generating first data representing an evaluation associated with one or more first labels by mentally comparing the one or more first labels to one or more second labels. Nothing in the claim provides significantly more than this. As such, the claim is not patent eligible.
Claim 20
Step 1: A machine, as above.
Step 2A Prong 1: The claim recites the same judicial exception as claim 19.
Step 2A Prong 2: This judicial exception is not integrated into a practical application. The claim further recites “wherein the one or more processors is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative Al operations; a system for performing operations using one or more large language models (LLMs); a system for performing operations using one or more visual language models (VLMs); a system for performing one or more conversational Al operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources,” however, this limitation amounts to mere instructions to apply a judicial exception on a generic computer programmed with a generic class of computer algorithms (MPEP 2106.05(f)).
Step 2B: The claim does not contain significantly more than the judicial exception. The analysis at this step mirrors that of Step 2A Prong 2 above.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and
103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis
(i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if
the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis
for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 6, 9-11, 13-14, and 18-20 are rejected under 35 U.S.C. 102(a)(1) and (a)(2) as being anticipated by Lyman et al. (US20200161005) (hereinafter “Lyman”).
Regarding claim 6, Lyman discloses “A system comprising:
one or more processors (Lyman, [0042]: “FIG. 1 presents a medical scan processing system 100, which can include one or more medical scan subsystems 101 that communicate bidirectionally with one or more client devices 120 via a wired and/or wireless network 150. Some or all of the subsystems 101 can utilize the same processing devices, memory devices, and/or network interfaces, for example, running on a same set of shared servers connected to network 150”) to:
generate first data representative of one or more first labels associated with one or more first sensor representations (Lyman, [0311-0312]: “As illustrated in FIG. 13A, the medical scan annotator system 106 can select a medical scan from the medical scan database 342 for transmission via network 150 to one or more client devices 120 associated with a selected user set 4010 corresponding to one or more users in the user database 344… The client device 120 of each user of the selected user set 4010 can display one or more received medical scans to the via the interactive interface 275 displayed by a display device corresponding to the client device 120, for example, by displaying medical scan image data 410 in conjunction with the medical scan assisted review system 102. The interactive interface 275 displayed by client devices 120 of each user in the selected user set 4010 can include a prompt to provide annotation data 4020 corresponding to the medical scan. This can include a prompt to provide a text and/or voice description via a keyboard and/or microphone associated with the client device. This can also include a prompt to indicate one or more abnormalities in the medical scan, for example, by clicking on or outlining a region corresponding to each abnormality via a mouse and/or touchscreen… The interactive interface 275 can present the medical scan by utilizing interface features indicated in the display parameter data 470 and/or the interface preference data 560 of the user, and/or the user can indicate the annotation data via the interactive interface 275 by utilizing interface features indicated in the display parameter data 470 and/or the interface preference data 560 of the user. For example, some or all of the annotation data 4020 can correspond to, or be automatically generated based on, user input to the interactive interface; Examiner notes that the annotation data 4020 generated based on the user inputs of a first user of the selected user set 4010 corresponds to “first data representative of one or more first labels,” and the medical scan image data 410 displayed on client device 120 of the first user corresponds to “one or more first sensor representations”);
generate, based at least on one or more user inputs, second data representative of one or more second labels associated with one or more second sensor representations that correspond to the one or more first sensor representations (Lyman, [0311-0312 (see excerpt above)]; Examiner notes that the annotation data 4020 generated based on the user inputs of a second user of the selected user set 4010 corresponds to “second data representative of one or more second labels,” and the medical scan image data 410 displayed on client device 120 of the second user corresponds to “one or more second sensor representations that correspond to the one or more first sensor representations”); and
determine, based at least on the one or more second labels, one or more values for one or more metrics associated with the one or more first labels (Lyman, [0314]: “The medical scan annotator system 106 can evaluate the set annotation data 4020 received from the selected user set 4010 to determine if a consensus is reached, and/or generate a final consensus annotation 4030, for example, by performing an annotation consensus function 4040. For example, consider a selected user set 4010 that includes three users. If two users annotate a medical scan as “normal” and the third user annotates the medical scan as “contains abnormality”, the annotation consensus function 4040 performed by medical scan annotator system 106 may determine that the final consensus annotation 4030 is “normal” by following a majority rules strategy” and [0324]: “The user profile entries 354 of each user in the selected user set 4010 and/or each expert user can be automatically updated by the medical scan annotator system 106 or another subsystem 101 by generating and/or updating performance score data 530 for each user based comparing their annotation to the final consensus annotation 4030. For example, the accuracy score data 531 of the performance score data 530 can be generated by calculating the Euclidian distance between a feature vector of a user's annotation and the feature vector of the consensus annotation as described previously, where a higher performance score is assigned to a user whose annotation is a smaller Euclidian distance from the consensus, and a lower performance score is assigned to a user whose annotation is a larger Euclidian distance from the consensus”; Examiner notes that calculating the Euclidian distance between the first user’s annotation and the consensus annotation to generate accuracy score data 531 corresponds to “determining one or more values for one or more metrics associated with the one or more first labels,” and the determination is “based at least on the one or more second labels” because the second user’s annotation is used to determine the consensus annotation).
Regarding claim 9, Lyman further discloses “wherein the one or more processors are further to:
generate one or more third labels based at least on the one or more second labels, wherein the determination of the one or more values for the one or more metrics is based at least on the one or more third labels” (Lyman, [0314]: “The medical scan annotator system 106 can evaluate the set annotation data 4020 received from the selected user set 4010 to determine if a consensus is reached, and/or generate a final consensus annotation 4030, for example, by performing an annotation consensus function 4040. For example, consider a selected user set 4010 that includes three users. If two users annotate a medical scan as “normal” and the third user annotates the medical scan as “contains abnormality”, the annotation consensus function 4040 performed by medical scan annotator system 106 may determine that the final consensus annotation 4030 is “normal” by following a majority rules strategy” and [0324]: “The user profile entries 354 of each user in the selected user set 4010 and/or each expert user can be automatically updated by the medical scan annotator system 106 or another subsystem 101 by generating and/or updating performance score data 530 for each user based comparing their annotation to the final consensus annotation 4030. For example, the accuracy score data 531 of the performance score data 530 can be generated by calculating the Euclidian distance between a feature vector of a user's annotation and the feature vector of the consensus annotation as described previously, where a higher performance score is assigned to a user whose annotation is a smaller Euclidian distance from the consensus, and a lower performance score is assigned to a user whose annotation is a larger Euclidian distance from the consensus”; Examiner notes that “final consensus annotation 4030” corresponds to “one or more third labels”).
Regarding claim 10, the rejection of claim 9 is incorporated. Lyman further discloses “wherein the generation the one or more third labels comprises generating the one or more third labels based at least on one or more of:
computing one or more averages associated with the one or more second labels;
computing one or more modes associated with the one or more second labels;
computing one or more medians associated with the one or more second labels; or
computing one or more amounts of overlap associated with the one or more second labels” (Lyman, [0316]: “The annotation consensus function 4040 can calculate the final consensus annotation 4030 itself by creating a consensus feature vector, where each attribute of the consensus feature vector is determined by calculating a mean, median or mode of each corresponding annotation feature extracted from all of the received annotation data 4020”; Examiner notes that the second label is included in the received annotation data 4020, thus the mean, median, and mode are “associated with the one or more second labels”).
Regarding claim 11, the rejection of claim 6 is incorporated. Lyman further discloses “wherein the determination of the one or more values for the one or more metrics comprises:
determining, based at least on the one or more second labels, one or more errors associated with the one or more first labels (Lyman, [0314]: “The medical scan annotator system 106 can evaluate the set annotation data 4020 received from the selected user set 4010 to determine if a consensus is reached, and/or generate a final consensus annotation 4030, for example, by performing an annotation consensus function 4040. For example, consider a selected user set 4010 that includes three users. If two users annotate a medical scan as “normal” and the third user annotates the medical scan as “contains abnormality”, the annotation consensus function 4040 performed by medical scan annotator system 106 may determine that the final consensus annotation 4030 is “normal” by following a majority rules strategy” and [0324]: “The user profile entries 354 of each user in the selected user set 4010 and/or each expert user can be automatically updated by the medical scan annotator system 106 or another subsystem 101 by generating and/or updating performance score data 530 for each user based comparing their annotation to the final consensus annotation 4030. For example, the accuracy score data 531 of the performance score data 530 can be generated by calculating the Euclidian distance between a feature vector of a user's annotation and the feature vector of the consensus annotation as described previously...”; Examiner notes that calculating the Euclidian distance between the first user’s annotation and the consensus annotation corresponds to “determining one or more errors associated with the one or more first labels,” and the determination is “based at least on the one or more second labels” because the second user’s annotation is used to determine the consensus annotation); and
determining, based at least on the one or more errors, the one or more values for the one or more metrics associated with the one or more first labels” (Lyman, [0324]: “...where a higher performance score is assigned to a user whose annotation is a smaller Euclidian distance from the consensus, and a lower performance score is assigned to a user whose annotation is a larger Euclidian distance from the consensus”).
Regarding claim 13, the rejection of claim 6 is incorporated. Lyman further discloses “wherein the one or more processors are further to generate the one or more second sensor representations based at least on replicating the one or more first sensor representations” (Lyman, [0311]: “As illustrated in FIG. 13A, the medical scan annotator system 106 can select a medical scan from the medical scan database 342 for transmission via network 150 to one or more client devices 120 associated with a selected user set 4010 corresponding to one or more users in the user database 344… The client device 120 of each user of the selected user set 4010 can display one or more received medical scans to the via the interactive interface 275 displayed by a display device corresponding to the client device 120, for example, by displaying medical scan image data 410 in conjunction with the medical scan assisted review system 102”; Examiner notes that the same medical scan is transmitted to one or more client devices, thus the second sensor representation displayed on the second user’s client device is a replica of the first sensor representation displayed on the first user’s client device).
Regarding claim 14, the rejection of claim 6 is incorporated. Lyman further discloses “wherein the one or more processors are further to:
determine that at least a portion of the one or more values for the one or more metrics are associated with a user (Lyman, [0324]: “The user profile entries 354 of each user in the selected user set 4010 and/or each expert user can be automatically updated by the medical scan annotator system 106 or another subsystem 101 by generating and/or updating performance score data 530 for each user based comparing their annotation to the final consensus annotation 4030. For example, the accuracy score data 531 of the performance score data 530 can be generated by calculating the Euclidian distance between a feature vector of a user's annotation and the feature vector of the consensus annotation as described previously, where a higher performance score is assigned to a user whose annotation is a smaller Euclidian distance from the consensus, and a lower performance score is assigned to a user whose annotation is a larger Euclidian distance from the consensus”); and
determining, based at least on the portion of the one or more values of the one or more metrics, a score indicating an accuracy associated with the user (Lyman, [0324]: “Aggregate performance data for each user can be generate and/or updated based on past accuracy and/or efficiency scores, based on how many scans have been annotated in total, based on measured improvement of the user over time, etc”).
Regarding claim 18, the rejection of claim 6 is incorporated. Lyman further discloses “wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing one or more simulation operations;
a system for performing one or more digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing one or more deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing one or more generative Al operations;
a system for performing operations using one or more large language models (LLMs);
a system for performing operations using one or more visual language models (VLMs);
a system for performing one or more conversational Al operations;
a system for generating synthetic data;
a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources” (Lyman, [0564]: “Note that if the processing module, module, processing circuit, and/or processing unit includes more than one processing device, the processing devices may be centrally located (e.g., directly coupled together via a wired and/or wireless bus structure) or may be distributedly located (e.g., cloud computing via indirect coupling via a local area network and/or a wide area network)”).
Regarding claim 19, Lyman discloses “One or more processors (Lyman, [0042]: “FIG. 1 presents a medical scan processing system 100, which can include one or more medical scan subsystems 101 that communicate bidirectionally with one or more client devices 120 via a wired and/or wireless network 150. Some or all of the subsystems 101 can utilize the same processing devices, memory devices, and/or network interfaces, for example, running on a same set of shared servers connected to network 150”) comprising:
processing circuitry (Lyman, [0564]: “As may also be used herein, the terms “processing module”, “processing circuit”, “processor”, “processing device” and/or “processing unit” may be a single processing device or a plurality of processing devices. Such a processing device may be a microprocessor, micro-controller, digital signal processor, graphics processing unit, microcomputer, central processing unit, field programmable gate array, programmable logic device, state machine, logic circuitry, analog circuitry, digital circuitry, and/or any device that manipulates signals (analog and/or digital) based on hard coding of the circuitry and/or operational instructions”) to
generate first data representing an evaluation associated with one or more first labels corresponding to one or more first sensor representations, wherein the first data is generated based at least on comparing the one or more first labels to one or more second labels corresponding to one or more second sensor representations as determined using one or more user inputs” (Lyman, [0054]: “The medical scan annotator system 106 can be operable to select a medical scan for transmission via a network to a first client device and a second client device for display via an interactive interface, and annotation data can be received from the first client device and the second client device in response. Annotation similarity data can be generated by comparing the first annotation data to the second annotation data, and consensus annotation data can be generated based on the first annotation data and the second annotation data in response to the annotation similarity data indicating that the difference between the first annotation data and the second annotation data compares favorably to an annotation discrepancy threshold” and [0312]: “For example, some or all of the annotation data 4020 can correspond to, or be automatically generated based on, user input to the interactive interface”; Examiner notes that first annotation data for the medical scan displayed on an interactive interface of the first client device corresponds to “one or more first labels corresponding to one or more first sensor representations,” second annotation data for the medical scan displayed on an interactive interface of the second client device corresponds to “one or more second labels corresponding to one or more second sensor representations as determined using one or more user inputs,” and annotation similarity data corresponds to “first data representing an evaluation associated with one or more first labels”).
Regarding claim 20, the rejection of claim 19 is incorporated. Lyman further discloses “wherein the one or more processors is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing one or more simulation operations;
a system for performing one or more digital twin operations;
a system for performing light transport simulation;
a system for performing collaborative content creation for 3D assets;
a system for performing one or more deep learning operations;
a system implemented using an edge device;
a system implemented using a robot;
a system for performing one or more generative Al operations;
a system for performing operations using one or more large language models (LLMs);
a system for performing operations using one or more visual language models (VLMs);
a system for performing one or more conversational Al operations;
a system for generating synthetic data;
a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content;
a system incorporating one or more virtual machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources” (Lyman, [0564]: “Note that if the processing module, module, processing circuit, and/or processing unit includes more than one processing device, the processing devices may be centrally located (e.g., directly coupled together via a wired and/or wireless bus structure) or may be distributedly located (e.g., cloud computing via indirect coupling via a local area network and/or a wide area network)”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 5, and 7-8 are rejected under 35 U.S.C. 103 as being unpatentable over Lyman in view of Huval (US20180373980).
Regarding claim 1, Lyman discloses “A method comprising:
…
generating, based at least on one or more first user inputs… one or more first human labels for… one or more first sensor representations (Lyman, [0311-0312]: “As illustrated in FIG. 13A, the medical scan annotator system 106 can select a medical scan from the medical scan database 342 for transmission via network 150 to one or more client devices 120 associated with a selected user set 4010 corresponding to one or more users in the user database 344… The client device 120 of each user of the selected user set 4010 can display one or more received medical scans to the via the interactive interface 275 displayed by a display device corresponding to the client device 120, for example, by displaying medical scan image data 410 in conjunction with the medical scan assisted review system 102. The interactive interface 275 displayed by client devices 120 of each user in the selected user set 4010 can include a prompt to provide annotation data 4020 corresponding to the medical scan. This can include a prompt to provide a text and/or voice description via a keyboard and/or microphone associated with the client device. This can also include a prompt to indicate one or more abnormalities in the medical scan, for example, by clicking on or outlining a region corresponding to each abnormality via a mouse and/or touchscreen… The interactive interface 275 can present the medical scan by utilizing interface features indicated in the display parameter data 470 and/or the interface preference data 560 of the user, and/or the user can indicate the annotation data via the interactive interface 275 by utilizing interface features indicated in the display parameter data 470 and/or the interface preference data 560 of the user. For example, some or all of the annotation data 4020 can correspond to, or be automatically generated based on, user input to the interactive interface; Examiner notes that generating annotation data 4020 based on the user inputs of a first user of the selected user set 4010 corresponds to “generating, based at least on one or more first user inputs, one or more first human labels,” and the medical scan image data 410 displayed on client device 120 of the first user corresponds to “one or more first sensor representations”);
generating, based at least on one or more second user inputs, one or more second human labels for one or more second sensor representations (Lyman, [0311-0312 (see excerpt above)]; Examiner notes that generating annotation data 4020 based on the user inputs of a second user of the selected user set 4010 corresponds to “generating, based at least on one or more second user inputs, one or more second human labels,” and the medical scan image data 410 displayed on client device 120 of the second user corresponds to “one or more second sensor representations”);
determining one or more consensus labels based at least on the one or more second human labels (Lyman, [0314]: “The medical scan annotator system 106 can evaluate the set annotation data 4020 received from the selected user set 4010 to determine if a consensus is reached, and/or generate a final consensus annotation 4030, for example, by performing an annotation consensus function 4040. For example, consider a selected user set 4010 that includes three users. If two users annotate a medical scan as “normal” and the third user annotates the medical scan as “contains abnormality”, the annotation consensus function 4040 performed by medical scan annotator system 106 may determine that the final consensus annotation 4030 is “normal” by following a majority rules strategy”; Examiner notes that “final consensus annotation 4030” corresponds to “one or more consensus labels” which is determined based at least on the one or more second human labels, as it is determined based on the set of annotation data 4020 which includes the second user’s annotation); and
determining, based at least on the one or more first human labels and the one or more consensus labels, one or more values for one or more metrics associated with the one or more first human labels” (Lyman, [0324]: “The user profile entries 354 of each user in the selected user set 4010 and/or each expert user can be automatically updated by the medical scan annotator system 106 or another subsystem 101 by generating and/or updating performance score data 530 for each user based comparing their annotation to the final consensus annotation 4030. For example, the accuracy score data 531 of the performance score data 530 can be generated by calculating the Euclidian distance between a feature vector of a user's annotation and the feature vector of the consensus annotation as described previously, where a higher performance score is assigned to a user whose annotation is a smaller Euclidian distance from the consensus, and a lower performance score is assigned to a user whose annotation is a larger Euclidian distance from the consensus”; Examiner notes that calculating the Euclidian distance between the first user’s annotation and the consensus annotation to generate accuracy score data 531 corresponds to “determining, based at least on the one or more first human labels and the one or more consensus labels, one or more values for one or more metrics associated with the one or more first human labels” ).
Lyman does not appear to explicitly disclose the further limitations of the claim.
However, Huval discloses “obtaining one or more automatically generated labels for one or more first sensor representations, the one or more automatically generated labels being determined using one or more machine learning models” (Huval, [0052]: “In one example, the remote computer system passes a first optical image through the neural network in Block S132 to: detect an object at a first automatically-defined location within the first optical image; to calculate a first confidence score that the first object is of a first type; to calculate a second confidence score that the first object is of a second type; etc. In this example, if the first confidence score exceeds a preset threshold score (e.g., 60%) and exceeds the second confidence score, the remote computer system can serve—to the annotation portal—the first optical image with a first automated label attributed to the first automatically-defined location to define the first object as of the first type”) and “generating, based at least on one or more first user inputs associated with the one or more automatically generated labels, one or more first human labels for the one or more first sensor representations” (Huval, [0052]: “…the annotation portal can then render the first optical image and the first label over or linked to the first automatically-defined location for confirmation or replacement with a label of another type by the human annotator. Therefore, in this example, the annotation portal can present a pre-generated automated label for an object detected at an automatically-defined location by the neural network; the human annotator can then confirm the type and location of the automated label, confirm the automated label but adjust (e.g., move, shift) the automatically-defined location for the automated label, or reject the automated label and replace the automated label with a manual label of a different type in a manually-defined location”; Examiner notes that the adjusted or replaced label generated by the human annotator based on input to the annotation portal presenting the automatically generated label corresponds to “one or more first human labels”).
Huval and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Lyman to include “obtaining one or more automatically generated labels for one or more first sensor representations, the one or more automatically generated labels being determined using one or more machine learning models,” and such that the one or more first human labels are generated for the one or more first sensor representations based at least on one or more first user inputs associated with the one or more automatically generated labels, as disclosed by Huval, and one would have been motivated to do so, as doing so would decrease a number of human labor hours and quantity of transmitted data per label while also maintaining a high label quality in order to assemble a large and accurate training set sufficient to train an effective and accurate neural network at a reduced cost (see Huval, [0012]).
Regarding claim 2, the rejection of claim 1 is incorporated. Lyman as modified by Huval further discloses “wherein the determining the one or more values for the one or more metrics comprises:
comparing the one or more first human labels to the one or more consensus labels (Lyman, [0324]: “The user profile entries 354 of each user in the selected user set 4010 and/or each expert user can be automatically updated by the medical scan annotator system 106 or another subsystem 101 by generating and/or updating performance score data 530 for each user based comparing their annotation to the final consensus annotation 4030”);
determining, based at least on the comparing, one or more errors associated with the one or more first human labels (Lyman, [0324]: “For example, the accuracy score data 531 of the performance score data 530 can be generated by calculating the Euclidian distance between a feature vector of a user's annotation and the feature vector of the consensus annotation as described previously”; Examiner notes that the Euclidian distance between the first human label and the consensus label corresponds to an error associated with the first human label); and
determining, based at least on the one or more errors, the one or more values for the one or more metrics associated with the one or more first human labels (Lyman, [0324]: “.... where a higher performance score is assigned to a user whose annotation is a smaller Euclidian distance from the consensus, and a lower performance score is assigned to a user whose annotation is a larger Euclidian distance from the consensus”).
Regarding claim 3, the rejection of claim 1 is incorporated. Lyman as modified by Huval further discloses “wherein the generating the one or more consensus labels comprises generating the one or more consensus labels based at least on one or more of:
computing one or more averages associated with the one or more second human labels;
computing one or more modes associated with the one or more second human labels;
computing one or more medians associated with the one or more second human labels; or
computing one or more amounts of overlap associated with the one or more second human labels” (Lyman, [0316]: “The annotation consensus function 4040 can calculate the final consensus annotation 4030 itself by creating a consensus feature vector, where each attribute of the consensus feature vector is determined by calculating a mean, median or mode of each corresponding annotation feature extracted from all of the received annotation data 4020”; Examiner notes that the second human label is included in the received annotation data 4020, thus the mean, median, or mode is “associated with the one or more second human labels”).
Regarding claim 5, the rejection of claim 1 is incorporated. Lyman as modified by Huval further discloses “determining, based at least on the one or more values for the one or more metrics, at least one of:
a first evaluation associated with a user that provided at least a portion of the one or more first user inputs (Lyman, [0324]: “Aggregate performance data for each user can be generate and/or updated based on past accuracy and/or efficiency scores, based on how many scans have been annotated in total, based on measured improvement of the user over time, etc”); or
a second evaluation associated with a group of users that provided the one or more first user inputs.”
Regarding claim 7, the rejection of claim 6 is incorporated. Lyman does not appear to explicitly disclose the further limitations of the claim.
However, Huval discloses “obtain third data representative of one or more third labels associated with... one or more first sensor representations, the one or more third labels being determined using one or more machine learning models (Huval, [0052]: “In one example, the remote computer system passes a first optical image through the neural network in Block S132 to: detect an object at a first automatically-defined location within the first optical image; to calculate a first confidence score that the first object is of a first type; to calculate a second confidence score that the first object is of a second type; etc. In this example, if the first confidence score exceeds a preset threshold score (e.g., 60%) and exceeds the second confidence score, the remote computer system can serve—to the annotation portal—the first optical image with a first automated label attributed to the first automatically-defined location to define the first object as of the first type”; Examiner notes that the automated label corresponds to “third data representative of one or more third labels”); and
receive one or more second inputs associated with the one or more third labels, wherein… generation of... first data is based at least on the one or more second inputs” (Huval, [0052]: “…the annotation portal can then render the first optical image and the first label over or linked to the first automatically-defined location for confirmation or replacement with a label of another type by the human annotator. Therefore, in this example, the annotation portal can present a pre-generated automated label for an object detected at an automatically-defined location by the neural network; the human annotator can then confirm the type and location of the automated label, confirm the automated label but adjust (e.g., move, shift) the automatically-defined location for the automated label, or reject the automated label and replace the automated label with a manual label of a different type in a manually-defined location”; Examiner notes that the adjusted or replaced label generated by the human annotator based on input to the annotation portal (corresponding to one or more second inputs) presenting the automatically generated label corresponds to “first data”).
Huval and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Lyman such that the one or more processors are further to “obtain third data representative of one or more third labels associated with the one or more first sensor representations, the one or more third labels being determined using one or more machine learning models; and receive one or more second inputs associated with the one or more third labels, wherein the generation of the first data is based at least on the one or more second inputs,” as disclosed by Huval, and one would have been motivated to do so, as doing so would decrease a number of human labor hours and quantity of transmitted data per label while also maintaining a high label quality in order to assemble a large and accurate training set sufficient to train an effective and accurate neural network at a reduced cost (see Huval, [0012]).
Regarding claim 8, the rejection of claim 7 is incorporated. Lyman as modified by Huval further discloses “wherein the one or more second inputs indicate one or more of:
that a first label of the one or more third labels is accurate (Huval, [0052]: “the human annotator can then confirm the type and location of the automated label”);
an update to a second label of the one or more third labels (Huval, [0052]: “...confirm the automated label but adjust (e.g., move, shift) the automatically-defined location for the automated label”);
that a third label of the one or more third labels needs to be removed (Huval, [0052]: “or reject the automated label and replace the automated label with a manual label of a different type in a manually-defined location”);
or that the one or more third labels is missing a fourth label.”
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Lyman in view of Huval and Kwant et al. (US20190102656) (hereinafter “Kwant”).
Regarding claim 4, the rejection of claim 1 is incorporated. Neither Lyman nor Huval appear to explicitly disclose the further limitations of the claim.
However, Kwant discloses “determining, based at least on... one or more values for... one or more metrics, a first score associated with an accuracy of training data that includes at least a portion of... one or more first sensor representations (Kwant, [0036-0037]: “In one embodiment, the system 100 uses the manually trained feature detector 103 on the manually labeled training data set (e.g., the training images) to generate automatically marked labels for the data items or images in the training data set. The system 100 then compares the automatically marked labels to the manually marked labels to identify any differences between the two sets of labels. In one embodiment, discrepancies between the manually marked labels and the automatically marked labels can indicate potential QA issues. For example, the system 100 can identify images or other data items in the training set whose manually marked labels and automatically marked labels differ by a predetermined threshold or criterion as having potential QA issues. In one embodiment, the system 100 can use any quality metric to determine the differences between the manually marked labels and the automatically marked labels. For example, when the labeled features are objects or features that can be represented as polygons, examples of such a quality metric can include, but is not limited to, a distance metric that measures the distance (e.g., in pixels or some other unit of measure) between the manually labeled polygons and the corresponding automatically labeled polygons”; Examiner notes that training images correspond to “training data that includes at least a portion of one or more first sensor representations,” quality metric corresponds to “a first score associated with an accuracy” and the distance between the manually labeled polygons and the automatically labeled polygons corresponds to “one or more values for one or more metrics”);
determining a second score associated with one or more product requirements corresponding to the training data (Kwant, [0037]: “...predetermined precision criteria”); and
determining, based at least on the first score and the second score, whether the training data satisfies the one or more product requirements” (Kwant, [0037]: “Based on the computed difference or precision between the two data sets, the system 100 can perform any type of number of QA processes such as, but not limited to: (1) automatically filtering data items (e.g., images) from the training set that do not meet predetermined precision criteria, and then retraining the feature detector 103 with the training data set remaining after filtering”).
Kwant and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Lyman/Huval with Kwant to include “determining, based at least on the one or more values for the one or more metrics, a first score associated with an accuracy of training data that includes at least a portion of the one or more first sensor representations; determining a second score associated with one or more product requirements corresponding to the training data; and determining, based at least on the first score and the second score, whether the training data satisfies the one or more product requirements,” and one would have been motivated to do so, as doing so would automate the quality assurance process for training data, leading to lower resource utilization and costs and improved training data quality (see Kwant, [0033-0034]).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Lyman in view of Herman et al. (US11308364) (hereinafter “Herman”).
Regarding claim 12, the rejection of claim 11 is incorporated. Lyman does not appear to explicitly disclose the further limitations of the claim.
However, Herman discloses “wherein... one or more errors are associated with at least one of:
a first object represented by... one or more first sensor representations not including a first label from... one or more first labels;
a second object represented by...one or more first sensor representations including a second label from... one or more second labels that is inaccurate (Herman, Col 9, lines 23-32: “When benchmark 416 includes one or more classes or categories associated with an entity in data sample 400, the distance may be calculated based on a comparison of the classes or categories with the label. If the label includes some or all of the classes or categories in benchmark 416, then the distance is set to a positive value (e.g., 1 if the label exactly matches benchmark 416 and between 0 and 1 if the label partially matches benchmark 416). If the label does not include any of the classes or categories in benchmark 416, the distance is set to 0” and Col 6, lines 61-63: “For example, data sample 400 includes one or more images, video frames, LIDAR point clouds, radar scans, and/or other types of data collected by sensors on one or more electronic devices and/or vehicles”; Examiner notes that “an entity in data sample 400” corresponds to “a second object represented by one or more first sensor representations,” “the label” corresponds to a second label from one or more second labels,” and the distance being set to 0 if the label does not include any of the classes or categories in benchmark 416 corresponds to an error associated with a second object including a second label from one or more first labels that is inaccurate (doesn’t match the benchmark));
or a third object represented by... one or more first sensor representations not including a third label from... one or more first labels.”
Herman and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Lyman such that the “one or more errors are associated with at least one of: a first object represented by the one or more first sensor representations not including a first label from the one or more first labels; a second object represented by the one or more first sensor representations including a second label from the one or more second labels that is inaccurate; or a third object represented by the one or more first sensor representations not including a third label from the one or more first labels,” as disclosed by Herman, and one would have been motivated to do so, as doing so would allow for continuous evaluation of users’ labeling performance and adjustment of the users’ labeling tasks based on the labeling performance which would in turn improve the quality and accuracy of the labels (see Herman, Col 1, lines 55-65).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Lyman in view of Tanner et al. (US20200356770) (hereinafter “Tanner”).
Regarding claim 15, the rejection of claim 6 is incorporated. Lyman further discloses “wherein the one or more processors are further to: determine that at least a portion of the one or more values for the one or more metrics are associated with a... [user] (Lyman, [0324]: “The user profile entries 354 of each user in the selected user set 4010 and/or each expert user can be automatically updated by the medical scan annotator system 106 or another subsystem 101 by generating and/or updating performance score data 530 for each user based comparing their annotation to the final consensus annotation 4030. For example, the accuracy score data 531 of the performance score data 530 can be generated by calculating the Euclidian distance between a feature vector of a user's annotation and the feature vector of the consensus annotation as described previously, where a higher performance score is assigned to a user whose annotation is a smaller Euclidian distance from the consensus, and a lower performance score is assigned to a user whose annotation is a larger Euclidian distance from the consensus”); and
determining, based at least on the portion of the one or more values of the one or more metrics, a score indicating an accuracy associated with the... [user] (Lyman, [0324]: “Aggregate performance data for each user can be generate and/or updated based on past accuracy and/or efficiency scores, based on how many scans have been annotated in total, based on measured improvement of the user over time, etc”).
Lyman does not appear to explicitly disclose “group of users.”
However, Tanner discloses “determine that at least a portion of... one or more values for... one or more metrics are associated with a group of users (Tanner, [0073]: “The annotation in FIG. 7B was performed by three annotators, which generated polygons 710B, 720B, and 730B. The intersection area is again 1 pixel in size”; Examiner notes that the intersection area of 1 pixel in size corresponds to a value for a metric associated with a group of users); and
determining, based at least on the portion of the one or more values of the one or more metrics, a score indicating an accuracy associated with the group of users” (Tanner, [0075]: “Table 2 calculates the annotation precision using the I/U method as well as the I/APOT method” and [0076]: “Having a metric for annotation accuracy that has includes greater precision has multiple applications: It allows for the quantitative comparison of a group of annotators compared to another group of annotators if both groups annotate the same images” and [0077]: “The Metric of intersection divided by the average pixels on target and the metric of intersection divided by union are measures of accuracy. However, in the context of multiple independent annotations, I/APOT is more precise than I/U”; Examiner notes that the annotation accuracy which is calculated using the intersection area corresponds to “a score indicating an accuracy associated with the group of users”).
Tanner and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Lyman with the teachings of Tanner such that “a user” is instead “a group of users,” and one would have been motivated to do so, as doing so would allow for the quantitative comparison of a group of annotators compared to another group of annotators if both groups annotate the same images, which would provide insight on the effects that different imagery characteristics have on the ability of people to accurately annotate targets, or the relative difficulty of a particular annotation (see Tanner, [0076]).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Lyman in view of Kwant.
Regarding claim 16, the rejection of claim 6 is incorporated. Lyman does not appear to explicitly disclose the further limitations of the claim.
However, Kwant discloses “determine, based at least on... one or more values for... one or more metrics, a first score associated with a first accuracy of training data that includes at least a portion of... one or more first sensor representations (Kwant, [0036-0037]: “In one embodiment, the system 100 uses the manually trained feature detector 103 on the manually labeled training data set (e.g., the training images) to generate automatically marked labels for the data items or images in the training data set. The system 100 then compares the automatically marked labels to the manually marked labels to identify any differences between the two sets of labels. In one embodiment, discrepancies between the manually marked labels and the automatically marked labels can indicate potential QA issues. For example, the system 100 can identify images or other data items in the training set whose manually marked labels and automatically marked labels differ by a predetermined threshold or criterion as having potential QA issues. In one embodiment, the system 100 can use any quality metric to determine the differences between the manually marked labels and the automatically marked labels. For example, when the labeled features are objects or features that can be represented as polygons, examples of such a quality metric can include, but is not limited to, a distance metric that measures the distance (e.g., in pixels or some other unit of measure) between the manually labeled polygons and the corresponding automatically labeled polygons”; Examiner notes that training images corresponds to “training data that includes at least a portion of one or more first sensor representations,” quality metric corresponds to “a first score associated with an accuracy” and the distance between the manually labeled polygons and the automatically labeled polygons corresponds to “one or more values for one or more metrics”);
determine a second score associated a second accuracy corresponding to one or one or more machine learning models (Kwant, [0035]: “To address these challenges, the system 100 of FIG. 1 introduces a capability to perform an automated QA process on the training data that is used to train a feature prediction model” and [0037]: “...predetermined precision criteria”; Examiner notes that the predetermined precision criteria corresponds to one or more machine learning models because it is an accuracy requirement for the training data used to train a feature prediction model); and
determining, based at least on the first score and the second score, whether the training data satisfies the second accuracy corresponding to the one or more machine learning models (Kwant, [0037]: “Based on the computed difference or precision between the two data sets, the system 100 can perform any type of number of QA processes such as, but not limited to: (1) automatically filtering data items (e.g., images) from the training set that do not meet predetermined precision criteria, and then retraining the feature detector 103 with the training data set remaining after filtering”).
Kwant and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Lyman with the teachings of Kwant to include the one or more processors being further to “determine, based at least on the one or more values for the one or more metrics, a first score associated with a first accuracy of training data that includes at least a portion of the one or more first sensor representations; determine a second score associated a second accuracy corresponding to one or one or more machine learning models; and determining, based at least on the first score and the second score, whether the training data satisfies the second accuracy corresponding to the one or more machine learning models,” and one would have been motivated to do so, as doing so would automate the quality assurance process for training data, leading to lower resource utilization and costs and improved training data quality (see Kwant, [0033-0034]).
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Lyman in view of Kwant and Swaninathan et al. (US20250355779) (hereinafter “Swaninathan”).
Regarding claim 17, the rejection of claim 16 is incorporated. Neither Lyman nor Kwant appear to explicitly disclose the further limitations of the claim.
However, Swaninathan discloses “determining, based at least on one or more product requirements corresponding to... one or more machine learning models, one or more second values associated with... [a] second accuracy” (Swaninathan, [0006]: “In a first aspect there is provided an apparatus comprising means for determining, for a given use case, a behavioural requirement policy for a machine learning model, means for providing an indication of the behavioural requirement policy to an analytics producer and means for receiving, from the analytics producer, a performance evaluation metric determined based on the behavioural requirement policy” and [0012]: “The performance evaluation metric may comprise at least one of precision, accuracy, recall, f1-score, mean squared error, mean absolute error and root mean squared error”; Examiner notes that the behavioural requirement policy for a machine learning model corresponds to “one or more product requirements corresponding to one or more machines learning models” and the value of the “accuracy” metric corresponds to “one or more second values associated with a second accuracy”); and
determining... [a] second score based at least on the one or more second values” (Swaninathanm, [0012]: “The performance evaluation metric may comprise at least one of precision, accuracy, recall, f1-score, mean squared error, mean absolute error and root mean squared error”; Examiner notes that “performance evaluation metric” corresponds to a second score).
Swaninathan and the instant application both relate to machine learning and are analogous. It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the combination of Lyman/Kwant with the teachings of Swaninathan such that the determination of the second score comprises “determining, based at least on one or more product requirements corresponding to the one or more machine learning models, one or more second values associated with the second accuracy; and determining the second score based at least on the one or more second values,” and one would have been motivated to do so, as doing so would allow for acheiving the best possible model for a given task (see Swaninathan, [0086]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GWYNEVERE A DETERDING whose telephone number is (571) 272-7657. The examiner can normally be reached Mon-Fri. 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/G.A.D./
Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights