DETAILED ACTION
This communication is in response to the amendment filed 09/11/25 in which claims 1, 8, 15, 21, and 27 were amended. Claims 1—33 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 09/11/25 has been entered.
Response to Arguments
With respect to the 103 rejection, Applicant argues:
Bernard in view of Chari does not teach to "determine respective amounts of unlabeled training data to select from different categories of training data according to a predefined target distribution of training data across the different categories when the selected unlabeled training data is combined with a set of already-labeled training data, wherein the different categories are determined from the predefined target distribution; automatically select the determined respective amounts of unlabeled training data based, at least in part, on metadata associated with the unlabeled training data indicative of the different categories; obtain labels for the automatically selected unlabeled training data to produce automatically selected labeled training data; and cause one or more neural networks to be trained based, at least in part, on the automatically selected labeled training data," as recited in amended claim 1. Applicant has amended claim 1 to clarify that the target distribution is predefined and the different categories are determined from the predefined target distribution. For example, as illustrated in Applicant's, the target distribution is received as an input to the system. Newly-cited Chari works in a fundamentally different manner. Instead of determining how to select unlabeled training data to meet a predefined target distribution where the categories are determined from the predefined target distribution, Chari re-categories (performs clustering) and resets the distribution for each iteration of Chari's process. See Chari, FIG. 1. For example, as shown at 106 in FIG. 1 and described at [0031], Chari re-clusters (re-categorizes) the data at each iteration. Thus, the clusters or categories used in Chari are explicitly not predefined. Moreover, Chari does not determine respective amounts of unlabeled training data to select from different categories of training data according to a predefined target distribution. Instead, as described at [0032], Chari determines a number of samples to drawn from each newly-determined cluster based on distributions recalculated at each iteration as an "estimation of class distribution in the previously labeled sample set" and a "class distribution of previously labeled samples in each cluster." Thus, Chari's process works in a manner explicitly contrary to Applicant's amended claim. Bernard is not cited for and does not teach these aspects of Applicant's amended claim 1. Accordingly, claim 1 is patentable over Bernard in view of Chari. Similar remarks apply with respect to amended independent claims 8 and 27. Similarly, with respect to amended independent claims 15 and 21, Bernard in view of Chari does not teach to "training data automatically selected, based at least in part, on metadata associated with the training data to have a distribution satisfying a predefined target distribution of training data across different categories when the automatically selected training data is combined with training data previously used to train the one or more neural networks, wherein the different categories are determined from the predefined target distribution."
Applicant’s arguments with respect to claims 1, 8, 15, 21, and 27 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Specifically, Bernard is remapped in combination with a new reference to Weider to teach the argued limitations.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1—33 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more:
[Claim 1] A processor, [Step 1: processor is a machine and therefore constitutes a statutory category] comprising:
one or more circuits to: [Step 2A, prong 2/Step 2B: generic computer component recited at a high level of generality is a mere instruction to apply the exception under MPEP 2106.05(f)]
determine respective amounts of unlabeled training data to select from different categories of training data according to a predefined target distribution of training data across the different categories when the selected unlabeled training data is combined with a set of already-labeled training data, wherein the different categories are determined from the predefined target distribution; [Step 2A, prong 1: determining amounts of data according to a target distribution may be performed manually by human judgement and, therefore, is a mental process]
automatically select the determined respective amounts of unlabeled training data based, at least in part, on metadata associated with the unlabeled training data indicative of the different categories; [Step 2A, prong 1: selecting amounts of training data based on metadata may be performed manually by human judgment and, therefore, is a mental process; Step 2A, prong 2: “automatically” is mere instruction to apply the exception under MPEP 2106.05(f)]
obtain labels for the automatically selected unlabeled training data [Step 2A, prong 2: obtaining labels is data gathering/insignificant extra-solution activity under MPEP 2106.05(g); Step 2B: receiving data is well understood routine and conventional under MPEP 2106.05(d)] to produce automatically selected labeled training data; and [Step 2A, prong 1: producing selected labeled training may be performed manually and, therefore, is a mental process; Step 2A prong 2: “automatically” is mere instruction to apply the exception under MPEP 2106.05(f)]
cause one or more neural networks to be trained based, at least in part, on the automatically selected labeled training data [Step 2A prong 2: using training data to train a neural network recited at a high level of generality is mere instruction to apply the exception under MPEP 2106.05(f)].
[Claim 2] The processor of claim 1, [Step 1: processor is a machine and therefore constitutes a statutory category] wherein the one or more circuits are further to: [Step 2A, prong 2/Step 2B: generic computer component recited at a high level of generality is a mere instruction to apply the exception under MPEP 2106.05(f)]
obtain a plurality of unlabeled training data; [Step 2A prong 2: data gathering is insignificant extra-solution activity under MPEP 2106.05(g)]
process the plurality of unlabeled training data into a set of groups based at least in part on the metadata, wherein each group of the set of groups corresponds to a different category of the different categories; and [Step 2A prong 1: processing unlabeled training data into a set of groups based on metadata may be performed manually and constitutes a mental process]
wherein to automatically select the determined respective amounts of the unlabeled training data the one or more circuits are further to automatically select the determined respective amounts based at least in part on the set of groups [Step 2A prong 2/Step 2B: recitation of “the one or more circuits are further to automatically” is mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)].
[Claim 3] The processor of claim 2, [Step 1: processor is a machine and therefore constitutes a statutory category] wherein a first group of the set of groups corresponds to a first combination of metadata values [Step 2A prong 1: processing unlabeled training data into a set of groups based on metadata may be performed manually and constitutes a mental process].
[Claim 4] The processor of claim 2, [Step 1: processor is a machine and therefore constitutes a statutory category] wherein the one or more circuits to [Step 2A prong 2/Step 2B: mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)] determine the respective amounts of unlabeled training data to select from different categories of training data are further to use one or more equation solvers to automatically determine respective amounts [Step 2A prong 1: determining an amount of training data to select to use equation solvers to determine the amounts recites a mental process; Step 2A prong 2/Step 2B: using equation solvers to automatically determine the amounts recites a mere instruction to apply the exception at a high level using a generic computer component].
[Claim 5] The processor of claim 1, [Step 1: processor is a machine and therefore constitutes a statutory category] wherein the one or more circuits are further to: [Step 2A prong 2/Step 2B: mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)]
train the one or more neural networks using at least the set of already-labeled training data and the automatically selected labeled training data [Step 2A prong 2/Step 2B: mere instruction to apply the exception in training step recited at a high level of generality].
[Claim 6] The processor of claim 1, [Step 1: processor is a machine and therefore constitutes a statutory category] wherein the unlabeled training data comprises one or more images captured from one or more vehicles [Step 2A prong 2/Step 2B: the type/source of data does not cause the data gathering activity to integrate the exception into a practical application under MPEP 2106.05(g)].
[Claim 7] The processor of claim 1, [Step 1: processor is a machine and therefore constitutes a statutory category] wherein the metadata indicates one or more operational design domain (ODD) values [Step 2A prong 2/Step 2B: generally linking the exception to a particular technological environment (operational design)].
[Claim 8] A system, comprising: [Step 1: system is a machine and therefore constitutes a statutory category]
one or more computers having one or more processors to: [Step 2A prong 2/Step 2B: mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)]
determine respective amounts of unlabeled training data to select from different categories of training data according to a predefined target distribution of training data across the different categories when the selected unlabeled training data is combined with a set of already-labeled training data, wherein the different categories are determined from the predefined target distribution; [Step 2A, prong 1: determining amounts of data according to a target distribution may be performed manually by human judgement and, therefore, is a mental process]
automatically select the determined respective amounts of unlabeled training data based, at least in part, on metadata associated with the unlabeled training data indicative of the different categories; [Step 2A, prong 1: selecting amounts of training data based on metadata may be performed manually by human judgment and, therefore, is a mental process; Step 2A, prong 2: “automatically” is mere instruction to apply the exception under MPEP 2106.05(f)]
obtain labels for the automatically selected unlabeled training data [Step 2A, prong 2: obtaining labels is data gathering/insignificant extra-solution activity under MPEP 2106.05(g); Step 2B: receiving data is well understood routine and conventional under MPEP 2106.05(d)] to produce automatically selected labeled training data; and [Step 2A, prong 1: producing selected labeled training may be performed manually and, therefore, is a mental process; Step 2A prong 2: “automatically” is mere instruction to apply the exception under MPEP 2106.05(f)]
cause one or more neural networks to be trained based, at least in part, on the automatically selected labeled training data [Step 2A prong 2: using training data to train a neural network recited at a high level of generality is mere instruction to apply the exception under MPEP 2106.05(f)].
[Claim 9] The system of claim 8, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the one or more processors are further to: [Step 2A prong 2/Step 2B: mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)]
obtain a set of unlabeled training data and associated metadata; [Step 2A prong 2: data gathering is insignificant extra-solution activity under MPEP 2106.05(g)]
parse the set of unlabeled training data to calculate subsets of unlabeled training data based at least in part on the associated metadata, wherein each subset of the set of groups corresponds to a different category of the different categories; and [Step 2A prong 1: processing unlabeled training data into a set of groups based on metadata may be performed manually and constitutes a mental process]
wherein to determine the respective amounts of unlabeled training data to select from different categories the one or more processors are to use one or more equation solvers to calculate numbers of assets for the subsets of unlabeled training data [Step 2A prong 1: determining an amount of training data to select to use equation solvers to determine the amounts recites a mental process; Step 2A prong 2/Step 2B: using equation solvers to automatically determine the amounts recites a mere instruction to apply the exception at a high level using a generic computer component].
[Claim 10] The system of claim 9, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the one or more processors are further to automatically [Step 2A prong 2/Step 2B: mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)] select the unlabeled training data from the subsets of unlabeled training data based at least in part on the calculated numbers of assets [Step 2A prong 1: mental process involving evaluation, opinion, judgment].
[Claim 11] The system of claim 9, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the numbers of assets are calculated based at least in part on one or more target proportions of the target distribution [Step 2A prong 1: mathematical calculation].
[Claim 12] The system of claim 8, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the unlabeled training data comprises one or more images captured from one or more medical devices [Step 2A prong 2/Step 2B: the type/source of data does not cause the data gathering activity to integrate the exception into a practical application under MPEP 2106.05(g)].
[Claim 13] The system of claim 8, [Step 1: system is a machine and therefore constitutes a statutory category] wherein output data of the one or more neural networks comprises one or more classifications of one or more objects depicted in one or more images [Step 2A prong 1: type of outputted data does not cause judicial exception to be integrated into a practical application; Step 2B: data outputting is well understood routine and conventional under MPEP 2106.05(g)].
[Claim 14] The system of claim 13, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the automatically selected labeled training data is automatically selected to cause the one or more neural networks to [Step 2A prong 2/Step 2B: mere instruction to apply exception using generic computer components] generate output data having one or more attributes comprising one or more accuracy values corresponding to the one or more classifications [Step 2A prong 2: data outputting/transmission is insignificant extra-solution activity; type of data does not cause the outputting step to integrate the judicial exception into a practical application; Step 2B: data outputting/transmission is well understood routine and conventional under MPEP 2106.05(d)].
[Claim 15] A processor, comprising: [Step 1: processor is a machine and therefore constitutes a statutory category]
one or more circuits to use one or more neural networks to [Step 2A prong 2/Step 2B: mere instruction to apply exception using generic computer components] generate output data [Step 2A prong 1: data outputting constitutes insignificant extra-solution activity; Step 2B: data outputting is well understood routine and conventional under MPEP 2106.05(g)] based, at least in part, on training data [Step 2A prong 2: data inputting is insignificant extra solution activity; Step 2B: receiving data is well understood routine and conventional under MPEP 2106.05(d)] automatically selected, based at least in part, on metadata associated with the training data to have a distribution satisfying a predefined target distribution of training data across different categories when the automatically selected training data is combined with training data previously used to train the one or more neural networks, wherein the different categories are determined from the predefined target distribution [Step 2A, prong 1: selecting amounts of training data based on metadata may be performed manually by human judgment and, therefore, is a mental process; Step 2A, prong 2/Step 2B: “automatically” is mere instruction to apply the exception under MPEP 2106.05(f)].
[Claim 16] The processor of claim 15, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the one or more circuits are further to:
obtain one or more images depicting one or more objects; and [Step 2A prong 2: data gathering is insignificant extra-solution activity under MPEP 2106.05(g); Step 2B: receiving data is well understood routine and conventional under MPEP 2106.05(d)]
use the one or more neural networks to generate the output data based on the one or more images [Step 2A prong 2/Step 2B: recitation of using one or more neural networks constitutes mere instruction to apply the exception under MPEP 2106.05 (f)].
[Claim 17] The processor of claim 16, [Step 1: system is a machine and therefore constitutes a statutory category] wherein:
the one or more neural networks include one or more object detection neural networks; and [Step 2A prong 2/Step 2B: recitation of “object detection” refers to field of use and technological environment (object detection) and does not cause integration into a practical application under MPEP 2106.05(h)]
the output data comprises data indicating one or more locations of the one or more objects [Step 2A prong 2: type of data outputted does not cause the data outputting to integrate the exception into a practical application under MPEP 2106.05(g); Step 2B: data transmitting/outputting is well understood routine and conventional under MPEP 2106.05(d)].
[Claim 18] The processor of claim 15, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the metadata indicates one or more conditions of the training data [Step 2A prong 2/Step 2B: the type of metadata does not cause the mental process to be practically integrated].
[Claim 19] The processor of claim 15, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the one or more circuits are further to [Step 2A prong 2/Step 2B: recitation of circuits is mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)] use one or more equation solvers to automatically select the training data, wherein the one or more equation solvers are based at least in part on a linear formulation [Step 2A prong 2/Step 2B: mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)].
[Claim 20] The processor of claim 15, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the training data comprises sensor data [Step 2A prong 2: type of training data does not cause the data inputting to be practically integrated under MPEP 2106.05(g); Step 2B: data inputting/receiving is well understood routine and conventional under MPEP 2106.05(d)].
[Claim 21] A system, comprising: [Step 1: system is a machine and therefore constitutes a statutory category]
one or more computers having one or more processors to [Step 2A prong 2/Step 2B: recitation of computers/processors is mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)] use one or more neural networks to [Step 2A prong 2/Step 2B: recitation of neural networks is mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)] generate output data [Step 2A prong 2: data outputting constitutes insignificant extra-solution activity; Step 2B: data outputting is well understood routine and conventional under MPEP 2106.05(g)] based, at least in part, on training data [Step 2A prong 2: data inputting is insignificant extra solution activity; Step 2B: receiving data is well understood routine and conventional under MPEP 2106.05(d)] automatically selected to have a distribution satisfying a predefined target distribution of training data across different categories when the automatically selected training data is combined with training data previously used to train the one or more neural networks, wherein the different categories are determined from the predefined target distribution [Step 2A, prong 1: selecting training data to cause output data to have one or more attributes may be performed manually by human judgment and, therefore, is a mental process; Step 2A, prong 2/Step 2B: “automatically” is mere instruction to apply the exception under MPEP 2106.05(f)].
[Claim 22] The system of claim 21, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the one or more processors are further to [Step 2A prong 2/Step 2B: recitation of computers/processors is mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)] use one or more solvers to automatically [Step 2A prong 2/Step 2B: recitation of solvers is mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)] select the training data, [Step 2A prong 1: Mental process] wherein the one or more solvers are based at least in part on a quadratic formulation [Step 2A prong 1: quadratic formulation refers to a numerical algorithm and therefore constitutes a mathematical concept].
[Claim 23] The system of claim 21, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the one or more processors are further to [Step 2A prong 2/Step 2B: recitation of computers/processors is mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)] use the one or more neural networks to [Step 2A prong 2/Step 2B: recitation of neural networks is mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)] generate the output data based at least in part on a set of images [Step 2A prong 2: data outputting is insignificant extra-solution activity under MPEP 2106.05(g); Step 2B: data transmitting is well understood routine and conventional under MPEP 2106.05(d)].
[Claim 24] The system of claim 21, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the output data comprises one or more results of the one or more neural networks [Step 2A prong 2: source of data does not cause data outputting to be practically integrated under MPEP 2106.05(g); transmitting data is well understood routine and conventional under MPEP 2106.05(d)].
[Claim 25] The system of claim 24, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the one or more attributes indicate one or more confidence values for the one or more results [Step 2A prong 2: type of data outputted does not cause the data outputting to practically integrate the exception under MPEP 2106.05(g); Step 2B: data transmitting is well understood routine and conventional under MPEP 2106.05(d)].
[Claim 26] The system of claim 21, [Step 1: system is a machine and therefore constitutes a statutory category] wherein the training data comprises one or more frames of one or more videos [Step 2A prong 2: type of data inputted does not cause the data inputting to practically integrate the exception under MPEP 2106.05(g); Step 2B: data receiving is well understood routine and conventional under MPEP 2106.05(d)].
[Claim 27] A machine-readable medium having stored thereon a set of instructions, [Step 1: machine-readable medium comprising instructions is a composition of matter and therefore constitutes a statutory category] which if performed by one or more processors, cause the one or more processors to at least: [Step 2A prong 2/Step 2B: recitation of computers/processors is mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)]
determine respective amounts of unlabeled training data to select from different categories of training data according to a predefined target distribution of training data across the different categories when the selected unlabeled training data is combined with a set of already-labeled training data, wherein the different categories are determined from the predefined target distribution; [Step 2A, prong 1: determining amounts of data according to a target distribution may be performed manually by human judgement and, therefore, is a mental process]
automatically select the determined respective amounts of unlabeled training data based, at least in part, on metadata associated with the unlabeled training data indicative of the different categories; [Step 2A, prong 1: selecting unlabeled training data based on metadata may be performed manually by human judgement and, therefore, is a mental process]
obtain labels for the automatically selected unlabeled training data; and [Step 2A prong 2: data gathering is insignificant extra-solution activity under MPEP 2106.05(g); Step 2B: receiving data is well understood routine and conventional under MPEP 2106.05(d)]
cause one or more neural networks to be trained using training data [Step 2A prong 2/Step 2B: recitation of neural networks to be trained using training data constitutes instruction to apply the exception using generic computer components recited at a high level of generality under MPEP 2106.05(f)] automatically selected based, at least in part, on metadata associated with the training data [Step 2A, prong 1: may be performed manually by human judgment and, therefore, is a mental process; Step 2A, prong 2/Step 2B: “automatically” is mere instruction to apply the exception under MPEP 2106.05(f)].
[Claim 28]. The machine-readable medium of claim 27, wherein the set of instructions further comprise instructions, [Step 1: machine-readable medium comprising instructions is a composition of matter and therefore constitutes a statutory category] which if performed by the one or more processors, cause the one or more processors to: [Step 2A prong 2/Step 2B: recitation of computers/processors is mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)]
obtain a distribution indicating one or more clauses; and [Step 2A prong 2: data inputting/receiving is insignificant extra-solution activity under MPEP 2106.05(g); type of data does not cause the data inputting to integrate the exception into a practical application under MPEP 2106.05(g); receiving data is well understood routine and conventional under MPEP 2106.05(d)]
automatically select the training data based at least in part on one or more proportions indicated by the one or more clauses [Step 2A prong 1: mental process; Step 2A prong 2/Step 2B: “automatically” is mere instruction to apply the exception under MPEP 2106.05(f)].
[Claim 29] The machine-readable medium of claim 27, wherein the set of instructions further comprise instructions, [Step 1: machine-readable medium comprising instructions is a composition of matter and therefore constitutes a statutory category] which if performed by the one or more processors, cause the one or more processors to: [Step 2A prong 2/Step 2B: recitation of computers/processors is mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)]
obtain one or more collections of assets corresponding to one or more operational design domain (ODD) values; [Step 2A prong 2: data inputting/receiving is insignificant extra-solution activity under MPEP 2106.05(g); type of data does not cause the data inputting to integrate the exception into a practical application under MPEP 2106.05(g); receiving data is well understood routine and conventional under MPEP 2106.05(d)]
select a first asset from a first collection of assets; [Step 2A prong 1: mental process]
compute one or more distances between the first asset and the first collection of assets; and [Step 2A prong 1: mathematical calculation]
select a second asset based at least in part on the one or more distances, wherein the training data comprises the first asset and the second asset [Step 2A prong 1: mental process].
[Claim 30] The machine-readable medium of claim 29, [Step 1: machine-readable medium comprising instructions is a composition of matter and therefore constitutes a statutory category] wherein the one or more distances are based on a temporal distance or a spatial distance [Step 2A prong 1: mathematical calculation].
[Claim 31] The machine-readable medium of claim 27, [Step 1: machine-readable medium comprising instructions is a composition of matter and therefore constitutes a statutory category] wherein the training data is selected based at least in part on labelled training data used to train the one or more neural networks [Step 2A prong 1: mental process].
[Claim 32] The machine-readable medium of claim 27, [Step 1: machine-readable medium comprising instructions is a composition of matter and therefore constitutes a statutory category] wherein the set of instructions further comprise instructions, which if performed by the one or more processors, cause the one or more processors to: [Step 2A prong 2/Step 2B: recitation of computers/processors is mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)]
provide the training data to one or more labelling entities [Step 2A prong 2: data outputting is insignificant extra-solution activity under MPEP 2106.05(g); Step 2B: data transmission is well understood routine and conventional under MPEP 2106.05(d)] to obtain one or more labels [Step 2A prong 2: data inputting is insignificant extra-solution activity under MPEP 2106.05(g); Step 2B: receiving data is well understood routine and conventional under MPEP 2106.05(d)];
cause the one or more neural networks to [Step 2A prong 2/Step 2B: recitation of neural networks is mere instruction to apply the exception using generic computer components under MPEP 2106.05(f)] process the training data to calculate one or more results [Step 2A prong 1: mathematical calculations]; and
update the one or more neural networks based at least in part on the one or more results and the one or more labels [Step 2A prong 2/Step 2B: updating neural network based on training data is how neural networks are trained and is mere instruction to apply the exception under MPEP 2106.05(f)].
[Claim 33] The machine-readable medium of claim 27, [Step 1: machine-readable medium comprising instructions is a composition of matter and therefore constitutes a statutory category] wherein the training data comprises one or more images captured from one or more autonomous devices [Step 2A prong 2: type of input data or source of data does not cause the data inputting to practically integrate the exception under MPEP 2106.05(g); Step 2B: receiving data is well understood routine and conventional under MPEP 2106.05(d)].
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 1, 5, 7-8, 12—25, 27—29, and 31-32 are rejected under 35 U.S.C. 103 as being unpatentable over Bernard (US 10,140,421 B1; published Nov. 27, 2018) in view of Chari (US 2013/0097103 A1; published Apr. 18, 2013).
Regarding claim 1, Bernard discloses [a] processor, comprising:
one or more circuits to: (72:30-34 (“The medical scan image analysis system can include a processing system that includes a processor and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations.”))
Bernard does not expressly disclose determine respective amounts of unlabeled training data to select from different categories of training data…(but see Chari ¶ 32 (“Once the data is clustered, in step 108, a number of data points (samples) to be selected (draw) from each cluster is determined. First, the number of desired samples to draw for each class is determined based on the estimation of class distribution in the previously labeled sample set. This process is described in detail below, however, in general this step determines the class distribution of previously labeled samples regardless of their membership to particular clusters. From this information, it is determined how many samples to select for each class. Using strategies for re-sampling, members of minority classes are over-sampled and members of majority classes under-sampled to converge on a balanced sample. Next, the class distribution of previously labeled samples in each cluster is computed. Then, based on the two class distributions, the number of desired samples to draw from each cluster is determined.”))
automatically select the determined respective amounts of unlabeled training data based, at least in part, on metadata associated with the unlabeled training data indicative of the different categories; (but see Chari ¶ 33 (“Finally, to minimize any sample bias introduced by the semi-supervised clustering, in step 110, maximum entropy sampling is performed to draw samples from each cluster. Drawing samples from a small number of clusters to ensure balancedness introduces a risk of drawing samples that are too similar to previous samples. Maximum entropy sampling ensures a diverse sample population for classifier training.”))
obtain labels for the automatically selected unlabeled training data to produce automatically selected labeled training data; and (but see Chari ¶ 33 (“The samples chosen from the clusters are then labeled and added to the training data set, and as highlighted above methodology 100 can be repeated until a desired amount of training data is obtained.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bernard to incorporate the teachings of Chari to generate balanced and class-independent training data from the training set of medical scans to train the medical scan image analysis function, at least because doing so would improve the performance of the function. See Chari ¶ 2.
Bernard does not expressly disclose that the unlabeled samples are selected from different categories according to a predefined target distribution of training data across the different categories when the selected unlabeled training data is combined with a set of already-labeled training data, wherein the different categories are determined from the predefined target distribution; (but see Weider ¶ 41 (“FIGS. 3A-3B depict example bar charts for displaying distribution in data to show how bias can be present in a dataset and affect outcome of a model. FIG. 3A displays a bar chart 300A that depicts an ideal distribution of data in a dataset based on a gender attribute of the dataset. This assumes that one of the attributes of a datapoint in the dataset is gender and gender is categorized by three categories: female, male and non-binary. The example also assumes that the dataset is used to train a model for determining loan approvals. For such a dataset, an ideal distribution based on gender may result in a female bar 310 that has an equal distribution to the male bar 320 and the non-binary bar 330. This means the number of datapoints that represent each of the categories of the gender attribute may be equal or be within a predetermined distribution threshold. As a result, the percentage of loans approved for people falling into each category may also be equal. Thus, the model trained by this dataset may generate outcomes that are consistent across the gender spectrum (e.g. 10% of loans submitted by applicants in each category are approved).”), ¶ 42 (“The ideal distribution depicted in FIG. 3A, however, rarely occurs in the real world. Often the dataset is representative of one category more than others. FIG. 3B depicts a bar chart 300B displaying a more realistic real-world distribution of data across the gender spectrum in a dataset. The bar chart 300B shows the female bar 340 represents 35% of the data, while the male bar 350 represents 55% of the data and the non-binary bar chart 360 represents only 10% of data. This shows a clear imbalanced distribution of data across the three categories. When such an imbalanced dataset is used to train a ML mode, the outcome is often severely biased. FIG. 3C depicts a bar chart 300C displaying such an outcome. The female bar 370 of bar chart 300C shows that the ML model rejects 97% of female applicants, while the male bar 380 displays how only 3% of the male applicants are rejected by the ML model. As the non-binary bar 390 shows, the percentage of people falling into the non-binary category that are rejected is even higher than the female applicants, with a 99% rejection rate. As such, imbalanced or biased distribution of input data in a dataset can significantly impact the outcome produced by a ML model trained with the imbalanced dataset.”), ¶ 43 (“To address such imbalanced distributions, the input dataset may be trimmed to select a subset of the dataset that represents a more balanced distribution. For example, the subset may be selected based on the size of the category having the smallest distribution. Referring to the imbalanced distribution of FIG. 3B, this may mean choosing the size of the non-binary category as the measuring point and selecting a dataset that corresponds with data in each of the female and male categories in numbers that are equal to or within a desired distribution of the non-binary category. For example, if the non-binary category includes 1000 datapoints from a total of 10,000 datapoints for the entire dataset, a subset may be selected such that each of the female, male and non-binary categories has 1000 datapoints. This is illustrated in FIG. 4 which depicts a bar chart 400 displaying a distribution of data across the gender spectrum in a corrected subset of data. As shown in FIG. 4, because of trimming of the dataset, the resulting subset shows a balanced distribution of data across the three categories of the spectrum. As a result, each of the categories of the corrected subset has about 33% of the data. In one implementation, after a correction is performed, the bias detection tool may be executed again to ensure that the new subset achieves its purposes and it does not generate new undesired imbalance in data. The process may be repeated iteratively until an acceptable corrected dataset is achieved.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Bernard to incorporate the teachings of Weider to choose the size of each category of unlabeled data according to a desired target distribution of the overall dataset, at least because doing so would correct datasets of labeled and unlabeled data for which bias is detected. See Weider ¶ 37 (“Datasets for which bias is examined, detected, and corrected by the bias detection and correction service may be used for training ML models by a training mechanism 224. The training mechanism 224 may use training datasets stored in the datastore 212 to provide initial and/or ongoing training for ML models. In one implementation, the training mechanism 224 may use labeled training data from the data store 212 train the ML models. The initial training may be performed in an offline or online stage. In another example, the training mechanism 224 may utilize unlabeled training data from the datastore 212 to train the ML model via an unsupervised learning mechanism. Unsupervised learning may allow the ML model to create and/or output its own labels. In an example, an unsupervised learning mechanism may apply reinforcement learning to maximize a given value function or achieve a desired goal.”).
Bernard further discloses cause one or more neural networks to be trained based, at least in part, on the automatically selected labeled training data (72:52-61 (“A training set of medical scans used to train one more medical scan image analysis functions can be received from one or more client devices via the network and/or can be retrieved from the medical scan database 342, for example, based on training set data 621 corresponding to medical scan image analysis functions. Training set criteria, for example, identified in training parameters 620 of the medical scan image analysis function, can be utilized to automatically identify and select medical scans to be included in the training set from a plurality of available medical scans.”)).
Regarding claim 5, Bernard, in view of Chari and Weider, discloses the invention of claim 1 as discussed above. Bernard further discloses wherein the one or more circuits are further to:
train the one or more neural networks using at least the set of already-labeled training data and the automatically selected labeled training data (73:59-62 (the plurality of training sets are created to generate corresponding neural network models)).
Regarding claim 7, Bernard, in view of Chari and Weider, discloses the invention of claim 1 as discussed above. Bernard further discloses wherein the metadata indicates one or more operational design domain (ODD) values (73:17-32 (the training set criteria can include requirements for one or more medical scan classification data)).
Regarding claim 8, Bernard discloses [a] system, comprising:
one or more computers having one or more processors to: (72:30-34 (“The medical scan image analysis system can include a processing system that includes a processor and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations.”)).
Bernard does not expressly disclose determine respective amounts of unlabeled training data to select from different categories of training data… (but see Chari ¶ 32 (“Once the data is clustered, in step 108, a number of data points (samples) to be selected (draw) from each cluster is determined. First, the number of desired samples to draw for each class is determined based on the estimation of class distribution in the previously labeled sample set. This process is described in detail below, however, in general this step determines the class distribution of previously labeled samples regardless of their membership to particular clusters. From this information, it is determined how many samples to select for each class. Using strategies for re-sampling, members of minority classes are over-sampled and members of majority classes under-sampled to converge on a balanced sample. Next, the class distribution of previously labeled samples in each cluster is computed. Then, based on the two class distributions, the number of desired samples to draw from each cluster is determined.”))
automatically select the determined respective amounts of unlabeled training data based, at least in part, on metadata associated with the unlabeled training data indicative of the different categories; (but see Chari ¶ 33 (“Finally, to minimize any sample bias introduced by the semi-supervised clustering, in step 110, maximum entropy sampling is performed to draw samples from each cluster. Drawing samples from a small number of clusters to ensure balancedness introduces a risk of drawing samples that are too similar to previous samples. Maximum entropy sampling ensures a diverse sample population for classifier training.”))
obtain labels for the automatically selected unlabeled training data to produce automatically selected labeled training data; and (but see Chari ¶ 33 (“The samples chosen from the clusters are then labeled and added to the training data set, and as highlighted above methodology 100 can be repeated until a desired amount of training data is obtained.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bernard to incorporate the teachings of Chari to generate balanced and class-independent training data from the training set of medical scans to train the medical scan image analysis function, at least because doing so would improve the performance of the function. See Chari ¶ 2.
Bernard does not expressly disclose that the unlabeled samples are selected from different categories according to a predefined target distribution of training data across the different categories when the selected unlabeled training data is combined with a set of already-labeled training data, wherein the different categories are determined from the predefined target distribution (but see Weider ¶ 41 (“FIGS. 3A-3B depict example bar charts for displaying distribution in data to show how bias can be present in a dataset and affect outcome of a model. FIG. 3A displays a bar chart 300A that depicts an ideal distribution of data in a dataset based on a gender attribute of the dataset. This assumes that one of the attributes of a datapoint in the dataset is gender and gender is categorized by three categories: female, male and non-binary. The example also assumes that the dataset is used to train a model for determining loan approvals. For such a dataset, an ideal distribution based on gender may result in a female bar 310 that has an equal distribution to the male bar 320 and the non-binary bar 330. This means the number of datapoints that represent each of the categories of the gender attribute may be equal or be within a predetermined distribution threshold. As a result, the percentage of loans approved for people falling into each category may also be equal. Thus, the model trained by this dataset may generate outcomes that are consistent across the gender spectrum (e.g. 10% of loans submitted by applicants in each category are approved).”), ¶ 42 (“The ideal distribution depicted in FIG. 3A, however, rarely occurs in the real world. Often the dataset is representative of one category more than others. FIG. 3B depicts a bar chart 300B displaying a more realistic real-world distribution of data across the gender spectrum in a dataset. The bar chart 300B shows the female bar 340 represents 35% of the data, while the male bar 350 represents 55% of the data and the non-binary bar chart 360 represents only 10% of data. This shows a clear imbalanced distribution of data across the three categories. When such an imbalanced dataset is used to train a ML mode, the outcome is often severely biased. FIG. 3C depicts a bar chart 300C displaying such an outcome. The female bar 370 of bar chart 300C shows that the ML model rejects 97% of female applicants, while the male bar 380 displays how only 3% of the male applicants are rejected by the ML model. As the non-binary bar 390 shows, the percentage of people falling into the non-binary category that are rejected is even higher than the female applicants, with a 99% rejection rate. As such, imbalanced or biased distribution of input data in a dataset can significantly impact the outcome produced by a ML model trained with the imbalanced dataset.”), ¶ 43 (“To address such imbalanced distributions, the input dataset may be trimmed to select a subset of the dataset that represents a more balanced distribution. For example, the subset may be selected based on the size of the category having the smallest distribution. Referring to the imbalanced distribution of FIG. 3B, this may mean choosing the size of the non-binary category as the measuring point and selecting a dataset that corresponds with data in each of the female and male categories in numbers that are equal to or within a desired distribution of the non-binary category. For example, if the non-binary category includes 1000 datapoints from a total of 10,000 datapoints for the entire dataset, a subset may be selected such that each of the female, male and non-binary categories has 1000 datapoints. This is illustrated in FIG. 4 which depicts a bar chart 400 displaying a distribution of data across the gender spectrum in a corrected subset of data. As shown in FIG. 4, because of trimming of the dataset, the resulting subset shows a balanced distribution of data across the three categories of the spectrum. As a result, each of the categories of the corrected subset has about 33% of the data. In one implementation, after a correction is performed, the bias detection tool may be executed again to ensure that the new subset achieves its purposes and it does not generate new undesired imbalance in data. The process may be repeated iteratively until an acceptable corrected dataset is achieved.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Bernard to incorporate the teachings of Weider to choose the size of each category of unlabeled data according to a desired target distribution of the overall dataset, at least because doing so would correct datasets of labeled and unlabeled data for which bias is detected. See Weider ¶ 37 (“Datasets for which bias is examined, detected, and corrected by the bias detection and correction service may be used for training ML models by a training mechanism 224. The training mechanism 224 may use training datasets stored in the datastore 212 to provide initial and/or ongoing training for ML models. In one implementation, the training mechanism 224 may use labeled training data from the data store 212 train the ML models. The initial training may be performed in an offline or online stage. In another example, the training mechanism 224 may utilize unlabeled training data from the datastore 212 to train the ML model via an unsupervised learning mechanism. Unsupervised learning may allow the ML model to create and/or output its own labels. In an example, an unsupervised learning mechanism may apply reinforcement learning to maximize a given value function or achieve a desired goal.”).
Bernard further discloses cause one or more neural networks to be trained based, at least in part, on the automatically selected labeled training data (72:52-61 (“A training set of medical scans used to train one more medical scan image analysis functions can be received from one or more client devices via the network and/or can be retrieved from the medical scan database 342, for example, based on training set data 621 corresponding to medical scan image analysis functions. Training set criteria, for example, identified in training parameters 620 of the medical scan image analysis function, can be utilized to automatically identify and select medical scans to be included in the training set from a plurality of available medical scans.”), 80:62-81:5 (“The inferred output vector of the inference data 1370 can include a plurality of abnormality probabilities mapped to a pixel location of each of a plurality of cross-sectional image slices of the new medical scan. For example, the inferred output vector can indicate a set of probability matrices 1371, where each matrix in the set corresponds to one of the plurality of image slices of the medical scan, where each matrix is a size corresponding to the number of pixels in each image slice, where each cell of each matrix corresponds to a pixel of the corresponding image slice, whose value is the abnormality probability of the corresponding pixel.”)).
Regarding claim 12, Bernard, in view of Chari and Weider, discloses the invention of claim 8 as discussed above. Bernard further discloses wherein the unlabeled training data comprises one or more images captured from one or more medical devices (72:52-55 (training set of medical scans received along with training set criteria)).
Regarding claim 13, Bernard, in view of Chari and Weider, discloses the invention of claim 8 as discussed above. Bernard further discloses wherein output data of the one or more neural networks comprises one or more classifications of one or more objects depicted in one or more images (78:25-31 (output feature vector includes data that will be inferred in a subsequent medical scan input and can include single output value indicating whether or not the medical scan includes an abnormality)).
Regarding claim 14, Bernard, in view of Chari and Weider, discloses the invention of claim 13 as discussed above. Bernard further discloses wherein the automatically selected labeled training data is automatically selected to cause the one or more neural networks to generate output data having one or more attributes comprising one or more accuracy values corresponding to the one or more classifications (80:62-81:5 (“The inferred output vector of the inference data 1370 can include a plurality of abnormality probabilities mapped to a pixel location of each of a plurality of cross-sectional image slices of the new medical scan. For example, the inferred output vector can indicate a set of probability matrices 1371, where each matrix in the set corresponds to one of the plurality of image slices of the medical scan, where each matrix is a size corresponding to the number of pixels in each image slice, where each cell of each matrix corresponds to a pixel of the corresponding image slice, whose value is the abnormality probability of the corresponding pixel.”)).
Regarding claim 15, Bernard discloses [a] processor, comprising:
one or more circuits (72:30-34 (“The medical scan image analysis system can include a processing system that includes a processor and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations.”)) to use one or more neural networks to generate output data based, at least in part, on training data automatically selected, based at least in part, on metadata associated with the training data (72:52-61 (“A training set of medical scans used to train one more medical scan image analysis functions can be received from one or more client devices via the network and/or can be retrieved from the medical scan database 342, for example, based on training set data 621 corresponding to medical scan image analysis functions. Training set criteria, for example, identified in training parameters 620 of the medical scan image analysis function, can be utilized to automatically identify and select medical scans to be included in the training set from a plurality of available medical scans.”), 80:62-81:5 (“The inferred output vector of the inference data 1370 can include a plurality of abnormality probabilities mapped to a pixel location of each of a plurality of cross-sectional image slices of the new medical scan. For example, the inferred output vector can indicate a set of probability matrices 1371, where each matrix in the set corresponds to one of the plurality of image slices of the medical scan, where each matrix is a size corresponding to the number of pixels in each image slice, where each cell of each matrix corresponds to a pixel of the corresponding image slice, whose value is the abnormality probability of the corresponding pixel.”)).
Bernard does not expressly disclose that the unlabeled samples are selected from different categories to have a distribution satisfying a predefined target distribution of training data across different categories when the automatically selected training data is combined with training data previously used to train the one or more neural networks, wherein the different categories are determined from the predefined target distribution (but see Weider ¶ 41 (“FIGS. 3A-3B depict example bar charts for displaying distribution in data to show how bias can be present in a dataset and affect outcome of a model. FIG. 3A displays a bar chart 300A that depicts an ideal distribution of data in a dataset based on a gender attribute of the dataset. This assumes that one of the attributes of a datapoint in the dataset is gender and gender is categorized by three categories: female, male and non-binary. The example also assumes that the dataset is used to train a model for determining loan approvals. For such a dataset, an ideal distribution based on gender may result in a female bar 310 that has an equal distribution to the male bar 320 and the non-binary bar 330. This means the number of datapoints that represent each of the categories of the gender attribute may be equal or be within a predetermined distribution threshold. As a result, the percentage of loans approved for people falling into each category may also be equal. Thus, the model trained by this dataset may generate outcomes that are consistent across the gender spectrum (e.g. 10% of loans submitted by applicants in each category are approved).”), ¶ 42 (“The ideal distribution depicted in FIG. 3A, however, rarely occurs in the real world. Often the dataset is representative of one category more than others. FIG. 3B depicts a bar chart 300B displaying a more realistic real-world distribution of data across the gender spectrum in a dataset. The bar chart 300B shows the female bar 340 represents 35% of the data, while the male bar 350 represents 55% of the data and the non-binary bar chart 360 represents only 10% of data. This shows a clear imbalanced distribution of data across the three categories. When such an imbalanced dataset is used to train a ML mode, the outcome is often severely biased. FIG. 3C depicts a bar chart 300C displaying such an outcome. The female bar 370 of bar chart 300C shows that the ML model rejects 97% of female applicants, while the male bar 380 displays how only 3% of the male applicants are rejected by the ML model. As the non-binary bar 390 shows, the percentage of people falling into the non-binary category that are rejected is even higher than the female applicants, with a 99% rejection rate. As such, imbalanced or biased distribution of input data in a dataset can significantly impact the outcome produced by a ML model trained with the imbalanced dataset.”), ¶ 43 (“To address such imbalanced distributions, the input dataset may be trimmed to select a subset of the dataset that represents a more balanced distribution. For example, the subset may be selected based on the size of the category having the smallest distribution. Referring to the imbalanced distribution of FIG. 3B, this may mean choosing the size of the non-binary category as the measuring point and selecting a dataset that corresponds with data in each of the female and male categories in numbers that are equal to or within a desired distribution of the non-binary category. For example, if the non-binary category includes 1000 datapoints from a total of 10,000 datapoints for the entire dataset, a subset may be selected such that each of the female, male and non-binary categories has 1000 datapoints. This is illustrated in FIG. 4 which depicts a bar chart 400 displaying a distribution of data across the gender spectrum in a corrected subset of data. As shown in FIG. 4, because of trimming of the dataset, the resulting subset shows a balanced distribution of data across the three categories of the spectrum. As a result, each of the categories of the corrected subset has about 33% of the data. In one implementation, after a correction is performed, the bias detection tool may be executed again to ensure that the new subset achieves its purposes and it does not generate new undesired imbalance in data. The process may be repeated iteratively until an acceptable corrected dataset is achieved.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Bernard to incorporate the teachings of Weider to choose the size of each category of unlabeled data according to a desired target distribution of the overall dataset, at least because doing so would correct datasets of labeled and unlabeled data for which bias is detected. See Weider ¶ 37 (“Datasets for which bias is examined, detected, and corrected by the bias detection and correction service may be used for training ML models by a training mechanism 224. The training mechanism 224 may use training datasets stored in the datastore 212 to provide initial and/or ongoing training for ML models. In one implementation, the training mechanism 224 may use labeled training data from the data store 212 train the ML models. The initial training may be performed in an offline or online stage. In another example, the training mechanism 224 may utilize unlabeled training data from the datastore 212 to train the ML model via an unsupervised learning mechanism. Unsupervised learning may allow the ML model to create and/or output its own labels. In an example, an unsupervised learning mechanism may apply reinforcement learning to maximize a given value function or achieve a desired goal.”).
Regarding claim 16, Bernard, in view of Weider, discloses the invention of claim 15 as discussed above. Bernard further discloses wherein the one or more circuits are further to:
obtain one or more images depicting one or more objects; and use the one or more neural networks to generate the output data based on the one or more images (80:62-81:5 (“The inferred output vector of the inference data 1370 can include a plurality of abnormality probabilities mapped to a pixel location of each of a plurality of cross-sectional image slices of the new medical scan. For example, the inferred output vector can indicate a set of probability matrices 1371, where each matrix in the set corresponds to one of the plurality of image slices of the medical scan, where each matrix is a size corresponding to the number of pixels in each image slice, where each cell of each matrix corresponds to a pixel of the corresponding image slice, whose value is the abnormality probability of the corresponding pixel.”)).
Regarding claim 17, Bernard, in view of Weider, discloses the invention of claim 16 as discussed above. Bernard further discloses wherein:
the one or more neural networks include one or more object detection neural networks; and (74:15-18 (“each of the medical scan image analysis functions associated with each neural network model can correspond to one of the plurality of neural network models generated by the medical scan image analysis system”))
the output data comprises data indicating one or more locations of the one or more objects (80:62-81:5 (“The inferred output vector of the inference data 1370 can include a plurality of abnormality probabilities mapped to a pixel location of each of a plurality of cross-sectional image slices of the new medical scan. For example, the inferred output vector can indicate a set of probability matrices 1371, where each matrix in the set corresponds to one of the plurality of image slices of the medical scan, where each matrix is a size corresponding to the number of pixels in each image slice, where each cell of each matrix corresponds to a pixel of the corresponding image slice, whose value is the abnormality probability of the corresponding pixel.”)).
Regarding claim 18, Bernard, in view of Weider, discloses the invention of claim 15 as discussed above. Bernard further discloses wherein the metadata indicates one or more conditions of the training data (73:1-2 (“requirements for medical scans in the training set such as requiring that the medical scan is assigned a truth flag”)).
Regarding claim 19, Bernard, in view of Weider, discloses the invention of claim 15 as discussed above. Bernard further discloses wherein the one or more circuits are further to use one or more equation solvers to automatically select the training data, wherein the one or more equation solvers are based at least in part on a linear formulation (73:17-32 (training set criteria can include quota or proportion requirements for one or more medical scan classification data)).
Regarding claim 20, Bernard, in view of Weider, discloses the invention of claim 15 as discussed above. Bernard further discloses wherein the training data comprises sensor data (5:25-33 (“FIG. 4A presents an embodiment of a medical scan entry 352, stored in medical scan database 342, included in metadata of a medical scan, and/or otherwise associated with a medical scan. A medical scan can include imaging data corresponding to a CT scan, x-ray, MRI, PET scan, Ultrasound, EEG, mammogram, or other type of radiological scan or medical scan taken of an anatomical region of a human body, animal, organism, or object and further can include metadata corresponding to the imaging data.”)).
Regarding claim 21, Bernard discloses [a] system, comprising:
one or more computers having one or more processors (72:30-34 (“The medical scan image analysis system can include a processing system that includes a processor and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations.”)) to use one or more neural networks to generate output data based, at least in part, on training data automatically selected (72:52-61 (“A training set of medical scans used to train one more medical scan image analysis functions can be received from one or more client devices via the network and/or can be retrieved from the medical scan database 342, for example, based on training set data 621 corresponding to medical scan image analysis functions. Training set criteria, for example, identified in training parameters 620 of the medical scan image analysis function, can be utilized to automatically identify and select medical scans to be included in the training set from a plurality of available medical scans.”), 80:62-81:5 (“The inferred output vector of the inference data 1370 can include a plurality of abnormality probabilities mapped to a pixel location of each of a plurality of cross-sectional image slices of the new medical scan. For example, the inferred output vector can indicate a set of probability matrices 1371, where each matrix in the set corresponds to one of the plurality of image slices of the medical scan, where each matrix is a size corresponding to the number of pixels in each image slice, where each cell of each matrix corresponds to a pixel of the corresponding image slice, whose value is the abnormality probability of the corresponding pixel.”)).
Bernard does not expressly disclose that the unlabeled samples are selected from different categories to have a distribution satisfying a predefined target distribution of training data across different categories when the automatically selected training data is combined with training data previously used to train the one or more neural networks, wherein the different categories are determined from the predefined target distribution (but see Weider ¶ 41 (“FIGS. 3A-3B depict example bar charts for displaying distribution in data to show how bias can be present in a dataset and affect outcome of a model. FIG. 3A displays a bar chart 300A that depicts an ideal distribution of data in a dataset based on a gender attribute of the dataset. This assumes that one of the attributes of a datapoint in the dataset is gender and gender is categorized by three categories: female, male and non-binary. The example also assumes that the dataset is used to train a model for determining loan approvals. For such a dataset, an ideal distribution based on gender may result in a female bar 310 that has an equal distribution to the male bar 320 and the non-binary bar 330. This means the number of datapoints that represent each of the categories of the gender attribute may be equal or be within a predetermined distribution threshold. As a result, the percentage of loans approved for people falling into each category may also be equal. Thus, the model trained by this dataset may generate outcomes that are consistent across the gender spectrum (e.g. 10% of loans submitted by applicants in each category are approved).”), ¶ 42 (“The ideal distribution depicted in FIG. 3A, however, rarely occurs in the real world. Often the dataset is representative of one category more than others. FIG. 3B depicts a bar chart 300B displaying a more realistic real-world distribution of data across the gender spectrum in a dataset. The bar chart 300B shows the female bar 340 represents 35% of the data, while the male bar 350 represents 55% of the data and the non-binary bar chart 360 represents only 10% of data. This shows a clear imbalanced distribution of data across the three categories. When such an imbalanced dataset is used to train a ML mode, the outcome is often severely biased. FIG. 3C depicts a bar chart 300C displaying such an outcome. The female bar 370 of bar chart 300C shows that the ML model rejects 97% of female applicants, while the male bar 380 displays how only 3% of the male applicants are rejected by the ML model. As the non-binary bar 390 shows, the percentage of people falling into the non-binary category that are rejected is even higher than the female applicants, with a 99% rejection rate. As such, imbalanced or biased distribution of input data in a dataset can significantly impact the outcome produced by a ML model trained with the imbalanced dataset.”), ¶ 43 (“To address such imbalanced distributions, the input dataset may be trimmed to select a subset of the dataset that represents a more balanced distribution. For example, the subset may be selected based on the size of the category having the smallest distribution. Referring to the imbalanced distribution of FIG. 3B, this may mean choosing the size of the non-binary category as the measuring point and selecting a dataset that corresponds with data in each of the female and male categories in numbers that are equal to or within a desired distribution of the non-binary category. For example, if the non-binary category includes 1000 datapoints from a total of 10,000 datapoints for the entire dataset, a subset may be selected such that each of the female, male and non-binary categories has 1000 datapoints. This is illustrated in FIG. 4 which depicts a bar chart 400 displaying a distribution of data across the gender spectrum in a corrected subset of data. As shown in FIG. 4, because of trimming of the dataset, the resulting subset shows a balanced distribution of data across the three categories of the spectrum. As a result, each of the categories of the corrected subset has about 33% of the data. In one implementation, after a correction is performed, the bias detection tool may be executed again to ensure that the new subset achieves its purposes and it does not generate new undesired imbalance in data. The process may be repeated iteratively until an acceptable corrected dataset is achieved.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Bernard to incorporate the teachings of Weider to choose the size of each category of unlabeled data according to a desired target distribution of the overall dataset, at least because doing so would correct datasets of labeled and unlabeled data for which bias is detected. See Weider ¶ 37 (“Datasets for which bias is examined, detected, and corrected by the bias detection and correction service may be used for training ML models by a training mechanism 224. The training mechanism 224 may use training datasets stored in the datastore 212 to provide initial and/or ongoing training for ML models. In one implementation, the training mechanism 224 may use labeled training data from the data store 212 train the ML models. The initial training may be performed in an offline or online stage. In another example, the training mechanism 224 may utilize unlabeled training data from the datastore 212 to train the ML model via an unsupervised learning mechanism. Unsupervised learning may allow the ML model to create and/or output its own labels. In an example, an unsupervised learning mechanism may apply reinforcement learning to maximize a given value function or achieve a desired goal.”).
Regarding claim 22, Bernard, in view of Weider, discloses the invention of claim 21 as discussed above. Bernard further discloses wherein the one or more processors are further to use one or more solvers to automatically select the training data, wherein the one or more solvers are based at least in part on a quadratic formulation (74:6-12 (“The medical scan classifications selected to segregate the medical scans can be automatically determined by the medical scan image analysis system, for example, where an unsupervised clustering algorithm is applied to the original training set to determine appropriate medical scan classifications based on the output of the unsupervised clustering algorithm.”) (An unsupervised clustering algorithm such as k-means employs a quadratic formulation of distance clustering.).
Regarding claim 23, Bernard, in view of Weider, discloses the invention of claim 21 as discussed above. Bernard further discloses wherein the one or more processors are further to use the one or more neural networks to generate the output data based at least in part on a set of images (80:62-81:5 (“The inferred output vector of the inference data 1370 can include a plurality of abnormality probabilities mapped to a pixel location of each of a plurality of cross-sectional image slices of the new medical scan. For example, the inferred output vector can indicate a set of probability matrices 1371, where each matrix in the set corresponds to one of the plurality of image slices of the medical scan, where each matrix is a size corresponding to the number of pixels in each image slice, where each cell of each matrix corresponds to a pixel of the corresponding image slice, whose value is the abnormality probability of the corresponding pixel.”)).
Regarding claim 24, Bernard, in view of Weider, discloses the invention of claim 21 as discussed above. Bernard further discloses wherein the output data comprises one or more results of the one or more neural networks (80:62-81:5 (“The inferred output vector of the inference data 1370 can include a plurality of abnormality probabilities mapped to a pixel location of each of a plurality of cross-sectional image slices of the new medical scan. For example, the inferred output vector can indicate a set of probability matrices 1371, where each matrix in the set corresponds to one of the plurality of image slices of the medical scan, where each matrix is a size corresponding to the number of pixels in each image slice, where each cell of each matrix corresponds to a pixel of the corresponding image slice, whose value is the abnormality probability of the corresponding pixel.”)).
Regarding claim 25, Bernard, in view of Weider, discloses the invention of claim 24 as discussed above. Bernard further discloses wherein the automatically selected training data is automatically selected to have one or more attributes that indicate one or more confidence values for the one or more results (80:62-81:5 (“The inferred output vector of the inference data 1370 can include a plurality of abnormality probabilities mapped to a pixel location of each of a plurality of cross-sectional image slices of the new medical scan. For example, the inferred output vector can indicate a set of probability matrices 1371, where each matrix in the set corresponds to one of the plurality of image slices of the medical scan, where each matrix is a size corresponding to the number of pixels in each image slice, where each cell of each matrix corresponds to a pixel of the corresponding image slice, whose value is the abnormality probability of the corresponding pixel.”)).
Regarding claim 27, Bernard discloses [a] machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least: (72:30-34 (“The medical scan image analysis system can include a processing system that includes a processor and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations.”)).
Bernard does not expressly disclose determine respective amounts of unlabeled training data to select from different categories of training data… (but see Chari ¶ 32 (“Once the data is clustered, in step 108, a number of data points (samples) to be selected (draw) from each cluster is determined. First, the number of desired samples to draw for each class is determined based on the estimation of class distribution in the previously labeled sample set. This process is described in detail below, however, in general this step determines the class distribution of previously labeled samples regardless of their membership to particular clusters. From this information, it is determined how many samples to select for each class. Using strategies for re-sampling, members of minority classes are over-sampled and members of majority classes under-sampled to converge on a balanced sample. Next, the class distribution of previously labeled samples in each cluster is computed. Then, based on the two class distributions, the number of desired samples to draw from each cluster is determined.”))
automatically select the determined respective amounts of unlabeled training data based, at least in part, on metadata associated with the unlabeled training data indicative of the different categories; (but see Chari ¶ 33 (“Finally, to minimize any sample bias introduced by the semi-supervised clustering, in step 110, maximum entropy sampling is performed to draw samples from each cluster. Drawing samples from a small number of clusters to ensure balancedness introduces a risk of drawing samples that are too similar to previous samples. Maximum entropy sampling ensures a diverse sample population for classifier training.”))
obtain labels for the automatically selected unlabeled training data; and (but see Chari ¶ 33 (“The samples chosen from the clusters are then labeled and added to the training data set, and as highlighted above methodology 100 can be repeated until a desired amount of training data is obtained.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bernard to incorporate the teachings of Chari to generate balanced and class-independent training data from the training set of medical scans to train the medical scan image analysis function, at least because doing so would improve the performance of the function. See Chari ¶ 2.
Bernard does not expressly disclose that the unlabeled samples are selected from different categories according to a predefined target distribution of training data across the different categories when the selected unlabeled training data is combined with a set of already-labeled training data, wherein the different categories are determined from the predefined target distribution (but see Weider ¶ 41 (“FIGS. 3A-3B depict example bar charts for displaying distribution in data to show how bias can be present in a dataset and affect outcome of a model. FIG. 3A displays a bar chart 300A that depicts an ideal distribution of data in a dataset based on a gender attribute of the dataset. This assumes that one of the attributes of a datapoint in the dataset is gender and gender is categorized by three categories: female, male and non-binary. The example also assumes that the dataset is used to train a model for determining loan approvals. For such a dataset, an ideal distribution based on gender may result in a female bar 310 that has an equal distribution to the male bar 320 and the non-binary bar 330. This means the number of datapoints that represent each of the categories of the gender attribute may be equal or be within a predetermined distribution threshold. As a result, the percentage of loans approved for people falling into each category may also be equal. Thus, the model trained by this dataset may generate outcomes that are consistent across the gender spectrum (e.g. 10% of loans submitted by applicants in each category are approved).”), ¶ 42 (“The ideal distribution depicted in FIG. 3A, however, rarely occurs in the real world. Often the dataset is representative of one category more than others. FIG. 3B depicts a bar chart 300B displaying a more realistic real-world distribution of data across the gender spectrum in a dataset. The bar chart 300B shows the female bar 340 represents 35% of the data, while the male bar 350 represents 55% of the data and the non-binary bar chart 360 represents only 10% of data. This shows a clear imbalanced distribution of data across the three categories. When such an imbalanced dataset is used to train a ML mode, the outcome is often severely biased. FIG. 3C depicts a bar chart 300C displaying such an outcome. The female bar 370 of bar chart 300C shows that the ML model rejects 97% of female applicants, while the male bar 380 displays how only 3% of the male applicants are rejected by the ML model. As the non-binary bar 390 shows, the percentage of people falling into the non-binary category that are rejected is even higher than the female applicants, with a 99% rejection rate. As such, imbalanced or biased distribution of input data in a dataset can significantly impact the outcome produced by a ML model trained with the imbalanced dataset.”), ¶ 43 (“To address such imbalanced distributions, the input dataset may be trimmed to select a subset of the dataset that represents a more balanced distribution. For example, the subset may be selected based on the size of the category having the smallest distribution. Referring to the imbalanced distribution of FIG. 3B, this may mean choosing the size of the non-binary category as the measuring point and selecting a dataset that corresponds with data in each of the female and male categories in numbers that are equal to or within a desired distribution of the non-binary category. For example, if the non-binary category includes 1000 datapoints from a total of 10,000 datapoints for the entire dataset, a subset may be selected such that each of the female, male and non-binary categories has 1000 datapoints. This is illustrated in FIG. 4 which depicts a bar chart 400 displaying a distribution of data across the gender spectrum in a corrected subset of data. As shown in FIG. 4, because of trimming of the dataset, the resulting subset shows a balanced distribution of data across the three categories of the spectrum. As a result, each of the categories of the corrected subset has about 33% of the data. In one implementation, after a correction is performed, the bias detection tool may be executed again to ensure that the new subset achieves its purposes and it does not generate new undesired imbalance in data. The process may be repeated iteratively until an acceptable corrected dataset is achieved.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Bernard to incorporate the teachings of Weider to choose the size of each category of unlabeled data according to a desired target distribution of the overall dataset, at least because doing so would correct datasets of labeled and unlabeled data for which bias is detected. See Weider ¶ 37 (“Datasets for which bias is examined, detected, and corrected by the bias detection and correction service may be used for training ML models by a training mechanism 224. The training mechanism 224 may use training datasets stored in the datastore 212 to provide initial and/or ongoing training for ML models. In one implementation, the training mechanism 224 may use labeled training data from the data store 212 train the ML models. The initial training may be performed in an offline or online stage. In another example, the training mechanism 224 may utilize unlabeled training data from the datastore 212 to train the ML model via an unsupervised learning mechanism. Unsupervised learning may allow the ML model to create and/or output its own labels. In an example, an unsupervised learning mechanism may apply reinforcement learning to maximize a given value function or achieve a desired goal.”).
Bernard further discloses cause one or more neural networks to be trained based, at least in part, on the automatically selected training data and obtained labels (72:52-61 (“A training set of medical scans used to train one more medical scan image analysis functions can be received from one or more client devices via the network and/or can be retrieved from the medical scan database 342, for example, based on training set data 621 corresponding to medical scan image analysis functions. Training set criteria, for example, identified in training parameters 620 of the medical scan image analysis function, can be utilized to automatically identify and select medical scans to be included in the training set from a plurality of available medical scans.”)).
Regarding claim 28, Bernard, in view of Chari and Weider, discloses the invention of claim 27 as discussed above. Bernard further discloses wherein the set of instructions further comprise instructions, which if performed by the one or more processors, cause the one or more processors to:
obtain the target distribution indicating one or more clauses; and (73:17-32 (“The training set criteria can include quota and/or proportion requirements for one or more medical scan classification data.”))
determine the respective amounts of unlabeled select the training data based at least in part on one or more proportions indicated by the one or more clauses (72:57-61 (“Training set criteria, for example, identified in training parameters 620 of the medical scan image analysis function, can be utilized to automatically identify and select medical scans to be included in the training set from a plurality of available medical scans.”)).
Regarding claim 29, Bernard, in view of Chari and Weider, discloses the invention of claim 27 as discussed above. Bernard does not expressly disclose wherein to automatically select a determined amount of the determined respective amounts of the determined respective amounts of unlabeled training data the set of instructions further comprise instructions, which if performed by the one or more processors, cause the one or more processors to:
select a first asset of unlabeled training data from a first category of the different categories; (but see Chari ¶ 29 (“In step 102, a small set of data (e.g., from about 5% to about 10% of the desired training data set), is selected (sampled) from Data Set U.”))
compute based, at least in part, on the metadata, one or more distances between the first asset and other assets of the first category; and (but see Chari ¶ 31 (“The remaining samples to be labeled are picked in an iterative fashion, where each iteration produces a fraction of the desired sample size. In each iteration, semi-supervised clustering is applied to the data, incorporating the labeled samples from previous iterations. See step 106. As is known in the art, semi-supervised clustering employs both labeled (e.g., known labels from the previous iterations) and unlabeled data for training. Specifically, in step 106, the data from Data Set U is clustered using a semi-supervised clustering process.”))
select a second asset from the first category based at least in part on the one or more distances, wherein the automatically selected unlabeled training data comprises the first asset and the second asset (but see Chari ¶ 31 (“The remaining samples to be labeled are picked in an iterative fashion, where each iteration produces a fraction of the desired sample size. In each iteration, semi-supervised clustering is applied to the data, incorporating the labeled samples from previous iterations. See step 106. As is known in the art, semi-supervised clustering employs both labeled (e.g., known labels from the previous iterations) and unlabeled data for training. Specifically, in step 106, the data from Data Set U is clustered using a semi-supervised clustering process.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bernard to incorporate the teachings of Chari to generate balanced and class-independent training data from the training set of medical scans to train the medical scan image analysis function, at least because doing so would improve the performance of the function. See Chari ¶ 2.
Regarding claim 31, Bernard, in view of Chari and Weider, discloses the invention of claim 27 as discussed above. Bernard further discloses wherein the part on labelled set of already-labeled training data was used to train the one or more neural networks (72:5-12 (“A medical scan image analysis system 112 can be used to generate and/or perform one or more medical scan image analysis functions by utilizing a computer vision-based learning algorithm on a training set of medical scans with known annotation data, diagnosis data, labeling and/or medical code data, . . .”)).
Regarding claim 32, Bernard, in view of Chari and Weider, discloses the invention of claim 27 as discussed above. Bernard further discloses wherein the set of instructions further comprise instructions, which if performed by the one or more processors, cause the one or more processors to:
provide the automatically selected unlabeled training data to one or more labelling entities to obtain the labels; (45:44-46 (“the medical scan report can be transmitted to a client device associated with an expert user or other medical professional”))
cause the one or more neural networks to process the automatically selected training data to calculate one or more results; and (79:9-21 (“In particular, determining the model parameter data 1355 can include utilizing a backpropagation strategy. The forward propagation algorithm can be performed on at least one input feature vector corresponding to at least one medical scan in the training set to propagate the at least one input feature vector through the plurality of neural network layers based on initial and/or default model parameter data 1355, such as an initial weight vector of initial weight values set by an administrator or chosen at random. The at least one output vector generated by performing the forward propagation algorithm on the at least one input feature vector can be compared to the corresponding at least one known output feature vector to determine an output error.”))
update the one or more neural networks based at least in part on the one or more results and the obtained labels (79:28-30 (“Next, gradient descent can be performed to determine an updated weight vector based on the output error or average output error.”)).
Claims 2 and 3 are rejected under 35 U.S.C. 103 as being unpatentable over Bernard, Chari, and Weider as applied to claim 1 above, and further in view of Tang (US 2019/0279293 A1; published Sep. 12, 2019).
Regarding claim 2, Bernard, in view of Chari and Weider, discloses the invention of claim 1 as discussed above. Bernard further discloses wherein the one or more circuits are further to:
obtain a plurality of unlabeled training data; (72:57-61 (“plurality of available medical scans”)).
Bernard teaches that training set criteria, identified in training parameters, can be utilized to automatically identify and select medical scans to be included in the training set, see 72:57-61, but does not expressly disclose process the plurality of unlabeled training data into a set of groups based at least in part on the metadata, wherein each group of the set of groups corresponds to a different category of the different categories; and (but see Tang ¶ 132 (“In step 1610, server system 105 may determine whether the associated metadata includes a word of interest based on the preliminary search. If the metadata does not include a word of interest (step 1610: No), server system 105 may continue to step 1612, classifying the image as unlabeled, and/or discarding it. If instead the metadata includes a word of interest (step 1610: Yes), server system 105 may continue to step 1614 and perform a second search for attributes in metadata. A “word of interest” may include any vehicle-related word such as “exterior,” “interior,” “car,” “vehicle,” “front view,” “side view,” “rear view,” “Ford,” “Honda,” “2006,” “2007,” “XL,” “4L,” “horsepower,” or the like. Such staggered search for metadata improves computer functionality by minimizing the resources that are used to classify images creating a stratified approach that quickly identifies images that are not of interest and then devoting more resources to images that overcome an initial threshold. Indeed, because the determination in step 1610 may be performed with specialized computers, such as FPGAs, specifically programmed to perform the keyword search in metadata, process 1600 may improve the overall speed of the server system 105.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bernard to incorporate the teachings of Tang to use image file metadata to classify images, at least because doing so would enable quickly label unlabeled images. See Tang ¶ 132.
Bernard further discloses wherein to automatically select the determined respective amounts of the unlabeled training data the one or more circuits are further to automatically select the determined respective amounts based at least in part on the set of groups (72-57-61 (the training set criteria are used to automatically identify and select medical scans to be included in the training set)).
Regarding claim 3, Bernard, in view of Chari, Weider, and Tang, discloses the invention of claim 2 as discussed above. Bernard further discloses wherein a first group of the set of groups corresponds to a first combination of metadata values (72:61-65 (“The training set criteria can be automatically generated based on, for example, previously learned criteria, and/or training set criteria can be received via the network, for example, from an administrator of the medical scan image analysis system. The training set criteria can include a minimum training set size. The training set criteria can include data integrity requirements for medical scans in the training set such as requiring that the medical scan is assigned a truth flag 461, requiring that performance score data 530 for a hospital and/or medical professional associated with the medical scan compares favorably to a performance score threshold, requiring that the medical scan has been reviewed by at least a threshold number of medical professionals, requiring that the medical scan and/or a diagnosis corresponding to a patient file of the medical scan is older than a threshold elapsed time period, or based on other criteria intended to insure that the medical scans and associated data in the training set is reliable enough to be considered “truth” data.”)).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Bernard, Chari, Weider, and Tang as applied to claim 2 above, and further in view of Frossard (US 2019/0147610 A1; published May 16, 2019).
Regarding claim 4, Bernard, in view of Chari, Weider, and Tang, discloses the invention of claim 2 as discussed above. Chari teaches computing the number of samples to select using a mathematical calculation, see ¶ 48, but neither Bernard nor Chari expressly disclose wherein the one or more circuits to determine the respective amounts of unlabeled training data to select from different categories of training data are further to use one or more equation solvers to automatically determine respective amounts (but see Frossard ¶ 88 (“In some embodiments, a min cost flow problem can be used as an alternative formulation for the linear program. A min cost flow problem can be solved using Bellman-Ford and/or Successive Shortest Paths (SSP) techniques. The same solution may be achieved with these techniques in example embodiments. In some implementations, a Gurobi solver can be used to solve a constrained linear program problem.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bernard to incorporate the teachings of Frossard to use a Gurobi solver to calculate the number of samples to select from each cluster, at least because doing so would enable performing the calculation with an off-the-shelf computation tool.
Claims 9—11 are rejected under 35 U.S.C. 103 as being unpatentable over Bernard, Chari, and Weider, as applied to claim 8 above, and further in view of Tang and Frossard.
Regarding claim 9, Bernard, in view of Chari and Weider, discloses the invention of claim 8 as discussed above. Bernard further discloses wherein the one or more processors are further to:
obtain a set of unlabeled training data and associated metadata; (72:52-55 (training set of medical scans received along with training set criteria)).
Bernard teaches that training set criteria, identified in training parameters, can be utilized to automatically identify and select medical scans to be included in the training set, see 72:57-61, but does not expressly disclose parse the set of unlabeled training data to calculate subsets of unlabeled training data based at least in part on the associated metadata, wherein each subset of the set of groups corresponds to a different category of the different categories; and (but see Tang ¶ 132 (“In step 1610, server system 105 may determine whether the associated metadata includes a word of interest based on the preliminary search. If the metadata does not include a word of interest (step 1610: No), server system 105 may continue to step 1612, classifying the image as unlabeled, and/or discarding it. If instead the metadata includes a word of interest (step 1610: Yes), server system 105 may continue to step 1614 and perform a second search for attributes in metadata. A “word of interest” may include any vehicle-related word such as “exterior,” “interior,” “car,” “vehicle,” “front view,” “side view,” “rear view,” “Ford,” “Honda,” “2006,” “2007,” “XL,” “4L,” “horsepower,” or the like. Such staggered search for metadata improves computer functionality by minimizing the resources that are used to classify images creating a stratified approach that quickly identifies images that are not of interest and then devoting more resources to images that overcome an initial threshold. Indeed, because the determination in step 1610 may be performed with specialized computers, such as FPGAs, specifically programmed to perform the keyword search in metadata, process 1600 may improve the overall speed of the server system 105.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bernard to incorporate the teachings of Tang to use image file metadata to classify images, at least because doing so would enable quickly label unlabeled images. See Tang ¶ 132.
Chari teaches computing the number of samples to select using a mathematical calculation, see ¶ 48, but neither Bernard nor Chari expressly disclose wherein to determine the respective amounts of unlabeled training data to select from different categories the one or more processors are to use one or more equation solvers to calculate numbers of assets for the subsets of unlabeled training data (but see Frossard ¶ 88 (“In some embodiments, a min cost flow problem can be used as an alternative formulation for the linear program. A min cost flow problem can be solved using Bellman-Ford and/or Successive Shortest Paths (SSP) techniques. The same solution may be achieved with these techniques in example embodiments. In some implementations, a Gurobi solver can be used to solve a constrained linear program problem.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bernard to incorporate the teachings of Frossard to use a Gurobi solver to calculate the number of samples to select from each cluster, at least because doing so would enable performing the calculation with an off-the-shelf computation tool.
Regarding claim 10, Bernard, in view of Chari, Weider, Tang, and Frossard, discloses the invention of claim 9 as discussed above. Bernard further discloses wherein the one or more processors are further to automatically select the unlabeled training data from the subsets of unlabeled training data based at least in part on the calculated numbers of assets (72:57-61 (training set criteria are used to automatically identify and select medical scans to be included in the training set)).
Regarding claim 11, Bernard, in view of Chari, Weider, Tang, and Frossard, discloses the invention of claim 9 as discussed above. Bernard further discloses wherein the numbers of assets are calculated based at least in part on one or more target proportions of the target distribution (73:17-32 (training set criteria can include quota or proportion requirements for one or more medical scan classification data)).
Claim 30 is rejected under 35 U.S.C. 103 as being unpatentable over Bernard and Weider, as applied to claim 29 above, and further in view of Guo (US 2011/0137903 A1; published Jun. 9, 2011).
Regarding claim 30, Bernard, in view of Weider, discloses the invention of claim 29 as discussed above. Bernard does not expressly disclose wherein the one or more distances are based on a temporal distance or a spatial distance (but see Guo ¶ 7 (“The second type of methods incorporates spatial distance explicitly in the similarity measure for a general clustering method (e.g., K-Means) and thus data items in the same cluster tend to be spatially close to each other.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bernard to incorporate the teachings of Guo to cluster the unlabeled data during semi-supervised clustering based on a spatial distance of the images, at least because doing so would enable an optimal grouping of the images into spatially contiguous regions. See Guo ¶ 4.
Claims 6, 26, and 33 are rejected under 35 U.S.C. 103 as being unpatentable over Bernard, Chari, and Weider, as applied to claim 1 and 27 above, and Bernard and Weider as applied to claim 21 above, and further in view of Gaidon (US 2020/0134379 A1; published Apr. 30, 2020).
Regarding claim 6, Bernard, in view of Chari and Weider, discloses the invention of claim 1 as discussed above. Bernard does not expressly disclose wherein the training data comprises one or more images captured from one or more vehicles. However, Gaidon teaches automated labeling of images obtained by vehicle sensors in order to develop machine learning algorithms. ¶ 14.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bernard to incorporate the teachings of Gaidon to automatically select training images captured by vehicle sensors for use in developing machine learning algorithms at least because manually labeling vehicle data is inefficient and not scalable.
Regarding claim 26, Bernard, in view of Weider, discloses the invention of claim 21 as discussed above. Bernard does not expressly disclose wherein the training data comprises one or more frames of one or more videos. However, Gaidon teaches automated labeling of video obtained by vehicle sensors in order to develop machine learning algorithms. ¶ 14.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bernard to incorporate the teachings of Gaidon to automatically select training images captured by vehicle sensors for use in developing machine learning algorithms at least because manually labeling vehicle data is inefficient and not scalable.
Regarding claim 33, Bernard, in view of Chari and Weider, discloses the invention of claim 27 as discussed above. Bernard does not expressly disclose wherein the unlabeled training data comprises one or more images captured from one or more autonomous devices. However, Gaidon teaches automated labeling of video obtained by vehicle sensors in order to develop machine learning algorithms. ¶ 14.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bernard to incorporate the teachings of Gaidon to automatically select training images captured by vehicle sensors for use in developing machine learning algorithms at least because manually labeling vehicle data is inefficient and not scalable.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Singh et al. teaches methods for mining minority class data samples for training a neural network;
Jeong, Jisoo, et al. "Consistency-based semi-supervised learning for object detection." Advances in neural information processing systems 32 (2019);
Berthelot, David, et al. "Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring." arXiv preprint arXiv:1911.09785 (2019).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAHID KHAN whose telephone number is (571)270-0419. The examiner can normally be reached M-F, 9-5 est.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Jung can be reached at (571)270-3779. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHAHID K KHAN/Primary Examiner, Art Unit 2146
Prosecution Insights