DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
35 U.S.C. 101 – Claims 1-20
Claims 13-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. Independent claim 13 recites “An adaptive data labeling system comprising of (i) a smart selection and initial labeling component, (ii) a tiered hardness based selection and consensus based automatic labeling component, and a targeted selection and automatic labeling component”, however, no processor is being recited to be part of this system as it only recites components, therefore, it is directed to a computer program, being software per se. A claim that recites a piece of software alone without any link to a hardware component is directed to non-statutory subject matter since there is no relationship between the computer software and hardware components which permits the functionality of the software to be realized. The claims lack the necessary physical articles or objects to constitute a machine or a manufacture within the meaning of 35 USC 101. Therefore, based on the broadest reasonable interpretation, the claim itself seems to be only directed to a system comprising software components. As such, it fails to fall within a statutory category under 35 USC 101.
Claims 1-12, 14-20 also stand rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception without significantly more. The examiner will quote all text directly copied from the application within double quotes (e.g. “text”), all text directly copied from references will be in boldface.
Step 1 of the subject matter eligibility test (see MPEP 2106.03)
Claims 1 through 9 are directed toward a system, claims 10 through 12 are directed toward a method, which describes one of the four statutory categories or patentable subject matter, i.e., a machine. Claims 14-20 as explained above, fail to fall within a statutory category under 35 USC 101, however, these claims can be amended to fall within a statutory category if they were to be directed towards a system including a processor, being a hardware component; therefore, Examiner will proceed to determine whether such an amended claim would qualify as eligible under Steps 2A and 2B. For the purposes of compact prosecution, the examiner will proceed with the analysis under step 2.
Regarding claim 1:
Step 2A of the subject matter eligibility test (see MPEP 2106.04).
Prong one:
Claim 1 recites (i.e. “sets forth” or “describes”), “select a subset of unlabeled data” - this limitation recites selecting a subset of data, which is considered a mental process that can be performed by a human using a pen and paper as a physical aid. Observation, evaluation, and judgements of human mind can be used here to merely select which subset of data is needed. In addition, the ‘smart selection component’ mentioned will be addressed at Prong 2 below.
Claim 1 recites, “enable a user to manually label the selected subset of unlabeled data” – this limitation recites directing humans to label data on their own – both of which are mental processes that can be performed by a human using observation, evaluation, and judgement. In addition, ’user labeling component’ will be addressed in prong two below.
Claim 1 recites, “select data samples based on hardness tiers” – this limitation recites selecting data based on a perceived “hardness” level which is considered a mental process that can be performed by a human using a pen and paper as a physical aid. Observation, evaluation, and judgements of human mind can be used here to merely decide which data to select. In addition, ‘tiered-hardness based selection component’ will be addressed in prong two below.
Claim 1 recites, “automatically label the selected data samples” – this limitation recites the process of labeling selected data which is a mental process that can be performed by a human using a pen and paper as a physical aid. Observation, evaluation, and judgements of human mind can be used here to merely decide which data to label. In addition, the ‘consensus-based auto-labeling component’ will be addressed in prong two below.
Claim 1 recites, “identify weaknesses in the machine learning model” – this limitation is recited at a high level of generality and is directed towards a judicial exception, mental process. A human using a pen and paper as a physical aid using observation, evaluation, and judgements could perform this limitation. In addition, the ‘automatic evaluation and error analysis component’ will be addressed in prong two below.
Claim 1 recites, “select additional unlabeled data samples similar to the identified weaknesses” – this limitation is a mental process which can be performed by a human using a pen and paper as a physical aid. Observation, evaluation, and judgements of human mind can be used to perform this task. In addition, the ‘targeted selection component’ will be addressed in prong two below.
Prong two:
Claim 1 recites, “one or more memories configured to store instructions” --- these one or more memories amount to mere instructions to apply an exception, as the use of a generic computer component or other machinery in its ordinary capacity (such as a memory to store data) amounts to invoking computers merely as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
Claim 1 recites, “one or more processors configured to, when executing the instruction, perform operations for a plurality of components, wherein the plurality of components include:” – these one or more processors amount to mere instructions to apply an exception, as the use of a generic computer component or other machinery in its ordinary capacity (such as a processor to perform a step of instruction) amounts to invoking computers merely as a tool to perform an existing process (see MPEP 2106.05(f) (2)).
Claim 1 limitations : “a smart selection component”, “a user labeling component”, “a tiered-hardness based selection component”, “a consensus-based auto-labeling component”, “an automatic evaluation and error analysis component”, “a targeted selection component”- these components are interpreted as software components being executed by the processor for the purposes of applying the mental processes as described above at Prong 1, therefore, they amount to mere instructions to implement an abstract idea on a computer (see MPEP 2106.05(f)).
Claim 1 recites, “a model training component configured to train a machine learning model using the labeled subset of data” – this limitation is recited at a high level of generality and is directed towards a particular technological environment or field of use. The examiner understands that the act of labeling data remains an abstract idea, while the use of this abstract idea to train a machine learning model is a field of use. This does not amount to more than the general use of training a machine learning model with a selected set of labeled data. The element alone or in combination with the rest of the claim fails to integrate the judicial exceptions into a practical application. Please see MPEP 2106.05(h)
Step 2B of the subject matter eligibility test (see MPEP 2106.05).
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application under step 2A: prong 2, the additional elements recited at claim 1 amount to generic computer components merely used as a tool to implement the abstract ideas and / or to perform an existing process, and applying the judicial exception to a field of use
Viewed as a whole, these additional claim element(s) do not provide meaningful limitation(s) to transform the abstract ideas into a patent eligible application such that the claim(s) amount to significantly more than the abstract ideas themselves. Therefore, the claim 1 is rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Claim 2 recites, “The system of claim 1, wherein the smart selection component is configured to select the subset of unlabeled data based on a diversity selection function or a representative selection function.” – the smart selection component is being limited by a diversity selection function and a representative selection function. These functions, as disclosed in the specification’s [0081] [0082] [0083] [0086] [0087] are mathematical formulae and are directed towards the abstract idea of mathematical calculations and formulas, being mathematical concepts (MPEP 2106.04(a)(2)(I)(C)).
Claim 3 recites, “The system of claim 2, wherein the diversity selection function is configured to select data instances that capture a wide range of different and distinct variations of the dataset.” –This diversity selection functions, as disclosed in the specifications [0086] [0087], are mathematical formulae and are directed towards the abstract idea of mathematical calculations and formulas, being mathematical concepts (MPEP 2106.04(a)(2)(I)(C)).
Claim 4 recites, “The system of claim 2, wherein the representative selection function is configured to select data instances that reflect the overall distribution and characteristics of the dataset.” - These representative selection functions, as disclosed in the specifications [0081] [0082] [0083], are mathematical formulae and are directed towards the abstract idea of mathematical calculations and formulas, being mathematical concepts (MPEP 2106.04(a)(2)(I)(C)).
Claim 5 recites, “The system of claim 1, wherein the tiered-hardness based selection component is configured to select data samples based on at least three hardness tiers, including a first hardness tier for easy samples, a second hardness tier for intermediate samples, and a third hardness tier for hard samples.” – this limitation is classifying data into three mental levels. This is a mental process which can be performed by a human using a pen and paper as a physical aid. Observation, evaluation, and judgements of human mind can be used here. The additional elements of at least three levels of mental scopes, alone or in combination, fail to integrate the judicial exception into a practical application. Please see MPEP 2106.04(a)(2)(III)(C).
Claim 6 recites, “The system of claim 5, wherein the consensus-based auto-labeling component is configured to automatically label the easy samples without human verification, and to automatically label the intermediate and hard samples with human verification.” – this limitation is merely automating an abstract idea (evaluating and judging a piece of data for a mental context or scope) on a computer, therefore, it amounts to a mere instruction to implement an abstract idea on a computer. Please see MPEP 2106.04(a)(2)(III)(C).
Claim 7 recites, “The system of claim 1, wherein the automatic evaluation and error analysis component is configured to identify weaknesses in the machine learning model by analyzing errors and low-confidence predictions of the machine learning model.” – this limitation is recited at a high level of generality and is directed towards a judicial exception, mental process. A human using a pen and paper as a physical aid using observation, evaluation, and judgements could perform this limitation. Please see MPEP 2106.04(a)(2)(III)(C).
Claim 8 recites, “The system of claim 1, wherein the targeted selection component is configured to select additional unlabeled data samples that are conceptually similar to the identified weaknesses.” – identifying conceptually similar data can be done in human mind with the help of a pen and paper and is directed to the abstract idea of a mental process. Please see MPEP 2106.04(a)(2)(III)(C).
Claim 9 recites, “The system of claim 1, further comprising a data augmentation component configured to modify the labeled subset of data to increase the diversity, richness, and quantity of labeled data.” – The examiner interprets this claim limitation as altering the data. Computer image manipulation of data recited at such high level of generality directs to insignificant extra-solution activity, under step 2A prong two, please see MPEP 2106.05(g). Data augmentation is a well-understood, routine, conventional activity (under step 2B) in machine learning systems as evidenced by Buslaev et al. from here on Buslaev (NPL: “Albumentations: fast and flexible image augmentations”, see Abstract: “Data augmentation is a commonly used technique for increasing both the size and the diversity of labeled training sets by leveraging input transformations that preserve output labels.”)
Referring to independent method claim 10, it is rejected on the same basis as independent claim 1, mutatis mutandis, since they are analogous claims.
Referring to the dependent claim 11, it is rejected on the same basis as dependent claims 5 and 6 combined, mutatis mutandis, since they are analogous claims.
Referring to dependent claim 12, this claim recites limitation, “further comprising augmenting the labeled subset of data using data augmentation techniques to increase the diversity and richness of the labeled data, wherein the data augmentation techniques include at least one of random cropping, scaling and padding, horizontal flipping, adjusting lighting, brightness, contrast, hue, or adding synthetic weather conditions.” -- The examiner interprets this claim limitation as altering the data via augmentation or transformation. Computer image manipulation recited at such high level of generality directs to insignificant extra-solution activity, under step 2A prong two, please see MPEP 2106.05(g). Data augmentation is a well-understood, routine, conventional activity (under step 2B) in machine learning systems as evidenced by Buslaev (NPL: “Albumentations: fast and flexible image augmentations”, see Abstract: “Data augmentation is a commonly used technique for increasing both the size and the diversity of labeled training sets by leveraging input transformations that preserve output labels.”)
Regarding claim 14:
Step 2A of the subject matter eligibility test (see MPEP 2106.04).
Prong one:
The claim restriction of dependent claim 14 limits the independent claim 13 by, “select a subset of unlabeled data for initial human labeling” – this limitation recites selecting a subset of data, which is considered a mental process that can be performed by a human using a pen and paper as a physical aid. Observation, evaluation, and judgements of human mind can be used here to merely select which subset of data is needed. In addition, the ‘data selection component’ mentioned will be addressed at Prong 2 below.
The claim restriction of dependent claim 14 limits the independent claim 13 by, “wherein a user labels the selected data instances” -- this limitation recites labeling data by humans – which is a mental processes that can be performed by a human using observation, evaluation, and judgement. In addition, the ’user labeling component’ will be addressed in prong two below.
The claim restriction of dependent claim 14 limits the independent claim 13 by, “use pseudo-labels from multiple models for consensus”– this limitation recites the process of labeling selected data which is a mental process that can be performed by a human using a pen and paper as a physical aid. Observation, evaluation, and judgements of human mind can be used here to merely decide which data to label. In addition, the ‘consensus based labeling component’ will be addressed in prong two.
The claim restriction of dependent claim 14 limits the independent claim 13 by, “train the models on the user labeled and system labeled instances” -- this limitation recites selecting data based on a perceived “hardness” level which is considered a mental process that can be performed by a human using a pen and paper as a physical aid. Observation, evaluation, and judgements of human mind can be used here to merely decide which data to select. In addition, the ‘tiered-hardness based selection component’ and the ‘model training and refinement component’ will be addressed in prong two below.
Prong two:
The limitations of claim 13, “(i) a smart selection and initial labeling component, (ii) a tiered hardness based selection and consensus based automatic labeling component, and a targeted selection and automatic labeling component” - these components are interpreted as software components for the purposes of applying the mental processes, as described above, amount to mere instructions to implement an abstract idea on a computer (see MPEP 2106.05(f)).
The claim limitations in dependent claim 14, “data selection component”, “user labeling component”, “consensus based labeling component”, “tiered hardness based selection component”, and “targeted selection component, and an evaluation and validation component” - these components are interpreted as software components for the purposes of applying the mental processes, as described above, amount to mere instructions to implement an abstract idea on a computer (see MPEP 2106.05(f)).
The dependent claim 14 is limiting the independent claim 13 by, “a data augmentation component configured to augment the labeled data with various data transformations to increase the amount of labeled data”. Computer image data manipulation recited at such high level of generality directs to insignificant extra-solution activity, under step 2A prong two, please see MPEP 2106.05(g).
Further by, “a model training and refinement component configured to train the models on the user labeled and system labeled instances” -- this limitation is recited at a high level of generality and is directed towards a particular technological environment or field of use. The examiner interprets that the act of labeling data remains an abstract idea, while the use of this abstract idea to train a machine learning model is a field of use. This does not amount to more than the general use of training a machine learning model with a selected set of labeled data. The element alone or in combination with the rest of the claim fails to integrate the judicial exceptions into a practical application. Please see MPEP 2106.05(h).
Step 2B of the subject matter eligibility test (see MPEP 2106.05).
The limitations of claim 13, “(i) a smart selection and initial labeling component, (ii) a tiered hardness based selection and consensus based automatic labeling component, and a targeted selection and automatic labeling component” - these components are interpreted as software components for the purposes of applying the mental processes, as described above, amount to mere instructions to implement an abstract idea on a computer (see MPEP 2106.05(f)).
The claim limitations “data selection component”, “user labeling component”, “consensus based labeling component”, “tiered hardness based selection component”, and “targeted selection component, and an evaluation and validation component” - these components are interpreted as software components for the purposes of applying the mental processes as described above amount to mere instructions to implement an abstract idea on a computer (see MPEP 2106.05(f)).
The dependent claim 14 is limiting the independent claim 13 by, “a data augmentation component configured to augment the labeled data with various data transformations to increase the amount of labeled data”. This is an insignificant extra-solution activity. Computer image manipulation of data recited at such high level of generality directs to insignificant extra-solution activity, under step 2A prong two, please see MPEP 2106.05(g). Data augmentation is a well-understood, routine, conventional activity (under step 2B) in machine learning systems as evidenced by Buslaev (NPL: “Albumentations: fast and flexible image augmentations”, see Abstract: “Data augmentation is a commonly used technique for increasing both the size and the diversity of labeled training sets by leveraging input transformations that preserve output labels.”)
Further by, “a model training and refinement component configured to train the models on the user labeled and system labeled instances” -- this limitation is recited at a high level of generality and is directed towards a particular technological environment or field of use. The examiner interprets that the act of labeling data remains an abstract idea, while the use of this abstract idea to train a machine learning model is a field of use. This does not amount to more than the general use of training a machine learning model with a selected set of labeled data. The element alone or in combination with the rest of the claim fails to integrate the judicial exceptions into a practical application. Please see MPEP 2106.05(h).
The dependent claim 15 limits claim 14 by “wherein the data selection component is configured to select a subset of diverse and representative data instances using submodular functions as choices of diversity functions.” These selection functions, as disclosed in the specifications [0086] [0087], are mathematical formulae and are directed towards the abstract idea of mathematical calculations and formulas, being mathematical concepts (MPEP 2106.04(a)(2)(I)(C)).
The dependent claim 16 recites limitation for claim 14, “wherein the data augmentation component is configured to perform random cropping, scaling and padding, horizontal flipping, lighting, brightness, contrast, hue, augmentations by adding different weather conditions, and combinations thereof.” The examiner interprets this claim limitation as altering the data via augmentation or transformation. Computer image manipulation recited at such high level of generality directs to insignificant extra-solution activity, under step 2A prong two, please see MPEP 2106.05(g). Data augmentation is a well-understood, routine, conventional activity (under step 2B) in machine learning systems as evidenced by Buslaev (NPL: “Albumentations: fast and flexible image augmentations”, see Abstract: “Data augmentation is a commonly used technique for increasing both the size and the diversity of labeled training sets by leveraging input transformations that preserve output labels.”)
Claim 17 recites the limitation of claim 14, “wherein the tiered hardness component is configured to select instances which are easy and can be automatically labeled, instances of intermediate hardness and that require a mix of human verification and automatic labeling, and instances of hard instances and that require manual labeling.” – the classification of data for claim 14 is being limited to at least three mental levels. This is a mental process which can be performed by a human using a pen and paper as a physical aid. Observation, evaluation, and judgements of human mind can be used here. Please see MPEP 2106.04(a)(2)(III)(C).
Claim 18 recites the limitation of claim 14, “wherein the consensus based labeling component is configured to use multiple machine learning models and use a consensus mechanism to only label the high confidence data instances.” -- this limitation is merely automating an abstract idea: evaluating and judging data for labeling on a computer; therefore, it amounts to a mere instruction to implement an abstract idea on a computer. Please see MPEP 2106.04(a)(2)(III)(C).
The dependent claim 19 limits claim 14 by, “wherein scenarios of existing pretrained models, and without unlabeled data, can be directly labeled using multiple pre-trained models”. The examiner interprets this as, scenarios of existing pretrained models, and with labeled data (“without unlabeled data” = with labeled data only), can be directly labeled using multiple pre-trained models. -- this limitation is merely automating an abstract idea (evaluating and judging a piece of data for labeling) on a computer, therefore, it amounts to a mere instruction to implement an abstract idea on a computer. Please see MPEP 2106.04(a)(2)(III)(C).
The dependent claim 20 limits claim 14 by, “wherein the adaptive data labeling system selects examples given an initial labeled set, a labeling model, and an unlabeled data pool using mechanisms of diversity and uncertainty.” This limitation invokes computers or other generic machinery components at a high level of generality merely as a tool to perform an abstract idea, selecting data to be labeled (a mental process), therefore, it amounts to mere instructions to implement an abstract idea or other exception on a computer (see MPEP 2106.05(f)).
Thus, none of the limitations do more to the judicial exception and the judicial exception is not integrated into a practical application (see MPEP 2106.04(d)(I)), failing step 2A prong 2. The limitations do not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-8, 10,11,13,15, 17-20 are rejected under 35 U.S.C. 102 (a)(1) and (a) (2) as being anticipated by Salman et al. (US Publication number US 20210406644) hereinafter Salman.
Referring to Claim 1, Salman teaches a system for adaptive data labelling to enhance machine learning precision, comprising:
one or more memories configured to store instructions (see Salman at [0206]: “FIG. 13 illustrates an example device 2500, with a processor 2502 and memory 2504 that can be configured to implement various embodiments of the active learning framework as discussed in this disclosure”. Examiner interprets Salman’s memory 2504 as the claimed memories for storing instructions); and
one or more processors configured to, when executing the instruction, perform operations for a plurality of components (see Salman at [0006] and fig 1: “FIG. 13 illustrates an example device 2500, with a processor 2502 and memory 2504 that can be configured to implement various embodiments of the active learning framework as discussed in this disclosure”. Examiner interprets Salman’s processor 2502 as the claimed processors to perform operations for a plurality of components), wherein the plurality of components include:
a smart selection component configured to select a subset of unlabeled data (see Salman at Fig. 1A and at [0006]: “ In each iteration of the active learning phase, the plurality of machine learning components (which have been trained from the previous iteration of the training phase) are configured to generate scores or metrics that are used to control the sampling (or selection) of unlabeled observations for labeling.”);
a user labeling component configured to enable a user to manually label the selected subset of unlabeled data (see Salman at fig. 1A and [0064]: “FIG. 1A illustrates a passive learning framework where in block 101 a human annotator labels a random batch of unlabeled observations”. Examiner interprets that the active learning component of Salman can direct human annotator to label a batch of unlabeled data.);
a model training component configured to train a machine learning model using the labeled subset of data (see Salman at Fig. 1A and [0064]: “The batch of labeled observations (data) generated in block 101 are then used to train a machine learning system in block 103”. Examiner interprets this training of a machine learning system using the labeled data initially from the human annotator, subsequently from the auto labeling via models);
a tiered-hardness based selection component configured to select data samples based on hardness tiers (see Salman at [0064]: “The active learning phase uses the trained machine learning system to generate a prediction for an unlabeled observation (data) and compute a metric (such as an uncertainty score) associated with the (label) prediction. The metrics associated with the predictions are then used to rank unlabeled observations, and the ranking of the unlabeled observations is used to control sampling or selection of the unlabeled observations for labeling” The examiner interprets the ranking/ uncertainty of the unlabeled data to be the level of difficulty of prediction);
a consensus-based auto-labeling component configured to automatically label the selected data samples (See Salman at [0067]: “The ranking of the unlabeled observations is used to control sampling of the unlabeled observations (data) for a batch-mode labeling process, which includes automatic labeling without human annotation”. Further see Salman at [0076]: “[in fig 2A2] … the inspection learning component can be modified to generate better uncertainty measurements, such as by employing an ensemble of learners” The examiner interprets the ensemble of models to form consensus for ranking data elements to label.);
an automatic evaluation and error analysis component configured to identify weaknesses in the machine learning model (See Salman at [0067]: “The operations can also include healing operations that use the trained machine learning components to possibly identify one or more problematic labels and corresponding observations (data) that are stored in the pool of labeled data for review by a human annotator and subsequent update or discarding of the label for the observation as dictated by input from the human annotator” The examiner interprets the problematic/ incorrect labeling as the measurement of weakness of the model and the act of identifying such weakness as the error analysis and evaluation process); and
a targeted selection component configured to select additional unlabeled data samples similar to the identified weaknesses (See Salman at [0067]: “[in fig 2B2] One or more composite scores (or weighted scores) are computed based on the metrics (typicality score, relevance score, uncertainty score) associated with the predictions for the unlabeled observations, and the composite scores for the unlabeled observations are then used to rank unlabeled observations (data). The ranking of the unlabeled observations is used to control sampling of the unlabeled observations for a batch-mode labeling process, which includes automatic labeling without human annotation as well as manual labeling involving human annotation.” Further, as mentioned earlier See Salman in [0067]: “The results of the labeling process are added to (stored in) the appropriate data pool. The operations can also include healing operations that use the trained machine learning components to possibly identify one or more problematic labels and corresponding observations that are stored in the pool of labeled data”) The examiner interprets the weighted scoring of the unlabeled data, selection of additional data based on prediction metrics, and the subsequent healing process as ways to target unlabeled data to improve upon the “weakness” of the model(s).
Referring to Claim 2, Salman teaches the system of claim 1, wherein the smart selection component is configured to select the subset of unlabeled data based on a diversity selection function or a representative selection function. (Salman teaches in [0084] “Furthermore, a diversity criterion can be used to select observations for automatic labeling. For example, unlabeled observations which are the centroids obtained by a k-means algorithm on top score observations described by a set of features can be selected for automatic labeling”. See Salman in [0201], “the informative observations can include one or more unlabeled observations which are highly relevant (e.g., have relatively high relevance scores) and which are highly representative (e.g., have a relatively high density score) and on which the inspection learning component 209C (figure 2A2) performs badly (e.g., have relatively high uncertainty scores and relatively low typicality scores) and thus from which we expect to learn the most when annotated.”) Thus illustrating the selection of diverse and representative data elements.
Referring to Claim 3, Salman teaches the system of claim 2, wherein the diversity selection function is configured to select data instances that capture a wide range of different and distinct variations of the dataset. (Salman teaches in [0084], “In the case that the observations are sampled in batches, it is desirable that the confident observations be diverse so that they cover a variety of situations.” The examiner interprets the diversity sampling in Salman is choosing a wide range of different and distinct variations of data as Salman states it is desirable to cover a variety of situations.)
Referring to Claim 4, Salman teaches the system of claim 2, wherein the representative selection function is configured to select data instances that reflect the overall distribution and characteristics of the dataset. (Salman shows in [0201], “observations (data) can include one or more unlabeled observations which are highly relevant (e.g., have relatively high relevance scores) and which are highly representative (e.g., have a relatively high density score)”. Salman in [0202] explains, “For example, the confident observations can include one or more unlabeled observations which are relevant (e.g., have a relatively high relevance score), representative (e.g., have a relatively high-density score), typical (e.g., have a relatively high typicality score) and have low uncertainty (e.g., have a relatively low uncertainty score).” thus illustrating the selection of highly representative data).
Referring to Claim 5, Salman teaches the system of claim 1, wherein the tiered-hardness based selection component is configured to select data samples based on at least three hardness tiers, including a first hardness tier for easy samples, a second hardness tier for intermediate samples, and a third hardness tier for hard samples. (Salman adopts the concept of uncertainty score for sampling (selecting) observations (data). Uncertainty score characterizes the confidence level for the associated predicted label and/or the reproducibility of the associated unlabeled observation [0007]. See Salman at [0012], “A confident observation can be identified from the uncertainty, typicality, density and/or relevance scores or metrics produced from the output of the plurality of machine learning components based on the unlabeled observation taken as input. For example, a confident observation can i) produce a predicted label output by the at least one primary learning component with a relatively low uncertainty score or have high reproducibility, ii) produce a predicted label output by the least one primary learning component with a relatively high typicality score and/or relatively high density score, and iii) have a relatively high relevance score.” Further, “An informative observation can be identified from the uncertainty, typicality, density and/or relevance scores or metrics produced from the output of the plurality of machine learning components based on the unlabeled observation taken as input. For example, an informative observation can i) produce a predicted label output by the inspection learning component with a relatively high uncertainty score or have low reproducibility, ii) produce a predicted label output by the inspection learning component with a relatively low typicality score (where such low typicality label is divergent from a distribution of labeled observations) and/or a relatively high density score (e.g., where the label is similar to other labeled or non-labeled observations), and iii) have a relatively high relevance score (e.g., not anomalous or off-topic).” See Salman at [0014], “The problematic labels can be identified from the uncertainty, typicality, density and/or relevance scores or metrics produced from the output of the plurality of machine learning components based on labeled observations taken as input. For example, a problematic label can i) have a relatively high uncertainty score or have low reproducibility, ii) have a relatively low typicality score and/or a relatively low-density score, and iii) a relatively low relevance score.” The examiner interprets the confident observations as easy, informative observations as intermediate, and problematic observations as hard data.)
Referring to Claim 6, Salman teaches the system of claim 5, wherein the consensus-based auto-labeling component is configured to automatically label the easy samples without human verification, and to automatically label the intermediate and hard samples with human verification, (See Salman at [0084] and figure 2A2: “the labeling component 215 can be configured such that a batch of confident observations are automatically labeled without human input (by an auto-labeler)”. See Salman in [0083] and figure 2A2, “the labeling component 215 can be configured such that informative observations are sampled (or selected) to be labeled by the human annotator(s).” See Salman in [0090] and figure 8, 9B, “In blocks 923 and 925 (block 807), the (human) annotator can select a problematic labeled observation from the queue and review the selected problematic labeled observation to decide whether it is valid or whether it should be rejected or discarded (block 810”).Further see Salman at [0076]: “[in fig 2A2] … the inspection learning component can be modified to generate better uncertainty measurements, such as by employing an ensemble of learners” The examiner interprets the use of ensemble of models to form a consensus for data labeling.)
Referring to Claim 7, Salman teaches the system of claim 1, wherein the automatic evaluation and error analysis component is configured to identify weaknesses in the machine learning model by analyzing errors and low-confidence predictions of the machine learning model. (See Salman at [0067]: “The operations can also include healing operations that use the trained machine learning components to possibly identify one or more problematic labels and corresponding observations that are stored in the pool of labeled data for review by a human annotator and subsequent update or discarding of the label for the observation as dictated by input from the human annotator” The examiner interprets the problematic labels as a symptom of weakness in a model and the act of identifying such weakness via the language model as the error analysis and evaluation process.)
Referring to Claim 8, Salman teaches the system of claim 1, wherein the targeted selection component is configured to select additional unlabeled data samples that are conceptually similar to the identified weaknesses. (See Salman at [0086]: “[in fig 7] depicts operations of the data analysis component 213, which can involve computing an informativeness weighted ranking for unlabeled observations (data) by computing and ranking composite scores for the unlabeled observations such that the top ranked observations have relatively high relevance scores, relatively low typicality scores and relatively high uncertainty scores (block 701). The informativeness weighted ranking can be used to shape the data into diverse and ordered batches of unlabeled observations (block 703)” Further, See Salman in [0086] and fig 7: “The operations of the data analysis component 213 can also involve computing a problematic weighted ranking for labeled observations by computing and ranking the labeled observations such that the top ranked labeled observations have relatively low relevance scores, relatively low typicality scores and relatively high uncertainty scores (block 711).” The problematic labels are later rejected or discarded with human annotator help [0090]).
Referring to independent method claim 10, it is rejected on the same basis as independent claim 1, mutatis mutandis, since they are analogous claims.
Referring to dependent method claim 11, it is rejected on the same basis as (limitations of) claims 5 and 6 (dependent claims of claim 1), mutatis mutandis, since they are analogous claims.
Referring to independent claim 13, Salman teaches, an adaptive data labeling system comprising of (i) a smart selection and initial labeling component (see Salman at Fig. 1A and at [0064]: “FIG. 1A illustrates a passive learning framework where a human annotator labels a random batch of unlabeled observations (initial data). The batch of labeled observations are then used to train a machine learning system”. Further in [0064] Salman discloses, “The active learning (model) phase uses the trained machine learning system to generate a prediction for an unlabeled observation (data)” Examiner interprets the framework of passive learning, with initial labeling and subsequent (repeated) auto labeling via the model for active learning, as the whole system.);
(ii) a tiered hardness based selection (see Salman at [0064]: “The active learning phase uses (the prediction module of) the trained machine learning system to generate a prediction for an unlabeled observation (data)” The examiner interprets the uncertainty ranking by the prediction component (block 119 of figure 1B) in Salman as the tiered hardness calculator);
and consensus based automatic labeling component, (See Salman at [0067]: “block 271 depicted in figure 2B2, batch labels unlabeled ‘ranked’ data”. Further see Salman at [0076]: “[in fig 2A2] … the inspection learning component 209C can be modified to generate better uncertainty measurements, such as by employing an ensemble of learners” The examiner interprets block 271 as the auto labeler, and the ensemble of models to form the consensus.);
and a targeted selection and automatic labeling component (See Salman at [0067]: “[in fig 2B2] One or more composite scores (or weighted scores) are computed based on the metrics (typicality score, relevance score, uncertainty score) associated with the predictions for the unlabeled observations (done via block 269)…automatic labeling (is done) without human annotation (using block 271) ...” Block 269 is the targeted selection module and block 271 is the auto labeling component, as interpreted by the examiner.)
Referring to claim 15, Salman teaches, data selection component is configured to select a subset of diverse and representative data instances using submodular functions as choices of diversity functions. (Salman teaches in [0084] “Furthermore, a diversity criterion can be used to select observations for automatic labeling. For example, unlabeled observations which are the centroids obtained by a k-means algorithm on top score observations described by a set of features can be selected for automatic labeling”. See Salman in [0201], “the informative observations can include one or more unlabeled observations which are highly relevant (e.g., have relatively high relevance scores) and which are highly representative (e.g., have a relatively high density score) and on which the inspection learning component 209C (figure 2A2) performs badly (e.g., have relatively high uncertainty scores and relatively low typicality scores) and thus from which we expect to learn the most when annotated.” -- Thus depicting the selection process of diverse and representative data.)
Referring to claim 17, Salman teaches, the tiered hardness component is configured to select instances which are easy and can be automatically labeled, instances of intermediate hardness and that require a mix of human verification and automatic labeling, and instances of hard instances and that require manual labeling. (See Salman at [0084] and figure 2A2: “the labeling component 215 can be configured such that a batch of confident observations are automatically labeled without human input (by an auto-labeler)”. See Salman in [0083] and figure 2A2, “the labeling component 215 can be configured such that informative observations are sampled (or selected) to be labeled by the human annotator(s).” See Salman in [0090] and figure 8, 9B, “In blocks 923 and 925 (block 807), the (human) annotator can select a problematic labeled observation from the queue and review the selected problematic labeled observation to decide whether it is valid or whether it should be rejected or discarded (block 810)”)
Referring to claim 18, Salman teaches, the consensus based labeling component is configured to use multiple machine learning models and use a consensus mechanism to only label the high confidence data instances. (See Salman at [0089]: “Furthermore, the labeling operations can be machine learning aided where most confident labels can be automatically labeled without human input and added to the final label(ed) data”. Further see Salman at [0076]: “[in fig 2A2] … the inspection learning component 209C can be modified to generate better uncertainty measurements, such as by employing an ensemble of learners” The examiner interprets the ensemble of models to form consensus for ranking data elements to label.)
Referring to claim 19, Salman teaches scenarios of existing pretrained models, and without unlabeled data, can be directly labeled using multiple pre-trained models. (See Salman at [0067]: in figure 2B1, the labeled data in the pool is sent through refinement using inspection learner and the relevance learner until a threshold on training is exhausted. “The operations can also include healing operations that use the trained machine learning components to possibly identify one or more problematic labels and corresponding observations that are stored in the pool of labeled data for review by a human annotator and subsequent update or discarding of the label for the observation as dictated by input from the human annotator (block 277 in fig. 2B2)” The examiner interprets this auto labeling process by trained models with already labeled data as an effort to improve accuracy and learning by the model(s).
Referring to claim 20, Salman teaches, the adaptive data labeling system selects examples given an initial labeled set, a labeling model, and an unlabeled data pool using mechanisms of diversity and uncertainty. (see Salman at Fig. 1A and at [0064]: “FIG. 1A illustrates a passive learning framework where a human annotator labels a random batch of unlabeled observations (initial data). The batch of labeled observations are then used to train a machine learning system”. Further see Salman [0067] and figure 2B1 where the unlabeled data pool is being used to predict the labels using block 265. Salman teaches in [0084], “In the case that the observations are sampled in batches, it is desirable that the confident observations be diverse so that they cover a variety of situations”. A metric named uncertainty score, disclosed in Salman [0007], is being used to find the uncertainty in predicting labels for the unlabeled data.)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 9 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Salman et al. , and further in view of Ionescu et al. (“Join Path-Based Data Augmentation for Decision Trees," 2022 IEEE 38th International Conference on Data Engineering Workshops (ICDEW), Kuala Lumpur, Malaysia, 2022, pp. 84-88, doi: 10.1109/ICDEW55742.2022.00018) -- hereinafter Ionescu.
Referring to Claim 9, Salman teaches the system of claim 1, however, it fails to teach: data augmentation component configured to modify the labeled subset of data to increase the diversity, richness, and quantity of labeled data.
Ionescu teaches, data augmentation component configured to modify the labeled subset of data to increase the diversity, richness, and quantity of labeled data. (see Ionescu [abstract]: “Automated data augmentation techniques can help increase the richness of training data, thus increasing the ML model accuracy.”) Further, Ionescu teaches that, (see Ionescu [Introduction]: “automatic data augmentation has emerged as a technique to improve the amount and diversity of training data through image transformation”) the examiner interprets “amount” as quantity.
It would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teachings of Salman with the above teachings of Ionescu, by the system for adaptive data labelling to enhance machine learning precision, as taught by Salman, and modify the labeled subset of data to increase the diversity, richness, and quantity of labeled data, as taught by Ionescu. The modification would have been obvious because one of ordinary skills in the art would be motivated to enhance the model training, in the absence of abundant high quality data for training, by incorporating data augmentation. (see Ionescu [abstract]: “Automated data augmentation techniques can help increase the richness of training data, thus increasing the ML model accuracy”) and (see Ionescu [Introduction]: “automatic data augmentation has emerged as a technique to improve the amount and diversity of training data through image transformation”)
Referring to claim 14, Salman teaches the adaptive data labeling system of claim 13 and the limitations of claim 14,
a data selection component configured to select a subset of unlabeled data for initial human labeling – (see Salman at Fig. 1A and at [0064]: “FIG. 1A illustrates a passive learning framework where in block 101 a human annotator labels a random batch of unlabeled observations”. Examiner interprets the ‘random batch’ of unlabeled observations as the ‘selected subset’ of unlabeled data as a user is only labeling one subset, not the entire set.)
a user labeling component, wherein a user labels the selected data instances -(see Salman at Fig. 1A and at [0064]: “FIG. 1A illustrates a passive learning framework where in block 101 a human annotator labels a random batch of unlabeled observations”. Examiner interprets the ‘random batch’ of unlabeled observations as the ‘selected’ set of unlabeled data)
a consensus based labeling component configured to use pseudo-labels from multiple models for consensus -- (See Salman at [0067]: “The ranking of the unlabeled observations is used to control sampling of the unlabeled observations (data) for a batch-mode labeling process, which includes automatic labeling without human annotation”. Further see Salman at [0076]: “[in fig 2A2] … the inspection learning component can be modified to generate better uncertainty measurements, such as by employing an ensemble of learners” The examiner interprets the ensemble of models forming consensus to predict pseudo labels for ranking data elements to be labeled.)
a tiered hardness based selection component -- (see Salman at [0064]: “The active learning phase uses the trained machine learning system to generate a prediction for an unlabeled observation (data) and compute a metric (such as an uncertainty score) associated with the (label) prediction. The metrics associated with the predictions are then used to rank unlabeled observations, and the ranking of the unlabeled observations is used to control sampling or selection of the unlabeled observations for labeling” The examiner interprets the ranking/ uncertainty score of the unlabeled data to be the level of difficulty or “hardness” for predicting labels.)
a model training and refinement component configured to train the models on the user labeled and system labeled instances - (see Salman at Fig. 1A and [0064]: “The batch of labeled observations (data) generated in block 101 are then used to train a machine learning system in block 103” Examiner interprets this process of active learning using the labeled data initially from the human annotator, subsequently from the auto labeling by models as the invention. See Salman in [0067]: “The results of the labeling process are added to (stored in) the appropriate data pool. The operations can also include healing operations that use the trained machine learning components to possibly identify one or more problematic labels and corresponding observations that are stored in the pool of labeled data” The examiner interprets the weighted scores of the unlabeled data, prediction metrics, and the healing process of labeled data as ways of “refinement” of the model(s).)
a targeted selection component, and an evaluation and validation component - (See Salman at [0067]: [in fig 2B2] component 269 is computing the composite scores for selecting data for either auto labeling or manual labeling. The examiner interprets this as the targeted selection component. See Salman [0054], “the framework can also include a mechanism for a human annotator to review predicted labels as it is faster to validate or reject a predicted label” The examiner interprets the mechanism inside the framework as the validation component.)
Salman does not teach, a data augmentation component configured to augment the labeled data with various data transformations to increase the amount of labeled data.
However, Ionescu teaches, a data augmentation component configured to augment the labeled data with various data transformations to increase the amount of labeled data. (See Ionescu [Introduction]: “automatic data augmentation has emerged as a technique to improve the amount and diversity of training data through image transformation” the examiner interprets “amount” as quantity of data.)
It would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teachings of Salman with the above teachings of Ionescu, by the adaptive data labelling system, as taught by Salman, a data augmentation component configured to augment the labeled data with various data transformations to increase the amount of labeled data, as taught by Ionescu. The modification would have been obvious because one of ordinary skills in the art would be motivated to enhance the model training, in the absence of abundant high quality data for training, by incorporating data augmentation. (see Ionescu [abstract]: “Automated data augmentation techniques can help increase the richness of training data, thus increasing the ML model accuracy”) and (see Ionescu [Introduction]: “automatic data augmentation has emerged as a technique to improve the amount and diversity of training data through image transformation”)
Claim 12 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Salman in view of Ionescu, in view of VAZQUEZ et al. (Canadian patent number CA-3070817-C)– hereinafter Vazquez, and further in view of Xu (US publication 20180373999) – hereinafter Xu.
Referring to Claim 12, Salman teaches the computer implemented method of claim 10, however, it fails to teach the limitation: “further comprising augmenting the labeled subset of data using data augmentation techniques to increase the diversity and richness of the labeled data, wherein the data augmentation techniques include at least one of random cropping, scaling and padding, horizontal flipping, adjusting lighting, brightness, contrast, hue, or adding synthetic weather conditions.”
Ionescu teaches data augmentation techniques to increase the diversity and richness of the labeled data. (See Ionescu [abstract]: “Automated data augmentation techniques can help increase the richness of training data, thus increasing the ML model accuracy”. Further, see Ionescu [Introduction]: “automatic data augmentation has emerged as a technique to improve the amount and diversity of training data through image transformation”).
It would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teachings of Salman with the above teachings of Ionescu, by the computer-implemented method for adaptive data labelling to enhance machine learning precision, as taught by Salman, augmenting the labeled subset of data using data augmentation techniques to increase the diversity and richness of the labeled data, as taught by Ionescu. The modification would have been obvious because one of ordinary skills in the art would be motivated to enhance the model training, in the absence of abundant high quality data for training, by incorporating data augmentation. (see Ionescu [abstract]: “Automated data augmentation techniques can help increase the richness of training data, thus increasing the ML model accuracy”) and (see Ionescu [Introduction]: “automatic data augmentation has emerged as a technique to improve the amount and diversity of training data through image transformation”)
However, Ionescu does not teach, “wherein the data augmentation techniques include at least one of random cropping, scaling and padding, horizontal flipping, adjusting lighting, brightness, contrast, hue, or adding synthetic weather conditions.”
VAZQUEZ teaches, wherein the data augmentation techniques include at least one of random cropping, scaling and padding, horizontal flipping, adjusting lighting, brightness, contrast, hue (see Vazquez at page 5, [20]: “In some embodiments of the method, the geometrical transformation comprises at least one of: rotation, translation, cropping, and zooming.” and “In some embodiments of the system, the color transformation comprises at least one of: a hue variation, a saturation variation, a contrast variation, and a brightness variation”. Further, see VAZQUEZ page 1, [20]: “A classic example of data augmentation is the inclusion in the training data of the horizontally flipped training images, because we know that in nature the horizontal mirror of an object is visually still a valid object”)
It would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the combination of Salman and Ionescu with the above teachings of Vazquez, by computer-implemented method for adaptive data labelling to enhance machine learning precision, as taught by Salman and Ionescu, wherein the data augmentation techniques include at least one of random cropping, sampling and padding, horizontal flipping, adjusting lightning, brightness, contrast, or hue, as taught by Vazquez. The modification would have been obvious because one of ordinary skills in the art would be motivated to enhance the model training, in the absence of abundant high quality data for training, by incorporating data augmentation (see Vazquez at page 5, [20]: “In some embodiments of the method, the geometrical transformation comprises at least one of: rotation, translation, cropping, and zooming.” and “In some embodiments of the system, the color transformation comprises at least one of: a hue variation, a saturation variation, a contrast variation, and a brightness variation”. Further, see VAZQUEZ page 1, [20]: “A classic example of data augmentation is the inclusion in the training data of the horizontally flipped training images, because we know that in nature the horizontal mirror of an object is visually still a valid object”)
However, Salman, Ionescu, and Vazquez do not teach, “adding synthetic weather conditions” -- Xu teaches augmenting for weather related image data in machine learning (see Xu [9]: “The data augmentation method can be applied to uses other than the fine tuning of DNN that processes an image captured by a target camera. For example, training and test images are taken from the same camera but at different times (e.g., day to night), different weather (e.g., cloudy against sunny)” etc.)
It would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the combination of Salman, Ionescu and Vazquez with the above teachings of Xu, by computer-implemented method for adaptive data labelling to enhance machine learning precision, as taught by Salman, Ionescu, and Vazquez, adding synthetic weather conditions, as taught by Xu. The modification would have been obvious because one of ordinary skills in the art would be motivated to enhance the model training, in the absence of abundant high quality data for training, by incorporating data augmentation. (See Xu [9]: “The data augmentation method can be applied to uses other than the fine tuning of DNN that processes an image captured by a target camera. For example, training and test images are taken from the same camera but at different times (e.g., day to night), different weather (e.g., cloudy against sunny)” etc.)
Referring to claim 16 which is a dependent of claim 14, Salman teaches the system of claim 14, however, it fails to teach the limitation (of claim 14), “a data augmentation component configured to augment the labeled data with various data transformations to increase the amount of labeled data” and the limitation of claim 16, “wherein the data augmentation component is configured to perform random cropping, scaling and padding, horizontal flipping, lighting, brightness, contrast, hue, augmentations by adding different weather conditions, and combinations thereof.”
Ionescu teaches data augmentation component configured to augment the labeled data with various data transformations to increase the amount of labeled data (See Ionescu [Introduction]: “automatic data augmentation has emerged as a technique to improve the amount and diversity of training data through image transformation”).
It would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teachings of Salman with the above teachings of Ionescu, by the adaptive data labeling system, as taught by Salman, to augment the labeled data with various data transformations to increase the amount of labeled data, as taught by Ionescu. The modification would have been obvious because one of ordinary skills in the art would be motivated to enhance the model training, in the absence of abundant high quality data for training, by incorporating data augmentation. (See Ionescu [Introduction]: “automatic data augmentation has emerged as a technique to improve the amount and diversity of training data through image transformation”).
However, Ionescu fails to teach, “wherein the data augmentation component is configured to perform random cropping, scaling and padding, horizontal flipping, lighting, brightness, contrast, hue, augmentations by adding different weather conditions, and combinations thereof”
VAZQUEZ teaches, (wherein the) data augmentation component is configured to perform random cropping, scaling and padding, horizontal flipping, lighting, brightness, contrast, hue (See page 5, [20]: “In some embodiments of the method, the geometrical transformation comprises at least one of: rotation, translation, cropping, and zooming.” Further, see page 1, [20]: “A classic example of data augmentation is the inclusion in the training data of the horizontally flipped training images, because we know that in nature the horizontal mirror of an object is visually still a valid object” Further, see page 5,[20]: “In some embodiments of the system, the color transformation comprises at least one of: a hue variation, a saturation variation, a contrast variation, and a brightness variation” Etc.)
It would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the combination of Salman and Ionescu with the above teachings of Vazquez, by an adaptive data labeling system, as taught by Salman and Ionescu, wherein the data augmentation component is configured to perform random cropping, scaling and padding, horizontal flipping, lighting, brightness, contrast, hue, as taught by Vazquez. The modification would have been obvious because one of ordinary skills in the art would be motivated to enhance the model training, in the absence of abundant high quality data for training, by incorporating data augmentation (see Vazquez at page 5, [20]: “In some embodiments of the method, the geometrical transformation comprises at least one of: rotation, translation, cropping, and zooming.” and “In some embodiments of the system, the color transformation comprises at least one of: a hue variation, a saturation variation, a contrast variation, and a brightness variation”. Further, see VAZQUEZ page 1, [20]: “A classic example of data augmentation is the inclusion in the training data of the horizontally flipped training images, because we know that in nature the horizontal mirror of an object is visually still a valid object”)
However, Vazquez does not teach, “augmentations by adding different weather conditions, and combinations thereof”
Xu teaches augmenting for weather related image data in machine learning (See Xu [9]: “The data augmentation method can be applied to uses other than the fine tuning of DNN that processes an image captured by a target camera. For example, training and test images are taken from the same camera but at different times (e.g., day to night), different weather (e.g., cloudy against sunny)” etc.)
It would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the combination of Salman, Ionescu and Vazquez with the above teachings of Xu, by having an adaptive data labeling system, as taught by Salman, Ionescu, and Vazquez, adding synthetic weather conditions, as taught by Xu. The modification would have been obvious because one of ordinary skills in the art would be motivated to enhance the model training, in the absence of abundant high quality data for training, by incorporating data augmentation. (See Xu [9]: “The data augmentation method can be applied to uses other than the fine tuning of DNN that processes an image captured by a target camera. For example, training and test images are taken from the same camera but at different times (e.g., day to night), different weather (e.g., cloudy against sunny)” etc.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AZHARUL KHAN whose telephone number is (571)272-0836. The examiner can normally be reached M-F 7 am to 5 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571)270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Azharul Khan/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126