DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 07/19/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Gupta et al. (Gupta), US Patent No. 11,544,566 B2.
As to independent claim 1, Gupta discloses a computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
acquiring at least one digital dataset (col. 7, lines 48-61: training data may include input data (dataset), both structured and unstructured data, wherein the training data may include images and documents; col. 8, lines 16-26: training data is retrieved form a public database);
training a neural network model using the dataset, the neural network model
comprising a plurality of layers, each of the layers comprising a plurality of layer
parameters (col. 7, lines 48-61: training data is used to train a model; col. 7, line 62 – col. 8, line 8: deep neural network architecture may have multiple layers, wherein each layer may have multiple neurons or nodes (parameters); col. 8, lines 16-26: collecting and passing training data through deep model layers in a deep network and the provenance data is collected on the training data for that iteration);
analyzing accuracy and loss of a partially trained version of the neural network
model following each of one or more epochs to identify when an epoch corresponds to an anomalous pattern, the epochs each representing a complete iteration of the dataset by the neural network model (col. 5, line 61 – col. 6, line 9: a reasoning model may be built to generate insights about deep learning models and the reasoning model may assist in detecting a non-optimal performance or a poor performance (anomalous pattern) of a model by analyzing model parameters. For example, the reasoning model can detect poor performance (anomalous pattern) in a model by analyzing model parameters using provenance data (collected and analyzed during the training phase of the deep learning model), wherein model parameters may include data distribution, adversarial samples, corrupted samples, underfitting, overfitting, model training issues or sampling issues; col. 6, lines 21-43: underfitting or overfitting (anomalous pattern) of a training model may be determined by analyzing node activations at various layers and at various iterations; col. 8, lines 27-44: number of iterations are evaluated to identify the accuracy of the collected data and the accuracy of the trained algorithm);
searching the layer parameters of the partially trained version of the neural
network model for the identified epoch corresponding to the anomalous pattern to
identify the layer parameters contributing to the anomalous pattern (col. 6, lines 21-43: if a node in a higher layer hierarchy activates for different class samples at initial iterations but does not activate for the different class samples at later iterations, then underfitting or overfitting may present);
modifying a subset of the layer parameters of the partially trained neural network
model with the layer parameters identified by the searching (col. 8, lines 27-55: training data relating to animal species as an example, and the collected provenance data is evaluated by the analytics model and the neuron weights have converged quickly after 5 iterations. Therefore, the fast convergence of neuron weights may indicate that the training data of the specified animal species is quality data. Neuron wights analysis may also include evaluating the weights of the neurons, and if the neuron or the filter is widely fluctuating, then a parameter may be need to be amended or altered, such as by changing the momentum. The neuron weight analysis may include comparing the weights of the neurons across the iterations and by plotting a graph to visually illustrate the neuron behavior. Neuron learning feature analysis may include identifying if the neuron is learning the same feature. Filters may be added or reduced at each layer, for example, if a neuron is learning the same filters, then those filters at that particular layer may be pruned.); and
applying the modified subset of the layer parameters to the neural network model (col. 9, lines 6-19: at 306, the model size is reduced using the generated model insights. The model insights generated at step 304 may be used to reduce the model size. At each iteration, the provenance data is analyzed, and the network size may be pruned or reduced. Upon a reduction of the network size, the model training is continued on the reduced network; col. 9, lines 42-49: at 308, a final trained model is created. For example, the final trained model may be created after n number of iterations and based on the analyses in the previous steps).
As to dependent claim 2, Gupta discloses wherein the epoch corresponding to the anomalous pattern is a subset of one or more partially trained versions of the neural network model (col. 9, lines 6-19).
As to dependent claim 3, Gupta discloses wherein searching the layer parameters of the partially trained version of the neural network model comprises performing a search operation to compare the layer parameters between partially trained versions of the neural network model and identify the layer parameters contributing to the anomalous pattern (col. 8, lines 45-56; col. 10, lines 3-21).
As to dependent claim 4, Gupta discloses wherein the search operation returns the subset of the layer parameters for one or more of the layers contributing to the anomalous pattern (col. 8, lines 45-56; col. 10, lines 3-21).
As to dependent claim 5, Gupta discloses wherein applying the modified subset of the layer parameters to the neural network model comprises loading the partially trained version of the neural network model with the subset of the layer parameters returned by the search operation to generate a modified version of the neural network model (col. 9, lines 6-19).
As to dependent claim 6, Gupta discloses further comprising generating a detailed report of accuracy and loss for the dataset for the modified version of the neural network model (col. 9, lines 42-49 and col. 11, lines 17-26).
As to dependent claim 7, Gupta discloses further comprising generating a detailed report of accuracy and loss for the dataset for each partially trained version of the neural network model (col. 3, line 51 – col. 4, line 7).
As to dependent claim 8, Gupta discloses wherein generating the detailed report includes positive and negative prediction details for each record in the dataset for all partially trained versions of the neural network model (col. 10, lines 3-21).
As to dependent claim 9, Gupta discloses wherein the anomalous pattern comprises at least one of underfitting and overfitting (col. 6, lines 21-43).
As to dependent claim 10, Gupta discloses further comprising determining the accuracy and loss of the neural network model after applying the modified subset of the layer parameters thereto (col. 9, lines 6-19).
As to dependent claim 11, Gupta discloses further comprising generating a record generalization score representing a prediction accuracy for each test record of the dataset (col. 8, lines 27-44 and col. 10, lines 3-21).
As to independent claim 12, Gupta discloses a computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
training a neural network model, the neural network model comprising a plurality
of layers, each of the layers comprising a plurality of layer parameters (col. 7, lines 48-61: training data is used to train a model; col. 7, line 62 – col. 8, line 8: deep neural network architecture may have multiple layers, wherein each layer may have multiple neurons or nodes (parameters); col. 8, lines 16-26: collecting and passing training data through deep model layers in a deep network and the provenance data is collected on the training data for that iteration);
saving a partially trained version of the neural network model on a computer-
readable storage medium following each of one or more epochs (col. 7, lines 8-33 and col. 12, lines 9-19: the deep learning program may interact with a database that may be embedded in various storage device), the epochs each representing a complete iteration of a training dataset by the neural network model col. 8, lines 27-44: number of iterations are evaluated to identify the accuracy of the collected data and the accuracy of the trained algorithm);
generating a detailed report of accuracy and loss for the dataset for each partially
trained version of the neural network model (col. 11, lines 17-25: at 410, generated insights are provided to the user, which may be produced to a user operating a computing device. The generated insights may include visual aids, graphs, charts, images, videos or typewritten messages. The generated insight output may produce a varying range of data analytics to the user based on the analyses at, for example, steps 304 and 408. The output data transmitted to the user may provide the user with insight to make decisions relating to model training, use and accuracy);
identifying, based on the accuracy and loss, when an epoch corresponds to an
anomalous pattern (col. 5, line 61 – col. 6, line 9: a reasoning model may be built to generate insights about deep learning models and the reasoning model may assist in detecting a non-optimal performance or a poor performance (anomalous pattern) of a model by analyzing model parameters. For example, the reasoning model can detect poor performance (anomalous pattern) in a model by analyzing model parameters using provenance data (collected and analyzed during the training phase of the deep learning model), wherein model parameters may include data distribution, adversarial samples, corrupted samples, underfitting, overfitting, model training issues or sampling issues; col. 6, lines 21-43: underfitting or overfitting (anomalous pattern) of a training model may be determined by analyzing node activations at various layers and at various iterations; col. 8, lines 27-44: number of iterations are evaluated to identify the accuracy of the collected data and the accuracy of the trained algorithm);;
searching the layer parameters of the partially trained version of the neural network model for the identified epoch corresponding to the anomalous pattern to
identify the layer parameters contributing at least one of underfitting and overfitting (col. 6, lines 21-43: if a node in a higher layer hierarchy activates for different class samples at initial iterations but does not activate for the different class samples at later iterations, then underfitting or overfitting may present);
modifying a subset of the layer parameters of the partially trained neural network
model with the layer parameters identified by the searching (col. 8, lines 27-55: training data relating to animal species as an example, and the collected provenance data is evaluated by the analytics model and the neuron weights have converged quickly after 5 iterations. Therefore, the fast convergence of neuron weights may indicate that the training data of the specified animal species is quality data. Neuron wights analysis may also include evaluating the weights of the neurons, and if the neuron or the filter is widely fluctuating, then a parameter may be need to be amended or altered, such as by changing the momentum. The neuron weight analysis may include comparing the weights of the neurons across the iterations and by plotting a graph to visually illustrate the neuron behavior. Neuron learning feature analysis may include identifying if the neuron is learning the same feature. Filters may be added or reduced at each layer, for example, if a neuron is learning the same filters, then those filters at that particular layer may be pruned.); and
determining the accuracy and loss of the neural network model after the subset of the layer parameters has been modified (col. 9, lines 6-49: at 306, the model size is reduced using the generated model insights, the model insights generated at step 304 may be used to reduce the model size. At each iteration, the provenance data is analyzed, and the network size may be pruned or reduced. Upon a reduction of the network size, the model training is continued on the reduced network. Over time, the neuron weights may be converging, thus allowing additional reasoning to remove the neurons at the lower weight layer to reduce the model size while retaining the accuracy of the model).
As to dependent claim 13, Gupta discloses wherein the computer-readable storage medium for saving the partially trained version of the neural network model comprises a local device or a network device (col. 13, lines 8-33).
As to dependent claim 14, Gupta discloses wherein the epoch corresponding to the anomalous pattern is a subset of one or more partially trained versions of the neural network model (col. 9, lines 6-19).
As to dependent claim 15, Gupta discloses wherein searching the layer parameters of the partially trained version of the neural network model comprises performing a search operation to compare the layer parameters between partially trained versions of the neural network model and identify the layer parameters contributing to the anomalous pattern (col. 8, lines 45-56; col. 10, lines 3-21).
As to dependent claim 16, Gupta discloses wherein the search operation returns the subset of the layer parameters for one or more of the layers contributing to the anomalous pattern (col. 8, lines 45-56; col. 10, lines 3-21).
As to dependent claim 17, Gupta discloses further comprising loading the partially trained version of the neural network model with the subset of the layer parameters returned by the search operation to generate a modified version of the neural network mode (col. 9, lines 6-19).
As to dependent claim 18, Gupta discloses further comprising saving the modified version of the neural network model on the computer-readable storage medium (col. 9, lines 42-49 and col. 11, lines 17-26).
As to dependent claim 19, Gupta discloses wherein generating the detailed report includes positive and negative prediction details for each record in the dataset for all partially trained versions of the neural network model (col. 10, lines 3-21).
As to dependent claim 20, Gupta discloses further comprising generating a record generalization score representing a prediction accuracy for each test record of the dataset (col. 8, lines 27-44 and col. 10, lines 3-21).
Conclusion
Any inquiry concerning this communication should be directed to CHAU T NGUYEN at telephone number (571)272-4092. The examiner can normally be reached on M-F from 8am to 5pm (PT).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated-interview-request-air-form.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula, can be reached at telephone number 5712724128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center and Private PAIR for authorized users only. Should you have questions about access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/CHAU T NGUYEN/Primary Examiner, Art Unit 2145
Prosecution Insights