Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-7, 11-17, and 21-26 are pending. This Office Action is responsive to the amendment filed on 03/30/2026, which has been entered in the above identified application.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-6 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites “the automated data collection tool”. There is insufficient antecedent basis for this limitation. At best, a previous recitation reads “an automated data collector”.
Claims 2-6 are rejected for their dependency on an indefinite independent claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4, 7, 11-12, 14, 17, 21-22 and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan et al. (US 10981272 B1, filed 12/18/2017), hereinafter Nagarajan; in view of Hendrycks et al. (“AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty”, published 02/17/2020), hereinafter Hendrycks; in further view of Silveira et al. (US 20200265511 A1, filed 02/16/2019). Nagarajan and Silveira were cited in previous Office Actions.
Regarding Claim 1, Nagarajan teaches a deep learning training and inference system for a primary machine learning system (Nagarajan: “The server system 120 can perform retrospective analysis and simulation of failed grasp attempts based on data shared by multiple robots. The server system 120 can train a grasp model in a manner that incorporates information gained from the varied situations and conditions encountered by all of the robots 110” [Col 5, Lines 11-16]), comprising:
at least one hardware processor (Nagarajan: “Embodiments of the invention may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium may be a non-transitory computer readable storage medium, a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus may include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.” [Col. 22, Lines 14-33]);
an automated data collector executable by the at least one hardware processor via execution of the instructions and receptive to incoming input data from a sensor data source and embeds one or more sensor data classifications associated with the incoming input data (Nagarajan: “the robot 210 receives input sensor data 260 from its sensors 213” [Col. 14, Lines 49-50], “sensor data 260 collected by sensors 213 on the robot 210 may be input to a classification module 272” [Col. 14, Lines 64-66]; “the computing system of the robot 110 may be configured to include a classification module. The classification module may accept, as input, information related to an object in the robot's surroundings, e.g., images or video of an object, or estimated dimensions of an object. The classification module may associate the identified object with a particular object class, e.g., to label the object as a cup, pen, chair, or other type of object. The classification module may be implemented as a machine learning model, such as a neural network.” [Col. 6, Line 64 – Col. 7, Line 6]; “For example, the grasp model may include a neural network that has been trained to output grasp parameters in response to input information about a grasping task, e.g., sensor data describing the object, data indicating a location or pose of the object, a classification of the object, etc.” [Col. 5, Lines 30-35]; In light of paragraph [0025] of the specification and [Fig. 3], which provide examples of classifications, BRI of “sensor data classification” refers to observations made from the sensor data of the environment of the sensor, the subject of the sensor data, the nature of the sensor data, etc.);
a data augmenter executable by the at least one hardware processor via execution of the instructions and receptive to the input data from the automated data collection tool to generate an augmented input data set from one or more predefined expansion operations upon the input data of an application of ordered variations to the input data (Nagarajan: “the server system 120 can alter or augment the sensor data used by the grasp model, for example, by adding noise or random changes, or by altering parameters of the task” [Col. 5, Lines 52-55]; “the server system 120 may create new simulation scenarios by augmenting the annotated grasp attempt data. For example, the server system 120 may add noise or randomness to sensor data used to generate inputs to the grasp model before simulating a grasp. The server system 120 may create several variations of a grasp scenario through different transformations or alterations to the sensor data. By simulating the grasp using augmented data, the neural network grasp model may be tested for a wider variety of situations and conditions.” [Col. 12, Lines 37-46]; “the server system 320 may augment some or all of the grasp attempt data 360 before using it to train the grasp model (374). For example, the server system 320 may filter or preprocess video image data or other data sets of the grasp attempt data 360. In some implementations, the server system 320 may add or subtract noise from some or all of the grasp attempt data 360, to evaluate the robustness of the grasp model for data of varying quality. In some implementations, the server system 320 may systematically modify some or all of the grasp attempt data 360 to emulate different grasp conditions. For example, the server system 320 may reduce the intensity for all pixels within the video images to emulate a grasp attempt in a low-light scenario.” [Col. 18, Lines 44-56]);
an adaptive trainer executable by the at least one hardware processor via execution of the instructions and receptive to the augmented input data set to improve performance with a new set of weight values being generated for the primary machine learning system from a training, validation and adaptation process with the primary machine learning system using the augmented input data set, the adaptive trainer being in communication with one or more trainers for the primary machine learning system to provide the augmented input data set thereto (Nagarajan: “After augmenting the data if desired, the server system 320 may use the grasp attempt data 360, the augmented data, and/or the annotation result to train a grasp model (376)” [Col. 18, Lines 59-61], “updating the grasp model may involve determining new synaptic weights, node connectivity, or other neural network parameters of the grasp model” [Col. 19, Lines 12-14]; “The server system annotates the data, trains and updates a neural network grasp model using the annotated data. In some examples, the server system may augment the data prior to updating the grasp model. The server system may simulate the grasp attempt using the updated grasp model to verify that the updated model provides improved grasp performance over the previous model. In some cases, the server system may iterate the process one or more times, augmenting the data, training the model, and verifying the updated model in repeated cycles. When the server system determines that it has generated a final updated neural network grasp model, it then distributes the updated model to the one or more robots.” [Col. 17, Lines 9-22]; “the server system 320 may iterate all or part of the process 370 to refine and/or verify the updated grasp model. For example, after updating and verifying the grasp model with one set of grasp attempt data 360, the server system 320 may simulate and verify the updated grasp model using another, different set of grasp attempt data 360. In some implementations, the server system 360 may augment the grasp attempt data 360 it used to update the grasp model then retrain and/or re-simulate the grasp attempt using the augmented data.” [Col. 19, Lines 25-34]); and
an inference model interface executable by the at least one hardware processor via execution of the instructions and in communication with the adaptive trainer to receive the new set of weight values for an inference model simulator emulating a native hardware environment of the primary machine learning system (Nagarajan: “the server system 360 may augment the grasp attempt data 360 it used to update the grasp model then retrain and/or re-simulate the grasp attempt using the augmented data” [Col. 19, Lines 31-34], “the server system may iterate the analysis process, e.g., augmenting data, training a model, and simulating scenarios repeatedly until the system generates a satisfactory updated grasp model” [Col. 1, Lines 59-62]),
the inference model simulator adjusting various tuning parameters during an inference model simulation process with the new set of weight values before reinvoking the training pipeline, the reinvoking being performed by the at least one hardware processor by execution of the instructions and being based upon measured performance of the inference model simulator using the new set of weight values after the various tuning parameters are adjusted for iteratively improving the primary machine learning system (Nagarajan: “For the inputs to the grasp model representing the conditions of the grasp attempt, the model outputs that resulted in the unsuccessful grasp can be specified to high cost or error, which can then be back propagated or otherwise used to update the neural network parameters.” [Col. 12, Lines 4-9]; “In some implementations, when attempting to grasp an object, robots 110 generate grasp parameters which may include, for example, a type of grasp to use for an object, a location on the object to initiate contact, an amount of force to apply during the grasp, a trajectory or approach for moving an actuator to the object, and so on. The robots 110 may determine these and other parameters for carrying out a grasp using a grasp model that may be stored locally in an on-board computing system of the robot 110. For example, the grasp model may include a neural network that has been trained to output grasp parameters in response to input information about a grasping task, e.g., sensor data describing the object, data indicating a location or pose of the object, a classification of the object, etc.” [Col. 5, Lines 22-35]; “If some of the simulations result in inappropriate grasp parameters, the server system 120 can continue training the model to enhancing robustness to noise and other variations in sensor data.” [Col. 12, Lines 46-50]; “In some implementations, the method further includes training the grasp model; determining, based on simulations of grasp attempts using the trained grasp model that are carried out based on the sensor data for the unsuccessful grasp attempts, that the trained grasp model does not provide at least a minimum level of performance; and, based on determining that the trained grasp model does not provide at least the minimum level of performance, continuing to train the grasp model.” [Col. 4, Lines 18-26]); and
a non-transitory memory storing instructions executable by the one or more hardware processors to implement the automated data collector, the data augmenter, the adaptive trainer, and the inference model (Nagarajan: “Implementations of the disclosed techniques may include hardware (e.g., a system including one or more computers), a method or process implemented at least partially in hardware, or a non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the computers to perform operations that carry out the disclosed techniques.” [Col. 4, Lines 46-52]).
However, Nagarajan fails to expressly disclose ordered variations comprising a hierarchical sequence in which output of a first predefined expansion operation is used as input to a second predefined expansion operation to generate at least second-level augmented input data variants; and the inference model interface selectively invoking one or more of the automated data collector to collect additional data, the data augmenter to increase the augmented input data set, and the adaptive trainer.
In the same field of endeavor, Hendrycks teaches ordered variations comprising a hierarchical sequence in which output of a first predefined expansion operation is used as input to a second predefined expansion operation to generate at least second-level augmented input data variants (Hendrycks: “AUGMIX is a data augmentation technique which improves model robustness and uncertainty esti mates, and slots in easily to existing training pipelines. At a high level, AugMix is characterized by its utilization of simple augmentation operations in concert with a consistency loss. These augmentation operations are sampled stochastically and layered to produce a high diversity of augmented images.” [Section 3. AugMix], See [Fig. 4], in which multiple augmentation operations are layered in sequence chains, then combined in parallel).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated ordered variations comprising a hierarchical sequence in which output of a first predefined expansion operation is used as input to a second predefined expansion operation to generate at least second-level augmented input data variants, as taught by Hendrycks to the system of Nagarajan because both of these systems are directed towards ensuring robustness of a trained model through data augmentation. In making this combination and applying sequences of augmentation operations that cascade into one another, it would allow the system of Nagarajan to “generate diverse transformations”, thereby avoiding “the memorization of fixed augmentations”, which is “a common failure mode of deep models in the arena of corruption robustness” (Hendrycks: [Section 3. AugMix]).
Nagarajan and Hendrycks still fail to expressly disclose the inference model interface selectively invoking one or more of the automated data collector to collect additional data, the data augmenter to increase the augmented input data set, and the adaptive trainer.
In the same field of endeavor, Silveira teaches the inference model interface selectively invoking one or more of the automated data collector to collect additional data, the data augmenter to increase the augmented input data set, and the adaptive trainer (Silveira: “After the model is constructed, the test data is fed into the model to test its accuracy (step 1610). In an embodiment the model is tested using mean absolute error, which examines each prediction in the model and provides an average error score for each prediction. If the error rate between the training and test dataset is below a predetermined threshold, the model has learned the dataset's pattern and passed the test.” [0113]; “If the model fails the test: the hyperparameters of the model are changed and/or the training and test data are re-randomized or more data is collected (either through obtaining more samples or through data augmentation), and the iterative analysis of the training data is repeated” [0114]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated the inference model interface selectively invoking one or more of the automated data collector to collect additional data, the data augmenter to increase the augmented input data set, and the adaptive trainer, as taught by Silveira to the system of Nagarajan and Hendrycks because both of these systems are directed towards iterative training and optimization of a machine learning model. In making this combination and selecting the next step in the iterative process based on the model performance, it would allow the system of Nagarajan and Hendrycks to “continuously improve[[s]] its model using feedback from application to new empirical data” (Silveira: [0115]).
Regarding Claim 2, Nagarajan, Hendrycks, and Silveira teach the system of Claim 1, wherein the sensor data source is connected to a microphone and the incoming input data is an audio data stream (Nagarajan: “a robot 110 typically has sensory capabilities, e.g., an image sensor, an audio microphone, lidar, radar, or other sensors” [Col. 6, Lines 1-3], “the sensor data that describes a grasp attempt may include one or more images captured during the grasp attempt, video data showing the grasp attempt, audio data indicating sounds occurring during the grasp attempt” [Col. 19, Line 65 – Col. 20, Line 2]).
Regarding Claim 4, Nagarajan, Hendrycks, and Silveira teach the system of Claim 2, wherein the augmented input data set is generated from the input data by applying an audio process thereto, the audio process being selected from a group consisting of: addition of noise, addition of reverberation, speed increase, and speed decrease (Nagarajan: “training and verifying the grasp model using augmented grasp attempt data, for example, grasp attempt data to which noise has been added or intensity levels adjusted” [Col. 2, Lines 27-31]).
Regarding Claims 7, 12, 14, 17, 22 and 24, they are method and article of manufacture claims that correspond to Claims 1, 2 and 4. Therefore, they are rejected for the same reasons as Claims 1, 2 and 4 above.
Regarding Claim 11, Nagarajan, Hendrycks, and Silveira teach the method of Claim 7, further comprising: simulating, with the inference model simulator, the native hardware environment of the primary machine learning system with the additional new set of weight values for the primary machine learning system in a subsequent training iteration, the inference model simulator adjusting various tuning parameters during the inference model simulation process (Nagarajan: “the server system may iterate the analysis process, e.g., augmenting data, training a model, and simulating scenarios repeatedly until the system generates a satisfactory updated grasp model” [Col. 1, Lines 59-62]; “In some implementations, when attempting to grasp an object, robots 110 generate grasp parameters which may include, for example, a type of grasp to use for an object, a location on the object to initiate contact, an amount of force to apply during the grasp, a trajectory or approach for moving an actuator to the object, and so on. The robots 110 may determine these and other parameters for carrying out a grasp using a grasp model that may be stored locally in an on-board computing system of the robot 110. For example, the grasp model may include a neural network that has been trained to output grasp parameters in response to input information about a grasping task, e.g., sensor data describing the object, data indicating a location or pose of the object, a classification of the object, etc.” [Col. 5, Lines 22-35]).
Regarding Claim 21, it is an article of manufacture claim that corresponds to Claim 11. Therefore, it is rejected for the same reason as Claim 11 above.
Claims 3, 13 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan in view of Hendrycks and Silveira, as applied to Claims 1, 7 and 17 above, in further view of Raux (US 20110224979 A1, filed 03/09/2010). Raux was cited in the previous Office Action.
Regarding Claim 3, Nagarajan, --Hendrycks, and Silveira teach the system of Claim 2. However, they fail to expressly disclose wherein the one or more sensor data classifications is selected from a group consisting of: distance to microphone, room size, speaker age, and speaker gender.
In the same field of endeavor, Raux teaches wherein the one or more sensor data classifications is selected from a group consisting of: distance to microphone, room size, speaker age, and speaker gender (Raux: “the biometric data are determined or estimated from visual information captured by the speech recognition device 100. The biometric data may include, but are not limited to, height, gender, weight, age, and ethnicity of a speaker” [0046]; “The feature extractor 360 also extracts environmental features 324. The environmental features 324 are correlated with reverberation of acoustic signals in the environment. To extract the environmental features 324, the feature extractor 360 may use Simultaneous Localization and Mapping (SLAM) or other computer vision algorithms to estimate the size or geometry of the room, the speaker's location and distance, and the location and distance of the microphone.” [0057]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the one or more sensor data classifications is selected from a group consisting of: distance to microphone, room size, speaker age, and speaker gender, as taught by Raux to the system of Nagarajan, Hendrycks, and Silveira because both of these systems are directed towards collecting and processing data from audio sensors. In making this combination and collecting classification data about the speaker, as taught by Raux, it would allow the system of Nagarajan, Hendrycks, and Silveira to adapt its speech recognition based on the features of the specific speaker, as “such biometric data [is] correlated with the vocal characteristics of the speakers” (Raux: [0046]).
Regarding Claims 13 and 23, they are method and article of manufacture claims that correspond to the system of Claim 3. Therefore, they are rejected for the same reason as Claim 3 above.
Claims 5, 15 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan in view of Hendrycks and Silveira, as applied to Claims 1, 7 and 17 above, in further view of Plumbey et al. (US 20210027864 A1, filed 03/29/2019), hereinafter Plumbey. Plumbey was cited in a previous Office Action.
Regarding Claim 5, Nagarajan, ---Hendrycks, and Silveira teach the system of Claim 1. However, they fail to expressly disclose wherein the one or more training tools for the primary machine learning system are specific to a training category, each of the one or more training tools independently iterating through a training, validation, and adaptation loop for a given one of the training categories.
In the same field of endeavor, Plumbey teaches wherein the one or more training tools for the primary machine learning system are specific to a training category, each of the one or more training tools independently iterating through a training, validation, and adaptation loop for a given one of the training categories (Plumbey: “An iterative procedure/feedback loop may be performed for generating the property model, the procedure including: generating a prediction result list for a plurality of compounds and their association with the particular property based on the property model; validating the property model based on compounds from the prediction result list having an association with the particular property; and updating the property model based on the property model validation. The procedure/loop may be repeated using the updated property model until it is determined the property model has been validly trained” [Abstract]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the one or more training tools for the primary machine learning system are specific to a training category, each of the one or more training tools independently iterating through a training, validation, and adaptation loop for a given one of the training categories, as taught by Plumbey to the system of Nagarajan, Hendrycks, and Silveira because both of these systems are directed towards iteratively training a machine learning model. In making this combination and iterating through a training/validation/updating loop to produce a trained model, as taught by Plumbey, it would allow the system of Nagarajan, Hendrycks, and Silveira to better train the model to handle the exponentially increasing complexity of the data when there is a large amount of data characteristics (Plumbey: see Paragraph [0004]).
Regarding Claims 15 and 25, they are method and article of manufacture claims that correspond to the system of Claim 5. Therefore, they are rejected for the same reason as Claim 5 above.
Claims 6, 16 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Nagarajan in view of Hendrycks and Silveira, as applied to Claims 1, 7 and 17 above, in further view of Li et al. (US 20210097443 A1, filed 09/27/2019), hereinafter Li. Li was cited in a previous Office Action.
Regarding Claim 6, Nagarajan, Hendrycks, and Silveira teach the system of Claim 1. However, they fail to expressly disclose wherein the inference tool generates a set of hyperparameter updates to the adaptive training tool, the set of hyperparameters governing the function of the adaptive training tool.
In the same field of endeavor, Li teaches wherein the inference tool generates a set of hyperparameter updates to the adaptive training tool, the set of hyperparameters governing the function of the adaptive training tool (Li: “A PBT system can train a machine learning model while simultaneously learning improved hyperparameter values. The PBT system can warm-start a model with updated model and hyperparameter values at the same time, instead of training the model, searching for hyperparameter values separately, and then re-training the model according to the new hyperparameter values” [0008]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the inference tool generates a set of hyperparameter updates to the adaptive training tool, the set of hyperparameters governing the function of the adaptive training tool, as taught by Li to the system of Nagarajan, Hendrycks, and Silveira because both of these systems are directed towards iteratively training a machine learning model. In making this combination and generating hyperparameter values to update the model training, as taught by Li, it would allow the system of Nagarajan, Hendrycks, and Silveira to compare iterations of the model through the training process and keep the better performers (Li: see Paragraphs [0006] – [0007]).
Regarding Claims 16 and 26, they are method and article of manufacture claims that correspond to the system of Claim 6. Therefore, they are rejected for the same reason as Claim 6 above.
Response to Arguments
The Examiner acknowledges the Applicant’s amendments to Claims 1, 7, 11, 17, and 21. It is noted that the status of Claim 17 currently reads “Previously Presented” despite the present amendments.
Applicant’s arguments, filed 03/30/2026, regarding the rejection of Claims 1-6 under 35 U.S.C. § 101 have been fully considered and are persuasive. The rejection has been withdrawn.
Applicant’s arguments, filed 03/30/2026, regarding the rejection of Claims 1-7, 11-17, and 21-26 under 35 U.S.C. § 103 have been fully considered and are found moot in light of the new ground(s) of rejection (see rejection above).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Amershi et al. (“Software Engineering for Machine Learning: A Case Study”) discusses best practices and strategies for constructing machine learning pipelines through a combination of data-oriented and model-oriented workflow stages with multiple feedback loops selected based on model diagnostics upon evaluation.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MEGAN E HWANG whose telephone number is (703)756-1377. The examiner can normally be reached Monday-Thursday 10:00AM-7:30PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.E.H./Examiner, Art Unit 2143
/JENNIFER N WELCH/Supervisory Patent Examiner, Art Unit 2143
Prosecution Insights