DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This action is responsive to applicant’s submission filed on 08/08/2025 has been entered. Claims 22, 24, 31, 38, 40, 41, and 42 are amended. Claims 31, 33, 35–36, 40, 42 and 44 are cancelled. Claim 45 is new.
Claims 22, 24–29, 38–39, 41, 43, and 45 are pending for examination.
Response to Arguments
In reference to 35 USC § 112(b)
Applicant’s arguments, filed on 08/08/2025, with respect to the claim § 112(b) rejections have been fully considered. The previous § 112(b) rejections have been withdrawn in view of the amendments.
In reference to 35 USC § 101
Applicant’s amendments and arguments, filed on 08/08/2025, with respect to the § 101 rejections have been fully considered and are persuasive.
Examiner notes that while the claims recite several limitations that are abstract ideas (mental processes), the claims as a whole are not directed to an abstract idea. Applicant amended the claims, which collectively now recite detailed devices and methods directed toward generating labeled data, in particular for training neural network models using unlabeled partitioned samples to prevent errors that exist in the labels at the beginning of the iterations. The newly amended independent claims now include "training neural network models to perform automated guidance of a robot or vehicle" and "performing automated guidance in the at least one of: the robot, or the vehicle the performing automated guidance including: recognizing objects in a single frame of sensor data representing an environment of the at least one of the robot or the vehicle using the nth trained first model: and recognizing attributes of the objects over time using the nth trained further model." These additional limitations are not abstract ideas (see MPEP 2106.04(a)). Thus, these limitations must be considered additional elements to the abstract idea. Examiner notes that these additional elements integrate the abstract idea into a practical application because the entire claim amounts to detailed methods that require implementing a specific combination of hardware with the methods of labeling unlabeled data using neural network to automatically perform guidance of a vehicle or robot (as opposed to a broad recitation at a high level of generality), and the specific combination of hardware and instructions recited in the additional element amounts to an improvement to the functioning of a computer/field, as set forth by MPEP 2106.05(a)), which states “the claim must include the components or steps of the invention that provide the improvement described in the specification.” Pursuant to this requirement set forth by the MPEP, examiner points out that the Specification states in at least [0005–0006, 0008, 0010, 0016, 0119–0129]: “The quality of the labels may affect the recognition performance of the trained models of the machine learning methods … The present provides a method for generating labels that is improved compared to the related art … An advantage of the example method of the present invention is that in the course of the iterative process the training of the first and of the further model is performed in every step using data that are disjunctive with respect to the data for which the respective model subsequently performs a prediction … Thus, it is possible to prevent errors that exist in the labels at the beginning of the iterations from propagating during the training of a respective model and further until the end of the iterative process.” Finally, as outlined in the Remarks submitted 08/08/2025, “Because the single-frame model and the time-based model are trained together but using disjunctively developed training sets according to the present invention, the overall system performance is improved.” Therefore, the additional elements reflect the improvement set forth and explains what the resulting improvement is.
Thus, the additional limitations do amount to significantly more, and the § 101 rejections are withdrawn.
In reference to 35 USC § 103
Applicant’s arguments filed on 08/08/2025, with respect to the newly amended limitations have been considered but are not persuasive. Applicant argues, beginning on Pg. 11 in the Remarks, that the “cited references do not disclose or suggest at least the above-highlighted claim features.” More specifically, applicant argues that “Chari's iterative method pulls in new unlabeled data with each iteration, and therefore does not repeatedly label the same unlabeled data sets with each iteration.” Examiner respectfully disagrees. Examiner contends that Chari, in at least paragraphs 28–31 and Figure 1, indeed teaches generating labels for a first subset of data and then generating data for an (n+1)th subset using a nth trained model including a neural network. Chari’s method includes creating clusters C1, C2, …. Ckcluster by predicting labels for the unlabeled data using a semi-supervised process including a model that is iteratively updated with training data (i.e., an nth trained model). Chari’s method is iterative and provides the labeled data to the next iteration (i.e., providing the labeled first subset as an nth labeled first subset). Furthermore, as pointed out by applicant, Chari in at least paragraph 31 “employs both labeled (e.g., known labels from the previous iterations) and unlabeled data for training.” Figure 1 clearly depicts semi-supervised clustering of the training data which creates subsets of labeled data using a model at that current iteration. Therefore, Chari teaches labeling more than once (e.g., expert and semi-supervised) i.e., labeling data from the same subset of data. Examiner notes, there is no specific requirement that the n+1 data cannot also include new data in addition to the same subset of data. The claim only requires the use of the same subset of data. Moreover, the broadest reasonable interpretation of the same subset of data includes Chari’s “initial sample set selection 102” which is iterated through the steps as shown in Figure 1. See § 103 rejections below for a detailed analysis.
Thus, the § 103 rejections are maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 22, 38–39, 41, 43, and 45 are rejected under 35 U.S.C. 103 as being unpatentable over Chari et al., (US Pub No: 20130097103 A1), hereinafter "Chari" in view of Cella et al,. (US 20180284758 A1), hereinafter “Cella”.
Regarding claim 22, Chari teaches:
Providing an unlabeled data set including a first subset of unlabeled data and a further subset of unlabeled data that is disjunctive with respect to the first subset of unlabeled data. Chari states, “An iterative method is applied herein, where, in each iteration, the present method draws (selects) a batch of samples (B), and domain experts provide the labels of the selected samples. Information embedded in the labeled sample is used to group together data elements which are very similar to the labeled sample using semi-supervised clustering. The class distribution in the clusters can then be estimated and used to perform a biased sampling of clusters to obtain a diverse balanced sample. Within each cluster, a diverse sample is obtained by using a maximum entropy sampling. The sample obtained at each iteration is then labeled and used in subsequent iterations.” (paragraph 28, emphasis added); Chari also states “Data is taken from an unlabeled data set. See ‘Unlabeled Data Set U’ in FIG. 1. As highlighted above, the starting point for the methodology is an initial (possibly empty, i.e., when an initial set is empty, no labeled data exists at the first iteration) set of labeled samples selected from the unlabeled data set. In step 102, a small set of data (e.g., from about 5% to about 10% of the desired training data set), is selected (sampled) from Data Set U. According to one exemplary embodiment, this initial sample set is created by random sampling” (paragraph 29, emphasis added). Chari selects a small subset of a larger subset of data (i.e., provides an unlabeled data set including a first dataset) which leaves the rest of Unlabeled Data Set U (i.e., a further subset of unlabeled data). The initial sample set is created randomly (i.e., the data is disjunctive).
Generating a labeled first subset of data by generating labels for the first subset of unlabeled data and providing the labeled first subset of data as an nth labeled first subset of data where n=1. Chari states, “In step 103, class labels of this small initial sample of the data are provided. According to an exemplary embodiment, the labels are provided by one or more domain experts (i.e., a person who is an expert in a particular area or topic) as is known in the art, e.g., by hand labeling the data” (paragraph 29). Chari also states “An iterative method is applied herein, where, in each iteration, the present method draws (selects) a batch of samples (B), and domain experts provide the labels of the selected samples. Information embedded in the labeled sample is used to group together data elements which are very similar to the labeled sample using semi-supervised clustering. The class distribution in the clusters can then be estimated and used to perform a biased sampling of clusters to obtain a diverse balanced sample. Within each cluster, a diverse sample is obtained by using a maximum entropy sampling. The sample obtained at each iteration is then labeled and used in subsequent iterations.” (paragraph 28, see also Figure 3). Chari’s method is iterative and provides the labeled data to the next iteration (i.e., providing the labeled first subset as an nth labeled first subset). Examiner notes, a person having ordinary skill in the art before the present application was filed would have known that the iterative number representation “N” can equal any number necessary to complete the task including N=1.
Implementing an iterative process, each nth iteration of the iterative process including the following steps for every n=1, 2, 3, ... N. Chari states, “The remaining samples to be labeled are picked in an iterative fashion, where each iteration produces a fraction of the desired sample size” (paragraph 31). Here, “sample size” refers to the amount of labeled data desired.
Generating an nth labeled further subset of data by predicting labels for the further subset of unlabeled data by using the nth trained model. Chari states, “In each iteration, semi-supervised clustering is applied to the data, incorporating the labeled samples from previous iterations. See step 106. As is known in the art, semi-supervised clustering employs both labeled (e.g., known labels from the previous iterations) and unlabeled data for training. Specifically, in step 106, the data from Data Set U is clustered using a semi-supervised clustering process. The result of the semi-supervised clustering is a plurality of clusters C1, C2, . . . , Ckcluster (see FIG. 1) which should have a biased class distribution.” (paragraph 31, see also Figure 1). Chari’s method includes creating clusters C1, C2, …. Ckcluster by predicting labels for the unlabeled data using a semi-supervised process including a model that is iteratively updated with training data (i.e., an nth trained model). Examiner notes the broadest reasonable interpretation of an nth trained model includes iterative states of the same base model e.g., base model + 10th training iteration is a distinct nth model.
Generating an (n+1)th labeled first subset of data by predicting labels for the first subset of unlabeled data by using the nth trained further model. Chari states, “In each iteration, semi-supervised clustering is applied to the data, incorporating the labeled samples from previous iterations. See step 106. As is known in the art, semi-supervised clustering employs both labeled (e.g., known labels from the previous iterations) and unlabeled data for training. Specifically, in step 106, the data from Data Set U is clustered using a semi-supervised clustering process. The result of the semi-supervised clustering is a plurality of clusters C1, C2, . . . , Ckcluster (see FIG. 1) which should have a biased class distribution.” (paragraph 31, see also Figure 1). Chari’s iterative method includes creating clusters C1, C2, …. Ckcluster (i.e., Cn+1) by predicting labels for the unlabeled data using a semi-supervised process including a model that is iteratively updated with training data (i.e., an nth trained model).
Training a first model using the nth labeled first subset of data as an nth trained first model, the nth trained first model including a neural network. Chari states “A simplistic approach is to assume that the class distribution of the cluster is exactly the same as the class distribution of the samples labeled in this cluster. This is based on the optimistic assumption that the semi-supervised clustering works perfectly and groups together elements which are similar to the labeled sample. First, determine how many samples one ideally wishes to draw from each class in this iteration from the total B samples to draw. Let li j be the number of instances of class j sampled after iteration i, and ρi j be the normalized proportion of samples with class label j, i.e., ρij=lij∑rlir. To increase the balancedness in the training step, one wants to select samples inversely proportional to their current distribution (see Liu, Chawla and Wu), i.e., nj=1-ρijl-1*B, where l is the number of classes and (l−1) is the normalization factor.” (paragraph 45, see also Figures 3 and 4). Chari’s method uses the clustered dataset (i.e., the nth labeled first subset; example C1) to train the ith iteration (i.e., nth trained first model) of the model. Examiner notes that Chari’s core methodologies include iterative machine learning using neural networks, and thus each model includes at least one neural network (see Chari paragraphs 0024, 0068, and 0072).
Training a further model using the nth labeled further subset of data as an nth trained further model, the nth trained further model including another neural network. Chari states “A simplistic approach is to assume that the class distribution of the cluster is exactly the same as the class distribution of the samples labeled in this cluster. This is based on the optimistic assumption that the semi-supervised clustering works perfectly and groups together elements which are similar to the labeled sample. First, determine how many samples one ideally wishes to draw from each class in this iteration from the total B samples to draw. Let li j be the number of instances of class j sampled after iteration i, and ρi j be the normalized proportion of samples with class label j, i.e., ρij=lij∑rlir. To increase the balancedness in the training step, one wants to select samples inversely proportional to their current distribution (see Liu, Chawla and Wu), i.e., nj=1-ρijl-1*B, where l is the number of classes and (l−1) is the normalization factor.” (paragraph 45, see also Figures 3 and 4). Chari also states “Next, the estimated class distribution in each cluster is used to select the appropriate number of samples from each class. Let θi j be the probability of drawing a sample with class label j from the previously labeled subset of cluster i. By assumption, this is exactly the probability of drawing a sample with class label j from the entire cluster i. Since it is desired to have nj samples with label j in this iteration” (paragraph 46). Chari’s method uses the clustered dataset (i.e., the nth labeled first subset; example C1) to train the ith iteration (i.e., nth trained further model) of the model. Examiner notes, a person having ordinary skill in the art before the present application was filed would have known that the iterative nth training process produces nth trained further outcomes (i.e., nth trained further models). Examiner notes that Chari’s core methodologies include iterative machine learning using neural networks, and thus each model includes at least one neural network (see Chari paragraphs 0024, 0068, and 0072.
Chari does not appear to explicitly teach the following limitations:
A method for training neural network models to perform automated guidance of a robot or vehicle;
Performing automated guidance in the at least one of: the robot, or the vehicle; the performing automated guidance including:
Recognizing objects in a single frame of sensor data representing an environment of the at least one of the robot or the vehicle using the nth trained first model:
Recognizing attributes of the objects over time using the nth trained further model
the unlabeled data set including measured values from at least one sensor, the at least one sensor including at least one of: a radar sensor, an optical camera, an ultrasonic sensor, a lidar sensor or an infrared sensor.
However, Cella teaches:
A method for training neural network models to perform automated guidance of a robot or vehicle Cella states, “Methods and systems are provided herein for data collection in industrial environments, as well as for improved methods and systems for using collected data to provide improved monitoring, control, and intelligent diagnosis of problems and intelligent optimization of operations in various heavy industrial environments. These methods and systems include methods, systems, components, devices, workflows, services, processes, and the like that are deployed in various configurations and locations, such as: (a) at the “edge” of the Internet of Things, such as in the local environment of a heavy industrial machine; (b) in data transport networks that move data between local environments of heavy industrial machines and other environments, such as of other machines or of remote controllers, such as enterprises that own or operate the machines or the facilities in which the machines are operated; and (c) in locations where facilities are deployed to control machines or their environments, such as cloud-computing environments and on-premises computing environments of enterprises that own or control heavy industrial environments or the machines, devices or systems deployed in them. These methods and systems include a range of ways for providing improved data include a range of methods and systems for providing improved data collection, as well as methods and systems for deploying increased intelligence at the edge, in the network, and in the cloud or premises of the controller of an industrial environment” (Cella, paragraph 0009).
Performing automated guidance in the at least one of: the robot, or the vehicle; the performing automated guidance including: Cella states, “The platform 100 may use pattern recognition in face detection related applications such as security systems, tracking, sports related applications, fingerprint analysis, medical and forensic applications, navigation and guidance systems, vehicle tracking, public infrastructure systems such as transport systems, license plate monitoring, and the like.” (Cella, paragraph 317).
Recognizing objects in a single frame of sensor data representing an environment of the at least one of the robot or the vehicle using the nth trained first model: Cella states, “For instance, most drones 11730 are equipped with some sort of visual sensor, either in visual light or infrared range, as well as certain forms of active guidance sensor technology such as light-pulse distance sensing, sonar-pulse sensing. In addition, drones 11730 can be equipped with additional sensors such as specific chemical sensors and magnetic sensors designed to analyze the materials of specific apparatus and machinery.” and “the algorithms used in supervised and unsupervised learning methods of pattern recognition enable the use of pattern recognition in various high precision applications. The platform 100 may use pattern recognition in face detection related applications such as security systems, tracking, sports related applications, fingerprint analysis, medical and forensic applications, navigation and guidance systems, vehicle tracking, public infrastructure systems such as transport systems, license plate monitoring, and the like” (Cella, paragraphs 0317 and 1240).
Recognizing attributes of the objects over time using the nth trained further model Cella states, “FIG. 156 shows an airborne drone 11730 data acquisition box with onboard sensors 11732 and four motors 11734 to provide lift and movement control. In embodiments, the drone 11730 has a charging dock capability and in embodiments, a battery changing capability so that the same drone 11730 can return to inspection after a brief return to base for battery replacement. The drone 11730 can travel from a location near the systems to be sensed. The drone 11730 can detect the presence of other sensor drone and avoid collisions based on both active sensors and network-coordinated flight plans. These sensor drones 11730 inspect and sense environmental and apparatus conditions based on scheduled tours of sensor reconnaissance” and “In embodiments, machine learning can vary and select landing and engagement modes by variation and selection, including testing security of various forms of attachment. Machine learning can be, or be initiated using, a set of rules for landing and engagement, a set of models (which may be populated with information about machines, infrastructure elements and other features of an industrial environment), a training set (including one created by having human operators land a set of drones and engage with sensors), or by deep learning approach fusing various vision and other sensors through a large set of trial landing and engagement events” (Cella, paragraphs 1240 and 1242).
the unlabeled data set including measured values from at least one sensor, the at least one sensor including at least one of: a radar sensor, an optical camera, an ultrasonic sensor, a lidar sensor or an infrared sensor. Cella states “In embodiments, the platform 100 may include the local data collection system 102 deployed in the environment 104 to monitor multiple signals that may be carried by a plurality of physical, electronic, and symbolic formats or signals. The platform 100 may employ signal processing including a plurality of mathematical, statistical, computational, heuristic, and linguistic representations and processing of signals and a plurality of operations needed for extraction of useful information from signal processing operations such as techniques for representation, modeling, analysis, synthesis, sensing, acquisition, and extraction of information from signals.”(Cella, paragraph 317, see also Figure 2, e.g., Camera and Sensor).
To conclude this analysis of claim 22, Chari and Cella are analogous art because they are both in the same field of endeavor of labelling unlabeled data and controlling machinery based on trained models. Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Chari and Cella before them, to include Cella’s teachings of using the labeled data and trained models to control machinery such as robots and/or vehicles. One would have been motivated to make such an inclusion to avoid detected objects and to mitigate operating failures (e.g., overrun of machines beyond their defined capacities). (Cella paragraph 316: “In embodiments, the platform 100 may include the local data collection system 102 deployed in the environment 104 to monitor signals from industrial machinery including failures that may be caused by premature bearing failure that may occur due to contamination or loss of bearing lubricant. In another example, a mechanical defect such as misalignment of bearings may occur. Many factors may contribute to the failure such as metal fatigue, therefore, the local data collection system 102 may monitor cycles and local stresses. By way of this example, the platform 100 may monitor the incorrect operation of machine parts, lack of maintenance and servicing of parts, corrosion of vital machine parts, such as couplings or gearboxes, misalignment of machine parts, and the like. Though the fault occurrences cannot be completely stopped, many industrial breakdowns may be mitigated to reduce operational and financial losses. The platform 100 provides real-time monitoring and predictive maintenance in many industrial environments wherein it has been shown to present a cost-savings over regularly-scheduled maintenance processes that replace parts according to a rigid expiration of time and not actual load and wear and tear on the element or machine. To that end, the platform 10 may provide reminders of, or perform some, preventive measures such as adhering to operating manual and mode instructions for machines, proper lubrication, and maintenance of machine parts, minimizing or eliminating overrun of machines beyond their defined capacities, replacement of worn but still functional parts as needed, properly training the personnel for machine use, and the like”).
Regarding claim 38, Cella teaches:
A device configured to train neural network models to perform automated guidance of a robot or vehicle. Cella states, “Methods and systems are provided herein for data collection in industrial environments, as well as for improved methods and systems for using collected data to provide improved monitoring, control, and intelligent diagnosis of problems and intelligent optimization of operations in various heavy industrial environments. These methods and systems include methods, systems, components, devices, workflows, services, processes, and the like that are deployed in various configurations and locations, such as: (a) at the “edge” of the Internet of Things, such as in the local environment of a heavy industrial machine; (b) in data transport networks that move data between local environments of heavy industrial machines and other environments, such as of other machines or of remote controllers, such as enterprises that own or operate the machines or the facilities in which the machines are operated; and (c) in locations where facilities are deployed to control machines or their environments, such as cloud-computing environments and on-premises computing environments of enterprises that own or control heavy industrial environments or the machines, devices or systems deployed in them. These methods and systems include a range of ways for providing improved data include a range of methods and systems for providing improved data collection, as well as methods and systems for deploying increased intelligence at the edge, in the network, and in the cloud or premises of the controller of an industrial environment” (Cella, paragraph 0009).
Regarding the remaining limitations of claim 38, although varying in scope, the remaining limitations of claim 38 are substantially the same as the limitations of claim 22, respectively. Thus, claim 38 is rejected using the same reasoning and analysis as claim 22 above, including the motivation to combine Chari and Cella.
Regarding claim 39, Chari in view of Cella teaches all the elements of claim 38 as outlined above. Chari also teaches:
A computing device configured to execute computer program instructions to perform the providing, generating, implementing and performing steps. Chari states, “Apparatus 1400 comprises a computer system 1410 and removable media 1450. Computer system 1410 comprises a processor device 1420, a network interface 1425, a memory 1430, a media interface 1435 and an optional display 1440” (paragraph 75). Examiner notes any general purpose computer is the broadest reasonable interpretation of this limitation.
A storage device configured to store the neural network and the another neural network. Chari states, “The memory 1430 could be implemented as an electrical, magnetic or optical memory, or any combination of these or other types of storage devices. Moreover, the term ‘memory’ should be construed broadly enough to encompass any information able to be read from, or written to, an address in the addressable space accessed by processor device 1420” (paragraph 78). Examiner notes that the broadest reasonable interpretation of this limitation is any general purpose computer memory, and all the options listed are capable of performing these functions.
Regarding claim 41, Cella teaches:
A non-transitory computer-readable storage medium on which is stored a computer program for training neural network models to perform automated guidance of a robot or vehicle. Cella states, “In embodiments, one or more non-transitory computer-readable media comprising computer executable instructions that, when executed, may cause at least one processor to perform actions comprising: providing a data collector communicatively coupled to a plurality of input channels; providing a data storage structured to store a plurality of collector route templates, sensor specifications for sensors that correspond to the input channels, wherein the plurality of collector route templates each comprise a different sensor collection routine; providing a data acquisition circuit structured to interpret a plurality of detection values, each of the plurality of detection values corresponding to at least one of the input channels; and providing a machine learning data analysis circuit structured to receive output data from the plurality of input channels and evaluate a current routing template collection routine based on the received output data received over time, wherein the machine learning data analysis circuit learns received output data patterns, wherein the data collector is configured to switch from the current routing template collection routine to an alternative routing template collection routine based on the learned received output data patterns. In embodiments, the instructions may be deployed locally on the data collector, such as deployed in part locally on the data collector and in part on a remote information technology infrastructure component apart from the collector, where each of the input channels correspond to a sensor located in the environment” (Cella, paragraph 0966).
Regarding the remaining limitations of claim 41, although varying in scope, the remaining limitations of claim 41 are substantially the same as the limitations of claim 22, respectively. Thus, claim 41 is rejected using the same reasoning and analysis as claim 22 above, including the motivation to combine Chari and Cella.
Regarding claim 43, Chari in view of Cella teaches all the elements of claim 22 as outlined above. Chari also teaches:
Generating training data for training a neural network. Chari states, “Given the above-described problems associated with the conventional approaches to creating training data sets for predictive modeling, the present techniques address the problem of selecting a good representative subset which is independent of both the original data distribution as well as the classifier that will be trained using the labeled data. Namely, presented herein are new strategies to generate training samples from unlabeled data which overcomes limitations in random and existing active sampling” (paragraph 23).
Regarding claim 45, Chari in view of Cella teaches all the elements of claim 22 as outlined above. Cella also teaches:
Wherein the recognizing objects over time using the nth further trained model is based on recognition output of the nth trained first model. Cella states, “In embodiments, the analytic system 4018 may apply to any of a wide range of analytic techniques, including statistical and econometric techniques (such as linear regression analysis, use similarity matrices, heat map based techniques, and the like), reasoning techniques (such as Bayesian reasoning, rule-based reasoning, inductive reasoning, and the like), iterative techniques (such as feedback, recursion, feed-forward and other techniques), signal processing techniques (such as Fourier and other transforms), pattern recognition techniques (such as Kalman and other filtering techniques), search techniques, probabilistic techniques (such as random walks, random forest algorithms, and the like), simulation techniques (such as random walks, random forest algorithms, linear optimization and the like), and others. This may include computation of various statistics or measures. In embodiments, the analytic system 4018 may be disposed, at least in part, on a data collection system 102, such that a local analytic system can calculate one or more measures, such as measures relating to any of the items noted throughout this disclosure. For example, measures of efficiency, power utilization, storage utilization, redundancy, entropy, and other factors may be calculated onboard, so that the data collection 102 can enable various cognitive and learning functions noted throughout this disclosure without dependence on a remote (e.g., cloud-based) analytic system” and “The machine learning facility may start with an initial configuration and vary parameters of the swarm relevant to any of the foregoing (also including varying the membership of the swarm), such as iterating based on feedback to the machine learning facility regarding measures of success (such as utilization measures, efficiency measures, measures of success in prediction or anticipation of states, productivity measures, yield measures, profit measures, and others). Over time, the swarm may be optimized to a favorable configuration to achieve the desired measure of success for an owner, operator, or host of an industrial environment or a machine, component, or process thereof” and “This method may, optionally, include processing the subset of data with at least one set of algorithms, models and pattern recognizers that corresponds to algorithms, models and pattern recognizers associated with processing the data captured with predefined lines of resolution covering a predefined frequency range” (Cella, paragraphs 0328, 0338, and 0033).
The same motivation used to combine Chari with Cella in claim 22 is equally applicable to claim 45.
Claims 24–29 are rejected under 35 U.S.C. 103 as Chari in view of Cella, and further in view of Zhang et al., (US Pub No: 20190228268 A1), hereinafter “Zhang”.
Regarding claim 24, Chari in view of Cella teaches all the elements of claim 22 as outlined above. Cella also teaches:
performing the automated guidance, based on the recognized objects, of the at least one of the robot or the vehicle; Cella states, “The platform 100 may use pattern recognition in face detection related applications such as security systems, tracking, sports related applications, fingerprint analysis, medical and forensic applications, navigation and guidance systems, vehicle tracking, public infrastructure systems such as transport systems, license plate monitoring, and the like.” (Cella, paragraph 317).
training an object recognition algorithm using the final labeled data set, the object recognition algorithm for use in performing automated guidance, based on the recognized object, of at least one of: a robot, or a vehicle. Cella states, “In embodiments, the platform 100 may include the local data collection system 102 deployed in the environment 104 to monitor signals from industrial machinery including failures that may be caused by premature bearing failure that may occur due to contamination or loss of bearing lubricant. In another example, a mechanical defect such as misalignment of bearings may occur. Many factors may contribute to the failure such as metal fatigue, therefore, the local data collection system 102 may monitor cycles and local stresses. By way of this example, the platform 100 may monitor the incorrect operation of machine parts, lack of maintenance and servicing of parts, corrosion of vital machine parts, such as couplings or gearboxes, misalignment of machine parts, and the like. Though the fault occurrences cannot be completely stopped, many industrial breakdowns may be mitigated to reduce operational and financial losses. The platform 100 provides real-time monitoring and predictive maintenance in many industrial environments wherein it has been shown to present a cost-savings over regularly-scheduled maintenance processes that replace parts according to a rigid expiration of time and not actual load and wear and tear on the element or machine. To that end, the platform 10 may provide reminders of, or perform some, preventive measures such as adhering to operating manual and mode instructions for machines, proper lubrication, and maintenance of machine parts, minimizing or eliminating overrun of machines beyond their defined capacities, replacement of worn but still functional parts as needed, properly training the personnel for machine use, and the like. In embodiments, the platform 100 may include the local data collection system 102 deployed in the environment 104 to monitor multiple signals that may be carried by a plurality of physical, electronic, and symbolic formats or signals. The platform 100 may employ signal processing including a plurality of mathematical, statistical, computational, heuristic, and linguistic representations and processing of signals and a plurality of operations needed for extraction of useful information from signal processing operations such as techniques for representation, modeling, analysis, synthesis, sensing, acquisition, and extraction of information from signals. In examples, signal processing may be performed using a plurality of techniques, including but not limited to transformations, spectral estimations, statistical operations, probabilistic and stochastic operations, numerical theory analysis, data mining, and the like. The processing of various types of signals forms the basis of many electrical or computational process. As a result, signal processing applies to almost all disciplines and applications in the industrial environment such as audio and video processing, image processing, wireless communications, process control, industrial automation, financial systems, feature extraction, quality improvements such as noise reduction, image enhancement, and the like. Signal processing for images may include pattern recognition for manufacturing inspections, quality inspection, and automated operational inspection and maintenance. The platform 100 may employ many pattern recognition techniques including those that may classify input data into classes based on key features with the objective of recognizing patterns or regularities in data. The platform 100 may also implement pattern recognition processes with machine learning operations and may be used in applications such as computer vision, speech and text processing, radar processing, handwriting recognition, CAD systems, and the like. The platform 100 may employ supervised classification and unsupervised classification. The supervised learning classification algorithms may be based to create classifiers for image or pattern recognition, based on training data obtained from different object classes. The unsupervised learning classification algorithms may operate by finding hidden structures in unlabeled data using advanced analysis techniques such as segmentation and clustering. For example, some of the analysis techniques used in unsupervised learning may include K-means clustering, Gaussian mixture models, Hidden Markov models, and the like. The algorithms used in supervised and unsupervised learning methods of pattern recognition enable the use of pattern recognition in various high precision applications. The platform 100 may use pattern recognition in face detection related applications such as security systems, tracking, sports related applications, fingerprint analysis, medical and forensic applications, navigation and guidance systems, vehicle tracking, public infrastructure systems such as transport systems, license plate monitoring, and the like.” (paragraphs 316–317, emphasis added).
using the trained object recognition algorithm to recognize objects in an environment of the at least one of: the robot, or the vehicle; Cella states “This may occur by triggering a rule that reflects a model or understanding of system behavior (e.g., recognizing a shift in operating mode that calls for different sensors as velocity of a shaft increases) or it may occur under control of a neural net (either in combination with a rule-based approach or on its own), where inputs are provided such that the neural net over time learns to select appropriate collection modes based on feedback as to successful outcomes (e.g., successful classification of the state of a system, successful prediction, successful operation relative to a metric). For example only, when an assembly line is reconfigured for a new product or a new assembly line is installed in a manufacturing facility, data from the current data collector(s) may not accurately predict the state or metric of operation of the system, thus, the self-organization functionality may begin to iterate to determine if a new data collector, type of sensed data, format of sensed data, etc. is better at predicting a state or metric. Based on offset system data, such as from a library or other data structure, certain sensors, frequency bands or other data collectors may be used in the system initially and data may be collected to assess performance. As the self-organization functionality iterates, other sensors/frequency bands may be accessed to determine their relative weight in identifying performance metrics. Over time, a new frequency band may be identified (or a new collection of sensors, a new set of configurations for sensors, or the like) as a better or more suitable gauge of performance in the system and the self-organization functionality may modify its data collector(s) based on this iteration. For example only, perhaps an older boring tool in an energy extraction environment dampens one or more vibration frequencies while a different frequency is of higher amplitude and present during optimal performance than what was seen in the present system. In this example, the self-organization functionality may alter the data collectors from what was originally proposed, e.g., by the data collection system, to capture the higher amplitude frequency that is present in the current system” (Cella, paragraph 1392, emphasis added).
The same motivation used to combine Chari with Cella used in claim 22 is equally applicable to claim 24.
Chari in view of Cella does not appear to explicitly teach:
Following the Nth iteration of the iterative process, training a final model using the Nth labeled first subset and/or the Nth labeled further subset, the final model including a final neural network;
generating a final labeled data set by predicting labels for the unlabeled data set using the final model
However, Zhang teaches:
Following the Nth iteration of the iterative process, training a final model using the Nth labeled first subset and/or the Nth labeled further subset the final model including a final neural network. Zhang states “FIG. 1 schematically illustrates the architecture of a two-stage CNN system according to embodiments of the present invention, including a first stage convolutional neural network 2 (“CNN-1”) and a second stage convolutional neural network 6 (“CNN-2”). For convenience, in this two-stage system, the first stage is referred to as the “coarse learning” stage and the second stage is referred to as the “fine tuning learning” stage.” (Zhang, paragraph 20). Zhang also states “In the fine-tuning learning stage, the second stage network CNN-2 receives the class score images 5 as input images to the network, as well as label data 7. The label data 7 is the same as the label data 3 used in the coarse learning stage, i.e., the label data for the original input images 1. Supervised learning is conducted using the input class score images 5 and corresponding label data 7 to learn the weights W2 8 of the second stage network CNN-2” (Zhang, paragraph 23). Zhang teaches that the CNN is iteratively trained and that it starts with coarse training (e.g., CNN-1) and a fine tuning training (i.e., training a final model using the Nth labeled data; e.g., CNN-2 i.e., a final neural network). Furthermore, examiner notes that Chari’s core methodologies include iterative machine learning using neural networks, and thus each model includes at least one neural network (see Chari paragraphs 0024, 0068, and 0072.
generating a final labeled data set by predicting labels for the data set using the final model. Zhang states, “For a system including three or more stages, the output image of the first, second, etc. stage CNNs may be referred to as ‘first stage class score image’, ‘second stage class score image’, etc. representing first stage preliminary probabilities, second stage preliminary probabilities, etc. of the image classification, and the output of the final stage may be referred to as the ‘final stage class score image’ representing the final probabilities of the image classification” (paragraph 30, emphasis added). Zhang teaches that its methods are iterative and produce a final stage class score image (i.e., generate a final labeled data set) which represents the final probability (i.e., predicted label) using the finely tuned model (i.e., the final model).
To conclude this analysis of this combination, Chari and Zhang are analogous art because they are both in the same field of endeavor of labelling unlabeled data. Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Chari and Zhang before them, to include Zhang’s training and use of models to provide labels in Chari’s method for generating labeled data. One would have been motivated to make such an inclusion to learn weights of the network, produce network parameters, and minimize loss from training using multiple iterations while labeling data (Zhang, paragraph 21: “The label data corresponding to each training image is a map having the same height and width as the training image where each pixel has a pixel value representing the desired classification result for the corresponding pixel of the training image. Supervised learning is conducted using the training images 1 and corresponding label data 3 to learn the weights W1 4 of the first stage network CNN-1. Generally speaking, a supervised learning algorithm processes labeled training data and produces network parameters that minimize a loss function on the training data through multiple iterations. Any suitable training algorithm may be used to train the first stage CNN”; see also Zhang paragraph 23: “In the fine-tuning learning stage, the second stage network CNN-2 receives the class score images 5 as input images to the network, as well as label data 7. The label data 7 is the same as the label data 3 used in the coarse learning stage, i.e., the label data for the original input images 1. Supervised learning is conducted using the input class score images 5 and corresponding label data 7 to learn the weights W2 8 of the second stage network CNN-2”).
Regarding claim 25, Chari in view of Cella teaches all the elements of claim 22 as outlined above. Chari also teaches:
The generation of the labeled first subset of data occurs by predicting labels. Chari states, “In each iteration, semi-supervised clustering is applied to the data, incorporating the labeled samples from previous iterations. See step 106. As is known in the art, semi-supervised clustering employs both labeled (e.g., known labels from the previous iterations) and unlabeled data for training. Specifically, in step 106, the data from Data Set U is clustered using a semi-supervised clustering process. The result of the semi-supervised clustering is a plurality of clusters C1, C2, . . . , Ckcluster (see FIG. 1) which should have a biased class distribution.” (paragraph 31, see also Figure 1). Chari’s method includes creating clusters C1, C2, …. Ckcluster by predicting labels for the unlabeled data using a semi-supervised process.
Regarding claim 26, Chari in view of Zhang and Cella teaches all the elements of claim 25. Chari also teaches:
The initial model is trained in a preceding step using a labeled initial subset of data. Chari states “A simplistic approach is to assume that the class distribution of the cluster is exactly the same as the class distribution of the samples labeled in this cluster. This is based on the optimistic assumption that the semi-supervised clustering works perfectly and groups together elements which are similar to the labeled sample. First, determine how many samples one ideally wishes to draw from each class in this iteration from the total B samples to draw. Let li j be the number of instances of class j sampled after iteration i, and ρi j be the normalized proportion of samples with class label j, i.e., ρij=lij∑rlir. To increase the balancedness in the training step, one wants to select samples inversely proportional to their current distribution (see Liu, Chawla and Wu), i.e., nj=1-ρijl-1*B, where l is the number of classes and (l−1) is the normalization factor.” (paragraph 45, see also Figures 1, 3, 4, and 5). Chari also states “Next, the estimated class distribution in each cluster is used to select the appropriate number of samples from each class. Let θi j be the probability of drawing a sample with class label j from the previously labeled subset of cluster i. By assumption, this is exactly the probability of drawing a sample with class label j from the entire cluster i. Since it is desired to have nj samples with label j in this iteration” (paragraph 46). Chari’s method uses the clustered dataset (i.e., the nth labeled first subset; example C1) to train the ith iteration (i.e., the initial model; e.g., i+1) of the model. Examiner notes that iterative processes naturally includes preceding steps when n equals n+1. Additionally, the BRI of initial model is any model used as a starting point when using more than one model or iterations of models.
The labeled initial subset of data being disjunctive with respect to the first subset of unlabeled data and the further subset of unlabeled data. Chari states “A simplistic approach is to assume that the class distribution of the cluster is exactly the same as the class distribution of the samples labeled in this cluster. This is based on the optimistic assumption that the semi-supervised clustering works perfectly and groups together elements which are similar to the labeled sample. First, determine how many samples one ideally wishes to draw from each class in this iteration from the total B samples to draw. Let li j be the number of instances of class j sampled after iteration i, and ρi j be the normalized proportion of samples with class label j, i.e., ρij=lij∑rlir. To increase the balancedness in the training step, one wants to select samples inversely proportional to their current distribution (see Liu, Chawla and Wu), i.e., nj=1-ρijl-1*B, where l is the number of classes and (l−1) is the normalization factor.” (paragraph 45, see also Figures 3, 4, and 5). Chari’s method selects subsets of samples inversely proportional to their current distribution (i.e., disjunctively).
Regarding claim 27, Chari in view of Zhang, and Cella teaches all the elements of claim 26. Chari also teaches:
The labeled initial subset of data is smaller than the first subset and/or smaller than the further subset. Chari states, “An iterative method is applied herein, where, in each iteration, the present method draws (selects) a batch of samples (B), and domain experts provide the labels of the selected samples. Information embedded in the labeled sample is used to group together data elements which are very similar to the labeled sample using semi-supervised clustering. The class distribution in the clusters can then be estimated and used to perform a biased sampling of clusters to obtain a diverse balanced sample. Within each cluster, a diverse sample is obtained by using a maximum entropy sampling. The sample obtained at each iteration is then labeled and used in subsequent iterations" (paragraph 28, see also Figure 1). Chari’s methods are iterative and build on each other from one iteration to the next. This process necessitates that each iteration is larger than the previous iteration (i.e., the initial subset is smaller).
Regarding claim 28, Chari in view of Cella teaches all the elements of claim 22 as outlined above. Chari also teaches:
Steps of the iterative process are carried out repeatedly for as long as a quality criterion and/or a termination criterion is not yet fulfilled. Chari states, “The samples chosen from the clusters are then labeled and added to the training data set, and as highlighted above methodology 100 can be repeated until a desired amount of training data is obtained” (paragraph 33).
Regarding claim 29, Chari in view of Zhang and Cella teaches all the elements of claim 25 as outlined above. Zhang also teaches:
The first model, the further model, the initial model, and/or the final model includes a deep neural network. Zhang states, “In the training process, the first stage convolutional neural network 2 (‘CNN-1’) receives training image data 1 as input” (Zhang, paragraph 21). Note that a convolutional neural network aka CNN is a type of deep neural network.
The same motivation used to combine Chari with Zhang in claim 24 is equally applicable to claim 29.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS SHINE whose telephone number is (571)272-2512. The examiner can normally be reached M-F, 9a-5p ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached on (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.B.S./Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126