DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-13 remain pending. Claims 1, 3, 7-8, and 10 have been amended.
Claim Objections
Claims 1-7 are objected to because of the following informalities: Claim 1 states “said reconstructed data setelement” in line 12. Applicant should remove the term “element” so that the claim reads “said reconstructed data set”. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3 are rejected under 35 U.S.C. 103 as being unpatentable over Park et al “A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencoder” in view of AlRegib et all (US 20220327389 A1) (Hereinafter referred to as Park and AlRegib respectively)
Regarding Claim 1, Park teaches a system for detecting off-nominal behavior (See at least Park Page 1 Abstract, the anomalous execution is interpreted as off-nominal behavior) comprising:
a trained manifold developed with a target behavior dataset (See at least Park Page 4 C. Training and Testing Framework and Page 3 Figure 2, the trained LSTM-VAE-based anomaly detector is interpreted as the trained manifold, which is developed with a sequence of non-anomalous/target behavior dataset) , wherein said target behavior data set corresponds to one or more tasks to be completed by a robotic system over a time interval (See at least Park Pages 1-2 Introduction, Page 4 C. Training and Testing Framework, and Page 3 Figure 2, the non-anomalous/target behavior dataset corresponds to a task to be completed by a robot in sequences over a time interval);
an encoder configured to receive at least one initial data set (See at least Park Page 3 C. An LSTM-based Variational Autoencoder and Figure 2, and Pages 6-7 Evaluation and Figure 6, the encoder receives an initial multimodal input at different times), said encoder being further configured to generate a lower dimensional data set from said at least one initial data set (See at least Park Page 1 Introduction, during encoding, the multimodal observations/initial data set is projected into a latent space, which is a lower dimensional data set), wherein said at least one initial data set corresponds to a performance of a first task of said one or more tasks over said time interval (See at least Park Page 1 Introduction, and Pages 5-7, V Experimental Setup and VI Evaluation, the initial data set/multimodal observation corresponds to a robot performing a task over a time interval);
a generator configured to generate a reconstructed data set by mapping said lower dimensional data set onto said trained manifold (See at least Park Page 1 Introduction, Page 2 A. Preliminary Autoencoder, and Page 3 Figure 2, the decoder/generator generates a reconstructed data set using the lower dimensional/compressed data set); and
…measure a divergence between said initial data set and said reconstructed data setelement along said time interval as said at least one initial data set is received, wherein a divergence beyond a pre-determined threshold is indicative of off-nominal behavior (See at least Park Page 4 IV Anomaly Detection and Pages 6-7 VI Evaluation and Figure 6, the deviation between the initial/observed data set and the reconstructed/predicted data set beyond a threshold value indicates anomalous execution/off-nominal behavior).
Park fails to explicitly disclose a classifier configured to measure a divergence between said initial data set and said reconstructed data set.
However, AlRegib teaches a classifier configured to measure a divergence between said initial data set and said reconstructed data set (See at least AlRegib Paragraphs 0005, 0007, 0009, 0030, and 0032 the discriminator, which is a type of classifier, measures the reconstruction error/divergence between the reconstructed data and the initial data).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in Park with AlRegib to utilize a classifier to measure a divergence between said initial data set and said reconstructed data set. Classifiers are utilized as an anomaly detector by comparing the reconstructed data with original data (See at least AlRegib Paragraph 0009). By using a classifier to measure the divergence, as taught by AlRegib, the system can detect an abnormality by measuring the divergence/reconstruction error between the initial data and the reconstructed data (See at least AlRegib Paragraph 0032), which would improve the autonomy of the system.
Regarding Claim 2, modified Park teaches said target behavior dataset includes only nominal behavior data (See at least Park Page 4 C. Training and Testing Framework, the target behavior dataset includes non-anomalous training data, which is interpreted as nominal behavior data).
Regarding Claim 3, modified Park teaches said at least one initial data set includes a plurality of measured state of said robotic system over said time interval (See at least Park Pages 6-7, VI Evaluation and Figure 6, the current observations are a plurality of measured states of the robot over said time interval) and said nominal behavior data corresponds to a set of target states for said robotic system over said time interval (See at least Park Page 4 C. Training and Testing Framework, the nominal behavior data/non-anomalous training corresponds to target states over said time interval).
Claims 4 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Park in view of AlRegib, and in further view of Kranski et al (US 20210197335 A1) (Hereinafter referred to as Kranski)
Regarding Claim 4, modified Park fails to disclose said nominal behavior data comprises simulated behavior data.
However, Kranski teaches said nominal behavior data comprises simulated behavior data (See at least Kranski Paragraphs 0022 and 0061, the nominal behavior data to complete a task is simulated).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Park with Kranski to have the nominal behavior data comprise simulated behavior data. This modification, as taught by Kranski, would allow the system to train the robot to perform the task through simulated attempts rather than actually controlling a real robot (See at least Kranski Paragraphs 0022 and 0061), which would improve the efficiency and cost of the training.
Regarding Claim 7, modified Park teaches a robotics control platform coupled to said robotic system (See at least Park Page 5 C. Experimental Procedure, the user interface is interpreted as the robotics control platform), wherein…cause said robotic system to terminate an operation upon…determines divergence of said initial data set and said reconstructed data set beyond said pre-determined threshold (See at least Park Page 1 Introduction, Page 4 IV Anomaly Detection and Pages 6-7 VI Evaluation and Figure 6, the robot is stopped during anomalous task execution, which is determined based on the deviation between the initial/observed data set and the reconstructed/predicted data set beyond a threshold value).
Modified Park fails to explicitly disclose said classifier determines a divergence of said initial data set and said reconstructed data set.
However, AlRegib teaches said classifier determines a divergence of said initial data set and said reconstructed data set (See at least AlRegib Paragraphs 0005, 0007, 0009, 0030, and 0032 the discriminator, which is a type of classifier, measures the reconstruction error/divergence between the reconstructed data and the initial data).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Park with AlRegib to utilize a classifier to determine a divergence between said initial data set and said reconstructed data set. Classifiers are utilized as an anomaly detector by comparing the reconstructed data with original data (See at least AlRegib Paragraph 0009). By using a classifier to measure the divergence, as taught by AlRegib, the system can detect an abnormality by measuring the divergence/reconstruction error between the initial data and the reconstructed data (See at least AlRegib Paragraph 0032), which would improve the autonomy of the system.
Modified Park fails to explicitly disclose wherein said robotics control platform is configured to cause said robotic system to terminate an operation.
However, Kranski teaches wherein said robotics control platform is configured to cause said robotic system to terminate an operation (See at least Kranski Paragraph 0061, the control platform/teaching subsystem stops/terminates an operation of the robot).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Park with Kranski to have the robotics control platform cause said robotic system to terminate an operation. This modification, as taught by Kranski, would allow an operator to stop the operation of the robot when the robot failed at completing the task, which would prevent damage to the robot (See at least Kranski Paragraph 0061).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Park in view of AlRegib, and in further view of Wang et al (US 20210197335 A1) (Hereinafter referred to as Wang)
Regarding Claim 5, modified Park fails to disclose said classifier is a fully-connected classifier.
However, Wang teaches said classifier is a fully-connected classifier (See at least Wang Paragraph 0013, the discriminator/classifier is a fully-connected neural network).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Park with Wang to have the classifier be a fully-connected classifier. Fully connected classifiers are conventional and well-understood in the art, and therefore, one of ordinary skill in the art would be motivated to utilize a fully connected classifier as the classifier.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Park in view of AlRegib, and in further view of Rozantsev et al (US 20220230376 A1) (Hereinafter referred to as Rozantsev)
Regarding Claim 6, modified Park fails to disclose said classifier is a long short-term memory classifier.
However, Rozantsev teaches said classifier is a long short-term memory classifier (See at least Rozantsev Paragraph 0040, the discriminator/classifier utilizes a LSTM unit).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Kranski with Rozantsev to have the classifier be a long short-term memory classifier. This modification, as taught by Rozantsev, would allow the system to efficiently gather information from the predictions of a prediction network for multiple consecutive time steps, and predict whether the sequences of input data is generated or coming from ground-truth data (See at least Rozantsev Paragraph 0040), thus, improving efficiency.
Claims 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over Park in view of AlRegib, Mantegazza et al “Sensing Anomalies as Potential Hazards: Datasets and Benchmarks”, and Hanspal et al (US 20240338561 A1) (Hereinafter referred to as Mantegazza and Hanspal respectively)
Regarding Claim 8, Park teaches a system for detecting off-nominal behavior (See at least Park Page 1 Abstract, the anomalous execution is interpreted as off-nominal behavior) comprising:
a robotic control platform configured to observe and control operations of a robotic system (See at least Park Page 1 Introduction, and Pages 5-6 V Experimental Setup, the long short-term memory-based variational autoencoder anomaly detector and user interface are interpreted as a robotic control platform, which observes the robot using the multimodal inputs and controls operations via the interface), said robotic control platform comprising:
a user interface configured to allow a user to remotely direct operations of said robotic system (See at least Park Page 5 C. Experimental Procedure, the user directs operations of the robot using the user interface);
a trained manifold developed with a target behavior dataset (See at least Park Page 4 C. Training and Testing Framework and Page 3 Figure 2, the trained LSTM-VAE-based anomaly detector is interpreted as the trained manifold, which is developed with a sequence of non-anomalous/target behavior dataset) , wherein said target behavior data set corresponds to one or more tasks to be completed by a robotic system over a time interval (See at least Park Pages 1-2 Introduction, Page 4 C. Training and Testing Framework, and Page 3 Figure 2, the non-anomalous/target behavior dataset corresponds to a task to be completed by a robot in sequences over a time interval);
an encoder configured to receive at least one initial data set (See at least Park Page 3 C. An LSTM-based Variational Autoencoder and Figure 2, and Pages 6-7 Evaluation and Figure 6, the encoder receives an initial multimodal input at different times), wherein said at least one initial data set corresponds to a performance of a first task of said one or more tasks over said time interval (See at least Park Page 1 Introduction, and Pages 5-7, V Experimental Setup and VI Evaluation, the initial data set/multimodal observation corresponds to a robot performing a task over a time interval), …said encoder being further configured to generate a lower dimensional data set from said at least one initial data set (See at least Park Page 1 Introduction, during encoding, the multimodal observations/initial data set is projected into a latent space, which is a lower dimensional data set);
a generator configured to generate a reconstructed data set by mapping said lower dimensional data set onto said trained manifold (See at least Park Page 1 Introduction, Page 2 A. Preliminary Autoencoder, and Page 3 Figure 2, the decoder/generator generates a reconstructed data set using the lower dimensional/compressed data set); and
…measure a divergence between said initial data set and said reconstructed data set along said time interval as said at least one initial data set is received, wherein a divergence beyond a pre-determined threshold is indicative of off-nominal behavior (See at least Park Page 4 IV Anomaly Detection and Pages 6-7 VI Evaluation and Figure 6, the deviation between the initial/observed data set and the reconstructed/predicted data set beyond a threshold value indicates anomalous execution/off-nominal behavior).
Park fails to explicitly disclose a classifier configured to measure a divergence between said initial data set and said reconstructed data set.
However, AlRegib teaches a classifier configured to measure a divergence between said initial data set and said reconstructed data set (See at least AlRegib Paragraphs 0005, 0007, 0009, 0030, and 0032 the discriminator, which is a type of classifier, measures the reconstruction error/divergence between the reconstructed data and the initial data).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in Park with AlRegib to utilize a classifier to measure a divergence between said initial data set and said reconstructed data set. Classifiers are utilized as an anomaly detector by comparing the reconstructed data with original data (See at least AlRegib Paragraph 0009). By using a classifier to measure the divergence, as taught by AlRegib, the system can detect an abnormality by measuring the divergence/reconstruction error between the initial data and the reconstructed data (See at least AlRegib Paragraph 0032), which would improve the autonomy of the system.
Modified Park fails to disclose said user interface being further configured to notify said user upon said classifier detecting said off-nominal behavior.
However, Mantegazza teaches said user interface being further configured to notify said user upon detecting said off-nominal behavior (See at least Mantegazza section 1, the anomaly/off-nominal behavior is reported to the user/operator).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Park with Mantegazza to notify a user upon the classifier detecting off-nominal behavior. By notifying the user of the off-nominal behavior, the user can analyze the reported off-nominal behavior and generate instructions accordingly (See at least Mantegazza section 1), which would also improve the safety and efficiency of the system.
Modified Park fails to disclose said user interface is further configured to allow said user to cause said at least one initial data set to be communicated to said encoder.
However, Hanspal teaches said user interface is further configured to allow said user to cause said at least one initial data set to be communicated to said encoder (See at least Hanspal Paragraphs 0064-0065, and 0084, the initial/input data set is communicated to the encoder according to the user input).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Park with Hanspal to allow said user to cause said at least one initial data set to be communicated to said encoder. This modification, as taught by Hanspal, would allow the user to train the encoder according to initial data set inputted by the user (See at least Hanspal Paragraphs 0064-0065, and 0084), thus, giving the user greater control over the system.
Regarding Claim 9, modified Park teaches said target behavior dataset includes only nominal behavior data (See at least Park Page 4 C. Training and Testing Framework, the target behavior dataset includes non-anomalous training data, which is interpreted as nominal behavior data).
Regarding Claim 10, modified Park teaches said at least one initial data set includes a plurality of measured state of said robotic system over said time interval (See at least Park Pages 6-7, VI Evaluation and Figure 6, the current observations are a plurality of measured states of the robot over said time interval) and said nominal behavior data corresponds to a set of target states for said robotic system over said time interval (See at least Park Page 4 C. Training and Testing Framework, the nominal behavior data/non-anomalous training corresponds to target states over said time interval).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Park in view of AlRegib Mantegazza, and Hanspal, and in further view of Wang
Regarding Claim 11, modified Park fails to disclose said classifier is a fully-connected classifier.
However, Wang teaches said classifier is a fully-connected classifier (See at least Wang Paragraph 0013, the discriminator/classifier is a fully-connected neural network).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Park with Wang to have the classifier be a fully-connected classifier. Fully connected classifiers are conventional and well-understood in the art, and therefore, one of ordinary skill in the art would be motivated to utilize a fully connected classifier as the classifier.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Park in view of AlRegib, Mantegazza, and Hanspal, and in further view of Rozantsev
Regarding Claim 12, modified Park fails to disclose said classifier is a long short-term memory classifier.
However, Rozantsev teaches said classifier is a long short-term memory classifier (See at least Rozantsev Paragraph 0040, the discriminator/classifier utilizes a LSTM unit).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Park with Rozantsev to have the classifier be a long short-term memory classifier. This modification, as taught by Rozantsev, would allow the system to efficiently gather information from the predictions of a prediction network for multiple consecutive time steps, and predict whether the sequences of input data is generated or coming from ground-truth data (See at least Rozantsev Paragraph 0040), thus, improving efficiency.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Park in view of AlRegib, Mantegazza, and Hanspal, and in further view of Kranski
Regarding Claim 13, modified Park fails to disclose said nominal behavior data comprises simulated behavior data.
However, Kranski teaches said nominal behavior data comprises simulated behavior data (See at least Kranski Paragraphs 0022 and 0061, the nominal behavior data to complete a task is simulated).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Park with Kranski to have the nominal behavior data comprise simulated behavior data. This modification, as taught by Kranski, would allow the system to train the robot to perform the task through simulated attempts rather than actually controlling a real robot (See at least Kranski Paragraphs 0022 and 0061), which would improve the efficiency and cost of the training.
Response to Arguments
Applicant’s arguments with respect to claims 1 and 8 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Applicant has amended the claims to specify that the data sets correspond to a robot performing a task over a time interval. This limitation is taught by Park, which teaches training a manifold using target behavior of a robot over a time interval, an encoder that encodes an initial data set that corresponds to a performance of a first task over said time interval, a generator/decoder that reconstructs the data set, and the divergence between the initial data set and reconstructed data set along said time interval is used to determine off-nominal/anomalous behavior. Therefore, the claims still stand rejected under 103.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ESVINDER SINGH whose telephone number is (571)272-7875. The examiner can normally be reached Monday-Friday: 9 am-5 pm est.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached at 571-270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ESVINDER SINGH/Examiner, Art Unit 3657 /ABBY LIN/Supervisory Patent Examiner, Art Unit 3657