Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged. The present application claims priority to U.S. Provisional application No. 63/398,648 filed on 08/17/2022 and U.S. Provisional application No. 63/414,056 filed on 10/17/2022. Applicant has not complied with one or more conditions for receiving the benefit of an earlier filing date under 35 U.S.C. 119(e) as follows:
The later-filed application must be an application for a patent for an invention which is also disclosed in the prior application (the parent or original nonprovisional application or provisional application). The disclosure of the invention in the parent application and in the later-filed application must be sufficient to comply with the requirements of 35 U.S.C. 112(a) or the first paragraph of pre-AIA 35 U.S.C. 112, except for the best mode requirement. See Transco Products, Inc. v. Performance Contracting, Inc., 38 F.3d 551, 32 USPQ2d 1077 (Fed. Cir. 1994).
The disclosure of the prior-filed application U.S. Provisional application No. 63/398,648, fails to provide adequate support or enablement in the manner provided by 35 U.S.C. 112(a) or pre-AIA 35 U.S.C. 112, first paragraph for one or more claims of this application. Dependent claims 5 and 15 recite, using respective similar language, “wherein skill discovery includes training a bidirectional skill embedding model, the skill matching model, and the low-level policies using the mutual information” and dependent claims 10 and 20 both recite “wherein the low-level policies are implemented as multilayer perceptron neural network models”.
The above recited features of claims 5, 10, 15, and 20 are not defined or described in the specification of U.S. Provisional application No. 63/398,648 in such a way as to reasonably convey to one skilled in the relevant art that the inventor, at the time the application was filed, had possession of the claimed invention.
For instance, the as-filed specification of the 63/398,648 provisional application fails to provide adequate support or enablement for at least the above-noted elements of claims 5, 10, 15, and 20. That is, the as-filed specification of the 63/398,648 provisional application fails to provide adequate support or enablement for at least the “wherein skill discovery includes training a bidirectional skill embedding model, the skill matching model, and the low-level policies using the mutual information” elements of claims 5 and 15, and the “wherein the low-level policies are implemented as multilayer perceptron neural network models” elements of claims 10 and 20.
Further, for example, the original specification of the 63/398,648 provisional application fails to disclose the training or use of a bidirectional skill embedding model as recited in claims 5 and 15 and the low-level policies being implemented as multilayer perceptron neural network models as recited in claims 10 and 20.
Thus, the as-filed specification of the 63/398,648 provisional application fails to provide adequate support or enablement for at least the above-noted elements of dependent claims 5, 10, 15, and 20. However, the disclosure of the prior-filed application, U.S. Provisional application No. 63/414,056 appears to provide adequate support for at least dependent claims 5 and 15.
Therefore, the effective filing date of claims 5 and 15 of the instant application is the filing date of the 63/414,056 provisional application, 10/07/2022. Examiner will consider if the 63/398,648 provisional application supports each of the other claims if a rejection would need to rely upon an intervening reference between the actual filing date of the 63/414,056 provisional application, 10/07/2022, and the 08/17/2022 filing of the 63/398,648 provisional application.
The disclosure of the prior-filed application, U.S. Provisional application No. 63/414,056, fails to provide adequate support or enablement in the manner provided by 35 U.S.C. 112(a) or pre-AIA 35 U.S.C. 112, first paragraph for one or more claims of this application. Dependent claims 10 and 20 recite, using respective similar language, “wherein the low-level policies are implemented as multilayer perceptron neural network models”.
The as-filed specification of the 63/414,056 provisional application fails to provide adequate support or enablement for at least the above-noted elements of claims 10 and 20. That is, the as-filed specification of the 63/414,056 provisional application fails to provide adequate support or enablement for at least the “wherein the low-level policies are implemented as multilayer perceptron neural network models” elements of claims 10 and 20.
For example, the original specification of the 63/414,056 provisional application fails to disclose the low-level policies being implemented as multilayer perceptron neural network models as recited in claims 10 and 20.
Thus, the as-filed specification of the 63/414,056 provisional application fails to provide adequate support or enablement for at least the above-noted elements of dependent claims 10 and 20.
Therefore, the effective filing date of claims 10 and 20 of the instant application is the filing date of the instant, non-provisional application, 08/16/2023. Examiner will consider if the 63/414,056 provisional application supports each of the other claims (aside from claims 5 and 15) if a rejection would need to rely upon an intervening reference between the actual filing date of the instant application, 08/16/2023 and the 10/07/2022 filing of the 63/414,056 provisional application.
Each claim will receive benefit of the earliest filing date above for which a continuous chain of support can be established for the entirety of the claim. As discussed above, the effective filing date for at least claims 5 and 15 of the instant application is the filing date of the 63/414,056 provisional application, 10/07/2022. As also discussed above, the effective filing date for at least claims 10 and 20 of the instant application is the filing date of the instant, non-provisional application, 08/16/2023.
Information Disclosure Statement
Acknowledgment is made of the information disclosure statements filed 08/16/2023 and 3/28/2025, which comply with 37 CFR 1.97. As such, the information disclosure statements have been placed in the application file and the information referred to therein has been considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION. —The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6-9 and 16-19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “sub-optimal” in claims 6, 7, 9, 16, 17, and 19 is a relative term which renders the claims indefinite. The term “sub-optimal” is not defined by the claims, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The specification does not establish a definition of what qualifies as sub-optimal when it comes to the outcomes and skills of noisy demonstrations recited in these claims. The specification fails to describe or define what is meant by the term “sub-optimal”. In particular, it is unclear what metrics or standards are used for ascertaining the requisite degree of optimality (or lack thereof) for the claimed “sub-optimal” in claims 6, 7, 9, 16, 17, and 19.
Claims 8 and 18, which depend directly from claims 7 and 17, respectively, are rejected under 35 U.S.C. 112(b) as being indefinite under the same rationale as claims 7 and 17.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Hua et al, NPL (Learning for a Robot: Deep Reinforcement Learning, Imitation Learning, Transfer Learning), published February 11, 2021, hereinafter "Hua", in view of Hausman et al, WIPO Publication No. WO2023/192497, provisional filed March 31, 2022, hereinafter "Hausman", and further in view of Pertsch et al, NPL (Accelerating Reinforcement Learning with Learned Skill Priors) published October 22, 2020, hereinafter “Pertsch”).
With regard to independent claim 1,
Hua teaches, “A computer-implemented method for training a model, comprising”, (Pg. 6, Ch. 3, subsection 3.1, paragraph 4; EN: This denotes creating and training a neural network in deep reinforcement learning). “performing skill discovery, using a set of demonstrations that includes known-good demonstrations and noisy demonstrations, to generate a set of skills;” (Pg.8, chapter. 4, paragraph one; EN: This denotes the use of expert demonstrations for use in generating skills in imitation learning). “and …, the skill matching model, and the low-level policies together in an end-to-end fashion in a second training” (Pg. 6, Ch. 3, subsection 3.1, paragraph 4; EN: This denotes end-to-end training for a skill matching model and for low-level policies in deep reinforcement learning).
However, Hua fails to explicitly disclose “training a unidirectional skill embedding model in a first training”, “while parameters of a skill matching model and low-level policies that relate skills to actions are held constant; and training the unidirectional skill embedding model…”.
Pertsch teaches, “training a unidirectional skill embedding model in a first training” (Pg.1-2, Introduction, paragraph 3; EN: This denotes the use of SPiRL (Skill-Prior RL), which can be used as a unidirectional skill embedding model due to its use of deep latent variable model that learns an embedding space of skills and skill prior from an offline agent experience).
Hua and Pertsch are considered to be analogous to the claimed invention due to the fact that they are both generally related to reinforcement learning, the transfer of skills, and training models based on those skills. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the various robot learning styles of Hua with the training of transferrable skills learned from prior task to newer task of Pertsch. One would be motivated to do so to improve skills used by robots, various models, and skill discovery, as suggested by Pertsch (see, e.g., Pertsch, pg. 8, conclusion, paragraph 2).
However, Hua and Pertsch both fail to explicitly disclose “while parameters of a skill matching model and low-level policies that relate skills to actions are held constant”.
Hausman teaches, “while parameters of a skill matching model and low-level policies that relate skills to actions are held constant” (paragraph 0040; EN: This denotes that the sentence encoder language model parameters can be frozen during training).
Hua, Pertsch, and Hausman are considered to be analogous to the claimed invention due to the fact that they are each generally related to reinforcement learning, the transfer of skills, and training models based on those skills. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the various robot learning styles of Hua and the prior over skills training of Pertsch with the control of robots through natural language inputs of Hausman. One would be motivated to do so to improve skills used by robots, various models, and skill discovery, as suggested by Hausman (see, e.g., Hausman, pg. 2, paragraph 0006).
With regard to independent claim 11,
Hua and Pertsch both fail to explicitly disclose “A system for training a model, comprising: a hardware processor”.
However, in the same field, analogous art Hausman teaches, “A system for training a model, comprising: a hardware processor;” (Paragraph 0126; EN: This denotes that a processor is able to execute stored instructions to perform various methods). “and a memory that stores a computer program which, when executed by the hardware processor, causes the hardware processor to” (Paragraph 0109; EN: This denotes memory used for storage of instructions that can be executed by a processor). Further claim limitations are similar in scope to claim 1 and are rejected under a similar rationale as discussed above with regard to claim 1.
Hua, Pertsch, and Hausman are considered to be analogous to the claimed invention due to the fact that they are each generally related to reinforcement learning, the transfer of skills, and training models based on those skills. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the various robot learning styles of Hua and the prior over skills training of Pertsch with the control of robots through natural language inputs of Hausman. One would be motivated to do so to improve skills used by robots, various models, and skill discovery as suggested by Hausman (see, e.g., Hausman, pg. 2, paragraph 0006).
Claim(s) 2, 3, 6-9, 12, 13, and 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over Hua et al, NPL (Learning for a Robot: Deep Reinforcement Learning, Imitation Learning, Transfer Learning), published February 11, 2021, Hausman et al, WIPO Patent 2023192497, provisional filed March 30, 2022, and Pertsch et al, NPL (Accelerating Reinforcement Learning with Learned Skill Priors) published October 22, 2020 as applied to claims 1 and 11 above, and further in view of Grigsby et al, NPL (A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets), published October 10, 2021.
As discussed above, Hua in view of Pertsch and Hausman teaches the method of claim 1.
With regard to dependent claim 2,
Hua in view of Pertsch and Hausman does not explicitly teach “The method of claim 1, wherein skill discovery includes sampling positive and negative candidates for transition samples taken from the set”.
Grigsby teaches, “The method of claim 1, wherein skill discovery includes sampling positive and negative candidates for transition samples taken from the set” (Pg. 4, Ch. 3, paragraph 1-2; EN: This denotes separating various samples from the expert demonstrations based on their respective quality).
Hua, Pertsch, Hausman, and Grigsby are considered to be analogous to the claimed invention due to the fact that they are each generally related to reinforcement learning and training models based on various demonstrations. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the various robot learning styles of Hua, the prior over skills training of Pertsch, the control of robots through natural language inputs of Hausman with the modifications to offline reinforcement learning methods of Grigsby. One would be motivated to do so to improve skills used by robots, various models, and skill discovery, as suggested by Grigsby (see, e.g., Grigsby, pg. 8-9, Ch. 5, paragraph 2).
With regard to dependent claim 3,
As discussed above, Hua in view of Pertsch, Hausman, and Grigsby teaches the method of claim 2.
Hua in view of Pertsch and Hausman does not explicitly teach “wherein the positive candidates are sampled from a same clustering group.”
Grigsby teaches, “The method of claim 2, wherein the positive candidates are sampled from a same clustering group” (Pg. 4, Ch. 3, paragraph 1-2, Fig. 2; EN: This denotes that positive samples can come from higher quality demonstrations). “and the negative candidates are sampled from a different clustering group” (Pg. 4, Ch. 3, paragraph 1-2, Fig. 2; EN: This denotes that negative samples can come from other groups and can have its own category).
The motivation to combine Hua, Pertsch, Hausman, and Grigsby is the same as discussed above with respect to claim 2
With regard to dependent claim 6,
As discussed above, Hua in view of Pertsch and Hausman teaches the method of claim 1.
Hua in view of Pertsch and Hausman does not explicitly teach “wherein skill discovery is performed on a set of demonstrations that includes expert demonstrations with known-good outcomes and noisy demonstrations with sub-optimal outcomes.”
Grigsby teaches, “The method of claim 1, wherein skill discovery is performed on a set of demonstrations that includes expert demonstrations with known-good outcomes and noisy demonstrations with sub-optimal outcomes” (Pg. 4, Ch. 3, paragraph 1-2; EN: This denotes that the expert demonstrations can provide sub-optimal outcomes).
With regard to dependent claim 7,
Hua further teaches, “The method of claim 6, wherein the expert demonstrations are made up of a set of expert skills” (Pg. 10, Ch. 4, subsection 4.3, paragraphs 2-3; EN: This denotes the use of unsupervised third-person imitation learning. This allows learning from videos or other observations, which can provide a framework for skill acquisition). “and wherein the noisy demonstrations are made up of a combination of expert skills and sub-optimal skills” (Pg. 10, Ch. 4, subsection 4.3, paragraph 2; EN: This denotes the use of unsupervised third-person imitation learning. This allows learning from videos or other observations, which can be noisy demonstrations and can provide a framework for skill acquisition).
With regard to dependent claim 8,
Hua, Hausman, and Grigsby fail to explicitly teach “The method of claim 7, wherein the first training is performed using only the expert skills”.
Pertsch teaches, “The method of claim 7, wherein the first training is performed using only the expert skills” (Pg. 3, Ch. 3, subsection 3.2, paragraph 2, Fig. 2; EN: This denotes training a stochastic latent variable model using an offline dataset. This allows it to learn from skill embeddings and prior skills).
With regard to dependent claim 9,
Hua further teaches, “The method of claim 7, wherein the second training is performed using the expert skills and the sub-optimal skills” (Pg. 6, Ch. 3, subsection 3.1, paragraph 4; EN: This denotes training a neural network over various different iterations).
With regard to dependent claim 12,
This claim is similar in scope to claim 2 and is rejected under a similar rationale.
With regard to dependent claim 13,
This claim is similar in scope to claim 3 and is rejected under a similar rationale.
With regard to dependent claim 16,
This claim is similar in scope to claim 6 and is rejected under a similar rationale.
With regard to dependent claim 17,
This claim is similar in scope to claim 7 and is rejected under a similar rationale.
With regard to dependent claim 18,
This claim is similar in scope to claim 8 and is rejected under a similar rationale.
With regard to dependent claim 19,
This claim is similar in scope to claim 9 and is rejected under a similar rationale.
Claim(s) 4, 5, 14, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Hua et al, NPL (Learning for a Robot: Deep Reinforcement Learning, Imitation Learning, Transfer Learning), published February 11, 2021, Hausman et al, WIPO Patent 2023192497, provisional filed March 30, 2022, Pertsch et al, NPL (Accelerating Reinforcement Learning with Learned Skill Priors) published October 22, 2020, and Grigsby et al, NPL (A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets), published October 10, 2021 as applied to claims 2 and 12 above, and further in view of Guo et al, NPL (Social Bots Detection via Fusing BERT and Graph Convolutional Networks), published December 27, 2021.
With regard to dependent claim 4,
As discussed above, Hua in view of Pertsch, Hausman, and Grigsby teach the method of claim 2.
Hua in view of Pertsch, Hausman, and Grigsby fail to explicitly teach “The method of claim 2, wherein the positive candidates and the negative candidates are used to update a compatibility based on a mutual information”.
Guo teaches, “The method of claim 2, wherein the positive candidates and the negative candidates are used to update a compatibility based on a mutual information” (Ch. 3, subsection 3.1 and 3.2; EN: This denotes using both positive and negative examples for sentence-level dichotomies, which eventually uses PPMI (point-wise mutual information)).
Hua, Pertsch, Hausman, Grigsby, and Guo are considered to be analogous to the claimed invention due to the fact that they are each generally related to machine learning and training models based on various demonstrations. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the various robot learning styles of Hua, the prior over skills training of Pertsch, the control of robots through natural language inputs of Hausman, the modifications to offline reinforcement learning methods of Grigsby, with the social robot detection of Guo. One would be motivated to do so to improve the analysis of various models and improving detection of social robots, as suggested by Guo (see, e.g., Guo, Ch. 5).
With regard to dependent claim 5,
As discussed above, Hua in view of Pertsch, Hausman, Grigsby, and Guo teach the method of claim 4.
Hua further discloses “The method of claim 4, wherein skill discovery includes …, the skill matching model, and the low-level policies …” (Pg. 6, Ch. 3, subsection 3.1, paragraph 4; EN: This denotes end-to-end training for a skill matching model and for low-level policies in deep reinforcement learning).
However, Hua in view of Pertsch, Hausman, and Grigsby fail to explicitly teach “training a bidirectional skill embedding model, … using the mutual information”
In the same field, analogous art Guo teaches, “The method of claim 4, wherein skill discovery includes training a bidirectional skill embedding model, the skill matching model, … using the mutual information” (Ch. 3, subsection 3.1 and 3.2; EN: This denotes the use of a BERT (Bidirectional Encoder Representations from Transformers) model to initialize a text-based GCN, which eventually uses PPMI (point-wise mutual information)).
The motivation to combine Hua, Pertsch, Hausman, Grigsby, and Guo is the same as discussed above with respect to claim 4.
With regard to dependent claim 14,
This claim is similar in scope to claim 4 and is rejected under a similar rationale.
With regard to dependent claim 15,
This claim is similar in scope to claim 5 and is rejected under a similar rationale.
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Hua et al, NPL (Learning for a Robot: Deep Reinforcement Learning, Imitation Learning, Transfer Learning), published February 11, 2021, Hausman et al, WIPO Patent 2023192497, provisional filed March 30, 2022, and Pertsch et al, NPL (Accelerating Reinforcement Learning with Learned Skill Priors) published October 22, 2020 as applied to claims 1 and 11 above, and further in view of Commons et al, U.S. Patent No. 11514305, filed January 19, 2018.
With regard to dependent claim 10,
As discussed above, Hua in view of Pertsch and Hausman teach the method of claim 1.
Hua in view of Pertsch and Hausman fail to explicitly teach “The method of claim 1, wherein the low-level policies are implemented as multilayer perceptron neural network models”.
Commons teaches, “The method of claim 1, wherein the low-level policies are implemented as multilayer perceptron neural network models” (Col.11, lines 17-22; EN: This denotes that an artificial neural network can model a multilayer perceptron).
Hua, Pertsch, Hausman, and Commons are considered to be analogous to the claimed invention due to the fact that they are all generally related to reinforcement learning and training models based on various demonstrations. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the various robot learning styles of Hua, the prior over skills training of Pertsch, the control of robots through natural language inputs of Hausman, with the policy based artificial neural networks of Commons. One would be motivated to do so to improve skills used by various models and improving usable policies, as suggested by Commons (see, e.g., Commons, Col. 39 lines: 49-61).
With regard to dependent claim 20,
This claim is similar in scope to claim 10 and is rejected under a similar rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Ghadirzadeh (NPL: Data-Efficient Visuomotor Policy Training Using Reinforcement Learning and Generative Models) teaches the use of a InfoGAN model, which uses disentangled latent representations to maximize mutual information between latent code and generated samples. The model can produce samples that resemble the original samples from the training data.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SETH CAPRIANO-UMARI SHELTON whose telephone number is (571)270-0213. The examiner can normally be reached 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Ell can be reached at (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SETH CAPRIANO-UMARI SHELTON/ Examiner, Art Unit 2141
/RANDALL K. BALDWIN/ Primary Examiner, Art Unit 2125
Prosecution Insights