Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Detailed Action
This action is in response to the claims filed 2/26/2026:
Claims 1 – 20 are pending.
Claims 1 – 9 are withdrawn.
Claims 10 and 15 are independent.
Claim Rejections - 35 USC § 101
101 Rejection
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 10-20 are rejected under 35 USC § 101 because the claimed invention is directed to non-statutory subject matter.
Regarding Claim 10: Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 10 is directed to a method, which is directed towards a process, one of the statutory categories.
Step 2A Prong One Analysis: Claim 10 under its broadest reasonable interpretation is a series of mental processes. For example, but for the generic computer components language, the above limitations in the context of this claim encompass machine learning processing, including the following:
based on the recorded data, simulating multiple different synthetic sets of interactions between the user and the computing device, wherein each synthetic set comprises a variation of the observed set of interactions at a different level of abstraction (observation, evaluation, and judgement),
based on the user feedback, selecting one of the multiple different synthetic sets of interactions (observation, evaluation, and judgement)
Therefore, claim 10 recites an abstract idea which is a judicial exception.
Step 2A Prong Two Analysis: Claim 10 recites additional elements “one or more processors” and “causing a machine learning model to be trained to generate output indicative of the selected synthetic set of interactions”. However, these additional features are computer components recited at a high-level of generality, such that they amount to no more than mere instructions to apply the judicial exception using a generic computer component. An additional element that merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea, does not integrate the judicial exception into a practical application (See MPEP 2106.05(f)). Claim 10 also recites additional elements “recording data indicative of an observed set of interactions between a user and a computing device”, “obtaining user feedback about each of the multiple different synthetic sets of interactions” which amounts to gathering and outputting data which is insignificant extra-solution activity (See MPEP 2106.05(g)). Therefore, claim 10 is directed to a judicial exception.
Step 2B Analysis: Claim 10 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the lack of integration of the abstract idea into a practical application, the additional elements recited in claim 10 amount to no more than mere instructions to apply the judicial exception using a generic computer component and insignificant extra-solution activity. The gathering and outputting of data is considered well-understood, routine, and conventional in the art (See MPEP 2106.05(d)(II)).
For the reasons above, claim 10 is rejected as being directed to non-patentable subject matter under §101. This rejection applies equally to independent claim 15 which recites a system, as well as to dependent claims 10-14 and 16-20.
The additional elements of independent claim 15 “one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to” amount to mere instructions to apply the judicial exception using generic computer components.
The additional limitations of the dependent claims are addressed briefly below:
Dependent claims 11 and 16 recite additional observation, evaluation, and judgement “the simulating is performed based on the machine learning model.”
Dependent claims 12 and 17 recite additional instructions to apply the judicial exception using generic computer components “the machine learning model is trained to facilitate intelligent process automation.”
Dependent claims 13 and 18 recite additional mathematical calculations and relationships “the machine learning model is trained to generate a probability distribution over an action space”.
Dependent claims 14, 19, and 20 recite additional instructions to apply the judicial exception using generic computer components “the machine learning model comprises a private machine learning model” as well as additional insignificant extra-solution activity of gathering and outputting data “providing parameters of the trained private machine learning model for federated learning of a global machine learning model” which is well-understood, routine, and conventional in the art (See MPEP 2106.05(d)(II)(i))
Therefore, when considering the elements separately and in combination, they do not add significantly more to the inventive concept. Accordingly, claims 10-20 are rejected under 35 U.S.C. § 101.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 10-13, and 15-18 are rejected under U.S.C. §102(a)(1) as being anticipated by Bıyık (“Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences”, 2021).
PNG
media_image1.png
298
774
media_image1.png
Greyscale
FIG. 1 of Bıyık
Regarding claim 10, Bıyık teaches A method implemented using one or more processors and comprising:([p. 11] "Participants provided demonstrations via teleoperation (using end-effector control) on a keyboard interface; each user was given some time to familiarize themselves with the teleoperation system before beginning the experiment" See also FIG. 2 which shows computer with processor connected to robot and keyboard.)
recording data indicative of an observed set of interactions between a user and a computing device;([p. 4] "DemPref starts with a set of offline trajectory demonstrations D. These demonstrations are collected passively: the robot lets the human show their desired behavior, and does not interfere or probe the user. We leverage these passive human demonstrations to initialize an informative but imprecise prior over the true reward weights ω." [p. 3] "Figure 2. Example of a demonstration (top) and preference query (bottom). During the demonstration the robot is passive, and the human teleoperates the robot to produce trajectory ξD from scratch")
based on the recorded data, simulating multiple different synthetic sets of interactions between the user and the computing device, ([p. 4] "The human starts by providing a set of high-level demonstrations (left), which are used to initialize the robot’s belief over the human’s reward function. The robot then fine-tunes this belief by asking questions (right): the robot actively generates a set of trajectories" Robot generated trajectory interpreted as synthetic set of interactions between the user and the computing device)
wherein each synthetic set comprises a variation of the observed set of interactions at a different level of abstraction;([p. 4] "Intuitively, demonstrations provide an informative, high-level understanding of what behavior the human wants; however, these demonstrations are often noisy, and may fail to cover some aspects of the reward function. By contrast, preferences are fine-grained: they isolate specific, ambiguous aspects of the human’s reward, and reduce the robot’s uncertainty over these regions. It therefore makes sense for the robot to start with high-level demonstrations before moving to fine-grained preferences" high level and fine grained are interpreted as different levels of abstraction)
obtaining user feedback about each of the multiple different synthetic sets of interactions;([p. 4] "The robot then fine-tunes this belief by asking questions (right): the robot actively generates a set of trajectories, and asks the human to indicate their favorite"')
based on the user feedback, selecting one of the multiple different synthetic sets of interactions; and([p. 3] "the preference query is active: the robot chooses two trajectories ξ1 and ξ2 to show to the human, and the human answers by selecting their preferred option" [p. 4] "The robot then fine-tunes this belief by asking questions (right): the robot actively generates a set of trajectories, and asks the human to indicate their favorite"')
causing a machine learning model to be trained to generate output indicative of the selected synthetic set of interactions.([p. 5] "DemPref iteratively performs two main tasks: actively choosing the right preference query Q to ask, and applying the human’s answer to update b" [p. 11] "After training, the robot was trained in simulation using Proximal Policy Optimization (PPO) with the reward function learned from each system").
Regarding claim 11, Bıyık teaches The method of claim 10, wherein the simulating is performed based on the machine learning model.(Bıyık [p. 4] "The human starts by providing a set of high-level demonstrations (left), which are used to initialize the robot’s belief over the human’s reward function. The robot then fine-tunes this belief by asking questions (right): the robot actively generates a set of trajectories" [p. 11] "After training, the robot was trained in simulation using Proximal Policy Optimization (PPO) with the reward function learned from each system").
Regarding claim 12, Bıyık teaches The method of claim 11, wherein the machine learning model is trained to facilitate intelligent process automation.(Bıyık [p. 5] "DemPref iteratively performs two main tasks: actively choosing the right preference query Q to ask, and applying the human’s answer to update b" [p. 11] "After training, the robot was trained in simulation using Proximal Policy Optimization (PPO) with the reward function learned from each system" [p. 3] "Our motivation, physical autonomous systems").
Regarding claim 13, Bıyık teaches The method of claim 11, wherein the machine learning model is trained to generate a probability distribution over an action space.(Bıyık [p. 4] "Let the belief b be a probability distribution over ω. We initialize b using the trajectory demonstrations, so that b0(ω) = P(ω | D) […] In order to evaluate Equation (3), we need a model of P(ξD | ω)—in other words, how likely is the demonstrated trajectory ξD given that the human’s reward weights are ω [...] A trajectory ξ ∈ Ξ is a finite sequence of state action pairs").
Regarding claim 15, claim 15 is directed towards a system for performing the method of claim 10. Therefore, the rejection applied to claim 10 also applies to claim 15. Claim 15 recites additional elements one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to (Bıyık [p. 11] "Participants provided demonstrations via teleoperation (using end-effector control) on a keyboard interface; each user was given some time to familiarize themselves with the teleoperation system before beginning the experiment" See also FIG. 2 which shows desktop computer having memory and processor connected to robot and keyboard.)).
Regarding claim 16, Bıyık teaches The system of claim 15, wherein the machine learning model is used to simulate the multiple different synthetic sets of interactions between the user and the computing device.(Bıyık [p. 4] "The human starts by providing a set of high-level demonstrations (left), which are used to initialize the robot’s belief over the human’s reward function. The robot then fine-tunes this belief by asking questions (right): the robot actively generates a set of trajectories" [p. 11] "After training, the robot was trained in simulation using Proximal Policy Optimization (PPO) with the reward function learned from each system").
Regarding claims 17 and 18, claims 17 and 18 are directed towards a system for performing the method of claims 12 and 13. Therefore the rejection applied to claims 12 and 13 also apply to claims 17 and 18.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 14, 19, and 20 are rejected under U.S.C. §103 as being unpatentable over the combination of Bıyık and Liu (“Federated imitation learning: A novel framework for cloud robotic systems with heterogeneous sensor data”, 2020).
Regarding claim 14, Bıyık teaches The method of claim 11, wherein the machine learning model comprises a private machine learning model, (Bıyık [p. 3] "MDP. Let us consider a fully observable dynamical system describing the evolution of the robot, which should ideally behave according to the human’s preferences. We formulate this system as a discrete-time Markov Decision Process (MDP) M").
However, Bıyık doesn't explicitly teach and the method further comprises providing parameters of the trained private machine learning model for federated learning of a global machine learning model..
Liu, in the same field of endeavor, teaches and the method further comprises providing parameters of the trained private machine learning model for federated learning of a global machine learning model. ([Abstract] "we present a novel framework named FIL. It provides a heterogeneous knowledge fusion mechanism for cloud robotic systems. Then, a knowledge fusion algorithm in FIL is proposed. It enables the cloud to fuse heterogeneous knowledge from local robots and generate guide models for robots with service requests. After that, we introduce a knowledge transfer scheme to facilitate local robots acquiring knowledge from the cloud").
Bıyık as well as Liu are directed towards autonomous imitation systems. Therefore, Bıyık as well as Liu are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Bıyık with the teachings of Liu by applying federated learning to the inverse reinforcement learning system in Bıyık. Liu provides as additional motivation for combination ([p. 6] “The results are summarized in Table I. It can be seen from the experimental results that the cloud knowledge improves the local controller that is trained using general imitation learning. The controller based on cloud policies performs better”).
Regarding claims 19 and 20, claims 19 and 20 are directed towards systems for implementing the method of claim 14. Therefore, the rejection applied to claim 14 also applies to claims 19 and 20.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Singh (US20230385085A1) is directed towards training imitation robots.
Cataltepe (US12175345B2) is directed towards continuous online learning of human interactions based on synthetic interactions.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY VINCENT BOSTWICK whose telephone number is (571)272-4720. The examiner can normally be reached M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached on (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SIDNEY VINCENT BOSTWICK/Examiner, Art Unit 2124
Prosecution Insights