Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Detailed Action
This action is in response to the claims filed 8/7/2026:
Claims 1 – 29 are pending.
Claims 1, 9, 16, and 23 are independent.
Claim Objections
Claims 16, 18, and 20 are objected to because of the following informalities: “the least one hardware-based processor” should read “the at least one hardware-based processor”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-29 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 1, 9, 16, and 23, “until the data stored in the buffer exceeds a threshold” is indefinite. Data itself is not itself an ordinarily scalar quantity, possible interpretations include: number of samples, number of timesteps, number of epochs, occupied buffer capacity, number of bytes, values of data stored in the buffer, or something else altogether. Since each of these interpretations are contradictory with other interpretations, the scope of the claim cannot reasonably be determined. In the interest of further examination the claim is interpreted as “until a value in the data stored in the buffer exceeds a threshold.”
Regarding claims 5, 19, and 26, “the set of angular velocities” lacks antecedent basis. “A set of angular velocities” is recommended.
Regarding claims 21 and 22, "the processor" lacks antecedent basis. Claim 16 from which claims 21 and 22 depend introduces "at least one hardware-based processor" such that it would be unclear to one of ordinary skill in the art which processor "the processor" refers to.
The remaining claims are rejected with respect to their dependence on the rejected claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 5-10, 16, 17, 19-24, and 26-29 are rejected under U.S.C. §103 as being unpatentable over the combination of Zhang (“Model-Free Attitude Control of Spacecraft Based on PID-Guide TD3 Algorithm”, 2020) and Passarin (EP4292945A1).
PNG
media_image1.png
788
1332
media_image1.png
Greyscale
FIG. 3 of Zhang
Regarding claim 1, Zhang teaches A method for implementing a machine learning model for controlling a satellite system comprising at least one satellite, the method comprising:([p. 1] "On-Orbit Servicing Spacecraft (OOSS)" [p. 2] "The paper mainly studies the attitude control problem of OOSS under complex situations such as uncertain inertia matrix")
receiving a set of control parameters associated with an orientation of the at least one satellite;([p. 4] "The agent receives observations and rewards from the environment and sends actions to the environment" [p. 3] "The input of the environment is the action output by the agent, and the output is the observed state and reward signal after the action is performed. For the spacecraft attitude control problem, the system states are selected as attitude angle […] and angular velocity [...] The system state s and the error signal e are used as the observations")
processing each of the set of control parameters via a classic control model at a control subsystem to generate a first set of actions ([p. 6] "The core idea of the PID-Guide TD3 algorithm is as follows. In the current state s, two actions are generated by the action network and PID controller, respectively. Then, the critical network is used to evaluate the two actions; the action with higher value will be actually executed" See FIG. 3)
to control the orientation of the at least one satellite;([p. 6] "two actions are generated by the action network and PID controller, respectively. Then, the critical network is used to evaluate the two actions; the action with higher value will be actually executed" [p. 3] "The action space is continuous three-dimensional control torque")
storing the first set of actions as data in a buffer, ([p. 5] "Initialize replay buffer R" [p. 7] "Store transition tuple (s, a, r, s′) in R" See FIG. 3 Replay Buffer output. Zhang generates a PID candidate action and an actor candidate action. Its critic evaluates the two and the transition subsequently stored in the replay buffer contains that executed action a. Therefore, when the PID candidate is selected by the critic, the stored tuple necessarily contains the PID generated action together with the corresponding state, reward, and next state as shown in FIG. 3)
when the data stored in the buffer exceeds the threshold:([p. 5] "Initialize replay buffer R" [p. 7] "Store transition tuple (s, a, r, s′) in R" [p. 3] "if the agent exceeds a certain threshold, it may receive a positive reward" r is the reward which is explicitly stored in the buffer)
iteratively implementing, based on each of the set of control parameters and the data stored in the buffer, the machine learning model at the control subsystem ([p. 4] "TD3 Algorithm. The agent receives observations and rewards from the environment and sends actions to the environment. The TD3 algorithm is used as the learning method of agent in this paper. TD3 and DDPG are both off-policy algorithms. TD3 creates an experience replay buffer to store historical experiences and then randomly sample transitions" [p. 5] "Actor network [...] selects the current action a according to the current state s, which is used to interact with the environment to generate the next state s' and reward R")
to control the orientation of the at least one satellite to stabilize the at least one satellite.([p. 6] "two actions are generated by the action network and PID controller, respectively. Then, the critical network is used to evaluate the two actions; the action with higher value will be actually executed" [p. 3] "The action space is continuous three-dimensional control torque").
However, Zhang does not explicitly teach wherein the processing and storing are iteratively repeated until the data stored in the buffer exceeds a threshold; and.
Passarin, in the same field of endeavor, teaches wherein the processing and storing are iteratively repeated until the data stored in the buffer exceeds a threshold; and([¶0041] "The effects of these control commands on the satellite (e.g. the updated attitude states of the satellite following application of the control signals) are evaluated 212 by determining a reward, r, for the set of parameters of the machine-learning model each time step" [¶0043] "θmax is a threshold attitude error" [¶0085] "The average set of parameter values may comprise an average (e.g. a mean) of the top µ performing machine-learning models, i.e. the µ models with the highest reward values, where µ is less than the population size. The average may comprise a weighted average/sum of the parameter sets, where the weights are based at least in part on the reward value for each parameter set included in the sum" [¶0003] "to store state-action-reward time series" Passarin explicitly gates selected parameter values based on reward, where lower reward values are not selected, and commands with attitude error above θmax are given zero reward).
Zhang as well as Passarin are directed towards reinforcement learning for controlling satellites. Therefore, Zhang as well as Passarin are analogous art in the same field of endeavor. Zhang teaches repeatedly generating PID-guided spacecraft control actions, storing resulting state-action-reward data in a replay buffer, and using that stored data to iteratively train the TD3 controller. Similarly, Passarin teaches iterative satellite attitude control in which reward data are determined based on attitude-performance thresholds and are used to drive subsequent machine-learning updates. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Passarin's threshold-based reward evaluation to Zhang's existing PID-guided replay-buffer process, resulting in repeated processing and storage until the stored reward data satisfies the threshold, followed by iterative implementation of the machine-learning attitude controller using the buffered data. Passarin provides as additional motivation for combination that this iterative process (¶0016] “controlling the attitude of a satellite in a way that improves over time”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 2, the combination of Zhang and Passarin teaches The method of claim 1, wherein the machine learning model corresponds to a reinforcement learning model.(Zhang [p. 1] "This paper is devoted to model-free attitude control of rigid spacecraft in the presence of control torque saturation and external disturbances. Specifically, a model-free deep reinforcement learning (DRL) controller is proposed").
Regarding claim 3, the combination of Zhang and Passarin teaches The method of claim 2, wherein the reinforcement learning model is based on an actor-critic model.(Zhang [p. 1] "Considering the continuity of state space and action space, the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm based on actor-critic architecture is adopted").
Regarding claim 5, the combination of Zhang and Passarin teaches The method of claim 1, wherein the set of control parameters comprises: a set of quaternions; and (Zhang [p. 2] "There are usually five ways to describe the attitude of the spacecraft, including […] Quaternions […] Different description methods can be converted between each other")
the set of angular velocities.(Zhang [p. 3] "ω=[ωxωyωz] T∈ℝ3 is the rotation angular velocity").
Regarding claim 6, the combination of Zhang and Passarin teaches The method of claim 1, wherein the processing each of the set of control parameters via the classic control model comprises: determining an error between a desired orientation and an actual orientation of the at least one satellite to generate a determined error; and (Zhang [p. 2] "p(t) is the attitude angle, […] pd is the target attitude angle" [p. 3] "system state s and the error signal e are used as the observations, where e=(pT-p,wT-w)^T")
generating feedback based on the determined error, wherein the first set of actions is generated based on the feedback to control the orientation of the at least one satellite.(Zhang [p. 6] "PID is a model-free controller based on the feedback of error" [p. 6] "two actions are generated by the action network and PID controller, respectively").
Regarding claim 7, the combination of Zhang and Passarin teaches The method of claim 1, further comprising: simulating, by a guidance and navigation control (GNC) processor, the first set of actions to determine the set of control parameters of the at least one satellite; and(Zhang [p. 3] "In order to implement DRL, the simulation environment of the spacecraft attitude motion should be established according to the attitude dynamic and kinematic equations of spacecraft [...] the input of the environment is the action output by the agent" [p. 9] "In order to verify the effectiveness and superiority of the proposed PID-Guide TD3 algorithm, the simulations are carried out in this section")
storing the set of control parameters of the at least one satellite after simulating the first set of actions as data in the buffer.(Zhang [p. 5] "Initialize replay buffer R" [p. 7] "Store transition tuple (s, a, r, s′) in R" See FIG. 3 Replay Buffer output. Zhang generates a PID candidate action and an actor candidate action. Its critic evaluates the two and the transition subsequently stored in the replay buffer contains that executed action a. Therefore, when the PID candidate is selected by the critic, the stored tuple necessarily contains the PID generated action together with the corresponding state, reward, and next state as shown in FIG. 3).
Regarding claim 8, the combination of Zhang and Passarin teaches The method of claim 1, wherein the iteratively implementing the machine learning model comprises: generating a second set of actions to control the orientation of the at least one satellite (Zhang [p. 6] "PID is a model-free controller based on the feedback of error" [p. 6] "two actions are generated by the action network and PID controller, respectively")
to stabilize the at least one satellite;(Zhang [p. 3] "TD3 adds noise to the target action to reduce sensitivity and instability […] The action space is continuous three-dimensional control torque" [p. 7] "Figure 5: Attitude stabilization using the DRL controller")
storing the second set of actions as data in the buffer, wherein the second set of actions is different than the first set of actions;(Zhang [p. 5] "Initialize replay buffer R" [p. 7] "Store transition tuple (s, a, r, s′) in R" See also FIG. 3. when the actor-generated action wins the critic comparison and is executed, that ML-generated action a is stored in the replay-buffer transition)
simulating, by a guidance and navigation control (GNC) processor, the second set of actions to determine the set of control parameters of the at least one satellite; and(Zhang [p. 3] "In order to implement DRL, the simulation environment of the spacecraft attitude motion should be established according to the attitude dynamic and kinematic equations of spacecraft [...] the input of the environment is the action output by the agent" [p. 9] "In order to verify the effectiveness and superiority of the proposed PID-Guide TD3 algorithm, the simulations are carried out in this section")
storing the set of control parameters of the at least one satellite after simulating the second set of actions as data in the buffer.(Zhang [p. 5] "Initialize replay buffer R" [p. 7] "Store transition tuple (s, a, r, s′) in R" See also FIG. 3. when the actor-generated action wins the critic comparison and is executed, that ML-generated action a is stored in the replay-buffer transition).
Regarding claim 9, Zhang teaches A method for controlling a satellite system comprising at least one satellite, the method comprising:([p. 1] "On-Orbit Servicing Spacecraft (OOSS)" [p. 2] "The paper mainly studies the attitude control problem of OOSS under complex situations such as uncertain inertia matrix")
receiving a set of control parameters associated with an orientation of the at least one satellite; ([p. 4] "The agent receives observations and rewards from the environment and sends actions to the environment" [p. 3] "The input of the environment is the action output by the agent, and the output is the observed state and reward signal after the action is performed. For the spacecraft attitude control problem, the system states are selected as attitude angle […] and angular velocity [...] The system state s and the error signal e are used as the observations")
processing each of the set of control parameters via a classic control model at a control subsystem ([p. 6] "The core idea of the PID-Guide TD3 algorithm is as follows. In the current state s, two actions are generated by the action network and PID controller, respectively. Then, the critical network is used to evaluate the two actions; the action with higher value will be actually executed" See FIG. 3)
to generate a first set of actions to control an orientation of the at least one satellite;([p. 6] "two actions are generated by the action network and PID controller, respectively. Then, the critical network is used to evaluate the two actions; the action with higher value will be actually executed" [p. 3] "The action space is continuous three-dimensional control torque")
storing the first set of actions as data in a buffer, ([p. 5] "Initialize replay buffer R" [p. 7] "Store transition tuple (s, a, r, s′) in R" See FIG. 3 Replay Buffer output. Zhang generates a PID candidate action and an actor candidate action. Its critic evaluates the two and the transition subsequently stored in the replay buffer contains that executed action a. Therefore, when the PID candidate is selected by the critic, the stored tuple necessarily contains the PID generated action together with the corresponding state, reward, and next state as shown in FIG. 3)
when the data stored in the buffer exceeds the threshold:([p. 5] "Initialize replay buffer R" [p. 7] "Store transition tuple (s, a, r, s′) in R" [p. 3] "if the agent exceeds a certain threshold, it may receive a positive reward" r is the reward which is explicitly stored in the buffer)
iteratively executing, based on each of the set of control parameters and the data stored in the buffer, a machine learning model at the control subsystem ([p. 4] "TD3 Algorithm. The agent receives observations and rewards from the environment and sends actions to the environment. The TD3 algorithm is used as the learning method of agent in this paper. TD3 and DDPG are both off-policy algorithms. TD3 creates an experience replay buffer to store historical experiences and then randomly sample transitions" [p. 5] "Actor network [...] selects the current action a according to the current state s, which is used to interact with the environment to generate the next state s' and reward R")
to control the orientation of the at least one satellite to stabilize the at least one satellite; and([p. 6] "two actions are generated by the action network and PID controller, respectively. Then, the critical network is used to evaluate the two actions; the action with higher value will be actually executed" [p. 3] "The action space is continuous three-dimensional control torque")
controlling, based on one of the processing of each of the set of control parameters via the classic control model and the executing of the machine learning model, the orientation of the at least one satellite to stabilize the at least one satellite.([p. 2] "to realize attitude stabilization control and attitude tracking control of spacecraft" [p. 6] "two actions are generated by the action network and PID controller, respectively. Then, the critical network is used to evaluate the two actions; the action with higher value will be actually executed" [p. 3] "The action space is continuous three-dimensional control torque" See FIG. 2).
However, Zhang does not explicitly teach wherein the processing and storing are iteratively repeated until the data stored in the buffer exceeds a threshold;.
Passarin, in the same field of endeavor, teaches wherein the processing and storing are iteratively repeated until the data stored in the buffer exceeds a threshold;([¶0041] "The effects of these control commands on the satellite (e.g. the updated attitude states of the satellite following application of the control signals) are evaluated 212 by determining a reward, r, for the set of parameters of the machine-learning model each time step" [¶0043] "θmax is a threshold attitude error" [¶0085] "The average set of parameter values may comprise an average (e.g. a mean) of the top µ performing machine-learning models, i.e. the µ models with the highest reward values, where µ is less than the population size. The average may comprise a weighted average/sum of the parameter sets, where the weights are based at least in part on the reward value for each parameter set included in the sum" [¶0003] "to store state-action-reward time series" Passarin explicitly gates selected parameter values based on reward, where lower reward values are not selected, and commands with attitude error above θmax are given zero reward).
Zhang as well as Passarin are directed towards reinforcement learning for controlling satellites. Therefore, Zhang as well as Passarin are analogous art in the same field of endeavor. Zhang teaches repeatedly generating PID-guided spacecraft control actions, storing resulting state-action-reward data in a replay buffer, and using that stored data to iteratively train the TD3 controller. Similarly, Passarin teaches iterative satellite attitude control in which reward data are determined based on attitude-performance thresholds and are used to drive subsequent machine-learning updates. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Passarin's threshold-based reward evaluation to Zhang's existing PID-guided replay-buffer process, resulting in repeated processing and storage until the stored reward data satisfies the threshold, followed by iterative implementation of the machine-learning attitude controller using the buffered data. Passarin provides as additional motivation for combination that this iterative process (¶0016] “controlling the attitude of a satellite in a way that improves over time”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 10, the combination of Zhang and Passarin teaches The method of claim 9, wherein the machine learning model corresponds to a reinforcement learning model, (Zhang [p. 1] "This paper is devoted to model-free attitude control of rigid spacecraft in the presence of control torque saturation and external disturbances. Specifically, a model-free deep reinforcement learning (DRL) controller is proposed").
Regarding claim, claim 16 is directed towards a system for performing the method of claim 1. Therefore, the rejection applied to claim 1 also applies to claim 16. Claim 16 also recites additional elements A system for implementing a machine learning model for controlling a satellite system comprising at least one satellite, the system comprising: at least one hardware-based processor and memory, wherein the memory comprises processor-executable instructions encoded on a non-transient processor-readable media, wherein the processor-executable instructions, when executed by the least one hardware-based processor, configure the system to: (Zhang [p. 9] "All training processes are done on the computer with an Intel Seon Silver 4210 CPU and an NVIDIA Quadro P2000 GPU. Python 3.7 is used as the project interpreter").
Similarly, regarding claims 17 and 19-22, claims 17 and 19-22 are directed towards a system for performing the methods of claims 2 and 5-8, respectively. Therefore, the rejections applied to claims 2 and 5-8 also apply to claims 17 and 19-22.
Regarding claim 23, claim 23 is directed towards a system for performing the method of claim 9. Therefore, the rejection applied to claim 9 also applies to claim 23. Claim 23 also recites additional elements A system for controlling a satellite system comprising at least one satellite, the system comprising: at least one hardware-based processor and memory, wherein the memory comprises processor-executable instructions encoded on a non-transient processor-readable media, wherein the processor-executable instructions, when executed by the processor, configure the system to: (Zhang [p. 9] "All training processes are done on the computer with an Intel Seon Silver 4210 CPU and an NVIDIA Quadro P2000 GPU. Python 3.7 is used as the project interpreter").
Regarding claim 24, the combination of Zhang and Passarin teaches The system of claim 23, wherein the machine learning model corresponds to a reinforcement learning model.(Zhang [p. 1] "This paper is devoted to model-free attitude control of rigid spacecraft in the presence of control torque saturation and external disturbances. Specifically, a model-free deep reinforcement learning (DRL) controller is proposed").
Regarding claim 26, the combination of Zhang and Passarin teaches The system of claim 23, wherein the set of control parameters comprises: a set of quaternions; and(Zhang [p. 2] "There are usually five ways to describe the attitude of the spacecraft, including […] Quaternions […] Different description methods can be converted between each other")
the set of angular velocities.(Zhang [p. 3] "ω=[ωxωyωz] T∈ℝ3 is the rotation angular velocity").
Regarding claim 27, the combination of Zhang and Passarin teaches The system of claim 23, wherein the processor-executable instructions, when executed by the processor, further configure the system to process each of the set of control parameters via the classic control model by configuring the system to: determine an error between a desired orientation and an actual orientation of the at least one satellite to generate a determined error; and(Zhang [p. 2] "p(t) is the attitude angle, […] pd is the target attitude angle" [p. 3] "system state s and the error signal e are used as the observations, where e=(pT-p,wT-w)^T")
generate feedback based on the determined error, wherein the first set of actions is generated based on the feedback to control the orientation of the at least one satellite.(Zhang [p. 6] "PID is a model-free controller based on the feedback of error" [p. 6] "two actions are generated by the action network and PID controller, respectively").
Regarding claim 28, the combination of Zhang and Passarin teaches The system of claim 23, wherein the processor-executable instructions, when executed by the processor, further configure the system to: execute the first set of actions to determine the set of control parameters of the at least one satellite; and(Zhang [p. 3] "In order to implement DRL, the simulation environment of the spacecraft attitude motion should be established according to the attitude dynamic and kinematic equations of spacecraft [...] the input of the environment is the action output by the agent" [p. 9] "In order to verify the effectiveness and superiority of the pro posed PID-Guide TD3 algorithm, the simulations are carried out in this section")
store the set of control parameters of the at least one satellite after executing the first set of actions as data in the buffer.(Zhang [p. 5] "Initialize replay buffer R" [p. 7] "Store transition tuple (s, a, r, s′) in R" See FIG. 3 Replay Buffer output. Zhang generates a PID candidate action and an actor candidate action. Its critic evaluates the two and the transition subsequently stored in the replay buffer contains that executed action a. Therefore, when the PID candidate is selected by the critic, the stored tuple necessarily contains the PID generated action together with the corresponding state, reward, and next state as shown in FIG. 3).
Regarding claim 29, the combination of Zhang and Passarin teaches The system of claim 23, wherein the processor-executable instructions, when executed by the processor, further configure the system to iteratively implement the machine learning model by configuring the system to:(Zhang [p. 6] "PID is a model-free controller based on the feedback of error" [p. 6] "two actions are generated by the action network and PID controller, respectively")
generate a second set of actions to control the orientation of the at least one satellite to stabilize the at least one satellite;(Zhang [p. 3] "TD3 adds noise to the target action to reduce sensitivity and instability […] The action space is continuous three-dimensional control torque" [p. 7] "Figure 5: Attitude stabilization using the DRL controller")
store the second set of actions as data in the buffer, wherein the second set of actions is different than the first set of actions(Zhang [p. 5] "Initialize replay buffer R" [p. 7] "Store transition tuple (s, a, r, s′) in R" See also FIG. 3. when the actor-generated action wins the critic comparison and is executed, that ML-generated action a is stored in the replay-buffer transition)
execute the second set of actions to determine the set of control parameters of the at least one satellite; and(Zhang [p. 3] "In order to implement DRL, the simulation environment of the spacecraft attitude motion should be established according to the attitude dynamic and kinematic equations of spacecraft [...] the input of the environment is the action output by the agent" [p. 9] "In order to verify the effectiveness and superiority of the proposed PID-Guide TD3 algorithm, the simulations are carried out in this section")
store the set of control parameters of the at least one satellite after executing the second set of actions as data in the buffer.(Zhang [p. 5] "Initialize replay buffer R" [p. 7] "Store transition tuple (s, a, r, s′) in R" See also FIG. 3. when the actor-generated action wins the critic comparison and is executed, that ML-generated action a is stored in the replay-buffer transition).
Claims 4, 11-15, 18, and 25 are rejected under U.S.C. §103 as being unpatentable over the combination of Zhang and Passarin and in further view of Habib (“Combined spacecraft orbit and attitude control through extended Kalman filtering of magnetometer, gyro, and GPS measurements”, 2013).
Regarding claim 4, the combination of Zhang and Passarin teaches The method of claim 1.
However, the combination of Zhang and Passarin doesn't explicitly teach further comprising: receiving magnetic field information, a set of angular velocities, and global navigation satellite system (GNSS) data associated with the at least one satellite; and
combining, at a sensor fusion subsystem, the magnetic field information, the set of angular velocities, and the GNSS data to generate the set of control parameters associated with the at least one satellite,
wherein the magnetic field information comprises at least Earth's magnetic field.
Habib, in the same field of endeavor, teaches The method of claim 1, further comprising: receiving magnetic field information, a set of angular velocities, and global navigation satellite system (GNSS) data associated with the at least one satellite; and ([p. 93] "magnetometer, gyro, and GPS" GPS is a GNSS)
combining, at a sensor fusion subsystem, the magnetic field information, the set of angular velocities, and the GNSS data to generate the set of control parameters associated with the at least one satellite, ([p. 93] "magnetometer, gyro, and GPS […] Both estimates of spacecraft attitude and orbital states are fed effectively to the attitude and orbit control algorithms. The spacecraft attitude and orbit control algorithms were able to bring the spacecraft from the detumbling conditions to nadir pointing and execute the required orbital maneuver at the same time")
wherein the magnetic field information comprises at least Earth's magnetic field. ([p. 90] "b.b is earth’s magnetic field component measured by the magnetometer in the corresponding direction").
The combination of Zhang and Passarin as well as Habib are directed towards statistical algorithms for controlling satellites. Therefore, the combination of Zhang and Passarin as well as Habib are reasonably pertinent analogous art. It would have been obvious before the effective filing date of the claimed invention to combine the reinforcement learning system of the combination of Zhang and Passarin with the input set of Habib. Habib provides as additional motivation for combination ([Abstract] “Gyro and magnetometer are utilized to provide measurements for the estimates of spacecraft attitude state vector whereas magnetometer and GPS are utilized to provide measurements for the estimates of spacecraft orbital state vector”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 11, the combination of Zhang and Passarin teaches wherein the reinforcement learning model is based on an actor-critic model.(Zhang [p. 1] "Considering the continuity of state space and action space, the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm based on actor-critic architecture is adopted").
However, the combination of Zhang and Passarin doesn't explicitly teach The method of claim 9, further comprising: receiving magnetic field information, a set of angular velocities, and global navigation satellite system (GNSS) data associated with the at least one satellite; and
combining, at a sensor fusion subsystem, the magnetic field information, the set of angular velocities, and the GNSS data to generate the set of control parameters associated with the at least one satellite,
wherein the magnetic field information comprises at least Earth's magnetic field.
Habib, in the same field of endeavor, teaches The method of claim 9, further comprising: receiving magnetic field information, a set of angular velocities, and global navigation satellite system (GNSS) data associated with the at least one satellite; and([p. 93] "magnetometer, gyro, and GPS" GPS is a GNSS)
combining, at a sensor fusion subsystem, the magnetic field information, the set of angular velocities, and the GNSS data to generate the set of control parameters associated with the at least one satellite, ([p. 93] "magnetometer, gyro, and GPS […] Both estimates of spacecraft attitude and orbital states are fed effectively to the attitude and orbit control algorithms. The spacecraft attitude and orbit control algorithms were able to bring the spacecraft from the detumbling conditions to nadir pointing and execute the required orbital maneuver at the same time")
wherein the magnetic field information comprises at least Earth's magnetic field. ([p. 90] "b.b is earth’s magnetic field component measured by the magnetometer in the corresponding direction").
The combination of Zhang and Passarin as well as Habib are directed towards statistical algorithms for controlling satellites. Therefore, the combination of Zhang and Passarin as well as Habib are reasonably pertinent analogous art. It would have been obvious before the effective filing date of the claimed invention to combine the reinforcement learning system of the combination of Zhang and Passarin with the input set of Habib. Habib provides as additional motivation for combination ([Abstract] “Gyro and magnetometer are utilized to provide measurements for the estimates of spacecraft attitude state vector whereas magnetometer and GPS are utilized to provide measurements for the estimates of spacecraft orbital state vector”). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 12, the combination of Zhang, Passarin, and Habib teaches The method of claim 11, wherein the set of control parameters comprises: a set of quaternions; and (Zhang [p. 2] "There are usually five ways to describe the attitude of the spacecraft, including […] Quaternions […] Different description methods can be converted between each other")
the set of angular velocities. (Zhang [p. 3] "ω=[ωxωyωz] T∈ℝ3 is the rotation angular velocity").
Regarding claim 13, the combination of Zhang, Passarin, and Habib teaches The method of claim 11, wherein the processing each of the set of control parameters via the classic control model comprises: determining an error between a desired orientation and an actual orientation of the at least one satellite to generate a determined error; and(Zhang [p. 2] "p(t) is the attitude angle, […] pd is the target attitude angle" [p. 3] "system state s and the error signal e are used as the observations, where e=(pT-p,wT-w)^T")
generating feedback based on the determined error, wherein the first set of actions is generated based on the feedback to control the orientation of the at least one satellite. (Zhang [p. 6] "PID is a model-free controller based on the feedback of error" [p. 6] "two actions are generated by the action network and PID controller, respectively").
Regarding claim 14, the combination of Zhang, Passarin, and Habib teaches The method of claim 11, further comprising: executing, by a guidance and navigation control (GNC) processor, the first set of actions to determine the set of control parameters of the at least one satellite; and (Zhang [p. 3] "In order to implement DRL, the simulation environment of the spacecraft attitude motion should be established according to the attitude dynamic and kinematic equations of spacecraft [...] the input of the environment is the action output by the agent" [p. 9] "In order to verify the effectiveness and superiority of the proposed PID-Guide TD3 algorithm, the simulations are carried out in this section")
storing the set of control parameters of the at least one satellite after executing the first set of actions as data in the buffer. (Zhang [p. 5] "Initialize replay buffer R" [p. 7] "Store transition tuple (s, a, r, s′) in R" See FIG. 3 Replay Buffer output. Zhang generates a PID candidate action and an actor candidate action. Its critic evaluates the two and the transition subsequently stored in the replay buffer contains that executed action a. Therefore, when the PID candidate is selected by the critic, the stored tuple necessarily contains the PID generated action together with the corresponding state, reward, and next state as shown in FIG. 3).
Regarding claim 15, the combination of Zhang, Passarin, and Habib teaches The method of claim 11, wherein the iteratively executing the machine learning model comprises: generating a second set of actions to control the orientation of the at least one satellite (Zhang [p. 6] "PID is a model-free controller based on the feedback of error" [p. 6] "two actions are generated by the action network and PID controller, respectively")
to stabilize the at least one satellite; (Zhang [p. 3] "TD3 adds noise to the target action to reduce sensitivity and instability […] The action space is continuous three-dimensional control torque" [p. 7] "Figure 5: Attitude stabilization using the DRL controller")
storing the second set of actions as data in the buffer, wherein the second set of actions is different than the first set of actions; (Zhang [p. 5] "Initialize replay buffer R" [p. 7] "Store transition tuple (s, a, r, s′) in R" See also FIG. 3. when the actor-generated action wins the critic comparison and is executed, that ML-generated action a is stored in the replay-buffer transition)
executing, by a guidance and navigation control (GNC) processor, the second set of actions to determine the set of control parameters of the at least one satellite; and (Zhang [p. 3] "In order to implement DRL, the simulation environment of the spacecraft attitude motion should be established according to the attitude dynamic and kinematic equations of spacecraft [...] the input of the environment is the action output by the agent" [p. 9] "In order to verify the effectiveness and superiority of the proposed PID-Guide TD3 algorithm, the simulations are carried out in this section")
storing the set of control parameters of the at least one satellite after executing the second set of actions as data in the buffer.(Zhang [p. 5] "Initialize replay buffer R" [p. 7] "Store transition tuple (s, a, r, s′) in R" See also FIG. 3. when the actor-generated action wins the critic comparison and is executed, that ML-generated action a is stored in the replay-buffer transition).
Regarding claim 18, claim 18 is directed towards a system for performing the method of claim 4. Therefore, the rejection applied to claim 4 also applies to claim 18.
Regarding claim 25, the combination of Zhang and Passarin teaches The system of claim 23.
However, the combination of Zhang and Passarin doesn't explicitly teach wherein the processor-executable instructions, when executed by the processor, further configure the system to: receive magnetic field information, a set of angular velocities, and global navigation satellite system (GNSS) data associated with the at least one satellite; and
combine the magnetic field information, the set of angular velocities, and the GNSS data to generate the set of control parameters associated with the at least one satellite,
wherein the magnetic field information comprises at least Earth's magnetic field.
Habib, in the same field of endeavor, teaches the processor-executable instructions, when executed by the processor, further configure the system to: receive magnetic field information, a set of angular velocities, and global navigation satellite system (GNSS) data associated with the at least one satellite; and ([p. 93] "magnetometer, gyro, and GPS" GPS is a GNSS)
combine the magnetic field information, the set of angular velocities, and the GNSS data to generate the set of control parameters associated with the at least one satellite, ([p. 93] "magnetometer, gyro, and GPS […] Both estimates of spacecraft attitude and orbital states are fed effectively to the attitude and orbit control algorithms. The spacecraft attitude and orbit control algorithms were able to bring the spacecraft from the detumbling conditions to nadir pointing and execute the required orbital maneuver at the same time")
The combination of Zhang and Passarin as well as Habib are directed towards statistical algorithms for controlling satellites. Therefore, the combination of Zhang and Passarin as well as Habib are reasonably pertinent analogous art. It would have been obvious before the effective filing date of the claimed invention to combine the reinforcement learning system of the combination of Zhang and Passarin with the input set of Habib. Habib provides as additional motivation for combination ([Abstract] “Gyro and magnetometer are utilized to provide measurements for the estimates of spacecraft attitude state vector whereas magnetometer and GPS are utilized to provide measurements for the estimates of spacecraft orbital state vector”). This motivation for combination also applies to the remaining claims which depend on this combination.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Tipaldi (“Reinforcement learning in spacecraft control applications: Advances, prospects, and challenges”, 2022) is directed towards reinforcement learning for satellite attitude pointing.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY VINCENT BOSTWICK whose telephone number is (571)272-4720. The examiner can normally be reached M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached on (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SIDNEY VINCENT BOSTWICK/Examiner, Art Unit 2124