DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR
1.17(e), was filed in this application after final rejection. Since this application is eligible for
continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely
paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114.
Applicant's submission filed on 06/18/2025 has been entered.
Examiner’s Notes
In regards to the 35 USC § 112(b) rejection, has been withdrawn in light of the instant amendments to the claims. In regards to 35 USC § 103 rejection, has been withdrawn, because Applicant’s argument filed 06/18/2025 were found persuasive, see the allowable subject matter section.
Response to Arguments
Applicant’s arguments, see “Regarding the above viewpoint, since Kasaragod fails to disclose the policy module, Kasaragod certainly fails to disclose calling a necessary function definition and an optimizer during a process of creating the policy module. Moreover, the hyperparameters in Kasaragod are used for performing simulation and training the reinforcement learning model usable to optimize the application, not for creating the learner. Kasaragod does not disclose when the strategy function was created, when the necessary function definitions were invoked, and the optimizer.” filed 06/18/2025, have been fully considered and are persuasive. The 35 USC § 103 rejection has been withdrawn.
Claim Objections
Claim 1 is objected to because of the following informalities: the claim recites: “… the policy module is configured to determine a policy based on a policy parameter read by the sixth processor, and formulate a decision-making behavior adopted by the intelligent agent with the feature calculated by the representer as an input; the decision-making behavior comprises an action selection policy and an environment interaction mode;” should read “… the policy module is configured to determine a policy based on a policy parameter read by the sixth processor, and formulate a decision-making behavior adopted by the intelligent agent with the feature calculated by the representer as an input; and the decision-making behavior comprises an action selection policy and an environment interaction mode;”
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
runner, model library, learner, representer, policy module, intelligent agent, empirical data pool, in claim 1.,
runner in claim 3.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1 – 3 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AlA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AlA the applicant regards as the invention.
Claim limitation:
runner, model library, learner, representer, policy module, intelligent agent, empirical data pool, in claim 1.,
runner in claim 3.
invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function.
The correct requirement for satisfying the definiteness requirement is that the corresponding structure (or material or acts) of a means- (or step-) plus-function limitation must be disclosed in the specification itself in a way that one skilled in the art will understand what structure (or material or acts) will perform the recited function. If there is no disclosure of structure, material or acts for performing the recited function, the claim fails to satisfy the requirements of 35 U.S.C. 112(b). See Atmel Corp. v. Information Storage Devices, Inc., 198 F.3d 1374, 1381, 53 USPQ2d 1225, 1230 (Fed. Cir. 1999).
Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Regarding dependent claim 2, the claim does not resolve the deficiencies noted above; and is therefore appropriately rejected.
Claim Rejections - 35 USC § 112: New Matter
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and
of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and
process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1 – 3 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AlA), first paragraph, as
failing to comply with the written description requirement. The claim(s) contains subject matter
which was not described in the specification in such a way as to reasonably convey to one skilled in
the relevant art that the inventor or a joint inventor, or for applications subject to pre-AlA 35 U.S.C.
112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Specifically, claim 1 recites: “A deep reinforcement learning intelligent decision-making platform based on a unified artificial intelligence (AI) framework, comprising: a multi-processor system; a first processor; a second processor; a third processor; a fourth processor; a fifth processor; a sixth processor and a seventh processor” that is considered new matter because the original disclosure does not appear to provide support for the inclusion of processors in the specification.
Regarding the dependent claim(s) 2 – 3, do not resolve the noted deficiencies and thus are appropriately rejected.
Allowable Subject Matter
The following is an examiner’s statement of reasons for allowance.
Claim(s) 1 – 3 would be allowable If rewritten to amended to overcome: claim objection, 35 USC § 112 rejection, new matter rejection set forth in this Office action, because Applicant’s remarks filed 06/18/2025 were found persuasive (see page 8 – 14 of filed remarks)
“2. Examiner states (page 11 of Office action dated 2/18/2025) that Kasaragod discloses ([0035]) “the customer submits, through the client device 102, a request [call] to the simulation management service 104 [the multi-processor system] to train a reinforcement learning model usable to optimize a robotic [an intelligent agent] device application [a learner]. The request may specify the name of the reinforcement learning model to be trained, as well as the computer-executable code 110 that defines the custom-designed reinforcement function for training the reinforcement learning model. The request to train the reinforcement learning model may also include a set of parameters [is configured to read the parameters of the deep reinforcement learning model], including a set of simulation parameters and a set of system parameters. The set of simulation parameters [from the model library according to the parameters] may include a set of hyperparameters for performing the simulation and training the reinforcement learning model usable to optimize the application [call a necessary function definition and an optimizer from the fifth processor during a process of creating the policy module and the learner]. For instance, the set of simulation parameters may include the batch size for the simulation, which may be used to determine the GPU requirements for the simulation. For example, the number of GPUs required for the simulation may increase in proportion to the batch size specified by the customer via the interface provided by the simulation management service 104.”
In summary, the reference made of record, fail to disclose the required claimed technical features recited by the independent claim limitation as a whole, see remark failed 06/18/2025.
Furthermore, the references of record alone or in combination disclose or suggest the combination of limitations found within the independent claims as a whole without hindsight reasoning. The dependent claims, being further limiting to the independent claims, definite, and enable by the Specification are also allowed.
Claim 1 recites:
A deep reinforcement learning intelligent decision-making platform based on a unified artificial intelligence (AI) framework, comprising: a first processor; a multi-processor system; a second processor; a third processor; a fourth processor; a fifth processor; a model library; and a runner;
the first processor is configured to select parameters of a deep reinforcement learning model, comprising an intelligent agent name, a representer name, a policy name, a learner name, an algorithmic parameter, an environment name, and a system parameter;
the multi-processor system is configured to read the parameters of the deep reinforcement learning model; call and create a representer, a policy module, a learner, and an intelligent agent from the model library according to the parameters; and call a necessary function definition and an optimizer from the fifth processor during a process of creating the policy module and the learner;
the third processor is configured to create parallel environments based on an original environment according to the parameters;
the fourth processor is configured to make the parallel environments to obtain made environments, and input the made environments and the intelligent agent into the runner; the runner is configured to compute an action output, and execute the action output in the made environments to realize intelligent decision-making;
the model library is configured to provide a user with the deep reinforcement learning model, and customize and optimize the deep reinforcement learning model according to different scenarios and task requirements; the model library consists of the representer, the policy module, the learner, and the intelligent agent; the representer is configured to be determined based on a representation parameter read by a sixth processor, and convert raw observation data in the made environments into a feature suitable for being processed by the deep reinforcement learning model for representation;
the policy module is configured to determine a policy based on a policy parameter read by the sixth processor, and formulate a decision-making behavior adopted by the intelligent agent with the feature calculated by the representer as an input; the decision-making behavior comprises an action selection policy and an environment interaction mode;
the learner is configured to be determined based on a learner parameter read by the sixth processor, formulate a learning rule based on empirical data and the action selection policy, so as to obtain an action-selection policy; the intelligent agent is configured to be determined based on an agent parameter read by the sixth processor, output an action and execute the decision-making behavior using the action-selection policy of the learner, and interact with a simulation environment; the first processor is also configured to configure parameters involved in decision-making algorithms and tasks in a YAML Ain't a Markup Language (YAML) format, and transfer configured parameters to the multi-processor system; the multi-processor system is configured to store a programming module required by different decision-making algorithms for solving different decision-making problems;
the multi-processor system includes the sixth processor, a seventh processor and an empirical data library; the sixth processor is configured to read a YAML file in the first processor, transfer a parameter read from the YAML file to the intelligent agent and the runner, transfer the parameter to the learner, the policy module, and the representer in turn through the intelligent agent, and transfer the parameter to the second processor, the third processor, and the fourth processor through the runner;
the seventh processor is configured to read a terminal command to support user's interaction with the deep reinforcement learning intelligent decision-making platform; the empirical data library is configured to store and manage empirical data from environment interactions; the empirical data library is configured to be associated with the learner through the intelligent agent to support an experience replay training and optimization process of the learner;
the second processor is configured to store original environment definitions for different simulation environments, comprising parameter acquisition, environment reset, action execution, environment rendering and global state acquisition functions of the original environment, and provide the third processor, the fourth processor, the intelligent agent and the policy module with a basic tool and parameters for simulation environment interaction; the third processor is configured to randomly create a plurality of environments to run in parallel according to the original environment to interact with the intelligent agent; and the fourth processor is configured to make a specific simulation environment according to the simulation scenarios and task requirements, to interact with the intelligent agent.
Closest prior arts disclose:
Kasaragod et al., Pub. No.: US20200167686A1, (2018-11-27).
Kasaragod et al., describes a simulation management service that enables customers to build and train reinforcement learning models for robotic device by defining a custom reinforcement function and simulation parameters through UI, evaluate the reinforcement function and update the RL model based on performance feedback from simulated actions and rewards. However, Kasaragod does not teach comprehensive and modular platform for deep reinforcement learning (DRL) that enables firms to efficiently build, configure and deploy intelligent decision-making systems to outline a structured framework where parameters defining the agent, learner, policy and environment are specified in a Ain't a Markup Language (YAML) configuration file. Dynamic assemble DRL components from a model library based on these parameters, create parallel environments for scalable training, and run agents that interact with simulated environment in real time.
Majumder et al., "Deep Reinforcement Learning." Deep Reinforcement Learning in Unity: With Unity ML Toolkit (2021): 305 - 447.
The article introduces the foundational concepts of deep reinforcement learning (RL) and how they integrate with the ML agent’s toolkit’s brain academy architecture. It emphasizes the role of Python-based RL algorithms, such as deep Q-Learning and PPO, in guiding agent behavior, while also exploring actor-critic methods in detail. To prepare for implementing these algorithms, the article covers deep learning fundamentals, using TensorFlow and Keras, including the creation of neural network models for vision tasks. However, Majumder does not teach comprehensive and modular platform for deep reinforcement learning (DRL) that enables firms to efficiently build, configure and deploy intelligent decision-making systems to outline a structured framework where parameters defining the agent, learner, policy and environment are specified in a Ain't a Markup Language (YAML) configuration file. Dynamic assemble DRL components from a model library based on these parameters, create parallel environments for scalable training, and run agents that interact with simulated environment in real time.
Chou, et al., "Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution.", (2017).
The article addresses the limitation of using a standard Gaussian distribution in reinforcement learning for real-world control problems, where actions are physically bounded, proposed replacing the Gaussian policy with a Beta distribution, which naturally respects action bounds. Analyze the bias and variance of policy gradients and demonstrate that the Beta policy is bias-free. However, Kasaragod does not teach comprehensive and modular platform for deep reinforcement learning (DRL) that enables firms to efficiently build, configure and deploy intelligent decision-making systems to outline a structured framework where parameters defining the agent, learner, policy and environment are specified in a Ain't a Markup Language (YAML) configuration file. Dynamic assemble DRL components from a model library based on these parameters, create parallel environments for scalable training, and run agents that interact with simulated environment in real time.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to Matiyas T Maru whose telephone number is (571)270-0902. The examiner
can normally be reached Monday - Friday 8:00 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to
use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
Michelle Bechtold can be reached on (571)431-0762. The fax phone number for the organization
where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users.
To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA)
or 571-272-1000.
/M.T.M./ Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148