DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application is being examined under the pre-AIA first to invent provisions.
Response to Amendment
As per the instant Application having Application number 18/286,609 the examiner acknowledges the applicant's submission of the amendment dated 04/26/2024. At this point, claims 1-4, 6-7, 11, 13, 14-20, the abstract, and the specification have been amended. Claims 21-28 have been cancelled. Claims 21-28 are pending.
Claim Objections
Claim 5 is objected to because of the following informalities: Claim 5 depends on itself (claim 5) when it appears to a dependence on claim 4 is intended. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to the mental process of making judgments without significantly more.
In determining whether the claims are subject matter eligible, the Examiner applies the 2019 USPTO Patent Eligibility Guidelines. (2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50, Jan. 7, 2019.)
Regarding claim 1:
Step 1: Is the claim to a process, machine, manufacture, or composition of matter? Yes—claim 1 a method.
Step 2A, prong one: Does the claim recite an abstract idea, law of nature or natural phenomenon? Yes. Claim 1 recites “A method for configuring a reinforcement learning agent to perform a task using a reward structure derived from a task-specific definition of metric importances, the method being performed by a computing unit executing a configurator component and comprising: obtaining a definition of metric importances specifying, for a plurality of performance-related metrics associated with the task, pairwise importance values each indicating a relative importance of one metric with respect to another metric of the plurality of performance-related metrics for the task; deriving a reward structure from the definition of metric importances, the reward structure defining, for each of the plurality of performance-related metrics, a reward to be attributed to an action taken by the reinforcement learning agent that yields a positive outcome in the respective performance-related metric; and configuring the reinforcement learning agent to employ the derived reward structure when performing the task.” This all recites the mental process of making judgments based on human intelligence which can be performed in the human mind and/or with the aid of pen and paper without significantly more.
Step 2A, prong two: Does the claim recite additional elements that integrate the judicial exception into a practical application? No – Although claim 1 recites “a reinforcement learning agent … a computing unit executing a configurator component… by the reinforcement learning agent”, they are recited at a high-level of generality such that it amounts to no more than a mere instructions to apply the exception using a generic machines that gather data and input into readily available routine into an off the shelf models which is no more than extra solution activity (see MPEP 2106.05 (f)). The receiving of input data and the sending output data amounts to no more than extra solution activity (see MPEP 2106.05(g)); 2106.05(d)(II)(iii) update log, electronic record keeping; 2106.05(d)(II)(iv) storing and retrieving data in memory).
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No – Although claim 1 recites “a reinforcement learning agent … a computing unit executing a configurator component… by the reinforcement learning agent”, they are recited at a high-level of generality such that it amounts to no more than a mere instructions to apply the exception using a generic machines that gather data and input into readily available routine into an off the shelf models which is no more than extra solution activity (see MPEP 2106.05 (f)). The receiving of input data and the sending output data amounts to no more than extra solution activity (see MPEP 2106.05(g)); 2106.05(d)(II)(iii) update log, electronic record keeping; 2106.05(d)(II)(iv) storing and retrieving data in memory). See Alice Corp. Pty. Ltd. v. CLS Bank Int’l, 573 U.S. 208, 134 S. Ct. 2347, 2360 (2014).
For the reasons above, claim 1 is rejected as being directed to non-patentable subject matter under §101. The additional limitations of the dependent claims are addressed briefly below:
Regarding dependent claim 2: “wherein deriving the reward structure from the definition of metric importances is performed using a multi-criteria decision-making, MCDM, technique.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 1.
Regarding dependent claim 3: “wherein the definition of metric importances is provided as a matrix A:A1...An where n is the number of metrics of the plurality of performance-related metrics and wij is the pairwise importance value indicating the relative importance of metric Ai with respect to metric Aj, where i=1, ..., n and j=1, ..., n” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 1 and further incorporates the abstract idea of mathematical concept of matrix math without significantly more.
Regarding dependent claim 4: “wherein deriving the reward structure from the matrix A includes solving the eigenvalue problem Aw=λw: [w11.Math.w1n.Math.⋱.Math.wn1.Math.wnn][w1.Math.wn]=λ[w1.Math.wn] where λ is the maximum eigenvalue of A and w=[w.sub.1 . . . w.sub.n] is the solution of the eigenvalue problem, wherein each weight w.sub.i is taken as the reward for the corresponding metric A.sub.i, where i=1, . . . , n.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 3 and further incorporates the abstract idea of mathematical concept of matrix math without significantly more.
Regarding dependent claim 5: “wherein w=[w.sub.1 . . . w.sub.n] is normalized by dividing each weight w.sub.i by the sum of the weights w.sub.1 . . . w.sub.n, where i=1, . . . , n.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 4 and further incorporates the abstract idea of mathematical concept of matrix math without significantly more.
Regarding dependent claim 6: “1herein the matrix A is a positive reciprocal matrix.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 3 and further incorporates the abstract idea of mathematical concept of matrix math without significantly more.
Regarding dependent claim 7: “wherein deriving the reward structure from the matrix A includes performing a consistency check of the matrix A using, as a measure of deviation of the matrix A from consistency, an inconsistency value defined by: λ-nn-1” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 4 and further incorporates the abstract idea of mathematical concept of matrix math without significantly more.
Regarding dependent claim 8: “wherein, if the inconsistency value is above a predefined threshold, deriving the reward structure from the matrix A includes identifying, among the pairwise importance values w.sub.ij of the matrix A, one or more entries causing inconsistency and perturbing the one or more entries to reduce the inconsistency.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 7 and further incorporates the abstract idea of mathematical concept of matrix math without significantly more.
Regarding dependent claim 9: “wherein identifying and perturbing one or more entries causing inconsistency is iteratively performed until the inconsistency value is below the predefined threshold” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 8.
Regarding dependent claim 10: “wherein, if the inconsistency value is above a predefined threshold, deriving the reward structure from the matrix A includes reconstructing the matrix A based on a set of distinct eigenvalues λ.sub.1, . . . , λ.sub.n and corresponding linearly independent eigenvectors v.sub.1, . . . , v.sub.n, wherein the matrix A is reconstructed as A=PDP.sup.−1 where matrix P is constructed by stacking v.sub.1, . . . , v.sub.n as column vectors and matrix D is D=(λ.sub.1, . . . , λ.sub.n).” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 7 and further incorporates the abstract idea of mathematical concept of matrix math without significantly more.
Regarding dependent claim 11: “wherein the definition of metric importances is derived from a requirements specification regarding the task to be performed by the reinforcement learning agent.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 1. The use of “reinforcement learning agent” is recited at a high-level of generality such that it amounts to no more than a mere instructions to apply the exception using a generic machines that gather data and input into readily available routine into an off the shelf models which is no more than extra solution activity (see MPEP 2106.05 (f)). The receiving of input data and the sending output data amounts to no more than extra solution activity (see MPEP 2106.05(g)); 2106.05(d)(II)(iii) update log, electronic record keeping; 2106.05(d)(II)(iv) storing and retrieving data in memory).
Regarding dependent claim 12: “herein the requirements specification is formulated using a formal requirements specification syntax, optionally an Easy Approach to Requirements Syntax, EARS, wherein at least portions of the requirements specification are pattern matched to derive the definition of metric importances.” –which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 11.
Regarding dependent claim 13: “wherein an explanation provided in response to a query requesting a reason why the reinforcement learning agent took a particular action is provided on the basis of the derived reward structure.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 1. The use of “reinforcement learning agent” is recited at a high-level of generality such that it amounts to no more than a mere instructions to apply the exception using a generic machines that gather data and input into readily available routine into an off the shelf models which is no more than extra solution activity (see MPEP 2106.05 (f)). The receiving of input data and the sending output data amounts to no more than extra solution activity (see MPEP 2106.05(g)); 2106.05(d)(II)(iii) update log, electronic record keeping; 2106.05(d)(II)(iv) storing and retrieving data in memory).
Regarding dependent claim 14: “ wherein the explanation is provided with reference to a formulation of a requirements specification regarding the task to be performed by the reinforcement learning agent, and optionally indicating that the particular action was taken in order to meet the formulation of the requirements specification.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 1. The use of “reinforcement learning agent” is recited at a high-level of generality such that it amounts to no more than a mere instructions to apply the exception using a generic machines that gather data and input into readily available routine into an off the shelf models which is no more than extra solution activity (see MPEP 2106.05 (f)). The receiving of input data and the sending output data amounts to no more than extra solution activity (see MPEP 2106.05(g)); 2106.05(d)(II)(iii) update log, electronic record keeping; 2106.05(d)(II)(iv) storing and retrieving data in memory).
Regarding dependent claim 15: “wherein the reinforcement learning agent is operable to perform the task in a plurality of deployment setups, wherein, for each of the plurality of deployment setups, a different definition of metric importances specific to the respective deployment setup is obtained and used to derive a different reward structure specific to the respective deployment setup, wherein the reinforcement learning agent is configured to employ one of the different reward structures depending on the deployment setup in which the reinforcement learning agent currently operates.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 1. The use of “reinforcement learning agent” is recited at a high-level of generality such that it amounts to no more than a mere instructions to apply the exception using a generic machines that gather data and input into readily available routine into an off the shelf models which is no more than extra solution activity (see MPEP 2106.05 (f)). The receiving of input data and the sending output data amounts to no more than extra solution activity (see MPEP 2106.05(g)); 2106.05(d)(II)(iii) update log, electronic record keeping; 2106.05(d)(II)(iv) storing and retrieving data in memory).
Regarding dependent claim 16: “wherein, when an operation of the reinforcement learning agent is changed to a different deployment setup, the reinforcement learning agent is automatically reconfigured to employ the different reward structure that corresponds to the different deployment setup.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 1. The use of “reinforcement learning agent” is recited at a high-level of generality such that it amounts to no more than a mere instructions to apply the exception using a generic machines that gather data and input into readily available routine into an off the shelf models which is no more than extra solution activity (see MPEP 2106.05 (f)). The receiving of input data and the sending output data amounts to no more than extra solution activity (see MPEP 2106.05(g)); 2106.05(d)(II)(iii) update log, electronic record keeping; 2106.05(d)(II)(iv) storing and retrieving data in memory).
Regarding dependent claim 17: “wherein the task to be performed by the reinforcement learning agent includes determining a network slice configuration for a mobile communication network, wherein, optionally, the plurality of performance-related metrics comprises at least one of a latency observed for a network slice, a throughput observed for a network slice, an elasticity for reconfiguring a network slice, and an explainability regarding a reconfiguration of a network slice.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 1. The use of “reinforcement learning agent” and “network slice” are recited at a high-level of generality such that it amounts to no more than a mere instructions to apply the exception using a generic machines that gather data and input into readily available routine into an off the shelf models which is no more than extra solution activity (see MPEP 2106.05 (f)). The receiving of input data and the sending output data amounts to no more than extra solution activity (see MPEP 2106.05(g)); 2106.05(d)(II)(iii) update log, electronic record keeping; 2106.05(d)(II)(iv) storing and retrieving data in memory).
Regarding dependent claim 18: “wherein the task to be performed by the reinforcement learning agent includes operating a robot, wherein, optionally, the plurality of performance-related metrics comprises at least one of an energy consumption of the robot, a movement accuracy of the robot, a movement speed of the robot, and a safety level provided by the robot.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 1. The use of “reinforcement learning agent” and “operating a robot” are recited at a high-level of generality such that it amounts to no more than a mere instructions to apply the exception using a generic machines that gather data and input into readily available routine into an off the shelf models which is no more than extra solution activity (see MPEP 2106.05 (f)). The receiving of input data and the sending output data amounts to no more than extra solution activity (see MPEP 2106.05(g)); 2106.05(d)(II)(iii) update log, electronic record keeping; 2106.05(d)(II)(iv) storing and retrieving data in memory).
Regarding dependent claim 19: “wherein the task to be performed by the reinforcement learning agent includes operating a robot, wherein, optionally, the plurality of performance-related metrics comprises at least one of an energy consumption of the robot, a movement accuracy of the robot, a movement speed of the robot, and a safety level provided by the robot.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 1. The use of “reinforcement learning agent” and “antenna tilt configuration” are recited at a high-level of generality such that it amounts to no more than a mere instructions to apply the exception using a generic machines that gather data and input into readily available routine into an off the shelf models which is no more than extra solution activity (see MPEP 2106.05 (f)). The receiving of input data and the sending output data amounts to no more than extra solution activity (see MPEP 2106.05(g)); 2106.05(d)(II)(iii) update log, electronic record keeping; 2106.05(d)(II)(iv) storing and retrieving data in memory).
Regarding dependent claim 20: “wherein the task to be performed by the reinforcement learning agent includes determining an offloading level for offloading of computational tasks of one computing device to one or more networked computing devices, wherein, optionally, the plurality of performance-related metrics comprises at least one of an energy consumption of the computing device, a latency observed by the computing device of receiving results of the computational tasks offloaded to the one or more networked computing devices, and a task accuracy achieved by the computing device when offloading the computational tasks to the one or more networked computing devices.” – which continues to recite the abstract idea of making judgments based on human intelligence without significantly more of claim 1. The use of “reinforcement learning agent” and “networked computing devices… computational tasks” are recited at a high-level of generality such that it amounts to no more than a mere instructions to apply the exception using a generic machines that gather data and input into readily available routine into an off the shelf models which is no more than extra solution activity (see MPEP 2106.05 (f)). The receiving of input data and the sending output data amounts to no more than extra solution activity (see MPEP 2106.05(g)); 2106.05(d)(II)(iii) update log, electronic record keeping; 2106.05(d)(II)(iv) storing and retrieving data in memory).
Taken alone, the additional elements of the dependent claims above do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation.
Would Be Allowable Subject Matter
Claims 1-20 would be allowable if rewritten or amended to overcome the rejection(s) under 35 U.S.C. 101 set forth in this Office action.
The following is a statement of reasons for the indication of allowable subject matter:
As per claim 1:
USCUMLIC et al., (US 2021/0409524 A1), part of the prior art made of record, teaches relating values with reinforcement learning of claim 1 through the use of machine learning related values in learning matrices for reinforcement learning in paragraph [0032].
Rickard (US 2004/0024756 A1), part of the prior art made of record, teaches pairwise importance with reinforcement learning of claim 1 through the pairwise importance comparisons and reinforcement architecture paragraphs [0122], [0142], and [0151-0156].
The primary reason for marking of would be allowable subject matter of independent claim 1, in the instant application, is the combination with the inclusion in these claims of the limitations of a method comprising:
“a reinforcement learning agent to perform a task using a reward structure derived from a task-specific definition of metric importances, the method being performed by a computing unit executing a configurator component and comprising: obtaining a definition of metric importances specifying, for a plurality of performance-related metrics associated with the task, pairwise importance values each indicating a relative importance of one metric with respect to another metric of the plurality of performance-related metrics for the task; deriving a reward structure from the definition of metric importances, the reward structure defining, for each of the plurality of performance-related metrics, a reward to be attributed to an action taken by the reinforcement learning agent that yields a positive outcome in the respective performance-related metric; and configuring the reinforcement learning agent to employ the derived reward structure when performing the task.”
The prior art of made of record above neither anticipates nor renders obvious the above-recited combinations. Specifically, though the prior art of made of record does teach the use of relating values for reinforcement learning and the use of pairwise importance it does not teach pairwise importance being used with reinforcement learning for performance related metrics and that these pairwise importance values for performance related metrics associated with a task is used for the rewards for configuring reinforcement learning for the tasks.
Dependent claim(s) 2-20 are marked as would be allowable at least for the reasons recited above as including all of the limitations of the would be allowable independent base claim 1 upon which claims 2-20 depend
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
USCUMLIC et al., (US 2021/0409524 A1), part of the prior art made of record, teaches relating values with reinforcement learning of claim 1 through the use of machine learning related values in learning matrices for reinforcement learning in paragraph [0032].
Rickard (US 2004/0024756 A1), part of the prior art made of record, teaches pairwise importance with reinforcement learning of claim 1 through the pairwise importance comparisons and reinforcement architecture paragraphs [0122], [0142], and [0151-0156].
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHANE D WOOLWINE whose telephone number is (571)272-4138. The examiner can normally be reached M-F 9:30-6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MIRANDA HUANG can be reached at (571) 270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SHANE D. WOOLWINE
Primary Examiner
Art Unit 2124
/SHANE D WOOLWINE/Primary Examiner, Art Unit 2124
Prosecution Insights