DETAILED ACTION
This office action is responsive to the response filed 5/18/2026. The application contains claims 1-2, and 4, all examined and rejected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Objections
Claims 2 objected to because of the following informalities: claim 2 recite “of learning an action value from the state information” instead of “learning an action value from the state information”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1 recites the limitation "the optimal action learning unit". There is insufficient antecedent basis for this limitation in the claim.
Claim 1 recites the limitation " the value function estimation unit". There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2, and 4 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. Claims 1-5 is rejected under 35 USC 101 because the claimed inventions are directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
While independent claims 1, and 2 are each directed to a statutory category, it recites a series of steps, which appears to be directed to an abstract idea (mental process, mathematical concept).
Claims 1-2, and 4 are rejected under 35 U.S.C. § 101 because the instant application is directed to non-patentable subject matter. Specifically, the claims are directed toward at least one judicial exception without reciting additional elements that amount to significantly more than the judicial exception. The rationale for this determination is in accordance with the guidelines of USPTO, applies to all statutory categories, and is explained in detail below.
When considering subject matter eligibility under 35 U.S.C. 101, (1) it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter. If the claim does fall within one of the statutory categories, (2a) it must then be determined whether the claim is directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea), and if so (2b), it must additionally be determined whether the claim is a patent-eligible application of the exception. If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim amounts to significantly more than the abstract idea itself. Examples of abstract ideas include certain methods of organizing human activities; a mental processes; and mathematical concepts, (2019 PEG)
STEP 1.
Per Step 1, the claims are determined to include machine and process, and as in independent Claim 1, 2 and in the therefrom dependent claims. Therefore, the claims are directed to a statutory eligibility category.
At step 2A, prong 1, The invention is directed to identifying features within received data that could be an indication of the probability of occurrence of a machine failure based on analyzed historic data which is akin to Mental Process (see Alice), As such, the claims include an abstract idea. When considering the limitations individually and as a whole the limitations directed to the abstract idea are:
Claim 1:
“estimate a discrete latent variable representing characteristics of features from the state information and the action information as a discrete latent variable estimation”, “learn an optimal action using the state information and the discrete latent variable as a first learning act”, “learn an action value from the state information and the action information as a second learning act”, “identify the discrete latent variable that maximizes the action value using a result” (Mental process, observation, evaluation and judgment); “put the identified discrete latent variable into the second learning act to update the value function as a value function update”, “put the updated value function into the estimation as a latent variable action update”, “repeat the value function update and the latent variable action update to learn the discrete latent variable and the optimal action as a third learning act” (Mathematical concept, Mental process);
“wherein, when z is the discrete latent variable, z′ is a next discrete latent variable, s is a state, s′ is a next state, Qw is an estimate of a Q value parameterized by a vector w, y is a target value, r is a reward in learning, γ is a discount factor, θ is a vector representing parameters of a policy, ϕ is a vector representing parameters of a model of a posterior distribution, (z˜)′ is the next discrete latent variable that has been estimated, fπ is a function that quantifies performance of a policy π, lcvae is a variational lower bound, and a is an action, the estimation includes calculating the latent variable using
PNG
media_image1.png
76
455
media_image1.png
Greyscale
the value function update step includes calculating the target value y using
PNG
media_image2.png
73
376
media_image2.png
Greyscale
the value function update includes updating an action value function by updating a critic that minimizes
PNG
media_image3.png
77
210
media_image3.png
Greyscale
, and the latent variable action update step includes updating a first model by updating an actor and a posterior distribution to maximize
PNG
media_image4.png
98
500
media_image4.png
Greyscale
(Mathematical Concept).
Claim 2:
“estimation step of estimating a discrete latent variable representing characteristics of features of the dataset from the state information and the action information included in the dataset”, “learning an optimal action using the state information and the estimated discrete latent variable as first learning act”, “of learning an action value from the state information and the action information as second learning act”, “identifying the discrete latent variable that maximizes the action value using a result of learning of the first learning act and a result of learning of the second learning act” (Mental process, observation, evaluation and judgment) (Mental process, observation, evaluation and judgment);
“putting the identified discrete latent variable into the second learning act to update the value function as a value function update”, “putting the updated value function into the estimation and the first learning step to update the discrete latent variable and the optimal action as a latent variable action update”, “repeating the value function update and the latent variable action update to learn the discrete latent variable and the optimal action as a third learning act” (Mathematical concept, Mental process);
“wherein, when z is the discrete latent variable, z′ is a next discrete latent variable, s is a state, s′ is a next state, Qw is an estimate of a Q value parameterized by a vector w, y is a target value, r is a reward in learning, γ is a discount factor, θ is a vector representing parameters of a policy, ϕ is a vector representing parameters of a model of a posterior distribution, (z˜)′ is the next discrete latent variable that has been estimated, fπ is a function that quantifies performance of a policy π, lcvae is a variational lower bound, and a is an action, the estimation includes calculating the latent variable using
PNG
media_image1.png
76
455
media_image1.png
Greyscale
the value function update step includes calculating the target value y using
PNG
media_image2.png
73
376
media_image2.png
Greyscale
the value function update includes updating an action value function by updating a critic that minimizes
PNG
media_image3.png
77
210
media_image3.png
Greyscale
, and the latent variable action update step includes updating a first model by updating an actor and a posterior distribution to maximize
PNG
media_image4.png
98
500
media_image4.png
Greyscale
(Mathematical Concept).
.
The claims recites additional elements as
Claim 1:
“a storage medium storing computer-readable instructions; and a processor coupled to the storage medium, the processor executing the computer-readable” (“Using a computer as a tool to perform a mental process”, MPEP 2106.04(a)(2)(III)(C));
“acquire a dataset including state information and action information on which a policy is to be learned as a dataset acquisition act” (insignificant extra-solution activity, MPEP 2106.05(g)).
Claim 2:
“acquiring a dataset including state information and action information on which a policy is to be learned” (insignificant extra-solution activity, MPEP 2106.05(g)).
This judicial exception is not integrated into a practical application. The elements are recited at a high level of generality, i.e. a generic computing system performing generic functions including generic processing of data. Accordingly the additional elements do not integrate the abstract into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Therefore the claims are directed to an abstract idea. (2019 Revised Patent Subject Matter Eligibility Guidance ("2019 PEG"). Thus, under Step 2A of the Mayo framework, the Examiner holds that the claims are directed to concepts identified as abstract.
STEP 2B.
Because the claims include one or more abstract ideas, the examiner now proceeds to Step 2B of the analysis, in which the examiner considers if the claims include individually or as an ordered combination limitations that are "significantly more" than the abstract idea itself. This includes analysis as to whether there is an improvement to either the "computer itself," "another technology," the "technical field," or significantly more than what is "well-understood, routine, or conventional" (WURC) in the related arts.
The instant application includes in Claim 1 additional steps to those deemed to be abstract idea(s).
When taken the steps individually, these steps are:
Claim 1:
“a storage medium storing computer-readable instructions; and a processor coupled to the storage medium, the processor executing the computer-readable” (“Using a computer as a tool to perform a mental process” (“Using a computer as a tool to perform a mental process”, MPEP 2106.05(f)(2));
“acquire a dataset including state information and action information on which a policy is to be learned as a dataset acquisition act” (Well-Understood Routine, Conventional Activity, sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i)).
Claim 2:
“acquiring a dataset including state information and action information on which a policy is to be learned” (Well-Understood Routine, Conventional Activity, sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i)).
In the instant case, Claims 1 and 2 are directed to above mentioned abstract idea. Technical functions such as receiving, and extracting are common and basic functions in computer technology. The individual limitations are recited at a high level and do not provide any specific technology or techniques to perform the functions claimed.
In addition, when the claims are taken as a whole, as an ordered combination, the combination of steps does not add "significantly more" by virtue of considering the steps as a whole, as an ordered combination. The instant application, therefore, still appears only to implement the abstract idea to the particular technological environments using what is well-understood, routine, and conventional in the related arts. The steps are still a combination made to the abstract idea. The additional steps only add to those abstract ideas using well understood and conventional functions, and the claims do not show improved ways of, for example, an unconventional non-routine functions for analyzing model operations or updating the model that could then be pointed to as being "significantly more" than the abstract ideas themselves.
Moreover, Examiner was not able to identify any "unconventional" steps, which, when considered in the ordered combination with the other steps, could have transformed the nature of the abstract idea previously identified. The instant application, therefore, still appears to only implement the abstract ideas to the particular technological environments using what is well-understood, routine, and conventional (WURC) in the related arts.
Further, note that the limitations, in the instant claims, are done by the generically
recited computing devices. The limitations are merely instructions to implement the abstract idea on a computing device that is recited in an abstract level and require no more than a generic computing devices to perform generic functions.
CONCLUSION
It is therefore determined that the instant application not only represents an abstract idea identified as such based on criteria defined by the Courts and on USPTO examination guidelines, but also lacks the capability to bring about "Improvements to another technology or technical field" (Alice), bring about "Improvements to the functioning of the computer itself" (Alice), "Apply the judicial exception with, or by use of, a particular machine" (Bilski), "Effect a transformation or reduction of a particular article to a different state or thing" (Diehr), "Add a specific limitation other than what is well-understood, routine and conventional in the field" (Mayo), "Add unconventional steps that confine the claim to a particular useful application" (Mayo), or contain "Other meaningful limitations beyond generally linking the use of the judicial exception to a particular technological environment" (Alice), transformed a traditionally subjective process performed by humans into a mathematically automated process executed on computers (McRO), or limitations directed to improvements in computer related technology, including claims directed to software (Enfish).
The dependent claims, when considered individually and as a whole, likewise do not provide "significantly more" than the abstract idea for similar reasons as the independent claim.
Claims 4 disclose “the learned policy is executed, not all the first learning acts are activated, the discrete latent variable is estimated according to a situation, and a lower policy corresponding to the estimated discrete latent variable is sequentially selected and activated” (Mental process); It does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea.
The dependent claim which impose additional limitations also fail to claim patent eligible subject matter because the limitations cannot be considered statutory. The dependent claim(s) have been examined individually and in combination with the preceding claims, however they do not cure the deficiencies of claims 1 and 2 ; where all claims are directed to the same abstract idea, "addressing each claim of the asserted patents [is] unnecessary." Content Extraction &. Transmission LLC v, Wells Fargo Bank, Natl Ass'n, 776 F.3d 1343, 1348 (Fed. Cir. 2014). If applicant believes the dependent claims are directed towards patent eligible subject matter, they are invited to point out the specific limitations in the claim that are directed towards patent eligible subject matter. Claims for the other statutory classes are similarly analyzed.
For at least these reasons, the claimed inventions of dependent claims 4, is directed or indirect to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more and are rejected under 35 USC 101.
Examiner notes
Examiner notes that the claims 1-2, and 4 are rejected under 35 USC 101. Currently, there is no art rejection applied as upon review of the evidence at hand, it is hereby concluded that the evidence obtained and made of record, alone or in combination, neither anticipates, reasonably teaches, nor renders obvious the below noted features of applicant's invention as the noted features amount to more than a predictable use of elements in the prior art. The features include “A learning device comprising: a storage medium storing computer-readable instructions; and a processor coupled to the storage medium, the processor executing the computer-readable instructions to: acquire a dataset including state information and action information on which a policy is to be learned as a dataset acquisition act; estimate a discrete latent variable representing characteristics of features from the state information and the action information as a discrete latent variable estimation; learn an optimal action using the state information and the discrete latent variable as a first learning act; learn an action value from the state information and the action information as a second learning act; identify the discrete latent variable that maximizes the action value using a result from the optimal action learning unit and a result from the value function estimation unit as an identification act; put the identified discrete latent variable into the second learning act to update the value function as a value function update; put the updated value function into the estimation and the first learning act to update the discrete latent variable and the optimal action as a latent variable action update; and repeat the value function update and the latent variable action update to learn the discrete latent variable and the optimal action as a third learning act, wherein, when z is the discrete latent variable, z' is a next discrete latent variable, s is a state, s' is a next state, Qw is an estimate of a Q value parameterized by a vector w, y is a target value, r is a reward in learning, y is a discount factor, 0 is a vector representing parameters of a policy, 5 is a vector representing parameters of a model of a posterior distribution, (z~)' is the next discrete latent variable that has been estimated, f' is a function that quantifies performance of a policy r, lcvae is a variational lower bound, and a is an action, the estimation includes calculating the latent variable using
PNG
media_image1.png
76
455
media_image1.png
Greyscale
the value function update step includes calculating the target value y using
PNG
media_image2.png
73
376
media_image2.png
Greyscale
the value function update step includes updating an action value function by updating a critic that minimizes
PNG
media_image3.png
77
210
media_image3.png
Greyscale
, and the latent variable action update step includes updating a first model by updating an actor and a posterior distribution to maximize
PNG
media_image4.png
98
500
media_image4.png
Greyscale
A remarkable art in this area, HIERARCHICAL REINFORCEMENT LEARNING VIA ADVANTAGE-WEIGHTED INFORMATION MAXIMIZATION (hereinafter D1) that teach hierarchical RL with discrete latent option and a gating policy that select option policy using an option value determined based on Q values. It also teaches an advantage weight importance weighting scheme used to estimate a mutual information objective for learning a latent representation based on state and action. However, it does not teach requirement 5 z' = argmax over candidate discrete latent at next state as required (D1 uses a softmax gating policy). D1 also does not teach CVAE-style variational lower bound objective Icvae (s, a;ϴ, ɸ) optimized with an actor as required by claims.
Another remarkable teaching, “Goal-Conditioned Variational Autoencoder Trajectory Primitives with Continuous and Discrete Latent Codes” disclose a framework for modeling demonstrated trajectories using a variational autoencoder (VAE) framework for modeling and generating robot trajectories, including training with a variational lower bound and discrete latent variables. However it does not teach reinforcement learning, compute Q learning target Y, use a double critic minimum, select next discrete latent variable using an argmax over Q function, and weight updates by policy performance.
Another remarkable teaching , “Categorical Reparameterization with Gumbel softmax” [herein D3] that teach training with discrete latent variable. However, D3 does not teach Q learning target y, or double critic min, or argmax next latent selection, and no policy performance weighting.
In addition to the above, the Examiner emphasizes the interrelation of the above distinguishing elements with the remainder of each respective claim element, and further notes that it is the interrelation that truly distinguishes applicant's invention from the evidence at hand.
However, the claim is still rejected under 35 U.S.C. 101 and further evaluation will be provided to the claims upon receiving the applicant’s response.
Response to Arguments
Examiner notes that the claim interpretation under 35 USC 112(f) is withdrawn based on the applicant’s amendments.
Applicant argue that acquiring data, estimating a discrete latent variable representing characteristics of features of the data, learning from the data, manipulating the data, and updating a model," as recited in independent claims 1 and 2. These acts require action by a processor/memory and cannot be practicably applied in the mind.
Examiner respectfully disagrees; the argued limitations are not classified as a mental process in the rejection. The argued limitations is insignificant extra-solution activity, MPEP 2106.05(g) that is Well-Understood Routine, Conventional Activity, sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i). And the usage of the computer is “Using a computer as a tool to perform a mental process”, MPEP 2106.04(a)(2)(III)(C).
Applicant argue that the claims as a whole show’s improvement to the functioning of a computer by integrating its various acts into a practical application of improving a learning method by learning from acquired data, manipulating the data, and updating a model with learned data.
Examiner respectfully disagrees; the argument is a conclusionary statement that does not clarify based on the specifications and as reflected in the claims, what is the state of art at the time of filling of the invention, and how the current claims as clarified in the specifications represent improvement to the functioning of a computer.
The 35 U.S.C. 102(a)(1) rejection is respectfully withdrawn based on the amendments.
Conclusion
The prior art made of record and not relied upon is considered pertinent to the applicant’s disclosure.
“Goal-Conditioned Variational Autoencoder Trajectory Primitives with Continuous and Discrete Latent Codes” disclose a framework for modeling demonstrated trajectories using a variational autoencoder (VAE) framework for modeling and generating robot trajectories, including training with a variational lower bound and discrete latent variables. However it does not teach reinforcement learning, compute Q learning target Y, use a double critic minimum, select next discrete latent variable using an argmax over Q function, and weight updates by policy performance.
Examiner has pointed out particular references contained in the prior arts of record in the body of this action for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and Figures may apply as well. It is respectfully requested from the applicant, in preparing the response, to consider fully the entire references as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior arts or disclosed by the examiner. It is noted that any citation to specific pages, columns, figures, or lines in the prior art references any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331-33, 216 USPQ 1038-39 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)).
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMED ABOU EL SEOUD whose telephone number is (303)297-4285. The examiner can normally be reached Monday-Thursday 9:00am-6:00pm MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMED ABOU EL SEOUD/Primary Examiner, Art Unit 2148
Prosecution Insights