DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on XXXXXXXXXXXXXX has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This action is responsive to the application filed September 17, 2024. Claims 1-20 are pending. Claims 1, 13, and 14 are independent. Claims 1-20 are rejected. Claims 1, 8, 13, 14, 19, and 20 are objected to.
Priority
This application is a continuation of PCT Application No. PCT/US2023/015369, filed March 16, 2023, which claims the benefit of U.S. Provisional Application No. 63/320,905, filed March 17, 2022. For purposes of this action, the effective filing date of claims 1-20 is taken to be March 17, 2022. Each reference applied below predates March 17, 2022, so the prior art status of the applied references does not depend on whether the provisional application provides support under 35 U.S.C. 112(a) for the pending claims.
Information Disclosure Statement
The information disclosure statements submitted: 12/31/2024 and 09/25/2026, have been considered by the Examiner and made of record in the application file.
Claim Objections
Claims 1-20 are objected to for the following reasons.
Claims 1 and 13 are objected to because of the following informality: the clause “performing an ongoing determining of a proper subset of the possible future states of the environment that is to be explored” is not separated from the following clause “determining that one or more inner look ahead search commencement criteria are satisfied” by any punctuation. A semicolon should follow “to be explored”.
Claim 14 is objected to because “computed-implemented system” appears to be a typographical error for “computer-implemented system.”
Claims 8 and 19 are objected to because “exceed a predetermined time length” should read “exceeds a predetermined time length,” since the subject of the verb is “a time”.
Claim 20 is objected to because “the updated current subset of the candidate solutions are sufficient” uses a plural verb with a singular subject. “Is sufficient” is suggested.
The remaining claims are objected to for depending upon an objected to independent claim.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (an abstract idea) without significantly more.
The claims are analyzed under the two-part framework of Alice Corp. Pty. Ltd. v. CLS Bank Int’l, 573 U.S. 208, 217-18 (2014), and Mayo Collaborative Servs. v. Prometheus Labs., Inc., 566 U.S. 66 (2012), as set out in MPEP 2106. Consistent with the memorandum of December 5, 2025 incorporating Ex parte Desjardins, Appeal No. 2024-000567 (PTAB Sept. 26, 2025) (Appeals Review Panel) (precedential), into MPEP 2106.04(d) and 2106.05(a), each claim has been evaluated as a whole, and the limitations Applicant may regard as technical have been considered individually and in combination rather than set aside at a high level of generality.
Step 1 - Statutory Category
Claims 1-12 recite “[a] system comprising: one or more computers and one or more storage devices storing instructions” and are directed to a machine. Claim 13 recites “[o]ne or more computer storage media storing instructions” and is directed to an article of manufacture. The specification states that “[a] computer storage medium is not a propagated signal” (Specification Pub. No.: US 2025/0013883 A1, ¶[0100]), so claim 13 does not read on a transitory signal per se. Claims 14-20 recite “[a] computed-implemented system” comprising “a machine learning subsystem and a search engine”. As explained in the § 112(f) claim interpretation below, those components are construed as one or more computers programmed to perform the disclosed algorithms (Specification, ¶¶[0038], [0089]-[0099], [0106]-[0107]), so claims 14-20 are directed to a machine and are not software per se. Each claim falls within a statutory category (Step 1: Yes). No signal-per-se or software-per-se rejection is made.
Step 2A, Prong One - Independent Claim 1
Claim 1 recites the following limitations, which together describe a planning procedure that can practically be performed in the human mind, with or without pen and paper, and therefore fall within the mental processes grouping of abstract ideas (observation, evaluation, judgment, opinion) (MPEP 2106.04(a)(2), subsection III):
“selecting an action to be performed by the agent in response to the current observation by performing multiple iterations of outer look ahead search to generate an evaluation of possible future states of the environment starting from the current environment state”;
“performing an ongoing determining of a proper subset of the possible future states of the environment that is to be explored”;
“determining that one or more inner look ahead search commencement criteria are satisfied and in response halting the ongoing determining of the proper subset”; and
“then, performing an inner look ahead search of only the proper subset of the possible future states of the environment, and not any remaining possible future states of the environment that are not in the proper subset, until one or more inner look ahead search termination criteria are satisfied”.
A person deciding what to do next in a game, a route, or a task does exactly these things. A chess player looking at a position (the current state) lists a handful of candidate continuations worth considering (the “proper subset” of future states to be explored), stops listing candidates once she judges she has enough or runs short of time (the “commencement criteria” and “halting”), reads out lines only among those candidates and not others (the “inner look ahead search of only the proper subset”), stops each line when it reaches a position she can evaluate (the “termination criteria”), repeats the exercise with a refreshed candidate list, and then picks a move (the “selecting”). Claim 1 places no limit on the number of states, the depth of search, or the manner in which the subset is determined, and it recites no particular data structure or computational technique that a person could not carry out mentally for a problem of modest size. The recitation of generic computers to perform the steps does not take them out of the mental processes grouping. MPEP 2106.04(a)(2), subsection III.C. The same limitations, viewed as the steps of a search algorithm over a state space, also describe a mathematical concept (a sequence of mathematical operations defining which states are evaluated and how an evaluation is produced). MPEP 2106.04(a)(2), subsection I. See also Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350, 1354 (Fed. Cir. 2016) (collecting information, analyzing it, and presenting results of the analysis are abstract mental processes).
Claim 1 recites an abstract idea (Step 2A, Prong One: Yes).
Step 2A, Prong Two - Independent Claim 1
The additional elements of claim 1, beyond the abstract idea, are: (i) “one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform a method”; (ii) “receiving a current observation characterizing a current environment state of the environment”; and (iii) the context that the actions are “to be performed by an agent interacting with an environment to cause the agent to perform a task”.
Element (i) is a generic computing environment recited at a high level of generality and amounts to instructions to implement the abstract idea on a computer. MPEP 2106.05(f). Element (ii) is data gathering needed to perform the abstract idea and is insignificant extra-solution activity. MPEP 2106.05(g). Element (iii) generally links the abstract idea to a field of use (agent control) without placing a meaningful limit on it. MPEP 2106.05(h). Claim 1 selects an action but does not require that any agent perform it, that any control signal be generated, or that anything in any environment change as a result.
The specification asserts that the disclosed techniques reduce computational resources and latency (Specification, ¶¶[0007], [0056]-[0058]). Under MPEP 2106.04(d)(1) and 2106.05(a), as revised to incorporate Desjardins, an asserted improvement is weighed only if the specification describes it in enough detail that one of ordinary skill would recognize it as a technical improvement and the claim itself reflects it. The specification attributes the improvement to a machine learning component: “By incorporating machine learning to determine which particular subsets of different states of the large state tree to explore, the described techniques reduce the amount of computational resources, time, or both consumed by the search process” (Specification, ¶[0007]); see also ¶¶[0048]-[0049] (restart engine 110 and machine learning model 120). Claim 1 recites no machine learning model and no restart engine. It recites only that some subset of states is chosen, that the choosing stops, and that the chosen subset is searched. Whatever efficiency follows from evaluating fewer states is an efficiency of the abstract search procedure itself, not an improvement in the functioning of the computer or in any other technology. See In re Board of Trustees of Leland Stanford Junior University, 989 F.3d 1367 (Fed. Cir. 2021) (an improvement in a mathematical process, even one that produces better results, is not an improvement in technology); SAP America, Inc. v. InvestPic, LLC, 898 F.3d 1161, 1163 (Fed. Cir. 2018). This distinguishes Desjardins, in which the claims reflected a specific change to how a machine learning model is trained that reduced the storage the system required; claim 1 contains no comparable change to how any computer or model operates. The claim is also unlike Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1336 (Fed. Cir. 2016), because it does not recite a specific data structure or computer operation that improves how the computer stores or retrieves data.
Considered individually and in combination, the additional elements do not integrate the abstract idea into a practical application (Step 2A, Prong Two: No). Claim 1 is directed to an abstract idea.
Step 2B - Independent Claim 1
For the reasons given under Prong Two, the generic computers and storage devices are mere instructions to apply the exception (MPEP 2106.05(f)), and the field-of-use context adds nothing inventive (MPEP 2106.05(h)). The re-evaluation of the observation-receiving step under Step 2B confirms it is well-understood, routine, conventional activity: receiving data over a network or from an input is a computer function the courts have recognized as well-understood, routine, and conventional (MPEP 2106.05(d), subsection II, item i), and the specification describes receiving observations only generically (Specification, ¶¶[0018]-[0021], [0060]). The ordered combination adds nothing that is not already present when the elements are considered individually. Claim 1 does not recite significantly more than the abstract idea (Step 2B: No).
Independent Claim 13
Claim 13 recites “[o]ne or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations” that are word-for-word the same as the method steps of claim 1, including “performing an ongoing determining of a proper subset of the possible future states of the environment that is to be explored”, “determining that one or more inner look ahead search commencement criteria are satisfied and in response halting the ongoing determining of the proper subset”, and “performing an inner look ahead search of only the proper subset of the possible future states of the environment”. The analysis of claim 1 at Step 2A, Prongs One and Two, and Step 2B applies equally. The computer storage media are a generic computer component recited to implement the abstract idea. MPEP 2106.05(f). Claim 13 is ineligible.
Independent Claim 14
Step 2A, Prong One. Claim 14 recites “solving a search problem by searching through a search space comprising a plurality of candidate solutions”, in each of multiple outer search iterations identifying “one or more new candidate solutions that are to be searched in the current outer search iteration” based on “data specifying a current subset of the plurality of candidate solutions that have already been searched in previous outer search iterations”, and “identify[ing] the respective candidate final solution to the search problem by performing a look ahead search of possible continuing solutions that start from at least the one or more new candidate solutions . . . until one or more look ahead search termination criteria are satisfied, and selecting, from the plurality of candidate solutions, and as the respective candidate final solution, a selected candidate solution as a result of performing the look ahead search”. Reviewing which candidates have already been tried, proposing new candidates, working each candidate forward, and choosing the best one is a mental process of evaluation and judgment (MPEP 2106.04(a)(2), subsection III), and as a formal search procedure over a solution space it is also a mathematical concept (MPEP 2106.04(a)(2), subsection I). The specification’s own examples of the “search problem” are combinatorial optimization problems such as assignment, packing, scheduling, and network flow problems (Specification, ¶[0088]), which are classic mathematical problems.
Step 2A, Prong Two. The additional elements are the “machine learning subsystem” that is “configured to . . . process the machine learning subsystem input to generate a machine learning subsystem output”, and the “search engine”. The machine learning subsystem is recited only by its input and output. The claim does not specify any model architecture, any training, or any change to how a model operates; the specification confirms that the model can be “any appropriate architecture,” including a neural network, a support vector machine, or a decision forest (Specification, ¶[0050]). Using a generic, trained machine learning model as a tool to propose candidates for an otherwise abstract search is an instruction to apply the exception with a generic computer tool. MPEP 2106.05(f); Recentive Analytics, Inc. v. Fox Corp., 134 F.4th 1205 (Fed. Cir. 2025) (claims that apply established machine learning methods to a new data environment, without disclosing improvements to the machine learning models themselves, are directed to an abstract idea). Unlike the claims in Desjardins, claim 14 does not recite any change to how the machine learning model is trained or how it operates. The search engine is a generic computer function for carrying out the recited search. Claim 14 does not recite control of any physical system. The additional elements do not integrate the abstract idea into a practical application.
Step 2B. For the same reasons, the machine learning subsystem and search engine, alone and in combination, are generic computer components performing their ordinary functions and do not supply an inventive concept. Claim 14 is ineligible.
Dependent Claims
Each dependent claim has been analyzed individually. Claims are grouped below only where the analysis is identical, and every claim number is listed.
Claim 2 adds “wherein the agent is a mechanical agent, the environment is a real-world environment, and the current observation comprises data from one or more sensors configured to sense the real-world environment”. Identifying the agent as mechanical and the environment as real-world is a field-of-use limitation (MPEP 2106.05(h)), and obtaining the observation from sensors is data gathering (MPEP 2106.05(g)). Claim 2 still requires only that an action be selected. It does not require that the mechanical agent be controlled to perform the action, so the claim does not apply the abstract idea to effect a particular physical result. Claim 2 is ineligible.
Claim 3 adds “wherein the agent is a computer program, the environment is a real-world environment, and the observation comprises data from one or more sensors configured to sense the real-world environment, and the agent performs the task by providing instructions specifying the selected action in the real-world environment”. The sensor data is data gathering (MPEP 2106.05(g)). “[P]roviding instructions specifying the selected action” is the output of the result of the abstract idea. The claim does not say to what the instructions are provided or require that anything act on them, and outputting the result of a mental process is insignificant post-solution activity (MPEP 2106.05(g)). The computer-program agent is a generic computer implementation (MPEP 2106.05(f)). Claim 3 is ineligible.
Claims 4 and 5 add “maintaining data that specifies a current subset of possible future states . . . that have already been visited”, “determining one or more unvisited, possible future states, beginning from one or more terminal states of any of the previous outer look ahead search iterations”, “adding the one or more unvisited, possible future states to the current subset” (claim 4), and that the unvisited states are those the environment “will transition into from a terminal state . . . in response to the agent performing a valid action” (claim 5). Keeping track of which positions have already been considered and extending the list from where the last line of analysis ended are further mental steps of the same planning process (MPEP 2106.04(a)(2), subsection III). “[M]aintaining data” on a generic storage device is a generic computer function (MPEP 2106.05(f)). Claims 4 and 5 are ineligible.
Claims 6 and 7 add determining that the proper subset “is sufficient in terms of generating the evaluation” (claim 6) and doing so by “evaluating, by computing a stopping function, the proper subset . . . to generate a binary classification result” (claim 7). Judging whether enough candidates have been gathered is a mental evaluation, and computing a function that returns a yes/no result is a mathematical calculation (MPEP 2106.04(a)(2), subsections I and III). The stopping function is not tied to any particular computer operation. Claims 6 and 7 are ineligible.
Claim 8 adds “determining that a time spent on determining the one or more unvisited, possible future states exceed a predetermined time length”. Checking whether a time budget has been used up is a mental observation and comparison (MPEP 2106.04(a)(2), subsection III). Claim 8 is ineligible.
Claim 9 adds “traversing a state tree starting from a root node of the state tree representing the current environment state until reaching a leaf node in the state tree”. Following a line of play from the current position until reaching a position not yet expanded is the same mental planning process, and a tree is an abstract organization of the information being evaluated. Claim 9 is ineligible.
Claims 10 and 11 add selecting the action “using statistics generated during the multiple iterations of outer look ahead search for the root node”, where the statistics are “a respective action score . . . which action score specifies a likelihood that the agent will complete the task if the action is performed” and the action with “a highest action score” is selected (claim 10), or “a respective visit count” and the action with “a highest visit count” is selected (claim 11). Tallying scores or counts and choosing the maximum is a mathematical calculation and a mental judgment (MPEP 2106.04(a)(2), subsections I and III). Claims 10 and 11 are ineligible.
Claim 12 adds “performing a Monte Carlo tree search guided by outputs of a neural network, wherein the neural network is configured to receive an input observation characterizing a state of the environment and to process the input observation in accordance with network parameters to generate a network output”. Monte Carlo tree search is a mathematical algorithm, and processing an input through a neural network in accordance with network parameters is a series of mathematical calculations (MPEP 2106.04(a)(2), subsection I). To the extent the neural network is treated as an additional element, it is recited at a high level of generality, with no architecture, training, or modification to how the network operates, and it is used as a tool to perform the abstract search. MPEP 2106.05(f); Recentive Analytics, 134 F.4th 1205. Claim 12 is ineligible.
Claims 15, 16, and 17 add maintaining and updating the set of already-searched candidate solutions (claim 15), “generating an evaluation result of the one or more new candidate solutions” and using it to select the selected candidate solution (claim 16), and identifying “a final solution to the search problem from the respective candidate final solutions” (claim 17). Each is a further step of recording, evaluating, and choosing among candidates, which are mental processes and mathematical concepts (MPEP 2106.04(a)(2), subsections I and III). Claims 15, 16, and 17 are ineligible.
Claims 18, 19, and 20 add “determining that one or more look ahead search commencement criteria are satisfied” (claim 18), where the criteria are that “a time spent on processing the machine learning system input . . . exceed[s] a predetermined time length” (claim 19), or that the candidate solutions “are sufficient”, determined by “computing a stopping function” that produces “a binary classification result that specifies whether the respective candidate final solution is a valid solution to the search problem” (claim 20). Deciding when to stop generating candidates, by elapsed time or by testing whether a valid answer has been found, is a mental judgment and a mathematical calculation (MPEP 2106.04(a)(2), subsections I and III). Claims 18, 19, and 20 are ineligible.
None of dependent claims 2-12 and 15-20 recites an additional element that, alone or in combination with the elements of its parent claim, integrates the abstract idea into a practical application or amounts to significantly more.
Suggested Amendment
The rejection would likely be overcome by amendments that (1) tie the claims to the improvement the specification describes and (2) apply the result to control a physical system. For example, claims 1 and 13 could be amended to recite that the ongoing determining of the proper subset is performed by processing, using a trained machine learning model, an input that includes data specifying the possible future states already visited in previous outer look ahead search iterations, to generate an output specifying one or more unvisited possible future states beginning from the terminal states at which the previous iterations terminated (Specification, ¶¶[0048]-[0049], [0066]-[0067], [0073]), and to recite that the system generates a control signal and causes a mechanical agent (for example, a robot or autonomous vehicle) to perform the selected action in the real-world environment (Specification, ¶¶[0017], [0022]-[0023], [0040]). Claim 14 would need a comparable limitation applying the identified final solution to control a physical system or process described in the specification. Any amendment should be accompanied by an explanation, with citations to the specification, of how the claimed arrangement reduces the computational resources consumed in selecting the action (Specification, ¶¶[0056]-[0058]). Applicant may also submit a declaration under 37 CFR 1.132 with objective evidence of the asserted technological improvement.
Claim Rejections - 35 USC §112(b)
The following is a quotation of 35 U.S.C. §112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 6, 8, 10, 19, and 20 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention. Claim 7 is rejected for depending upon rejected parent claim 6.
Claim 6 - Term of Degree
Claim 6 recites “determining that the proper subset of the possible future states of the environment that is to be explored is sufficient in terms of generating the evaluation of the possible future states of the environment starting from the current environment state”. “Sufficient” is a term of degree. Neither claim 6 nor the specification supplies a standard for measuring when a subset is “sufficient in terms of generating the evaluation.” The specification states only that the system “can evaluate the proper subset . . . by computing a stopping function to generate a binary classification result” and that the stopping function “can be implemented as a trained classifier model, or can be realized deterministically as a parametric equation” (Specification, ¶[0077]), without identifying what the classifier or equation measures or what threshold separates a sufficient subset from an insufficient one. One of ordinary skill therefore cannot determine with reasonable certainty which subsets meet the limitation. Nautilus, Inc. v. Biosig Instruments, Inc., 572 U.S. 898, 901 (2014); Interval Licensing LLC v. AOL, Inc., 766 F.3d 1364 (Fed. Cir. 2014); MPEP 2173.05(b). For examination, claim 6 is interpreted as met by any determination, according to any criterion, that the subset is adequate to proceed to the inner search. Claim 7, which requires computing a stopping function that produces a binary result, supplies a determinable test and is not included in this rejection.
Suggested fix: incorporate the subject matter of claim 7 into claim 6 or recite the measurable condition that defines sufficiency (for example, that a value estimate derived from the proper subset satisfies a threshold).
Claim 8 - Antecedent Basis
Claim 8 depends from claim 1 and recites “determining that a time spent on determining the one or more unvisited, possible future states exceed a predetermined time length”. The phrase “the one or more unvisited, possible future states” lacks antecedent basis. Claim 1 does not introduce any unvisited possible future states; that term first appears in claim 4. MPEP 2173.05(e). It is unclear whether claim 8 requires the determining of unvisited states recited in claim 4 or measures the time spent on the “ongoing determining of a proper subset” recited in claim 1. For examination, claim 8 is interpreted as measuring the time spent on the ongoing determining of the proper subset.
Suggested fix: make claim 8 depend from claim 4, or replace the phrase with “the ongoing determining of the proper subset.”
Claim 10 - Inconsistent Recitation of Performed Actions
Claim 10 recites statistics for “each outgoing edge connected to the root node that represents a corresponding action that was performed by the agent in response to the current observation”. Claim 10 and its parent claims are directed to selecting, in response to the current observation, the action the agent is to perform. Read literally, “was performed by the agent in response to the current observation” requires that the agent has already performed, in response to the same observation, each action represented by an outgoing edge, which conflicts with the selection of a single action to be performed in response to that observation. The specification uses the same phrase (Specification, ¶[0083]) but also explains that the root-node statistics are compiled during the look ahead search, and it describes the corresponding visit count as representing “a number of times that the action has been considered by the agent as an action that might be performed in response to the current observation” (Specification, ¶[0084]). Parallel claim 11 likewise uses the phrase “considered by the agent as an action that might be performed”. The metes and bounds of claim 10 are unclear. For examination, claim 10 is interpreted consistently with claim 11 as reciting actions evaluated during the look ahead search.
Suggested fix: replace “was performed by the agent” with “was considered, during the multiple iterations of outer look ahead search, as an action to be performed by the agent.”
Claims 19 and 20 - Antecedent Basis
Claims 19 and 20 each depend from claim 14 and each recite “the one or more look ahead search commencement criteria”. Claim 14 does not recite any commencement criteria; the term first appears in claim 18. The phrase lacks antecedent basis. MPEP 2173.05(e).
Claim 19 further recites “the machine learning system input”. Claim 14 recites “a machine learning subsystem input”, not a machine learning system input. It is unclear whether the two terms refer to the same input.
Claim 20 further recites “the updated current subset of the candidate solutions” (twice). Claim 14 recites “a current subset of the plurality of candidate solutions” but never recites updating that subset or an “updated” current subset. Updating is recited only in claim 15, from which claim 20 does not depend. The phrase lacks antecedent basis.
For examination, claims 19 and 20 are interpreted as depending from claim 18, “the machine learning system input” is interpreted as the machine learning subsystem input of claim 14, and “the updated current subset” is interpreted as the current subset of claim 14 together with the new candidate solutions generated in the current outer search iteration.
Suggested fix: amend claims 19 and 20 to depend from claim 18 (and, for claim 20, from a claim that recites updating the current subset, such as claim 15), and change “machine learning system input” in claim 19 to “machine learning subsystem input.”
Claims Reviewed Without a § 112(b) Rejection
Claims 1-5, 7, 9, and 11-18 have been reviewed for definiteness, and no § 112(b) rejection is made. The informalities noted in the claim objections above (claims 1 and 13 punctuation; claim 14 “computed-implemented”) do not render the claims indefinite. In claim 4, “determining the proper subset” is understood to refer to the “ongoing determining of a proper subset” recited in claim 1.
35 U.S.C. 112(f) – Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f):
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) except as otherwise indicated in an Office action.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f).
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) because the claim limitations use a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function, and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
“a machine learning subsystem” that is “configured to identify a proper subset of the plurality of candidate solutions” and, in each outer search iteration, to “receive a machine learning subsystem input”, “process the machine learning subsystem input to generate a machine learning subsystem output”, and “search candidate solutions to identify a respective candidate final solution using the search engine” (claim 14; also claims 15 and 18-20); and
“a search engine” that is “configured to, in each current outer search iteration: identify the respective candidate final solution to the search problem by performing a look ahead search” (claim 14; also claims 17-20).
“Subsystem” and “engine” are nonce terms that do not connote a definite structure to one of ordinary skill; the modifiers “machine learning” and “search” describe what the components do, not what they are. Williamson v. Citrix Online, LLC, 792 F.3d 1339, 1350 (Fed. Cir. 2015) (en banc in relevant part); MPEP 2181, subsection I. Because these limitations are computer-implemented functions, the corresponding structure is a general-purpose computer programmed with the algorithm disclosed for performing each function. MPEP 2181, subsection II.B. The specification discloses the corresponding structure as one or more computers (Specification, ¶¶[0038], [0101], [0106]-[0107]) programmed to perform the algorithm of FIG. 5 and ¶¶[0089]-[0099], using a machine learning model such as a neural network, support vector machine, or decision forest (Specification, ¶[0050]) for the machine learning subsystem, and a search-based optimization algorithm such as a depth-first, breadth-first, greedy, A*, alpha-beta, or Monte Carlo tree search for the search engine (Specification, ¶¶[0052]-[0053], [0097]). That disclosure is adequate, and no rejection under 35 U.S.C. 112(a) or (b) is made on this basis.
If Applicant does not intend these limitations to be interpreted under 35 U.S.C. 112(f), Applicant may amend the claim limitations to avoid interpretation under 35 U.S.C. 112(f) (for example, by reciting one or more computers and one or more storage devices storing instructions that cause the computers to perform the recited operations, as in claim 1), or present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function.
No limitation of claims 1-13 is interpreted under 35 U.S.C. 112(f). Those claims recite “one or more computers and one or more storage devices storing instructions” (claims 1-12) or “computer storage media storing instructions” executed by “one or more computers” (claim 13) as the structure performing every recited operation.
Claim Rejections - 35 U.S.C. § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-9, 11, and 13-20 are rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser (US 2023/0073326 A1) in view of Caldwell (US 2023/0041975 A1).
Claim 1 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 1
Claim 1 recites “A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform a method for selecting, from a set of actions, actions to be performed by an agent interacting with an environment to cause the agent to perform a task”. Schrittwieser teaches this feature. Schrittwieser describes “methods for selecting, from a set of actions, actions to be performed by an agent interacting with an environment to cause the agent to perform a task” (Schrittwieser, ¶[0009]), implemented in a “system implemented as computer programs on one or more computers in one or more locations” (Schrittwieser, ¶[0029; Fig. 1]), with embodiments including “corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods” (Schrittwieser, ¶[0011]).
Claim 1 recites “receiving a current observation characterizing a current environment state of the environment”. Schrittwieser teaches this feature: “The system receives a current observation (e.g., an image or a video frame) characterizing a current environment state of the environment (202)” (Schrittwieser, ¶[0089]).
Claim 1 recites “selecting an action to be performed by the agent in response to the current observation by performing multiple iterations of outer look ahead search to generate an evaluation of possible future states of the environment starting from the current environment state”. Schrittwieser teaches this feature: “The system then performs multiple planning iterations to generate plan data that indicates a respective value to performing the task of the agent performing each of the set of actions in the environment and starting from the current environment state. Each planning iteration generally involves performing a look ahead search, e.g., a Monte-Carlo tree search” (Schrittwieser, ¶[0091]), and “After performing the multiple planning iterations as described above to generate plan data, the system proceeds to select an action to be performed by the agent in response to the current observation using the plan data (414)” (Schrittwieser, ¶[0118]). Schrittwieser’s planning iterations are the claimed outer look ahead search iterations, and its plan data, which indicates the value of the future states reached from the current state, is the claimed evaluation of possible future states.
Claim 1 recites “wherein performing the multiple iterations of outer look ahead search comprises, in each outer look ahead search iteration: performing an ongoing determining of a proper subset of the possible future states of the environment that is to be explored”. Schrittwieser teaches this feature: “The system samples a proper subset of the set of actions (408). The system can do this by generating a sampling distribution from the score distribution, and then sampling a fixed number of samples from the sampling distribution” (Schrittwieser, ¶[0114]), and “For each sampled action, the system adds, to the state tree, a respective outgoing edge from the leaf node that represents the sampled action” (Schrittwieser, ¶[0115]). Schrittwieser explains that “the planning engine 120 uses the action sampling engine 160 to select a proper subset of the actions in the set of possible actions and to perform planning by using only the actions in the proper subset” (Schrittwieser, ¶[0042]). As explained in the common claim interpretation above, the sampled actions determine which successor states are added to the tree to be explored, so drawing the fixed number of samples one after another, and adding the corresponding states, is an ongoing determining of a proper subset of the possible future states that is to be explored.
Claim 1 recites “determining that one or more inner look ahead search commencement criteria are satisfied and in response halting the ongoing determining of the proper subset”. Primary prior-art reference Schrittwieser does not explicitly teach “determining that one or more inner look ahead search commencement criteria are satisfied and in response halting the ongoing determining of the proper subset.” Schrittwieser draws “a fixed number of samples” (Schrittwieser, ¶[0114]), so its sampling ends once that number is drawn, but it does not describe a determination that a criterion has been met that halts the determination in response. However, secondary prior-art reference Caldwell teaches this feature as follows: “For example, the tree search may continue to alternately determine action nodes and prediction nodes until a termination event, such as determining a predefined number of action nodes, a time limit for the computation, a computation or storage limit, a displacement along the route is achieved, a number of time steps has been reached (e.g., actions up to a time horizon have been determined), determining an action that reaches a target position, determining that a path to a target position is less than a cost threshold, or the like” (Caldwell, ¶[0116]). Caldwell determines the vehicle path through the generated nodes “as the sets of nodes are generated and/or termination event occurs” (Caldwell, ¶[0117]), and “[t]he guidance system may conduct a search for the path from the root node to a last layer of the data structure 332” (Caldwell, ¶[0118]). Caldwell’s alternating generation of action nodes and prediction nodes (predicted future environment states) is an ongoing determination of the future states to be explored. Its termination event is a criterion whose satisfaction is determined, and node generation stops in response. The path determination performed when the termination event occurs is a search over the nodes so generated. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Schrittwieser so that, in each planning iteration, the ongoing sampling and addition of states continues until a determined termination event of the kind Caldwell teaches (a predefined number of nodes, a time limit for the computation, or a computation limit) and is halted in response, before the search proceeds, because Schrittwieser seeks to select actions “with reduced latency and reduced consumption of computational resources” (Schrittwieser, ¶[0014]) and Caldwell’s termination events include “a time limit for the computation, a computation or storage limit” (Caldwell, ¶[0116]), which on their face cap the resources spent generating nodes, in a system that seeks to “reduce the computational load for determining a path for controlling an autonomous vehicle” (Caldwell, ¶[0022]). The rationale is MPEP 2143(I)(D), with (G): Schrittwieser’s planner is a known method ready for the improvement of an explicitly bounded state-determination phase, Caldwell supplies the known technique and states the reason for it, and the predictable result is a planner whose determination of the subset stops upon a detected count, time, or computation criterion. KSR, 550 U.S. at 416.
Claim 1 recites “and then, performing an inner look ahead search of only the proper subset of the possible future states of the environment, and not any remaining possible future states of the environment that are not in the proper subset, until one or more inner look ahead search termination criteria are satisfied”. Schrittwieser teaches this feature: “At each planning iteration, the system traverses, using statistics for node-edge pairs in the state tree, the state tree starting from a root node of the state tree representing the current environment state until reaching a leaf node in the state tree (404)” (Schrittwieser, ¶[0112]), and “the system begins the look ahead search starting from a root node of the state tree (which corresponds to the hidden state generated at step 202) and continues the look ahead search until a possible future state that satisfies termination criteria is encountered. For example, the look ahead search may be a Monte-Carlo tree search and the criteria may be that the future state is represented by a leaf node in the state tree” (Schrittwieser, ¶[0092]). The traversal follows only edges present in the state tree, and Schrittwieser expands that tree “using only the sampled actions, i.e., rather than using every possible action in the entire action space” (Schrittwieser, ¶[0014]). The search therefore visits only the states in the proper subset and not the remaining possible future states, and it stops when the leaf-node termination criterion is met. As explained in the common claim interpretation, this search follows the determination of the subset. Caldwell likewise conducts its path search after node generation terminates and only over “the different sets of nodes from the root node to an action node in a deepest layer of the data structure 332” (Caldwell, ¶[0118]). No secondary reference is needed to meet this limitation.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 2
Claim 2 recites “wherein the agent is a mechanical agent, the environment is a real-world environment, and the current observation comprises data from one or more sensors configured to sense the real-world environment”. Schrittwieser teaches this feature: “In some implementations, the environment 102 is a real-world environment and the agent 108 is a mechanical agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle navigating through the environment” (Schrittwieser, ¶[0058]), and “the observations 104 may include, e.g., one or more of images, object position data, and sensor data to capture observations as the agent as it interacts with the environment, for example sensor data from an image, distance, or position sensor or from an actuator” (Schrittwieser, ¶[0059]). No secondary reference is needed to meet this limitation. The remaining limitations are met as set forth for claim 1.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 3
Claim 3 recites “wherein the agent is a computer program, the environment is a real-world environment, and the observation comprises data from one or more sensors configured to sense the real-world environment, and the agent performs the task by providing instructions specifying the selected action in the real-world environment”. Schrittwieser teaches this feature. Schrittwieser describes an electronic agent whose observations “may include data from one or more sensors monitoring part of a plant or service facility such as current, voltage, power, temperature and other sensors” (Schrittwieser, ¶[0064]), and explains that “the agent may control actions in a real-world environment including items of equipment, for example in a data center, in a power/water distribution system, or in a manufacturing plant or service facility,” where “[t]he actions may include actions controlling or imposing operating conditions on items of equipment of the plant/facility, and/or actions that result in changes to settings in the operation of the plant/facility e.g. to adjust or turn on/off components of the plant/facility” (Schrittwieser, ¶[0068]). Schrittwieser also describes an agent in a real-world environment that “manages distribution of tasks across computing resources,” where “the actions may include assigning tasks to particular computing resources” (Schrittwieser, ¶[0076]). An electronic agent that imposes operating conditions, changes settings, or assigns tasks is a computer program, and imposing a setting or assigning a task is providing instructions that specify the selected action in the real-world environment. No secondary reference is needed to meet this limitation. The remaining limitations are met as set forth for claim 1.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 4
Claim 4 recites “maintaining data that specifies a current subset of possible future states of the environment that have already been visited in previous outer look ahead search iterations”. Schrittwieser teaches this feature: “The system 100 can maintain the plan data 122 at a memory device accessible to the system 100. While logically described as a tree, the plan data 122 generated by using the planning engine 120 may be represented by any of a variety of convenient data structures, e.g., as multiple triples or as an adjacency list” (Schrittwieser, ¶[0034]). The maintained state tree contains the nodes (states) added and traversed in earlier planning iterations.
Claim 4 recites “determining one or more unvisited, possible future states, beginning from one or more terminal states of any of the previous outer look ahead search iterations at which the previous outer look ahead search iteration terminated”. Schrittwieser teaches this feature. Each search ends at a leaf node, “a node in the state tree that has no child nodes, i.e., is not connected to any other nodes by an outgoing edge” (Schrittwieser, ¶[0112]; see ¶[0092]), and the leaf is then expanded: “For each sampled action, the system adds, to the state tree, a respective outgoing edge from the leaf node that represents the sampled action” (Schrittwieser, ¶[0115]), with the dynamics model generating “a hidden state corresponding to a predicted next environment state that the environment would transition into if the agent performed the input action when the environment is in the input environment state” (Schrittwieser, ¶[0093]). The leaf at which the previous search terminated is the terminal state of the previous iteration, and the predicted next states reached by the sampled actions are unvisited possible future states determined beginning from it.
Claim 4 recites “adding the one or more unvisited, possible future states to the current subset of possible future states of the environment that have already been visited in the previous outer look ahead search iterations”. Schrittwieser teaches this feature: “The system updates the state tree based on the sampled actions (410). For each sampled action, the system adds, to the state tree, a respective outgoing edge from the leaf node that represents the sampled action” (Schrittwieser, ¶[0115]). No secondary reference is needed to meet the limitations added by claim 4. The remaining limitations are met as set forth for claim 1.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 5
Claim 5 recites “wherein the one or more unvisited, possible future states comprise future states that the environment will transition into from a terminal state of any of the previous outer look ahead search iterations in response to the agent performing a valid action in the set of actions when the environment is in the terminal state”. Schrittwieser teaches this feature: “to expand the leaf node, the system may add a new edge to the state tree for an action that is a possible (or valid) action to be performed by the agent (referred to below as ‘an input action’) in response to a leaf environment state represented by the leaf node” (Schrittwieser, ¶[0092]), and the dynamics model outputs “a hidden state corresponding to a predicted next environment state that the environment would transition into if the agent performed the input action when the environment is in the input environment state” (Schrittwieser, ¶[0093]). No secondary reference is needed to meet this limitation. The remaining limitations are met as set forth for claims 1 and 4.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 6
Claim 6 recites “wherein determining that the one or more inner look ahead search commencement criteria are satisfied comprises: determining that the proper subset of the possible future states of the environment that is to be explored is sufficient in terms of generating the evaluation of the possible future states of the environment starting from the current environment state”. Claim 6 is interpreted as set forth in the § 112(b) rejection above. Primary prior-art reference Schrittwieser does not explicitly teach “determining that the proper subset of the possible future states of the environment that is to be explored is sufficient in terms of generating the evaluation,” however, secondary prior-art reference Caldwell teaches this feature as follows: node generation continues “until a termination event, such as . . . determining an action that reaches a target position, determining that a path to a target position is less than a cost threshold, or the like” (Caldwell, ¶[0116]). Caldwell stops generating nodes when the nodes already generated contain an action reaching the target or a path to the target whose cost is below a threshold, that is, when the generated set is sufficient to produce the evaluation the search is meant to produce. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to use, as the criterion for halting Schrittwieser’s per-iteration determination of the subset, Caldwell’s test of whether the states already determined are adequate to yield an acceptable result, because Caldwell uses that test to stop spending computation on node generation once further generation is unnecessary (Caldwell, ¶¶[0022], [0116]), which serves Schrittwieser’s goal of reduced latency and computation (Schrittwieser, ¶[0014]). The rationale is MPEP 2143(I)(D), with (G), and the predictable result is a planner that halts state generation once the generated states suffice. KSR, 550 U.S. at 416. The remaining limitations are met as set forth for claim 1.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 7
Claim 7 recites “evaluating, by computing a stopping function, the proper subset of the possible future states of the environment, to generate a binary classification result that specifies whether the proper subset of the possible future states of the environment is sufficient”. Primary prior-art reference Schrittwieser does not explicitly teach “evaluating, by computing a stopping function, the proper subset . . . to generate a binary classification result,” however, secondary prior-art reference Caldwell teaches this feature as follows: the termination event includes “determining that a path to a target position is less than a cost threshold” (Caldwell, ¶[0116]). Comparing the cost of a path formed from the generated nodes with a threshold is computing a function over the generated states whose output is one of two values: terminate (the generated set suffices) or continue. Caldwell also applies a two-valued feasibility test to a path before searching: “before conducting a full search for a path, the guidance component may comprise determining if a previous path and/or previous connection determined by the guidance system is feasible (e.g., satisfies current constraints such as velocity, maximum steering angle, and/or boundaries; is impact-free; has a cost that is less than a cost threshold)” (Caldwell, ¶[0119]). That test is applied to a previously determined path, and it is cited here only to show that Caldwell uses yes/no tests on candidate paths. The specification confirms that a stopping function includes one “realized deterministically as a parametric equation” (Specification, ¶[0077]). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to implement the sufficiency determination applied to Schrittwieser for claim 6 as Caldwell’s cost-threshold test producing a yes/no result, because a binary test is the direct way to decide whether to continue or stop generation, and Caldwell uses such a test as a termination event (Caldwell, ¶[0116]; see ¶[0119]). The rationale is MPEP 2143(I)(D), with (G), and the predictable result is a yes/no stopping test applied to the generated subset. KSR, 550 U.S. at 416. The remaining limitations are met as set forth for claims 1 and 6.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 8
Claim 8 recites “determining that a time spent on determining the one or more unvisited, possible future states exceed a predetermined time length”. Claim 8 is interpreted as set forth in the § 112(b) rejection above. Primary prior-art reference Schrittwieser does not explicitly teach “determining that a time spent on determining the one or more unvisited, possible future states exceed a predetermined time length,” however, secondary prior-art reference Caldwell teaches this feature as follows: node generation continues “until a termination event, such as determining a predefined number of action nodes, a time limit for the computation, a computation or storage limit” (Caldwell, ¶[0116]). A time limit for the computation that generates the nodes is a predetermined time length, and the termination event occurs when the time spent generating nodes exceeds it. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to halt Schrittwieser’s per-iteration determination of new states when the time spent on it exceeds a predetermined limit, as Caldwell does, because an agent controlled in real time must select its action within a bounded time. Caldwell expressly provides “a time limit for the computation” as an event that ends node generation (Caldwell, ¶[0116]), and Schrittwieser seeks reduced latency (Schrittwieser, ¶[0014]). The rationale is MPEP 2143(I)(D), with (G), and the predictable result is a time-bounded state-generation phase. KSR, 550 U.S. at 416. The remaining limitations are met as set forth for claim 1.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 9
Claim 9 recites “traversing a state tree starting from a root node of the state tree representing the current environment state until reaching a leaf node in the state tree”. Schrittwieser teaches this feature: “At each planning iteration, the system traverses, using statistics for node-edge pairs in the state tree, the state tree starting from a root node of the state tree representing the current environment state until reaching a leaf node in the state tree (404)” (Schrittwieser, ¶[0112]). No secondary reference is needed to meet this limitation. The remaining limitations are met as set forth for claim 1.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 11
Claim 11 recites “selecting the action to be performed by the agent in response to the current observation using statistics generated during the multiple iterations of outer look ahead search for the root node of the state tree that represents the current observation”. Schrittwieser teaches this feature (Schrittwieser, ¶[0037], quoted above for claim 10).
Claim 11 recites “wherein the statistics comprise, for each outgoing edge connected to the root node that represents a corresponding action that was considered by the agent as an action that might be performed in response to the current observation, a respective visit count for the action represented by the edge, which visit count represents a number of times that the action has been considered by the agent as an action that might be performed in response to the current observation”. Schrittwieser teaches this feature: “the plan data 122 can include, for each outgoing edge of a root node of the state tree, . . . (ii) a visit count N for the action represented by the edge that represents a number of times that the action was selected during the plurality of planning iterations” (Schrittwieser, ¶[0032]). An action selected during a planning iteration is an action considered as one that might be performed.
Claim 11 recites “wherein selecting the action to be performed by the agent comprises selecting the action that has a highest visit count”. Schrittwieser teaches this feature: “the action corresponding to the outgoing edge from the root node that has the highest visit count in the plan data, may be selected as the actual action 110 to be performed by the agent in response to the current observation” (Schrittwieser, ¶[0038]). No secondary reference is needed to meet the limitations added by claim 11. The remaining limitations are met as set forth for claims 1 and 9.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 13
Claim 13 is the storage-media counterpart of claim 1. Claim 13 recites “One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for selecting, from a set of actions, actions to be performed by an agent interacting with an environment to cause the agent to perform a task”. Schrittwieser teaches this feature: embodiments include “computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods” (Schrittwieser, ¶[0011]), executed on “one or more computers” (Schrittwieser, ¶[0029]).
Claim 13 further recites “receiving a current observation characterizing a current environment state of the environment”; “selecting an action to be performed by the agent in response to the current observation by performing multiple iterations of outer look ahead search to generate an evaluation of possible future states of the environment starting from the current environment state”; “performing an ongoing determining of a proper subset of the possible future states of the environment that is to be explored”; “determining that one or more inner look ahead search commencement criteria are satisfied and in response halting the ongoing determining of the proper subset”; and “then, performing an inner look ahead search of only the proper subset of the possible future states of the environment, and not any remaining possible future states of the environment that are not in the proper subset, until one or more inner look ahead search termination criteria are satisfied”. These operations are identical to the method steps of claim 1 and are taught by Schrittwieser in view of Caldwell for the reasons, with the citations, motivation to combine, and rationale, set forth for claim 1.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 14
Claim 14 recites “A computed-implemented system for solving a search problem by searching through a search space comprising a plurality of candidate solutions”. Schrittwieser teaches this feature. Schrittwieser’s system is implemented “on one or more computers” (Schrittwieser, ¶[0029]) and solves the problem of choosing an action by searching sequences of actions: “at each planning iteration, the system 100 generates a sequence of actions that progress the environment 102 to new, predicted (i.e., hypothetical) future states starting from the state represented by the current observation 104” (Schrittwieser, ¶[0030]). Each sequence of actions, with its succession of predicted states, is a candidate solution, consistent with Applicant’s description of a candidate solution as “a succession of multiple environment states” (Specification, ¶[0047]).
Claim 14 recites “a machine learning subsystem and a search engine”. Schrittwieser teaches this feature: “the planning engine 120 includes a representation model 130, a dynamics model 140, a prediction model 150 and, in some cases, the action sampling engine 160” (Schrittwieser, ¶[0043]), which are “machine learning model[s]” (Schrittwieser, ¶¶[0044], [0046], [0048]), and the look ahead tree search (Schrittwieser, ¶¶[0031], [0091]). Schrittwieser’s computers programmed with a trained model and a Monte Carlo tree search are the same structure identified above under § 112(f) for these limitations.
Claim 14 recites “wherein the machine learning subsystem is configured to identify a proper subset of the plurality of candidate solutions”. Schrittwieser teaches this feature: “the planning engine 120 uses the action sampling engine 160 to select a proper subset of the actions in the set of possible actions and to perform planning by using only the actions in the proper subset” (Schrittwieser, ¶[0042]). Restricting the actions available at each node restricts the action sequences that can be explored to a proper subset of all sequences.
Claim 14 recites “the machine learning subsystem configured to, in each current outer search iteration of multiple outer search iterations: receive a machine learning subsystem input that includes data specifying a current subset of the plurality of candidate solutions that have already been searched in previous outer search iterations”. Schrittwieser teaches this feature: “The system processes (406), using the prediction model and in accordance with trained values of the prediction model parameter, a hidden state corresponding to an environment state represented by the leaf node” (Schrittwieser, ¶[0113]). The leaf’s hidden state is produced by applying the dynamics model along the action sequence from the root to the leaf (Schrittwieser, ¶[0100], describing hidden states s1-s3 generated along the sequence a1-a3), so the input to the prediction model is data specifying the action sequence that has already been searched in previous planning iterations up to that leaf.
Claim 14 recites “process the machine learning subsystem input to generate a machine learning subsystem output that includes data specifying one or more new candidate solutions that are to be searched in the current outer search iteration”. Schrittwieser teaches this feature. The prediction model generates “a predicted policy output that defines a score distribution over the set of actions” (Schrittwieser, ¶[0113]); the system “samples a proper subset of the set of actions (408)” from it (Schrittwieser, ¶[0114]); and “[f]or each sampled action, the system adds, to the state tree, a respective outgoing edge from the leaf node that represents the sampled action” (Schrittwieser, ¶[0115]). Each new edge extends the searched sequence into a new candidate sequence, which the following search explores (see the common claim interpretation above).
Claim 14 recites “search candidate solutions to identify a respective candidate final solution using the search engine, wherein the searched candidate solutions include the one or more new candidate solutions”. Schrittwieser teaches searching candidate solutions that include the new ones: the traversal of the state tree “starting from a root node of the state tree representing the current environment state until reaching a leaf node” (Schrittwieser, ¶[0112]) can pass through the newly added edges. Schrittwieser associates each new edge with a prior probability “derived from the predicted policy output” (Schrittwieser, ¶[0116]), and the traversal selects edges by maximizing an upper confidence bound that includes that prior probability (Schrittwieser, ¶[0035]). Newly added edges are therefore candidates for selection in subsequent searches. The identification of a candidate final solution is addressed with the final limitation below.
Claim 14 recites “wherein the search engine is configured to, in each current outer search iteration: identify the respective candidate final solution to the search problem by performing a look ahead search of possible continuing solutions that start from at least the one or more new candidate solutions specified by the machine learning subsystem output, until one or more look ahead search termination criteria are satisfied”. Schrittwieser teaches performing the look ahead search: “the system begins the look ahead search starting from a root node of the state tree . . . and continues the look ahead search until a possible future state that satisfies termination criteria is encountered” (Schrittwieser, ¶[0092]). A search that passes through a newly added edge and continues to a leaf explores continuing solutions that start from the new candidate solution.
Claim 14 recites “and selecting, from the plurality of candidate solutions, and as the respective candidate final solution, a selected candidate solution as a result of performing the look ahead search”. Primary prior-art reference Schrittwieser does not explicitly teach “selecting, from the plurality of candidate solutions, and as the respective candidate final solution, a selected candidate solution” in each iteration. Schrittwieser’s traversal in each planning iteration selects a sequence of edges to explore (Schrittwieser, ¶[0098]), but Schrittwieser chooses a sequence as an answer to the planning problem only once, after all iterations (Schrittwieser, ¶[0104]). However, secondary prior-art reference Caldwell teaches this feature as follows: “In some examples, the guidance component may trace one or more potential paths through the sets of action nodes as they are generated and back-trace a lowest-cost path . . . . Regardless, the guidance system may determine a contiguous path through the sets of nodes that is associated with a lowest cost and/or that makes the most progress along the route” (Caldwell, ¶[0117]). Caldwell selects, from among the candidate paths generated so far and as the nodes are generated, the lowest-cost path as the current solution. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have Schrittwieser’s planner select, at the end of each iteration’s search, the best candidate sequence found so far as that iteration’s candidate final solution, as Caldwell selects its lowest-cost path while nodes are generated, because Caldwell’s approach keeps a usable solution available whenever node generation is cut off by a time or computation limit (Caldwell, ¶¶[0116]-[0117]), which is necessary for an agent that must act within a bounded time. The rationale is MPEP 2143(I)(D), with (G), and the predictable result is a planner that holds a best-so-far solution after each iteration. KSR, 550 U.S. at 416.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 15
Claim 15 recites “maintain the data specifying the current subset of the plurality of candidate solutions that have already been searched in the previous outer search iterations, including updating the current subset of the plurality of candidate solutions by adding the one or more new candidate solutions”. Schrittwieser teaches this feature: “The system 100 can maintain the plan data 122 at a memory device accessible to the system 100” (Schrittwieser, ¶[0034]), and “[t]he system updates the state tree based on the sampled actions (410). For each sampled action, the system adds, to the state tree, a respective outgoing edge from the leaf node that represents the sampled action” (Schrittwieser, ¶[0115]). No secondary reference is needed to meet this limitation. The remaining limitations are met as set forth for claim 14.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 16
Claim 16 recites “generating an evaluation result of the one or more new candidate solutions by evaluating the one or more new candidate solutions with respect to solving the search problem; and using the evaluation results to select the selected candidate solution”. Schrittwieser teaches this feature: “the system then evaluates the leaf node and updates the statistics data for the edges traversed during the search based on the model outputs. The system may use the score corresponding to the new edge from the score distribution defined by the prediction model output as the prior probability P for the new edge. The system may also determine the action score Q for the new edge from the value output the prediction model network” (Schrittwieser, ¶[0095]), and the value-based selection uses those values (Schrittwieser, ¶[0104]). Caldwell likewise determines “a cost associated with a candidate action” (Caldwell, ¶[0019]) and uses it to select the lowest-cost path (Caldwell, ¶[0117]). No additional secondary reference is needed to meet this limitation. The remaining limitations are met as set forth for claim 14.
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 17
Claim 17 recites “wherein the search engine is further configured to identify a final solution to the search problem from the respective candidate final solutions generated in the multiple outer search iterations”. Schrittwieser teaches this feature: “the system can select a1 as the action to be performed, assuming the sequence of actions a1 - a3 has the maximum associated value output among all different sequences of actions that have been generated over multiple planning iterations” (Schrittwieser, ¶[0104]). No secondary reference is needed to meet this limitation. The remaining limitations are met as set forth for claim 14.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 18
Claim 18 recites “wherein searching the candidate solutions to identify the respective candidate final solution using the search engine comprises determining that one or more look ahead search commencement criteria are satisfied”. Primary prior-art reference Schrittwieser does not explicitly teach “determining that one or more look ahead search commencement criteria are satisfied,” however, secondary prior-art reference Caldwell teaches this feature as follows: “the tree search may continue to alternately determine action nodes and prediction nodes until a termination event, such as determining a predefined number of action nodes, a time limit for the computation, a computation or storage limit” (Caldwell, ¶[0116]), and the path is determined “as the sets of nodes are generated and/or termination event occurs” (Caldwell, ¶[0117]; see ¶[0118]). A termination event that ends node generation and upon which the path through the generated nodes is determined is a look ahead search commencement criterion. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to begin the search in each iteration of Schrittwieser upon determining that such a criterion is satisfied, for the reasons given for claim 1 (Schrittwieser, ¶[0014]; Caldwell, ¶¶[0022], [0116]). The rationale is MPEP 2143(I)(D), with (G), and the predictable result is a search commenced upon a detected count, time, or computation criterion. KSR, 550 U.S. at 416. The remaining limitations are met as set forth for claim 14.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 19
Claim 19 recites “determining that a time spent on processing the machine learning system input to generate the machine learning subsystem output exceed a predetermined time length”. Claim 19 is interpreted as set forth in the § 112(b) rejection above. Primary prior-art reference Schrittwieser does not explicitly teach “determining that a time spent on processing the machine learning system input to generate the machine learning subsystem output exceed a predetermined time length,” however, secondary prior-art reference Caldwell teaches this feature as follows: node generation continues “until a termination event, such as determining a predefined number of action nodes, a time limit for the computation” (Caldwell, ¶[0116]). In the combination, the computation that generates new candidates is Schrittwieser’s processing of the leaf hidden state by the prediction model and the sampling of actions (Schrittwieser, ¶¶[0113]-[0115]), and Caldwell’s time limit bounds that computation. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to end the generation of new candidates when the time spent on it exceeds a predetermined limit, for the reasons given for claim 8 (Caldwell, ¶¶[0022], [0116]; Schrittwieser, ¶[0014]). The rationale is MPEP 2143(I)(D), with (G), and the predictable result is a time-bounded candidate-generation phase. KSR, 550 U.S. at 416. The remaining limitations are met as set forth for claims 14 and 18.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell.
Claim 20
Claim 20 recites “determining that the updated current subset of the candidate solutions are sufficient in terms of generating an evaluation result of all of the plurality of candidate solutions in the search space, wherein determining that the updated current subset of the candidate solutions are sufficient comprises: evaluating, by computing a stopping function, the respective candidate final solution that has been identified from the updated current subset of the candidate solutions, to generate a binary classification result that specifies whether the respective candidate final solution is a valid solution to the search problem”. Claim 20 is interpreted as set forth in the § 112(b) rejection above. Primary prior-art reference Schrittwieser does not explicitly teach this feature, however, secondary prior-art reference Caldwell teaches this feature as follows: node generation continues until a termination event such as “determining an action that reaches a target position, determining that a path to a target position is less than a cost threshold” (Caldwell, ¶[0116]). Caldwell thus evaluates the path identified from the generated nodes (the candidate final solution) with a test whose result is either valid (it reaches the target, or its cost is below the threshold) or not, and it treats a valid path as a reason to stop generating nodes, that is, as showing that the generated nodes suffice. Caldwell also describes a yes/no feasibility test on a path, “determining if a previous path and/or previous connection determined by the guidance system is feasible (e.g., satisfies current constraints such as velocity, maximum steering angle, and/or boundaries; is impact-free; has a cost that is less than a cost threshold)” (Caldwell, ¶[0119]). That test is applied to a previously determined path before a full search, which confirms that validity tests on candidate paths are part of Caldwell’s search. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to test, in each iteration of Schrittwieser as modified for claim 14, whether the candidate final solution identified from the updated set of candidates is a valid solution and to treat a valid result as a sufficient set, as Caldwell does, because doing so stops the generation of further candidates as soon as a usable solution exists (Caldwell, ¶¶[0116], [0119]; Schrittwieser, ¶[0014]). The rationale is MPEP 2143(I)(D), with (G), and the predictable result is a binary validity test on the candidate final solution. KSR, 550 U.S. at 416. The remaining limitations are met as set forth for claims 14 and 18.
Claims 10 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser (US 2023/0073326 A1) in view of Caldwell (US 2023/0041975 A1) and Simonyan (US 2020/0143239 A1).
Claim 10
Claim 10 recites “selecting the action to be performed by the agent in response to the current observation using statistics generated during the multiple iterations of outer look ahead search for the root node of the state tree that represents the current observation”. Schrittwieser teaches this feature: “the plan data 122 can include statistics data that has been compiled during planning for each outgoing edge of the root node of the state tree, i.e., the node that corresponds to the state characterized by the current observation 104, and the system 100 can select the actual action 110 based on the statistics data for the node-edge pairs corresponding to the root node” (Schrittwieser, ¶[0037]).
Claim 10 recites “wherein the statistics comprise, for each outgoing edge connected to the root node that represents a corresponding action that was performed by the agent in response to the current observation, a respective action score for the action represented by the edge”. Claim 10 is interpreted as set forth in the § 112(b) rejection above. Schrittwieser teaches this feature: “the plan data 122 can include, for each outgoing edge of a root node of the state tree, (i) an action score Q for the action represented by the edge” (Schrittwieser, ¶[0032]).
Claim 10 recites “which action score specifies a likelihood that the agent will complete the task if the action is performed”. Primary prior-art reference Schrittwieser does not explicitly teach an action score that “specifies a likelihood that the agent will complete the task if the action is performed”; Schrittwieser’s action score “represents the current estimate of the return that will be received if the action is performed” (Schrittwieser, ¶[0033]). However, secondary prior-art reference Simonyan teaches this feature as follows: “the action score for an action represents the current estimate of the return that will be received if the action is performed” (Simonyan, ¶[0069]), and that estimate is “the new average of the predicted expected returns of all searches that involved traversing the edge” (Simonyan, ¶[0098]). The predicted expected return is trained toward a target return, where “[t]he target return can be based on whether the terminal state of the environment achieved the specified result, e.g., 1 if the terminal state achieves the result and 0 or −1 if the terminal state does not achieve the result” (Simonyan, ¶[0076]), and “the reinforcement learning system selects actions in order to maximize the likelihood of completing the task, i.e., of achieving the result” (Simonyan, ¶[0030]). When the return is 1 for achieving the result and 0 otherwise, the averaged predicted return for an action is an estimate of the likelihood that the task will be completed if the action is performed. With the 1/-1 alternative, the score is a linear rescaling of that likelihood. Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to use in Schrittwieser’s planner an action score computed from Simonyan’s success-based return, so that the score estimates the likelihood of completing the task, because Schrittwieser expressly incorporates Simonyan’s look ahead search (Schrittwieser, ¶[0036]) and Simonyan uses this return for tasks where “the only reward being received being at a terminal state of any given episode of interactions and indicating whether the specified result was achieved or not” (Simonyan, ¶[0053]). The rationale is MPEP 2143(I)(A), with (G), because Schrittwieser’s own text points to Simonyan’s statistics, and the predictable result is an action score that estimates the chance of task completion. KSR, 550 U.S. at 416.
Claim 10 recites “wherein selecting the action to be performed by the agent comprises selecting the action that has a highest action score”. Schrittwieser selects by value: “the system can make this selection by determining, from the sequences of actions in the plan data, a sequence of actions that has a maximum associated value output and thereafter selecting, as the action to be performed by the agent in response to the current observation, the first action in the determined sequence of actions” (Schrittwieser, ¶[0104]). Schrittwieser does not state in terms that it selects the root-node action whose own action score is highest. However, Simonyan selects edges by the highest score among a node’s outgoing edges: “the system selects the edge to be traversed using the edge data for the outgoing edges from the in-tree node . . . . The system may select the edge based on the action score” (Simonyan, ¶[0089]), and “[t]he system then selects the edge with the highest adjusted action score as the edge to be traversed from the in-tree node” (Simonyan, ¶[0091]). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to select, as the action to be performed, the root-node action having the highest action score, because Schrittwieser already selects by maximum value among the root’s alternatives (Schrittwieser, ¶[0104]) and bases that selection on the root-node edge statistics, including the action score Q (Schrittwieser, ¶¶[0032], [0037]). Simonyan teaches choosing among a node’s outgoing edges by the highest action score (Simonyan, ¶¶[0089], [0091]), and Schrittwieser incorporates Simonyan’s search by reference (Schrittwieser, ¶[0036]). The rationale is MPEP 2143(I)(E), with (G): selecting by highest action score is one of a small number of known ways to pick the root action from its edge statistics, alongside the highest visit count (Schrittwieser, ¶[0038]) and the maximum value (¶[0104]). It yields the predictable result of choosing the action with the best estimated chance of completing the task. KSR, 550 U.S. at 416. The remaining limitations are met as set forth for claims 1 and 9.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Schrittwieser in view of Caldwell and Simonyan.
Claim 12
Claim 12 recites “wherein performing the inner look ahead search comprises performing a Monte Carlo tree search guided by outputs of a neural network”. Schrittwieser teaches this feature: “the system 100 can generate the plan data 122 by performing a look ahead search guided by the outputs of the planning engine 120. . . . For example, the look ahead search may be a tree search, e.g., a Monte-Carlo tree search” (Schrittwieser, ¶[0031]), where “[t]he representation, dynamics, and prediction models can each be implemented as a respective neural network” (Schrittwieser, ¶[0051]).
Claim 12 recites “wherein the neural network is configured to receive an input observation characterizing a state of the environment and to process the input observation in accordance with network parameters to generate a network output that specifies an action to be performed by the agent in response to the input observation”. Primary prior-art reference Schrittwieser does not explicitly teach that the network guiding the search receives an observation as its input; Schrittwieser’s prediction model receives a hidden state produced from the observation by a representation model (Schrittwieser, ¶¶[0044], [0048]). However, secondary prior-art reference Simonyan teaches this feature as follows: “the action selection neural network 130 is a neural network that is configured to receive a network input including an observation and to process the network input in accordance with parameters of the action selection neural network (‘network parameters’) to generate a network output” (Simonyan, ¶[0048]); “[t]he action selection output defines an action selection policy for selecting an action to be performed by the agent in response to the input observation” (Simonyan, ¶[0049]); and “the action selection subsystem 120 performs the look ahead search by performing a tree search guided by the outputs of the neural network 130” (Simonyan, ¶[0066]). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to guide the Monte Carlo tree search of Schrittwieser, as modified by Caldwell, with Simonyan’s observation-input action selection network, because Schrittwieser expressly incorporates Simonyan’s look ahead search (Schrittwieser, ¶[0036]), and Simonyan’s network is the known alternative for environments in which a simulator can supply the observations of future states (Simonyan, ¶¶[0060]-[0062]; Schrittwieser, ¶[0008]). The rationale is MPEP 2143(I)(B), with (G): substituting one known guiding network for another performing the same function yields the predictable result of a network-guided Monte Carlo tree search. KSR, 550 U.S. at 416. The remaining limitations are met as set forth for claim 1.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
PERTINENT PRIOR ART – Patent Literature / Non-Patent Literature (NPL)
The prior art made of record and not relied upon is considered pertinent to Applicant’s disclosure. It is made of record for compact prosecution.
Redding, US 2018/0089563 A1, published March 29, 2018, first-named inventor Joshua D. Redding, assignee Apple Inc. (issued as US 10,769,525 B2). Redding describes a behavior planner for an autonomous vehicle that generates conditional action sequences with Monte Carlo tree search, using heuristics from neural networks “for limiting the sizes of the trees that are constructed, limiting the breadth of a tree search at a given level by identifying a subset of nodes which correspond to ‘more useful’ actions, and/or for limiting the depth of a tree search based on assigned node values” (Redding, ¶[0028]; see ¶[0052]). It also notes that such algorithms “can be exited gracefully at any time” (Redding, ¶[0051]). Redding thus teaches learned restriction of a tree search to a subset of states. It does not describe repeated outer iterations in each of which the determination of the subset is halted upon a commencement criterion before an inner search of only that subset, and it is cumulative of Schrittwieser for the subset restriction.
Graepel, US 2018/0032864 A1, published February 1, 2018, first-named inventor Thore Kurt Hartwig Graepel, applicant Google Inc. (issued as US 10,867,242 B2 to DeepMind Technologies Limited, cited in Applicant’s IDS). Graepel describes selecting agent actions by traversing a state tree from the root “until the environment reaches a leaf state” (Graepel, ¶[0085]) and expanding the leaf by adding “a respective new edge to the state tree for each action that is a valid action” with priors from a policy neural network (Graepel, ¶¶[0090]-[0091]). Graepel teaches the basic network-guided tree search that underlies claims 9-12. It expands every valid action rather than a determined proper subset of future states, and it does not teach halting state determination upon a commencement criterion.
Ramamoorthy, US 2021/0370980 A1, published December 2, 2021, first-named inventor Subramanian Ramamoorthy, assignee Five AI Limited. Ramamoorthy describes autonomous vehicle planning in which “[t]he tree search algorithm may be a Monte Carlo Tree Search (MCTS) algorithm” (Ramamoorthy, ¶[0025]) and in which the search “may be performed multiple times for multiple constructed game trees with different sampled goals, and the sequence of AV maneuvers may be determined by the AV planner based on a statistical analysis of paths within the multiple constructed game trees” (Ramamoorthy, ¶[0059]). Ramamoorthy shows repeated tree searches whose results are aggregated to choose an action, which bears on the outer/inner iteration structure of claims 1, 13, and 17. It does not teach determining a proper subset of future states, halting that determination upon a criterion, or confining each search to the determined subset.
V. Lisý, B. Bošanský, R. Vaculín and M. Pěchouček, "Agent subset adversarial search for complex non-cooperative domains," Proceedings of the 2010 IEEE Conference on Computational Intelligence and Games, Copenhagen, Denmark, 2010, pp. 211-218, doi: 10.1109/ITW.2010.5593351.
Prior Art of Record
Schrittwieser, US 2023/0073326 A1, published March 9, 2023, first-named inventor Julian Schrittwieser, assignee DeepMind Technologies Limited (U.S. Application No. 17/794,797, national stage of PCT/IB2021/050691 filed January 28, 2021; foreign priority to Greek Application No. 20200100037, filed January 28, 2020). Basis: 35 U.S.C. 102(a)(2). Schrittwieser describes selecting an action for an agent by performing multiple planning iterations of Monte Carlo tree search from the node representing the current observation. At each iteration it traverses the state tree from the root to a leaf, samples a proper subset of a fixed number of actions from a learned policy output, and expands the tree using only the sampled actions, so that planning explores only a subset of the reachable states. It then selects the action from root-node statistics such as visit counts or values.
Caldwell, US 2023/0041975 A1, published February 9, 2023, first-named inventor Timothy Caldwell, assignee Zoox, Inc. (U.S. Application No. 17/394,334, filed August 4, 2021; issued as US 11,932,282 B2). Basis: 35 U.S.C. 102(a)(2). Caldwell describes a tree search for controlling an autonomous vehicle that alternately generates candidate-action nodes and predicted-environment-state nodes from a root node representing the current sensed state. Node generation continues until a termination event such as a predefined number of nodes or a time limit, after which a search is conducted over the generated nodes for a lowest-cost path, including feasibility and cost-threshold checks on candidate paths.
Simonyan, US 2020/0143239 A1, published May 7, 2020, first-named inventor Karen Simonyan, assignee DeepMind Technologies Limited (issued as US 11,449,750 B2). Basis: 35 U.S.C. 102(a)(1). Simonyan describes a look ahead tree search guided by an action selection neural network that receives an observation and generates a policy output and a predicted return, with edge statistics (action score, visit count, prior probability) and a target return of 1 when the specified result is achieved and 0 or -1 otherwise. Schrittwieser incorporates Simonyan by reference as describing the look ahead search it uses (Schrittwieser, ¶[0036]).
Analogous art. Each reference is in the same field of endeavor as the claimed invention, namely selecting actions for an agent (including robots and autonomous vehicles) by look ahead search over possible future states of an environment (Specification, ¶¶[0002], [0017], [0045]; Schrittwieser, ¶¶[0006], [0058]; Caldwell, ¶¶[0013], [0015]; Simonyan, ¶¶[0030], [0066]). Caldwell is also reasonably pertinent to the problem the claims address, which is limiting the computation spent on building the set of future states to be searched (Specification, ¶¶[0007], [0074]; Caldwell, ¶¶[0022], [0116]). MPEP 2141.01(a); In re Bigio, 381 F.3d 1320, 1325 (Fed. Cir. 2004).
THIS ACTION IS MADE FINAL
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
THIS ACTION IS MADE FINAL
Applicant’s amendment necessitated new grounds of rejection and FINAL Rejection.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW T. SITTNER whose telephone number is (571) 270-7137 and email: matthew.sittner@uspto.gov. The examiner can normally be reached on Monday-Friday, 8:00am - 5:00pm (Mountain Time Zone). Please schedule interview requests via email: matthew.sittner@uspto.gov
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sarah M. Monfeldt can be reached on (571) 270-1833.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW T SITTNER/
Primary Examiner, Art Unit 3629b
Prosecution Insights