Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for National Stage under 35 U.S.C. 371. The certified copy of parent Application No. PCT/JP2022/018429, filed on 04/21/2022, was entered on 10/17/2024.
Status of Claims
Claims 1-4 and 6-13 were rejected in the Non-Final Office action mailed on 09/29/2025. Applicant’s amended claimset, entered on 12/23/2025, amended Claims 1, 4, 6, 9, 10, and 13. Herein this Final Office Action, Claims 1-4 and 6-13 are rejected.
Response to Arguments
Applicant’s arguments filed 12/23/2025, with respect to Rejections under 35 U.S.C. 112(b) for Claims 1-4 and 6-13, have been fully considered and are persuasive. The previous rejection under 35 U.S.C. 112(b) has been withdrawn. However, the amendments yield a rejection under 35 U.S.C. 112(b) on new grounds.
Applicant’s arguments filed 12/23/2025, with respect to Rejections under 35 U.S.C. 101 for Claims 1-4 and 6-13, have been fully considered and are not persuasive.
On Pages 6-7, Applicant argues that the claims do not recite an abstract idea in Step 2A Prong One. On Pages 6, Applicant states “The Office Action asserted that claim 1 falls within the 'Mental Process' grouping of abstract ideas. (Office Action, at page 5).” Applicant further argues that the amended limitation of “the reinforcement learning is performed to reduce a loss function based on a characteristic of a dense embedding layer and a reward" is not a mental process. Examiner does not agree.
Examiner’s previous office action, and herein, did not categorize any of the limitations of the recited abstract idea as “mental processes.” The previous office action, and herein, categorizes the recited abstract idea as “certain methods of organizing human activity” or “mathematical concepts.”
Examiner further notes that the analysis of Step 2A Prong One asks if the claim “recites” an abstract idea, not if any limitation of the claim does not recite an abstract idea. Examiner has identified additional elements (not a part of the recited abstract idea) that are further analyzed.
On Pages 7-8, Applicant argues that the abstract idea is integrated into a practical application at Step 2A Prong Two. Applicant argues that the claims as a whole provides “specific improvements to the technical field of providing delivery planning to delivery vehicles with giving considerations to the status of the delivery vehicles.” Applicant argues that the limitation of “the reinforcement learning is performed to reduce a loss function based on a characteristic of a dense embedding layer and a reward" “reduces computational cost while improving the efficiency.” Applicant distinguishes instant claims from the ineligible claims in Electric Power Group by asserting the limitation of “the reinforcement learning is performed to reduce a loss function based on a characteristic of a dense embedding layer and a reward" is not recited at a high level of generality. Examiner does not agree.
PEG Example 47 Claim 2 states “A method of using an artificial neural network (ANN) comprising: . . . (c) training, by the computer, the ANN based on the input data and a selected training algorithm to generate a trained ANN, wherein the selected training algorithm includes a backpropagation algorithm and a gradient descent algorithm; . . .” PEG Example 47 Claim 2 Step 2A, Prong One states “Step (c) requires specific mathematical calculations (a backpropagation algorithm and a gradient descent algorithm) to perform the training of the ANN and therefore encompasses mathematical concepts.”
MPEP 2106.05(a) states that “the examiner should not determine the claim improves technology [if the improvement is set forth in a conclusory manner].”
Examiner responds that the claimed training to “reduce a loss function” recites mathematical concepts, i.e. an abstract idea, similar to PEG Example 47 Claim 2. Applicant’s assertion of reducing “computation cost while improving the efficiency” is conclusory, and therefore fails to provide an eligible improvement under MPEP 2106.05(a). The claim limitation provides no detail as to how the learning reduces the loss function or how it interacts with the recited computer components. Thus, the rejection remains.
On Pages 8-9, Applicant argues that the claims provide an inventive concept in Step 2B by reciting additional elements that are unconventional or otherwise more than what is well-understood, routine, conventional activity in the field. Applicant argues that the claims, as a whole, “recites a particular solution to address the computer-centric challenge of providing delivery planning to delivery vehicles with giving considerations to the status of the delivery vehicles.” Applicant argues that the limitation of “the reinforcement learning is performed to reduce a loss function based on a characteristic of a dense embedding layer and a reward" is not well-understood, routine, or conventional. Applicant argues that the use of “machine learning” is a patent eligible particular machine under MPEP 2106.05. Examiner does not agree.
MPEP 2106.05(d) states “When making a determination whether the additional elements in a claim amount to significantly more than a judicial exception, the examiner should evaluate whether the elements define only well-understood, routine, conventional activity. In this respect, the well-understood, routine, conventional consideration overlaps with other Step 2B considerations, particularly the improvement consideration (see MPEP § 2106.05(a)), the mere instructions to apply an exception consideration (see MPEP § 2106.05(f)), and the insignificant extra-solution activity consideration (see MPEP § 2106.05(g)). Thus, evaluation of those other considerations may assist examiners in making a determination of whether a particular element or combination of elements is well-understood, routine, conventional activity.”
MPEP 2106.05(b) states “It is noted that while the application of a judicial exception by or with a particular machine is an important clue, it is not a stand-alone test for eligibility. . . Examiners may find it helpful to evaluate other considerations such as the mere instructions to apply an exception consideration (see MPEP § 2106.05(f)), the insignificant extra-solution activity consideration (see MPEP § 2106.05(g)), and the field of use and technological environment consideration (see MPEP § 2106.05(h)), when making a determination of whether an element (or combination of elements) is a particular machine.”
MPEP 2106.05(b)I states “It is important to note that a general purpose computer that applies a judicial exception, such as an abstract idea, by use of conventional computer functions does not qualify as a particular machine. Ultramercial, Inc. v. Hulu, LLC, 772 F.3d 709, 716-17, 112 USPQ2d 1750, 1755-56 (Fed. Cir. 2014). See also TLI Communications LLC v. AV Automotive LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (mere recitation of concrete or tangible components is not an inventive concept); Eon Corp. IP Holdings LLC v. AT&T Mobility LLC, 785 F.3d 616, 623, 114 USPQ2d 1711, 1715 (Fed. Cir. 2015) (noting that Alappat’s rationale that an otherwise ineligible algorithm or software could be made patent-eligible by merely adding a generic computer to the claim was superseded by the Supreme Court’s Bilski and Alice Corp. decisions). If applicant amends a claim to add a generic computer or generic computer components and asserts that the claim recites significantly more because the generic computer is 'specially programmed' (as in Alappat, now considered superseded) or is a 'particular machine' (as in Bilski), the examiner should look at whether the added elements integrate the exception into a practical application or provide significantly more than the judicial exception. Merely adding a generic computer, generic computer components, or a programmed computer to perform generic computer functions does not automatically overcome an eligibility rejection. Alice Corp. Pty. Ltd. v. CLS Bank Int’l, 573 U.S. 208, 223-24, 110 USPQ2d 1976, 1983-84 (2014). See In re Alappat, 33 F.3d 1526, 1545, 31 USPQ2d 1545, 1558 (Fed. Cir. 1994); In re Bilski, 545 F.3d 943, 88 USPQ2d 1385 (Fed. Cir. 2008).”
As discussed above, the additional elements are recited at a high degree of generality, such that they merely link the recited abstract idea to the technology (MPEP 2106.05(h)) and merely apply the abstract idea (MPEP 2106.05(f)), and do not provide an improvement in technology (MPEP 2106.05(a)). This determination is further supported by the lack of particular machine used to implement the abstract idea. As shown in MPEP 2106.05(b)I, merely programming a computer to perform generic computer functions (i.e. calculations outlined in the 35 U.S.C. 101 rejection section) does not overcome eligibility. Thus, the rejection remains.
Applicant’s arguments filed 12/23/2025, with respect to Rejections under 35 U.S.C. 103 for Claims 1-4 and 6-13, have been fully considered and are not persuasive.
On Pages 9-10, Applicant argues that the limitation of "each agent in the each of the plurality of actor networks is configured to act alternatively for each time step and the state of the certain moving body and the state of the plurality of nodes are simultaneously updated" is not taught by the prior art of record. Specifically, Applicant argues that “However, as noted by the Examiner Interview Summary dated December 18, 2025, the combination of Hamzehi-007 and Hamzehi-652 fails to teach or suggest "each agent in the each of the plurality of actor networks is configured to act alternatively for each time step and the state of the certain moving body and the state of the plurality of nodes are simultaneously updated," as recited by amended claim 1, as nowhere in Hamzehi-007 and Hamzehi-652 teaches the feature of each agent acting alternatively for each time step and simultaneously updating the state of the moving body and the nodes.” Examiner does not agree.
Examiner responds that no agreement for withdrawal was made during the interview. Examiner stated that “Although further search and consideration is needed, the proposed amendments introduce new claim limitations that were not previously examined, and therefore overcome the art rejection as written. No agreement for allowance met.” (Emphasis added). Thus, Examiner did not agree that Hamzehi-007 and Hamzehi-652 fails to teach the quoted limitation.
Additionally, as discussed in greater detail in the 35 U.S.C. 112(b) rejection section below, upon examination, Applicant used “alternatively” (as synonymous with “alternately”) as a special definition contrary to its ordinary meaning. As further discussed in the 35 U.S.C. 112(b) rejection section below, this limitation is interpreted as synonymous with “alternately” herein, which deviates from Examiners assumption of no special definition during the interview.
In light of the further search and consideration, Hamzehi-007 teaches the limitation of each agent acting alternatively for each time step and simultaneously updating the state of the moving body and the nodes, as discussed in greater detail in the rejection section below.
Claim Interpretation
Claims 1-4 and 6-13 recite (either directly or via dependency) a “route.” Representative independent Claim 1 recites “solving a vehicle routing problem for determining a route for providing a service to a plurality of nodes by a plurality of moving bodies . . .” Thus, “a route” is for “providing a service” (e.g. transportation service) to “a plurality of nodes by a plurality of moving bodies” (emphasis added). Therefore, the scope of “a route” is not restricted to the path of a single moving body but includes the path of the plurality of moving bodies, analogous to a routing plan for the fleet of moving bodies.
Claims 1-4 and 6-13 recite (either directly or via dependency) “delivery” in reference to planning and operations of a vehicle or moving body. Applicant’s specification is clear that “delivery” refers to the delivery of a product, baggage or similar object (¶¶2-4 and ¶17), which is consistent with the plain meaning of the word. Thus, the scope of the broadest reasonable interpretation of “delivery” does not include transportation services for humans or persons (e.g. ride hailing or ride sharing).
Claims 1-4 and 6-13 recite (either directly or via dependency) a “moving body.” The broadest reasonable interpretation of “moving body,” read in light of the specification (¶17 shows that the “moving body” can be a “vehicle” or a “person” which delivers a baggage.), includes a vehicle or a person, which would move the deliverable object.
Claims 3, 8, and 12 recite an “algorithm calculation unit.” The broadest reasonable interpretation of an “algorithm calculation unit” includes a “model,” i.e. a conceptual “unit” for “algorithm calculation” analogous to an “algorithm calculation feature,” “algorithm calculation function,” or “algorithm calculation operation,” read in light of the specification (¶32 shows “The algorithm calculation unit 130 is a model . . .” ¶25 shows “[T]he algorithm calculation unit 130 may be implemented in a certain computer, . . .”).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1-4 and 6-13 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Where applicant acts as his or her own lexicographer to specifically define a term of a claim contrary to its ordinary meaning, the written description must clearly redefine the claim term and set forth the uncommon definition so as to put one reasonably skilled in the art on notice that the applicant intended to so redefine that claim term. Process Control Corp. v. HydReclaim Corp., 190 F.3d 1350, 1357, 52 USPQ2d 1029, 1033 (Fed. Cir. 1999). The term “alternatively” in Claims 1-4 and 6-13, i.e. last paragraph of the independent claims, is used by the claim to mean “alternately,” i.e. to occur in sequence, in succession, or in turn, See Specification ¶¶19-22 showing intent to use “alternatively” synonymously with “alternately,” while the accepted meaning of “alternatively” indicates mutually exclusive choice, i.e. alternative options, one of which is selected. The term is indefinite because the specification does not clearly redefine the term. Further examination of Claims 1-4 and 6-13 herein will be based on interpreting “alternatively” as if it recited “alternately.”
Additionally, Claims 3, 8, and 12 recites the limitation " the algorithm calculation unit" in the last paragraph. There is insufficient antecedent basis for this limitation in the claim. Further examination herein will interpret this limitation as “an algorithm calculation unit.”
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-4 and 6-13 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1
Claims 1-3 and 6 recite a device (i.e. a machine or manufacture), Claims 4 and 7-9 recite a method (i.e. a process), and Claims 10-13 recite a non-transitory recording medium (i.e. a machine or manufacture). Therefore, Claims 1-13 all fall within the one of the four statutory categories of invention of 35 U.S.C. 101.
Step 2A, Prong One
Independent Claim 1 recites the abstract idea of “. . . to perform a set of operations, the set of operations comprising:
solving a vehicle routing problem for determining a route for providing a service to a plurality of nodes by a plurality of moving bodies using . . . reinforcement learning by an Actor-Critic method,
wherein the reinforcement learning is performed to reduce a loss function based on a characteristic of a dense embedding layer and a reward, and
determining for each of a plurality of [sub-models] corresponding to the plurality of moving bodies, and each [sub-model] determines the route based on a state of a certain moving body and a state of the plurality of nodes,
wherein each agent in the each of the plurality of actor networks is configured to act alternatively for each time step and the state of the certain moving body and the state of the plurality of nodes are simultaneously updated.”
The limitations stated above are processes/ functions that under broadest reasonable interpretation covers (1) determining a route for providing a service by solving a VRP using reinforcement learning by an Actor-Critic method, (2) reinforcement learning reduces a loss function based on a characteristic and a reward, (3) a plurality of models, each used to determine the route based on certain information, and (4) each agent acts alternatively (i.e. “alternately” see 35 U.S.C. 112(b) rejection section) at each time step and the state of the nodes are updated simultaneously, all of which are commercial or legal interactions (i.e. determining a route for providing a service is at least “sales activities or behaviors”), which are certain methods of organizing human activity, an abstract idea, under MPEP 2106.04(a)(2)II, and mathematical relationships (i.e. the models and sub-models define a relationship between variables) and mathematical calculations (i.e. solving the VRP by using a plurality of sub-models based on certain state information and the actor-critic method of reinforcement learning, reinforcement learning to reduce a loss function based on certain parameters, and performing the calculation a certain way such as solving for certain variables simultaneously and adjusting certain inputs alternately), which are mathematical concepts, an abstract idea, under MPEP 2106.04(a)(2)III. The mere the recitation of generic computer components (i.e., the “delivery planning device,” “a neural network that performs [learning],” and “actor network”) implementing the identified abstract idea does not take the claim out of the certain methods of organizing human activity or mathematical concepts groupings. MPEP 2106.04(d). If a claim limitation, under its broadest reasonable interpretation, covers “commercial or legal interactions,” “mathematical relationships,” and “mathematical calculations,” but for the recitation of generic computer components, then it falls in the certain methods of organizing human activity or mathematical concepts groupings of abstract ideas. MPEP 2106.04. Therefore, Claim 1 recites an abstract idea.
Step 2A, Prong Two
The judicial exception is not integrated into a practical application. Claim 1 as a whole amounts to: (i) merely invoking generic components as a tool to perform the abstract idea or “apply it” (or an equivalent) and (ii) generally links the use of a judicial exception to a particular technological environment or field of use. The claim recites the additional elements of:
(i) delivery planning device,
(ii) processor and
(iii) memory,
(iv) a neural network (to the extent that the neural network is a part of a computer), and
(v) an actor network (to the extent that the actor network is a part of a computer).
The additional elements of (i) delivery planning device (Fig.1 and ¶24 shows that the “delivery planning device 100” includes “an algorithm calculation unit 130.” ¶25 shows “The delivery planning device 100 may be implemented by one device (computer) or may be implemented by a plurality of devices.”), (ii) processor (Fig. 5 and ¶¶86-89 shows “CPU 1004.”), (iii) memory (Fig. 5 and ¶¶86-89 shows “memory device 1003.”), (iv) a neural network (Fig. 2 and ¶32 shows “The algorithm calculation unit 130 is a model of a neural network that performs reinforcement learning of the Actor-Critic method.” See ¶¶9-10 showing Non-Patent Literation (2018) discussing the Actor-Critic method of solving a VRP.), and (v) an actor network (Fig. 2 and ¶¶33-38 shows “actor network 131.”), are recited at a high-level of generality, such that, when viewed as whole/ordered combination (Fig. 1-2, ¶24, and ¶¶32-38 shows elements in combination.), they amount to no more than mere instruction to apply the judicial exception using generic computer components or “apply it” (See MPEP 2106.05(f)).
The (i) delivery planning device, (ii) processor, (iii) memory, (iv) a neural network, and (v) an actor network, when viewed as whole/ordered combination (Fig. 1-2 and 5, ¶24, ¶¶32-38, and ¶¶85-89 shows elements in combination.), does no more than generally link the use of the judicial exception to a particular technological environment or field of use (i.e. computer environment) (See MPEP 2106.05(h)).
Accordingly, these additional elements, when viewed as a whole/ordered combination (Fig. 1-2 and 5, ¶24, ¶¶32-38, and ¶¶85-89 shows elements in combination.), do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Thus, the claim is directed to an abstract idea.
Step 2B
As discussed above with respect to Step 2A Prong Two, the additional elements amount to no more than: (i) “apply it” (or an equivalent) and (ii) generally link the use of a judicial exception to a particular technological environment or field of use, and are not a practical application of the abstract idea. The same analysis applies here in Step 2B, i.e., (i) merely invoking the generic components as a tool to perform the abstract idea or “apply it” (See MPEP 2106.05(f)) and (ii) generally linking the use of a judicial exception to a particular technological environment or field of use (See MPEP 2106.05(h)), does not integrate the abstract idea into a practical application at Step 2A or provide an inventive concept at Step 2B.
Furthermore, the (i) delivery planning device, (ii) a neural network, and (iii) an actor network, when viewed as whole/ordered combination (Fig. 1-2 and 5, ¶24, ¶¶32-38, and ¶¶85-89 shows elements in combination.) are recited at a high-level of generality and performs generic computer functions (i.e., “ii. Performing repetitive calculations,” “iii. Electronic recordkeeping [(e.g. updating an activity log)],” and “iv. Storing and retrieving information in memory”) that are well-understood, routine and conventional activities previously known in the industry (See MPEP 2106.05(d)(II)).
Therefore, the additional elements of the (i) delivery planning device, (ii) processor, (iii) memory, (iv) a neural network, and (v) an actor network, do not integrate the abstract idea into a practical application at Step 2A or provide an inventive concept at Step 2B. Thus, even when viewed as a whole/ordered combination (Fig. 1-2 and 5, ¶24, ¶¶32-38, and ¶¶85-89 shows elements in combination.), nothing in the claims adds significantly more (i.e., an inventive concept) to the abstract idea. Thus, the claim is ineligible.
Dependent Claims 2-3 and 6 recite the abstract idea of:
“. . . wherein a state of each moving body includes at least a position and a loading amount, and a state of each node includes at least a position and a demand.” (Claim 2).
“. . . wherein the algorithm calculation unit performs repeatedly the processing of determining an action and updating a state for each moving body in each time step.” (Claim 3).
“. . . whereinis outputted by solving a vehicle routing problem (VRP) problem based on information on each node and each of the plurality of the moving bodies Claim 6).
Dependent Claims 2-3 and 6, have been given the full two-prong analysis including analyzing the further elements and limitations, both individually and in combination. When analyzed individually and in combination, these claims are also held to be patent ineligible under 35 U.S.C. 101. The further limitation of Claims 2-3 and 6 fail to establish claims that are not directed to an abstract idea because the further limitations include (1) the state of each body and node is limited to certain information, (2) repeatedly processing and updating the state in each time step, and (3) outputting a delivery plan by solving a VRP problem based on certain information, which merely further limit the abstract idea itself. The elements of Claims 2-3 and 6 (i.e. elements of Claim 1) fails to establish claims that are not directed to an abstract idea because the elements merely recite the generic computer components of Claim 1 and generally link the abstract idea to a particular technology or field of use (i.e. computer environment) just as in Claim 1. The organization of the further limitations of Claims 2-3 and 6 fail to integrate an abstract idea into a practical application just as discussed above for Claim 1. Additionally, performing the abstract idea of Claim 1 as recited in each of the further limitations of Claims 2-3 and 6, individually or in combination, does not (1) impose any meaningful limits on practicing the abstract ideas, or (2) provide improvements to the functioning of computing systems or to another technology or technical field, just as discussed above regarding Claim 1. Therefore, Claims 2-3 and 6 amount to mere instructions to implement the abstract idea (1) using generic computer components—using the computer, in its ordinary capacity, as a tool to perform the abstract idea, and (2) generally linked to a particular technology or field of use. Because the claims merely use a computer, in its ordinary capacity in a particular field of use, as a tool to perform the abstract idea cannot provide an inventive concept, the elements and limitations of Claims 2-3 and 6 fail to establish that the claims provide an inventive concept, just as in Claim 1. Therefore, Claims 2-3 and 6 fails the Subject Matter Eligibility Test and are consequently rejected under 35 U.S.C. 101.
Claims 4 and 7-9 recite elements and limitations that are substantially similar to Claims 1-3 and 6. Claims 4 and 7-9 recite a method (i.e. “A delivery planning method executed by a delivery planning device”) embodied by the elements and limitations of Claims 1-3 and 6. Therefore, Claims 4 and 7-9 are rejected under 35 U.S.C. 101 just as Claims 1-3 and 6 are rejected under 35 U.S.C. 101 as discussed below.
Claims 10-13 (i.e. “A delivery planning method executed by a delivery planning device”) recite elements and limitations that are substantially similar to Claims 4 and 7-9 (i.e. “A computer-readable non-transitory recording medium storing computer- executable program instructions that when executed by a processor cause a computer [(i.e. analogous to delivery planning device of Claim 4)] to execute a delivery planning method . . .”). Therefore, Claims 10-13 are rejected under 35 U.S.C. 101 just as Claims 4 and 7-9 are rejected under 35 U.S.C. 101 as discussed above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4 and 6-13 are rejected under 35 U.S.C. 103 as being unpatentable over EP-3806007-A1 (“Hamzehi-007” Assigned to “Bayerische Moteren Werke AG” and Published 04/14/2021) in view of EP-3916652-A1 (“Hamzehi-652” Assigned to “Bayerische Moteren Werke AG” and Published 12/01/2021).
Regarding Claim 1, Hamzehi-007 teaches “A [transportation] planning device comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, causes the device to perform a set of operations” (Fig. 1a-1c and ¶27 shows “The system 100 [(i.e. planning device)] comprises one or more processing modules 14 [(i.e. processor)] and one or more storage modules 16 [(i.e. memory)] that are coupled to the one or more processing modules 14. The system 100 may optionally comprise one or more interfaces 12 that may be coupled to the one or more processing modules. In general, the system 100 may be configured to execute the method of Figs. 1a and/or 1b.” (Emphasis added). Although Hamzehi-007 shows that the transportation task includes ride hailing or ride sharing in ¶30, Hamzehi-007 does not explicitly teach that the transportation task could include deliveries.) “the set of operations comprising:”
“solving a vehicle routing problem for determining a route for providing a service to a plurality of nodes by a plurality of moving bodies using a neural network that performs reinforcement learning by an Actor-Critic method” (¶30 shows “[A] vehicular task may be a transportation task, e.g. a transportation task of a ride hailing service or of a ride sharing service.” Thus, a transportation task teaches providing a service. Fig. 1b and ¶26 shows “obtaining 110 information on the plurality of vehicular tasks,” “obtaining 120 information on the plurality of vehicles” ((i.e. plurality of moving bodies), “providing 130 the information on the plurality of vehicular tasks and the information on the plurality of vehicles as input to a machine-learning model,” and “assigning 140 the plurality of vehicles to the plurality of vehicular tasks based on an output of the machine-learning model,” and “selecting 145 at least a subset of the plurality of assignments based on a selection criterion, the selection criterion being based on the favorability of the respective assignment.” The resulting assignment of transportation tasks to vehicles teaches determining a route for providing a service. ¶31 shows “The information on the plurality of vehicular tasks may comprise, for each vehicular tasks, the specifics of the vehicular task, e.g. a location of the vehicular task (e.g. starting position and destination in case of a transportation task, a position of a charging or maintenance station) [(i.e. a plurality of nodes)], a time of the vehicular task (e.g. now, at a specific time or within a specific time interval, a maximal delay until start of the vehicular task, a maximal time for picking up additional passengers) and a duration of the vehicular task.” ¶¶28-29 and ¶70 shows that the methods and system of Fig. 1a-1c solves a “Vehicle Routing Problem.” ¶¶36-37 shows that the “machine-learning model” includes “an artificial neural network.” ¶¶83-86 shows the “Actor-Critic Training” algorithm and procedure for the machine learning model.),
“wherein the reinforcement learning is performed to reduce a loss function based on a characteristic of a dense embedding layer and a reward” (This limitation is given its broadest reasonable interpretation in light of the specification (See ¶38 “In the actor network 131, the state of the environment and the state of the delivery vehicle are input to each LSTM cell for each time step. In each agent, the feature amount obtained by the LSTM cell is input to the attention layer. An output from each attention layer is input to a Softmax calculation unit (Softmax), and a value calculated by Softmax is output through Masking and used for reward calculation. In the critic network 132, a loss (Loss Function) is obtained based on the feature amount obtained from the input data by the dense embedding layer and the reward, and learning is performed to reduce the loss.” See also ¶¶35-38 discussing “dense embedding layer). Hamzehi-007 ¶84 shows “The critic network may reduce the variance of occurring gradients during the training process and is trained with stochastic gradient descent on a mean absolute error objective between the actual reward [(i.e. a reward)] and the prediction estimate θ C, [(i.e. a characteristic of a dense embedding layer)] . . .” See also ¶74 showing “The goal is to learn the parameters of a stochastic policy p(Y|X) that assigns high probabilities to routes, while minimizing the distances and maximizing the current fuel state of the vehicle fleet by learning from X sample sequences.” and ¶¶73-82 further discussing use of vectors, encoding, and reducing function in the embedding layer.), and
“determining for each of a plurality of actor networks corresponding to the plurality of moving bodies, learning. As laid out above, in reinforcement learning, one or more software actors (called "software agents") [(i.e. a plurality of actor networks corresponding to the plurality of moving bodies)] are trained to take actions in an environment. In embodiments, the environment may be a simulated environment that is based on the plurality of vehicles [(i.e. based on state of a certain moving boding)], the plurality of vehicular tasks [(i.e. based on state of plurality of nodes)], and a road environment, which may optionally comprise other vehicles or parameters such as traffic as well. The one or more software actors may assign the plurality of vehicular tasks to the plurality of vehicles [(each actor network determines the route based on state of a certain moving body and state of the plurality of nodes)]. Based on the taken actions, i.e. the plurality of assignments, a reward is calculated. In embodiments, the reward function of the reinforcement learning may be based on the objective function. In other words, the objective function may be used to calculate the reward in the reinforcement learning-based training of the machine-learning model. Reinforcement learning is based on training the one or more software agents to choose the actions such, that the cumulative reward is increased, leading to software agents that become better at the task they are given (as evidenced by increasing rewards). In embodiments, the assignments between the vehicles and the vehicular tasks may be repeatedly performed. Between the repetitions, the machine-learning model may be adjusted based on the assignments that have led to a high (or low, depending on implementation) reward.” ¶71 shows “Fig. 4 shows a schematic flow chart of a reinforcement Q (Quality)-learning approach. At reference sign 410, R and Q are initialized, where Q is a matrix that defines the states and actions, and R is the reward function. At 420, the current state is observed. At 430, a random or greedy action is selected from Q. At 440, the action of Q is performed. At 450, the reward is calculated according to R, at 460, the Q-matrix is updated, and the approach returns to reference sign 420 and observes the current state.” ¶72 shows that the operations of Fig. 4 shown in ¶71, which describe the reinforcement learning (i.e. actor-critic method) are “in connection with Figs. 1a to 2c.”),
“wherein each agent in the each of the plurality of actor networks is configured to act alternatively for each time step and the state of the certain moving body and the state of the plurality of nodes are simultaneously updated” (¶83 shows “The following algorithm shows the Actor-Critic Training Procedure:”
PNG
media_image1.png
447
374
media_image1.png
Greyscale
. ¶83 and ¶85 shows that the Actor function (i.e. Ac) is updated for each time step (i.e. train step). Fig. 4 and ¶71 shows “At reference sign 410, R and Q are initialized, where Q is a matrix that defines the states and actions, and R is the reward function. At 420, the current state is observed. At 430, a random or greedy action is selected from Q. At 440, the action of Q is performed. At 450, the reward is calculated according to R, at 460, the Q-matrix is updated, and the approach returns to reference sign 420 and observes the current state.” Because the “observed current state [420]” is updated based on “updated Q matrix [460]” and Q defines both the “states” (plural, i.e. of each agent) and “actions,” Hamzehi-007 teaches that “the state of the certain moving body and the state of the plurality of nodes are simultaneously updated [(i.e. updated, together within a single cycle of Fig. 4)].” See also ¶73 showing nodes include vehicles, requests, and charge stations. Additionally, because the “observed current state [420]” is updated based on “updated Q matrix [460],” which is the result of “perform action [440],” Hamzehi-007 teaches “each agent in the each of the plurality of actor networks is configured to act alternatively [(i.e. “alternately” per 35 U.S.C. 112(b) rejection section above)] for each time step.” Put plainly action of each agent for each cycle of Fig. 4 (i.e. time step) are based on the previous cycle of Fig. 4, and therefore, are performed “[alternately] for each time step.”).
Hamzehi-007 does not explicitly teach, but Hamzehi-652 teaches that “[transportation] planning” includes “delivery planning” (¶4 shows “Routing problems estimate the optimum path to visit a plurality of waypoints based on graph models and maps,” and waypoints may include “sightseeing points, delivery locations, or bus stops.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Hamzehi-652 with Hamzehi-007 because Hamzehi-652 teaches that the advantages of actor-critic machine learning can be applied to delivery planning to better improve delivery operations (¶2 and ¶6). Thus, combining Hamzehi-652 with Hamzehi-007 furthers the interest taught in Hamzehi-652, and therefore, would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 2, Hamzehi-007 and Hamzehi-652 teach “The delivery planning device according to claim 1,” as shown above.
Hamzehi-007 further teaches “wherein a state of each moving body includes at least a position and a loading amount, and a state of each node includes at least a position and a demand” (¶32 shows “The information on the plurality of vehicles may comprise, for each vehicle [(i.e. state of each moving body)], information on temporary properties of the vehicles, such as the current position of the vehicle [(i.e. position information)], a number of taken or available seats of the vehicle [(i.e. loading amount)], a charging status, fuel status or available of the vehicle etc., and/or information on permanent features of the vehicle, such as a boot capacity of the vehicle [(i.e. loading amount)], a number of seats of the vehicle [(i.e. loading amount)], a vehicle type of the vehicle.” ¶31 shows that each task obtained in 110 represents a need for transportation service (i.e. demand) from a starting position to a destination (i.e. position), to be performed within a time frame, which represents urgency (i.e. demand). Additionally, “vehicular tasks” include charging the vehicle at a charging station based on vehicle’s charge (¶¶30-31) and relocation of the vehicle to a new location based on predicted demand (¶30 and ¶54).).
Regarding Claim 3, Hamzehi-007 and Hamzehi-652 teach “The delivery planning device according to claim 1,” as shown above.
Hamzehi-007 further discloses “wherein the algorithm calculation unit performs repeatedly the processing of determining an action and updating a state for each moving body in each time step” (¶54 shows “In embodiments, the assignments between the vehicles and the vehicular tasks may be repeatedly performed. Between the repetitions, the machine-learning model may be adjusted based on the assignments that have led to a high (or low, depending on implementation) reward.” ¶71 shows “In embodiments, reinforcement learning may be used to perform the assignment. Fig. 4 shows a schematic flow chart of a reinforcement Q (Quality)-learning approach. At reference sign 410, R and Q are initialized, where Q is a matrix that defines the states and actions, and R is the reward function. At 420, the current state is observed. At 430, a random or greedy action is selected from Q. At 440, the action of Q is performed. At 450, the reward is calculated according to R, at 460, the Q-matrix is updated, and the approach returns to reference sign 420 and observes the current state.” (emphasis added). ¶56 shows that for each “epoch” the evaluation of assignment may be performed separately for each “batch” (i.e. time step). See also ¶55-57 and ¶¶82-85 showing details training the model for a given epoch.).
Regarding Claim 4, Hamzehi-007 teaches “A [transportation] planning method executed by a [transportation] planning device” (Fig. 1a-1c and ¶27 shows “The system 100 [(i.e. planning device)] comprises one or more processing modules 14 and one or more storage modules 16 that are coupled to the one or more processing modules 14. The system 100 may optionally comprise one or more interfaces 12 that may be coupled to the one or more processing modules. In general, the system 100 may be configured to execute the method of Figs. 1a and/or 1b.” (Emphasis added). Although Hamzehi-007 shows that the transportation task includes ride hailing or ride sharing in ¶30, Hamzehi-007 does not explicitly teach that the transportation task could include deliveries.) “the [transportation] planning method comprising:”
“solving a vehicle routing problem for determining a route for providing a service to a plurality of nodes by a plurality of moving bodies using a neural network that performs reinforcement learning by an Actor-Critic method” (¶30 shows “[A] vehicular task may be a transportation task, e.g. a transportation task of a ride hailing service or of a ride sharing service.” Thus, a transportation task teaches providing a service. Fig. 1b and ¶26 shows “obtaining 110 information on the plurality of vehicular tasks,” “obtaining 120 information on the plurality of vehicles” ((i.e. plurality of moving bodies), “providing 130 the information on the plurality of vehicular tasks and the information on the plurality of vehicles as input to a machine-learning model,” and “assigning 140 the plurality of vehicles to the plurality of vehicular tasks based on an output of the machine-learning model,” and “selecting 145 at least a subset of the plurality of assignments based on a selection criterion, the selection criterion being based on the favorability of the respective assignment.” The resulting assignment of transportation tasks to vehicles teaches determining a route for providing a service. ¶31 shows “The information on the plurality of vehicular tasks may comprise, for each vehicular tasks, the specifics of the vehicular task, e.g. a location of the vehicular task (e.g. starting position and destination in case of a transportation task, a position of a charging or maintenance station) [(i.e. a plurality of nodes)], a time of the vehicular task (e.g. now, at a specific time or within a specific time interval, a maximal delay until start of the vehicular task, a maximal time for picking up additional passengers) and a duration of the vehicular task.” ¶¶28-29 and ¶70 shows that the methods and system of Fig. 1a-1c solves a “Vehicle Routing Problem.” ¶¶36-37 shows that the ”machine-learning model” includes “an artificial neural network.” ¶¶83-86 shows the “Actor-Critic Training” algorithm and procedure for the machine learning model.),
“wherein the reinforcement learning is performed to reduce a loss function based on a characteristic of a dense embedding layer and a reward” (This limitation is given its broadest reasonable interpretation in light of the specification (See ¶38 “In the actor network 131, the state of the environment and the state of the delivery vehicle are input to each LSTM cell for each time step. In each agent, the feature amount obtained by the LSTM cell is input to the attention layer. An output from each attention layer is input to a Softmax calculation unit (Softmax), and a value calculated by Softmax is output through Masking and used for reward calculation. In the critic network 132, a loss (Loss Function) is obtained based on the feature amount obtained from the input data by the dense embedding layer and the reward, and learning is performed to reduce the loss.” See also ¶¶35-38 discussing “dense embedding layer). Hamzehi-007 ¶84 shows “The critic network may reduce the variance of occurring gradients during the training process and is trained with stochastic gradient descent on a mean absolute error objective between the actual reward [(i.e. a reward)] and the prediction estimate θ C, [(i.e. a characteristic of a dense embedding layer)] . . .” See also ¶74 showing “The goal is to learn the parameters of a stochastic policy p(Y|X) that assigns high probabilities to routes, while minimizing the distances and maximizing the current fuel state of the vehicle fleet by learning from X sample sequences.” and ¶¶73-82 further discussing use of vectors, encoding, and reducing function in the embedding layer.), and
“ determining for each actor network in a plurality of actor networks corresponding to the plurality of moving bodies determines the route based on a state of a certain moving body and a state of the plurality of nodes” (¶54 shows “In at least some embodiments, the machine-learning model is trained 220 using reinforcement learning. As laid out above, in reinforcement learning, one or more software actors (called "software agents") [(i.e. a plurality of actor networks corresponding to the plurality of moving bodies)] are trained to take actions in an environment. In embodiments, the environment may be a simulated environment that is based on the plurality of vehicles [(i.e. based on state of a certain moving boding)], the plurality of vehicular tasks [(i.e. based on state of plurality of nodes)], and a road environment, which may optionally comprise other vehicles or parameters such as traffic as well. The one or more software actors may assign the plurality of vehicular tasks to the plurality of vehicles [(each actor network determines the route based on state of a certain moving body and state of the plurality of nodes)]. Based on the taken actions, i.e. the plurality of assignments, a reward is calculated. In embodiments, the reward function of the reinforcement learning may be based on the objective function. In other words, the objective function may be used to calculate the reward in the reinforcement learning-based training of the machine-learning model. Reinforcement learning is based on training the one or more software agents to choose the actions such, that the cumulative reward is increased, leading to software agents that become better at the task they are given (as evidenced by increasing rewards). In embodiments, the assignments between the vehicles and the vehicular tasks may be repeatedly performed. Between the repetitions, the machine-learning model may be adjusted based on the assignments that have led to a high (or low, depending on implementation) reward.” ¶71 shows “Fig. 4 shows a schematic flow chart of a reinforcement Q (Quality)-learning approach. At reference sign 410, R and Q are initialized, where Q is a matrix that defines the states and actions, and R is the reward function. At 420, the current state is observed. At 430, a random or greedy action is selected from Q. At 440, the action of Q is performed. At 450, the reward is calculated according to R, at 460, the Q-matrix is updated, and the approach returns to reference sign 420 and observes the current state.” ¶72 shows that the operations of Fig. 4 shown in ¶71, which describe the reinforcement learning (i.e. actor-critic method) are “in connection with Figs. 1a to 2c.”),
“wherein each agent in the each of the plurality of actor networks is configured to act alternatively for each time step and the state of the certain moving body and the state of the plurality of nodes are simultaneously updated” (¶83 shows “The following algorithm shows the Actor-Critic Training Procedure:”
PNG
media_image1.png
447
374
media_image1.png
Greyscale
. ¶83 and ¶85 shows that the Actor function (i.e. Ac) is updated for each time step (i.e. train step). Fig. 4 and ¶71 shows “At reference sign 410, R and Q are initialized, where Q is a matrix that defines the states and actions, and R is the reward function. At 420, the current state is observed. At 430, a random or greedy action is selected from Q. At 440, the action of Q is performed. At 450, the reward is calculated according to R, at 460, the Q-matrix is updated, and the approach returns to reference sign 420 and observes the current state.” Because the “observed current state [420]” is updated based on “updated Q matrix [460]” and Q defines both the “states” (plural, i.e. of each agent) and “actions,” Hamzehi-007 teaches that “the state of the certain moving body and the state of the plurality of nodes are simultaneously updated [(i.e. updated, together within a single cycle of Fig. 4)].” See also ¶73 showing nodes include vehicles, requests, and charge stations. Additionally, because the “observed current state [420]” is updated based on “updated Q matrix [460],” which is the result of “perform action [440],” Hamzehi-007 teaches “each agent in the each of the plurality of actor networks is configured to act alternatively [(i.e. “alternately” per 35 U.S.C. 112(b) rejection section above)] for each time step.” Put plainly action of each agent for each cycle of Fig. 4 (i.e. time step) are based on the previous cycle of Fig. 4, and therefore, are performed “[alternately] for each time step.”).
Hamzehi-007 does not explicitly teach, but Hamzehi-652 teaches that “[transportation] planning” includes “delivery planning” (¶4 shows “Routing problems estimate the optimum path to visit a plurality of waypoints based on graph models and maps,” and waypoints may include “sightseeing points, delivery locations, or bus stops.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Hamzehi-652 with Hamzehi-007 because Hamzehi-652 teaches that the advantages of actor-critic machine learning can be applied to delivery planning to better improve delivery operations (¶2 and ¶6). Thus, combining Hamzehi-652 with Hamzehi-007 furthers the interest taught in Hamzehi-652, and therefore, would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 6, Hamzehi-007 and Hamzehi-652 teach “The delivery planning device according to claim 1,” as shown above.
Hamzehi-007 further teaches that “whereinis outputted by solving a vehicle routing problem (VRP) problem based on information on each node and each of the plurality of the moving bodies “obtaining 120 information on the plurality of vehicles” (i.e. information on each moving body), “providing 130 the information on the plurality of vehicular tasks and the information on the plurality of vehicles as input to a machine-learning model,” and “assigning 140 the plurality of vehicles to the plurality of vehicular tasks based on an output of the machine-learning model,” and “selecting 145 at least a subset of the plurality of assignments based on a selection criterion, the selection criterion being based on the favorability of the respective assignment.” The resulting assignment of transportation tasks to vehicles teaches outputting a transportation plan by solving a VRP. See also ¶42 showing “The method comprises assigning 140 the plurality of vehicles to the plurality of vehicular tasks based on an output of the machine-learning model. As laid out above, the machine-learning model may be used to evaluate the plurality of assignments.”).
Hamzehi-007 does not explicitly teach, but Hamzehi-652 teaches that “[transportation] plan” includes “delivery plan” (¶4 shows “Routing problems estimate the optimum path to visit a plurality of waypoints based on graph models and maps,” and waypoints may include “sightseeing points, delivery locations, or bus stops.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Hamzehi-652 with Hamzehi-007 because Hamzehi-652 teaches that the advantages of actor-critic machine learning can be applied to delivery planning to better improve delivery operations (¶2 and ¶6). Thus, combining Hamzehi-652 with Hamzehi-007 furthers the interest taught in Hamzehi-652, and therefore, would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 7, Hamzehi-007 and Hamzehi-652 teach “The delivery planning method according to claim 4,” as shown above.
Hamzehi-007 further teaches “wherein a state of each moving body includes at least a position and a loading amount, and a state of each node includes at least a position and a demand” (¶32 shows “The information on the plurality of vehicles may comprise, for each vehicle [(i.e. state of each moving body)], information on temporary properties of the vehicles, such as the current position of the vehicle [(i.e. position information)], a number of taken or available seats of the vehicle [(i.e. loading amount)], a charging status, fuel status or available of the vehicle etc., and/or information on permanent features of the vehicle, such as a boot capacity of the vehicle [(i.e. loading amount)], a number of seats of the vehicle [(i.e. loading amount)], a vehicle type of the vehicle.” ¶31 shows that each task obtained in 110 represents a need for transportation service (i.e. demand) from a starting position to a destination (i.e. position), to be performed within a time frame, which represents urgency (i.e. demand). Additionally, “vehicular tasks” include charging the vehicle at a charging station based on vehicle’s charge (¶¶30-31) and relocation of the vehicle to a new location based on predicted demand (¶30 and ¶54).).
Regarding Claim 8, Hamzehi-007 and Hamzehi-652 teach “The delivery planning method according to claim 4,” as shown above.
Hamzehi-007 further teaches “wherein the algorithm calculation unit performs repeatedly the processing of determining an action and updating a state for each moving body in each time step” (¶54 shows “In embodiments, the assignments between the vehicles and the vehicular tasks may be repeatedly performed. Between the repetitions, the machine-learning model may be adjusted based on the assignments that have led to a high (or low, depending on implementation) reward.” ¶71 shows “In embodiments, reinforcement learning may be used to perform the assignment. Fig. 4 shows a schematic flow chart of a reinforcement Q (Quality)-learning approach. At reference sign 410, R and Q are initialized, where Q is a matrix that defines the states and actions, and R is the reward function. At 420, the current state is observed. At 430, a random or greedy action is selected from Q. At 440, the action of Q is performed. At 450, the reward is calculated according to R, at 460, the Q-matrix is updated, and the approach returns to reference sign 420 and observes the current state.” (emphasis added). ¶56 shows that for each “epoch” the evaluation of assignment may be performed separately for each “batch” (i.e. time step). See also ¶55-57 and ¶¶82-85 showing details training the model for a given epoch.).
Regarding Claim 9, Hamzehi-007 and Hamzehi-652 teach “The delivery planning method according to claim 4,” as shown above.
Hamzehi-007 further teaches “wherein outputting a [transportation] plan by solving a vehicle routing problem (VRP) problem based on information on each node and each of the plurality of the moving bodies ” (¶¶28-29 and ¶70 shows that the methods and system of Fig. 1a-1c solves a “Vehicle Routing Problem.” ¶30 shows “[A] vehicular task may be a transportation task, e.g. a transportation task of a ride hailing service or of a ride sharing service.” Thus, a transportation task teaches providing a service. Fig. 1b and ¶26 shows “obtaining 110 information on the plurality of vehicular tasks” (i.e. information on each node), “obtaining 120 information on the plurality of vehicles” (i.e. information on each moving body), “providing 130 the information on the plurality of vehicular tasks and the information on the plurality of vehicles as input to a machine-learning model,” and “assigning 140 the plurality of vehicles to the plurality of vehicular tasks based on an output of the machine-learning model,” and “selecting 145 at least a subset of the plurality of assignments based on a selection criterion, the selection criterion being based on the favorability of the respective assignment.” The resulting assignment of transportation tasks to vehicles teaches outputting a transportation plan by solving a VRP. See also ¶42 showing “The method comprises assigning 140 the plurality of vehicles to the plurality of vehicular tasks based on an output of the machine-learning model. As laid out above, the machine-learning model may be used to evaluate the plurality of assignments.”).
Hamzehi-007 does not explicitly teach, but Hamzehi-652 teaches that “[transportation] plan” includes “delivery plan” (¶4 shows “Routing problems estimate the optimum path to visit a plurality of waypoints based on graph models and maps,” and waypoints may include “sightseeing points, delivery locations, or bus stops.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Hamzehi-652 with Hamzehi-007 because Hamzehi-652 teaches that the advantages of actor-critic machine learning can be applied to delivery planning to better improve delivery operations (¶2 and ¶6). Thus, combining Hamzehi-652 with Hamzehi-007 furthers the interest taught in Hamzehi-652, and therefore, would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 10, Hamzehi-007 teaches “A computer-readable non-transitory recording medium storing computer- executable program instructions that when executed by a processor cause a computer to execute a [transportation] planning method” (Fig. 1a-1c and ¶27 shows “The system 100 [(i.e. planning device)] comprises one or more processing modules 14 and one or more storage modules 16 that are coupled to the one or more processing modules 14. The system 100 may optionally comprise one or more interfaces 12 that may be coupled to the one or more processing modules. In general, the system 100 may be configured to execute the method of Figs. 1a and/or 1b.” (Emphasis added). Although Hamzehi-007 shows that the transportation task includes ride hailing or ride sharing in ¶30, Hamzehi-007 does not explicitly teach that the transportation task could include deliveries.) “comprising:”
“solving a vehicle routing problem for determining a route for providing a service to a plurality of nodes by a plurality of moving bodies using a neural network that performs reinforcement learning by an Actor-Critic method” (¶30 shows “[A] vehicular task may be a transportation task, e.g. a transportation task of a ride hailing service or of a ride sharing service.” Thus, a transportation task teaches providing a service. Fig. 1b and ¶26 shows “obtaining 110 information on the plurality of vehicular tasks,” “obtaining 120 information on the plurality of vehicles” ((i.e. plurality of moving bodies), “providing 130 the information on the plurality of vehicular tasks and the information on the plurality of vehicles as input to a machine-learning model,” and “assigning 140 the plurality of vehicles to the plurality of vehicular tasks based on an output of the machine-learning model,” and “selecting 145 at least a subset of the plurality of assignments based on a selection criterion, the selection criterion being based on the favorability of the respective assignment.” The resulting assignment of transportation tasks to vehicles teaches determining a route for providing a service. ¶31 shows “The information on the plurality of vehicular tasks may comprise, for each vehicular tasks, the specifics of the vehicular task, e.g. a location of the vehicular task (e.g. starting position and destination in case of a transportation task, a position of a charging or maintenance station) [(i.e. a plurality of nodes)], a time of the vehicular task (e.g. now, at a specific time or within a specific time interval, a maximal delay until start of the vehicular task, a maximal time for picking up additional passengers) and a duration of the vehicular task.” ¶¶28-29 and ¶70 shows that the methods and system of Fig. 1a-1c solves a “Vehicle Routing Problem.” ¶¶36-37 shows that the ”machine-learning model” includes “an artificial neural network.” ¶¶83-86 shows the “Actor-Critic Training” algorithm and procedure for the machine learning model.),
“wherein the reinforcement learning is performed to reduce a loss function based on a characteristic of a dense embedding layer and a reward” (This limitation is given its broadest reasonable interpretation in light of the specification (See ¶38 “In the actor network 131, the state of the environment and the state of the delivery vehicle are input to each LSTM cell for each time step. In each agent, the feature amount obtained by the LSTM cell is input to the attention layer. An output from each attention layer is input to a Softmax calculation unit (Softmax), and a value calculated by Softmax is output through Masking and used for reward calculation. In the critic network 132, a loss (Loss Function) is obtained based on the feature amount obtained from the input data by the dense embedding layer and the reward, and learning is performed to reduce the loss.” See also ¶¶35-38 discussing “dense embedding layer). Hamzehi-007 ¶84 shows “The critic network may reduce the variance of occurring gradients during the training process and is trained with stochastic gradient descent on a mean absolute error objective between the actual reward [(i.e. a reward)] and the prediction estimate θ C, [(i.e. a characteristic of a dense embedding layer)] . . .” See also ¶74 showing “The goal is to learn the parameters of a stochastic policy p(Y|X) that assigns high probabilities to routes, while minimizing the distances and maximizing the current fuel state of the vehicle fleet by learning from X sample sequences.” and ¶¶73-82 further discussing use of vectors, encoding, and reducing function in the embedding layer.),
“ determining for each actor network in a plurality of actor networks corresponding to the plurality of moving bodies actor network determines the route based on state of a certain moving body and state of the plurality of nodes)]. Based on the taken actions, i.e. the plurality of assignments, a reward is calculated. In embodiments, the reward function of the reinforcement learning may be based on the objective function. In other words, the objective function may be used to calculate the reward in the reinforcement learning-based training of the machine-learning model. Reinforcement learning is based on training the one or more software agents to choose the actions such, that the cumulative reward is increased, leading to software agents that become better at the task they are given (as evidenced by increasing rewards). In embodiments, the assignments between the vehicles and the vehicular tasks may be repeatedly performed. Between the repetitions, the machine-learning model may be adjusted based on the assignments that have led to a high (or low, depending on implementation) reward.” ¶71 shows “Fig. 4 shows a schematic flow chart of a reinforcement Q (Quality)-learning approach. At reference sign 410, R and Q are initialized, where Q is a matrix that defines the states and actions, and R is the reward function. At 420, the current state is observed. At 430, a random or greedy action is selected from Q. At 440, the action of Q is performed. At 450, the reward is calculated according to R, at 460, the Q-matrix is updated, and the approach returns to reference sign 420 and observes the current state.” ¶72 shows that the operations of Fig. 4 shown in ¶71, which describe the reinforcement learning (i.e. actor-critic method) are “in connection with Figs. 1a to 2c.”),
“wherein each agent in the each of the plurality of actor networks is configured to act alternatively for each time step and the state of the certain moving body and the state of the plurality of nodes are simultaneously updated” (¶83 shows “The following algorithm shows the Actor-Critic Training Procedure:”
PNG
media_image1.png
447
374
media_image1.png
Greyscale
. ¶83 and ¶85 shows that the Actor function (i.e. Ac) is updated for each time step (i.e. train step). Fig. 4 and ¶71 shows “At reference sign 410, R and Q are initialized, where Q is a matrix that defines the states and actions, and R is the reward function. At 420, the current state is observed. At 430, a random or greedy action is selected from Q. At 440, the action of Q is performed. At 450, the reward is calculated according to R, at 460, the Q-matrix is updated, and the approach returns to reference sign 420 and observes the current state.” Because the “observed current state [420]” is updated based on “updated Q matrix [460]” and Q defines both the “states” (plural, i.e. of each agent) and “actions,” Hamzehi-007 teaches that “the state of the certain moving body and the state of the plurality of nodes are simultaneously updated [(i.e. updated, together within a single cycle of Fig. 4)].” See also ¶73 showing nodes include vehicles, requests, and charge stations. Additionally, because the “observed current state [420]” is updated based on “updated Q matrix [460],” which is the result of “perform action [440],” Hamzehi-007 teaches “each agent in the each of the plurality of actor networks is configured to act alternatively [(i.e. “alternately” per 35 U.S.C. 112(b) rejection section above)] for each time step.” Put plainly action of each agent for each cycle of Fig. 4 (i.e. time step) are based on the previous cycle of Fig. 4, and therefore, are performed “[alternately] for each time step.”).
Hamzehi-007 does not explicitly teach, but Hamzehi-652 teaches that “[transportation] planning” includes “delivery planning” (¶4 shows “Routing problems estimate the optimum path to visit a plurality of waypoints based on graph models and maps,” and waypoints may include “sightseeing points, delivery locations, or bus stops.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Hamzehi-652 with Hamzehi-007 because Hamzehi-652 teaches that the advantages of actor-critic machine learning can be applied to delivery planning to better improve delivery operations (¶2 and ¶6). Thus, combining Hamzehi-652 with Hamzehi-007 furthers the interest taught in Hamzehi-652, and therefore, would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 11, Hamzehi-007 and Hamzehi-652 teach “The computer-readable non-transitory recording medium according to claim 10 wherein the delivery planning method according to claim 10,” as shown above.
Hamzehi-007 further teaches “a state of each moving body includes at least a position and a loading amount, and a state of each node includes at least a position and a demand” (¶32 shows “The information on the plurality of vehicles may comprise, for each vehicle [(i.e. state of each moving body)], information on temporary properties of the vehicles, such as the current position of the vehicle [(i.e. position information)], a number of taken or available seats of the vehicle [(i.e. loading amount)], a charging status, fuel status or available of the vehicle etc., and/or information on permanent features of the vehicle, such as a boot capacity of the vehicle [(i.e. loading amount)], a number of seats of the vehicle [(i.e. loading amount)], a vehicle type of the vehicle.” ¶31 shows that each task obtained in 110 represents a need for transportation service (i.e. demand) from a starting position to a destination (i.e. position), to be performed within a time frame, which represents urgency (i.e. demand). Additionally, “vehicular tasks” include charging the vehicle at a charging station based on vehicle’s charge (¶¶30-31) and relocation of the vehicle to a new location based on predicted demand (¶30 and ¶54).).
Regarding Claim 12, Hamzehi-007 and Hamzehi-652 teach “The computer-readable non-transitory recording medium according to claim 10 wherein the delivery planning method according to claim 10,” as shown above.
Hamzehi-007 further teaches “the algorithm calculation unit performs repeatedly the processing of determining an action and updating a state for each moving body in each time step” (¶54 shows “In embodiments, the assignments between the vehicles and the vehicular tasks may be repeatedly performed. Between the repetitions, the machine-learning model may be adjusted based on the assignments that have led to a high (or low, depending on implementation) reward.” ¶71 shows “In embodiments, reinforcement learning may be used to perform the assignment. Fig. 4 shows a schematic flow chart of a reinforcement Q (Quality)-learning approach. At reference sign 410, R and Q are initialized, where Q is a matrix that defines the states and actions, and R is the reward function. At 420, the current state is observed. At 430, a random or greedy action is selected from Q. At 440, the action of Q is performed. At 450, the reward is calculated according to R, at 460, the Q-matrix is updated, and the approach returns to reference sign 420 and observes the current state.” (emphasis added). ¶56 shows that for each “epoch” the evaluation of assignment may be performed separately for each “batch” (i.e. time step). See also ¶55-57 and ¶¶82-85 showing details training the model for a given epoch.).
Regarding Claim 13, Hamzehi-007 and Hamzehi-652 teach “The computer-readable non-transitory recording medium according to claim 10 wherein the delivery planning method according to claim 10,” as shown above.
Hamzehi-007 further teaches “outputting a [transportation] plan by solving a vehicle routing problem (VRP) problem based on information on each node and each of the plurality of the moving bodies
Hamzehi-007 does not explicitly teach, but Hamzehi-652 teaches that “[transportation] plan” includes “delivery plan” (¶4 shows “Routing problems estimate the optimum path to visit a plurality of waypoints based on graph models and maps,” and waypoints may include “sightseeing points, delivery locations, or bus stops.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Hamzehi-652 with Hamzehi-007 because Hamzehi-652 teaches that the advantages of actor-critic machine learning can be applied to delivery planning to better improve delivery operations (¶2 and ¶6). Thus, combining Hamzehi-652 with Hamzehi-007 furthers the interest taught in Hamzehi-652, and therefore, would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure and is as follows:
“Dense and Sparse Embeddings: A Comprehensive Overview” (Lokhandwala, 08/30/2024, https://mlokhandwalas.medium.com/dense-and-sparse-embeddings-a-comprehensive-overview-c5f6473ee9d0) defines a dense embedding layer.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW PARKER GOODMAN whose telephone number is (571) 272-5698. The examiner can normally be reached on Monday-Thursday from 9:30 AM ET to 6:00 PM ET. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jeffrey Zimmerman, can be reached at telephone number (571) 272-4602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://portal.uspto.gov/external/portal. Should you have questions about access to the Private PAIR
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
/MATTHEW PARKER GOODMAN/Examiner, Art Unit 3628
/JEFF ZIMMERMAN/Supervisory Patent Examiner, Art Unit 3628
Prosecution Insights