Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendments filed 4/22/2026 have been entered. The provide amendments to claims 1, 12, 16 and cancel claims 2 and 13. Claims 1, 3-12, and 14-20 are pending.
Response to Arguments
Applicant’s arguments with respect to 35 U.S.C § 103 filed 4/22/2026 (pages 6-11 of applicant’s arguments) have been fully considered but they are not persuasive.
Applicant argues the combination of Meghani and Zak fails to teach the model of claim 1, specifically that neither of Meghani nor Zak teaches “wherein the modeling comprises modeling with a statistical distribution of the service times based, at least in part, on travel times derived from the locations of the machines and the locations of the service personnel.” The examiner respectfully disagrees. Meghani states “The detailed planning and scheduling involves creating a level 3 hierarchy, assigning components and activities for these components and assigning time frames to these activities, assigning resources to these activities, and aggregating all of these at level 3 nodes for job lengths and the resource needs.” ([0072]) where activities are “activities that will need to be ultimately completed to finish the job listed in level 1.” ([0070). They are creating time frames for the activities that are need to complete a job. This would include travel times. Given the broadest reasonable interpretation of “statistical distribution” the data mentioned by Meghani would fall into the category of a statistical distribution. Applicant also challenges the combination of Meghani and Zak saying it is improper. Both Meghani and Zak deal with artificial intelligence schedulers that are trained using reinforcement learning. Thus, a person of ordinary skill in the art would be motivated to take certain aspects from one invention and integrate them into another invention.
Applicant argues the combination of Meghani and Zak fails to teach the simulation process of claim 1, specifically that neither of Meghani nor Zak teaches “the simulations using different samples from the statistical distribution for the service times”. The examiner respectfully disagrees. Meghani states “The reinforcement engine is trained using large amounts of data that is generated by a data simulation service 710 for different projects and different combinations of work packages, resource needs, timelines, resources/constraints available etc. For example, the data may be generated for close to several hundreds of projects of different sizes, with each project consisting of varying work packages from several hundreds to thousands in numbers of different components, activities, resource requirements, available constraints of equipment's, cranes etc. with the intention of generating industry resemblance data with varying requirements. The data generated by the data simulation service 710 also includes the various domain factors of seasons, complexity and density of installations, height of installations and other such to cover various variable factors which can be seen in the industry while doing the actual planning and constructions.” ([0105]). Given the broadest reasonable interpretation of statistical distribution, it is not limited to a probability distribution. Thus Meghani teaches the simulations for different samples of a statistical distribution.
Applicant argues the combination of Meghani and Nag fails to teach the claimed model of claim 12, specifically that neither Meghani nor Nag teaches: “modeling the service management system, the modeling using a model with state parameters and state transition parameters for the service management system, wherein the state parameters comprise machines, locations of the machines, service personnel, and locations of the service personnel, and wherein the state transition parameters comprise service times and travel times”. The examiner respectfully disagrees. The newly amended claim 12 includes elements of the previously presented claim 13. Zak was used to teach parts of previously present claim 13 as Zak teaches personnel and geographical location. Applicant also argues that the combination of Meghani and Nag is improper. Similarly to the combination of Meghani and Zak, Meghani and Nag are both artificial intelligence schedulers trained by reinforcement learning. Thus, a person of ordinary skill in the art would be motivated to take certain aspects from one invention and integrate them into another invention.
Applicant argues the combination of Meghani and Zak does not teach the system, wherein the stored policy was learned by a specific, technically-defined process as recited in claim 16, specifically that neither Meghani nor Zak teaches “wherein the gamification comprises use of the simulation with a model fit to the service environment, the simulations having used perturbation of distributions and/or sampling parameters of the model as fit to the service environment and resulting changes in performance indicators as rewards in the reinforcement machine learning”. The examiner respectfully disagrees. Meghani states “FIG. 8B shows a feedback loop recurring in reinforcement algorithms. Referring to FIG. 8B, an agent 880 takes an action “a.sub.t” 884 in an environment 886 and receives a reward “r.sub.t” 888 in a state “s.sub.t” 890. Both the Q-learning method and the neural network architechture are trained to predict the Q-function Q(s,a)—e.g., the reward 888 of taking action “a.sub.t” 884 in state “s.sub.t” 890. The Q-table represents the Q-function as a look-up table. That is, the Q-table is an exact mapping of states seen during training to actions, and thus, does not adapt to unseen states and cannot provide predictions on such. However, neural networks that operate as function approximators come into play. Neural networks have the advantages of providing the flexibility to adapt to unseen states, and they scale well to large state action spaces. In the same way as using a Q-table, the reward function for a neural network is used to update the Q-values. A Q-table is learned by using the reward function and the Bellmann equation. A neural network is also learned by using the reward function, but then uses backpropagation to update the network weights.” ([0114]). Thus the model is updated based on rewards from reinforcement machine learning.
Thus, the 103 rejection is maintained.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7, 11, and 16-19 are rejected under 35 U.S.C. 103 as being unpatentable over Meghani (US 2023/0096811 A1) in view of Zak (US 2025/0046436 A1 with PCT filed 12/8/2022).
Regarding claim 1, Meghani teaches:
A method for machine training an artificial intelligence to make recommendations in a service management system, the method comprising: ([0016] “In accordance with another embodiment of the invention, a method for generating an optimum task schedule for fulfilling a large-scale capital project using reinforcement learning”)
modeling the service management system, the modeling being a model including machines, …, service personnel, …, and service times, wherein the modeling comprises modeling with a statistical distribution of the service times based, at least in part, on travel times derived from the locations of the machines and the locations of the service personnel; ([0105] “The reinforcement engine is trained using large amounts of data that is generated by a data simulation service 710 for different projects and different combinations of work packages, resource needs, timelines, resources/constraints available etc.” and [0096] “The projects may include building factories, hospitals, warehouses, shipyards, and the like. The four projects have various requirements of resources, personnel, time demands, and the like.” Also [0072] ““The detailed planning and scheduling involves creating a level 3 hierarchy, assigning components and activities for these components and assigning time frames to these activities, assigning resources to these activities, and aggregating all of these at level 3 nodes for job lengths and the resource needs.” And [0070] ““activities that will need to be ultimately completed to finish the job listed in level 1.”)
machine training, by a processor, the artificial intelligence with reinforcement learning, the artificial intelligence being trained to make the recommendations for service by the service personnel of the machines based on simulations from the modeling of the service system, the simulations using different samples from the statistical distribution for the service times, and based on rewards from a performance indicator from the service times; and ([0078] “The trained reinforcement engine 240 receives the data inputs 210, 220, and 230 and generates schedule alternatives based on a reward function which codifies which and how many constraints are violated versus how many are optimized. This may be set by the simulation environment where the constraints data 220 are given as above, and the agent within the trained reinforcement engine 240 could then explore the action space towards first a feasible and—in a second or more steps—even optimal solution to a schedule 250.” And [0105] ““The reinforcement engine is trained using large amounts of data that is generated by a data simulation service 710 for different projects and different combinations of work packages, resource needs, timelines, resources/constraints available etc. For example, the data may be generated for close to several hundreds of projects of different sizes, with each project consisting of varying work packages from several hundreds to thousands in numbers of different components, activities, resource requirements, available constraints of equipment's, cranes etc. with the intention of generating industry resemblance data with varying requirements. The data generated by the data simulation service 710 also includes the various domain factors of seasons, complexity and density of installations, height of installations and other such to cover various variable factors which can be seen in the industry while doing the actual planning and constructions.”)
storing a policy of the artificial intelligence as trained by the machine training. ([0078] “The trained reinforcement engine 240 receives the data inputs 210, 220, and 230 and generates schedule alternatives based on a reward function which codifies which and how many constraints are violated versus how many are optimized.” This implies the AI policy was stored to be used later).
Meghani does not teach modeling the service management system, the modeling being a model including locations of the machines and locations of the service personnel. However Zak does:
modeling the service management system, the modeling being a model including locations of the machines and locations of the service personnel. ([0009] “A method to optimize human resource management within emergency medical services (EMS) according to one approach may have the steps of: receiving inputs from at least one or more the data sources selected from the list comprising traffic conditions, weather, incident location of emergency or non-emergency call, call type, dispatch type, latitude and longitude of incident location, age, sex, chief complaint, incident date and time, holiday, day of the week, call classification, emergency department population status, incoming EMS service requests, available medical consumables, available medical non consumables, available staff, Cellular triangulation of staff, Cellular triangulation of ambulance or other mobile EMS equipment, Cellular triangulation of service base sites, GPS location of staff, GPS location of the service base sites, GPS location of ambulance or other mobile EMS equipment, identified location of needed services, dispatch requests, time of dispatch requests, latitude and longitude of dispatch requests, hospital census counts, duration of patient admittance in hospital, admitting diagnosis in hospital, discharge diagnosis in hospital, unit transition patterns of patients, unit transition date and time, unit admitting date and time, admitting unit in hospital, on-site manufacturing capabilities; “)
Meghani and Zak are considered analogous art to the claimed invention because they are in the same field of endeavor being reinforcement learning. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the reinforcement learning Meghani with the location tracking of Zak. One would want to do this to create better simulations for the reinforcement learning.
Regarding claim 7, Meghani in view of Zak teaches claim 1 as outlined above. Meghani further teaches:
machine training comprises estimating states, taking actions, and receiving the rewards based on the simulations. ([0114] “Referring to FIG. 8B, an agent 880 takes an action “a.sub.t” 884 in an environment 886 and receives a reward “r.sub.t” 888 in a state “s.sub.t” 890. Both the Q-learning method and the neural network architecture are trained to predict the Q-function Q(s,a)—e.g., the reward 888 of taking action “a.sub.t” 884 in state “s.sub.t” 890.”)
Regarding claim 11, Meghani in view of Zak teaches claim 1 as outlined above. Meghani further teaches:
re-training the policy of the artificial intelligence based on review results for the recommendations by a service manager. ([0090] “The AI auto-scheduler 200 may be further trained by providing user feedback and industry feedback. For example, the AI auto-scheduler 200 may receive feedback relating to execution of the sequenced third level items in the work package and automatically update the sequence of uncompleted third level items and resources needed for completion of such items.”)
Regarding claim 16, Meghani teaches:
A system for machine-learned model service assistance, the system comprising: ([0008] “a system for generating task schedules using an electronic device includes: a processor, the processor comprising neural networks; a memory coupled to the processor”)
a memory configured to store a policy of the machine-learned model, the policy having been learned by reinforcement machine learning in a gamification using simulation of a service environment in combination with the reinforcement machine learning of the policy, wherein the gamification comprises use of the simulation with a model fit to the service environment, the simulations having used perturbation of distributions and/or sampling of parameters of the model as fit to the service environment and resulting changes in performance indicators as rewards in the reinforcement machine learning; ([0078] “The trained reinforcement engine 240 receives the data inputs 210, 220, and 230 and generates schedule alternatives based on a reward function which codifies which and how many constraints are violated versus how many are optimized. This may be set by the simulation environment where the constraints data 220 are given as above, and the agent within the trained reinforcement engine 240 could then explore the action space towards first a feasible and—in a second or more steps—even optimal solution to a schedule 250.” This implies the AI policy was stored to be used later. Also [0114] “FIG. 8B shows a feedback loop recurring in reinforcement algorithms. Referring to FIG. 8B, an agent 880 takes an action “a.sub.t” 884 in an environment 886 and receives a reward “r.sub.t” 888 in a state “s.sub.t” 890. Both the Q-learning method and the neural network architechture are trained to predict the Q-function Q(s,a)—e.g., the reward 888 of taking action “a.sub.t” 884 in state “s.sub.t” 890. The Q-table represents the Q-function as a look-up table. That is, the Q-table is an exact mapping of states seen during training to actions, and thus, does not adapt to unseen states and cannot provide predictions on such. However, neural networks that operate as function approximators come into play. Neural networks have the advantages of providing the flexibility to adapt to unseen states, and they scale well to large state action spaces. In the same way as using a Q-table, the reward function for a neural network is used to update the Q-values. A Q-table is learned by using the reward function and the Bellmann equation. A neural network is also learned by using the reward function, but then uses backpropagation to update the network weights.” This shows the model is updated based on rewards from reinforcement machine learning.
a processor configured to input measurements from the service environment to the policy and to output a recommendation from the policy in response to the input of the measurements; and ([0078] ““The trained reinforcement engine 240 receives the data inputs 210, 220, and 230 and generates schedule alternatives based on a reward function which codifies which and how many constraints are violated versus how many are optimized.”)
Meghani does not teach:
a display configured to display the recommendation from the policy.
However, Zak does:
a display configured to display the recommendation from the policy. ([0009] “displaying to a user the results of the comparison, including the one or more comparison outcomes, and one or more predicted assessment values that relate to the comparison of inputs or to the predicted evolution of data over a time interval using machine learning.”)
Meghani and Zak are considered analogous art to the claimed invention because they are in the same field of endeavor being reinforcement learning. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the reinforcement learning Meghani with the display of Zak. One would want to do this to be able to display the recommendation/scheduling produced by Meghani.
Regarding claim 17, Meghani in view of Zak teaches claim 16 as outlined above. Meghani further teaches:
the policy was learned using rewards based on a key performance indicator of the service environment, ([0079] “Reinforcement learning works by exploring possible actions and receiving feedback for each action in the form of a reward, and by that implicitly learning the underlying logic and dynamics of a system to eventually outperform classical approaches. The agent selects a single data input and sorts it into possible schedules. After each individual action or after several individual actions, e.g. whenever an individual IWP is “completed” (i.e. when no more components get added to the same IWP and the IWP is ready for scheduling) this is sent into a simulation which gives a reward. The reward is a real scalar but can take several aspects into account. A simulation engine could check the output 250 by the agent on whether it is feasible, e.g. whether any physical constraints are violated. Furthermore, the simulation environment puts out a state to the agent. This state defines the remaining components, resources, constraints, etc. The agent then selects a new action based on the updated reward and state.”)
Meghani does not teach:
wherein the display of the recommendation includes an expected value of the key performance indicator given the recommendation and a period for the expected value.
However, Zak does:
wherein the display of the recommendation includes an expected value of the key performance indicator given the recommendation and a period for the expected value. ([0408] “In parallel the patient condition profile is continuously updated during triage and displayed to the dispatcher to aid in determining the right response based on characteristics of the patient condition profile. If it is determined that an EMS unit is needed, the closest available unit is then searched and once a unit become available, it is dispatched to the location of the 911 call. This method performs continuous intelligent analytics to adapt to changes in the EMS unit and receiving entity profile with relation to changing patient condition throughout the patient trajectory. Some inputs to this method may include patient assessment findings, complaints, point of care diagnostics, and the like.”).
Regarding claim 18, Meghani in view of Zak teaches claim 16 as outlined above. Meghani further teaches:
the policy was learned using rewards based on a key performance indicator of the service environment, ([0079] “Reinforcement learning works by exploring possible actions and receiving feedback for each action in the form of a reward, and by that implicitly learning the underlying logic and dynamics of a system to eventually outperform classical approaches. The agent selects a single data input and sorts it into possible schedules. After each individual action or after several individual actions, e.g. whenever an individual IWP is “completed” (i.e. when no more components get added to the same IWP and the IWP is ready for scheduling) this is sent into a simulation which gives a reward. The reward is a real scalar but can take several aspects into account. A simulation engine could check the output 250 by the agent on whether it is feasible, e.g. whether any physical constraints are violated. Furthermore, the simulation environment puts out a state to the agent. This state defines the remaining components, resources, constraints, etc. The agent then selects a new action based on the updated reward and state.”)
Meghani does not teach:
wherein the display of the recommendation includes display of a value of the key performance indicator with no change and a value of the key performance indicator when the recommendation is followed.
However, Zak does:
wherein the display of the recommendation includes display of a value of the key performance indicator with no change and a value of the key performance indicator when the recommendation is followed. ([0408] “In parallel the patient condition profile is continuously updated during triage and displayed to the dispatcher to aid in determining the right response based on characteristics of the patient condition profile. If it is determined that an EMS unit is needed, the closest available unit is then searched and once a unit become available, it is dispatched to the location of the 911 call. This method performs continuous intelligent analytics to adapt to changes in the EMS unit and receiving entity profile with relation to changing patient condition throughout the patient trajectory. Some inputs to this method may include patient assessment findings, complaints, point of care diagnostics, and the like.”).
Regarding claim 19, Meghani in view of Zak teaches claim 16 as outlined above. Zak further teaches:
the processor is configured to adapt the display of the recommendation with a priority based on frequency of assessment of results by a service manager. ([0178] “Suggestions are presented in the map such as optimal ambulance location, specific locations of higher priority call types, and the like.”)
Claims 3-6, 8, 12-14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Meghani in view of Zak and Nag (US 2020/0065703 A1).
Regarding claim 3, Meghani in view of Zak teaches claim 1 as outlined above. Neither Meghani nor Zak teach:
machine training comprises machine training with an adversarial machine-learned agent configured by past training to perturb values of parameters of the model in the simulations such that an adverse reward is received for the adversarial machine-learned agent where the artificial intelligence fails to improve the rewards for the artificial intelligence.
However, Nag does:
machine training comprises machine training with an adversarial machine-learned agent configured by past training to perturb values of parameters of the model in the simulations such that an adverse reward is received for the adversarial machine-learned agent where the artificial intelligence fails to improve the rewards for the artificial intelligence. ([0095] “In this approach, during training, the automated reinforcement-learning-based application manager is directed to select non-optimal, potentially disadvantageous actions at various points in time in order to push the control trajectories into otherwise unexplored regions of the system-state space. In essence, this approach uses the disadvantages suffered by a conventionally trained automated reinforcement-learning-based application manager during live control of a computing environment as advantages during adversarial training. As a result of the disadvantageous actions taken during adversarial training, the automated reinforcement-learning-based application manager is forced to visit a much larger subset of the system states within the system-state space and therefore gain much broader experience, which, in turn, guarantees that the control policy learned during adversarial training is significantly more robust and complete then control policies learned during conventional training.”)
Meghani, Zak and Nag are considered analogous art to the claimed invention because they are in the same field of endeavor being reinforcement learning. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the reinforcement learning Meghani with the location tracking of Zak with the state transition and simulation perturbing of Nag. One would want to do this for better simulations that can be used in reinforcement learning.
Regarding claim 4, Meghani in view of Zak teaches claim 1 as outlined above. Nag further teaches:
modeling comprises representing the service management system as a random process defined over states of the machines, locations of the machines, service personnel, locations of the service personnel, the service times, service personnel shifts, and service agreement information with state transition functions defining probabilities of change in the states. ([0068] “For example, a transition from state 1420 to state 1422 as a result of action 1424 produces observation 1426, while transition from state 1420 to state 1421 via action 1424 produces observation 1428. A second additional detail is that each state transition is associated with a probability.” And Meghani and Zak teaches the machines, locations of the machines, service personnel, locations of the service personnel, the service times, service personnel shifts, and service agreement information as mentioned above.)
Regarding claim 5, Meghani in view of Zak and Nag teaches claim 4 as outlined above. Nag further teaches:
modeling comprises refining the states and the state transition functions based on matching observations from the modeling of the service management system to observations from the service management system. ([0068] “ As indicated by expressions 1434, the function O returns the probability that a particular observation o is returned by the environment given a particular action and the state to which the environment transitions following execution of the action. In other words, in general, there are many possible observations o that might be generated by the environment following transition to a particular state through a particular action, and each possible observation is associated with a probability of occurrence of the observation given a particular state transition through a particular action.”)
Regarding claim 6, Meghani in view of Zak and Nag teaches claim 5 as outlined above. Meghani further teaches:
refining comprises refining based on actions and resulting values of the performance indicator. ([0079] “A simulation engine could check the output 250 by the agent on whether it is feasible, e.g. whether any physical constraints are violated. Furthermore, the simulation environment puts out a state to the agent. This state defines the remaining components, resources, constraints, etc. The agent then selects a new action based on the updated reward and state.”)
Regarding claim 8, Meghani in view of Zak teaches claim 1 as outlined above. Nag further teaches:
machine training comprises the reinforcement learning using perturbation of the modeling in the simulations, the perturbations being for different initial conditions and/or state transitions. ([0095] “In this approach, during training, the automated reinforcement-learning-based application manager is directed to select non-optimal, potentially disadvantageous actions at various points in time in order to push the control trajectories into otherwise unexplored regions of the system-state space. In essence, this approach uses the disadvantages suffered by a conventionally trained automated reinforcement-learning-based application manager during live control of a computing environment as advantages during adversarial training. As a result of the disadvantageous actions taken during adversarial training, the automated reinforcement-learning-based application manager is forced to visit a much larger subset of the system states within the system-state space and therefore gain much broader experience, which, in turn, guarantees that the control policy learned during adversarial training is significantly more robust and complete then control policies learned during conventional training.”)
Regarding claim 12, Meghani teaches:
A method for machine training an artificial intelligence to make recommendations in a service management system, the method comprising: ([0016] “In accordance with another embodiment of the invention, a method for generating an optimum task schedule for fulfilling a large-scale capital project using reinforcement learning”)
storing the policy as trained by the machine training. ([0078] “The trained reinforcement engine 240 receives the data inputs 210, 220, and 230 and generates schedule alternatives based on a reward function which codifies which and how many constraints are violated versus how many are optimized.” This implies the AI policy was stored to be used later).
Meghani does not teach:
modeling the service management system, the modeling using a model with state parameters and state transition parameters for the service management system wherein the state parameters comprise machines, locations of the machines, service personnel, and locations of the service personnel, and wherein the state transition parameters comprise service times and travel times;
machine training, by a processor, a policy with reinforcement learning, the policy being trained to make the recommendations based on simulations using the model, the simulations perturbing sampling of distributions and/or selection of distributions for the state parameters and/or the state transition parameters; and
However, Nag does:
modeling the service management system, the modeling using a model with state parameters and state transition parameters for the service management system wherein the state parameters comprise machines, locations of the machines,…, and wherein the state transition parameters comprise service times and travel times; ([0076] “First, reinforcement learning is used to train an environment simulator 2002 by one or both of operating the simulator against a live-distributed-system environment 2004 or against a simulated distributed-system environment that replays archived data generated by a live distributed system to the simulator 2006.” Where Meghani teaches: [0096] “The projects may include building factories, hospitals, warehouses, shipyards, and the like. The four projects have various requirements of resources, personnel, time demands, and the like.”)
machine training, by a processor, a policy with reinforcement learning, the policy being trained to make the recommendations based on simulations using the model, the simulations perturbing sampling of distributions and/or selection of distributions for the state parameters and/or the state transition parameters; and ([0067] “In the reinforcement-learning approach, the environment is considered to inhabit a particular state at each point in time. The state may be represented by one or more numeric values or character-string values, but generally is a function of hundreds, thousands, millions, or more different variables. The observations generated by the environment and transmitted to the manager reflect the state of the environment at the time that the observations are made. The possible state transitions can be described by a state-transition diagram for the environment. FIG. 14A illustrates a portion of a state-transition diagram.”)
Meghani and Nag are considered analogous art to the claimed invention because they are in the same field of endeavor being reinforcement learning. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the reinforcement learning Meghani with the state transition and simulation perturbing of Nag. One would want to do this for better simulations that can be used in reinforcement learning.
Neither Meghani nor Nag teach modeling the service management system, the modeling being a model including locations of the machines and locations of the service personnel. However Zak does:
modeling the service management system, the modeling being a model including locations of the machines and locations of the service personnel. ([0009] “A method to optimize human resource management within emergency medical services (EMS) according to one approach may have the steps of: receiving inputs from at least one or more the data sources selected from the list comprising traffic conditions, weather, incident location of emergency or non-emergency call, call type, dispatch type, latitude and longitude of incident location, age, sex, chief complaint, incident date and time, holiday, day of the week, call classification, emergency department population status, incoming EMS service requests, available medical consumables, available medical non consumables, available staff, Cellular triangulation of staff, Cellular triangulation of ambulance or other mobile EMS equipment, Cellular triangulation of service base sites, GPS location of staff, GPS location of the service base sites, GPS location of ambulance or other mobile EMS equipment, identified location of needed services, dispatch requests, time of dispatch requests, latitude and longitude of dispatch requests, hospital census counts, duration of patient admittance in hospital, admitting diagnosis in hospital, discharge diagnosis in hospital, unit transition patterns of patients, unit transition date and time, unit admitting date and time, admitting unit in hospital, on-site manufacturing capabilities; “)
Meghani, Zak and Nag are considered analogous art to the claimed invention because they are in the same field of endeavor being reinforcement learning. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the reinforcement learning Meghani with the location tracking of Zak with the state transition and simulation perturbing of Nag. One would want to do this for better simulations that can be used in reinforcement learning.
Regarding claim 14, Meghani in view of Zak and Nag teaches claim 12 as outlined above. Nag further teaches:
machine training comprises machine training with an adversarial machine-learned agent configured by past training to perturb values of the state parameters and/or the state transition parameters of the model in the simulations such that an adverse reward is received where the policy fails to improve rewards of the reinforcement learning. ([0095] “In this approach, during training, the automated reinforcement-learning-based application manager is directed to select non-optimal, potentially disadvantageous actions at various points in time in order to push the control trajectories into otherwise unexplored regions of the system-state space. In essence, this approach uses the disadvantages suffered by a conventionally trained automated reinforcement-learning-based application manager during live control of a computing environment as advantages during adversarial training. As a result of the disadvantageous actions taken during adversarial training, the automated reinforcement-learning-based application manager is forced to visit a much larger subset of the system states within the system-state space and therefore gain much broader experience, which, in turn, guarantees that the control policy learned during adversarial training is significantly more robust and complete then control policies learned during conventional training.”)
Regarding claim 20, Meghani in view of Zak teaches claim 16 as outlined above. Neither of them teach:
the gamification comprised use of the simulation with a model fit to the service environment, the simulations having used perturbation of distributions and/or sampling of parameters of the model as fit to the service environment and resulting changes in performance indicators as rewards in the reinforcement machine learning.
However, Nag teaches
the gamification comprised use of the simulation with a model fit to the service environment, the simulations having used perturbation of distributions and/or sampling of parameters of the model as fit to the service environment and resulting changes in performance indicators as rewards in the reinforcement machine learning. ([0067] “In the reinforcement-learning approach, the environment is considered to inhabit a particular state at each point in time. The state may be represented by one or more numeric values or character-string values, but generally is a function of hundreds, thousands, millions, or more different variables. The observations generated by the environment and transmitted to the manager reflect the state of the environment at the time that the observations are made. The possible state transitions can be described by a state-transition diagram for the environment. FIG. 14A illustrates a portion of a state-transition diagram.”)
Meghani, Zak and Nag are considered analogous art to the claimed invention because they are in the same field of endeavor being reinforcement learning. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the reinforcement learning Meghani with the location tracking of Zak with the state transition and simulation perturbing of Nag. One would want to do this for better simulations that can be used in reinforcement learning.
Claims 9 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Meghani in view of Zak and Gopalan (US 2018/0285772 A1).
Regarding claim 9, Meghani in view of Zak teaches claim 1 as outlined above. Neither Meghani or Zak teaches:
comprising updating the model with statistical testing of the service times and/or other parameters of the model.
However, Gopalan does:
comprising updating the model with statistical testing of the service times and/or other parameters of the model. ([0013] “In addition, examples of the present disclosure may include a root cause analysis (RCA)—which identifies what features are most responsible for the accuracy of the machine learning model with respect to the service (e.g., classification, clustering, mapping, etc.). RCA includes a quantitative statistical analysis on historical data to determine relevance scores on the impact of the features/variables to the accuracy of the machine learning model.”)
Meghani, Zak and Gopalan are considered analogous art to the claimed invention because they are in the same field of endeavor being reinforcement learning. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the reinforcement learning Meghani with the location tracking of Zak with the retraining of Gopalan One would want to do this for another method of training the model
Regarding claim 10, Meghani in view of Zak teaches claim 1 as outlined above. Gopalan further teaches:
comprising re-training the policy of the artificial intelligence when an actual distribution of a parameter of the model is a threshold difference from a distribution or distributions used in the machine training. ([0016] “retraining a machine learning model when a counter for a likelihood of new data based upon a training data set being less than a first threshold exceeds a second threshold may operate in accordance with the present disclosure.”)
Claims 15 is rejected under 35 U.S.C. 103 as being unpatentable over Meghani in view of Zak, Nag and Gopalan.
Regarding claim 15, Meghani in view of Zak and Nag teaches claim 12 as outlined above. None of them teach:
comprising updating the model with statistical testing the state transition parameters of the model and/or with replacement of values of the state parameters.
However, Gopalan does:
comprising updating the model with statistical testing the state transition parameters of the model and/or with replacement of values of the state parameters. ([0013] “In addition, examples of the present disclosure may include a root cause analysis (RCA)—which identifies what features are most responsible for the accuracy of the machine learning model with respect to the service (e.g., classification, clustering, mapping, etc.). RCA includes a quantitative statistical analysis on historical data to determine relevance scores on the impact of the features/variables to the accuracy of the machine learning model.”)
Meghani, Nag and Gopalan are considered analogous art to the claimed invention because they are in the same field of endeavor being reinforcement learning. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the reinforcement learning Meghani with the state transition and simulation perturbing of Nag with the retraining of Gopalan One would want to do this for another method of training the model.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL P GRUSZKA whose telephone number is (571)272-5259. The examiner can normally be reached M-F 9:00 AM - 6:00 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL GRUSZKA/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights