DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5 and 17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 5, the claim recites the limitation "the reward value" in line 5. There is insufficient antecedent basis for this limitation in the claim.
Regarding claim 17, the claim recites the limitation "the location data elements" in line 4. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 9-11, 16-17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Mguni et al (United States Patent Application Publication No. US 2021/0319362 A1) [hereinafter “Mguni”] in view of Matzliach et al. (Cooperative Detection of Multiple Targets by the Group of Mobile Agents) [hereinafter “Matzliach”].
Regarding claim 1, Mguni teaches a method of iteratively controlling movement of a plurality of agents by a corresponding plurality of processors (see [0017]), wherein each iteration comprises:
receiving, by at least one agent, an initial map (see at least [0033]-[0036])
applying, by the at least one agent, a Neural Network model on the map to produce one or more Predicted Cumulative Reward values, wherein said PCR values (i) correspond to respective one or more optional movement actions of the at least one agent, and (ii) predict a future cumulative reward representing aggregation of data in the probability map by the plurality of agents (see [0025]-[0037]);
selecting, by the at least one agent, a movement action of the one or more optional movement actions, based on the PCR values (see [0036]-[0038]);
moving the at least one agent according to the selected movement action (see [0036]-[0038]).
Mguni does not teach the initial map is a probability map comprising one or more probability values, each representing probability of location of one or more targets in an area of interest. Mguni further does not teach receiving, by the at least one agent, from at least one first sensor associated with the agent, a target signal indicating a location of at least one target of the one or more targets; and updating the probability map by the at least one agent, based on the received target signal.
Matzliach also generally teaches a method for controlling a swarm of agents (see Abstract and Introduction). Matzliach teaches the swarm can be used to locate a target in an environment, such that the agents are provided with an initial probability map comprising one or more probability values, each representing probability of location of one or more targets in an area of interest (see page 4). Matzliach teaches, similar to Mguni, selecting a movement action for the agents based on an expected reward value associated with the action (see pages 5 and 7-11). Mguni teaches that after the agent moves, the agent observes the environment to detect the target and updates the probability map (see pages 4-5).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by Mguni such that the initial map is a probability map comprising one or more probability values, each representing probability of location of one or more targets in an area of interest and to include steps of receiving, by the at least one agent, from at least one first sensor associated with the agent, a target signal indicating a location of at least one target of the one or more targets; and updating the probability map by the at least one agent, based on the received target signal, in view of Matzliach, as Mguni teaches an effective method of iteratively moving agents in an environment towards a goal and Matzliach teaches similar control of a swarm of agents is effective in locating a target in an environment (see pages 2-3 of Matzliach). Additionally, Mguni teaches that the reward can be based on observation of the environment (see [0045]).
Regarding claim 2, the combination of Mguni and Matzliach further teaches each iteration further comprises receiving, from at least one second sensor associated with the agent, at least one location data element, representing a respective location of the at least one agent (see [0017] and [0033] of Mguni).
Regarding claim 3, the combination of Mguni and Matzliach further teaches the NN model is configured to produce the one or more PCR values based on the probability map and the at least one location data element (see [0028]-[0033] of Mguni and pages 3-5 of Matzliach).
Regarding claim 4, the combination of Mguni and Matzliach further teaches each iteration further comprises: calculating a reward value, representing an amount of data that is added in the updated probability map following the movement of the at least one agent (see pages 7-9 of Matzliach); based on the reward value, calculating an error value that corresponds to the selected movement action, wherein said error value represents an error in the predicted PCR value, and updating one or more weights of the NN model so as to minimize the error value (see [0036]-[0040] of Mguni).
Regarding claim 5, the combination of Mguni and Matzliach further teaches each iteration corresponds to movement of a specific, respective agent (see [0035] of Mguni), and wherein each iteration further comprises: moving the respective agent according to the selected movement action; updating the weights of the NN model based on the reward value, as calculated following movement of the respective agent; and distributing the updated weights of the NN model among the plurality of agents (see [0032]-[0044] of Mguni).
Regarding claim 6, the combination of Mguni and Matzliach further teaches each iteration corresponds to movement of a specific, respective agent (see [0035] of Mguni), and wherein each iteration further comprises: moving the respective agent according to the selected movement action (see [0036]-[0037] of Mguni); updating the probability map based on the target signal of the respective agent; and distributing the updated probability map among the plurality of agents (see [0032]-[0044] of Mguni).
Regarding claim 7, the combination of Mguni and Matzliach further teaches each iteration corresponds to movement of a specific, respective agent (see [0035] of Mguni), and wherein each iteration further comprises: distributing the location data elements of the respective agent, among the plurality of agents; and further applying the NN model on the location data elements of two or more agents, to produce said PCR values (see [0032]-[0044] and [0067]-[0072] of Mguni).
Regarding claim 9, the combination of Mguni and Matzliach further teaches the NN is a reinforcement learning network, and wherein each PCR value represents a cumulative value of rewards of future iterations that is expected until a predefined stop condition is met (see [0020] and [0031]-[0045] of Mguni).
Regarding claim 10, the combination of Mguni and Matzliach further teaches the stop condition comprises having a predefined number of targets represented in the probability map, by a respective number of probability values, that exceed a predefined threshold (see [0020] of Mguni and pages 14-15 of Matzliach).
Regarding claim 11, the combination of Mguni and Matzliach, as applied to claim 1 above, teaches a system for iteratively controlling at least one agent of a plurality of mobile agents, wherein each agent comprises a non-transitory memory device, wherein modules of instruction code are stored, and at least one processor associated with the memory device (see [0017] of Mguni), and configured to execute the modules of instruction code, whereupon execution of said modules of instruction code, the at least one processor is configured to:
receive an initial probability map, comprising one or more probability values, each representing probability of location of one or more targets in an area of interest (see at least [0033]-[0036] of Mguni, page 4 of Matzliach, and the rejection of claim 1 above);
apply a Neural Network model on the probability map to produce, based on the one or more probability values, one or more Predicted Cumulative Reward values, wherein each PCR value (i) corresponds to respective one or more optional movement actions of the at least one agent and (ii) predicts a future cumulative reward representing aggregation of data in the probability map by the plurality of agents (see [0025]-[0037] of Mguni);
select a movement action of the one or more optional movement actions, based on the PCR values (see [0036]-[0038] of Mguni)
move the at least one agent according to the selected movement action (see [0036]-[0038] of Mguni);
receive, from at least one first sensor associated with the agent, a target signal indicating a location of at least one target of the one or more targets (see pages 4-5 and 7-11 of Matzliach and the rejection of claim 1 above); and
update the probability map, based on the received target signal (see pages 4-5 and 7-11 of Matzliach and the rejection of claim 1 above).
Regarding claim 16, the combination of Mguni and Matzliach further teaches each iteration corresponds to movement of a specific, respective selected agent (see [0035] of Mguni), and wherein the at least one processor of the selected agent is configured to, in each iteration: move the respective agent according to the selected movement action (see [0036]-[0037] of Mguni); update the probability map based on the target signal of the respective agent; and distribute the updated probability map among the plurality of agents (see [0032]-[0044] of Mguni).
Regarding claim 17, the combination of Mguni and Matzliach further teaches each iteration corresponds to movement of a specific, respective selected agent (see [0035] of Mguni), and wherein the at least one processor of the selected agent is configured to, in each iteration: distribute the location data elements of the respective agent, among the plurality of agents; and further apply the NN model on the location data elements of two or more agents, to produce said PCR values (see [0032]-[0044] and [0067]-[0072] of Mguni).
Regarding claim 19, the combination of Mguni and Matzliach further teaches the NN is a reinforcement learning network, and wherein each PCR value represents a cumulative value of rewards of future iterations, that is expected until a predefined stop condition is met (see [0020] and [0031]-[0045] of Mguni).
Allowable Subject Matter
Claims 21-25 are allowed.
Claims 8 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is an examiner’s statement of reasons for allowance:
Claim 21 recites the following (emphasis added):
A method of distributed controlling of movement of a plurality of agents, wherein the method comprises:
associating each agent of the plurality of agents with a respective turn;
at each turn, performing the following steps by a processor of the associated agent:
receiving a probability map, comprising one or more probability values, each representing probability of location of one or more targets in an area of interest;
applying a Neural Network (NN) model on the probability map to produce one or more Predicted Cumulative Reward (PCR) values, wherein each PCR value (i) corresponds to a respective optional movement action of the associated agent, and (ii) predicts a future cumulative reward representing aggregation of data in the probability map by a subset of the plurality of agents;
moving the associated agent based on the one or more PCR values;
receiving a signal indicating a location of at least one target in the area of interest;
updating the probability map, based on the received signal; and
transferring the turn to one or more subsequent agents of the plurality of agents.
As noted above, the combination of Mguni and Matzliach teach much of the claimed invention. However, neither Mguni or Matzliach, both alone and in combination, teach that each agent is assigned a designated turn and the turn is transferred to another one or more agents after an iteration performed by one agent.
Mguni specifically wants each agent to act on their own accord in the iterative process, not pausing to allow another agent a turn (see [0019]). Similarly, Matzliach teaches the benefit of having the agents working in parallel rather than in sequential turns (see page 15). Accordingly, there is evidence of non-obviousness for assigning each agent a respective turn and transferring the turn to one or more subsequent agents of the plurality of agents.
Claim 8 recites the following (emphasis added):
The method of claim 1, wherein each iteration corresponds to movement of a specific, first agent, and wherein each iteration further comprises selecting a second agent for a subsequent iteration, based on at least one of: (i) a distance to the first agent from a predefined location in the area of interest, (ii) a distance of the second agent from a predefined location in the area of interest, and (iii) a distance between the first agent and the second agent.
Claim 18 recites similar limitations.
As noted above, neither Mguni or Matzliach, both alone and in combination, teach that the plurality of agents in the swarm are given turns to perform the iterative process. Rather, both teach all agents perform their operations iteratively in parallel with one another.
Accordingly, claims 21-25 are allowed and claims 8 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Guo et al. (US 2023/0083486 A1) generally teaches:
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an environment representation neural network of a reinforcement learning system controls an agent to perform a given task. In one aspect, the method includes: receiving a current observation input and a future observation input; generating, from the future observation input, a future latent representation of the future state of the environment; processing, using the environment representation neural network, to generate a current internal representation of the current state of the environment; generating, from the current internal representation, a predicted future latent representation; evaluating an objective function measuring a difference between the future latent representation and the predicted future latent representation; and determining, based on a determined gradient of the objective function, an update to the current values of the environment representation parameters.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANSHUL SOOD whose telephone number is (571)272-9411. The examiner can normally be reached Monday-Thursday 7-5 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hitesh Patel can be reached at (571) 270-5442. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANSHUL SOOD/ Primary Examiner, Art Unit 3667
Prosecution Insights