DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Claim Interpretation
The term “workflow” holds an ordinary and customary meaning to those of ordinary skill in the art. The term has been interpreted as some combination of activities which seek to accomplish an objective. In light of [0089] of Applicant’s originally filed specification, the activities are not inherently constrained by any particular chronological or other order
The term “work primitive” holds an ordinary and customary meaning to those of ordinary skill in the art. The term has been interpreted as any activity (or “step”, “sub-task”, etc.) that may be performed in combination with other activities to form a larger activity, etc. [0089] of Applicant’s originally filed specification relates.
The term “reusable” holds an ordinary and customary meaning to those of ordinary skill in the art. The term has been interpreted as meaning something capable of being used again or repeatedly. Examiner notes that [0090] of Applicant’s originally filed specification is not considered to amount to a special definition.
The term “percept” holds an ordinary and customary meaning to those of ordinary skill in the art. The term has been interpreted as meaning an impression of an object obtained by use of the senses. Examiner notes that [0099] of Applicant’s originally filed specification is not considered to amount to a special definition.
The term “access” holds multiple ordinary and customary meanings to those of ordinary skill in the art. One might mean to actually open or load (a computer file, etc.), but another might mean to be able to use, enter, or get near.
The term “goal state” holds an ordinary and customary meaning to those of ordinary skill in the art. Furthermore, the term is always relative to another state, etc. Therefore, the term does not appear to hold any inherent chronological or similar meaning without explicit further definition within the claims. The term has been interpreted as meaning “desired or intended state” or similar.
The term Boolean function holds an ordinary and customary meaning to those of ordinary skill in the art. However, while a Boolean function might generally be considered as requiring a binary input as well as a binary output, Applicant explicitly discloses a function that checks for a value within a threshold or range as a Boolean function. See [0128] of Applicant’s originally filed specification, “Such a Boolean function could entail … checking whether the position and/or orientation of the head is within a threshold”. Therefore “Boolean function” only appears to refer to the outputs being in a binary format. Furthermore, the additional recitation of “which returns either true or false” does not appear to further define the term. The Boolean function should return a binary value and therefore always have what could be considered a “true” or “false” value.
MPEP 2111.01 relates.
The nature of “satisfying” and not “satisfying” a metric, whether something is truly based on another, and what exactly constitutes a metric, particularly a combined metric, appears to be more fluid than implied by the claims in light of Applicant’s disclosure.
For example, in [0171] Applicant indicates that “a combined metric may not require that each metric be satisfied. With reference to the above exemplary scenario, the percept “is_self_looking_down()” can be replaced by the percept “is_self_looking_up()”. Given the right configuration and/or nature of sensors (e.g. image sensors with a wide field of view), the robot can be determined as sufficiently “looking down” by determining that the robot is NOT “looking up”.”
This clearly indicates that the metric of interest remains that of “is_self_looking_down()”, and is merely being determined in another manner. Furthermore, it indicates that the true metric is not “is_self_looking_up()”, but is “NOT is_self_looking_up()”. This furthermore indicates that the “combined metric” only requires a loose association with the supposedly combined metrics, as these metrics might be freely manipulated in the consideration of the combined metric, for example through arbitrary use of “NOT”.
For example, the disclosure does not recite retaining is_self_looking_down() and a separate combined metric function wherein 0 = true and 1 = false instead or similar, but instead makes it explicitly clear that it is a sufficiently equivalent function and uses the common NOT programming operator.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1 – 18 and 20 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over Claims 1 – 6 and 8 – 20 of copending Application No. 18/655,623 (reference application). Although the claims at issue are not identical, they are not patentably distinct from each other because they only effectively differ in their category of invention (MPEP 2106). The instant application claims are directed towards a machine, in particular a system, while the reference application claims are directed towards a method. The instant application claims and reference application recite the same general structures of a robot body, sensor, and controller and only differ in that the instant application claims further describe the controller as having a storage medium with instructions thereon. Said instructions are equivalent to the steps of the method claims of the reference application and the method claims clearly recite that each step is performed by a particular component, therefore one of ordinary skill in the art prior to the effective filing date of the claimed invention would find the claims functionally equivalent. Computers and robot controllers are well-understood to operate based off of stored instructions.
Claim 19 is provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 13 (primary) and 11 of copending Application No. copending Application No. 18/655,623 in view of at least Oleynik (US 20230031545 A1).
With respect to Claim 19, Claim 13 of the reference application as shown above is functionally equivalent to Claim 12 of the instant application and Claim 11 of the reference application already discloses the combined metric as a Boolean function. Therefore, there is only minor modification required to teach the minor limitation differences of Claim 19. Appropriate disclosure may be found below with respect to Oleynik, which makes clear the use of Boolean type functions for review of conditions. See also variously the references related to PDDL and similar cited in the Conclusion section. PDDL similarly provides single and combination metrics as Boolean functions. The body of art makes clear that Boolean type evaluation of conditions, predicates, “percepts”, etc. alone or in combination is well-known and routine.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7 – 10 and 12 – 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding Claims 7, 8, and 9, the claims recite the limitation “at least one respective aspect of the first state representation”. It is unclear what “respective” adds to the claim, as “the first state representation” or any “aspect” thereof is not claimed with any particularity. If it refers to some aspect that is supposed to be related to a given percept, no such limitation has been previously provided and it lacks sufficient antecedent basis.
In the interest of compact prosecution, the limitation has been interpreted as reciting “at least one aspect of the first state representation”.
Relatedly, Claims 8 and 10 recite the limitation “each respective aspect of the first state representation” which exhibits the same issue.
Claim 8, after “to:”, has been interpreted as instead reading:
“determine that the first state representation satisfies the combined metric if each respective metric of each respective percept of the plurality of percepts is satisfied by the first state representation; and
determine that the first state representation does not satisfy the combined metric if each respective metric of each respective percept of the plurality of percepts is not satisfied by the first state representation”
Claim 10, has been interpreted as reading:
“which takes the indications of metric satisfaction or metric non-satisfaction as input”
instead of:
“which takes the indications of metric satisfaction or metric non-satisfaction for each respective aspect of the first state representation as input”
Regarding Claim 12, the claim recites the limitation “the respective percept”. There is insufficient antecedent basis for this limitation in the claim. Furthermore, the claim recites “for each reusable work primitive … apply the respective percept”. The claim previously only recites “each percept in the plurality of percepts associated with a respective reusable work primitive”. This means that each percept must have a work primitive it is associated with, but does not mean that each work primitive must have a percept associated with it, or even that each percept is associated with a different work primitive. Consequently, it is not known which percept is referred to, as there may be multiple or none, and in the case of none, it is not possible to perform the final claim limitation.
In the interest of compact prosecution, the claim has been interpreted as reciting “each reusable work primitive having an associated percept in the plurality of percepts” or similar instead of “each percept in the plurality of percepts associated with a respective reusable work primitive.
Regarding Claims 13 – 20, the claims depend from claim(s) rejected above and inherit the deficiencies of said claim(s) as described above. Therefore, Claims 13 – 20 are rejected under the same logic presented above.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 – 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Independent Claim 12 is rejected below and is considered representative of independent Claim 1.
Claim 12 recites:
A robot system comprising:
a robot body;
at least one sensor;
a robot controller which includes at least one processor and at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one processor-readable storage medium storing processor-executable instructions which when executed by the at least one processor cause the robot system to:
identify, by the at least one processor, a workflow to complete a work objective, the workflow comprising a plurality of reusable work primitives available in a library of reusable work primitives;
access each reusable work primitive in the plurality of reusable work primitives;
access a plurality of percepts, each percept in the plurality of percepts associated with a respective reusable work primitive in the plurality of reusable work primitives, each percept in the plurality of percepts comprising a respective metric for evaluating a state representation in relation to the respective reusable work primitive; and
perform the workflow,
wherein for each reusable work primitive in the plurality of reusable work primitives:
the processor-executable instructions further cause the robot system to capture, by the at least one sensor, respective sensor data at a respective time;
the processor-executable instructions further cause the robot system to determine, by the at least one processor, a respective state representation for the respective time, based on the respective sensor data; and
the processor-executable instructions further cause the robot system to apply the respective percept to the respective state representation to determine whether the respective metric is satisfied.
101 Analysis – Step 1: Statutory Category – Yes
The claim recites a method including at least one step. The claim falls within one of the four statutory categories. MPEP 2106.03 relates.
101 Analysis – Step 2A Prong One Evaluation: Judicial Exception – Yes – Mental Processes
In Step 2A, Prong one of the 2019 Patent Eligibility Guidance (PEG), a claim is to be analyzed to determine whether it recites subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) mental processes, and/or c) certain methods of organizing human activity.
The Office submits that the foregoing bolded limitation(s) constitutes judicial exceptions in terms of “mental processes”. MPEP 2106.04(a)(2) relates.
The claim recites limitations of “access”, “identify”, “determine”, and “apply”, with no particular limitations provided which render them unable to be performed in the human mind or by a human using a pen and paper. The act of accessing might simply be a person recalling or remembering information, or even simply having access thereof. The acts of identifying, determining, and applying are all simple judgements. Furthermore, as the claims do not particularly claim a “work primitive”, “workflow”, or similar, under the broadest reasonable interpretation of the claims the “workflow” might also readily be performed in the human mind.
These limitations, as drafted, are a basic process or processes that, under their broadest reasonable interpretation, covers performance of the limitations in the mind but for the recitation of performing these limitations by robot components. That is, other than reciting the bolded limitations nothing in the claim elements precludes the step from practically being performed in the mind. The mere nominal recitation of being performed by a generic computer or components thereof does not take the claim limitations out of the mental process grouping.
Thus, the claim recites mental processes.
101 Analysis – Step 2A Prong Two Evaluation: Practical Application – No
In Step 2A, Prong two of the 2019 PEG, a claim is to be evaluated whether, as a whole, it integrates the recited judicial exception into a practical application. As noted in MPEP 2106.04(d), it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception. The courts have indicated that additional elements such as: merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
The Office submits that the foregoing underlined limitation(s) recite additional elements that do not integrate the recited judicial exception into a practical application.
The claim recites additional elements of “a robot body”, “at least one sensor”, “a robot controller”, “at least one processor”, “at least one non-transitory processor-readable storage medium”, and “instructions” as well as the activity to “capture … sensor data”.
These operations are no more than mere data gathering and outputting which is a judicially recognized insignificant extra-solution activity. See MPEP 2106.05(g).
The “robot body”, “at least one sensor”, “a robot controller”, “at least one processor”, “at least one non-transitory processor-readable storage medium”, and “instructions” appear to merely describe how to generally “apply” the otherwise mental judgements using a generic or general-purpose computer or components thereof. It is recited at a high level of generality and is merely automating certain activities.
The activity to “capture … sensor data” is mere data gathering, a form of insignificant extra-solution activity.
The activity to “perform the workflow” if considered as not possible to perform in the human mind and not an abstract idea, would still appear to be generally linking the abstract idea to a particular technological environment of field of use. Furthermore, even the “link” appears tenuous at best and amounts to merely co-existing in the claim. There does not appear to be any required interaction between the abstract idea and performance of the workflow. Furthermore, even if the judgements were clearly recited as being with respect to the workflow, it would at most appear to be only that; a judgement of the workflow. Nothing is done with or effected by making the judgements.
Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
101 Analysis – Step 2B Evaluation: Inventive Concept – No
In Step 2B of the 2019 PEG, a claim is to be evaluated as to whether the claim, as a whole, amounts to significantly more than the recited exception, i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. MPEP 2106.05 relates.
Under the 2019 PEG, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B.
As discussed with respect to Step 2A Prong Two, the additional elements in the claim, if any, amount to no more than mere instructions to apply the exception using a generic computer. The same analysis applies here in 2B, i.e., mere instructions to apply an exception on a generic computer cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B. Applicant’s specification does not provide any indication that the data-processing hardware is anything other than a conventional computer or components thereof.
Thus, the claim is ineligible.
With respect to Claim 1, the claim recites effectively broader variations of the same limitations and subject matter Claim 12. Claim 1 is therefore rejected under the same logic as Claim 12 above.
With respect to the dependent claims of independent claims 1 and 12 (Claims 2 – 11, and 13 – 20), the claims merely recite additional details to the mental processes recited in Claim 1 or 12 and or additional generic computing components or other extra-solution activities which do not meaningfully alter the rejections made above with respect to Claim 12.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1 – 20 are rejected under 35 U.S.C. 102(a)(1) and 102(a)(2) as being anticipated by Oleynik (US 20230031545 A1).
Regarding Claim 1, Oleynik discloses:
A robot system comprising:
a robot body (See at least “robotic dual-arm system” ([0306]) in Figure 7D);
at least one sensor (See at least sensor units 66 ([0306]) in Figure 7D);
a robot controller (See at least computer (14) ([0306]) in Figure 7D) which includes at least one processor and (See at least onboard processing unit 16 ([0306]) in Figure 7D) at least one non-transitory processor-readable storage medium (See at least media 18 ([0306]) in Figure 7D) communicatively coupled to the at least one processor (See at least Figure 7D), the at least one processor-readable storage medium storing processor-executable instructions (See at created recipe script 19 ([0306]) in Figure 7D) which when executed by the at least one processor cause the robot system to:
access a first reusable work primitive (See at least [0208] disclosing action primitives and [0224] disclosing minimanipulations) from a library of reusable work primitives (See at least [0017] “A minimanipulation library provides a large suite of higher-level sensing-and-execution sequences that are common building blocks for complex tasks, such as cooking, taking care of the infirm, or other tasks performed by the next generation of humanoid robots. More specifically, unlike the previous art, the present disclosure provides the following distinctive features. First, a potentially very large library of pre-defined/pre-learned sensing-and-action sequences called minimanipulations”);
access a first percept associated with the first reusable work primitive, wherein the first percept comprises a first metric for evaluating a state representation in relation to the first reusable work primitive (See at least [0017] “Second, each mini-manipulation encodes preconditions required for the sensing-and-action sequences to produce successfully the desired functional results (i.e. the postconditions) with a well-defined probability of success (e.g. 100% or 97% depending on the complexity and difficulty of the minimanipulation) … Seventh, the assembly of minimanipulations into end-to-end-tasks is performed by robotic planning, taking into account the preconditions and postconditions of the component minimanipulations”);
capture, by the at least one sensor, first sensor data at a first time (See at least [0279] “The quality check refers to three-dimensional vision sensors in the standardized robotic kitchen 50, which monitor and adjust in real time each manipulation action during the food preparation process to correct any deviation and avoid a flawed result”);
determine, by the at least one processor, a first state representation for the first time, based on the first sensor data (See at least [0636] “A first database (database 1) 3194 contains the library of all minimanipulations (MM) known to the robot, including for each MM, a triple <PRE, ACT, POST>, where PRE={s.sub.1, s.sub.2, . . . , s.sub.n} is a set of items in the world state that must be true before the actions ACT=[a.sub.1, a.sub.2, . . . , a.sub.k] can take place, and result in a set of changes to the world state denoted as POST={p.sub.1, p.sub.2, . . . , p.sub.m}. In a preferred embodiment, the MMs are index by … by sensors and actuators they involved”); and
apply, by the at least one processor, the first percept to the first state representation to determine whether the first metric is satisfied (See at least [0381] “Generalized Minimanipulations: A generalized minimanipulation comprises a well-defined sequence of sensing and actuator actions with an expected functional outcome. Associated with each minimanipulation we have a set of pre-conditions and a set of post-conditions. The pre-conditions assert what must be true in the world state in order to enable the minimanipulation to take place. The postconditions are changes to the world state brought about by the minimanipulations”).
Regarding Claim 2, Oleynik discloses:
The robot system of claim 1 wherein the processor-executable instructions further cause the robot system to:
if the first metric is satisfied, output an indication of metric satisfaction; and
if the first metric is not satisfied, output an indication of metric non-satisfaction (Examiner notes that “an indication” is particularly broad, and “satisfaction” is similarly broad. Effectively, the metric simply appears to need to be processed, applied, etc.
See again at least [0636])
Regarding Claim 3, Oleynik discloses:
The robot system of claim 1, wherein the processor-executable instructions further cause the robot system to, prior to the first time:
capture, by the at least one sensor, second sensor data at a second time before the first time;
determine, by the at least one processor, a goal state based on the second sensor data (The nature of “based on” is not claimed with any particularity. Therefore, any association appears to satisfy this requirement. See already disclosed portions of Claim 1.
See alternatively also that the “recipe” to be replicated is based on demonstrated data. The conditions of minimanipulations, etc. are based on prior recorded data);
select, by the at least one processor, the first percept and the associated first reusable work primitive by (Note use of “by” provides that the previous portion is accomplished by the following) identifying the first metric as indicative of whether the robot system is in the goal state (All conditions are evaluated, see again at least [0636]); and
perform the first reusable work primitive to (Use of “to” appears to only indicate an intended result or purpose rather than a positively recited limitation) attempt to transition the robot system towards the goal state (See at least [0390] “In a third embodiment a minimanipulation is successful if its POST conditions match PRE conditions of the next minimanipulation in the robotic task or alternatively the consideration of the context of conditions generated from demonstration and replication of related actions).
Regarding Claim 4, Oleynik discloses:
The robot system of claim 1, wherein the processor-executable instructions further cause the robot system to:
determine, by the at least one processor, a goal state (The nature of “based on” is not claimed with any particularity. Therefore, any association appears to satisfy this requirement. See already disclosed portions of Claim 1, for example [0636].
See alternatively also that the “recipe” to be replicated is based on demonstrated data. The conditions of minimanipulations, etc. are based on prior recorded data);
select, by the at least one processor, the first percept and the associated first reusable work primitive by (Note use of “by” provides that the previous portion is accomplished by the following) identifying the first metric as indicative of whether the robot system is in the goal state (All conditions are evaluated, see again at least [0636]);
if the first state representation satisfies the first metric, determine that the robot system is in the goal state; and
if the first state representation does not satisfy the first metric, perform the first reusable work primitive to (Use of “to” appears to only indicate an intended result or purpose rather than a positively recited limitation) attempt to transition the robot system towards the goal state (See at least [0390] “In a third embodiment a minimanipulation is successful if its POST conditions match PRE conditions of the next minimanipulation in the robotic task, or alternatively the consideration of the context of conditions generated from demonstration and replication of related actions, or again alternatively that the claim is a comprising claim and not exclusive of other actions on the “if” clauses).
Regarding Claim 5, Oleynik discloses:
The robot system of claim 1, wherein the first percept is a Boolean function which returns either true or false (See at least [388] – [0390] providing three examples of condition review, wherein the result is a binary success or unsuccessful, or similar. Examiner notes that these appear to match Applicant’s disclosure in [0128] of Applicant’s originally filed specification)
Regarding Claim 6, Oleynik discloses:
The robot system of claim 1, wherein the first metric indicates a success state representation, and the first metric is satisfied if the first state representation matches the success state representation (See at least [388] “[0388] In one embodiment, when specifying the threshold performance of a minimanipulation, whether generalized or basic, the measurements are performed on the POST conditions, comparing the actual result to the optimal result. For instance, in the task of assembly if a part is positioned within 1% of its desired orientation and location and the threshold of performance was 2%, then the minimanipulation is successful. Similarly, if the threshold were 0.5% in the above example, then the minimanipulation is unsuccessful”).
Regarding Claim 7, Oleynik discloses:
The robot system of claim 1, wherein:
the processor-executable instructions which cause the robot system to access a first reusable work primitive from a library of reusable work primitives cause the robot system to: access a plurality of reusable work primitives from the library of reusable work primitives, including the first reusable work primitive (See again at least [0017] noting the plurality of “minimanipulations” and “primitives”);
the processor-executable instructions which cause the robot system to access the first percept cause the robot system to: access a plurality of percepts including the first percept, each percept of the plurality of percepts associated with a respective reusable work primitive of the plurality of reusable work primitives, wherein each percept comprises a respective metric for evaluating a respective state representation in relation to the respective reusable work primitive (See again at least [0636], in particular “for each MM, a triple <PRE, ACT, POST>” (emphasis added));
the processor-executable instructions which cause the robot system to apply the first percept to the first state representation cause the robot system to, for each percept in the plurality of percepts:
apply the respective percept to at least one respective aspect of the first state representation to (This use of “to” indicates an intended result or purpose rather than a positively recited limitation) determine whether the respective metric is satisfied (See again at least [0636] and [0381]. Each condition is evaluated, and each minimanipulation may have a plurality of pre and post conditions); and
the processor-executable instructions further cause the robot system to determine whether the first state representation satisfies a combined metric represented by the plurality of percepts, based on whether the respective metrics are satisfied (See at least “PRE={s.sub.1, s.sub.2, . . . , s.sub.n}” and “POST={p.sub.1, p.sub.2, . . . , p.sub.m}” of [0636]. Applicant does not define “combined metric” beyond the vague phrasing of “represented by the plurality of percepts” wherein what constitutes “representing” is not provided).
Regarding Claim 8, Oleynik discloses:
The robot system of claim 7, wherein the processor-executable instructions which cause the robot system to determine whether the first state representation satisfies the combined metric cause the robot system to:
determine that the first state representation satisfies the combined metric if each respective aspect of the first state representation satisfies the respective metric of the respective percept applied to the respective aspect of the first state representation; and
determine that the first state representation does not satisfy the combined metric if at least one respective aspect of the first state representation does not satisfy the respective metric of the respective percept applied to the respective aspect of the first state representation (See again at least “a set of items in the world state that must be true” of [0636]).
Regarding Claim 9, Oleynik discloses:
The robot system of claim 7, wherein:
the processor-executable instructions which cause the robot system to apply the first percept to the first state representation further cause the robot system to, for each percept in the plurality of percepts:
if the at least one respective aspect of the first state representation satisfies the respective metric, output an indication of metric satisfaction; and
if the at least one respective aspect of the first state representation does not satisfy the respective metric, output an indication of metric non-satisfaction; and
the processor-executable instructions which cause the robot system to determine whether the first state representation satisfies a combined metric represented by the plurality of percepts cause the at least one processor to determine whether the first state representation satisfies the combined metric based on respective indications of metric satisfaction or metric non-satisfaction for each percept (See again at least “a set of items in the world state that must be true” of [0636]).
Regarding Claim 10, Oleynik discloses:
The robot system of claim 9, wherein:
the processor-executable instructions which cause the robot system to determine whether the first state representation satisfies the combined metric cause the robot system to: evaluate a Boolean logic function which takes the indications of metric satisfaction or metric non-satisfaction for each respective aspect of the first state representation as input (See again at least “PRE={s.sub.1, s.sub.2, . . . , s.sub.n} is a set of items in the world state that must be true before the actions ACT=[a.sub.1, a.sub.2, . . . , a.sub.k] can take place” of [0636], [0017] “Second, each mini-manipulation encodes preconditions required for the sensing-and-action sequences”, and [0299] “This software module uses data … in order to ascertain that the configuration of the robotic kitchen systems and process matches that required by the recipe script (database); if not, it enacts modifications to the commanded system-configuration values to ensure the task is completed successfully”. See also [0627] as needed for more detailed disclosure. Therefore, it is clear that a result of the check of the set of items in the world state is binary).
Regarding Claim 11, Oleynik discloses:
The robot system of claim 1, wherein the processor-executable instructions further cause the robot system to:
capture, by the at least one sensor, second sensor data at a second time subsequent the first time;
determine, by the at least one processor, a second state representation for the second time, based on the second sensor data (Sensor data and world state data are constantly taken in real-time and updated for use in the system. See at least [0302], e.g. “Output data from both engines 316 and 318 are then used to feed the scene modeler and content classifier 320, where the 3D world model is created with all the key content required for executing the robotic cooking script executor”); and
apply, by the at least one processor, the first percept to the second state representation to determine whether the first metric is satisfied (This claim may be satisfied a number of ways.
First, a given recipe process may use the same condition more than once. For example, a pre and post condition might be shared, or a given minimanipulation or action repeated without failure (e.g. [0288], [0399], or [0440]) or repeated due to failure (e.g. [0613]).
Second, any repetition of the entire recipe would read on this limitation (e.g. [0411] for testing or [0029] for the capacity to produce the same recipe on command)).
Regarding Claim 12, Oleynik discloses:
A robot system comprising:
a robot body (See at least “robotic dual-arm system” ([0306]) in Figure 7D);
at least one sensor (See at least sensor units 66 ([0306]) in Figure 7D);
a robot controller (See at least computer (14) ([0306]) in Figure 7D) which includes at least one processor (See at least onboard processing unit 16 ([0306]) in Figure 7D) and at least one non-transitory processor-readable storage medium (See at least media 18 ([0306]) in Figure 7D) communicatively coupled to the at least one processor (See at least Figure 7D), the at least one processor-readable storage medium storing processor-executable instructions (See at created recipe script 19 ([0306]) in Figure 7D) which when executed by the at least one processor cause the robot system to:
identify, by the at least one processor, a workflow to complete a work objective (See at least [0240] “Recipe Script—refers to a recipe script as a sequence in time containing a structure and a list of commands and execution primitives (simple to complex command software) that, when executed by the robotic kitchen elements (robot-arm, automated equipment, appliances, tools, etc.) in a given sequence, should result in the proper replication and creation of the same dish as prepared by the human chef in the studio-kitchen”), the workflow comprising a plurality of reusable work primitives available in a library of reusable work primitives (See at least [0017] “A minimanipulation library provides a large suite of higher-level sensing-and-execution sequences that are common building blocks for complex tasks, such as cooking, taking care of the infirm, or other tasks performed by the next generation of humanoid robots. More specifically, unlike the previous art, the present disclosure provides the following distinctive features. First, a potentially very large library of pre-defined/pre-learned sensing-and-action sequences called minimanipulations”);
access each reusable work primitive in the plurality of reusable work primitives (See again at least [0240]));
access a plurality of percepts, each percept in the plurality of percepts associated with a respective reusable work primitive in the plurality of reusable work primitives, each percept in the plurality of percepts comprising a respective metric for evaluating a state representation in relation to the respective reusable work primitive (See at least [0017] “Second, each mini-manipulation encodes preconditions required for the sensing-and-action sequences to produce successfully the desired functional results (i.e. the postconditions) with a well-defined probability of success (e.g. 100% or 97% depending on the complexity and difficulty of the minimanipulation) … Seventh, the assembly of minimanipulations into end-to-end-tasks is performed by robotic planning, taking into account the preconditions and postconditions of the component minimanipulations”); and
perform the workflow (See at least abstract “a robotic execution module configured for executing the minimanipulation steps by the robotic platform to accomplish a functional result”),
wherein for each reusable work primitive in the plurality of reusable work primitives:
the processor-executable instructions further cause the robot system to capture, by the at least one sensor, respective sensor data at a respective time (See at least [0279] “The quality check refers to three-dimensional vision sensors in the standardized robotic kitchen 50, which monitor and adjust in real time each manipulation action during the food preparation process to correct any deviation and avoid a flawed result”);
the processor-executable instructions further cause the robot system to determine, by the at least one processor, a respective state representation for the respective time, based on the respective sensor data (See at least [0636] “A first database (database 1) 3194 contains the library of all minimanipulations (MM) known to the robot, including for each MM, a triple <PRE, ACT, POST>, where PRE={s.sub.1, s.sub.2, . . . , s.sub.n} is a set of items in the world state that must be true before the actions ACT=[a.sub.1, a.sub.2, . . . , a.sub.k] can take place, and result in a set of changes to the world state denoted as POST={p.sub.1, p.sub.2, . . . , p.sub.m}. In a preferred embodiment, the MMs are index by … by sensors and actuators they involved”); and
the processor-executable instructions further cause the robot system to apply the respective percept to the respective state representation to determine whether the respective metric is satisfied (See at least [0381] “Generalized Minimanipulations: A generalized minimanipulation comprises a well-defined sequence of sensing and actuator actions with an expected functional outcome. Associated with each minimanipulation we have a set of pre-conditions and a set of post-conditions. The pre-conditions assert what must be true in the world state in order to enable the minimanipulation to take place. The postconditions are changes to the world state brought about by the minimanipulations”).
Regarding Claim 13, Oleynik discloses:
The robot system of claim 12, wherein the processor-executable instructions further cause the robot system to, for each reusable work primitive in the plurality of reusable work primitives, prior to each respective time:
determine, by the at least one processor, a respective goal state for the respective reusable work primitive based on the workflow (See again at least [0636]. Conditions are known before taking sensor data); and
perform the respective reusable work primitive to (Use of “to” appears to only indicate an intended result or purpose rather than a positively recited limitation) attempt to transition the robot system towards the respective goal state as part of performing the workflow (Appears inherent to the limitation of Claim 12 of “perform the workflow”).
Regarding Claim 14, Oleynik discloses:
The robot system of claim 12, wherein the processor-executable instructions further cause the robot system to, for each reusable work primitive in the plurality of reusable work primitives:
determine, by the at least one processor, a respective goal state (See again at least [0636]);
if the respective state representation satisfies the respective metric, determine that the robot system is in the respective goal state (See again at least [0636]); and
if the respective state representation does not satisfy the respective metric, perform the respective reusable work primitive to (Use of “to” appears to only indicate an intended result or purpose rather than a positively recited limitation) attempt to transition the robot system towards the respective goal state as part of performing the workflow (Appears inherent to the limitation of Claim 12 of “perform the workflow”).
Regarding Claim 15, Oleynik discloses:
The robot system of claim 12, wherein each percept is a Boolean function which returns either true or false (See at least [388] – [0390] providing three examples of condition review, wherein the result is a binary success or unsuccessful, or similar. Examiner notes that these appear to match Applicant’s disclosure in [0128] of Applicant’s originally filed specification).
Regarding Claim 16, Oleynik discloses:
The robot system of claim 12, wherein:
the processor-executable instructions further cause the robot system to determine whether a combined metric is satisfied based on whether the respective metrics are satisfied (See again at least “a set of items in the world state that must be true” of [0636]); and
the combined metric is represented by the plurality of percepts (See again at least [0636], in particular “for each MM, a triple <PRE, ACT, POST>” (emphasis added), “PRE={s.sub.1, s.sub.2, . . . , s.sub.n}”, and “POST={p.sub.1, p.sub.2, . . . , p.sub.m}”) and is for (This is a statement of purpose rather than a positively recited limitation) evaluating whether a combined state representation in relation to the work objective is satisfied (Applicant does not define “combined metric” beyond the vague phrasing of “represented by the plurality of percepts” wherein what constitutes “representing” is not provided. Relatedly, “combined state representation” is not particularly described or defined. Any “representation” wherein it relates to some combination reads on this term).
Regarding Claim 17, Oleynik discloses:
The robot system of claim 16, wherein the processor-executable instructions which cause the robot system to determine whether the combined metric is satisfied cause the robot system to:
determine that the combined metric is satisfied if each respective state representation satisfies the respective metric of the respective percept applied to the respective state representation (See at least [0381] “The pre-conditions assert what must be true in the world state in order to enable the minimanipulation to take place”); and
determine that the combined metric is not satisfied if at least one respective state representation does not satisfy the respective metric of the respective percept applied to the respective state representation (See again [0381]. Alternatively, see [0627]).
Regarding Claim 18, Oleynik discloses:
The robot system of claim 16, wherein:
the processor-executable instructions which cause the robot system to, for each reusable work primitive in the plurality of reusable work primitives, apply the respective percept to the respective state representation further cause the robot system to, for each reusable work primitive in the plurality of reusable work primitives:
if the respective state representation satisfies the respective metric, output an indication of metric satisfaction for the respective percept satisfaction (Examiner notes that “an indication” is particularly broad, and “satisfaction” is similarly broad. Effectively, the metric simply appears to need to be processed, applied, etc.
See again at least [0636]); and
if the respective state representation does not satisfy the respective metric, output an indication of metric non-satisfaction for the respective percept satisfaction (Examiner notes that “an indication” is particularly broad, and “satisfaction” is similarly broad. Effectively, the metric simply appears to need to be processed, applied, etc.
See again at least [0636]); and
the processor-executable instructions which cause the robot system to determine whether the combined metric is satisfied cause the robot system to: determine whether the combined metric is satisfied based on respective indications of metric satisfaction or metric non-satisfaction for each percept (See again at least “a set of items in the world state that must be true” of [0636]).
Regarding Claim 19, Oleynik discloses:
The robot system of claim 18, wherein:
the processor-executable instructions which cause the robot system to determine whether the combined metric is satisfied cause the robot system to: evaluate a Boolean logic function which takes the indications of metric satisfaction or metric non-satisfaction for each respective percept as input (See again at least “PRE={s.sub.1, s.sub.2, . . . , s.sub.n} is a set of items in the world state that must be true before the actions ACT=[a.sub.1, a.sub.2, . . . , a.sub.k] can take place” of [0636], [0017] “Second, each mini-manipulation encodes preconditions required for the sensing-and-action sequences”, and [0299] “This software module uses data … in order to ascertain that the configuration of the robotic kitchen systems and process matches that required by the recipe script (database); if not, it enacts modifications to the commanded system-configuration values to ensure the task is completed successfully”. See also [0627] as needed for more detailed disclosure. Therefore, it is clear that a result of the check of the set of items in the world state is binary).
Regarding Claim 20, Oleynik discloses:
The robot system of claim 12, wherein the processor-executable instructions further cause the robot system to:
capture, by the at least one sensor, additional sensor data subsequent the respective times;
determine, by the at least one processor, an additional state representation based on the additional sensor data; and
apply, by the at least one processor, a final percept of the plurality of percepts to the additional state representation to determine whether the respective metric for the final percept is satisfied (This claim may be satisfied a number of ways, particularly as “additional” and “final” are extremely broad and there is effectively no relationship to the other claim limitations other than the sensor data being chronologically later in appearance.
For example:
The process may simply be repeated (e.g. [0411] for testing or [0029] for the capacity to produce the same recipe on command).
A given prior portion of a process may be considered as separate. See discussion of “stages” for example in [0658].
The process may be altered in real-time (e.g. [0659]).
A given minimanipulation or action repeated without failure (e.g. [0288], [0399], or [0440]) or repeated due to failure (e.g. [0613])).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Examiner notes that PDDL (Planning Domain Definition Language) and associated works appears to generally encompass the general concepts presented within the claims.
Ijiri et al. (US 20210283771 A1) which has been used as a primary reference in combination with Natarajan et al. (US 20210229281 A1) and Oleynik to reject equivalent claims (see double patenting rejection above) in copending Application No. 18/655,623.
Jeon et al. (Jeon, Jeongmin, et al. "Primitive action based combined task and motion planning for the service robot." Frontiers in Robotics and AI 9 (2022): 713470) which discloses “a CTAMP system in which a symbolic action sequence is generated in task planning, and each action is verified geometrically in motion planning using the off-the-shelf planners and reasoners. The approach is that a set of action models is defined with PDDL in the interface module (action library) and the required information to each planner is automatically provided by the interface module”. See in particular Section 3 and Figure 2.
Kattepur et al. (US 20210049037 A1) which discloses the use of conditioned basic actions. [0043] “For this purpose, the system 100 may also maintain a domain file consisting: (a) Action description, describing a possible action that may be performed in relation to an environment, (b) Precondition, a conjunction of atoms which specify what must be true before an operator may be applied, (c) Effects, is a conjunction of literals which define situation changes after application of an operator”.
Spies et al. (US 20230321826 A1) which discloses sequences of primitives having conditions including a clear combined condition set. See for example Figure 3 and [0026] “Symbolic states 320 that are determined by each sensor modality 330 to be “true” are indicated by a box with a cross-hatched pattern. Symbolic states 320 that are determined to be “false” by a sensor modality 330 are indicated by a box with an empty fill pattern. No box is shown when a sensor modality 330 does not evaluate the corresponding symbolic state 320. In this example, predicates are defined to maximize the performance of each sensor modality 330”
González et al. (González-Santamarta, Miguel Á., et al. "SAILOR: Perceptual Anchoring For Robotic Cognitive Architectures." arXiv preprint arXiv:2303.08204 (2023).) which discloses the use of PDDL which uses conditioned actions and provides perceptual anchoring such that the conditioned actions can be grounded in the perception.
Kroemer et al. (Kroemer, Oliver, Scott Niekum, and George Konidaris. "A review of robot learning for manipulation: Challenges, representations, and algorithms." Journal of machine learning research 22.30 (2021): 1-82.) which discusses primitive actions, pre-conditions, post-conditions, effects, predicates, etc. See for example Section 7.
Zhang et al. (Zhang, Xiaohan, et al. "Grounding classical task planners via vision-language models." arXiv preprint arXiv:2304.08587 (2023).) which uses a vision language model to evaluate the precondition and effects during execution of a task using actions having said features.
Fox et al. (Fox, Maria, Jonathan Gough, and Derek Long. "Detecting execution failures using learned action models." Proceedings of The National Conference on Artificial Intelligence. Vol. 22. No. 2. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2007.) which discloses means for detecting when planned behavior has diverged from expected behavior.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW C GAMMON whose telephone number is (571)272-4919. The examiner can normally be reached M - F 10:00 - 6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ADAM MOTT can be reached on (571) 270-5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW C GAMMON/Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657