DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The amendment filed 5/5/2026 has been entered. Claims 1-7 are amended. Claims 8-20 are newly added. Claims 1-20 are pending in the application. Applicant’s amendments to the specification, drawings, and claims have overcome each and every objection and 112(a) and (b) rejection set forth in the Non-Final Office Action mailed 11/5/2025 except as otherwise indicated below. New objections and 112(b) rejections are provided in view of the amendments.
Applicant’s amendments remove the “anti-collision subsystem” and “self-protection module” language previously interpreted under 112(f).
Applicant’s arguments, see pages 14-15, with respect to the 112(a) rejection of claim 1’s use of Artificial Intelligence (AI) algorithms if fully considered and is persuasive. The respective 112(a) rejection is withdrawn.
Applicant’s arguments, see page 16, with respect to the 112(a) rejection of claim 1 for lack of description for what the resources are is fully considered but is unpersuasive. The applicant points to paragraph [0021] as a concreter example of resource data. However, the paragraph does not actually describe what the resource are, only how they are obtained. The respective 112(a) rejection is maintained.
Applicant’s arguments, see pages 16-19, with respect to the cited prior art not teaching the amended subject matter is fully considered and is persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Oleynik (US 20190291277 A1), Willför (US 20160207198 A1) and Gautier (US 20240001542 A1).
Claim Objections
Claim 12 is objected to because of the following informalities:
Claim 12 recites “training the A in simulation environment.” This should recite “training the AI in simulation environment.”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-7 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 recites “the control platform comprising consumption of process, product, environment and resources data” and “(vi) Control and monitoring of resources.” The description lacks description for what the resources are. Resources is a very broad term and its not understood to one of ordinary skill in the art as to what resource data comprises and what is being controlled in the step of “(vi) Control and monitoring of resources.” Paragraph [0030] recites “the resources and parameters of the autonomous systems during the execution of the generated trajectories are controlled and monitored, with the purpose of ensuring the integrity of the product and of the resources used.” Paragraph [0035] recites “the generation of data for history, which is subsequently used for generating new, improved trajectories, provides greater precision in the use of resources and better product quality.” Accordingly, it is described that the resources are related to a trajectory. However, the specification repeats the term “resources” numerous times without providing any examples or definitions. Accordingly claim 1 is rejected under 112(a) as lacking written description. Claims 2-7 are also rejected because they do not resolve the deficiencies of claim 1.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5, 6, 9, 12, 13, and 16-18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 5, 13, and 16 recite “a plurality of sensors.” It is unclear whether these sensors are the same sensors as those claimed in the parent claims 1 and 10. For examination purposes, these dependent claims are interpreted as reciting “[[a]] the plurality of sensors.”
Claims 6 and 17 recite “a warning area for recalculation of trajectory and an immediate stop area.” It is unclear whether these areas are the same as the “alert and stop zones” recited in parent claims 1 and 10. If the areas and zones are distinct from each other, then there is a 112(a) new matter issue because the original disclosure does not support different alert zones, stop zones, warning areas, and immediate stop areas. However, for examination purposes, the warning area is interpreted as the alert zone and the immediate stop area is interpreted as the stop zone. Claim 18 is also rejected because it does not resolve the deficiencies of claim 17.
Claims 9 and 12 recite “a digital twin of the product/process.” It is unclear whether this digital twin is the same as the one claimed in the parent claims 1 and 10. For examination purposes, these dependent claims are interpreted as reciting “[[a]] the digital twin of the product/process.”
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-8, 10-11, and 13-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Oleynik (US 20190291277 A1) in view of Willför (US 20160207198 A1) and Gautier (US 20240001542 A1).
Regarding Claim 1,
Oleynik teaches
A control platform for autonomous systems, (“Systems and methods are provided for operating universal robotic assistant systems.” See at least [0008])
the control platform consuming process, product, environment and resources data of a digital model for automatic generation of trajectories for autonomous systems, (“A minimanipulation library provides a large suite of higher-level sensing-and-execution sequences that are common building blocks for complex tasks.” See at least [0041], wherein the minimanipulation library is the data of a digital model.; “a database library structure 972 of minimanipulation objects for use in the standardized robotic kitchen. The database library structure 972 shows several fields for entering and storing information for a particular minimanipulation, including (1) the name of the minimanipulation, (2) the assigned code of the minimanipulation, (3) the code(s) of standardized equipment and tools associated with the performance of the minimanipulation, (4) the initial position and orientation of the manipulated (standard or non-standard) objects (ingredients and tools), (5) parameters/variables defined by the user (or extracted from the recorded recipe during execution), (6) sequence of robotic hand movements (control signals for all servos) and connecting feedback parameters (from any sensor or video monitoring system) of minimanipulations on the timeline.” See at least [0535]; See at least [0786], describing metrics such as time required and energy-expended corresponding to stored mini-manipulation action primitive (AP) components, wherein the metrics are historical resource data in a database.; “The method of mini-manipulation command generation for one or both the macro- or micro-manipulation subsystems, comprises receiving a high-level task execution command, identifying individual subtasks which will be mapped to the applicable robotic subsystems, generation of individual performance criteria and measurable success end-state criteria for each of the above subtasks, selection of one or more in either a stand-alone or combination, of the most suitable action primitive candidates, evaluation of these action primitive alternatives for maximizing or minimizing such measures as execution-time, energy expended, robot reachability, collision avoidance or any other task-critical criteria, generation of either or both macro- and/or micro-manipulation subsystem trajectories in one or more motion spaces.” See at least [1289])
the control platform for autonomous systems comprising at least one processor and/or processing circuit connected to at least one memory storing information, the processor and/or processing circuit at least in part configured by the information the at least one memory stores to perform operations comprising: (“a computer device, as shown in 4324, on which computer-executable instructions to perform the methodologies discussed herein may be installed and run. … The example computer system 4324 includes a processor 4326 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 4328 and a static memory 30, which communicate with each other via a bus 4332. … The disk drive unit 4340 includes a machine-readable medium 244 on which is stored one or more sets of instructions (e.g., software 4346) embodying any one or more of the methodologies or functions described herein.” See at least [1271-1272])
(i) use a digital model and physical media reference data to provide a digital twin of the product/process; (“the position data and the image data are obtained from the one or more sensors, wherein the one or more sensors comprises at least one of a navigation system and one or more image capturing devices. In some embodiments, detecting the one or more objects is based on at least one of the type of the current environment, the environment data corresponding to the current environment, and object data.” See at least [0012-0013], wherein the sensor data is physical media reference data.; “Upon completion of the chef studio recipe creation and cooking process by the chef, the robotic cooking engine generates a simulation visualization program 1954 replicating the movement and media data used for later recipe replication by a remote standardized robotic kitchen system. Based on the raw and processed data, and a confirmation of the simulated recipe execution visualization by the chef, hardware-specific applications are developed and integrated for different (mobile) operating systems” See at least [0583], wherein the simulation is a digital twin.)
(ii) application of historical data from processes, product, environment and resources located in a database; (“integration of electronic libraries of mini-manipulations with transformed robotic instructions for replicating movements, processes, and techniques with real-time electronic adjustments.” See at least [0002], wherein the electronic libraries of mini-manipulations includes historical processes, product, environment and resources data according to the citations above (See at least [0535] and [0786]).)
(iii) control of digital information, the physical media reference data and the historical data combined for automatic generation of trajectories for autonomous systems providing a digital twin of the product/process; (“During the food preparation process, the robotic apparatus 75 uses touch signals generated by sensors in the fingertips and the palms of a robot's hands to detect force, temperature, humidity and toxicity as the robot replicates step-by-step movements and compares the sensed values with the tactile profile of the chef's studio cooking program. Visual sensors help the robot to identify the surroundings and take appropriate cooking actions. The robotic apparatus 75 analyzes the image of the immediate environment from the visual sensors and compares it with the saved image of the chef's studio cooking program, so that appropriate movements are made to achieve identical results.” See at least [0509]; “Raw data is collected at each point in time to allow the raw data to be processed to be able to extract the shape, dimension, location and orientation of all objects of importance to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50 in a step 1162. The processed data is further analyzed by the computer system to allow the controller of the standardized robotic kitchen to adjust robotic arm and hand trajectories and minimanipulations, by modifying the control signals defined by the robotic script. Adaptations to the recipe-script execution and thus control signals is essential in successfully completing each stage of the replication for a particular dish, given the potential for variability for many variables (ingredients, temperature, etc.).” See at least [0547]; See at least [0583] for the simulation/digital twin.; Examiner Interpretation: Adapting the control of the robot using the stored minimanipulation data (historical data) and the raw sensor data (reference data) is equivalent to the control of digital information.)
(v) automatically generate the trajectories of the autonomous systems; (“Raw data is collected at each point in time to allow the raw data to be processed to be able to extract the shape, dimension, location and orientation of all objects of importance to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50 in a step 1162. The processed data is further analyzed by the computer system to allow the controller of the standardized robotic kitchen to adjust robotic arm and hand trajectories and minimanipulations, by modifying the control signals defined by the robotic script.” See at least [0547])
(vi) use of Artificial Intelligence (AI) algorithms for control and modification of the automatically generated trajectories of the autonomous systems; (“The present disclosure relates to fields of robotics and artificial intelligence (AI). … integration of electronic libraries of mini-manipulations with transformed robotic instructions for replicating movements, processes, and techniques with real-time electronic adjustments.” See at least [0002]; “learning algorithms monitor each and every motion/interaction sequence and perform simple variable-perturbations to ascertain outcome to decide on if/how/when/what variable(s) and sequence(s) to modify in order to achieve a higher level of execution fidelity at levels ranging from low-to high-levels of various MMLs.” See at least [0052]; Also see at least [0383], [0992], and [0996-0998] for reinforcement learning.)
(vii) control and monitoring of resources and parameters; and (viii) automatic inspection and generation of historical data from processes, product, environment and resources for database feedback. (“The architecture of the software-module/action layer provides a framework that allows the inclusion of: (1) refined Endeffector sensing (for refined and more accurate real-world interface sensing); (2) introduction of the macro-(overall sensing by and from the articulated base) and micro-(local task-specific sensing between the endeffectors and the task-/cooking-specific elements) tiers to allow continuous minimanipulation libraries to be used and updated (via learning) based on a physical split between coarse and fine manipulation (and thus positioning, force/torque control, product-handling and process monitoring);” See at least [0767]; “Said motion commands are sequentially fed to an execution block 3613, which controls all instrumented articulated and actuated joints in at least joint- or Cartesian space to ensure the movements track the commanded trajectories in position/velocity and/or torque/force. A feedback sensing block 3614 provides feedback data from all sensors to the execution block 3613 as well as an environment perception block/module 3611 for further processing. Feedback is not only provided to allow tracking the internal state of variables, but also sensory data from sensor measuring the surrounding environment and geometries. Feedback data from said module 3614 is used by the execution module 3613 to ensure actual values track their commanded setpoints, as well as an environment perception module 3611 to image and map, model and identify the state of each articulated element, the overall configuration of the robot as well as the state of the surrounding environment the robot is operating in. Additionally, said feedback data is also provided to a learning module 3615 responsible for tracking the overall performance of the system and comparing it to known required performance metrics, allowing one or more learning methods to develop a continuously updated set of descriptors that define all mini-manipulations contained within their respective mini-manipulation library 3630, in this case the macro-level mini-manipulation sublibrary 3631.” See at least [0772]; Examiner Interpretation: The feedback/sensing is the monitoring and inspection. The feedback data is generated for updating the minimanipulation library/database (learning) and therefore is generated for database feedback.)
Oleynik does not explicitly teach, but Willför teaches
(iv) action of a volumetric anti-collision tester with (“the device further includes a camera 20 configured to capture an image of the real robot and its environment, and the augmented reality unit 18 is configured to register the graphical representations of the safety volumes on the image of the real robot and its environment to provide the composited augmented reality image. The display unit 5 visualizes the view of the camera 20 combined with computer generated graphics of the safety volumes.” See at least [0055]; “FIG. 5 shows an example of graphical presentation of a moving safety volume 60 along a movement path 62 defined by a plurality of target points 64.” See at least [0070] and fig. 5; “If the safety volume and the fixed safety volume instead cease to overlap each other on a part of the robot path, it is necessary to change the size of the fixed safety volume or change the programmed movement path to prevent the mechanical unit from reaching the border of the fixed safety volume during execution of the programmed path and by that being emergency stopped.” See at least [0084])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of Oleynik to further include the teachings of Willför with a reasonable expectation of success such that “programming and verification of the fixed safe modules will be more intuitive and faster.” (See at least [0017])
Willför also does not explicitly teach, but Gautier teaches
alert zones (“This alert sub-module 92 allows to inform the user 7 that the motion required for the robotic device or devices 2 to complete the trajectory of the effector or effectors 4 enters an alert area in which the trajectory is not technically feasible by the robotic device or devices 2. This alert sub-module 92 may also inform the user 7 that the trajectory of the effector or effectors 4 may enter an alert area where the trajectory collides with an element 31 of the workspace 3. In order for the alert to be given to the user 7, the alert sub-module 92 may comprise a visual alert unit and/or a haptic alert unit and/or an acoustic unit.” See at least [0092])
monitoring from a plurality of sensors (“the trajectory capture module 8 may comprise at least one video sensor 81, such as a camera (FIG. 6). This video sensor or sensors 81 captures images of the positions that the pointer or pointers 6 take while being driven. The trajectory of the pointer 6 or of each of the pointers 6 is then determined by processing the images captured by the video sensors.” See at least [0083])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of Oleynik and Willför to further include the teachings of Gautier with a reasonable expectation of success to alert the user of conflicts such that “a user can easily and quickly program one or more robotic devices of a robotic system” (See at least [0014]) and “the user can ensure that the trajectories they program are compatible with the workspace.” (See at least [0017])
Regarding Claim 2,
Oleynik further teaches
wherein the physical media reference data comprises results from the plurality of sensors displayed in real environments wherein the autonomous systems perform the automatically generated trajectories. (“the position data and the image data are obtained from the one or more sensors, wherein the one or more sensors comprises at least one of a navigation system and one or more image capturing devices. In some embodiments, detecting the one or more objects is based on at least one of the type of the current environment, the environment data corresponding to the current environment, and object data.” See at least [0012-0013], wherein the sensor data is physical media reference data.; “Raw data is collected at each point in time to allow the raw data to be processed to be able to extract the shape, dimension, location and orientation of all objects of importance to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50 in a step 1162. The processed data is further analyzed by the computer system to allow the controller of the standardized robotic kitchen to adjust robotic arm and hand trajectories … The process of recipe-script execution based on key measurable variables is an essential part of the use of the augmented (also termed multi-modal) sensor system 20 during the execution of the replicating steps for a particular dish in a standardized robotic kitchen 50.” See at least [0547])
Regarding Claim 3,
Oleynik further teaches
wherein the operations further include comparing physical media reference data with digital model data for feeding the control platform with confirmations or alterations of the automatically generated trajectories providing a digital twin of the product/process. (“During the food preparation process, the robotic apparatus 75 uses touch signals generated by sensors in the fingertips and the palms of a robot's hands to detect force, temperature, humidity and toxicity as the robot replicates step-by-step movements and compares the sensed values with the tactile profile of the chef's studio cooking program. Visual sensors help the robot to identify the surroundings and take appropriate cooking actions. The robotic apparatus 75 analyzes the image of the immediate environment from the visual sensors and compares it with the saved image of the chef's studio cooking program, so that appropriate movements are made to achieve identical results.” See at least [0509]; “Raw data is collected at each point in time to allow the raw data to be processed to be able to extract the shape, dimension, location and orientation of all objects of importance to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50 in a step 1162. The processed data is further analyzed by the computer system to allow the controller of the standardized robotic kitchen to adjust robotic arm and hand trajectories and minimanipulations, by modifying the control signals defined by the robotic script. Adaptations to the recipe-script execution and thus control signals is essential in successfully completing each stage of the replication for a particular dish, given the potential for variability for many variables (ingredients, temperature, etc.).” See at least [0547]; “plans can be made once the future environment is known sufficiently enough. The system simulates the movement of the robot, objects and the change in environment caused by a certain APSB.” See at least [0858])
Regarding Claim 4,
Oleynik further teaches
wherein the historical data are used as reference for industrial process parameters for new products. (“FIG. 23 is a flow diagram illustrating the process 926 of identifying a non-standard object through three-dimensional modeling and reasoning. At step 928, the computer 16 detects a non-standard object by a sensor, such as an ingredient that may have a different size, different dimensions, and/or different weight. At step 930, the computer 16 identifies the non-standard object with three-dimensional modeling sensors 66 to capture shape, dimensions, orientation and position information and robotic hands 72 make a real-time adjustment to perform the appropriate food preparation tasks (e.g. cutting or picking up a piece of steak). … A minimanipulation or an action primitive may involve the robotic hand 72 and a standard object, or the robotic hand 72 and a nonstandard object. … The parameters for a particular minimanipulation may differ depending on the complexity and objects that are necessary to perform the minimanipulation. In this example, four parameters are identified: the starting XYZ position coordinates in the volume of the standardized kitchen module, the speed, the object size, and the object shape. Both the object size and the object shape may be defined or described by non-standard parameters.” See at least [0532-0535], wherein the identified object or non-standard object is a new product.)
Regarding Claim 5,
Modified Oleynik does not explicitly teach, but Willför teaches
wherein the volumetric anti-collision tester comprises (“the device further includes a camera 20 configured to capture an image of the real robot and its environment, and the augmented reality unit 18 is configured to register the graphical representations of the safety volumes on the image of the real robot and its environment to provide the composited augmented reality image. The display unit 5 visualizes the view of the camera 20 combined with computer generated graphics of the safety volumes.” See at least [0055]; “FIG. 5 shows an example of graphical presentation of a moving safety volume 60 along a movement path 62 defined by a plurality of target points 64.” See at least [0070] and fig. 5; “If there is an overlap between the moving safety zone and the obstacle, there is a risk for collision between the mechanical unit and the obstacle. Then the path of the mechanical unit has to be reprogrammed.” See at least [0021])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Oleynik to further include the teachings of Willför with a reasonable expectation of success such that “programming and verification of the fixed safe modules will be more intuitive and faster.” (See at least [0017])
Willför also does not explicitly teach, but Gautier teaches
a plurality of sensors which monitor an environment (“the trajectory capture module 8 may comprise at least one video sensor 81, such as a camera (FIG. 6). This video sensor or sensors 81 captures images of the positions that the pointer or pointers 6 take while being driven. The trajectory of the pointer 6 or of each of the pointers 6 is then determined by processing the images captured by the video sensors.” See at least [0083])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Oleynik and Willför to further include the teachings of Gautier with a reasonable expectation of success to alert the user of conflicts such that “a user can easily and quickly program one or more robotic devices of a robotic system” (See at least [0014]) and “the user can ensure that the trajectories they program are compatible with the workspace.” (See at least [0017])
Regarding Claim 6,
Modified Oleynik does not explicitly teach, but Willför teaches
an immediate stop area.(“If the safety volume and the fixed safety volume instead cease to overlap each other on a part of the robot path, it is necessary to change the size of the fixed safety volume or change the programmed movement path to prevent the mechanical unit from reaching the border of the fixed safety volume during execution of the programmed path and by that being emergency stopped.” See at least [0084])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Oleynik to further include the teachings of Willför with a reasonable expectation of success such that “programming and verification of the fixed safe modules will be more intuitive and faster.” (See at least [0017])
Willför also does not explicitly teach, but Gautier teaches
wherein the volumetric anti-collision tester further comprises a warning area for recalculation of trajectory (“This alert sub-module 92 allows to inform the user 7 that the motion required for the robotic device or devices 2 to complete the trajectory of the effector or effectors 4 enters an alert area in which the trajectory is not technically feasible by the robotic device or devices 2. This alert sub-module 92 may also inform the user 7 that the trajectory of the effector or effectors 4 may enter an alert area where the trajectory collides with an element 31 of the workspace 3. In order for the alert to be given to the user 7, the alert sub-module 92 may comprise a visual alert unit and/or a haptic alert unit and/or an acoustic unit.” See at least [0092]; “The post-processing module 10 further comprises an input sub-module 104 configured to allow a user 7 to modify points of the trajectory stored by the trajectory capture module 8 and/or modify the parameters of the trajectory stored by the trajectory capture module 8 to obtain said compatible trajectory for the effector 4 or for each of the effectors 4 of the robotic device 2 or of each of the robotic devices 2 and/or said compatible parameters.” See at least [0098])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Oleynik and Willför to further include the teachings of Gautier with a reasonable expectation of success to alert the user of conflicts such that “a user can easily and quickly program one or more robotic devices of a robotic system” (See at least [0014]) and “the user can ensure that the trajectories they program are compatible with the workspace.” (See at least [0017])
Regarding Claim 7,
Oleynik further teaches
wherein the digital model is virtual, and the Artificial Intelligence (AI) algorithms, for control and modification of the automatically generated trajectories of the autonomous systems, are configured to use data originating from the virtual digital model, the plurality of sensors and the volumetric anti-collision tester for learning modifications in the trajectories and behavior of the autonomous systems. (“two robotic hands that closely resemble functional human hands with access to one or more libraries of minimanipulations, and standardized three-dimensional (3D) vision devices for creating dynamic virtual 3D-vision model of operation volume.” See at least [0750]; “executing said commands through position or velocity or joint or force based control at the joint-actuator level, and providing sensory data back to the macro-manipulation control and perception subsystems, while also monitoring all processes to allow for learning algorithms to provide improvements to the mini-manipulation macro-level command-library to improve future performance based on criteria such as execution-time, energy-expended, collision-avoidance, singularity-avoidance and workspace-reachability.” See at least [0798], wherein the learning algorithms are AI algorithms.)
Regarding Claim 8,
Oleynik further teaches
and using the data originating from the digital model, the plurality of sensors and information of the volumetric anti-collision tester for learning modifications in the trajectories and behavior of the autonomous systems so the platform for autonomous systems makes adjustments in the data to be used henceforth. (“two robotic hands that closely resemble functional human hands with access to one or more libraries of minimanipulations, and standardized three-dimensional (3D) vision devices for creating dynamic virtual 3D-vision model of operation volume.” See at least [0750]; “executing said commands through position or velocity or joint or force based control at the joint-actuator level, and providing sensory data back to the macro-manipulation control and perception subsystems, while also monitoring all processes to allow for learning algorithms to provide improvements to the mini-manipulation macro-level command-library to improve future performance based on criteria such as execution-time, energy-expended, collision-avoidance, singularity-avoidance and workspace-reachability.” See at least [0798], wherein the learning algorithms are AI algorithms.)
Oleynik does not explicitly teach, but Willför teaches
wherein the operations further include receiving a trajectory recalculation or stop command issued to the platform related to controlled zones, (“If there is an overlap between the moving safety zone and the obstacle, there is a risk for collision between the mechanical unit and the obstacle. Then the path of the mechanical unit has to be reprogrammed.” See at least [0021]; “If the safety volume and the fixed safety volume instead cease to overlap each other on a part of the robot path, it is necessary to change the size of the fixed safety volume or change the programmed movement path to prevent the mechanical unit from reaching the border of the fixed safety volume during execution of the programmed path and by that being emergency stopped.” See at least [0084])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of Oleynik to further include the teachings of Willför with a reasonable expectation of success such that “programming and verification of the fixed safe modules will be more intuitive and faster.” (See at least [0017])
Regarding Claim 10,
Oleynik teaches
A control platform method for autonomous systems (“Systems and methods are provided for operating universal robotic assistant systems.” See at least [0008])
comprising performing with at least one processor and/or processing circuit connected to at least one memory storing information, the processor and/or processing circuit at least in part configured by the information the at least one memory stores to perform operations comprising: (“a computer device, as shown in 4324, on which computer-executable instructions to perform the methodologies discussed herein may be installed and run. … The example computer system 4324 includes a processor 4326 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 4328 and a static memory 30, which communicate with each other via a bus 4332. … The disk drive unit 4340 includes a machine-readable medium 244 on which is stored one or more sets of instructions (e.g., software 4346) embodying any one or more of the methodologies or functions described herein.” See at least [1271-1272])
consuming process, product, environment and resources data of a digital model for automatic generation of trajectories for autonomous systems; (“A minimanipulation library provides a large suite of higher-level sensing-and-execution sequences that are common building blocks for complex tasks.” See at least [0041], wherein the minimanipulation library is the data of a digital model.; “a database library structure 972 of minimanipulation objects for use in the standardized robotic kitchen. The database library structure 972 shows several fields for entering and storing information for a particular minimanipulation, including (1) the name of the minimanipulation, (2) the assigned code of the minimanipulation, (3) the code(s) of standardized equipment and tools associated with the performance of the minimanipulation, (4) the initial position and orientation of the manipulated (standard or non-standard) objects (ingredients and tools), (5) parameters/variables defined by the user (or extracted from the recorded recipe during execution), (6) sequence of robotic hand movements (control signals for all servos) and connecting feedback parameters (from any sensor or video monitoring system) of minimanipulations on the timeline.” See at least [0535]; See at least [0786], describing metrics such as time required and energy-expended corresponding to stored mini-manipulation action primitive (AP) components, wherein the metrics are historical resource data in a database.; “The method of mini-manipulation command generation for one or both the macro- or micro-manipulation subsystems, comprises receiving a high-level task execution command, identifying individual subtasks which will be mapped to the applicable robotic subsystems, generation of individual performance criteria and measurable success end-state criteria for each of the above subtasks, selection of one or more in either a stand-alone or combination, of the most suitable action primitive candidates, evaluation of these action primitive alternatives for maximizing or minimizing such measures as execution-time, energy expended, robot reachability, collision avoidance or any other task-critical criteria, generation of either or both macro- and/or micro-manipulation subsystem trajectories in one or more motion spaces.” See at least [1289])
use a digital model and physical media reference data to provide a digital twin of the product/process; (“the position data and the image data are obtained from the one or more sensors, wherein the one or more sensors comprises at least one of a navigation system and one or more image capturing devices. In some embodiments, detecting the one or more objects is based on at least one of the type of the current environment, the environment data corresponding to the current environment, and object data.” See at least [0012-0013], wherein the sensor data is physical media reference data.; “Upon completion of the chef studio recipe creation and cooking process by the chef, the robotic cooking engine generates a simulation visualization program 1954 replicating the movement and media data used for later recipe replication by a remote standardized robotic kitchen system. Based on the raw and processed data, and a confirmation of the simulated recipe execution visualization by the chef, hardware-specific applications are developed and integrated for different (mobile) operating systems” See at least [0583], wherein the simulation is a digital twin.)
applying historical data from processes, product, environment and resources located in a database; (“integration of electronic libraries of mini-manipulations with transformed robotic instructions for replicating movements, processes, and techniques with real-time electronic adjustments.” See at least [0002], wherein the electronic libraries of mini-manipulations includes historical processes, product, environment and resources data according to the citations above (See at least [0535] and [0786]).)
controlling digital information, the physical media reference data and the historical data combined for automatic generation of trajectories for autonomous systems; (“During the food preparation process, the robotic apparatus 75 uses touch signals generated by sensors in the fingertips and the palms of a robot's hands to detect force, temperature, humidity and toxicity as the robot replicates step-by-step movements and compares the sensed values with the tactile profile of the chef's studio cooking program. Visual sensors help the robot to identify the surroundings and take appropriate cooking actions. The robotic apparatus 75 analyzes the image of the immediate environment from the visual sensors and compares it with the saved image of the chef's studio cooking program, so that appropriate movements are made to achieve identical results.” See at least [0509]; “Raw data is collected at each point in time to allow the raw data to be processed to be able to extract the shape, dimension, location and orientation of all objects of importance to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50 in a step 1162. The processed data is further analyzed by the computer system to allow the controller of the standardized robotic kitchen to adjust robotic arm and hand trajectories and minimanipulations, by modifying the control signals defined by the robotic script. Adaptations to the recipe-script execution and thus control signals is essential in successfully completing each stage of the replication for a particular dish, given the potential for variability for many variables (ingredients, temperature, etc.).” See at least [0547]; See at least [0583] for the simulation/digital twin.; Examiner Interpretation: Adapting the control of the robot using the stored minimanipulation data (historical data) and the raw sensor data (reference data) is equivalent to the control of digital information.)
automatically generating the trajectories for autonomous systems; (“Raw data is collected at each point in time to allow the raw data to be processed to be able to extract the shape, dimension, location and orientation of all objects of importance to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50 in a step 1162. The processed data is further analyzed by the computer system to allow the controller of the standardized robotic kitchen to adjust robotic arm and hand trajectories and minimanipulations, by modifying the control signals defined by the robotic script.” See at least [0547])
using of Artificial Intelligence (AI) algorithms for control and modification of the automatically generated trajectories of the autonomous systems; (“The present disclosure relates to fields of robotics and artificial intelligence (AI). … integration of electronic libraries of mini-manipulations with transformed robotic instructions for replicating movements, processes, and techniques with real-time electronic adjustments.” See at least [0002]; “learning algorithms monitor each and every motion/interaction sequence and perform simple variable-perturbations to ascertain outcome to decide on if/how/when/what variable(s) and sequence(s) to modify in order to achieve a higher level of execution fidelity at levels ranging from low-to high-levels of various MMLs.” See at least [0052]; Also see at least [0383], [0992], and [0996-0998] for reinforcement learning.)
controlling and monitoring of resources and parameters; and automatically inspecting and generating historical data from processes, product, environment and resources for database feedback. (“The architecture of the software-module/action layer provides a framework that allows the inclusion of: (1) refined Endeffector sensing (for refined and more accurate real-world interface sensing); (2) introduction of the macro-(overall sensing by and from the articulated base) and micro-(local task-specific sensing between the endeffectors and the task-/cooking-specific elements) tiers to allow continuous minimanipulation libraries to be used and updated (via learning) based on a physical split between coarse and fine manipulation (and thus positioning, force/torque control, product-handling and process monitoring);” See at least [0767]; “Said motion commands are sequentially fed to an execution block 3613, which controls all instrumented articulated and actuated joints in at least joint- or Cartesian space to ensure the movements track the commanded trajectories in position/velocity and/or torque/force. A feedback sensing block 3614 provides feedback data from all sensors to the execution block 3613 as well as an environment perception block/module 3611 for further processing. Feedback is not only provided to allow tracking the internal state of variables, but also sensory data from sensor measuring the surrounding environment and geometries. Feedback data from said module 3614 is used by the execution module 3613 to ensure actual values track their commanded setpoints, as well as an environment perception module 3611 to image and map, model and identify the state of each articulated element, the overall configuration of the robot as well as the state of the surrounding environment the robot is operating in. Additionally, said feedback data is also provided to a learning module 3615 responsible for tracking the overall performance of the system and comparing it to known required performance metrics, allowing one or more learning methods to develop a continuously updated set of descriptors that define all mini-manipulations contained within their respective mini-manipulation library 3630, in this case the macro-level mini-manipulation sublibrary 3631.” See at least [0772]; Examiner Interpretation: The feedback/sensing is the monitoring and inspection. The feedback data is generated for updating the minimanipulation library/database (learning) and therefore is generated for database feedback.)
Oleynik does not explicitly teach, but Willför teaches
acting of a volumetric anti-collision tester with alert and/or stop zones by self-protection in monitoring from (“the device further includes a camera 20 configured to capture an image of the real robot and its environment, and the augmented reality unit 18 is configured to register the graphical representations of the safety volumes on the image of the real robot and its environment to provide the composited augmented reality image. The display unit 5 visualizes the view of the camera 20 combined with computer generated graphics of the safety volumes.” See at least [0055]; “FIG. 5 shows an example of graphical presentation of a moving safety volume 60 along a movement path 62 defined by a plurality of target points 64.” See at least [0070] and fig. 5; “If the safety volume and the fixed safety volume instead cease to overlap each other on a part of the robot path, it is necessary to change the size of the fixed safety volume or change the programmed movement path to prevent the mechanical unit from reaching the border of the fixed safety volume during execution of the programmed path and by that being emergency stopped.” See at least [0084])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of Oleynik to further include the teachings of Willför with a reasonable expectation of success such that “programming and verification of the fixed safe modules will be more intuitive and faster.” (See at least [0017])
Willför also does not explicitly teach, but Gautier teaches
alert zones (“This alert sub-module 92 allows to inform the user 7 that the motion required for the robotic device or devices 2 to complete the trajectory of the effector or effectors 4 enters an alert area in which the trajectory is not technically feasible by the robotic device or devices 2. This alert sub-module 92 may also inform the user 7 that the trajectory of the effector or effectors 4 may enter an alert area where the trajectory collides with an element 31 of the workspace 3. In order for the alert to be given to the user 7, the alert sub-module 92 may comprise a visual alert unit and/or a haptic alert unit and/or an acoustic unit.” See at least [0092])
monitoring from a plurality of sensors (“the trajectory capture module 8 may comprise at least one video sensor 81, such as a camera (FIG. 6). This video sensor or sensors 81 captures images of the positions that the pointer or pointers 6 take while being driven. The trajectory of the pointer 6 or of each of the pointers 6 is then determined by processing the images captured by the video sensors.” See at least [0083])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of Oleynik and Willför to further include the teachings of Gautier with a reasonable expectation of success to alert the user of conflicts such that “a user can easily and quickly program one or more robotic devices of a robotic system” (See at least [0014]) and “the user can ensure that the trajectories they program are compatible with the workspace.” (See at least [0017])
Regarding Claim 11,
Oleynik further teaches
using data originating from the digital model, the plurality of sensors and information of the volumetric anti-collision tester for learning modifications in the trajectories and behavior of the autonomous systems, so that the platform for autonomous systems makes adjustments in data to be used henceforth. (“two robotic hands that closely resemble functional human hands with access to one or more libraries of minimanipulations, and standardized three-dimensional (3D) vision devices for creating dynamic virtual 3D-vision model of operation volume.” See at least [0750]; “executing said commands through position or velocity or joint or force based control at the joint-actuator level, and providing sensory data back to the macro-manipulation control and perception subsystems, while also monitoring all processes to allow for learning algorithms to provide improvements to the mini-manipulation macro-level command-library to improve future performance based on criteria such as execution-time, energy-expended, collision-avoidance, singularity-avoidance and workspace-reachability.” See at least [0798], wherein the learning algorithms are AI algorithms.)
Oleynik does not explicitly teach, but Willför teaches
wherein the operations further include receiving a trajectory recalculation or stop command issued to the platform related to controlled zones, (“If there is an overlap between the moving safety zone and the obstacle, there is a risk for collision between the mechanical unit and the obstacle. Then the path of the mechanical unit has to be reprogrammed.” See at least [0021]; “If the safety volume and the fixed safety volume instead cease to overlap each other on a part of the robot path, it is necessary to change the size of the fixed safety volume or change the programmed movement path to prevent the mechanical unit from reaching the border of the fixed safety volume during execution of the programmed path and by that being emergency stopped.” See at least [0084])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of Oleynik to further include the teachings of Willför with a reasonable expectation of success such that “programming and verification of the fixed safe modules will be more intuitive and faster.” (See at least [0017])
Regarding Claim 13,
Oleynik further teaches
wherein the physical media reference data comprises results from a plurality of sensors displayed in environments, and the method further includes the autonomous systems performing the automatically generated trajectories. (“the position data and the image data are obtained from the one or more sensors, wherein the one or more sensors comprises at least one of a navigation system and one or more image capturing devices. In some embodiments, detecting the one or more objects is based on at least one of the type of the current environment, the environment data corresponding to the current environment, and object data.” See at least [0012-0013], wherein the sensor data is physical media reference data.; “Raw data is collected at each point in time to allow the raw data to be processed to be able to extract the shape, dimension, location and orientation of all objects of importance to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50 in a step 1162. The processed data is further analyzed by the computer system to allow the controller of the standardized robotic kitchen to adjust robotic arm and hand trajectories … The process of recipe-script execution based on key measurable variables is an essential part of the use of the augmented (also termed multi-modal) sensor system 20 during the execution of the replicating steps for a particular dish in a standardized robotic kitchen 50.” See at least [0547])
Regarding Claim 14,
Oleynik further teaches
further including comparing the physical media reference data with digital model data for feeding the control of platform with confirmations or alterations of the generated trajectories. (“During the food preparation process, the robotic apparatus 75 uses touch signals generated by sensors in the fingertips and the palms of a robot's hands to detect force, temperature, humidity and toxicity as the robot replicates step-by-step movements and compares the sensed values with the tactile profile of the chef's studio cooking program. Visual sensors help the robot to identify the surroundings and take appropriate cooking actions. The robotic apparatus 75 analyzes the image of the immediate environment from the visual sensors and compares it with the saved image of the chef's studio cooking program, so that appropriate movements are made to achieve identical results.” See at least [0509]; “Raw data is collected at each point in time to allow the raw data to be processed to be able to extract the shape, dimension, location and orientation of all objects of importance to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50 in a step 1162. The processed data is further analyzed by the computer system to allow the controller of the standardized robotic kitchen to adjust robotic arm and hand trajectories and minimanipulations, by modifying the control signals defined by the robotic script. Adaptations to the recipe-script execution and thus control signals is essential in successfully completing each stage of the replication for a particular dish, given the potential for variability for many variables (ingredients, temperature, etc.).” See at least [0547]; “plans can be made once the future environment is known sufficiently enough. The system simulates the movement of the robot, objects and the change in environment caused by a certain APSB.” See at least [0858])
Regarding Claim 15,
Oleynik further teaches
further including using the historical data as reference for industrial process parameters for new products. (“FIG. 23 is a flow diagram illustrating the process 926 of identifying a non-standard object through three-dimensional modeling and reasoning. At step 928, the computer 16 detects a non-standard object by a sensor, such as an ingredient that may have a different size, different dimensions, and/or different weight. At step 930, the computer 16 identifies the non-standard object with three-dimensional modeling sensors 66 to capture shape, dimensions, orientation and position information and robotic hands 72 make a real-time adjustment to perform the appropriate food preparation tasks (e.g. cutting or picking up a piece of steak). … A minimanipulation or an action primitive may involve the robotic hand 72 and a standard object, or the robotic hand 72 and a nonstandard object. … The parameters for a particular minimanipulation may differ depending on the complexity and objects that are necessary to perform the minimanipulation. In this example, four parameters are identified: the starting XYZ position coordinates in the volume of the standardized kitchen module, the speed, the object size, and the object shape. Both the object size and the object shape may be defined or described by non-standard parameters.” See at least [0532-0535], wherein the identified object or non-standard object is a new product.)
Regarding Claim 16,
Modified Oleynik does not explicitly teach, but Willför teaches
further including using (“the device further includes a camera 20 configured to capture an image of the real robot and its environment, and the augmented reality unit 18 is configured to register the graphical representations of the safety volumes on the image of the real robot and its environment to provide the composited augmented reality image. The display unit 5 visualizes the view of the camera 20 combined with computer generated graphics of the safety volumes.” See at least [0055]; “FIG. 5 shows an example of graphical presentation of a moving safety volume 60 along a movement path 62 defined by a plurality of target points 64.” See at least [0070] and fig. 5; “If there is an overlap between the moving safety zone and the obstacle, there is a risk for collision between the mechanical unit and the obstacle. Then the path of the mechanical unit has to be reprogrammed.” See at least [0021])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Oleynik to further include the teachings of Willför with a reasonable expectation of success such that “programming and verification of the fixed safe modules will be more intuitive and faster.” (See at least [0017])
Willför also does not explicitly teach, but Gautier teaches
a plurality of sensors to monitor an environment (“the trajectory capture module 8 may comprise at least one video sensor 81, such as a camera (FIG. 6). This video sensor or sensors 81 captures images of the positions that the pointer or pointers 6 take while being driven. The trajectory of the pointer 6 or of each of the pointers 6 is then determined by processing the images captured by the video sensors.” See at least [0083])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Oleynik and Willför to further include the teachings of Gautier with a reasonable expectation of success to alert the user of conflicts such that “a user can easily and quickly program one or more robotic devices of a robotic system” (See at least [0014]) and “the user can ensure that the trajectories they program are compatible with the workspace.” (See at least [0017])
Regarding Claim 17,
Modified Oleynik does not explicitly teach, but Willför teaches
an immediate stop area.(“If the safety volume and the fixed safety volume instead cease to overlap each other on a part of the robot path, it is necessary to change the size of the fixed safety volume or change the programmed movement path to prevent the mechanical unit from reaching the border of the fixed safety volume during execution of the programmed path and by that being emergency stopped.” See at least [0084])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Oleynik to further include the teachings of Willför with a reasonable expectation of success such that “programming and verification of the fixed safe modules will be more intuitive and faster.” (See at least [0017])
Willför also does not explicitly teach, but Gautier teaches
further comprising using a warning area for recalculation of trajectory (“This alert sub-module 92 allows to inform the user 7 that the motion required for the robotic device or devices 2 to complete the trajectory of the effector or effectors 4 enters an alert area in which the trajectory is not technically feasible by the robotic device or devices 2. This alert sub-module 92 may also inform the user 7 that the trajectory of the effector or effectors 4 may enter an alert area where the trajectory collides with an element 31 of the workspace 3. In order for the alert to be given to the user 7, the alert sub-module 92 may comprise a visual alert unit and/or a haptic alert unit and/or an acoustic unit.” See at least [0092]; “The post-processing module 10 further comprises an input sub-module 104 configured to allow a user 7 to modify points of the trajectory stored by the trajectory capture module 8 and/or modify the parameters of the trajectory stored by the trajectory capture module 8 to obtain said compatible trajectory for the effector 4 or for each of the effectors 4 of the robotic device 2 or of each of the robotic devices 2 and/or said compatible parameters.” See at least [0098])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Oleynik and Willför to further include the teachings of Gautier with a reasonable expectation of success to alert the user of conflicts such that “a user can easily and quickly program one or more robotic devices of a robotic system” (See at least [0014]) and “the user can ensure that the trajectories they program are compatible with the workspace.” (See at least [0017])
Regarding Claim 18,
Oleynik further teaches
further including the Artificial Intelligence (AI) algorithms, for control and modification of the automatically generated trajectories of the autonomous systems, using data originating from the digital model, the plurality of sensors and the volumetric anti-collision testing for learning modifications in the trajectories and behavior of the autonomous systems. (“two robotic hands that closely resemble functional human hands with access to one or more libraries of minimanipulations, and standardized three-dimensional (3D) vision devices for creating dynamic virtual 3D-vision model of operation volume.” See at least [0750]; “executing said commands through position or velocity or joint or force based control at the joint-actuator level, and providing sensory data back to the macro-manipulation control and perception subsystems, while also monitoring all processes to allow for learning algorithms to provide improvements to the mini-manipulation macro-level command-library to improve future performance based on criteria such as execution-time, energy-expended, collision-avoidance, singularity-avoidance and workspace-reachability.” See at least [0798], wherein the learning algorithms are AI algorithms.)
Regarding Claim 19,
Oleynik teaches
A non-transitory memory that stores information for a control platform for autonomous systems, the information controlling at least one processor and/or processing circuit connected to the non-transitory memory storing information to perform operations comprising: (“a computer device, as shown in 4324, on which computer-executable instructions to perform the methodologies discussed herein may be installed and run. … The example computer system 4324 includes a processor 4326 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 4328 and a static memory 30, which communicate with each other via a bus 4332. … The disk drive unit 4340 includes a machine-readable medium 244 on which is stored one or more sets of instructions (e.g., software 4346) embodying any one or more of the methodologies or functions described herein. … The term “machine-readable medium” shall also be taken to include any tangible medium that is capable of storing a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure.” See at least [1271-1273])
consuming process, product, environment and resources data of a digital model for automatic generation of trajectories for autonomous systems; (“A minimanipulation library provides a large suite of higher-level sensing-and-execution sequences that are common building blocks for complex tasks.” See at least [0041], wherein the minimanipulation library is the data of a digital model.; “a database library structure 972 of minimanipulation objects for use in the standardized robotic kitchen. The database library structure 972 shows several fields for entering and storing information for a particular minimanipulation, including (1) the name of the minimanipulation, (2) the assigned code of the minimanipulation, (3) the code(s) of standardized equipment and tools associated with the performance of the minimanipulation, (4) the initial position and orientation of the manipulated (standard or non-standard) objects (ingredients and tools), (5) parameters/variables defined by the user (or extracted from the recorded recipe during execution), (6) sequence of robotic hand movements (control signals for all servos) and connecting feedback parameters (from any sensor or video monitoring system) of minimanipulations on the timeline.” See at least [0535]; See at least [0786], describing metrics such as time required and energy-expended corresponding to stored mini-manipulation action primitive (AP) components, wherein the metrics are historical resource data in a database.; “The method of mini-manipulation command generation for one or both the macro- or micro-manipulation subsystems, comprises receiving a high-level task execution command, identifying individual subtasks which will be mapped to the applicable robotic subsystems, generation of individual performance criteria and measurable success end-state criteria for each of the above subtasks, selection of one or more in either a stand-alone or combination, of the most suitable action primitive candidates, evaluation of these action primitive alternatives for maximizing or minimizing such measures as execution-time, energy expended, robot reachability, collision avoidance or any other task-critical criteria, generation of either or both macro- and/or micro-manipulation subsystem trajectories in one or more motion spaces.” See at least [1289])
use a digital model and physical media reference data to provide a digital twin of the product/process; (“the position data and the image data are obtained from the one or more sensors, wherein the one or more sensors comprises at least one of a navigation system and one or more image capturing devices. In some embodiments, detecting the one or more objects is based on at least one of the type of the current environment, the environment data corresponding to the current environment, and object data.” See at least [0012-0013], wherein the sensor data is physical media reference data.; “Upon completion of the chef studio recipe creation and cooking process by the chef, the robotic cooking engine generates a simulation visualization program 1954 replicating the movement and media data used for later recipe replication by a remote standardized robotic kitchen system. Based on the raw and processed data, and a confirmation of the simulated recipe execution visualization by the chef, hardware-specific applications are developed and integrated for different (mobile) operating systems” See at least [0583], wherein the simulation is a digital twin.)
applying historical data from processes, product, environment and resources located in a database; (“integration of electronic libraries of mini-manipulations with transformed robotic instructions for replicating movements, processes, and techniques with real-time electronic adjustments.” See at least [0002], wherein the electronic libraries of mini-manipulations includes historical processes, product, environment and resources data according to the citations above (See at least [0535] and [0786]).)
controlling digital information, the physical media reference data and the historical data combined for automatic generation of trajectories for autonomous systems; (“During the food preparation process, the robotic apparatus 75 uses touch signals generated by sensors in the fingertips and the palms of a robot's hands to detect force, temperature, humidity and toxicity as the robot replicates step-by-step movements and compares the sensed values with the tactile profile of the chef's studio cooking program. Visual sensors help the robot to identify the surroundings and take appropriate cooking actions. The robotic apparatus 75 analyzes the image of the immediate environment from the visual sensors and compares it with the saved image of the chef's studio cooking program, so that appropriate movements are made to achieve identical results.” See at least [0509]; “Raw data is collected at each point in time to allow the raw data to be processed to be able to extract the shape, dimension, location and orientation of all objects of importance to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50 in a step 1162. The processed data is further analyzed by the computer system to allow the controller of the standardized robotic kitchen to adjust robotic arm and hand trajectories and minimanipulations, by modifying the control signals defined by the robotic script. Adaptations to the recipe-script execution and thus control signals is essential in successfully completing each stage of the replication for a particular dish, given the potential for variability for many variables (ingredients, temperature, etc.).” See at least [0547]; See at least [0583] for the simulation/digital twin.; Examiner Interpretation: Adapting the control of the robot using the stored minimanipulation data (historical data) and the raw sensor data (reference data) is equivalent to the control of digital information.)
automatically generating the trajectories for autonomous systems; (“Raw data is collected at each point in time to allow the raw data to be processed to be able to extract the shape, dimension, location and orientation of all objects of importance to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50 in a step 1162. The processed data is further analyzed by the computer system to allow the controller of the standardized robotic kitchen to adjust robotic arm and hand trajectories and minimanipulations, by modifying the control signals defined by the robotic script.” See at least [0547])
using of Artificial Intelligence (AI) algorithms for control and modification of the automatically generated trajectories of the autonomous systems; (“The present disclosure relates to fields of robotics and artificial intelligence (AI). … integration of electronic libraries of mini-manipulations with transformed robotic instructions for replicating movements, processes, and techniques with real-time electronic adjustments.” See at least [0002]; “learning algorithms monitor each and every motion/interaction sequence and perform simple variable-perturbations to ascertain outcome to decide on if/how/when/what variable(s) and sequence(s) to modify in order to achieve a higher level of execution fidelity at levels ranging from low-to high-levels of various MMLs.” See at least [0052]; Also see at least [0383], [0992], and [0996-0998] for reinforcement learning.) controlling and monitoring of resources and parameters; and automatically inspecting and generating historical data from processes, product, environment and resources for database feedback. (“The architecture of the software-module/action layer provides a framework that allows the inclusion of: (1) refined Endeffector sensing (for refined and more accurate real-world interface sensing); (2) introduction of the macro-(overall sensing by and from the articulated base) and micro-(local task-specific sensing between the endeffectors and the task-/cooking-specific elements) tiers to allow continuous minimanipulation libraries to be used and updated (via learning) based on a physical split between coarse and fine manipulation (and thus positioning, force/torque control, product-handling and process monitoring);” See at least [0767]; “Said motion commands are sequentially fed to an execution block 3613, which controls all instrumented articulated and actuated joints in at least joint- or Cartesian space to ensure the movements track the commanded trajectories in position/velocity and/or torque/force. A feedback sensing block 3614 provides feedback data from all sensors to the execution block 3613 as well as an environment perception block/module 3611 for further processing. Feedback is not only provided to allow tracking the internal state of variables, but also sensory data from sensor measuring the surrounding environment and geometries. Feedback data from said module 3614 is used by the execution module 3613 to ensure actual values track their commanded setpoints, as well as an environment perception module 3611 to image and map, model and identify the state of each articulated element, the overall configuration of the robot as well as the state of the surrounding environment the robot is operating in. Additionally, said feedback data is also provided to a learning module 3615 responsible for tracking the overall performance of the system and comparing it to known required performance metrics, allowing one or more learning methods to develop a continuously updated set of descriptors that define all mini-manipulations contained within their respective mini-manipulation library 3630, in this case the macro-level mini-manipulation sublibrary 3631.” See at least [0772]; Examiner Interpretation: The feedback/sensing is the monitoring and inspection. The feedback data is generated for updating the minimanipulation library/database (learning) and therefore is generated for database feedback.
Oleynik does not explicitly teach, but Willför teaches
acting of a volumetric anti-collision tester with alert and/or stop zones by self-protection in monitoring from (“the device further includes a camera 20 configured to capture an image of the real robot and its environment, and the augmented reality unit 18 is configured to register the graphical representations of the safety volumes on the image of the real robot and its environment to provide the composited augmented reality image. The display unit 5 visualizes the view of the camera 20 combined with computer generated graphics of the safety volumes.” See at least [0055]; “FIG. 5 shows an example of graphical presentation of a moving safety volume 60 along a movement path 62 defined by a plurality of target points 64.” See at least [0070] and fig. 5; “If the safety volume and the fixed safety volume instead cease to overlap each other on a part of the robot path, it is necessary to change the size of the fixed safety volume or change the programmed movement path to prevent the mechanical unit from reaching the border of the fixed safety volume during execution of the programmed path and by that being emergency stopped.” See at least [0084])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of Oleynik to further include the teachings of Willför with a reasonable expectation of success such that “programming and verification of the fixed safe modules will be more intuitive and faster.” (See at least [0017])
Willför also does not explicitly teach, but Gautier teaches
alert zones (“This alert sub-module 92 allows to inform the user 7 that the motion required for the robotic device or devices 2 to complete the trajectory of the effector or effectors 4 enters an alert area in which the trajectory is not technically feasible by the robotic device or devices 2. This alert sub-module 92 may also inform the user 7 that the trajectory of the effector or effectors 4 may enter an alert area where the trajectory collides with an element 31 of the workspace 3. In order for the alert to be given to the user 7, the alert sub-module 92 may comprise a visual alert unit and/or a haptic alert unit and/or an acoustic unit.” See at least [0092])
monitoring from a plurality of sensors (“the trajectory capture module 8 may comprise at least one video sensor 81, such as a camera (FIG. 6). This video sensor or sensors 81 captures images of the positions that the pointer or pointers 6 take while being driven. The trajectory of the pointer 6 or of each of the pointers 6 is then determined by processing the images captured by the video sensors.” See at least [0083])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of Oleynik and Willför to further include the teachings of Gautier with a reasonable expectation of success to alert the user of conflicts such that “a user can easily and quickly program one or more robotic devices of a robotic system” (See at least [0014]) and “the user can ensure that the trajectories they program are compatible with the workspace.” (See at least [0017])
Regarding Claim 20,
Oleynik further teaches
using data originating from the digital model, the plurality of sensors and information of the volumetric anti-collision tester for learning modifications in the trajectories and behavior of the autonomous systems, so that the platform for autonomous systems makes adjustments in data to be used henceforth. (“two robotic hands that closely resemble functional human hands with access to one or more libraries of minimanipulations, and standardized three-dimensional (3D) vision devices for creating dynamic virtual 3D-vision model of operation volume.” See at least [0750]; “executing said commands through position or velocity or joint or force based control at the joint-actuator level, and providing sensory data back to the macro-manipulation control and perception subsystems, while also monitoring all processes to allow for learning algorithms to provide improvements to the mini-manipulation macro-level command-library to improve future performance based on criteria such as execution-time, energy-expended, collision-avoidance, singularity-avoidance and workspace-reachability.” See at least [0798], wherein the learning algorithms are AI algorithms.)
Oleynik does not explicitly teach, but Willför teaches
wherein the operations further include receiving a trajectory recalculation or stop command issued to the platform related to controlled zones, (“If there is an overlap between the moving safety zone and the obstacle, there is a risk for collision between the mechanical unit and the obstacle. Then the path of the mechanical unit has to be reprogrammed.” See at least [0021]; “If the safety volume and the fixed safety volume instead cease to overlap each other on a part of the robot path, it is necessary to change the size of the fixed safety volume or change the programmed movement path to prevent the mechanical unit from reaching the border of the fixed safety volume during execution of the programmed path and by that being emergency stopped.” See at least [0084])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of Oleynik to further include the teachings of Willför with a reasonable expectation of success such that “programming and verification of the fixed safe modules will be more intuitive and faster.” (See at least [0017])
Claim(s) 9 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Oleynik (US 20190291277 A1) in view of Willför (US 20160207198 A1), Gautier (US 20240001542 A1), and Toshev (US 20210397195 A1).
Regarding Claim 9,
Modified Oleynik does not explicitly teach, but Toshev teaches
wherein the AI is trained in a simulation environment providing a digital twin of the product/process using reinforcement learning to make decisions on the motion of each degree of freedom of the autonomous systems which allow building the trajectories up to final objectives without colliding. (“the robot 525 may have multiple degrees of freedom and each of the actuators may control actuation of the robot 525 within one or more of the degrees of freedom responsive to the control commands. As used herein, the term actuator encompasses a mechanical or electrical device that creates motion (e.g., a motor), in addition to any driver(s) that may be associated with the actuator and that translate received control commands into one or more signals for driving the actuator. Accordingly, providing a control command to an actuator may comprise providing the control command to a driver that translates the control command into appropriate signals for driving an electrical or mechanical device to create desired motion.” See at least [0085]; “using a trained low-level policy model to generate corresponding low-level action output that is based on the high-level action and that is optimized, according to the low-level policy model, to reach the navigation target most quickly and without collision. … Performing the reinforcement training includes: using simulated data, generated by a robot simulator, in generating rewards based on a reward function; and using the rewards to update the low-level policy model. The reward function penalizes robot collision, while optionally rewarding faster speeds and/or shorter distances in reaching navigation targets.” See at least [0097-0098])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Oleynik to further include the teachings of Toshev with a reasonable expectation of success to generate quicker trajectories without collision. (See at least [0097-0098])
Regarding Claim 12,
Modified Oleynik does not explicitly teach, but Toshev teaches
wherein the operations further include training the A in simulation environment as a digital twin of the product/process, using reinforcement learning to learn to make decisions on motion of each degree of freedom of the autonomous systems which allow building the trajectories up to a final objective without colliding. (“the robot 525 may have multiple degrees of freedom and each of the actuators may control actuation of the robot 525 within one or more of the degrees of freedom responsive to the control commands. As used herein, the term actuator encompasses a mechanical or electrical device that creates motion (e.g., a motor), in addition to any driver(s) that may be associated with the actuator and that translate received control commands into one or more signals for driving the actuator. Accordingly, providing a control command to an actuator may comprise providing the control command to a driver that translates the control command into appropriate signals for driving an electrical or mechanical device to create desired motion.” See at least [0085]; “using a trained low-level policy model to generate corresponding low-level action output that is based on the high-level action and that is optimized, according to the low-level policy model, to reach the navigation target most quickly and without collision. … Performing the reinforcement training includes: using simulated data, generated by a robot simulator, in generating rewards based on a reward function; and using the rewards to update the low-level policy model. The reward function penalizes robot collision, while optionally rewarding faster speeds and/or shorter distances in reaching navigation targets.” See at least [0097-0098])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Oleynik to further include the teachings of Toshev with a reasonable expectation of success to generate quicker trajectories without collision. (See at least [0097-0098])
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Karston G Evans whose telephone number is (571)272-8480. The examiner can normally be reached Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached at (571)270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KARSTON G. EVANS/Examiner, Art Unit 3657
Prosecution Insights