DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The following NON-FINAL Office Action is in response to Applicant’s communication filed 06/09/2026 regarding Application 18/832,501.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/09/2026 has been entered.
Status of Claim(s)
Claim(s) 1-11 is/are currently pending and are rejected as follows
Response to Arguments – 103 Rejection
Applicant’s arguments in regards to the previously applied 103 rejection are rendered moot in view of the amended prior art rejection below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Versace (US 2017/0076194 A1) in view of Luvaas (US 11,435,871 B1)
Claim(s) 1, 10, and 11 –
Versace discloses the following limitations
At least one memory storing a computer-readable code (Versace: Paragraph 42, "The platform 100 includes a user interface 102 that enables a user to define a hardware-agnostic brain, a processor 104 that implements the hardware-agnostic brain (which may include processes and programs implementing Artificial Intelligence (AI)/Artificial Neural Network (ANN)/Deep Neural Network (DNN) processing), a memory 103 to store instructions for defining and executing the hardware-agnostic brain (including instructions implementing AI/ ANN/DNN and synaptic weights defining ANN/DNN structures), and a communications interface 105 for communicating with the robot 106. The user interface 102 allows the user to create actionable tasks and/or usable workflows for the robot 106. The platform 100 interprets and implements these workflows as a hardware-agnostic brain 104 that interprets data from the robot 106 and input entered via the user interface 102, then performs one or more corresponding actions. The platform 100 can be implemented in any suitable computing device, including but not limited to a tablet computer (e.g., an iPad), a smartphone, a single-board computer, a desktop computer, a laptop, either local or in the cloud, etc. The platform 100 may provide the user interface 102 as a Graphical User Interface (GUI) via a touchscreen or other suitable display.")
At least one processor to call the computer-readable code (Versace: Paragraph 42, "The platform 100 includes a user interface 102 that enables a user to define a hardware-agnostic brain, a processor 104 that implements the hardware-agnostic brain (which may include processes and programs implementing Artificial Intelligence (AI)/Artificial Neural Network (ANN)/Deep Neural Network (DNN) processing), a memory 103 to store instructions for defining and executing the hardware-agnostic brain (including instructions implementing AI/ ANN/DNN and synaptic weights defining ANN/DNN structures), and a communications interface 105 for communicating with the robot 106. The user interface 102 allows the user to create actionable tasks and/or usable workflows for the robot 106. The platform 100 interprets and implements these workflows as a hardware-agnostic brain 104 that interprets data from the robot 106 and input entered via the user interface 102, then performs one or more corresponding actions. The platform 100 can be implemented in any suitable computing device, including but not limited to a tablet computer (e.g., an iPad), a smartphone, a single-board computer, a desktop computer, a laptop, either local or in the cloud, etc. The platform 100 may provide the user interface 102 as a Graphical User Interface (GUI) via a touchscreen or other suitable display.")
A node library with behavior tree nodes for constructing behavior trees and various types of data blocks, the behavior tree comprising: …function block nodes, each function block node correspondingly implementing a service operation, each data block being configured to present corresponding data in the service operation of the function block node connected thereto (Versace: Paragraph 92, "FIGS. 9A and 9B show a dedicated GUI display screen 900 that provides part of the "Configure" component. It appears if the user selects the "add brain to robot" button 704 on the navigation page 700. The screen 900 shows several icons representing various GUI functionalities including an "add brain" button 902 and buttons associated with previously defined brains, shown in FIGS. 9A and 9B as "888" brain 904a, "AllDisplays" brain 904b, "AudioResponse Test" brain 904c, and "Button Stimulus" brain 904d (collectively, previously defined brains 904). The previously defined brains 904 can be created and stored locally by the user or accessed or downloaded via a cloud resource, such as a "brain store" or a free sharing framework. Screen 900 also includes a search input 906 that enables to the user to search for a particular brain and/or filter brains by name, robot type, or other brain characteristic. Brains can be "swapped" via a GUI on the iOS device as described below."; Paragraph 93, "Each brain (including each previously defined brain 904) may have an xml representation that can be shared across one or more devices (robots) simultaneously, sequentially, or both simultaneously and sequentially. For instance, a particular brain can be swapped among robots and/or transmitted to multiple robots via a GUI executing on a iOS device, Android device, or other suitable computing device."; Paragraph 94, "The user can apply one brain to many robots, one brain to many different types of robots, and/or many brains to one robot via screen 900 without having to know or understand the specifics of the brain commands, the robots' capabilities, or how to program the robots. If the user selects a brain that is incompatible with the selected robot, the GUI may present a message warning of the incompatibilities. For example, if the selected robot is a ground robot and the brain includes a behavior for a UA V, such as a "Fly Up" command, the system warns the user that the brain and/or its behavior(s) has one or more incompatibilities with the selected robot."; Paragraph 100, "FIGS. l lA-1 lE show a Behavior Editor 1100 that enables viewing, adding, editing, and deleting of behaviors 10 for use in creating and editing brains 40. As shown in FIG. l lA, available stimuli 20 are represented on a stimulus panel 1120 and available responses 30 are represented on a response panel 1130 displayed on either side of the behavior 10 being created/edited. The available stimuli 20 and responses 30 may be retrieved from a library stored in a local or cloud-based memory."; Paragraph 127, "The library, which may be stored in a memory or database, that handles generalizing across robotic structures has to make specific effort to abstract away the heterogeneous communication protocols. Each of these communication protocols has their own set of inherent properties. For example, UDP is connectionless and tends to be unreliable while TCP is connection-based and tends to be reliable. To abstract away these differences while maintaining a single API for all robots, helper objects are provided in the library to add some of those properties to communication protocols that don't have them inherently. For example, there is a reliable UDP stream to allow us to use communication paradigms that require reliability. This allows us to treat heterogeneous communication protocols as functionally similar which provides more flexibility for what algorithms can be used on robots.")
generating a behavior tree corresponding to a workflow on the basis of a behavior tree construction operation entered by a user on a graphical user interface, the behavior tree construction operation comprising selecting and connecting nodes from a node library…and the behavior tree comprising function block nodes, each function block node correspondingly implementing a service operation; (Versace: Paragraph 42, "The platform 100 includes a user interface 102 that enables a user to define a hardware-agnostic brain, a processor 104 that implements the hardware-agnostic brain (which may include processes and programs implementing Artificial Intelligence (AI)/Artificial Neural Network (ANN)/Deep Neural Network (DNN) processing), a memory 103 to store instructions for defining and executing the hardware-agnostic brain (including instructions implementing AI/ ANN/DNN and synaptic weights defining ANN/DNN structures), and a communications interface 105 for communicating with the robot 106. The user interface 102 allows the user to create actionable tasks and/or usable workflows for the robot 106. The platform 100 interprets and implements these workflows as a hardware-agnostic brain 104 that interprets data from the robot 106 and input entered via the user interface 102, then performs one or more corresponding actions. The platform 100 can be implemented in any suitable computing device, including but not limited to a tablet computer (e.g., an iPad), a smartphone, a single-board computer, a desktop computer, a laptop, either local or in the cloud, etc. The platform 100 may provide the user interface 102 as a Graphical User Interface (GUI) via a touchscreen or other suitable display."; Paragraph 54, "FIGS. 3A-6 illustrate fundamental building blocks for defining hardware-agnostic robot brains. As described in greater detail below, each brain comprises one or more behaviors. Each behavior, in tum, comprises one or more stimuli and one or more responses. The stimuli may represent changes in environmental parameters that the robot can detect with on-board or networked sensors; the status of the robot's internal and external components; and commands and signals received from other sources. The responses represent actions that the robot can take. Together, the stimuli and actions that make up a particular behavior specify how the robot works and functions."; Paragraph 102, "Stimuli can be linked by AND/OR logical conditions. Types of stimuli include but are not limited to: user input, such as touchscreen swipes, tilts, button pushes, etc.; machine vision (e.g., OpenCV), AI/ANN/DNN-related input (e.g., color, motion, face, object, and/or scene detection, robot-generated map); and quantitative sensor readings as well as device status from robot or controlling device, e.g. an iPad (e.g., WiFi signal strength and time of day). In some implementations there may be sub-dialogs for settings ( e.g., at what battery level should a stimulus be activated). The setting may be displayed without the need to open the sub-dialog, or the user may open the sub-dialog for editing. Machine vision stimuli may include selection of particular colors the robot can detect to generate a response. Other implementations can include objects, people, scenes, either stored in the knowledge base of the robot, objects the user has trained the brain to recognize, objects that have been trained by other users, object learned by other robots, or knowledge bases available in cloud resources."; Paragraph 121, "FIG. 14 shows a process layout for constructing a generalized API 70 suitable for interfacing between a robot or robot-specific API and a hardware-agnostic robot brain. This generalized API 70 is constructed in four process layers. In some embodiments, a single block is taken from Layer 1 of the process layout of API 70 that represents choosing a specific robot 72. In some embodiments, one or more blocks can be taken from Layer 2, as this is the step that configures hardware capabilities 74 of the chosen robot 72. Layer 3 is determined by the robot's movement capabilities 76, such as for example whether the robot 72 is a ground based robot, a UAV or a UUV. The final process step for Layer 4 is added for all robots as general commands 78, regardless of the selections and/or combinations of the previous process layers.")
adding and connecting a data block to each function block node in the behavior tree, each data block configured to present a corresponding data in the service operation of the function block connected thereto (Versace: Paragraph 42, "The platform 100 includes a user interface 102 that enables a user to define a hardware-agnostic brain, a processor 104 that implements the hardware-agnostic brain (which may include processes and programs implementing Artificial Intelligence (AI)/Artificial Neural Network (ANN)/Deep Neural Network (DNN) processing), a memory 103 to store instructions for defining and executing the hardware-agnostic brain (including instructions implementing AI/ ANN/DNN and synaptic weights defining ANN/DNN structures), and a communications interface 105 for communicating with the robot 106. The user interface 102 allows the user to create actionable tasks and/or usable workflows for the robot 106. The platform 100 interprets and implements these workflows as a hardware-agnostic brain 104 that interprets data from the robot 106 and input entered via the user interface 102, then performs one or more corresponding actions. The platform 100 can be implemented in any suitable computing device, including but not limited to a tablet computer (e.g., an iPad), a smartphone, a single-board computer, a desktop computer, a laptop, either local or in the cloud, etc. The platform 100 may provide the user interface 102 as a Graphical User Interface (GUI) via a touchscreen or other suitable display."; Paragraph 54, "FIGS. 3A-6 illustrate fundamental building blocks for defining hardware-agnostic robot brains. As described in greater detail below, each brain comprises one or more behaviors. Each behavior, in tum, comprises one or more stimuli and one or more responses. The stimuli may represent changes in environmental parameters that the robot can detect with on-board or networked sensors; the status of the robot's internal and external components; and commands and signals received from other sources. The responses represent actions that the robot can take. Together, the stimuli and actions that make up a particular behavior specify how the robot works and functions."; Paragraph 102, "Stimuli can be linked by AND/OR logical conditions. Types of stimuli include but are not limited to: user input, such as touchscreen swipes, tilts, button pushes, etc.; machine vision (e.g., OpenCV), AI/ANN/DNN-related input (e.g., color, motion, face, object, and/or scene detection, robot-generated map); and quantitative sensor readings as well as device status from robot or controlling device, e.g. an iPad (e.g., WiFi signal strength and time of day). In some implementations there may be sub-dialogs for settings ( e.g., at what battery level should a stimulus be activated). The setting may be displayed without the need to open the sub-dialog, or the user may open the sub-dialog for editing. Machine vision stimuli may include selection of particular colors the robot can detect to generate a response. Other implementations can include objects, people, scenes, either stored in the knowledge base of the robot, objects the user has trained the brain to recognize, objects that have been trained by other users, object learned by other robots, or knowledge bases available in cloud resources."; Paragraph 121, "FIG. 14 shows a process layout for constructing a generalized API 70 suitable for interfacing between a robot or robot-specific API and a hardware-agnostic robot brain. This generalized API 70 is constructed in four process layers. In some embodiments, a single block is taken from Layer 1 of the process layout of API 70 that represents choosing a specific robot 72. In some embodiments, one or more blocks can be taken from Layer 2, as this is the step that configures hardware capabilities 74 of the chosen robot 72. Layer 3 is determined by the robot's movement capabilities 76, such as for example whether the robot 72 is a ground based robot, a UA Vora UUV. The final process step for Layer 4 is added for all robots as general commands 78, regardless of the selections and/or combinations of the previous process layers.") Examiner interprets the stimuli, responses, and final combined blocks to the equivalent to the data and function block nodes both under broadest reasonable interpretation and in view of Applicant’s specification.
Versace does not explicitly disclose the following, however, in analogous art of workflow creation, Luvaas discloses the limitations below:
… flow control nodes selected from the node library and configured to implement logic controls in the workflow independently of specific service operations… (Luvaas: Column 8 lines 42-50, “FIG. 3 illustrates an example of a workflow development, according to certain embodiments of the present disclosure. A workflow 300 can be visualized in a workflow assembly area of a workflow assembly applications as a set of icons and connections therebetween, where each icon corresponds to a node of the workflow. As illustrated in FIG. 3, the visualization includes a start indicator 310, an icon 320, a connector line 330, a branch 340, a flow control indicator 350, a flow control label 360, and an end indicator 370.”; Column 12 lines 18-33, “FIG. 8 illustrates an example of a result available upon an execution of a workflow, according to certain embodiments of the present disclosure. In example, the workflow is defined by using a workflow assembly application. In a workflow assembly window 810, the workflow assembly application presents options to execute the workflow. A user selection 820 of one of these options (e.g., selecting a trial run or a run) triggers an execution of the workflow. The execution can be performed at a remote system (e.g., the computer system 220 of FIG. 2) or locally at the user device executing the workflow assembly application (e.g., the user device 210 of FIG. 2, where the user device 210 can make calls to the library 226 of the workflow platform 221). The execution results in a result panel 830 that is presented in an area within the workflow assembly window 810 (e.g., an area superimposed over a workflow assembly area).”; Column 11 lines 8-26, “Visual guidance is shown to the user while a node's icon is added from the menu area to the workspace assembly area (e.g., via a drag-and-drop interaction) in the form of drop zones that identify the possible placement locations where the icon can be dropped. These drop zones are displayed based on the node type of the node being added to the workflow, the node types of nodes already in the workflow, and the position of such nodes relative to each other or the terminus indicators. Such drop zones determination can rely on assembly rules, whereby the node type and the process identifier are used in a look up of the assembly rules. Drop zones appear overlaid on top of applicable flow control indicators and to either side of existing node icons where a branch in the workflow would be allowed. These drop zones give a clear visual indicator of where the node's icon can be placed based on otherwise obscure or complex assembly rules. Flow control indicators where the node cannot be placed do not display drop zones and do not allow icon-dropping into that location or show node previews at that location.”; Column 16 line 54 – Column 17 line 15, “FIG. 12 illustrates an example of modules of a workflow assembly application 1210, according to certain embodiments of the present disclosure. The modules can be implemented as program codes and include a workflow UI 1220, a user interaction layer 1222, a UI rendering layer 1230, and a redux store 1240 having a view model 1242 and a workflow model 1244. The workflow UI 1220 is configured to present a workflow assembly window. The user interaction layer 1222 is configured to detect user interactions (e.g., drag-and-drop interactions) and provide related information to the UI rendering layer 1230 such that previews and visual guidance can be provided and to the view model 1242 such that the icons representing the node and the overall visualized workflow is updated (in which case the view model 1242 can update the workflow model 1244). The UI rendering layer 1230 is configured to render the workflow including, for instance, the icons representing its nodes, the connector lines, the flow control indicators, the flow control labels, the visual guidance, and the previews. The view model 1142 stores node view objects that can be used to view (e.g., visualize) the nodes of the process. The workflow model 1244 can be a JSON model that stores JSON objects, each corresponding to a node and its related properties (e.g., connection to other nodes, reference to executable code, etc.). Updates from the user interaction layer 1222 can drive updates to the view model 1242 and, in turn, to the workflow model 1244. Updates to the workflow model 1244 can be sent to a computer system that provides backend functionalities, such as the computer system 220 of FIG. 2.”)
…the node library comprising flow control nodes and function block nodes, the behavior tree comprising flow control nodes selected from the node library and configured to implement logic controls in the workflow independently of specific service operations… (Luvaas: Column 8 lines 42-50, “FIG. 3 illustrates an example of a workflow development, according to certain embodiments of the present disclosure. A workflow 300 can be visualized in a workflow assembly area of a workflow assembly applications as a set of icons and connections therebetween, where each icon corresponds to a node of the workflow. As illustrated in FIG. 3, the visualization includes a start indicator 310, an icon 320, a connector line 330, a branch 340, a flow control indicator 350, a flow control label 360, and an end indicator 370.”; Column 12 lines 18-33, “FIG. 8 illustrates an example of a result available upon an execution of a workflow, according to certain embodiments of the present disclosure. In example, the workflow is defined by using a workflow assembly application. In a workflow assembly window 810, the workflow assembly application presents options to execute the workflow. A user selection 820 of one of these options (e.g., selecting a trial run or a run) triggers an execution of the workflow. The execution can be performed at a remote system (e.g., the computer system 220 of FIG. 2) or locally at the user device executing the workflow assembly application (e.g., the user device 210 of FIG. 2, where the user device 210 can make calls to the library 226 of the workflow platform 221). The execution results in a result panel 830 that is presented in an area within the workflow assembly window 810 (e.g., an area superimposed over a workflow assembly area).”; Column 11 lines 8-26, “Visual guidance is shown to the user while a node's icon is added from the menu area to the workspace assembly area (e.g., via a drag-and-drop interaction) in the form of drop zones that identify the possible placement locations where the icon can be dropped. These drop zones are displayed based on the node type of the node being added to the workflow, the node types of nodes already in the workflow, and the position of such nodes relative to each other or the terminus indicators. Such drop zones determination can rely on assembly rules, whereby the node type and the process identifier are used in a look up of the assembly rules. Drop zones appear overlaid on top of applicable flow control indicators and to either side of existing node icons where a branch in the workflow would be allowed. These drop zones give a clear visual indicator of where the node's icon can be placed based on otherwise obscure or complex assembly rules. Flow control indicators where the node cannot be placed do not display drop zones and do not allow icon-dropping into that location or show node previews at that location.”; Column 16 line 54 – Column 17 line 15, “FIG. 12 illustrates an example of modules of a workflow assembly application 1210, according to certain embodiments of the present disclosure. The modules can be implemented as program codes and include a workflow UI 1220, a user interaction layer 1222, a UI rendering layer 1230, and a redux store 1240 having a view model 1242 and a workflow model 1244. The workflow UI 1220 is configured to present a workflow assembly window. The user interaction layer 1222 is configured to detect user interactions (e.g., drag-and-drop interactions) and provide related information to the UI rendering layer 1230 such that previews and visual guidance can be provided and to the view model 1242 such that the icons representing the node and the overall visualized workflow is updated (in which case the view model 1242 can update the workflow model 1244). The UI rendering layer 1230 is configured to render the workflow including, for instance, the icons representing its nodes, the connector lines, the flow control indicators, the flow control labels, the visual guidance, and the previews. The view model 1142 stores node view objects that can be used to view (e.g., visualize) the nodes of the process. The workflow model 1244 can be a JSON model that stores JSON objects, each corresponding to a node and its related properties (e.g., connection to other nodes, reference to executable code, etc.). Updates from the user interaction layer 1222 can drive updates to the view model 1242 and, in turn, to the workflow model 1244. Updates to the workflow model 1244 can be sent to a computer system that provides backend functionalities, such as the computer system 220 of FIG. 2.”)
parsing the behavior tree connected with data block, deploying the workflow corresponding to the behavior tree to a runtime of a corresponding workcell so that the runtime executes the workflow to control each resource in the workcell to execute the service operation according to the workflow, and providing corresponding data obtained in an execution process of the service operation to the corresponding data block for display (Luvaas: Column 5 line 51 – Column 6 line 4, “The options 140 can trigger the execution of the workflow (or, more specifically, the parsing of the workflow model and execution of the referenced executable codes according to the execution sequence and the configuration information of each node). FIG. 1 illustrates two options: a run and a trial run. The run represents manual trigger for executing the workflow at runtime by using the data set(s) to which the workflow applies. In comparison, the trial run represents another manual trigger, but for executing the workflow by using a portion of the data set(s) (e.g., instead of all the data entries, the trial run uses the first two-hundred data entries (or some other configured number) or a data byte amount (e.g., one gigabyte of the data)). This way, the trial run allows the validation of the workflow by using the real, applicable data, but without the need to process the entire data set(s). Other triggers for the execution are possible. For example, a schedule with recurrent runtime execution. In another example, a data event can be subscribed to and, upon receiving, an indication of its occurrence, a runtime execution can be automatically performed.”; Column 7 line 54 – Column 8 line 13, “The workflow platform 221 can store multiple workflow models 222, some of which may be received from the user device 210, and others may be received from other user devices and available to the user device 210. The workflow platform 221 can also maintain assembly rules 224, a subset or the entirety of which can be sent to the workflow assembly application 212 to support the workflow creating. Further, the workflow platform 221 can maintain a library 226 of executable codes, of which can correspond to a node type. Executable code can be application program code or code of an API for an application that may reside on a remote system. The executable codes can be predefined by a service provider of the workflow platform 221 or collected over time from one or more users. The workflow platform 221 can also include an execution engine 228 for runtime executions of the workflow models 222. For instance, the workflow models 222 include a copy of the workflow model 218. The execution engine 228 parses this copy, determines the executable codes and their execution sequence and parameters, and uses the library 226 to then execute each executable code according to the execution sequence and the parameters of that executable code. Data generated by the execution of the executable code can be stored in the data store 240 and, depending on the execution sequence, can be input to the next executable code. The final data output can be stored in the data destination 250.”; Column 16 lines 39-46, “At operation 1108, the device presents the report. For example, the result is presented in a result panel of the workflow assembly window. At operation 1110, the device receives a request to view a node log message. The node log message can correspond to the portion of the report specific to a node of the workflow and can include, for instance, an error report about the processing by the node and/or a link to the relevant node-processed data.”)
Versace discloses a method for building decision trees for workflow execution. Luvaas discloses an interface for building, connecting and running nodes for a workflow. By the effective filing date of Applicant’s claimed invention it would have been obvious to one of ordinary skill in the art to combine the methods of Versace with the teachings of Luvaas in order to improve the ease of access, and reduce inaccuracies in workflow assembly as disclosed by Luvaas (Luvaas: Column 3 lines 47-63, “Because the manual input can be avoided, the accuracy of the workflow can be improved, whereby user-induced errors may be eliminated. Further, the workflow assembly application provides a user-intuitive and guided tool to develop a properly executable workflow. The user need not be a programmer and need not have to create new program applications. Various different types of workflows can be developed, stored, modified over time, and made available to or shared between multiple users, thereby providing a scalable workflow platform.”)
Claim(s) 2 –
Versace in view of Luvaas disclose the limitations of claim 1
Versace further discloses the following:
a data block label for indicating a type of the data block (Versace: Paragraph 102, "Stimuli can be linked by AND/OR logical conditions. Types of stimuli include but are not limited to: user input, such as touchscreen swipes, tilts, button pushes, etc.; machine vision (e.g., OpenCV), AI/ANN/DNN-related input (e.g., color, motion, face, object, and/or scene detection, robot-generated map); and quantitative sensor readings as well as device status from robot or controlling device, e.g. an iPad (e.g., WiFi signal strength and time of day). In some implementations there may be sub-dialogs for settings ( e.g., at what battery level should a stimulus be activated). The setting may be displayed without the need to open the sub-dialog, or the user may open the sub-dialog for editing. Machine vision stimuli may include selection of particular colors the robot can detect to generate a response. Other implementations can include objects, people, scenes, either stored in the knowledge base of the robot, objects the user has trained the brain to recognize, objects that have been trained by other users, object learned by other robots, or knowledge bases available in cloud resources."; Paragraph 104, "As shown in FIGS. l lA-1 lE, responses 30 are depicted as triangles arranged sequentially (in a line) and selectable from the sliding panel 1130 on the right. Once conditions to satisfy a stimulus ( or multiple stimuli, e.g., three stimuli arranged in AND statements) are met, one or more responses are triggered. These responses can be executed sequentially, and while being executed, other stimuli processing can be prevented from gaining access to other stimuli using a scheduler as explained below with respect to FIGS. ISA and 15B. In other implementations, the sequence of responses can be broken by intervening stimuli. Responses are converted from robot-agnostic to robot-specific in software as explained below with respect to FIG. 14. For example, a "Move forward for 2 meters" on a ground robot, and the same command on a drone, will result in two very different set of motor commands for a robot, which will need to be handled in software to achieve equivalent behavioral results.")
a data block body with a display area for presenting specific data (Versace: Paragraph 105, "Responses 10 can include changing the status of the display of the robot (when available), specific movement of the robot, sounds (e.g., speech), tilt/rotations of the robot, picture/video, turning on/off lights (e.g., LED), pausing the robot, drone-specific operations (e.g., take off). In this example, available responses include display 30a)(e.g., if the robot has a screen, it can be a picture/video/image on the screen, color, text, etc.), light 30b (e.g., turn on a lightemitting diode (LED)), move 30c (e.g., trigger a walking or rolling motor), sound 30d (e.g., record sound with a microphone or emit sound with a speaker), tilt 30e (e.g., with an appropriate tilt actuator), drone 30f (e.g., fly in a certain direction, speed, or altitude), camera 30g (e.g., acquire still or video images with an image sensor), and pause 30h (e.g., stop moving). Additionally, custom actions can be available from the cloud, an on-line store, or other users."; Paragraph 107, "Stimuli 20 and responses 30 can be re-arranged by dragging and dropping in the interface 1100, and a specific response can formed by the user recording specific movement by the robot performed under the control of the user, and saved as custom movements. For example, in FIGS. l lA and l lB, the user selects and adds the location stimulus 20a and the vision stimulus 20d to the behavior 10 and possibly adjust the parameters of these stimuli 20 using the stimuli/response editor 1140. In FIG. l lC, the user adds a Move Response 30c to the behavior 10 and selects the direction and duration of the movement using the stimuli/response editor 1140. FIG. l lD shows that the user has added the image acquisition response 30g and the Drone Response 30f, which enables the use to select take off or land using the stimuli/response editor 1140. And FIG. l lE shows a Save button 1150 that enables the user to name and save the behavior 10, which can then be used in a brain 40, exported, exchanged, and posted on a cloud server store."; Paragraph 110, "In general, the interface 1200 may enable use of a dial format and/or swipe mode on a single screen. For instance, dials may provide indications of possible robot actions and/or easily recognizable symbols or icons ( e.g., in addition to or instead of text). The user interface may give the user the ability to playback a behavior via button press, to show and/or hide a heads-up display (HUD), and/or to customize a HUD. In some implementations, supported controls may include but are not limited to: two-dial control; swipe control; two-dial control and swipe control on the same screen; tilt control (e.g., using the iPad sensors, move the robot in the direction of a device tilt); and voice commands. For swipe control, the robot may move in the direction of the swipe and may continue moving until the user lifts his or her swiping finger. The interface may enable the user to create a pattern, by swiping, for the robot to follow. (In some implementations the interface may show a trail on the screen in the direction of the swipe.) Similarly, vertical flying control altitude may utilize two finger gestures. Similarly, voice commands may encompass a plurality of actions. Other commands may include: device-type commands ( e.g., forward, stop, right, left, faster), pet-related commands (e.g., come, heel), and other commands (e.g., wag, to move the iPhone in a Romotive Romo back and forth or to roll an Orbotix Sphero back and forth).")
Claim(s) 3 –
Versace in view of Luvaas disclose the limitations of claims 1-2
Versace does not explicitly disclose the following, however, in analogous art of generating workflows Luvaas teaches the limitations below:
wherein the size of the display area is adjustable (Luvaas: Column 6 lines 3-16, “Although FIG. 1 shows that an icon has the same representation in the menu area 120 and the workflow assembly area 130, the embodiments of the present disclosure are not limited as such. For example, the representation of the icon in the menu area 120 can include a graphical element with, optionally, text indicating a node type (e.g. a display name). When the icon is added to the workflow in the workflow assembly area 130, its presentation can be updated. For instance, its presentation is expanded to include additional information indicating that the corresponding node has been added such to give the appearance to the user that the node now appears in the workflow. This appearance can include the graphical element (resized), the display name, and a selectable option (e.g., for a spillover menu) with a surrounding bounding box.”; Column 9 line 59 – Column 10 line 6, “In another example, zoom in and zoom out functionalities are possible. A zoom out can result in representing a set of nodes of the workflow with a single icon (e.g., a container). This zoomed out icon can provide contexts about the represented nodes (e.g., information, such as a list of the nodes or their node types). Upon a zoom in, the presentation of this icon can be expanded to shown the icons corresponding to the nodes. The zoom in and out functionality need not alter the workflow model and correspond to a mere visual encapsulation, whereby the rendering of this workflow model on the GUI can be adjusted based on the zoom level. The workflow assembly application can allow the user to change the view zoom level. Zooming out allows the user to view the complete contents of complex workflows while zooming in reveals increasing fidelity and node details.”)
Versace discloses a method for building decision trees for workflow execution. Luvaas discloses an interface for building, connecting and running nodes for a workflow. By the effective filing date of Applicant’s claimed invention it would have been obvious to one of ordinary skill in the art to combine the methods of Versace with the teachings of Luvaas in order to improve the ease of access, and reduce inaccuracies in workflow assembly as disclosed by Luvaas (Luvaas: Column 3 lines 47-63, “Because the manual input can be avoided, the accuracy of the workflow can be improved, whereby user-induced errors may be eliminated. Further, the workflow assembly application provides a user-intuitive and guided tool to develop a properly executable workflow. The user need not be a programmer and need not have to create new program applications. Various different types of workflows can be developed, stored, modified over time, and made available to or shared between multiple users, thereby providing a scalable workflow platform.”)
Claim(s) 4 –
Versace in view of Luvaas disclose the limitations of claims 1-2
Versace further discloses the following:
wherein the type of the data block comprises some or all of data pair, datasheet, image, video, and chart (Versace: Paragraph 62, "FIG. SB shows an example of how behaviors (neurons) in a brain can be connected to each other or "chained together." In this case, a first behavior 51 Oa produces a particular output ( e.g., motion in a particular direction or to particular GPS coordinates) that serves as a stimulus 520a for a second behavior 510b. If the second behavior's other stimulus 520b (e.g., recognition of a particular object in imagery acquired by a camera) is present, then the second behavior 510b produces its own response 530, which in turn may stimulate another behavior. Thus, triggering of the first behavior 51 0a is a stimulus for a second behavior Slob."; Paragraph 135, "Additionally, the robotic brain may be configured with an arbitrary number of behaviors (e.g., pairs of stimulus/response sets 160). Behaviors can be created and edited by the user based on stimuli/responses defined above ( e.g., stimuli directly based on reading and preprocessing of robot sensors). They can also be chosen from a collection of stimuli/responses directly generated by machine vision ( e.g., Open CV) AI/ANN/DNN algorithms in the sensory objects module 110 and navigation modules 130. For example, a particular behavior can be defined to include predetermined stimuli, such as time of the day (e.g., it's 2:00 PM as determined by the Robot processor, or the controlling cell phone), or a stimulus learned by the sensory system 110 (e.g. what "John" looks like). Similarly, a response associated with the behavior and executed in response to the stimulus can be defined from the navigation system 130 as "go to the kitchen." The resulting behavior would cause the robot to go to the kitchen (as learned by the navigation system) when the robot sees John, as learned by elaborating video/audio sensory information and/or other signals (e.g., wireless signals originating from John's phone).”)
Claim(s) 5 –
Versace in view of Luvaas disclose the limitations of claim 1
Versace further discloses the following:
the workflow is an OT domain workflow (Versace: Paragraph 130, "The brains (collection of behaviors) described herein can be combined and associated with other forms of autonomous behaviors, such as autonomous sensory object recognition (such as but not limited to audition, vision, radio signals, LIDAR, or other point-cloud input, as well as any combination of the above sensors), in at least the following ways."; Paragraph 135, "Additionally, the robotic brain may be configured with an arbitrary number of behaviors (e.g., pairs of stimulus/response sets 160). Behaviors can be created and edited by the user based on stimuli/responses defined above ( e.g., stimuli directly based on reading and preprocessing of robot sensors). They can also be chosen from a collection of stimuli/responses directly generated by machine vision ( e.g., Open CV) AI/ANN/DNN algorithms in the sensory objects module 110 and navigation modules 130. For example, a particular behavior can be defined to include predetermined stimuli, such as time of the day (e.g., it's 2:00 PM as determined by the Robot processor, or the controlling cell phone), or a stimulus learned by the sensory system 110 (e.g. what "John" looks like). Similarly, a response associated with the behavior and executed in response to the stimulus can be defined from the navigation system 130 as "go to the kitchen." The resulting behavior would cause the robot to go to the kitchen (as learned by the navigation system) when the robot sees John, as learned by elaborating video/audio sensory information and/or other signals (e.g., wireless signals originating from John's phone)."; Paragraph 137, "In the example instantiation in FIG. ISA, at least two autonomy modules 110, 120 and several user-defined behaviors 160 can control robotic effectors via the scheduler 140. For example, the sensory module 110 could command the robot to make a camera movement to learn more about an object visual appearance with a right movement of the robot, the navigation system 130 may command the robot to explore the environment with a left movement of the robot, and the behavior 160 may command the robot to go backward following the appearance of a soccer ball. As an example instantiation, the scheduler can use a neural-like competitive cueing network ( or ANN, or DNN) to appropriately sequence actions based on their relative importance and timing.")
the workcell is a workcell in an OT domain (Versace: Paragraph 130, "The brains (collection of behaviors) described herein can be combined and associated with other forms of autonomous behaviors, such as autonomous sensory object recognition (such as but not limited to audition, vision, radio signals, LIDAR, or other point-cloud input, as well as any combination of the above sensors), in at least the following ways."; Paragraph 135, "Additionally, the robotic brain may be configured with an arbitrary number of behaviors (e.g., pairs of stimulus/response sets 160). Behaviors can be created and edited by the user based on stimuli/responses defined above ( e.g., stimuli directly based on reading and preprocessing of robot sensors). They can also be chosen from a collection of stimuli/responses directly generated by machine vision ( e.g., Open CV) AI/ANN/DNN algorithms in the sensory objects module 110 and navigation modules 130. For example, a particular behavior can be defined to include predetermined stimuli, such as time of the day (e.g., it's 2:00 PM as determined by the Robot processor, or the controlling cell phone), or a stimulus learned by the sensory system 110 (e.g. what "John" looks like). Similarly, a response associated with the behavior and executed in response to the stimulus can be defined from the navigation system 130 as "go to the kitchen." The resulting behavior would cause the robot to go to the kitchen (as learned by the navigation system) when the robot sees John, as learned by elaborating video/audio sensory information and/or other signals (e.g., wireless signals originating from John's phone)."; Paragraph 137, "In the example instantiation in FIG. ISA, at least two autonomy modules 110, 120 and several user-defined behaviors 160 can control robotic effectors via the scheduler 140. For example, the sensory module 110 could command the robot to make a camera movement to learn more about an object visual appearance with a right movement of the robot, the navigation system 130 may command the robot to explore the environment with a left movement of the robot, and the behavior 160 may command the robot to go backward following the appearance of a soccer ball. As an example instantiation, the scheduler can use a neural-like competitive cueing network ( or ANN, or DNN) to appropriately sequence actions based on their relative importance and timing.")
the resource is an OT resource (Versace: Paragraph 130, "The brains (collection of behaviors) described herein can be combined and associated with other forms of autonomous behaviors, such as autonomous sensory object recognition (such as but not limited to audition, vision, radio signals, LIDAR, or other point-cloud input, as well as any combination of the above sensors), in at least the following ways."; Paragraph 135, "Additionally, the robotic brain may be configured with an arbitrary number of behaviors (e.g., pairs of stimulus/response sets 160). Behaviors can be created and edited by the user based on stimuli/responses defined above ( e.g., stimuli directly based on reading and preprocessing of robot sensors). They can also be chosen from a collection of stimuli/responses directly generated by machine vision (e.g., OpenCV) AI/ANN/DNN algorithms in the sensory objects module 110 and navigation modules 130. For example, a particular behavior can be defined to include predetermined stimuli, such as time of the day (e.g., it's 2:00 PM as determined by the Robot processor, or the controlling cell phone), or a stimulus learned by the sensory system 110 ( e.g. what "John" looks like). Similarly, a response associated with the behavior and executed in response to the stimulus can be defined from the navigation system 130 as "go to the kitchen." The resulting behavior would cause the robot to go to the kitchen (as learned by the navigation system) when the robot sees John, as learned by elaborating video/audio sensory information and/or other signals (e.g., wireless signals originating from John's phone)."; Paragraph 137, "In the example instantiation in FIG. ISA, at least two autonomy modules 110, 120 and several user-defined behaviors 160 can control robotic effectors via the scheduler 140. For example, the sensory module 110 could command the robot to make a camera movement to learn more about an object visual appearance with a right movement of the robot, the navigation system 130 may command the robot to explore the environment with a left movement of the robot, and the behavior 160 may command the robot to go backward following the appearance of a soccer ball. As an example instantiation, the scheduler can use a neural-like competitive cueing network ( or ANN, or DNN) to appropriately sequence actions based on their relative importance and timing.")
Claim(s) 6 –
Versace in view of Luvaas disclose the limitations of claims 1 and 5
Versace further discloses the following:
generating a microservice on the basis of the behavior tree, so that an IT device triggers a runtime of the workcell to execute the OT domain workflow by calling the microservice (Versace: Paragraph 131, "FIG. ISA depicts one example application where a (real or virtual) robot 100 is controlled by at least two autonomy modules (a sensory object module 110 and a motivation module 120) and several user-defined behaviors 160. The robot implements the autonomous, user-defined behaviors using various on-board sensors (e.g., cameras, accelerometer, gyro, IR, etc) that form a robot sensory system I 00 and actuators/effectors (e.g., motors in tracks/propellers). The robot may also be linked to other sensors on associated hardware (e.g., a cell phone mounted on the robot, or a controlling iPad) that provide sensory input to an artificial robotic brain executing machine vision ( e.g., Open CV), AI, ANN, and DNN algorithms. These algorithms may be executed by:"; Paragraph 135, "Additionally, the robotic brain may be configured with an arbitrary number of behaviors (e.g., pairs of stimulus/response sets 160). Behaviors can be created and edited by the user based on stimuli/responses defined above ( e.g., stimuli directly based on reading and preprocessing of robot sensors). They can also be chosen from a collection of stimuli/responses directly generated by machine vision ( e.g., Open CV) AI/ANN/DNN algorithms in the sensory objects module 110 and navigation modules 130. For example, a particular behavior can be defined to include predetermined stimuli, such as time of the day (e.g., it's 2:00 PM as determined by the Robot processor, or the controlling cell phone), or a stimulus learned by the sensory system 110 (e.g. what "John" looks like). Similarly, a response associated with the behavior and executed in response to the stimulus can be defined from the navigation system 130 as "go to the kitchen." The resulting behavior would cause the robot to go to the kitchen (as learned by the navigation system) when the robot sees John, as learned by elaborating video/audio sensory information and/or other signals (e.g., wireless signals originating from John's phone)."; Paragraph 138, "FIG. 15B is a flowchart that illustrates an example scheduling process executed by the scheduler 140. The scheduler 140 starts by sorting its inputs and then computing the relative weight of each input. Inputs can come from a variety of sources, including on-board sensors 100, off-board sensors, and user inputs, and the scheduler can scale from having one input to many. Generally speaking, the input sources include modules currently executing algorithms ( e.g., the navigation module 130), the motivation of the robot (motivation module 120), command packets coming from a controller, and the currently executing brain (behaviors 160). Inputs with the highest weight execute, while inputs with lower weights that do not conflict with other inputs execute if they pass through a series of checks."; Paragraph 142, "Beyond the basic series of weights, the scheduler 140 also executes one or more sorting steps. The first step involves sorting commands that use discrete hardware resources from commands that affect things like settings and parameter adjustment ( operation 854). Settings changes are parsed and checked for conflict ( operation 856). If there are no conflicts, then all settings changes push (operation 858). If there are conflicts and there are weights that can be used to break the conflict, they are used. If everything is weighted identically and two settings conflict, than neither executes or a symmetrybreaking procedure may be applied (e.g., most-used behavior wins). Many of these settings packets can be executed simultaneously. Next, the packets that affect discrete system resources are further sorted based on the affected resource(s) ( operation 860). Commands that can inherently affect each other but don't necessarily do so are kept together. For example, audio playback and audio recording may be kept in the same stream, as certain devices cannot record and playback and even if the option is available there are still constraints to deal with to avoid feedback."; Paragraph 146, "The scheduling process of FIG. 15B allows the robot to look for an object in the environment, then step backward as required by the user-defined behavior, then go on exploring the environment. In order to determine the relative importance of actions, the scheduler 140 may use the graphical placement of behaviors in the brain to determine the relative importance of each behavior in the brain. In other implementations, a user may be able to provide positive and/or negative reinforcement (e.g., during a training process with the robot) in order to train the robot to develop an understanding of which behaviors and/or responses to prioritize over others. In another implementation, an ANN/DNN autonomously prioritizes scheduling based on learning and experience. In another implementation, the user may manually define the importance of each behavior, e.g., determining which behavior gets the precedence over other behaviors when both behaviors comprise stimuli which would activate their two different sets of responses in reaction a single event. For example, when an image contains two stimuli ( e.g., a face and the color red) which simultaneously activate two sets of responses, the user may manually predetermine when behavior 1 is engaged and when behavior 2 may be performed if behavior 1 is not complete (e.g., the user may indicate that behavior 2 may interrupt behavior 1, may start after behavior 1 completes, and/or the like).")
Claim(s) 7 –
Versace in view of Luvaas disclose the limitations of claims 1 and 5-6
Versace does not explicitly disclose the following, however, in analogous art of generating workflows Luvaas teaches the limitations below:
providing the corresponding data in the executing process of the service operation to the corresponding data block for display directly, or, providing the corresponding data obtained in the process of the service operation to the corresponding data block for display through the microservice Luvaas: Column 5 line 51 – Column 6 line 4, “The options 140 can trigger the execution of the workflow (or, more specifically, the parsing of the workflow model and execution of the referenced executable codes according to the execution sequence and the configuration information of each node). FIG. 1 illustrates two options: a run and a trial run. The run represents manual trigger for executing the workflow at runtime by using the data set(s) to which the workflow applies. In comparison, the trial run represents another manual trigger, but for executing the workflow by using a portion of the data set(s) (e.g., instead of all the data entries, the trial run uses the first two-hundred data entries (or some other configured number) or a data byte amount (e.g., one gigabyte of the data)). This way, the trial run allows the validation of the workflow by using the real, applicable data, but without the need to process the entire data set(s). Other triggers for the execution are possible. For example, a schedule with recurrent runtime execution. In another example, a data event can be subscribed to and, upon receiving, an indication of its occurrence, a runtime execution can be automatically performed.”; Column 7 line 54 – Column 8 line 13, “The workflow platform 221 can store multiple workflow models 222, some of which may be received from the user device 210, and others may be received from other user devices and available to the user device 210. The workflow platform 221 can also maintain assembly rules 224, a subset or the entirety of which can be sent to the workflow assembly application 212 to support the workflow creating. Further, the workflow platform 221 can maintain a library 226 of executable codes, of which can correspond to a node type. Executable code can be application program code or code of an API for an application that may reside on a remote system. The executable codes can be predefined by a service provider of the workflow platform 221 or collected over time from one or more users. The workflow platform 221 can also include an execution engine 228 for runtime executions of the workflow models 222. For instance, the workflow models 222 include a copy of the workflow model 218. The execution engine 228 parses this copy, determines the executable codes and their execution sequence and parameters, and uses the library 226 to then execute each executable code according to the execution sequence and the parameters of that executable code. Data generated by the execution of the executable code can be stored in the data store 240 and, depending on the execution sequence, can be input to the next executable code. The final data output can be stored in the data destination 250.”; Column 16 lines 39-46, “At operation 1108, the device presents the report. For example, the result is presented in a result panel of the workflow assembly window. At operation 1110, the device receives a request to view a node log message. The node log message can correspond to the portion of the report specific to a node of the workflow and can include, for instance, an error report about the processing by the node and/or a link to the relevant node-processed data.”)
Versace discloses a method for building decision trees for workflow execution. Luvaas discloses an interface for building, connecting and running nodes for a workflow. By the effective filing date of Applicant’s claimed invention it would have been obvious to one of ordinary skill in the art to combine the methods of Versace with the teachings of Luvaas in order to improve the ease of access, and reduce inaccuracies in workflow assembly as disclosed by Luvaas (Luvaas: Column 3 lines 47-63, “Because the manual input can be avoided, the accuracy of the workflow can be improved, whereby user-induced errors may be eliminated. Further, the workflow assembly application provides a user-intuitive and guided tool to develop a properly executable workflow. The user need not be a programmer and need not have to create new program applications. Various different types of workflows can be developed, stored, modified over time, and made available to or shared between multiple users, thereby providing a scalable workflow platform.”)
Claim(s) 8 –
Versace in view of Luvaas disclose the limitations of claims 1 and 5-6
Versace further discloses the following:
registering the microservice on a knowledge middle platform, so that the IT device triggers the runtime of the workcell to execute the OT domain workflow by calling the microservice through the knowledge middle platform (Versace: Paragraph 92, "FIGS. 9A and 9B show a dedicated GUI display screen 900 that provides part of the "Configure" component. It appears if the user selects the "add brain to robot" button 704 on the navigation page 700. The screen 900 shows several icons representing various GUI functionalities including an "add brain" button 902 and buttons associated with previously defined brains, shown in FIGS. 9A and 9B as "888" brain 904a, "AllDisplays" brain 904b, "AudioResponse Test" brain 904c, and "Button Stimulus" brain 904d (collectively, previously defined brains 904). The previously defined brains 904 can be created and stored locally by the user or accessed or downloaded via a cloud resource, such as a "brain store" or a free sharing framework. Screen 900 also includes a search input 906 that enables to the user to search for a particular brain and/or filter brains by name, robot type, or other brain characteristic. Brains can be "swapped" via a GUI on the iOS device as described below."; Paragraph 93, "Each brain (including each previously defined brain 904) may have an xml representation that can be shared across one or more devices (robots) simultaneously, sequentially, or both simultaneously and sequentially. For instance, a particular brain can be swapped among robots and/or transmitted to multiple robots via a GUI executing on a iOS device, Android device, or other suitable computing device."; Paragraph 94, "The user can apply one brain to many robots, one brain to many different types of robots, and/or many brains to one robot via screen 900 without having to know or understand the specifics of the brain commands, the robots' capabilities, or how to program the robots. If the user selects a brain that is incompatible with the selected robot, the GUI may present a message warning of the incompatibilities. For example, if the selected robot is a ground robot and the brain includes a behavior for a UAV, such as a "Fly Up" command, the system warns the user that the brain and/or its behavior(s) has one or more incompatibilities with the selected robot."; Paragraph 118, "FIG. 13 shows an example robot knowledge center. In this example, the knowledge of the robot is divided in people, objects, and places. In this particular example, people and objects views are populated automatically, e.g., by the sensory module 110 (people, objects) and navigation module 130 (places) in the system of FIG. ISA, which is described in greater detail below. The user can select, e.g., via a touch screen on a tablet or a mouse, a specific view of a person or object, and provide a verbal or iconic label. Additionally, the user can take multiple views of an object, person, location (map), or a scene (e.g., multiple views of the kitchen) and group them in a single entity that combines all those views (e.g., several views of "John", or a cup, or "John's kitchen"), e.g. via a drag/drop interface. The user can also edit the map generated by module 130 (places), by providing verbal and iconic ( e.g., color) labels to specific areas of the environment mapped by the robot. These verbally or iconically defined objects, people, and places can be used by the stimulus/response system.")
Claim(s) 9 –
Versace in view of Luvaas disclose the limitations of claims 1, 5-6, and 8
Versace further discloses the following:
providing the corresponding data obtained in the execution processes of the service operation to the knowledge middle platform and (Versace: Paragraph 137, "In the example instantiation in FIG. 15A, at least two autonomy modules 110, 120 and several user-defined behaviors 160 can control robotic effectors via the scheduler 140. For example, the sensory module 110 could command the robot to make a camera movement to learn more about an object visual appearance with a right movement of the robot, the navigation system 130 may command the robot to explore the environment with a left movement of the robot, and the behavior 160 may command the robot to go backward following the appearance of a soccer ball. As an example instantiation, the scheduler can use a neural-like competitive cueing network ( or ANN, or DNN) to appropriately sequence actions based on their relative importance and timing."; Paragraph 139, "Each source coming into the scheduler 140 has more than one associated weight that gets combined into a final weight used by the scheduler 140. Each packet received by the scheduler 140 may have a specific weight for its individual command and a global weight provided by the scheduler 140 for that specific input. For example, if the scheduler 140 receives two motor commands from a controller-a first motor command with a global system weight of 0.2 and a specific weight of 0.4 and a second motor command with an global system weight of 0.1 and a specific weight of 0.9-it executes the second motor command as the combined weight of the second motor command is greater than that of the first motor command.")
performing, by the knowledge middle platform, processing comprising filtering on the data, and then providing the data block for display directly or through the microservice (Versace: Paragraph 142, "Beyond the basic series of weights, the scheduler 140 also executes one or more sorting steps. The first step involves sorting commands that use discrete hardware resources from commands that affect things like settings and parameter adjustment (operation 854). Settings changes are parsed and checked for conflict ( operation 856). If there are no conflicts, then all settings changes push ( opera ti on 8 5 8). If there are conflicts and there are weights that can be used to break the conflict, they are used. If everything is weighted identically and two settings conflict, than neither executes or a symmetry-breaking procedure may be applied (e.g., most-used behavior wins). Many of these settings packets can be executed simultaneously. Next, the packets that affect discrete system resources are further sorted based on the affected resource(s) (operation 860). Commands that can inherently affect each other but don't necessarily do so are kept together. For example, audio playback and audio recording may be kept in the same stream, as certain devices cannot record and playback and even if the option is available there are still constraints to deal with to avoid feedback."; Paragraph 145, "Different input sources can also communicate to each other and adjust the weights of the other subsystems. For instance, if the motivation system 120 is really interested in navigating, but it wants to navigate in a different direction, it can adjust the weights of the navigation packets being sent into the scheduler 140 by signaling the navigation system 130."; Paragraph 146, "The scheduling process of FIG. 15B allows the robot to look for an object in the environment, then step backward as required by the user-defined behavior, then go on exploring the environment. In order to determine the relative importance of actions, the scheduler 140 may use the graphical placement of behaviors in the brain to determine the relative importance of each behavior in the brain. In other implementations, a user may be able to provide positive and/or negative reinforcement (e.g., during a training process with the robot) in order to train the robot to develop an understanding of which behaviors and/or responses to prioritize over others. In another implementation, an ANN/DNN autonomously prioritizes scheduling based on learning and experience. In another implementation, the user may manually define the importance of each behavior, e.g., determining which behavior gets the precedence over other behaviors when both behaviors comprise stimuli which would activate their two different sets ofresponses in reaction a single event. For example, when an image contains two stimuli ( e.g., a face and the color red) which simultaneously activate two sets of responses, the user may manually predetermine when behavior I is engaged and when behavior 2 may be performed if behavior I is not complete (e.g., the user may indicate that behavior 2 may interrupt behavior 1, may start after behavior I completes, and/or the like).")
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Jain (US 2014/0173720 A1) disclose a method for controlling state of features at a runtime
Tu (US 2016/0092059 A1) discloses a web-based user interface dataflow editor
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Philip N Warner whose telephone number is (571)270-7407. The examiner can normally be reached Monday-Friday 7am-4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jerry O’Connor can be reached at 571-272-6787. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Philip N Warner/Examiner, Art Unit 3624
/Jerry O'Connor/Supervisory Patent Examiner,Group Art Unit 3624
Prosecution Insights