DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
This office action is in response to amendments filed 06/28/2025. Claims 1-2, 5-11, 14-15 are pending.
Applicant’s arguments and amendments to the claims with respect to prior art rejections of Claims 1-2, 5-11, 14-15 under 35 USC 102/103 have been fully considered and are persuasive. The rejections of Claims 1-2, 5-11, 14-15 under 35 USC 102/103 have been withdrawn. However, upon further consideration, a new rejection is made in view of Ramanujam et al (US 20200101611, hereinafter Ramanujam)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 5-11, 14-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Thackston et al (US 20200114515, hereinafter Thackston) in view of Ramanujam et al (US 20200101611, hereinafter Ramanujam).
Regarding Claim 1, Thackston teaches:
An information processing device comprising: (see at least " In an embodiment, the interface device 180 includes at least one processor 302 and memory (non-volatile memory 308 and/or volatile memory 310). The interface device 180 may include non-volatile memory 308 (ROM, flash memory, etc.), volatile memory 310 (RAM, etc.), or a combination thereof. In some embodiments, the at least one processor 302 is coupled to the non-volatile memory 308 and/or volatile memory 310. " in par. 0041)
at least one memory storing instructions, and at least one processor configured to execute the instructions to; (see at least " In an embodiment, the interface device 180 includes at least one processor 302 and memory (non-volatile memory 308 and/or volatile memory 310). The interface device 180 may include non-volatile memory 308 (ROM, flash memory, etc.), volatile memory 310 (RAM, etc.), or a combination thereof. In some embodiments, the at least one processor 302 is coupled to the non-volatile memory 308 and/or volatile memory 310. " in par. 0041)
receive an input for modifying the operation sequence for the robot via an input device; and (see at least “The one or more input devices 138 may be provided so that the user may interact with the robot 100, such as to navigate menus, make selections, set preferences, and other functionality described herein.” In par. 0025 and "The interface device 180 may further include one or more input devices 306, which can include, by way of example, any type of mouse, keyboard, disk/media drive, memory stick/thumb-drive, memory card, pen, joystick, gamepad, touch-input device, biometric scanner, voice/auditory input device, motion-detector, camera, scale, etc. In some embodiments, one or more input devices 306 may constitute a control device." in par. 0043)
acquire object information indicating information on an object or a virtual object in an operation space of the robot based on the input via the input device (see at least "One or more object attributes pertaining to occupancy, force, and/or deformation may be determined. An object attribute (such as occupancy) may be obtained/determined, for example, by scanning the object with one or more proximity sensors 154. An object attribute may also be obtained and/or updated from an external data source, such as a database... In another non-limiting example, one or more attributes may be input and/or modified by a user. For example, an attribute may be modifiable to permit a certain amount of force and/or deformation of the object, permit touching of the object, and/or all restrictions may be removed." in par. 0050).
acquire attribute information relating to an object or a virtual object in an operation space of the robot, based on the input via the input device. (see at least "One or more object attributes pertaining to occupancy, force, and/or deformation may be determined. An object attribute (such as occupancy) may be obtained/determined, for example, by scanning the object with one or more proximity sensors 154. An object attribute may also be obtained and/or updated from an external data source, such as a database... In another non-limiting example, one or more attributes may be input and/or modified by a user. For example, an attribute may be modifiable to permit a certain amount of force and/or deformation of the object, permit touching of the object, and/or all restrictions may be removed." in par. 0050).
combine the object information and the attribute information, and generate an attribute signal in which the object information and the attribute information are combined (see at least " An object attribute may also be obtained and/or updated from an external data source, such as a database. By way of non-limiting example, an object may have one or more attribute values associated with one or more attributes retrieved from/stored in a data source and/or updated from a data source, such as a database. Continuing with this example, an object may comprise a plurality of force attributes or a plurality of deformation attributes, wherein each attribute applies to a different portion of the object. In another non-limiting example, one or more attributes may be input and/or modified by a user. For example, an attribute may be modifiable to permit a certain amount of force and/or deformation of the object, permit touching of the object, and/or all restrictions may be removed." in par. 0050) ; and
transmit the attribute signal to a controller of the robot (see at least "For example, the user command may be modified before being output by the interface device 180 to the robot. In another example, the processor 130 of the robot, having access to object attributes, may modify user commands." in par. 0055) .
Thackston does not appear to explicitly teach all of the following, but Ramanujam does teach:
display an operation sequence of the robot (see at least "The web UI 110 also displays the simulated execution of the robotic solution based on the execution of the reusable robotic solution recipe by the simulator 114." in par. 0041) ;
and receive an input for modifying the displayed operation sequence for the robot via an input device (see at least " In one embodiment, the web UI 110 also allows a user to add, remove, or modify existing environment components in the simulated environment displayed at the web UI 110. After a change to the environment components, a user can simulate execution of the robotic solution to determine whether the robotic solution recipe can be executed when the environment components are changed." in par. 0042 and “The software framework 500 also allows a user to add 508, modify 510, or edit 512 hardware or environment components and check whether the robotic solution executes when any of the hardware components are modified. The software framework 500 also allows modifying robotic solution execution information to change the joint angle of the robot arm 502.” In par. 0068 ) ;
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device taught by Thackston to incorporate the teachings of Ramanujam wherein the operation sequence of the robot is simulated and displayed on a web UI for the user to modify. The motivation to incorporate the teachings of Ramanujam would be to enable a user planning a robot operation to instantly see how changes in the environment can affect the plan (see par. 0004), which improves the user experience.
Regarding Claim 2, Thackston as modified by Ramanujam (references to Thackston) teaches:
The information processing device according to claim 1 (see Claim 1 analysis),
wherein the at least one processor configured to execute the instructions to (see at least "At block 400, a teleoperative user's control of a robot may be established, such as through an input device and/or a control device (e.g. the interface device 180), along with an interface that may provide a virtual representation of an environment of a robot, such as a virtual representation displayed on a display of the one or more output devices of the interface device 180. A robot may move within the environment, move with respect to objects within the environment and/or move with respect to objects within the environment. Any suitable type of data and/or sensor(s) (such as the proximity sensor 154 discussed above) may be utilized to obtain and produce a representation of the environment of the robot, which may be referred to as a virtual representation." in par. 0047)
receive a signal for executing an operation of the robot being input in regard to display result; and (see at least "At block 400, a teleoperative user's control of a robot may be established, such as through an input device and/or a control device (e.g. the interface device 180), along with an interface that may provide a virtual representation of an environment of a robot, such as a virtual representation displayed on a display of the one or more output devices of the interface device 180. A robot may move within the environment, move with respect to objects within the environment and/or move with respect to objects within the environment. " in par. 0047)
transmit a control signal of the operation sequence to the robot, based on the input. (see at least "At block 406, a user command to move with respect to the object may be received. User input/commands may be include anything that the robot would be capable of performing, including by way of non-limiting example to move forward, to move in reverse, and/or to move laterally, to rotate, to increase/decrease the robot's height, and/or to utilize any hardware the robot would use for moving with respect to objects (such as with the interaction effectuator 157 and/or the motorized wheel assembly 158 depicted above with respect to FIG. 2). As discussed in more detail below with respect to FIGS. 6A-C, a robot may move with respect to an object (such as grasping a coffee cup on a table). " in par. 0054)
Thackston does not appear to explicitly teach all of the following, but Ramanujam does teach:
display an operation sequence of the robot (see at least "The web UI 110 also displays the simulated execution of the robotic solution based on the execution of the reusable robotic solution recipe by the simulator 114." in par. 0041) ;
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device taught by Thackston to incorporate the teachings of Ramanujam wherein the operation sequence of the robot is simulated and displayed on a web UI. The motivation to incorporate the teachings of Ramanujam would be to enable a user planning a robot operation to instantly see how changes in the environment can affect the plan (see par. 0004), which improves the user experience.
Regarding Claim 5, Thackston as modified by Ramanujam teaches:
The information processing device according to claim 2 (see Claim 2 analysis), wherein the at least one processor configured to execute the instructions to:
Thackston does not appear to explicitly teach all of the following, but Ramanujam does teach:
convert a control signal of an operation sequence of the robot to a display signal for displaying the operation sequence, and supply the display signal to a display unit. (see at least "The web UI 110 also displays the simulated execution of the robotic solution based on the execution of the reusable robotic solution recipe by the simulator 114." in par. 0041) ;
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device taught by Thackston to incorporate the teachings of Ramanujam wherein the operation sequence of the robot is simulated and displayed on a web UI. The motivation to incorporate the teachings of Ramanujam would be to enable a user planning a robot operation to instantly see how changes in the environment can affect the plan (see par. 0004), which improves the user experience.
Regarding Claim 6, Thackston as modified by Ramanujam teaches:
The information processing device according to claim 1 (see Claim 1 analysis), wherein the at least one processor configured to execute the instructions to;
Thackston does not appear to explicitly teach all of the following, but Ramanujam does teach:
receive the input, generate a signal for displaying in real time an operation on the object or the virtual object, or an attribute of the object or the virtual object, and supply the signal to display unit. (see at least " Next the simulated execution of the robotic solution is displayed at the web UI (314). In one embodiment, a user may add, remove, or modify environment component or hardware component at the web UI. For example, the user may add a wall (“environment component”) and modify (change rotation angle or increase/decrease size) the wall in the displayed simulated execution. The user may also remove the wall. Allowing the user to change hardware component and environment component to easily check whether the robotic solution executes when the hardware component or environment component is changed." in par. 0059)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device taught by Thackston to incorporate the teachings of Ramanujam wherein the operation sequence of the robot is simulated and displayed on a web UI. The motivation to incorporate the teachings of Ramanujam would be to enable a user planning a robot operation to instantly see how changes in the environment can affect the plan (see par. 0004), which improves the user experience.
Regarding Claim 7, Thackston as modified by Ramanujam teaches:
The information processing device according to claim 6 (see Claim 6 analysis),
Thackston further teaches: wherein, in a case where receiving the input relating to selection of the object or the virtual object is received, the display unit selectively displays a plurality of attributes of the object or the virtual object (see at least " In another non-limiting example, one or more attributes may be input and/or modified by a user. For example, an attribute may be modifiable to permit a certain amount of force and/or deformation of the object, permit touching of the object, and/or all restrictions may be removed." in par. 0050).
Regarding Claim 8, Thackston as modified by Ramanujam (references to Thackston) teaches:
The information processing device according to claim 1 (see Claim 1 analysis), wherein the attribute information is based on a relationship between the robot and the object or the virtual object (see at least " In another non-limiting example, one or more attributes may be input and/or modified by a user. For example, an attribute may be modifiable to permit a certain amount of force and/or deformation of the object, permit touching of the object, and/or all restrictions may be removed." in par. 0050 and “For example, the space occupied by an object in the environment of a robot may be represented by an attribute relating to the occupancy associated with the virtual representation of the object within the virtual environment. This may mean, for example, that the robot is not permitted to come into contact with the virtual representation of the object (i.e., the robot cannot touch the virtual representation of the object in the virtual environment).” In par. 0051, “An attribute related to the object may relate to an amount of force exerted upon the object, such as by a robot.” In par. 0052 and “An attribute may also relate to a permissible amount of deformation of the object. For example, an object such as a bean bag may be flexible, such that the bean bag may become deformed based upon a certain amount of force.” In par. 0053) .
Regarding Claim 9, Thackston as modified by Ramanujam (references to Thackston) teaches:
The information processing device according to claim 8 (see Claim 8 analysis), wherein the attribute information includes information indicating that the robot is able to pass through the virtual object, and information indicating that the robot is unable to pass through the virtual object (see at least " In another non-limiting example, one or more attributes may be input and/or modified by a user. For example, an attribute may be modifiable to permit a certain amount of force and/or deformation of the object, permit touching of the object, and/or all restrictions may be removed." in par. 0050 and “For example, the space occupied by an object in the environment of a robot may be represented by an attribute relating to the occupancy associated with the virtual representation of the object within the virtual environment. This may mean, for example, that the robot is not permitted to come into contact with the virtual representation of the object (i.e., the robot cannot touch the virtual representation of the object in the virtual environment).” In par. 0051) .
Regarding Claim 10, Thackston as modified by Ramanujam (references to Thackston) teaches:
The information processing device according to claim 8 (see Claim 8 analysis), wherein the attribute information includes information indicating that the robot is able to execute a task relating to the object, and information indicating that the robot is unable to execute a task relating to the object (see at least "in some embodiments, where the virtual representation of the object may be a point cloud representation, the robot may not be permitted to come into contact with a boundary of the point cloud representing the object. This may in turn prohibit the robot from also coming into contact with any spaces between points within the virtual representation of the object. In some embodiments, occupancy may be a default attribute, such that a robot will be prohibited from contacting/touching object unless one or more object attributes are modified/updated." in par. 0051 and “Continuing with this embodiment, the object 702 has a deformation attribute, such that although the amount of force provided by the user command has not increased, the amount of deformation of the object 702 has approached or exceeded a deformation threshold of the object 702, and therefore the user command is overridden with a modified user command 706 that stops the robot 100 from squeezing the object 702. In another embodiment, a force attribute may be utilized to impose a force threshold such that the robot would not be able to increase its squeezing force to be able to further deform the object 702, depending upon one or more constituent materials of the object.” In par. 0064) .
Regarding Claim 11, Thackston as modified by Ramanujam (references to Thackston) teaches:
The information processing device according to claim 8 (see Claim 8 analysis), wherein the attribute information includes information indicating that the object is able to be moved by the robot, and information indicating that the object is unable to be moved by the robot (see at least ""n some embodiments, where the virtual representation of the object may be a point cloud representation, the robot may not be permitted to come into contact with a boundary of the point cloud representing the object. This may in turn prohibit the robot from also coming into contact with any spaces between points within the virtual representation of the object. In some embodiments, occupancy may be a default attribute, such that a robot will be prohibited from contacting/touching object unless one or more object attributes are modified/updated." in par. 0051) .
Regarding Claim 14, Thackston as modified by Ramanujam (references to Thackston) also teaches:
An information processing method comprising: (see at least "method" in par. 0004)
Implementing, step by step, each function of the device of Claim 1 (see Claim 1 analysis for rejection of the device)
Regarding Claim 15, Thackston as modified by Ramanujam (references to Thackston) also teaches:
A non-transitory computer-readable medium storing a program that causes a computer to execute: (see at least " the memory module 132 of the robot 100 is coupled to the communication path 128 and communicatively coupled to the processor 130. The memory module 132 may comprise RAM, ROM, flash memories, hard drives, or any non-transitory memory device capable of storing machine-readable instructions such that the machine-readable instructions can be accessed and executed by the processor 130." in par. 0023)
Implementing, step by step, each function of the device of Claim 1 (see Claim 1 analysis for rejection of the device)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN M KATZ whose telephone number is (571)272-2776. The examiner can normally be reached Mon-Thurs. 8:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on (571) 270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN M KATZ/Examiner, Art Unit 3657