DETAILED ACTION
Remarks
This Final office action is in response to the RCE amendments filled on 05/01/2026. Claims 1, 3, 6, 14 and 15 are currently amended. Claims 2, 4, 7-10, 12 and 13 are canceled. Claims 1, 3, 5, 6, 11 and 14-16 are pending and examined below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 3, 5 and 16 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by US 2019/0291277 (“Oleynik”).
Regarding claim 1, Oleynik discloses a method for guiding a robot arm (see at least [0496], where “The RGB-D sensor 500 provides guidance to the robotic hand 72 in moving the robotic hand 72 toward the direction of the object and to make necessary adjustments to grab an object.”; see also [0515]), comprising:
providing a visual cue associated with and in close physical proximity to an object with a camera mechanically coupled to the robot arm (see at least [0952], where “the general-purpose vision subsystem 5002r-5 of the robotic assistant 5002r, and the cloud computing system 5006. The general-purpose vision subsystem 5002r-5 includes and/or is associated with cameras 5002r-4, CPU 5002r-2a, and GPU 5002r-2b. As discussed herein, the cameras can include cameras that exclusively correspond to the robotic assistant 5002r (e.g., cameras embedded in the robot anatomy 5002r-1).”; visual cue associated with and in close physical proximity to an object is interpreted as surrounding environment of the object. see also [0011], fig 7B and [0967-966]);
training by guiding the robot arm from a first position to a target position near the object based on external input (training the robot arm by manipulating it until it has a successful interaction. see at least [0999], where “Training the robot to achieve successful cases (e.g., executions) of the desired interaction can include first imaging the object to be interacted with by one or more of the cameras 5002r-4 of the robotic assistant. In some embodiments, this includes moving the object or moving the object to a desired relative position of the cameras 5002r-4 and parts (e.g., end effectors) of the robotic assistant 5002r, where the object is imaged…the robotic assistant 5002r iteratively attempts to perform an interaction (or portion thereof) until the gauging point is consistently reached as measured by the cameras 5002r-4 and/or other sensors, thereby indicating that the robotic assistant has repeatedly successfully performed that aspect of the interaction.”) comprising at least one of (i) physical manipulation of the robot arm by a human while recording a trajectory of an end effector of the robot arm to track a pose of the end effector relative to a starting pose (see at least [0042], [0992] and [0999]) or (ii) control of the robot arm while recording joint angles of the robot arm, a trajectory from the end effector, and video input from the camera;
recording, by the camera, during multiple episodes of the training step, trajectory data during movement of the robot arm from the first position to the target position and storing the trajectory data and video input in memory (see at least [0707], where “Once the learning process has been completed, results are stored as a collection of action primitives that together are known to accomplish the desired functional result.”; see also [0996], where “When each command of the interaction is successfully performed by the robotic assistant, the underlying instructions and/or parameters such as the space coordinates (e.g., X, Y, Z, axis) of each part of the robotic assistant at each time period can be stored and/or used to program the robotic assistant 5002r accordingly.”);
receiving an instruction to move the robot arm to the object (see at least [0933], where “the robotic assistant system 5002r can be deployed within the robotic assisted environment 5002 to perform a recipe, which is a series of interactions configured to achieve a desired object…It should be understood that interactions refer to actions or manipulations performed by the robotic assistant 5002r to or with, among other things, the objects in the robotic assisted environment 5002 and/or workspace 5002w. FIG. 165 is a flow chart 6000 illustrating a process for executing an interaction using the robotic assistant 5002r”);
providing, in response to the instruction to move the robot arm, to a controller a sequence of robot arm trajectory instructions relative to the position and orientation of the visual cue based on input from the camera and trajectory data (see at least [0677], where “At a high level, this is achieved by downloading the task-descriptive libraries containing the complete set of minimanipulation datasets required by the robotic system, and providing them to a robot controller for execution…Updated parameter data is then used to rebuild the modified minimanipulation parameter set for re-execution as well as for updating/rebuilding a particular minimanipulation routine, which is provided back to the original library routines as a modified/re-tuned library for future use by other robotic systems. The system monitors all minimanipulation steps until the final result is achieved and once completed, exits the robotic execution loop to await further commands or human input.”; see also [1029], where “The one or more positioning techniques comprises at least one of object template matching technique and marker-based technique…Positioning (e.g., to position 0) can be performed using a 3D object template of the object to be interacted with…the object template is used to compare the images of the object obtained from the cameras (e.g., cameras embedded in or of the end effector), as positioned and oriented at step 8052, to the data definition of the object stored in the 3D object template…the one or more processors may position the one or more manipulation devices to a first position proximal to the target object. Thereafter, the one or more processors may receive the one or more images, in real-time, of the target object from at least one of image capturing devices associated with the one or more manipulation devices…the one or more processors may compare the object template of the target object with the one or more images of the target object. Further to comparison, the one or more processors may perform at least one of: adjusting position of the one or more manipulation devices towards the optimal standard position based on position of the one or more manipulation devices in previous iteration and reiterating steps of receiving and comparing, when the comparison results in mismatch; or inferring that the one or more manipulation devices reached the optimal standard position when the comparison results in a match.”) in either (a) a continuous mode with real-time adjustment based on ongoing video input from the camera (see at least [1042], where “the one or more processors may shift the one or more manipulation devices along an X-axis and/or Y-axis of the real-time image of the target object until a center of the triangle-shaped marker is positioned at the center of the real-time image of the target object captured by the camera of the one or more manipulation devices.”; see also [1079], where “The one or more processors may identify image co-ordinates of corners of square slots in the chessboard marker in real-time images of the target object, i.e. the internal corners of the chessboard marker are located from the captured imaged, which is analyzed to identify points of interest therein such as the white points shown in FIG. 185A.”; see also [1126], where “Images of failed and successful manipulations can be continuously obtained and transmitted to a central system (e.g., the central use cases laboratory) for storage, analysis and training.”) or (b) an instance mode with initial observation of the visual cue followed by predicted actions without requiring continuous visibility of the object to automatically guide the robot arm to the target position.
Regarding claim 3, Oleynik further discloses a method wherein the sequence of robot arm trajectory instructions are calculated based on a current visual cue and a current position of the robot arm, and wherein the recorded visual que and the recorded trajectory data use computer vision that outputs the trajectory instructions (see at least [0534], where “At step 956, the computer 16 monitors the food preparation process via a multimodal sensor that generates raw data supplied to abstraction software where the robotic apparatus 75 compares real-world output against controlled data based on multimodal sensory data (visual, audio, and any other sensory feedback). At step 958, the computer 16 determines if there any differences between the controlled data and the multimodal sensory data. At step 960, the computer 16 analyzes whether the multimodal sensory data deviates from the controlled data. If there is a deviation, at step 962, the computer 16 makes an adjustment to re-calibrate the robotic arm 70, the robotic hand 72, or other elements.”).
Regarding claim 5, Oleynik further discloses a method wherein the robot arm comprises an end effector capable of interacting with the object when the robot arm is in the target position (see at least [1002], where “In turn, at step 8052, the embedded processor of the end effector 5002r-1c of the robotic assistant 5002r uses this information to initiate its interaction responsibilities, by positioning the end effector 5002r-1c and/or object to be interacted with at a preliminary position relative to one another.”).
Regarding claim 16, Oleynik further discloses a method comprising: during execution of the sequence of robot arm trajectory instructions, detecting, from the video input and the visual cue, a new obstacle has entered a planned path of the robot arm; and recalculating a collision-free trajectory in real time based on current observations of the visual cue without relying on predictions of consequences from prior actions (see at least [0812], [0814], [0833] and [0856]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2019/0291277 (“Oleynik”), as applied to claim 5 above, and further in view of US 2019/0255703 (“Izhikevich”).
Regarding claim 6, Oleynik further discloses a method wherein the end effector is a gripper (see at least [0911], where “the end effectors can refer to a grouping of parts (e.g., robotic shoulder, arm, wrist, hand, palm, fingers, grippers, and/or objects connected thereto) or to a single part (e.g., gripper) of the robotic assistant 5002r that are disposed at a distalmost position relative to the torso 5002r-1b of the robotic assistant 5002r.”), and the object is a utensils or cooking pots (see at least [0441], where “The robotic arms 70 and the robotic hands 72 operate the standardized handles and utensils 80 in the cooking process. In one embodiment, one of the robotic hands 72 is fitted with a standardized handle, which is attached to a fork head, a knife head, and a spoon head for selection as required.”).
Oleynik does not disclose the following limitation:
the object is a movable portion of a coffee making apparatus and the method further comprises detecting movement of the movable portion of the coffee making apparatus caused by the robot arm, another robot, or a human operator using the visual cue of the coffee making apparatus and adapting the sequence of robot arm trajectory instructions based on the detected movement and current observations.
However, Izhikevich discloses a method wherein the object is a movable portion of a coffee making apparatus and the method further comprises detecting movement of the movable portion of the coffee making apparatus caused by the robot arm, another robot, or a human operator using the visual cue of the coffee making apparatus and adapting the sequence of robot arm trajectory instructions based on the detected movement and current observations (see at least [0245], where “Such methodology may be of use in predicting a sequence of actions/behaviors at time scales where the previous actions and/or behavior may affect the next actions/behavior, e.g., when a robotic manipulator arm may be tasked to fill with coffee three cups one a tray: completion of the first action (filling one cup) may leave the plant (arm) closes to another cup rather than the starting location (e.g., coffee machine). The use of feedback may reduce coffee serving time by enabling the arm to move from one cup to another without returning to the coffee machine in between. In some implementations (not shown), action feedback (e.g., 2016 in FIG. 20A) may be provided to other predictors configured to predict control input for other tasks (e.g., filling the coffee pot with water, and/or placing the pot into the coffee maker).”; see also [0006] and [0022]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Oleynik to incorporate the teachings of Izhikevich by including the above feature for providing automated coffee making by a robot.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2019/0291277 (“Oleynik”), as applied to claim 5 above, and further in view of US 2020/0170723 (“Crawford”).
Regarding claim 11, Oleynik does not disclose claim 11. However, Crawford discloses a method wherein end-effector pose may be computed through forward kinematics when the computer vision outputs joint angles, or joint angles are computed through inverse kinematics when the computer vision outputs an end-effector pose (see at least [0141], where “Loop 1102 also includes step 1118 determining current location of the single robot marker 1018 in the robot coordinate system from forward kinematics.”; see also [0198]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Oleynik to incorporate the teachings of Crawford by including the above feature for providing trajectory by simple mathematical calculation.
Claim(s) 14 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2019/0291277 (“Oleynik”), as applied to claim 1 above, and further in view of US 2020/0205603 (“Chen”), and further in view of US 6,494,238 (“Sindermann”).
Regarding claim 14, Oleynik further discloses a method wherein the object is a beverage cup and the target position comprises pouring liquid into the beverage cup from a pitcher held by the end effector of the robot arm (see at least [0036], where “robot-arms and wrists, based on fairly abstraction/high-level commands (e.g. “grab the pot by the handle”, “pour out the contents”, “grab the spoon off the countertop and stir the soup”, etc.”).
Oleynik does not disclose the following limitation:
during execution of the sequence of robot arm trajectory instructions, using the camera to determine a current liquid fill level in the beverage cup from the video input; and if the current liquid fill level exceeds a predetermined level, modifying the sequence of trajectory instructions in real time to tilt the pitcher away from the beverage cup to prevent overflow of the liquid from the beverage cup.
However, Chen discloses a method comprising: during execution of the sequence of robot arm trajectory instructions, (see at least [0010], where “the data comprises a moving route and a moving speed of the water injection unit moved by the robotic manipulator; an amount of water poured from the water injection unit”; see also [0026], where “The pump 62 may adjust and deliver a predetermined amount of the water from the tank 61 to the water injection unit 3 where the water would be poured out according to the at least one controlling signal.”; see also [0027] and [0033]).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Oleynik to incorporate the teachings of Chen by including the above feature for providing automated beverage services.
Oleynik in view of Chen does not disclose the following limitation:
using the camera to determine a current liquid fill level in the beverage cup from the video input; and if the current liquid fill level exceeds a predetermined level, modifying the sequence of trajectory instructions in real time to tilt the pitcher away from the beverage cup to prevent overflow of the liquid from the beverage cup.
However, Sindermann discloses a method comprising: using the camera to determine a current liquid fill level in the beverage cup from the video input; and if the current liquid fill level exceeds a predetermined level, modifying the sequence of trajectory instructions in real time to tilt the pitcher away from the beverage cup to prevent overflow of the liquid from the beverage cup (see at least claim 1, where “minimize overflow of fluid from the open filled beverage containers; apparatus to terminate the flow of fluid medium into the open filled beverage containers; and apparatus to seal open filled beverage containers filled with liquid beverage filling material”; see also col 3, lines 64-66, where “the measuring/sensing stations 34 and 35 operate with high frequency determination/measuring, or they are configured as light gates, or light barriers, or camera systems.”; see also col 6, lines 1-15).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified Oleynik in view of Chen to incorporate the teachings of Sindermann by including the above feature for providing clean and spill free automated beverage services.
Regarding claim 15, Sindermann further discloses a method wherein the current liquid fill level is determined directly from the video input and the observed position and orientation of the visual cue (see at least col 6, lines 7-12, where “To capture such a spot, the measuring/sensing station 35 is preferably embodied by a camera (video camera), whereby the comparison of the actual value with the preset value, or duty value, for example, is done through comparison of the image delivered by this camera with a preset image.”).
Response to Arguments
Applicant’s arguments with respect to claim 1, 3, 5, 6, 11 and 14-16 have been considered but are moot because the arguments do not apply to the new combination used in the current rejection that is due to the newly added claim amendments.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOHANA TANJU KHAYER whose telephone number is (408)918-7597. The examiner can normally be reached on Monday - Thursday, 7 am-5.30 pm, PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached on 571-270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SOHANA TANJU KHAYER/Primary Examiner, Art Unit 3657
Prosecution Insights