DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
(Submitted 1/23/2026)
In regard to 103 rejections
- The applicant argues from Pages 10-15 that the references do not teach the amended claims 1, 26 and 28. The applicant added a new claim 29.
Examiner’s Response:
The examiner submits that the applicant arguments is MOOT as a result of the amendment and new
ground of rejection that does not rely on any reference applied in the prior rejection of record for any
teaching or matter specifically challenged in the argument. Two new references “ Iqbal” and
“Homberg” teaches the claim amendments. However, the examiner does not concede the
arguments of the applicant. A new reference “OLEY” and “Yugachi” teaches the new claim 29. The
examiner submits that in according to the text specification, the evaluation of interference does not
automatically represent the entirety of the 6D target space unless the method explicitly accounts for
and samples the full phase space. However, the examiner notes to the applicant new claim 29 based on
the Fig 20 of the specification. The examiner submits that the primary reference “Iqbal” is a very strong
reference that teaches the independent claims 1, 26 and 28 in view of “Homberg”.
In CONCLUSION, the examiner rejects claims 1, 17-29 under 103 as a NON FINAL REJECTION under RCE.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, and 17-28 are rejected under 35 U.S.C. 103 unpatentable over
Shariq Iqbal et.al. (hereinafter Iqbal) US 2020/0061811 A1,
in view of Bianca Homberg et.al. (hereinafter Homberg) US 2019/0337152 A1.
In regard to claim 1: (Currently Amended):
Iqbal discloses:
A machine learning data generation device, comprising a central processing unit and a memory which are configured to:
[0078]; [0082];[0075]
- arrange a plurality of virtual subject models in a randomly piled state in a virtual space, the virtual subject models are virtual models of a plurality of subjects of picking up operation of a robot
[0183]:
In at least one embodiment, 3D flash LIDAR systems include, without limitation, a solid-state 3D staring array LIDAR camera with no moving parts other than a fan (e.g., a non-scanning LIDAR device). In at least one embodiment, flash LIDAR device may use a 5 nanosecond class I (eye-safe) laser pulse per frame and may capture reflected laser light in form of 3D range point clouds and co-registered intensity data. [BRI: A 3D range point cloud is indeed a collection of 3D points in space, each with coordinates (X, Y, Z) and often additional attributes and is an unordered, sparse representation of the 3D world, meaning the points are not arranged in a fixed grid or mesh but are scattered in 3D . Perhaps known to the POSITA that in virtual environments, arranging a plurality of virtual subject models in a randomly piled state means placing multiple 3D digital objects (such as workpieces, parts, or other virtual items) in a virtual space so that their positions and orientations are not fixed or ordered, but instead appear irregularly stacked or piled — similar to how real-world objects might be dumped or randomly arranged.]
generate a virtual image, based on the plurality of the randomly piled virtual subject models, the virtual image is obtained by virtually generating an image, image is taken by a camera on the plurality of subjects;
[0076]:
In at least one embodiment, the process begins at block 702 with the computer system generating a plurality of simulations. In at least one embodiment, the simulations correspond to a physical system to be controlled. In at least one embodiment, for example, a manipulator in the simulation matches a manipulator on a corresponding robot to be controlled. In at least one embodiment, the simulations are simulations of an articulated robot and an object to be grasped by the robot. In at least one embodiment, each simulation includes various textures, objects, and patterns that may block, obscure, or clutter the image collected by a virtual camera located on the wrist of the articulated robot
[0076]:
In at least one embodiment, the images are generated from the point of view of the simulated virtual camera mounted on the rest of the articulated robot and oriented in the direction of the robotic gripper used to grasp the object. In at least one embodiment, the images are produced with a camera model that matches a physical camera on the robot to be controlled. [BRI: an image collected by a virtual camera on the wrist, produced with a camera model matching the physical camera, is a virtual image in the sense that it is a synthetic projection from a 3D model using the same camera parameters.
[0285]:
tile-based rendering, in which rendering operations for a scene are subdivided in image space, for example to exploit local spatial coherence within a scene
rendering operations for a scene are subdivided in image space, or example to exploit local spatial coherence within a scene
[0076]:
In at least one embodiment, at block 704, the simulation generates images for each simulation that are used to train the machine-learning system. In at least one embodiment, the images are generated from the point of view of the simulated virtual camera mounted on the rest of the articulated robot and oriented in the direction of the robotic gripper used to grasp the object. In at least one embodiment, the images are produced with a camera model that matches a physical camera on the robot to be controlled. [BRI: virtual camera with a camera model matching a physical camera, and you render a scene subdivided in image space, you can indeed generate a virtual image from a plurality of randomly piled virtual subject models. The camera model ensures realism, and the image-space subdivision enables efficient rendering of complex, multi-object scenes
- generate a range of target values in 6-dimensional space of the virtual operation command
[0059]:
In at least one embodiment, the controller computer system obtains an image from a camera with a view of the work area, and determines a 6-d pose of the object 110 from the image. In at least one embodiment, the 6-d pose is a xyz position and a rotational orientation of the object 110
[0183]:
In at least one embodiment, flash LIDAR device may use a 5 nanosecond class I (eye-safe) laser pulse per frame and may capture reflected laser light in form of 3D range point clouds and co-registered intensity data
[0124]:
generate a 3D map of environment of vehicle 1200, including a distance estimate for all points in image [BRI: a 3D map with distance estimates for all points in the image provides the spatial and metric data define the range of target values in the 6D space of the robot’s operation]
- by virtually generating a picking up operation command for the robot based on a range of target values in the 6-dimensional space of the operation of the robot given on the subject and one or more of the plurality of the randomly piled virtual subject models
[0082]:
in at least one embodiment, the system above can be adapted to park an autonomous vehicle, accurately position an electric vehicle for charging, pick up a pallet with a forklift. In at least one embodiment, techniques described above can be adapted to operate a pick and place robot used for electronics assembly
[0080]:
the machine-learning computer system determines a particular action that will improve the position of the robotic gripper with respect to producing a successful grasp.
[0074]:
In at least one embodiment, after training in simulation, a policy produced by the training is able to grasp the intended object in the real world without fine-tuning. In at least one embodiment, the action space is gripper-centric, allowing the robot to perform not only top-down grasps (as illustrated in FIG. 5) but also of grasps from other orientations (as illustrated in FIG. 6).
- evaluate an outcome of the picking up operation of the robot in response to the virtual operation command in the virtual space by determining whether an end effector of the robot interferes or not with the plurality of the randomly piled virtual subject models in the virtual space
[0070]:
FIG. 3 illustrates an example of a machine-learning system that can direct a robot to perform a grasp of an object, according to at least one embodiment.
[0080]:
the machine-learning computer system determines a particular action that will improve the position of the robotic gripper with respect to producing a successful grasp.
[0082]:
in at least one embodiment, the system above can be adapted to park an autonomous vehicle, accurately position an electric vehicle for charging, pick up a pallet with a forklift. In at least one embodiment, techniques described above can be adapted to operate a pick and place robot used for electronics assembly
[0122]:
In at least one embodiment, cameras with a field of view that include portions of environment in front of vehicle 1200 (e.g., front-facing cameras) may be used for surround view, to help identify forward facing paths and obstacles,
[0178]:
create a focused beam pattern, designed to record vehicle's 1200 surroundings at higher speeds with minimal interference from traffic in adjacent lanes.
- wherein the end effector is in a position and an angle designated by a target value in the 6-dimensional space within the range of target values in the 6-dimensional space of the virtual operation command;
[0055]:
In at least one embodiment, techniques described herein utilize a deep reinforcement learning approach to grasp semantically meaningful objects in a geometrically consistent way. In at least one embodiment, the system is trained in simulation, with sim-to-real transfer accomplished by using a simulator that both models physical contact between the robot and the object to be grasped, and produces photorealistic imagery. In at least one embodiment, the system provides an example of end-to-end semantic grasping (mapping input pixels to output motor commands in Cartesian or other coordinate systems). In at least one embodiment, by using a camera positioned on the manipulator, the system is not limited to top-down grasps and is capable of grasping objects from any angle. In at least one embodiment, the system is able to grasp objects from multiple pre-defined object-centric orientations, such as from the side or top. When coupled with a real-time 6-DoF object pose estimator, at least one embodiment is capable of grasping objects from any position and orientation within the graspable workspace. In at least one embodiment, results in both simulation and the real-world demonstrate the effectiveness of the approach. [BRI: an end-to-end semantic grasping is a robotic learning approach that directly maps raw sensor data (e.g., monocular images) into a complete grasp plan for a user-specified object class and an end-to-end semantic grasping system represents and enables the pick-up operation]
[0064]:
In at least one embodiment, a semantic grasp technique allows a robot to learn to grasp a specific type of object—indicated by the user—from a cluttered bin. Although such approaches address the question of which object to grasp
[0094]:
converging towards a model, such as trained neural network 1008, suitable to generating correct answers, such as in result 1014, based on known input data, such as new data 1012
[BRI: the reference does teach question and answer]
Iqbal does not explicitly disclose:
and generate machine learning data by pairing the virtual image as question data with answer data [[,]],
and generate machine learning data by pairing the virtual image as question data with answer data, the answer data is obtained by incorporating the outcome of the picking up operation of the robot
However, Homberg discloses:
and generate machine learning data by pairing the virtual image as question data with answer data [[,]],
[0096]:
Referring back to FIG. 5, at block 510, method 500 further includes determining, by inputting the second sensor data into the classifier, a property of the object. The classifier is an algorithm or function that maps inputted sensor data to one or more object classes. The classes may be associated with semantic labels. For instance, a bowl may be further classified as a plastic bowl or a glass bowl, as a full bowl or an empty bowl, or as a bowl intended for solid food or a bowl intended for drinking. In further examples, the object classes output by a classifier may correspond to different types of intended robot behavior. For instance, the output of the classifier may answer questions such as: can a bowl be picked up by a robot?, can a bowl be washed by a robot?, or is a bowl in a state such that it is ready to be cleared by a robot? The output of the classifier may be used to control the robot to interact with the object. [BRI: perhaps known to a POSITA that a “semantic label” represents a tag assigned to a word or phrase in a sentence to indicate its role or meaning within the sentence. Answering a question is question and answer pairing]
[BRI: 6D pose can provide probability tables (including multi-dimensional ones) to represent a range of possible target values in the 6D space, with each “action phase” corresponding to a conditional or joint distribution over pose. This approach is standard in uncertainty-aware robotics and can be adapted to virtual operation command systems for richer, more robust decision-making]
[0093]:
In some examples, the robot 602 may process sensor data from one or more sensors on robot head 604 in order to determine a bounding box around bowl 622. In particular, the bounding box may be a virtual representation of a volume that fully surrounds the bowl 622. The bounding box may then be used by robot 602 in order to determine how to position gripper 606 above bowl 622 to collect additional sensor data. In particular, gripper 606 may be positioned so that a sensor on gripper 606 is horizontally aligned with the bounding box and above the top of the bounding box by predetermined height 624. In other examples, gripper 606 may be positioned so that some other point on gripper 606 other than the sensor is above bowl 622 by the predetermined height 624.
- wherein the answer data is obtained by incorporating the outcome of the picking up operation of the robot into the range of target values of the virtual operation command.
[0093]:
In some examples, the robot 602 may process sensor data from one or more sensors on robot head 604 in order to determine a bounding box around bowl 622. In particular, the bounding box may be a virtual representation of a volume that fully surrounds the bowl 622.
[0065]:
Short-range time-of-flight sensor 310 may include a narrow light source 312 and a light detector 314 to measure how long it takes laser light projected by light source 312 to bounce back after hitting an object. This time may be used to accurately determine a range or distance to a nearest object from short-range time-of-flight sensor 310 based on the known speed of light. As an example, short-range time-of-flight sensor 310 may have a range of about 1 centimeter up to 20 centimeters from the palm of the gripper. Additionally, short-range time-of-flight sensor 310 may have a relatively narrow field of view (e.g., 40 degrees) in order to detect objects within a cone of sensing range extending out from light detector 314. Based on its range, short-range time-of-flight sensor 310 may be most effective for determining information about grasped objects. [BRI: a gripper represents the end-effector. A short-range ToF sensor that cover a 1 cm to 20 cm target range, making it suitable for virtual operation command indeed represents the range]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Iqbal and Homberg.
Iqbal teaches machine learning data generation and virtual models, a 6-dimensional space for target value for the robot, generating virtual image, based on the plurality of the randomly piled virtual subject models.
Homberg teaches question and answer pairing and label attached to the answer data.
One of ordinary skill would have motivation to combine Iqbal and Homberg that can use accurate sensor data (Homberg [0073])
In regard to claim 17: (Previously Presented)
Iqbal discloses:
wherein the subjects are mechanical parts with a protruding portion.
[0061:
FIG. 2 illustrates an example of a robotic gripper 202, according to at least one embodiment. In at least one embodiment, the robotic gripper 202 is attached to a gripper 204 via a wrist joint 206. In at least one embodiment, the gripper 204 is a claw, articulated hand, vise, or other manipulator for grasping or interacting with an object 208. [BRI: a gripper — whether claw, articulated hand, vise, or other grasping manipulator is considered a protruding portion of the robot. This is because it extends from the main body or arm of the robot and is specifically designed to interact with or manipulate objects]
In regard to claim 18: (Currently Amended)
Iqbal does not explicitly disclose:
- comprises a position range in which [[an]] the end effector of the robot successfully picks up the subject.
However, Homberg discloses:
- comprises a position range in which [[an]] the end effector of the robot successfully picks up the subject.
[0030]:
By provisioning a robotic gripping device with one or more non-contact sensors, the robot may be able to collect more accurate sensor data about an object such as a bowl or plate from a close-up point of view. In particular, the robot may be configured to position the gripper so that a sensor (e.g., an infrared microcamera) on the gripper is positioned at a predetermined pose relative to the object in order to collect sensor data
[0065]
determine a range or distance to a nearest object from short-range time-of-flight sensor 310 based on the known speed of light. As an example, short-range time-of-flight sensor 310 may have a range of about 1 centimeter up to 20 centimeters from the palm of the gripper.
[0026]:
Robotic end effectors may be used in many situations to allow a robotic device to interact with an environment by pushing, pulling, grasping, holding, or otherwise interacting with one or more objects in the environment. For instance, a robotic device may include a robotic gripper having one or more digits that can be actuated to change their shape, thereby allowing the robotic gripper to interact with the environment.
[0029]:
Some robot tasks directed toward manipulating an object in an environment may involve picking up and moving the object, cleaning the object, refilling the object, or otherwise changing the state of the object.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Iqbal and Homberg.
Iqbal teaches machine learning data generation and virtual models, a 6-dimensional space for target value for the robot, generating virtual image, based on the plurality of the randomly piled virtual subject models.
Homberg teaches question and answer pairing and label attached to the answer data.
One of ordinary skill would have motivation to combine Iqbal and Homberg that can use accurate sensor data (Homberg [0073])
In regard to claim 19: (Currently Amended)
Iqbal does not explicitly disclose:
wherein the range of target values of the operation comprises an angle range
in which an end effector of the robot successfully picks up the subject.
However, Homberg discloses:
wherein the range of target values of the operation comprises an angle range
in which an end effector of the robot successfully picks up the subject.
[0065]
determine a range or distance to a nearest object from short-range time-of-flight sensor 310 based on the known speed of light. As an example, short-range time-of-flight sensor 310 may have a range of about 1 centimeter up to 20 centimeters from the palm of the gripper. Additionally, short-range time-of-flight sensor 310 may have a relatively narrow field of view (e.g., 40 degrees) in order to detect objects within a cone of sensing range extending out from light detector 314. Based on its range, short-range time-of-flight sensor 310 may be most effective for determining information about grasped objects.
In [0014]:
For example, in some kinesthetic PbD implementations, a user may manually move a robot to execute a task (e.g., rotate an arm of the robot, open an end-effector to receive an object, close an end-effector to grasp the object, move to a target location, release the object etc.), and the robot may learn from these movements to execute the task independently after the demonstration. [ BRI: the above does represents a “pick up” operation]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Iqbal and Homberg.
Iqbal teaches machine learning data generation and virtual models, a 6-dimensional space for target value for the robot, generating virtual image, based on the plurality of the randomly piled virtual subject models.
Homberg teaches question and answer pairing and label attached to the answer data.
One of ordinary skill would have motivation to combine Iqbal and Homberg that can use accurate sensor data (Homberg [0073])
In regard to claim 20: (Previously Presented)
Iqbal discloses:
wherein the range of target values in the 6-dimensional space of the operation is specified by a user.
[0066]:
In at least one embodiment, various techniques described herein provide an approach to geometry-aware semantic grasping that learns to grasp specific objects along specific grasp directions, a method to generate physically plausible, photorealistic synthetic data to train policies that transfer to the real world without a special domain adaptation step, and a system that combines the learned camera-in-hand policies for local control with global 6-DoF pose estimation from a fixed camera in order to grasp objects anywhere in the graspable workspace.
[0109]:
In at least one embodiment, vehicle 1200 may be an airplane, robotic vehicle, or other kind of vehicle.
[0176]:
In at least one embodiment, vehicle 1200 may further include GNSS sensor(s) 1258 (e.g., GPS and/or assisted GPS sensors), to assist in mapping, perception, occupancy grid generation, and/or path planning functions.
[BRI: a geometry-aware semantic grasping often represents grasp possibilities in 6D space (3D position + 3D orientation) because that’s the natural way to describe a grasp. The “geometry-aware” part ensures these representations capture both the physical shape and semantic meaning of the object, enabling robust and context-appropriate grasp planning
In regard to claim 22: (Currently Amended)
Iqbal does not explicitly disclose:
- obtain the answer data by deleting a portion in which the picking up operation of the robot cannot be executed due to the interference from each of the ranges of target value in the 6- dimensional space of the virtual operation command respectively, each of the ranges are generated on each of the plurality of the randomly piled virtual subject models.
However, Homberg discloses:
- obtain the answer data by deleting a portion in which the picking up operation of the robot cannot be executed due to the interference from each of the ranges of target value in the 6- dimensional space of the virtual operation command respectively, each of the ranges are generated on each of the plurality of the randomly piled virtual subject models.
[0029]:
a robot may be configured to perform tasks related to manipulating objects in an environment in which people are present, such as a home, an office space, a restaurant, or a different building. Some robot tasks directed toward manipulating an object in an environment may involve picking up and moving the object, cleaning the object, refilling the object, or otherwise changing the state of the object. In such examples, challenges exist in enabling the robot to collect sufficient sensor data about relevant properties of the object to determine how to manipulate the object. Further challenges exist in enabling the robot to accurately infer a user-preferred manner in which to manipulate the object.
[0093]:
In some examples, the robot 602 may process sensor data from one or more sensors on robot head 604 in order to determine a bounding box around bowl 622. In particular, the bounding box may be a virtual representation of a volume that fully surrounds the bowl 622.
[0096]:
the output of the classifier may answer questions such as: can a bowl be picked up by a robot?, can a bowl be washed by a robot?, or is a bowl in a state such that it is ready to be cleared by a robot? The output of the classifier may be used to control the robot to interact with the object.
[0027]:
In the field of robotics, and robotic grippers in particular, the control system of a robotic device may operate more effectively when provided with information regarding the environment in the area surrounding each component of the robotic device. To provide this information, different types of sensors may be placed on or included in one or more components. However, increasing the number of sensors also means increasing the complexity of the system, as well as increasing the number of possible points of failure. [BRI: a sensor failure can cause the pick up operation not be executed]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Iqbal and Homberg.
Iqbal teaches machine learning data generation and virtual models, a 6-dimensional space for target value for the robot, generating virtual image, based on the plurality of the randomly piled virtual subject models.
Homberg teaches question and answer pairing and label attached to the answer data.
One of ordinary skill would have motivation to combine Iqbal and Homberg that can use accurate sensor data (Homberg [0073])
In regard to claim 23: (Previously Presented)
Iqbal discloses:
- obtain the answer data as union of ranges obtained by deleting the portion in which the picking up operation of the robot cannot be executed from each of the ranges of target value in the 6-dimensional space of the virtual operation command obtained on each of the plurality of the randomly piled virtual subject models respectively.
[0094]:
generating correct answers, such as in result 1014, based on known input data, such as new data 1012.
[0066]:
combines the learned camera-in-hand policies for local control with global 6-DoF pose estimation from a fixed camera in order to grasp objects anywhere in the graspable workspace. [ BRI: in robotic manipulation, different ranges of detection refer to the spatial and angular limits within which a sensor (e.g., a camera) can reliably detect and estimate an object’s 6-DoF pose. Combining learned camera-in-hand policies for local control with global 6-DoF pose estimation from a fixed camera, represents merging two sources of information. Thus, combining these two systems and their detection ranges can represent the union of ranges, enabling reliable grasping anywhere in the graspable workspace by merging long-range global pose estimation with short-range local control]
In regard to claim 24: (Previously Presented)
Iqbal discloses:
and evaluate an achievement status of the picking up operation based on a result of a computer simulation.
[0076]:
In at least one embodiment, the process begins at block 702 with the computer system generating a plurality of simulations. In at least one embodiment, the simulations correspond to a physical system to be controlled. In at least one embodiment, for example, a manipulator in the simulation matches a manipulator on a corresponding robot to be controlled.
[0081]:
the machine-learning computer system directs the robot to grasp the object. In at least one embodiment, the machine-learning computer system may evaluate the success or failure of the grasp using force and/or position sensors on the gripper. In at least one embodiment, success or failure of the grasp is determined using the in-hand camera.
[0080]:
machine-learning computer system determines a particular action that will improve the position of the robotic gripper with respect to producing a successful grasp
[0081]:
in at least one embodiment, success or failure of the grasp is determined based on a Q-score calculated after articulating the robotic gripper
In regard to claim 25: (Previously Presented)
Iqbal discloses:
[0072]:
the environment surrounding the object 504 includes a variety of other objects such as a soda can 506 and a pen 508.
[0079]:
In at least one embodiment, the 6D-pose includes a three-axis translation and three-axis rotation for the object.
[0078]:
the object is identified as a symmetric, asymmetric, or semi-symmetric object.
In regard to claim 26: (Currently Amended)
Iqbal discloses:
- A machine learning device, comprising a central processing unit and a memory which are configured to:
[0078]; [0082];[0075]
- arrange a plurality of virtual subject models in a randomly piled state in a virtual space, the virtual subject models are virtual models of a plurality of subjects of picking up operation of a robot;
[0183]:
In at least one embodiment, 3D flash LIDAR systems include, without limitation, a solid-state 3D staring array LIDAR camera with no moving parts other than a fan (e.g., a non-scanning LIDAR device). In at least one embodiment, flash LIDAR device may use a 5 nanosecond class I (eye-safe) laser pulse per frame and may capture reflected laser light in form of 3D range point clouds and co-registered intensity data
- generate a virtual image based on the plurality of the randomly piled virtual subject models, the virtual image is obtained by virtually generating a image, the image is taken by a camera on the plurality of subjects;
[0076]:
In at least one embodiment, the process begins at block 702 with the computer system generating a plurality of simulations. In at least one embodiment, the simulations correspond to a physical system to be controlled. In at least one embodiment, for example, a manipulator in the simulation matches a manipulator on a corresponding robot to be controlled. In at least one embodiment, the simulations are simulations of an articulated robot and an object to be grasped by the robot. In at least one embodiment, each simulation includes various textures, objects, and patterns that may block, obscure, or clutter the image collected by a virtual camera located on the wrist of the articulated robot
[0076]:
In at least one embodiment, the images are generated from the point of view of the simulated virtual camera mounted on the rest of the articulated robot and oriented in the direction of the robotic gripper used to grasp the object. In at least one embodiment, the images are produced with a camera model that matches a physical camera on the robot to be controlled.
[0285]:
tile-based rendering, in which rendering operations for a scene are subdivided in image space, for example to exploit local spatial coherence within a scene
rendering operations for a scene are subdivided in image space, or example to exploit local spatial coherence within a scene
[0076]:
In at least one embodiment, at block 704, the simulation generates images for each simulation that are used to train the machine-learning system. In at least one embodiment, the images are generated from the point of view of the simulated virtual camera mounted on the rest of the articulated robot and oriented in the direction of the robotic gripper used to grasp the object. In at least one embodiment, the images are produced with a camera model that matches a physical camera on the robot to be controlled.
- generate a range of target values in 6-dimensional space of a virtual operation command by virtually generating a picking up operation command for the robot based on a range of target values in the 6-dimensional space of the picking up operation of the robot given on the subject and one or more of the plurality of the randomly piled virtual subject
[0059]:
In at least one embodiment, the controller computer system obtains an image from a camera with a view of the work area, and determines a 6-d pose of the object 110 from the image. In at least one embodiment, the 6-d pose is a xyz position and a rotational orientation of the object 110
[0183]:
In at least one embodiment, flash LIDAR device may use a 5 nanosecond class I (eye-safe) laser pulse per frame and may capture reflected laser light in form of 3D range point clouds and co-registered intensity data
[0124]:
generate a 3D map of environment of vehicle 1200, including a distance estimate for all points in image
[0082]:
in at least one embodiment, the system above can be adapted to park an autonomous vehicle, accurately position an electric vehicle for charging, pick up a pallet with a forklift. In at least one embodiment, techniques described above can be adapted to operate a pick and place robot used for electronics assembly
[0080]:
the machine-learning computer system determines a particular action that will improve the position of the robotic gripper with respect to producing a successful grasp.
[0074]:
In at least one embodiment, after training in simulation, a policy produced by the training is able to grasp the intended object in the real world without fine-tuning. In at least one embodiment, the action space is gripper-centric, allowing the robot to perform not only top-down grasps (as illustrated in FIG. 5) but also of grasps from other orientations (as illustrated in FIG. 6).
- evaluate an outcome of the picking up operation of the robot in response to the virtual operation command in the virtual space by determining whether an end effector of the robot interferes or not with the plurality of the randomly piled virtual subject models in the virtual space:
0070]:
FIG. 3 illustrates an example of a machine-learning system that can direct a robot to perform a grasp of an object, according to at least one embodiment.
[0080]:
the machine-learning computer system determines a particular action that will improve the position of the robotic gripper with respect to producing a successful grasp.
[0082]:
in at least one embodiment, the system above can be adapted to park an autonomous vehicle, accurately position an electric vehicle for charging, pick up a pallet with a forklift. In at least one embodiment, techniques described above can be adapted to operate a pick and place robot used for electronics assembly
[0122]:
In at least one embodiment, cameras with a field of view that include portions of environment in front of vehicle 1200 (e.g., front-facing cameras) may be used for surround view, to help identify forward facing paths and obstacles,
[0178]:
create a focused beam pattern, designed to record vehicle's 1200 surroundings at higher speeds with minimal interference from traffic in adjacent lanes.
- wherein the end effector is in a position and an angle designated by a target value in the 6-dimensional space within the range of target values in the 6-dimensional space of the virtual operation command;
[0055]:
In at least one embodiment, techniques described herein utilize a deep reinforcement learning approach to grasp semantically meaningful objects in a geometrically consistent way. In at least one embodiment, the system is trained in simulation, with sim-to-real transfer accomplished by using a simulator that both models physical contact between the robot and the object to be grasped, and produces photorealistic imagery. In at least one embodiment, the system provides an example of end-to-end semantic grasping (mapping input pixels to output motor commands in Cartesian or other coordinate systems). In at least one embodiment, by using a camera positioned on the manipulator, the system is not limited to top-down grasps and is capable of grasping objects from any angle. In at least one embodiment, the system is able to grasp objects from multiple pre-defined object-centric orientations, such as from the side or top. When coupled with a real-time 6-DoF object pose estimator, at least one embodiment is capable of grasping objects from any position and orientation within the graspable workspace. In at least one embodiment, results in both simulation and the real-world demonstrate the effectiveness of the approach.
[0064]:
In at least one embodiment, a semantic grasp technique allows a robot to learn to grasp a specific type of object—indicated by the user—from a cluttered bin. Although such approaches address the question of which object to grasp
[0094]:
converging towards a model, such as trained neural network 1008, suitable to generating correct answers, such as in result 1014, based on known input data, such as new data 1012
- execute a computer simulation of the picking up operation based on the virtual operation command in the virtual space with use of a virtual operating machine model and the plurality of the randomly piled virtual subject models, the virtual operating machine model is a virtual model of the robot;
[0247]:
virtualized graphics execution environment is presented in which resources of graphics processing engines 1731-1732, N are shared with multiple applications or virtual machines (VMs).
- evaluate an achievement status of the picking up operation based on a result of a computer simulation;
[0076] :
In at least one embodiment, the process begins at block 702 with the computer system generating a plurality of simulations. In at least one embodiment, the simulations correspond to a physical system to be controlled. In at least one embodiment, for example, a manipulator in the simulation matches a manipulator on a corresponding robot to be controlled.
[0081]:
the machine-learning computer system directs the robot to grasp the object. In at least one embodiment, the machine-learning computer system may evaluate the success or failure of the grasp using force and/or position sensors on the gripper. In at least one embodiment, success or failure of the grasp is determined using the in-hand camera.
[0080]:
machine-learning computer system determines a particular action that will improve the position of the robotic gripper with respect to producing a successful grasp
[0081]:
in at least one embodiment, success or failure of the grasp is determined based on a Q-score calculated after articulating the robotic gripper
- train a neural network model by use of the machine learning data, with determining a permission/prohibition of learning, a direction of learning from a positive direction and a negative direction, or an intensity of learning, according to an evaluation of the achievement status of the picking up operation in the machine learning data.
[0280]:
Inference and/or training logic 915 are used to perform inferencing and/or training operations associated with one or more embodiments.
[0280]:
in at least one embodiment, inference and/or training logic 915 may be used in integrated circuit 1800 for inferencing or predicting operations based, at least in part, on weight parameters calculated using neural network training operations, n
[0080]:
machine-learning computer system determines a particular action that will improve the position of the robotic gripper with respect to producing a successful grasp
[0081]:
the machine-learning computer system directs the robot to grasp the object. In at least one embodiment, the machine-learning computer system may evaluate the success or failure of the grasp using force and/or position sensors on the gripper. In at least one embodiment, success or failure of the grasp is determined using the in-hand camera.
[0081]:
in at least one embodiment, success or failure of the grasp is determined based on a Q-score calculated after articulating the robotic gripper
[0135]:
In at least one embodiment, streaming microprocessors may include independent parallel integer and floating-point data paths to provide for efficient execution of workloads with a mix of computation and addressing calculations.
Iqbal does not explicitly disclose:
- generate machine learning data by pairing the virtual image as question data with answer data[[,]] wherein the answer data is obtained by incorporating the outcome of the picking up operation of the robot into the range of target values of the virtual operation command;
However, Homberg discloses:
and generate machine learning data by pairing the virtual image as question data with answer data [[,]],
[0096]:
Referring back to FIG. 5, at block 510, method 500 further includes determining, by inputting the second sensor data into the classifier, a property of the object. The classifier is an algorithm or function that maps inputted sensor data to one or more object classes. The classes may be associated with semantic labels. For instance, a bowl may be further classified as a plastic bowl or a glass bowl, as a full bowl or an empty bowl, or as a bowl intended for solid food or a bowl intended for drinking. In further examples, the object classes output by a classifier may correspond to different types of intended robot behavior. For instance, the output of the classifier may answer questions such as: can a bowl be picked up by a robot?, can a bowl be washed by a robot?, or is a bowl in a state such that it is ready to be cleared by a robot? The output of the classifier may be used to control the robot to interact with the object.
[0093]:
In some examples, the robot 602 may process sensor data from one or more sensors on robot head 604 in order to determine a bounding box around bowl 622. In particular, the bounding box may be a virtual representation of a volume that fully surrounds the bowl 622. The bounding box may then be used by robot 602 in order to determine how to position gripper 606 above bowl 622 to collect additional sensor data. In particular, gripper 606 may be positioned so that a sensor on gripper 606 is horizontally aligned with the bounding box and above the top of the bounding box by predetermined height 624. In other examples, gripper 606 may be positioned so that some other point on gripper 606 other than the sensor is above bowl 622 by the predetermined height 624.
- wherein the answer data is obtained by incorporating the outcome of the picking up operation of the robot into the range of target values of the virtual operation command.
[0093]:
In some examples, the robot 602 may process sensor data from one or more sensors on robot head 604 in order to determine a bounding box around bowl 622. In particular, the bounding box may be a virtual representation of a volume that fully surrounds the bowl 622.
[0065]:
Short-range time-of-flight sensor 310 may include a narrow light source 312 and a light detector 314 to measure how long it takes laser light projected by light source 312 to bounce back after hitting an object. This time may be used to accurately determine a range or distance to a nearest object from short-range time-of-flight sensor 310 based on the known speed of light. As an example, short-range time-of-flight sensor 310 may have a range of about 1 centimeter up to 20 centimeters from the palm of the gripper. Additionally, short-range time-of-flight sensor 310 may have a relatively narrow field of view (e.g., 40 degrees) in order to detect objects within a cone of sensing range extending out from light detector 314. Based on its range, short-range time-of-flight sensor 310 may be most effective for determining information about grasped objects.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Iqbal and Homberg.
Iqbal teaches machine learning data generation and virtual models, a 6-dimensional space for target value for the robot, generating virtual image, based on the plurality of the randomly piled virtual subject models.
Homberg teaches question and answer pairing and label attached to the answer data.
One of ordinary skill would have motivation to combine Iqbal and Homberg that can use accurate sensor data (Homberg [0073])
In regard to claim 27: (Previously Presented)
Iqbal discloses:
- generate the virtual subject models for a plurality of variations of the subject.
[0072]:
the environment surrounding the object 504 includes a variety of other objects such as a soda can 506 and a pen 508.
[0079]:
In at least one embodiment, the 6D-pose includes a three-axis translation and three-axis rotation for the object.
[0078]:
the object is identified as a symmetric, asymmetric, or semi-symmetric object.
In regard to claim 28: (Currently Amended)
Iqbal discloses:
- A machine learning data generation method of causing a computer to execute:
[0078]; [0082];[0075]
- arranging a plurality of virtual subject models in a randomly piled state in a virtual space, the virtual subject models are virtual models of a plurality of subjects of picking up operation of a robot;
[0183]:
In at least one embodiment, 3D flash LIDAR systems include, without limitation, a solid-state 3D staring array LIDAR camera with no moving parts other than a fan (e.g., a non-scanning LIDAR device). In at least one embodiment, flash LIDAR device may use a 5 nanosecond class I (eye-safe) laser pulse per frame and may capture reflected laser light in form of 3D range point clouds and co-registered intensity data.
- generating a virtual image based on the plurality of the randomly piled virtual subject models, the virtual image is obtained by virtually generating a image, the image is taken by a camera on the plurality of subjects;
[0076]:
In at least one embodiment, the process begins at block 702 with the computer system generating a plurality of simulations. In at least one embodiment, the simulations correspond to a physical system to be controlled. In at least one embodiment, for example, a manipulator in the simulation matches a manipulator on a corresponding robot to be controlled. In at least one embodiment, the simulations are simulations of an articulated robot and an object to be grasped by the robot. In at least one embodiment, each simulation includes various textures, objects, and patterns that may block, obscure, or clutter the image collected by a virtual camera located on the wrist of the articulated robot
[0076]:
In at least one embodiment, the images are generated from the point of view of the simulated virtual camera mounted on the rest of the articulated robot and oriented in the direction of the robotic gripper used to grasp the object. In at least one embodiment, the images are produced with a camera model that matches a physical camera on the robot to be controlled.
[0285]:
tile-based rendering, in which rendering operations for a scene are subdivided in image space, for example to exploit local spatial coherence within a scene
rendering operations for a scene are subdivided in image space, or example to exploit local spatial coherence within a scene
[0076]:
In at least one embodiment, at block 704, the simulation generates images for each simulation that are used to train the machine-learning system. In at least one embodiment, the images are generated from the point of view of the simulated virtual camera mounted on the rest of the articulated robot and oriented in the direction of the robotic gripper used to grasp the object. In at least one embodiment, the images are produced with a camera model that matches a physical camera on the robot to be controlled.
- generating a range of target values in the 6-dimensional space of a virtual operation command by virtually generating a picking up operation command for the robot based on a range of target values in the 6-dimensional space of the picking up operation of the robot given on the subject and one or more of the plurality of the randomly piled virtual subject models
[0059]:
In at least one embodiment, the controller computer system obtains an image from a camera with a view of the work area, and determines a 6-d pose of the object 110 from the image. In at least one embodiment, the 6-d pose is a xyz position and a rotational orientation of the object 110
[0183]:
In at least one embodiment, flash LIDAR device may use a 5 nanosecond class I (eye-safe) laser pulse per frame and may capture reflected laser light in form of 3D range point clouds and co-registered intensity data
[0124]:
generate a 3D map of environment of vehicle 1200, including a distance estimate for all points in image
[0082]:
in at least one embodiment, the system above can be adapted to park an autonomous vehicle, accurately position an electric vehicle for charging, pick up a pallet with a forklift. In at least one embodiment, techniques described above can be adapted to operate a pick and place robot used for electronics assembly
[0080]:
the machine-learning computer system determines a particular action that will improve the position of the robotic gripper with respect to producing a successful grasp.
[0074]:
In at least one embodiment, after training in simulation, a policy produced by the training is able to grasp the intended object in the real world without fine-tuning. In at least one embodiment, the action space is gripper-centric, allowing the robot to perform not only top-down grasps (as illustrated in FIG. 5) but also of grasps from other orientations (as illustrated in FIG. 6).
- evaluating an outcome of the picking up operation of the robot in response to the virtual operation command in the virtual space by determining whether an end effector of the robot interferes or not with the plurality of the randomly piled virtual subject models in the virtual space;
[0070]:
FIG. 3 illustrates an example of a machine-learning system that can direct a robot to perform a grasp of an object, according to at least one embodiment.
[0080]:
the machine-learning computer system determines a particular action that will improve the position of the robotic gripper with respect to producing a successful grasp.
[0082]:
in at least one embodiment, the system above can be adapted to park an autonomous vehicle, accurately position an electric vehicle for charging, pick up a pallet with a forklift. In at least one embodiment, techniques described above can be adapted to operate a pick and place robot used for electronics assembly
[0122]:
In at least one embodiment, cameras with a field of view that include portions of environment in front of vehicle 1200 (e.g., front-facing cameras) may be used for surround view, to help identify forward facing paths and obstacles,
[0178]:
create a focused beam pattern, designed to record vehicle's 1200 surroundings at higher speeds with minimal interference from traffic in adjacent lanes.
- wherein the end effector is in a position and an angle designated by a target value in the 6-dimensional space within the range of target values in the 6-dimensional space of the virtual operation command;
[0055]:
In at least one embodiment, techniques described herein utilize a deep reinforcement learning approach to grasp semantically meaningful objects in a geometrically consistent way. In at least one embodiment, the system is trained in simulation, with sim-to-real transfer accomplished by using a simulator that both models physical contact between the robot and the object to be grasped, and produces photorealistic imagery. In at least one embodiment, the system provides an example of end-to-end semantic grasping (mapping input pixels to output motor commands in Cartesian or other coordinate systems). In at least one embodiment, by using a camera positioned on the manipulator, the system is not limited to top-down grasps and is capable of grasping objects from any angle. In at least one embodiment, the system is able to grasp objects from multiple pre-defined object-centric orientations, such as from the side or top. When coupled with a real-time 6-DoF object pose estimator, at least one embodiment is capable of grasping objects from any position and orientation within the graspable workspace. In at least one embodiment, results in both simulation and the real-world demonstrate the effectiveness of the approach.
[0064]:
In at least one embodiment, a semantic grasp technique allows a robot to learn to grasp a specific type of object—indicated by the user—from a cluttered bin. Although such approaches address the question of which object to grasp
[0094]:
converging towards a model, such as trained neural network 1008, suitable to generating correct answers, such as in result 1014, based on known input data, such as new data 1012
Iqbal does not explicitly disclose:
- and generating machine learning data by pairing the virtual image as question data with answer data;
- [[,]] wherein the answer data is obtained by incorporating the outcome of the picking up operation of the robot into the range of target values of the virtual operation command.
However, Homberg discloses:
and generate machine learning data by pairing the virtual image as question data with answer data [[,]],
[0096]:
Referring back to FIG. 5, at block 510, method 500 further includes determining, by inputting the second sensor data into the classifier, a property of the object. The classifier is an algorithm or function that maps inputted sensor data to one or more object classes. The classes may be associated with semantic labels. For instance, a bowl may be further classified as a plastic bowl or a glass bowl, as a full bowl or an empty bowl, or as a bowl intended for solid food or a bowl intended for drinking. In further examples, the object classes output by a classifier may correspond to different types of intended robot behavior. For instance, the output of the classifier may answer questions such as: can a bowl be picked up by a robot?, can a bowl be washed by a robot?, or is a bowl in a state such that it is ready to be cleared by a robot? The output of the classifier may be used to control the robot to interact with the object.
[0093]:
In some examples, the robot 602 may process sensor data from one or more sensors on robot head 604 in order to determine a bounding box around bowl 622. In particular, the bounding box may be a virtual representation of a volume that fully surrounds the bowl 622. The bounding box may then be used by robot 602 in order to determine how to position gripper 606 above bowl 622 to collect additional sensor data. In particular, gripper 606 may be positioned so that a sensor on gripper 606 is horizontally aligned with the bounding box and above the top of the bounding box by predetermined height 624. In other examples, gripper 606 may be positioned so that some other point on gripper 606 other than the sensor is above bowl 622 by the predetermined height 624.
- wherein the answer data is obtained by incorporating the outcome of the picking up operation of the robot into the range of target values of the virtual operation command.
[0093]:
In some examples, the robot 602 may process sensor data from one or more sensors on robot head 604 in order to determine a bounding box around bowl 622. In particular, the bounding box may be a virtual representation of a volume that fully surrounds the bowl 622.
[0065]:
Short-range time-of-flight sensor 310 may include a narrow light source 312 and a light detector 314 to measure how long it takes laser light projected by light source 312 to bounce back after hitting an object. This time may be used to accurately determine a range or distance to a nearest object from short-range time-of-flight sensor 310 based on the known speed of light. As an example, short-range time-of-flight sensor 310 may have a range of about 1 centimeter up to 20 centimeters from the palm of the gripper. Additionally, short-range time-of-flight sensor 310 may have a relatively narrow field of view (e.g., 40 degrees) in order to detect objects within a cone of sensing range extending out from light detector 314. Based on its range, short-range time-of-flight sensor 310 may be most effective for determining information about grasped objects.
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Iqbal and Homberg.
Iqbal teaches machine learning data generation and virtual models, a 6-dimensional space for target value for the robot, generating virtual image, based on the plurality of the randomly piled virtual subject models.
Homberg teaches question and answer pairing and label attached to the answer data.
One of ordinary skill would have motivation to combine Iqbal and Homberg that can use accurate sensor data (Homberg [0073])
Claim 29 are rejected under 35 U.S.C. 103 unpatentable over
Shariq Iqbal et.al. (hereinafter Iqbal) US 2020/0061811 A1,
in view of Bianca Homberg et.al. (hereinafter Homberg) US 2019/0337152 A1.
further in view Mark OLEYNIK et.al. (hereinafter OLEY) US 2019/0291277 A1.
further in view of Yudai Yuguchi et.al. (hereinafter Yuguchi) US 11803189 B2.
In regard to claim 29: (New)
Iqbal discloses:
- The machine learning data generation device according to wherein if the end effector of the
robot
[0069]:
In at least one embodiment, the learned policy maps 50×50 images downsampled from the eye-in-hand camera mounted on the wrist of the robot to a continuous 4D value representing an action in an end-effector-centric Cartesian coordinate system (that is, 3 translation values and 1 rotation around the wrist axis).
- the answer data is the entirety of the range of target values in the 6-dimensional space of the virtual operation command;
[0094]:
in at least one embodiment, training framework 1004 includes tools to monitor how well untrained neural network 1006 is converging towards a model, such as trained neural network 1008, suitable to generating correct answers, such as in result 1014,
[0059]:
In at least one embodiment, the controller computer system obtains an image from a camera with a view of the work area, and determines a 6-d pose of the object 110 from the image. In at least one embodiment, the 6-d pose is a xyz position and a rotational orientation of the object 110. In at least one embodiment, the controller computer system selects a grasp pose of the object 110 based on the 6-d pose of the object 110.
[0066]:
In at least one embodiment, various techniques described herein provide an approach to geometry-aware semantic grasping that learns to grasp specific objects along specific grasp directions, a method to generate physically plausible, photorealistic synthetic data to train policies that transfer to the real world without a special domain adaptation step, and a system that combines the learned camera-in-hand policies for local control with global 6-DoF pose estimation from a fixed camera in order to grasp objects anywhere in the graspable workspace.
[0109]:
In at least one embodiment, vehicle 1200 may be an airplane, robotic vehicle, or other kind of vehicle.
[0176]:
In at least one embodiment, vehicle 1200 may further include GNSS sensor(s) 1258 (e.g., GPS and/or assisted GPS sensors), to assist in mapping, perception, occupancy grid generation, and/or path planning functions.
[BRI: a geometry-aware semantic grasping often represents grasp possibilities in 6D space (3D position + 3D orientation) showing the range within the entirety of 6-D space]
Iqbal and Homberg do not explicitly disclose:
- wherein if the end effector of the robot does not interfere with the plurality of the randomly piled virtual subject models in the virtual space,
- wherein if the end effector of the robot interferes with the plurality of the randomly piled virtual subject models in the virtual space in a portion of the range of the target values
- less the portion
- wherein if the end effector of the robot interferes with the plurality of the randomly piled virtual subject models in the virtual space in an entirety of the range of target values in the 6- dimensional space of the virtual operation command, the entirety of the range of target values in the 6-dimensional space of the virtual operation command is deleted from the answer data.
However, OLEY discloses:
- wherein if the end effector of the robot does not interfere with the plurality of the randomly piled virtual subject models in the virtual space,
[0392]:
Number of Axes—three axes are required to reach any point in space. To fully control the orientation of the end of the arm (i.e. the wrist), three additional rotational axes (yaw, pitch, and roll) are required.
[0610]:
In task 2565 the robot carries out typical fetching, grasping and transportation of one or more items, completing the tasks using object recognition and environmental sensing, localization and mapping algorithms to optimize movements along obstacle-free paths
[BRI: an entire space is used for navigation- Point A1 of spec Fig 20]
- wherein if the end effector of the robot interferes with the plurality of the randomly piled virtual subject models in the virtual space in a portion of the range of target values in the 6- dimensional space of the virtual operation command, the answer data is the range of target values in the 6-dimensional space of the virtual operation command less the portion
[0392]:
Number of Axes—three axes are required to reach any point in space. To fully control the orientation of the end of the arm (i.e. the wrist), three additional rotational axes (yaw, pitch, and roll) are required.
0724]:
FIG. 103 is a block diagram illustrating the first minimanipulation of stride 3371 pose with the right and left leg in the set of minimanipulations for humanoid to walk in accordance with the present disclosure. As can be seen, the left and right leg, knee, and foot are arranged in a XYZ initial target position. The position may be based on the distance to the ground between the foot and the ground, the angle of the knee with respect to the ground and the overall height of the leg depending on the stepping technique and any potential obstacles [BRI: while the full space is six-dimensional, minimanipulation typically works in a smaller, constrained subset of that space — not because the space itself is reduced in dimensionality, but because the operational domain is limited to a smaller volume ]
[0874]:
in the case when the list of unsatisfied constraints is unavailable we reduce the number of constraints in a pre-planned manner where we remove the maximum number of constraints at a time. [BRI: the potential obstacle can indeed represent the end effector’s footprint in virtual space, and this is a standard practice in robotics simulation and motion planning to prevent self-interference and ensure safe operation
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Iqbal, Homberg and OLEY.
Iqbal teaches machine learning data generation and virtual models, a 6-dimensional space for target value for the robot, generating virtual image, based on the plurality of the randomly piled virtual subject models.
Homberg teaches question and answer pairing and label attached to the answer data.
OLEY teaches obstacle-free path and potential obstacle.
One of ordinary skill would have motivation to combine Iqbal, Homberg and OLEY that learns from its execution-state to improve existing minimanipulation descriptors [OLEY [0769]).
Iqbal, Homberg and OLEY do not explicitly disclose:
- wherein if the end effector of the robot interferes with the plurality of the randomly piled virtual subject models in the virtual space in an entirety of the range of target values in the 6- dimensional space of the virtual operation command, the entirety of the range of target values in the 6-dimensional space of the virtual operation command is deleted from the answer data.
However, Yuguchi discloses:
- wherein if the end effector of the robot interferes with the plurality of the randomly piled virtual subject models in the virtual space in an entirety of the range of target values in the 6- dimensional space of the virtual operation command, the entirety of the range of target values in the 6-dimensional space of the virtual operation command is deleted from the answer data.
[Col 4, lines 44-46]:
As shown in FIG. 1, the robot apparatus 1000 includes an arm 100. The arm 100 is used to grip an object, detect the state of an object, or perform other things. [BRI: pick up operations]
[Col 5, lines 22-24]:
In this event, shape approximation information (such as a bounding box) of a three-dimensional shape obtained by approximating the position and/or shape of a relevant element
[Col 17, lines 34-39]:
an operation plan can be made with consideration of the positions and states of constituent components of the robot apparatus 1000 and relevant elements such as a grip object; thus, a safer operation plan can be made.
[Col 17, lines 47-54]:
In step S26, the robot apparatus 1000 performs traveling or stopping on the basis of the plan drafted in step S24 of the path on which the robot apparatus 1000 moves. Specifically, in step S26, the movement or stopping of the robot apparatus 1000 is performed by a mechanism such as wheels on the underside. The stopping includes also a situation where emergency stopping is performed due to some kind of external factor or internal factor. [ BRI: If the robot detects that its current workspace is completely blocked by obstacles, or if the environment is designed to exclude certain poses (e.g., safety zones), the planner may delete the entire range of poses that would place the end-effector in unsafe or inaccessible regions. In emergency situations (e.g., power loss, sensor failure, or manual stop), the robot may enter a safe state where it is instructed to avoid all poses. This can involve clearing the task space range to prevent unintended movement]
[Col 20, lines 44-52]:
FIG. 9D shows a state where, at time t_13, the robot apparatus 1000 is moving on the basis of the movement plan 78 of the mobile moving body 70 made at time t_12. Although FIG. 9A to FIG. 9D describe an example in which a path not in contact with the mobile obstacle 70 is generated from the prediction of the movement track for the mobile obstacle 70, a judgment of causing the robot apparatus 1000 to stop may be made at the time point when the existence of the mobile obstacle 70 is recognized.
[Col 5, lines 26-32]:
information derived by the robot apparatus, configuration information or the like) of the robot apparatus 1000, in a space obtained or detected from environment information; thereby, shape, position, and attitude information of the relevant element is removed from the environment information [BRI: task space refers to the set of all possible positions, orientations, and attitudes that a system (e.g., a robotic arm) can achieve relative to its environment. This space is typically defined by the environment information — the data describing the surroundings, such as the positions of obstacles, the shape and size of the workspace, and the constraints on motion. Removing these three elements from the environment information does indeed represent deleting the entire task space]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Iqbal, Homberg, OLEY and Yuguchi.
Iqbal teaches machine learning data generation and virtual models, a 6-dimensional space for target value for the robot, generating virtual image, based on the plurality of the randomly piled virtual subject models.
Homberg teaches question and answer pairing and label attached to the answer data.
OLEY teaches obstacle-free path and potential obstacle.
Yuguchi teaches removing the simulation data if there is collision.
One of ordinary skill would have motivation to combine Iqbal, Homberg and Yuguchi that can provide optimal control of the robot (Yuguchi [Col 25, lines 4-14])
Conclusion
Any inquiry concerning this communication or earlier communications from the
examiner should be directed to TIRUMALE KRISHNASWAMY RAMESH whose telephone number is (571)272-4605. The examiner can normally be reached by phone.
Examiner interviews are available via telephone, in-person, and video conferencing
using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached on phone (571-272-3768). The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be
obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit:
https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for
information about filing in DOCX format. For additional questions, contact the Electronic
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO
Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIRUMALE K RAMESH/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights