DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The amendment filed 3/12/2026 has been entered. Claims 1-5, 11, 13, and 15 are amended. Claims 17-19 are newly added. Claims 1-19 remain pending in the application.
Applicant’s arguments, see pages 9-11, with respect to Ooba not teaching the amended claims have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Ooba (IDS: US 20200401801 A1) in view of Krishnaswamy (US 10322802 B1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-4, 6-8, 12-13, 15-16, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ooba (IDS: US 20200401801 A1) in view of Krishnaswamy (US 10322802 B1).
Regarding Claims 1 and 13,
Ooba teaches
(Claim 1) A processing device, comprising: circuitry configured to (“The robot device 5 of the present embodiment includes a monitor device that monitors a motion of the robot device 5. … The monitor device includes an arithmetic processing device.” See at least [0038])
(Claim 13) A processing method for causing a computer to: (“An aspect of the present disclosure is a monitor device for monitoring a motion of a robot device including a robot and an operation tool attached to the robot. The monitor device includes a camera for capturing a motion image of a motion of the robot device and a status detector for detecting an operational status of the robot device. The monitor device includes an arithmetic processing device for acquiring a motion image captured by the camera and the output of the status detector. The arithmetic processing device includes a storage unit for storing predetermined information and a storage control unit for operating the information stored in the storage unit. The arithmetic processing device includes an extraction unit for extracting part of the information stored in the storage unit and a determination unit for determining an operational status of the robot device. The storage control unit performs control for storing a motion image in the storage unit, the image being captured by the camera and attached with a time or an elapsed time from a predetermined reference time, and control for storing a variable number in the storage unit, the variable number being acquired from the output of the status detector and attached with a time or the elapsed time.” See at least [0008]
obtain a plurality of first events indicating that a holding part of a handling robot has passed through respective predetermined passing-through positions; (“FIG. 5 is an explanatory drawing of the control period transmitted by the motion control unit of the controller. FIG. 5 indicates movement points P1, P2, and P3 described in the motion program 41. The robot 1 is controlled so as to move, for example, a tool center point from the movement point P1 to the movement point P2. Furthermore, the robot 1 is controlled so as to move the tool center point from the movement point P2 to the movement point P3. In the motion program 41, the positions of the movement points P1, P2, and P3 and the postures of the robot 1 at the respective positions are determined.” See at least [0063]; “the acceleration and position being recorded in the monitoring information file 73. Variable numbers included in the video 71 and the monitoring information file 73 are attached with times, achieving a strict correspondence between a time included in the video 71 and a time stored in the monitoring information file 73.” See at least [0059]; Examiner Interpretation: The movement points and interpolation points are passing-through positions. The recorded positions are the plurality of events.
obtain a plurality of images of the handling robot (“The camera 10 captures images of a state in which the robot device 5 performs an operation to the workpieces 81 and 82.” See at least [0044])
identify, when obtaining a second event indicating an abnormality in the handling robot, a first period that is between two of the plurality of first events and includes occurrence timing of the second event, or a second period that is from one of the plurality of first events immediately before the occurrence timing to the occurrence timing; … and extract a subset of images, from the set of images, that were obtained in the first period or the second period. (“The extraction unit 34 acquires the time of deviation from the reference operational status. Subsequently, from the storage unit 42, the extraction unit 34 extracts a motion image in a predetermined period including a period preceding the deviation from the reference operational status. The extraction unit 34 extracts a motion image including a motion image preceding the occurrence of an abnormality in the robot device 5. The predetermined period preceding the occurrence of an abnormality may be, for example, a period of one to ten minutes from the deviation from the reference operational status. In the present embodiment, the extraction unit 34 extracts a motion image in a predetermined period including the deviation from the reference operational status.” See at least [0054]; “A motion image immediately before the deviation of the robot device 5 from the reference operational status and a variable number acquired from the output of the status detector are stored in the storage unit 42. … For example, the operator acquires the time of the deviation of an acceleration from the determination range from the monitoring information file 73, thereby confirming a status immediately before the time of the deviation of the acceleration from the determination range in the video 71.” See at least [0059]; Examiner Interpretation: The described period preceding the deviation from the reference operational status is interpreted as either of the claimed first or second period under broad reasonable interpretation because the period both includes the deviation and is a period from an recorded position immediately before the deviation to the deviation timing.)
Ooba does not explicitly teach, but Krishnaswamy teaches
obtain a plurality of images of the handling robot photographed by a plurality of photographing devices, each of the photographing devices continuously photographing a corresponding area; (“systems and techniques relating to gathering visual information using combinations of fixed sensors and on-demand sensors. The systems and techniques can be implemented in a workspace of a material handling system … The visual information can also be used to maintain and update a global view map of the workspace. The fixed sensors can be positioned at fixed locations throughout the workspace to monitor certain areas (e.g., high-traffic areas). The on-demand sensors can be selectively deployed throughout the workspace to collect additional visual information on an as-needed basis (e.g., as requested, as inferred, as offered, etc.).” See at least col. 2, lines 43-58; “Such sensors can include optical sensors such as imaging devices, depth sensors, visible light cameras, infrared cameras, RGB cameras, depth aware cameras, infrared laser projectors, ultraviolet sensitive cameras, scanning sensors, light filters, and any combination of the foregoing.” See at least col. 5, lines 29-34; Also see at least fig. 15 illustrating the robots in view of the fixed sensors.; Examiner Interpretation: The monitoring of areas and maintaining/updating a map of the workspace by the cameras demonstrates that the photographing devices continuously photograph corresponding areas.)
determine at least one of the plurality of photographing devices based on an area in which the abnormality occurs; extract, from the plurality of images, a set of images photographed by the determined photographing device; (“Turning now to the view 1502, in this view, the item 1510 has fallen from the grasp of the robotic manipulator 1506b (e.g., illustrated by the dashed line) to a location on a surface 1514, as illustrated by the arrow. A controller of the robotic manipulator 1506b and/or the management module 15 may use the fixed sensor package 1508b and/or the arm sensor package 1512 to search for the item 1510. In this example, it is assumed that this search does not reveal the item 1510 (e.g., because the item 1510 has fallen out of the fields of view of the sensors, 1508, 1512). Because the robotic manipulator 1506b cannot find the item 1510, the robotic manipulator 1506b can send a request to the management module 15 for assistance searching for the item 1510. The request may indicate characteristics about the item 1510, when it was dropped, an approximate location, and other relevant contextual details. Based on the request, the management module 15 can identify an on-demand sensor to assist with the search. In this example, the management module 15 has identified the robotic manipulator 1506a including the arm sensor package 1512a and the fixed sensor package 1508a as likely sensors for identifying the item 1510. Thus, the management module 15 may use one or more of the sensor packages 1508a, 1512a to search for the item 1510.” See at least col. 28, line 48 through col. 29, line 4 and fig. 15; Examiner Interpretation: The approximate location of the dropped item is interpreted as the area in which abnormality occurs. The determined likely sensors for identifying the item in response to the request are the determined at least one of the plurality photographing devices.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of Ooba to further include the teachings of Krishnaswamy with a reasonable expectation of success to improve the sensing capabilities of the robot system and the ability of the robot system to resolve problems. (See at least col. 2, lines 43-58 and col. 6, lines 7-20)
Regarding Claim 2,
Ooba does not explicitly teach, but Krishnaswamy teaches
wherein the circuitry is further configured to, when obtaining the second event, estimate an area, among the plurality of areas, in which the holding part is present at the occurrence timing, and one of the plurality of photographing devices that photographs the estimated area is selected as the determined photographing device. (“the fixed on-demand sensor 106b may be a camera of a robotic manipulator that is typically used to detect objects that the robotic manipulator will pick up, but may be deployed for other purposes when requested. For example, assume that two robotic manipulators are working near each other and a first robotic manipulator detects that it has dropped an item but the item cannot be found in a field of view of a camera of the first robotic manipulator. In this example, in response to a request from the first robotic manipulator, a second robotic manipulator may position its camera toward the first robotic manipulator to search for the dropped item. Once identified, an appropriate action can be performed.” See at least col. 6, lines 7-20, wherein the camera on the second robot is the selected photographing device.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Ooba to further include the teachings of Krishnaswamy with a reasonable expectation of success to improve the sensing capabilities of the robot system and the ability of the robot system to resolve problems. (See at least col. 2, lines 43-58 and col. 6, lines 7-20)
Regarding Claim 3,
Ooba does not explicitly teach, but Krishnaswamy teaches
wherein the circuitry is further configured to, when obtaining a second event indicating a drop of an item held by the holding part, estimate an area, among the plurality of areas, in which the dropping item is present, and one of the plurality of photographing devices that photographs the estimated area is selected as the determined photographing device. (“Turning now to the view 1502, in this view, the item 1510 has fallen from the grasp of the robotic manipulator 1506b (e.g., illustrated by the dashed line) to a location on a surface 1514, as illustrated by the arrow. A controller of the robotic manipulator 1506b and/or the management module 15 may use the fixed sensor package 1508b and/or the arm sensor package 1512 to search for the item 1510. In this example, it is assumed that this search does not reveal the item 1510 (e.g., because the item 1510 has fallen out of the fields of view of the sensors, 1508, 1512). Because the robotic manipulator 1506b cannot find the item 1510, the robotic manipulator 1506b can send a request to the management module 15 for assistance searching for the item 1510. The request may indicate characteristics about the item 1510, when it was dropped, an approximate location, and other relevant contextual details. Based on the request, the management module 15 can identify an on-demand sensor to assist with the search. In this example, the management module 15 has identified the robotic manipulator 1506a including the arm sensor package 1512a and the fixed sensor package 1508a as likely sensors for identifying the item 1510. Thus, the management module 15 may use one or more of the sensor packages 1508a, 1512a to search for the item 1510.” See at least col. 28, line 48 through col. 29, line 4 and fig. 15)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Ooba to further include the teachings of Krishnaswamy with a reasonable expectation of success to improve the sensing capabilities of the robot system and the ability of the robot system to resolve problems. (See at least col. 2, lines 43-58 and col. 6, lines 7-20)
Regarding Claim 4
Ooba does not explicitly teach, but Krishnaswamy teaches
wherein a plurality of sections are set in an installation place of the handling robot, each of the plurality of sections is associated with any of the plurality of photographing devices, (“systems and techniques relating to gathering visual information using combinations of fixed sensors and on-demand sensors. The systems and techniques can be implemented in a workspace of a material handling system … The visual information can also be used to maintain and update a global view map of the workspace. The fixed sensors can be positioned at fixed locations throughout the workspace to monitor certain areas (e.g., high-traffic areas). The on-demand sensors can be selectively deployed throughout the workspace to collect additional visual information on an as-needed basis (e.g., as requested, as inferred, as offered, etc.).” See at least col. 2, lines 43-58; “Such sensors can include optical sensors such as imaging devices, depth sensors, visible light cameras, infrared cameras, RGB cameras, depth aware cameras, infrared laser projectors, ultraviolet sensitive cameras, scanning sensors, light filters, and any combination of the foregoing.” See at least col. 5, lines 29-34)
the circuitry is further configured to, when obtaining a second event indicating a drop of an item held by the holding part, determine a section, among the plurality of sections, into which the item has dropped, and one of the plurality of photographing devices that is associated with the determined section is selected as the determined photographing device.. (“Turning now to the view 1502, in this view, the item 1510 has fallen from the grasp of the robotic manipulator 1506b (e.g., illustrated by the dashed line) to a location on a surface 1514, as illustrated by the arrow. A controller of the robotic manipulator 1506b and/or the management module 15 may use the fixed sensor package 1508b and/or the arm sensor package 1512 to search for the item 1510. In this example, it is assumed that this search does not reveal the item 1510 (e.g., because the item 1510 has fallen out of the fields of view of the sensors, 1508, 1512). Because the robotic manipulator 1506b cannot find the item 1510, the robotic manipulator 1506b can send a request to the management module 15 for assistance searching for the item 1510. The request may indicate characteristics about the item 1510, when it was dropped, an approximate location, and other relevant contextual details. Based on the request, the management module 15 can identify an on-demand sensor to assist with the search. In this example, the management module 15 has identified the robotic manipulator 1506a including the arm sensor package 1512a and the fixed sensor package 1508a as likely sensors for identifying the item 1510. Thus, the management module 15 may use one or more of the sensor packages 1508a, 1512a to search for the item 1510.” See at least col. 28, line 48 through col. 29, line 4 and fig. 15; Examiner Interpretation: The approximate location is interpreted as the determined section.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Ooba to further include the teachings of Krishnaswamy with a reasonable expectation of success to improve the sensing capabilities of the robot system and the ability of the robot system to resolve problems. (See at least col. 2, lines 43-58 and col. 6, lines 7-20)
Regarding Claim 6,
Ooba further teaches
A processing system comprising: the processing device according to claim 1; (“The robot device 5 of the present embodiment includes a monitor device that monitors a motion of the robot device 5. … The monitor device includes an arithmetic processing device.” See at least [0038])
and a terminal device connected to the processing device via a network, the processing device being configured to transmit the extracted image to the terminal device. (“The monitoring information file and video extracted by the extraction unit may be transmitted to another device via a communication device. For example, the information can be transmitted to an operator in the section for analyzing an abnormality of the robot device.” See at least [0089])
Regarding Claim 7,
Ooba further teaches
further comprising: a photographing device configured to obtain the plurality of images by photographing the handling robot. (“The camera 10 captures images of a state in which the robot device 5 performs an operation to the workpieces 81 and 82.” See at least [0044]; Also see at least fig. 3 illustrating an example of a captured image including claws 3 of the robot.)
Regarding Claim 8,
Ooba teaches
A handling system comprising: the processing device according to claim 1; (“The robot device 5 of the present embodiment includes a monitor device that monitors a motion of the robot device 5. … The monitor device includes an arithmetic processing device.” See at least [0038])
a photographing device configured to obtain the plurality of images by photographing the handling robot; (“The camera 10 captures images of a state in which the robot device 5 performs an operation to the workpieces 81 and 82.” See at least [0044]; Also see at least fig. 3 illustrating an example of a captured image including claws 3 of the robot.)
the handling robot; (“A first robot device 5 includes a hand 2 acting as an operation tool (end effector) and a robot 1 that moves the hand 2.” See at least [0017])
and a control device configured to control the handling robot so as to make the holding part pass through the passing-through positions. (“A controller for the robot device 5 includes a robot controller 4.” See at least [0030]; “FIG. 5 is an explanatory drawing of the control period transmitted by the motion control unit of the controller. FIG. 5 indicates movement points P1, P2, and P3 described in the motion program 41. The robot 1 is controlled so as to move, for example, a tool center point from the movement point P1 to the movement point P2. Furthermore, the robot 1 is controlled so as to move the tool center point from the movement point P2 to the movement point P3. In the motion program 41, the positions of the movement points P1, P2, and P3 and the postures of the robot 1 at the respective positions are determined.” See at least [0063])
Regarding Claim 12,
Ooba further teaches
wherein the handling robot is a picking robot. (“The hand 2 is an operation tool that grips and releases a workpiece 81. The hand 2 has a plurality of claws 3. The hand 2 is formed so as to open and close the claws 3. The claws 3 catch the workpiece 81 so as to grip the workpiece 81. The hand 2 of the present embodiment has the claws 3 but the configuration is not limited thereto. The hand may have any configuration formed so as to grip the workpiece. For example, a hand that grips a workpiece by suction or a magnetic force may be used.” See at least [0020])
Regarding Claim 15,
Ooba does not explicitly teach, but Krishnaswamy teaches
wherein a plurality of sections are set in an installation place of the handling robot, each of the plurality of sections is associated with any of the plurality of photographing devices, (“systems and techniques relating to gathering visual information using combinations of fixed sensors and on-demand sensors. The systems and techniques can be implemented in a workspace of a material handling system … The visual information can also be used to maintain and update a global view map of the workspace. The fixed sensors can be positioned at fixed locations throughout the workspace to monitor certain areas (e.g., high-traffic areas). The on-demand sensors can be selectively deployed throughout the workspace to collect additional visual information on an as-needed basis (e.g., as requested, as inferred, as offered, etc.).” See at least col. 2, lines 43-58; “Such sensors can include optical sensors such as imaging devices, depth sensors, visible light cameras, infrared cameras, RGB cameras, depth aware cameras, infrared laser projectors, ultraviolet sensitive cameras, scanning sensors, light filters, and any combination of the foregoing.” See at least col. 5, lines 29-34)
the computer is further configured to when obtaining a second event indicating a drop of an item held by the holding part, determine a section, among the plurality of sections, into which the item has dropped, and one of the plurality of photographing devices that is associated with the determined section is selected as the determined photographing device. (“Turning now to the view 1502, in this view, the item 1510 has fallen from the grasp of the robotic manipulator 1506b (e.g., illustrated by the dashed line) to a location on a surface 1514, as illustrated by the arrow. A controller of the robotic manipulator 1506b and/or the management module 15 may use the fixed sensor package 1508b and/or the arm sensor package 1512 to search for the item 1510. In this example, it is assumed that this search does not reveal the item 1510 (e.g., because the item 1510 has fallen out of the fields of view of the sensors, 1508, 1512). Because the robotic manipulator 1506b cannot find the item 1510, the robotic manipulator 1506b can send a request to the management module 15 for assistance searching for the item 1510. The request may indicate characteristics about the item 1510, when it was dropped, an approximate location, and other relevant contextual details. Based on the request, the management module 15 can identify an on-demand sensor to assist with the search. In this example, the management module 15 has identified the robotic manipulator 1506a including the arm sensor package 1512a and the fixed sensor package 1508a as likely sensors for identifying the item 1510. Thus, the management module 15 may use one or more of the sensor packages 1508a, 1512a to search for the item 1510.” See at least col. 28, line 48 through col. 29, line 4 and fig. 15; Examiner Interpretation: The approximate location is interpreted as the determined section.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Ooba to further include the teachings of Krishnaswamy with a reasonable expectation of success to improve the sensing capabilities of the robot system and the ability of the robot system to resolve problems. (See at least col. 2, lines 43-58 and col. 6, lines 7-20)
Regarding Claim 16,
Ooba further teaches
A non-transitory computer-readable storage medium storing a program for causing the computer to perform the processing method (“The arithmetic processing device includes a storage unit for storing predetermined information and a storage control unit for operating the information stored in the storage unit.” See at least [0008]; “The arithmetic processing device includes RAM (Random Access Memory) and ROM (Read Only Memory) that are connected to the CPU via a bus. … The storage unit 42 may include a storage medium capable of storing information, such as a volatile memory, a nonvolatile memory, or a hard disk.” See at least [0030-0031])
See the rejection above of claim 13 in view of Ooba and Krishnaswamy for the processing method according to claim 13.
Regarding Claim 18,
Ooba further teaches
wherein the handling robot is configured to convey an item from a first place to a second place, (“The robot device 5 of the present embodiment attaches the workpiece 81 to the workpiece 82. … The robot device 5 grips the workpiece 81 disposed on a work table, which is not illustrated in the figures. Subsequently, the position and posture of the robot 1 are changed so as to place the workpiece 81 on the top surface of the workpiece 82 as indicated by an arrow 91.” See at least [0022] and fig. 6)
Ooba does not explicitly teach, but Krishnaswamy teaches
and the plurality of photographing devices are configured to photograph the plurality of areas between the first place and the second place. (“The process 1900 may begin at 1902 by maintaining a first version of a digital map of a workspace based at least in part on first visual information. In some examples, the first visual information may be received from a portion of a plurality of fixed sensors. The digital map may include a plurality of regions corresponding to a plurality of areas within the workspace. In some examples, a plurality of mobile drive units may move about the workspace transporting a plurality of inventory holders that are configured to carry inventory items. The workspace may correspond to one or more floors in a warehouse. The first visual information may be received from one or more fixed sensors disposed at fixed locations within the workspace.” See at least col. 33, lines 16-28)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Ooba to further include the teachings of Krishnaswamy with a reasonable expectation of success to improve the sensing capabilities of the robot system and the ability of the robot system to resolve problems. (See at least col. 2, lines 43-58 and col. 6, lines 7-20)
Claim(s) 5 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ooba (IDS: US 20200401801 A1) in view of Krishnaswamy (US 10322802 B1), Cui (Translated CN 114851208 A), and Hickman (US 20190077019 A1).
Regarding Claim 5,
Modified Ooba does not explicitly teach, but Cui teaches
wherein the second event is generated when an item held by the holding part has dropped (“determine the electronic device in the process of grabbing and placing the current object to be grasped by the control grabbing module, whether the grabbing failure event occurs (i.e., whether there is an event of object falling).” See at least pg. 17, paragraph 7)
and the circuitry is further configured to, when the item has dropped, extract the image obtained in the first period, (“the electronic device each time control grabbing module for grabbing the current to be captured object after placing, can through the third image information after placing the object in the discharging device and the second image information of the object to be captured currently obtained before placing, to monitor whether there is a grabbing failure event, so that the user can timely find the object falling event in the object grabbing process.” See at least pg. 17, paragraph 8; Examiner Interpretation: The images are obtained in the first period because they are obtained in a period from before to after the grabbing failure event.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Ooba to further include the teachings of Cui with a reasonable expectation of success “so that the user can timely find the object falling event in the object grabbing process.” (See at least pg. 17, paragraph 8)
Cui also does not explicitly teach, but Hickman teaches
wherein the second event is generated … when the holding part has come into contact with an object other than the item to be held, (“a computing device may obtain accelerometer data for at least part of the prior time period, and the computing device may determine that the accelerometer data is indicative of a collision between the physical object and the robotic device at a particular time.” See at least [0057])
and when the holding part has come into contact with the object, extract the image obtained in the second period. (“responsive to determining that the log of sensor data relates to the collision, the computing device can then generate a training data set for the collision from the log of sensor data. The training data set may include multiple image frames of the sequence of image frames that are prior to an image frame in the sequence of image frames that corresponds to the particular time.” See at least [0019])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Ooba and Cui to further include the teachings of Hickman with a reasonable expectation of success to facilitate training “to predict when future collisions will occur and to take preventive actions to avoid colliding with physical objects,” and therefore improve safety. (See at least [0018])
Regarding Claim 14,
Modified Ooba does not explicitly teach, but Cui teaches
wherein the second event is generated when an item held by the holding part has dropped (“determine the electronic device in the process of grabbing and placing the current object to be grasped by the control grabbing module, whether the grabbing failure event occurs (i.e., whether there is an event of object falling).” See at least pg. 17, paragraph 7)
the computer is further configured to, when the item has dropped, extract the image obtained in the first period, (“the electronic device each time control grabbing module for grabbing the current to be captured object after placing, can through the third image information after placing the object in the discharging device and the second image information of the object to be captured currently obtained before placing, to monitor whether there is a grabbing failure event, so that the user can timely find the object falling event in the object grabbing process.” See at least pg. 17, paragraph 8; Examiner Interpretation: The images are obtained in the first period because they are obtained in a period from before to after the grabbing failure event.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of Ooba to further include the teachings of Cui with a reasonable expectation of success “so that the user can timely find the object falling event in the object grabbing process.” (See at least pg. 17, paragraph 8)
Cui also does not explicitly teach, but Hickman teaches
wherein the second event is generated … when the holding part has come into contact with an object other than the item to be held, (“a computing device may obtain accelerometer data for at least part of the prior time period, and the computing device may determine that the accelerometer data is indicative of a collision between the physical object and the robotic device at a particular time.” See at least [0057])
and the computer is further configured to, when the holding part has come into contact with the object, extract the image obtained in the second period. (“responsive to determining that the log of sensor data relates to the collision, the computing device can then generate a training data set for the collision from the log of sensor data. The training data set may include multiple image frames of the sequence of image frames that are prior to an image frame in the sequence of image frames that corresponds to the particular time.” See at least [0019])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of Ooba and Cui to further include the teachings of Hickman with a reasonable expectation of success to facilitate training “to predict when future collisions will occur and to take preventive actions to avoid colliding with physical objects,” and therefore improve safety. (See at least [0018])
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ooba (IDS: US 20200401801 A1) in view of Krishnaswamy (US 10322802 B1) and Shankar (US 20190005374 A1).
Regarding Claim 9,
Ooba further teaches
further comprising: a terminal device connected to the processing device via a network, (“The monitoring information file and video extracted by the extraction unit may be transmitted to another device via a communication device. For example, the information can be transmitted to an operator in the section for analyzing an abnormality of the robot device.” See at least [0089])
an item conveyed by the handling robot, (“The hand 2 is an operation tool that grips and releases a workpiece 81.” See at least [0020])
Ooba does not explicitly teach, but Shankar teaches
wherein the terminal device is configured to display a user interface including the extracted image, (“the user interface input is generated at a client computing device based on a user indicating the second bounding area via the client computing device. The second bounding can be indicated via the client device during display of an image that is based on the sensor data. In some version of those implementations, the method further includes providing the image to the client device in response to initial user interface input that indicates the robotic action is incorrect.” See at least [0025])
and a correction section for inventory information of the item, and change the inventory information of the item in accordance with input to the correction section from a user. (“The correction instance engine 130 can determine the further user interface input 103B2 indicates a correction to the action being performed on the object 252.sub.D. For example, the correction instance engine 130 can determine that the “hat” classification, indicated by the user interface input 103B2, conflicts with the incorrectly determined “dish” classification. In response, the correction instance engine 130 can interact with the local update module 126 to cause the local update module 126 to update the features model 160N. For example, the correction instance engine 130 can provide, to the local update module 126, an indication that the “bowl” classification was incorrect and that “hat” is instead the correct classification.” See at least [0071])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Ooba to further include the teachings of Shankar with a reasonable expectation of success because “such adjustment(s) to the classification model can immediately improve performance of future classifications” and “such improved classification performance further leads to improvement in performance of various robotic actions that depend (directly or indirectly) on classifications of objects.” (See at least [0008])
Claim(s) 10-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ooba (IDS: US 20200401801 A1) in view of Krishnaswamy (US 10322802 B1) and Stubbs (US 9694494 B1).
Regarding Claim 10,
Ooba further teaches
further comprising: a terminal device connected to the processing device via a network, (“The monitoring information file and video extracted by the extraction unit may be transmitted to another device via a communication device. For example, the information can be transmitted to an operator in the section for analyzing an abnormality of the robot device.” See at least [0089])
Ooba does not explicitly teach, but Stubbs teaches
wherein the terminal device is configured to display a user interface including the extracted image and an operation section configured to initialize a motion of the handling robot, and initialize the motion of the handling robot in accordance with input to the operation section from a user. (“a graphical user interface that enables the operator 706 to view and control in real-time the robotic manipulator 110. The grasp management service 102 may be capable of providing such an interface. Using the interface, the operator 706 may view on an output device 708 (e.g., a display device) aspects of the grasping station 704. For example, the output device 708 may display a streaming video feed of the grasping station 704. … the graphical user interface enables a display of a generated representation of the grasping station 704. For example, using the graphical user interface, the operator 706 may select, from a three-dimensional model representative of an item 712, one or more graspable surfaces on the item 712 (e.g., an outside surface and an inside surface). Using these graspable surfaces, a grasp may be generated. In some examples, the operator 706 may select the graspable surfaces and a possible grasp from a list of grasps. The robotic manipulator 110 may then attempt to grasp the item 712 using the selected grasping surfaces and grasp. In some examples, the operator 706 may directly instruct the robotic manipulator 110 how to pick up the item 712 by controlling the movements of the robotic manipulator 110. Both examples may be considered teaching acts, and the information associated with the attempted grasps can be saved and used to generate other grasps, as described herein.” See at least col. 15, lines 27-54)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Ooba to further include the teachings of Stubbs with a reasonable expectation of success to facilitate robot teaching and to increase the probability of a successful grasp. (See at least col. 15, lines 27-54 and col. 16, lines 26-45)
Regarding Claim 11,
Modified Ooba does not explicitly teach, but Stubbs teaches
wherein the control device is further configured to obtain item data at least one selected from the group consisting of a size of an item to be held, a form of the item, a weight of the item, and a material of the item, and change, in accordance with the item data, at least one selected from the group consisting of a holding power for the item, a holding method for the item, and a conveyance speed for the item. (“the item database 232 may include an item identifier (e.g., a unique product identifier), a description of the item, one or more stock images of the item, a surface model of the item or a link to the surface model of the item, a primitive shape model of the item or a link to the primitive shape model, a bounding box representation of the item, one or more actual images of the item (e.g., taken as it entered a facility), dimensions of the item (e.g., height, width, length), a location of a center of mass, a total weight, and any other suitable of information related to the item. This information can be used as part of determining a set of grasps for picking up the item and validating those grasps.” See at least col. 7, lines 52-64; “When a new item is encountered and it is desired to pick up the item using “arm tool 1” of the set of end of arm tools 524, the grasp database 522 may be used to identify features of the new item that have primitive shapes that are similar to any of the set of primitive shapes 526 and sized similar to the versions 528. In this manner, even without having previously attempted to grasp the new item, a grasp associated with the appropriately sized version of a primitive shape may be identified from the grasp database 522 and used to grasp a feature of the new item.” See at least col. 14, lines 12-21)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Ooba to further include the teachings of Stubbs with a reasonable expectation of success to facilitate automated robot handling of a larger variety of items. (See at least col. 7, lines 52-64 and col. 14, lines 12-21)
Claim(s) 17 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ooba (IDS: US 20200401801 A1) in view of Krishnaswamy (US 10322802 B1) and Kurokawa (US 20200047302 A1).
Regarding Claim 17,
Modified Ooba does not explicitly teach, but Kurokawa teaches
wherein the circuitry is further configured to associate each of the plurality of images with a label indicating a period in which the image is captured, (“Each image frame includes a timestamp” See at least [0064])
and extract the subset of images using the associated label. (“the information processing device can use, … the abnormality occurrence time … to thereby extract video data.” See at least [0098]; “Information processing device 80 may use timestamps included in a plurality of image frames constituting video data as well as time information included in operational information to synchronize video data with operational information that span the set period Ts. Specifically, information processing device 80 may process an image frame having a timestamp representing the time when information processing device 80 received from press controller 32 a bit indicative of abnormality” See at least [0128])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Ooba to further include the teachings of Kurokawa with a reasonable expectation of success to timestamp image frames to facilitate identifying video data spanning a period including a time of occurrence of an abnormality to be checked afterwards. (See at least [0012-0013])
Regarding Claim 19,
Modified Ooba does not explicitly teach, but Kurokawa teaches
wherein each of the plurality of images is associated with a label indicating a period in which the image is captured, (“Each image frame includes a timestamp” See at least [0064])
and the subset of images is extracted using the associated label. (“the information processing device can use, … the abnormality occurrence time … to thereby extract video data.” See at least [0098]; “Information processing device 80 may use timestamps included in a plurality of image frames constituting video data as well as time information included in operational information to synchronize video data with operational information that span the set period Ts. Specifically, information processing device 80 may process an image frame having a timestamp representing the time when information processing device 80 received from press controller 32 a bit indicative of abnormality” See at least [0128])
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to modify the teachings of modified Ooba to further include the teachings of Kurokawa with a reasonable expectation of success to timestamp image frames to facilitate identifying video data spanning a period including a time of occurrence of an abnormality to be checked afterwards. (See at least [0012-0013])
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Karston G Evans whose telephone number is (571)272-8480. The examiner can normally be reached Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached at (571)270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.G.E./Examiner, Art Unit 3657 /ABBY LIN/Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights