DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 8 is objected to because of the following informalities:
In line 1, it appears the word “to” should be before “handle”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1-22 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Relative to claim 1 and 21-22, operating a robotic structure according to the item label is ambiguous. The specification describes the robotic arm handling an item based on a location of a label on the item (Para. 00115). It is not clear as to what Applicant means by “according to the item label”.
Relative to claim 19, “operating the robotic structure, in response to said determination, to move one or more items in the workspace” after determining that no strategy is available to grasp any item from the workspace based on sensor data is unclear.
The specification describes the robot to identify and grasp another item if the system cannot determine a grasp strategy, or to send an alert to a human worker for assistance. Does Applicant mean “moving one or more other items in the workspace”? Does Applicant mean that the robot moves the item with no grip strategy to another location? The specification does not support moving the item where no grasp strategy can be determined to another location.
Also, “no strategy is available to grasp any item” is also unclear. Does Applicant mean “no strategy is available to grasp an item”?
The remaining claims not listed above are rejected because of their dependency from claim 1.
Appropriate clarification is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-15, and 19-22 (as understood by the Examiner) is/are rejected under 35 U.S.C. 103 as being unpatentable over Wicks et al (US PG. 2019/0344448) in view of Diankov et al (US PG. Pub. 2020/0238519). Relative to claims 1-3, 5-15, and 19-20, Wicks discloses:
Claim 1) A system (100)(Fig. 1), comprising:
a communication interface (Para. 0042); and
a processor coupled to the communication interface (Para. 0049) and configured to:
receive sensor data via the communication interface, the sensor data comprises image data associated with a workspace comprising an item (110)(vision system 115 includes sensors for capturing image data of the environment surrounding the robot, Para. 0048, 0071)(Fig. 1);
use the sensor data to determine an item surface comprising an item label (0048);
in response to determining the item surface comprising the item label, operate a robotic structure handle (104, 106)(Fig. 1) the item according to the item label (Para. 0049; 0071, system considers item label information to determine a manipulation strategy), including operating the robotic structure (104, 106) based on a placement strategy determined based at least in part on the item label (Para. 0049).
Claim 2) operating a robotic structure handle (104, 106) the item according to
the item label comprises:
operating the robotic structure handle (104, 106) the item according to the item surface comprising the item label (Para. 0070), including operating the robotic structure to pick the item, move the item within the workspace, and place the item according to an orientation in which the item label faces (Para. 0071).
Claim 5) operating the robotic structure handle (104, 106) the item according to the item label comprises: handling the item according to the item surface comprising the item label (Para. 0050).
Claim 7) operating the robotic structure handle (104, 106) the item according to the item label comprises: controlling the robotic structure (104) to move the item and cause a scanner to scan the item label based at least in part on the item surface comprising the item label (Para. 0050, 0070).
Claim 9) operating a robotic structure handle (104, 106) the item according to the item label comprises: operating the robotic structure (104) to place the item according to a placement strategy that is determined based at least in part on the item label (Para. 0051).
Claim 10) handling the item according to the item surface comprising the item label comprises: determining to locally scan the item label locally using a sensor system (115) based at least in part on a location of the item label (Para. 0070; 0075, sensors/cameras 115 that are separate from the robot can scan item label, as shown in Fig. 1); causing the sensor system (115) to locally scan the item label (Para. 0071).
Claim 11) after the item label is locally scanned, the item is handled by the robotic structure (104); and the item is placed on a conveyor (130)(Para. 0047-0048).
Claim 14) handling the item (110) according to a location of the item label comprises: determining to scan the item label with a downstream sensor (see 115, Fig. 1) as the item is carried on a conveyor after the item is picked from the workspace and placed on a conveyor (130)(Para. 0050).
Claim 20) the robotic structure (104) comprises a robotic arm (104)(Fig. 1).
Wicks does not expressly disclose:
Claim 1) the sensor data includes three dimensional image data associated with a workspace,
Claim 3) the item is routed based on downstream sensor data.
Claim 6) handling the item according to the item surface comprising the item label comprises: controlling the robotic structure to grasp the item offset from the item label;
Claim 8) operating a robotic structure handle the item label according to the item label comprises: invoking human intervention based at least in part on the item label.
Claim 12) the sensor system comprises a multi-view array of sensors, the sensor system being configured to obtain label information across a plurality of possible item orientations (Para. 0048).
Claim 13) the processor determines to locally scan the label based at least in part on a determination that the robotic structure is unable to rotate the item to an orientation in which a downstream sensor is able to read the item label after the item is placed on a conveyor.
Claim 15) the processor determines to scan the item label with the downstream sensor if the robotic structure is able to successfully place the item on the conveyor in an orientation in which the item label is viewable by the downstream sensor.
Claim 19) the processor is configured to determine, based at least in part on the sensor data, that no strategy is available to grasp any item from the workspace and to operate the robotic structure, in response to said determination, to move one or more items in the workspace.
Diankov teaches:
Claim 1) the sensor data includes three dimensional image data associated with a workspace (Para. 0088, 0113)(Fig. 2),
Claim 6) handling the item according to the item surface comprising the item label comprises: controlling the robotic structure to grasp the item offset from the item label (Para. 0165, robot eliminates approaches that cause the end effector to cover the location 334);
Claim 12) the sensor system comprises a multi-view array of sensors, the sensor system being configured to obtain label information across a plurality of possible item orientations (0127, 0128, sensors 416, 417 may capture top and bottom surfaces of item)(Fig. 4B).
Claim 13) the processor (Para. 0071, 0032) determines to locally scan the label based at least in part on a determination that the robotic structure (robotic arm 414)(Fig. 4B) is unable to rotate the item to an orientation in which a downstream sensor is able to read the item label after the item is placed on a conveyor (Para. 0128).
Claim 15) the processor determines to scan the item label with the downstream sensor (416, 417) if the robotic structure (414) is able to successfully place the item on the conveyor in an orientation in which the item label is viewable by the downstream sensor (Para. 0123, confidence measure for the pose is high)(Fig. 4B).
Claim 19) the processor is configured to determine, based at least in part on the sensor data, that no strategy is available to grasp any item from the workspace and to operate the robotic structure (414), in response to said determination, to move one or more items in the workspace (Para. 0199-0200, robot chooses a remedial action if a grasp strategy cannot be determines, such as cancelling the control sequence (picking up) of the item that has no grasp strategy, and selects a next operation object item).
Diankov teaches: the sensor data includes three dimensional image data associated with a workspace, controlling the robotic structure to grasp the item offset from the item label, the sensor system comprises a multi-view array of sensors, locally scanning the label based on a determination that the robotic structure is unable to rotate the item to an orientation in which a downstream sensor is able to read the item label, scanning the item label with the downstream sensor if the robotic structure (414) is able to successfully place the item on the conveyor, and determining that no strategy is available to grasp any item from the workspace, and operating the robotic structure to move items in the workspace described above, for the purpose of providing a control method of a robotic system that manipulates an operation object such as an article, that can realize a high degree of cooperation between humans and robots (Para. 0006).
It would have been obvious to one of ordinary skill in the art on or before the time of the filing to modify the system of Wicks with the sensor data includes three dimensional image data associated with a workspace, controlling the robotic structure to grasp the item offset from the item label, the sensor system comprises a multi-view array of sensors, locally scanning the label based on a determination that the robotic structure is unable to rotate the item to an orientation in which a downstream sensor is able to read the item label, scanning the item label with the downstream sensor if the robotic structure (414) is able to successfully place the item on the conveyor, and determining that no strategy is available to grasp any item from the workspace, and operating the robotic structure to move items in the workspace described above as taught in Diankov for the purpose of providing a control method of a robotic system that manipulates an operation object such as an article, that can realize a high degree of cooperation between humans and robots.
Relative to claim 8, Wicks in view of Diankov discloses all claim limitations mentioned above, but does not expressly disclose: operating a robotic structure to handle the item label according to the item label comprises: invoking human intervention based at least in part on the item label.
Wicks in view of Diankov teaches operating a robotic structure (414) to handle the item label according to the item label comprises invoking human intervention based at least in part on the item label as a matter of design choice.
Diankov teaches that involving human intervention for addressing pose determination errors as well-known means to adjust, re-execute, complement, stop or otherwise correct the system to improve accuracy (Para. 0061, 0201).
It would have been obvious to one of ordinary skill in the art on or before the effective filing date of the claimed invention to modify the system of Wicks in view of Diankov with the invoking human intervention based at least in part on the item label since this is well-known in the art of controlling a robotic arm for gripping objects.
Relative to claims 21-22, the disclosure of Wicks includes:
A method and computer program product embodied in a non-transitory computer readable medium, comprising computer instructions for;
receiving sensor data of a workspace (100) comprising an item (Para. 0071);
using the sensor data to determine an item surface comprising an item label (Para. 0048);
in response to determining the item surface comprising the item label (Para. 0071; 0049),
operating a robotic structure handle (104, 106) the item according to the item label, including operating the robotic structure (104) based on a placement strategy determined based at least in part on the item label (Para. 0051).
Wicks does not expressly disclose: the sensor data includes three-dimensional image data associated with a workspace.
Diankov teaches: the sensor data includes three-dimensional image data associated with a workspace (Para. 0113), for the purpose of providing a control method of a robotic system that manipulates an operation object such as an article, that can realize a high degree of cooperation between humans and robots.
It would have been obvious to one of ordinary skill in the art on or before the time of the filing to modify the system of Wicks with the sensor data includes three-dimensional image data as taught in Diankov for the purpose of providing a control method of a robotic system that manipulates an operation object such as an article, that can realize a high degree of cooperation between humans and robots.
Claim(s) 3-4 and 16-17 (as understood by the Examiner) is/are rejected under 35 U.S.C. 103 as being unpatentable over Wicks in view of Diankov as applied to claim above, and further in view of Kawano (US PG. Pub. 2015/0057793). Relative to claims 4, and 16-17, Wicks in view of Diankov discloses all claim limitations mentioned above, including:
Claim 3) a downstream sensor (see sensor/camera 115, positioned separate from robot 104, cameras may be installed around an area of the materials handling environment, Fig. 1) reads the item label facing a particular direction to obtain downstream sensor data (Para. 0071).
Claim 4) obtaining label information from the item label based at least in part on a downstream sensor data (sensors for determining characteristics are positioned at locations within the material handling system 100, Para. 0048, 0075).
Wicks in view of Diankov does not expressly disclose:
Claim 3) the item is routed based on downstream sensor data.
Claim 4) the processor is further configured to: and operate a conveyor system or other robotic structure to route the item based on the label information;
claim 16) the item label includes label information comprising routing information; or
claim 17) the processor uses the routing information to sort or route the item to an intermediate or final destination.
Kawano teaches:
Claim 3) the item is routed based on downstream sensor data (Para. 0016, 0018-0019).
Claim 4) the processor is further configured to: and operate a conveyor system (312)(Fig. 3) or other robotic structure to route the item based on the label information (Para. 0016, 0018-0019);
claim 16) the item label includes label information comprising routing information (Para. 0016, 0018-0019); and
claim 17) the processor (Para. 0032) uses the routing information to sort or route the item to an intermediate or final destination (Para. 0016, 0018-0019), for the purpose of providing an automated system and method for picking products in a materials handling facility that is capable of reliably and adaptively picking up items of various types, shapes, sizes, etc. (Para. 0002)
It would have been obvious to one of ordinary skill in the art on or before the time of the filing to modify the system of Wicks in view of Diankov with the routing the item based on the label information, the label information comprising routing information described above, as taught in Kawano for the purpose of providing an automated system and method for picking products in a materials handling facility that is capable of reliably and adaptively picking up items of various types, shapes, sizes, etc.
Claim(s) 18 (as understood by the Examiner) is/are rejected under 35 U.S.C. 103 as being unpatentable over Wicks in view of Diankov as applied to claim 1 above, and further in view of Wagner (US PG. Pub. 2018/0333749). Relative to claim 18, Wicks in view of Diankov discloses all claim limitations mentioned above, but does not expressly disclose: the processor is configured to use a model to detect unstable flow of at least a subset of a plurality of items, and to pause autonomous operation in response to detecting said unstable flow.
Wagner teaches: the processor (included in Ref. 78) is configured to use a model to detect unstable flow of at least a subset of a plurality of items (Para. 0045; 0055), and to pause autonomous operation in response to detecting said unstable flow (speed of conveyor 16 may be adjusted to slow down if moving too fast), for the purpose of providing automated and robotic systems for use in environments requiring a variety of objects to be sorted and/or otherwise distributed to several output destinations that is more efficient, space saving, and reduces costs (Para. 0002, 0008).
It would have been obvious to one of ordinary skill in the art on or before the time of the filing to modify the system of Wicks in view of Diankov with the processor configured to use a model to detect unstable flow of a plurality of items, and pausing autonomous operation in response to detecting said unstable flow described above as taught in Wagner, for the purpose of providing automated and robotic systems for use in environments requiring a variety of objects to be sorted and/or otherwise distributed to several output destinations that is more efficient, space saving, and reduces costs.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Sciog et al (US PG. Pub. 2019/0030712)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YOLANDA RENEE CUMBESS whose telephone number is (571)270-5527. The examiner can normally be reached M-F 10-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gene Crawford can be reached at 571-272-6911. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YOLANDA R CUMBESS/Primary Examiner, Art Unit 3651
Prosecution Insights