DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This is a nonfinal in response to the RCE filed on 01/15/2026. Claims 15-31 remain pending. Claims 15-19, 21-24, 26-27, and 29-30 have been amended.
Claim Objections
Claims 22 and 29-31 are objected to because of the following informalities:
Claim 22 states “in a context of determining the at least one object parameter of the object, and at least one further object parameter regarding each of the further objects is also ascertained”. The claim is not written in grammatically correct English. Applicant should remove the term “and” so that the claim reads “in a context of determining the at least one object parameter of the object, at least one further object parameter regarding each of the further objects is also ascertained”.
Claims 29 and 30 recite “wherein the data processor one of (i) comprises an edge device, or (ii) is configured as an edge device”. The claim is not written in grammatically correct English. Applicant should remove the term “one of” so that the claim reads “wherein the data processor (i) comprises an edge device, or (ii) is configured as an edge device”.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 15, 21, and 23-25 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Goldberg et al (US 20190210223 A1) (Hereinafter referred to as Goldberg)
Regarding Claim 15, Goldberg teaches a method for ascertaining control data for a gripping device for gripping an object (See at least Goldberg Paragraph 0028 and Figure 1), the method comprising:
capturing an image of the object (See at least Goldberg Paragraph 0075, the imaging device detects the object);
determining at least one object parameter for the captured object (See at least Goldberg Paragraph 0075, the pose, which is an object parameter, is determined using the image sensor);
ascertaining control data for the gripping device for gripping the object at at least one predetermined gripping point (See at least Goldberg Paragraphs 0028, 0032, 0042, 0065, and Figure 2, control data/signals for gripping the object at the gripping point, which is predetermined based on the planned grasp set for the stable poses, is ascertained), each predetermined gripping point of the object being effected utilizing information limited to possible stable poses of the object (See at least Goldberg Paragraphs 0028, 0042, 0065, and Figures 1-2, the grasp set for gripping the object at the gripping point is computed for each stable pose);
controlling the gripping device to grip the object at the at least one predetermined gripping point (See at least Goldberg Paragraphs 0028, 0032, and Figure 1, the gripping device/robot grips the object at the gripping point using the predetermined/planned grasp set for the stable poses);
wherein each possible stable pose of the object is established such that all pose data of the object which are convertible into one another via at least one of a displacement and a rotation about a surface normal of a support surface on which the object lies are assigned to said possible stable pose (See at least Goldberg Paragraphs 0028, 0042, 0044, 0075 and Figure 1, the finite set of stable poses are determined by rotating the object about the axis perpendicular to the work surface); and
wherein at least one of (i) said determining the at least one object parameter and (ii) said ascertaining the control data for the gripping device is effected utilizing information regarding the possible stable pose of the object (See at least Goldberg Paragraphs 0028, 0032, and 0042, the control data is ascertained using the possible stable poses of the object).
Regarding Claim 21, Goldberg teaches at least one of said determining the at least one object parameter, ascertaining ID information, ascertaining position data, determining a pose of the object, determining a virtual bounding box around the object and/or determining a stable pose adopted by the object are effected utilizing the information regarding the possible stable pose (See at least Goldberg Paragraph 0042 and Figure 2, the contact point, which is position data, is ascertained using the possible stable pose).
Regarding Claim 23, Goldberg teaches a method for gripping an object (See at least Goldberg Paragraphs 0028, 0032, and Figure 1), wherein at least one predetermined gripping point of the object is ascertained (See at least Goldberg Paragraphs 0028, 0042, 0065, and Figure 2, the gripping/contact point of the object is determined using the planned/predetermined grasp set) and the object is subsequently gripped by a gripping device in accordance with the method as claimed in claim 15 (See at least Goldberg Paragraphs 0028, 0032, and Figure 1, the gripping device/robot grips the object at the gripping point using the predetermined/planned grasp set for the stable poses); and wherein the gripping device engages at the at least one predetermined gripping point when gripping the object (See at least Goldberg Paragraphs 0028, 0032, 0042, 0065, and Figure 2, the robot/gripping device grips the object at the planned/predetermined contact/gripping point).
Regarding Claim 24, Goldberg teaches a system for gripping an object (See at least Goldberg Paragraph 0028 and Figure 1), the system comprising:
an optical capture device for capturing an image of the object (See at least Goldberg Paragraph 0075, the imaging device detects the object); and
a data processor for at least one of determining at least one object parameter of the object and ascertaining control data for a gripping device for gripping the object (See at least Goldberg Paragraphs 0028, 0032, 0042, 0075, and 0111, the processor determines the pose of the object, which is an object parameter, and ascertains control data/signals for the robot/gripping device);
wherein the data processor of the system is configured to:
ascertain control data for the gripping device for gripping the object at at least one predetermined gripping point (See at least Goldberg Paragraphs 0028, 0032, 0042, 0065, and Figure 2, control data/signals for gripping the object at the gripping point, which is predetermined based on the planned grasp set for the stable poses, is ascertained), each predetermined gripping point of the object being effected utilizing information limited to possible stable poses of the object (See at least Goldberg Paragraphs 0028, 0042, 0065, and Figures 1-2, the grasp set for gripping the object at the gripping point is computed for each stable pose);
wherein each possible stable pose of the object is established such that all pose data of the object which are convertible into one another via at least one of a displacement and a rotation about a surface normal of a support surface on which the object lies are assigned to the possible stable pose (See at least Goldberg Paragraphs 0028, 0042, 0044, 0075 and Figure 1, the finite set of stable poses are determined by rotating the object about the axis perpendicular to the work surface);
wherein at least one of (i) said determining the at least one object parameter and (ii) said ascertaining the control data for the gripping device is effected utilizing information regarding the possible stable pose of the object (See at least Goldberg Paragraphs 0028, 0032, and 0042, the control data is ascertained using the possible stable poses of the object); and
wherein the data processor is further configured to control the gripping device to grip the object at the at least one predetermined gripping point (See at least Goldberg Paragraphs 0028, 0032, 0042, 0065, and Figure 2, the robot/gripping device grips the object at the planned/predetermined contact/gripping point);
Regarding Claim 25, Goldberg teaches the gripping device (See at least Goldberg Paragraphs 0028, 0032, and Figure 1, the robot is interpreted as a gripping device).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Goldberg in view of Oyama et al (US 20230104802 A1) (Hereinafter referred to as Oyama)
Regarding Claim 16, Goldberg teaches said ascertaining the at least one predetermined gripping point further comprises:
…determining at least one model gripping point from the 3D model of the object (See at least Goldberg Paragraphs 0028, 0030, 0042, and Figure 1, the contact/gripping points for the mesh model are determined using the 3d mesh model of the object); and
determining the at least one predetermined gripping point of the object utilizing the model gripping point (See at least Goldberg Paragraphs 0028, 0030, 0032, 0042, and Figure 1, the predetermined gripping point for the planned grasp set is determined utilizing the gripping/contact point on the mesh model).
Goldberg fails to disclose selecting a 3D model for the object using the at least one object parameter.
However, Oyama teaches selecting a 3D model for the object using the at least one object parameter (See at least Oyama Paragraphs 0033, 0050, 0058, 0063, and 0068, the recognition unit selects the 3d model for the object to be used by the operation sequence generation unit using the image of the object, which includes the object parameters).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in Goldberg with Oyama to select a 3D model for the object using the at least one object parameter. Oyama teaches selecting a 3D model for the object by imaging the object and comparing it with object model information to recognize the type, the position, the posture, the ongoing (currently-executing) operation and the like of the object (See at least Oyama Paragraphs 0033 and 0050). This modification allows the system to generate an abstract model for the sequence between the robot and the object using the selected 3D model of the object based on the detected object parameters (See at least Oyama Paragraphs 0058, 0063, and 0068), which would improve the planning for the grasping operation.
Claims 17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Goldberg in view of Danielczuk et al (US 20220152826 A1) (Hereinafter referred to as Danielczuk)
Regarding Claim 17, Goldberg fails to disclose wherein the utilization of information regarding the possible stable pose is configured as utilization of a machine learning model; and
wherein the ML model is at least one of trained and configured to, via application of a ML method, ascertain information regarding the possible stable pose.
However, Danielczuk teaches wherein the utilization of information regarding the possible stable pose is configured as utilization of a machine learning model (See at least Danielczuk Paragraph 0087, the scene collision network is interpreted as machine learning model, which utilizes information regarding the stable poses); and
wherein the ML model is at least one of trained and configured to, via application of a ML method, ascertain information regarding the possible stable pose (See at least Danielczuk Paragraphs 0060, 0087, and Figure 9, the scene collision network/ML model is trained via the ML method in Figure 9 to ascertain trajectory information regarding grasping the object at the stable pose).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in Goldberg with Danielczuk to train the ML model to ascertain information regarding the at least one possible stable pose. This modification, as taught by Danielczuk, would allow the system to train the ML model to grasp the object at a stable pose while avoiding collisions (See at least Danielczuk Paragraphs 0060, 0087, and Figure 9), which would improve the safety of the system.
Regarding Claim 20, Goldberg teaches determining the at least one object parameter further comprises: ascertaining position data of the object (See at least Goldberg Paragraph 0075, the detected pose identifies the present position of the object).
Goldberg fails to explicitly disclose the position data further comprises information regarding a stable pose adopted by the object.
However, Danielczuk teaches the position data further comprises information regarding a stable pose adopted by the object (See at least Danielczuk Paragraphs 0068 and 0087, the camera images the object placed at the origin, wherein a stable pose is adopted by the object).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in Goldberg with Danielczuk to have the position data further comprise information regarding a stable pose adopted by the object. This modification, as taught by Danielczuk, would allow the system to utilize the position data to grasp the object at a stable pose while avoiding collisions (See at least Danielczuk Paragraphs 0060, 0067, 0087, and Figure 9), which would improve the safety of the system.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Goldberg in view of Oyama, and in further view of Danielczuk
Regarding Claim 18, modified Goldberg fails to disclose wherein the utilization of information regarding the possible stable pose is configured as utilization of a machine learning model; and
wherein the ML model is at least one of trained and configured to, via application of a ML method, ascertain information regarding the possible stable pose.
However, Danielczuk teaches wherein the utilization of information regarding the possible stable pose is configured as utilization of a machine learning model (See at least Danielczuk Paragraph 0087, the scene collision network is interpreted as machine learning model, which utilizes information regarding the stable poses); and
wherein the ML model is at least one of trained and configured to, via application of a ML method, ascertain information regarding the possible stable pose (See at least Danielczuk Paragraphs 0060, 0087, and Figure 9, the scene collision network/ML model is trained via the ML method in Figure 9 to ascertain trajectory information regarding grasping the object at the stable pose).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in Goldberg with Danielczuk to train the ML model to ascertain information regarding the at least one possible stable pose. This modification, as taught by Danielczuk, would allow the system to train the ML model to grasp the object at a stable pose while avoiding collisions (See at least Danielczuk Paragraphs 0060, 0087, and Figure 9), which would improve the safety of the system.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Goldberg in view of Oyama, and in further view of Wagner et al (US 20200164531 A1) (Hereinafter referred to as Wagner)
Regarding Claim 19, even though Goldberg teaches ascertaining the at least one gripping point of the object using the model gripping point (See at least Goldberg Paragraphs 0028, 0030, 0032, 0042, and Figure 1, the predetermined gripping point for the planned grasp set is determined utilizing the gripping/contact point on the mesh model), modified Goldberg fails to disclose said ascertaining the at least one predetermined gripping point of the object using the model gripping point is one of (i) effected utilizing a further ML model which is trained and configured to, via application of the ML method, determine transformation data regarding possible transformations of a predefined or predefinable initial position into possible poses of the object and (ii) effected aided by application of an image evaluation method to the captured image of the object.
However, Wagner teaches ascertaining a gripping point of the object using the model gripping point is effected aided by application of an image evaluation method to the captured image of the object (See at least Wagner Paragraphs 0007, 0040, and Figure 3, an image evaluation method is used to determine a grasp location for the object in the captured image).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Goldberg with Wagner to ascertain the at least one predetermined gripping point of the object using the model gripping point effected aided by application of an image evaluation method to the captured image of the object. This modification, as taught by Wagner, would allow the system to determine the gripping point by capturing an image of the object and evaluating the image for good and bad gripping points based on the current pose of the object (See at least Wagner Paragraphs 0007, 0040-0042, and Figure 3), which would improve the gripping point selection process.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Goldberg in view of Zhu et al (US 20210308866 A1) (Hereinafter referred to as Zhu)
Regarding Claim 22, Goldberg fails to disclose when capturing the image of the object, further objects are captured and, in a context of determining the at least one object parameter of the object, and at least one further object parameter regarding each of the further objects is also ascertained; and wherein after ascertaining object parameters regarding the object and the further objects, selection of the object is effected.
However, Zhu teaches when capturing the image of the object, further objects are captured (See at least Zhu Paragraphs 0020-0021 and Figure 1, a plurality of workpieces/objects are captured) and, in a context of determining the at least one object parameter of the object, and at least one further object parameter regarding each of the further objects is also ascertained (See at least Zhu Paragraphs 0020-0021 and Figure 1, the position and orientation for the plurality of workpieces/objects are determined, which is interpreted as further object parameters); and wherein after ascertaining object parameters regarding the object and the further objects, selection of the object is effected (See at least Zhu Paragraphs 0020-0021, 0047 and Figure 6, the selection of the object/workpiece is effected after imaging the objects/workpieces).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in Goldberg with Zhu to select the object after ascertaining object parameters regarding the object and the further objects. This modification, as taught by Zhu, would allow the system to select and pick up an individual object when there are a plurality of objects to choose from (See at least Zhu Paragraphs 0020-0021, 0047, and Figure 1), which would improve the picking operation performed by the gripping device.
Claims 26-27 are rejected under 35 U.S.C. 103 as being unpatentable over Goldberg in view of Kapoor et al (US 20170075331 A1) (Hereinafter referred to as Kapoor)
Regarding Claims 26-27, Goldberg fails to disclose the data processor is one of (i) configured as a modular programmable logic controller having a central module and a further module or (ii) comprises the programmable logic controller; and
wherein said determining the at least one object parameter of the object is effected utilizing the further module.
However, Kapoor teaches the data processor is one of (i) configured as a modular programmable logic controller having a central module and a further module (See at least Kapoor Paragraphs 0027-0029, 0070, and Figure 2, the processing unit/data processor is configured as modular PLC having an application programming interface, which is interpreted as a central module, and a pick scheduler, which is interpreted as a further module) or comprises the programmable logic controller (See at least Kapoor Paragraphs 0027-0029, and Figure 2, the processing unit/data processor is part of the PLC); and
wherein said determining the at least one object parameter of the object is effected utilizing the further module (See at least Kapoor Paragraph 0040, the further module/pick scheduler determines the position of the workpiece, which is interpreted as the object parameter).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in Goldberg with Kapoor to have the data processor be configured as a modular programmable logic controller with a further module for determining the object parameter. PLCs, as taught by Kapoor, are configurable by a user and can be employed in a complex environment utilizing a large number of robots to generate motion commands (See at least Kapoor Paragraphs 0030 and 0034). By having the PLC be modular, as taught by Kapoor, the functionality is enhanced (See at least Kapoor Paragraph 0070). By using the further module of the PLC to determine the object parameter, the positions and orientation of each individual workpiece is determined (See at least Kapoor Paragraph 0040), which would allow the PLC to generate the sequence with which the robot will manipulate the workpieces (See at least Kapoor Paragraph 0041), which improves the efficiency of the system.
Claim 28 is rejected under 35 U.S.C. 103 as being unpatentable over Goldberg in view Kapoor, and in further view of Yuvaraj et al (US 20200234071 A1) (Hereinafter referred to as Yuvaraj)
Regarding Claim 28, modified Goldberg fails to disclose at least one of:(i) said determining the at least one object parameter of the object is effected utilizing a machine learning model, the further module comprising the ML model and (ii) said ascertaining the control data for the gripping device is effected utilizing a further ML model, the further module comprising the further ML model.
However, Yuvaraj teaches ascertaining the control data for the gripping device is effected utilizing a further ML model (See at least Yuvaraj Paragraphs 0042, 0052, 0128, 0130-0131, and Figure 2 and 9e, ascertaining the control data for the gripping device/material handling apparatus is effected utilizing the machine learning unit/model), the further module comprising the further ML model (See at least Yuvaraj Paragraph 0052 and Figure 2, the PLC includes the machine learning unit/model in the learning and classification unit/further module).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Goldberg with Yuvaraj to ascertain the control data for the gripping device utilizing a further ML model in the further module. Yuvaraj teaches a ML model in the further module that is trained to classify candidate regions, which is used to ascertain control data for the gripping device (See at least Yuvaraj Paragraphs 0052, 0067-0068, 0128, and 0130-0131). This modification would improve the classification performed by the PLC.
Claims 29-31 are rejected under 35 U.S.C. 103 as being unpatentable over Goldberg in view Al-Qunaieer (US 20210287040 A1) (Hereinafter referred to as Al-Qunaieer)
Regarding Claims 29-30, Goldberg fails to disclose the data processor one of (i) comprises an edge device or (ii) is configured as an edge device and wherein said determining the at least one object parameter of the object is effected utilizing the edge device.
However, Al-Qunaieer teaches the data processor one of (i) comprises an edge device or (ii) is configured as an edge device (See at least Al-Qunaieer Paragraph 0027 and Figure 2, the computing system/data processing device is configured as an edge device) and wherein said determining the at least one object parameter of the object is effected utilizing the edge device (See at least Al-Qunaieer Paragraphs 0021-0023, 0027 and Figures 1a and 2, the computing system/edge device determines the characteristics of the object, which are interpreted as the object parameters, from the image data).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in Goldberg with Al-Qunaieer to have the data processor be configured as an edge device that is utilized to determine the object parameter. Edge devices are routine and well-understood in the art as an option for computing devices that process data (See at least Al-Qunaieer Paragraph 0027). By having the edge device determine the object parameter, as taught by Al-Qunaieer, object parameters such as shape, size, texture, and color, can be extracted (See at least Al-Qunaieer Paragraphs 0021-0023 and 0039-0042), thus, improving object detection.
Regarding Claim 31, modified Goldberg fails to disclose at least one of:(i) said determining the at least one object parameter of the object is effected utilizing a ML model, the edge device comprising the ML model and (ii) ascertaining the control data for the gripping device comprises utilizing a further ML model, the edge device comprising the further ML model.
However, Al-Qunaieer teaches determining the at least one object parameter of the object is effected utilizing a ML model, the edge device comprising the ML model (See at least Al-Qunaieer Paragraphs 0019, 0021-0023, 0027, 0057, and Figures 1a and 2, the computing system/edge device includes a machine learning model, which is utilized to determine the object parameters/characteristics).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Goldberg with Al-Qunaieer to have the edge device comprise a ML model that is utilized to determine the object parameter. By having the edge device comprise a ML model that determines the object parameter, as taught by Al-Qunaieer, the edge device can be trained to extract object parameters such as shape, size, texture, and color (See at least Al-Qunaieer Paragraphs 0003, 0021-0023, 0035-0036, and 0039-0042), thus, improving object detection.
Response to Arguments
Applicant’s arguments with respect to claims 15 and 24 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Applicant has amended the independent claims to clarify that the gripping point is predetermined, and is effected utilizing information limited to possible stable poses of the object. These limitations are taught by Goldberg, which teaches generating predetermined gripping points for an object using the stable poses of the object. The predetermined gripping points are to be used by a robot for execution. Therefore, the claims still stand rejected under 102 and 103.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ESVINDER SINGH whose telephone number is (571)272-7875. The examiner can normally be reached Monday-Friday: 9 am-5 pm est.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached at 571-270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ESVINDER SINGH/ Examiner, Art Unit 3657