DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The specification is objected to as failing to provide proper antecedent basis for the claimed subject matter. See 37 CFR 1.75(d)(1) and MPEP § 608.01(o). Correction of the following is required: clarify instances where the second imager takes the first image as in claim 9.
It is understood that a first imager takes a first image at a first position and a first attitude while a second imager takes a second image at a second position and a second attitude. There is improper antecedent basis for a second imager to take a first image. It is recommended to identify the imagers simply as overhead or hand, leaving the sequence of the images the imagers take as first image, second image, third image…etc. Thus, the hand imager can take a first image at a first position and attitude.
Status of Claims
This communication is a first office action, non-final rejection on the merits. Claims 1-16 as filed, are currently pending and have been considered below.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 9 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 9 recites the limitations “obtain an overhead image captured by a first imager” in line 3 and “calculated, based on the overhead image, a first position and a first attitude with which the first image is captured” in lines 6-7. Furthermore, the specification discloses the overhead camera taking the overhead image as being “referred to as a first imager” in [0030], thus appearing to consider the overhead image and the first image taken by the first imager to be synonymous. Based on examiner’s best understanding, it is reasonably interpreted that this establishes the first image taken by the first imager is intended to be the overhead image and the claim will be interpreted as the first image and the overhead image being the same image.
However, claim 9 lines 8-9 further recites “control the robot such that a second imager attached to the robot captures an image with the first position and the first attitude” and lines 10-11 recites “obtains, as the first image, the image captured by the second imager.” It is now unclear which imager is taking the first image, as applicant claims the first/overhead image being taken by the first imager and the second imager.
For examination purposes, examiner will assume the first imager takes a first image, the first image and the overhead image refer to the same image, and the second imager takes a second image.
Examiner will read claim 9 as
The robot control apparatus according to claim 1, wherein the controller is configured to
obtain an overhead image captured by a first imager located in such a way as to capture an image of at least a part of an operation range of the robot, wherein the first image is the overhead image,
calculate, on a basis of the overhead image, a first position and a first attitude with which the overhead image is captured,
control the robot such that a second imager attached to the robot captures the second image with the first position and the first attitude, and
obtain the second image captured by the second imager.
Examiner notes wherein the claims have been addressed below, in view of the prior art record, as best understood by the Examiner in light of the 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph rejections provided herein.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, and 9-16 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Oda et al. (Japanese Patent application JP 2022033167 hereinafter “Oda”). For clarity of English language translation, paragraph and image references will be made to US Patent Application 20220318942 which claims priority to Oda.
Regarding Claim 1, Oda discloses in figure 1 and 0147
A robot control apparatus (10) comprising:
a controller that controls (300) a robot (100) including a holder (102),
wherein the controller is configured to
obtain a first image (I1) of at least one holding target (W),
select a selected target (W1) to be held by the holder from the at least one holding target recognized in the first image
determine a holding position (K1) on the selected target on a basis of a second image (I2) of the selected target, and
cause the holder to hold the selected target at the determined holding position.
Oda further describes the control of the above apparatus in process steps S101-S110 found in figure 5 and 0072-0109.
Regarding Claim 9, Oda discloses all of the limitations of claim 1 and further discloses
obtain an overhead image captured by a first imager located in such a way as to capture an image of at least a part of an operation range of the robot, wherein the first image is the overhead image,
In figure 1 and 0063 Oda describes “The CPU 251 executes the program 261, and thereby causes the camera 401 to capture an image of an area in which the plurality of workpieces W exists, and detects a workpiece W that can be picked.” The fact that the workpiece W “can be picked” indicates that the image includes at least part of the operational range of the robot.
calculate, on a basis of the overhead image, a first position and a first attitude with which the first image is captured,
From the first image (I1) taken from the first imager (401) Oda determines the height of the workpieces (figure 9A), the location of the container (301), and an axis of the world coordinate frame (CO). It is understood to those of ordinary skill in the art that in order to obtain such information, the position and attitude of the first imager is known.
As detailed earlier, it is understood by the examiner that the hand imager may take an overhead image to use as the first image. Thus, Oda further discloses
control the robot such that a second imager attached to the robot captures the second image with the first position and the first attitude, and
obtain the second image captured by the second imager.
In 0129 Oda details how the second imager may take an overhead image to use as the first image I1 “In this case, the position of the camera 402 may be adjusted by the motion of the robot 100, and the image I1 may be produced by the camera 402.”
Regarding Claim 10, Oda discloses all of the limitations of claim 1 and further discloses in figure 6B and 0074
wherein the controller is configured to generate (S102) a recognition box surrounding the at least one holding target (A,B,C, or D) as a result of the recognition of the at least one holding target (WI).
Regarding Claim 11, Oda discloses all of the limitations of claim 1 and further discloses
wherein the controller is configured to select the selected target on a basis of a score indicating a result of the recognition of the at least one holding target.
In figure 9B and 0068 Oda details the workpiece priority ranking and score “The priority determination portion 214 assigns priorities to the plurality of candidate areas detected by the workpiece detection portion 211, in the order of easiness for the robot 100 to pick the workpiece; and extracts a candidate area with a top priority.”
Regarding Claim 12, Oda discloses all of the limitations of claim 11 and further discloses
evaluate holdability on a basis of the score and
selects the selected target on a basis of a result of the evaluation.
In equation 3 and 0095 Oda details how the holdability, or priority, is determined by a score and selects a target workpiece based on said score.
Regarding Claim 13, Oda discloses all of the limitations of claim 11 and further discloses in 0081 and 0087
generate a mask (WI) covering each of the at least one holding target in the first image as the result of the recognition of the at least one holding target, and
calculate the score (x.sub.1) on a basis of the mask.
Regarding Claim 14, Oda discloses all of the limitations of claim 1 and further discloses with the fourth embodiment and in 0159
select the at least one holding target as the selected target on a basis of the first image and
determine whether to capture the second image.
Oda details how the determination to capture the second image is based on the success of the pattern matching process “if the pattern matching can be performed on a search area of the image I1 obtained by the image capture operation of the camera 401, the camera 402, that is, the image I2 may not be used.”
Regarding Claim 15, Oda discloses all of the limitations of claim 14 and further discloses in the fourth embodiment and figure 15
wherein, when determining that the second image need not be captured, the controller is configured to
determine the holding position on the selected target on a basis of the first image (S401-S408), and
cause the holder to hold the selected target at the determined holding position (S409).
Regarding Claim 16, Oda discloses in in figure 5
A robot control method for controlling a robot including a holder, the robot control method comprising:
obtaining a first image of at least one holding target (S101);
selecting, from the at least one holding target recognized in the first image, the at least one holding target to be held by the holder as a selected target (S105);
determining a holding position on the selected target on a basis of a second image of the selected target (S108); and
causing the holder to hold the selected target at the determined holding position (S110-S111).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Oda.
Regarding Claim 2, Oda discloses all of the limitations of claim 1 and further teaches in figure 7B and 0079
wherein the controller is configured to
detect a normal line (C1) direction (a1) of a holding surface (F1) of the selected target (W) and further teaches in figure 20 and 0135
determine the holding position (K1) on a basis of the second image captured (I2)
Oda does not teach that the second image be captured from the detected normal line direction. However, Oda teaches determining the angle of the normal line of the workpiece before taking the second image (0106). Oda further teaches in 0102 “When the camera 402 is positioned close to the workpiece W, it is preferable that the center of the image capture area of the camera 402 be aligned with the center of the real-space area corresponding to the search area.” Thus, it would have known to one of ordinary skill in the art to use the method disclosed by Oda to additionally align the second imager in the holding surface normal line direction a1 to determine holding positions. This enables a better hold of the holder, specifically better adherence of a sucking grasper (102).
Claims 3-8 are rejected under 35 U.S.C. 103 as being unpatentable over Oda in view of Ito et al (Japanese Patent application JP 2010006590 hereinafter “Ito”).
Regarding Claim 3, Oda discloses all of the limitations of claim 1 and further discloses
calculate an attitude of the selected target in the second image
In 0070 Oda states “The measurement portion 215 performs the pattern matching process on the image I2 obtained from the camera 402, and thereby obtains three-dimensional information on the position and posture of the workpiece W.” Oda does not teach where to dispose the selected target. However, Ito teaches
determine, on a basis of the attitude, an area where the selected target is to be disposed.
Ito pertains to a robotic article sorting device and details in figures 1 (particularly part 14) and 2 and 0040 how the device places articles W with the surface S1 facing upward in region 24, and places articles W with the surface S2 facing upward in region 22. Thus, it would have been known to one of ordinary skill in the art to employ the sorting method of Ito to determine where to place the workpieces selected and grasped by the device disclosed by Oda.
Regarding Claim 4, Oda discloses all of the limitations of claim 1 and further discloses
calculate an attitude of the selected target in the second image as described in claim 3 above. Oda does not teach where to dispose the selected target. However, Ito teaches
determine, on a basis of the attitude, how to dispose the selected target in 0040 where it is disclosed that the articles W with the surface S1 facing upward are placed in region 24, and the articles W with the surface S2 facing upward are placed in region 22 in parallel and mirrored to the articles in region 24. Thus, it would have been known to one of ordinary skill in the art to employ the sorting method of Ito to determine how to place the workpieces selected and grasped by the device disclosed by Oda.
Regarding Claim 5, Oda in view of Ito discloses all of the limitations of claim 4, and Ito further discloses in 0027 and 0040
dispose the selected target in a first pattern when a front surface of the selected target is shown, and
dispose the selected target in a second pattern when a back surface of the selected target is shown.
In Ito, the work articles W have the same size and shape, each having a first surface S1 that is opposite the second surface S2, or front/back.
Regarding Claim 6, Oda in view of Ito discloses all of the limitations of claim 5, and Ito further discloses in 0040
wherein the second pattern is set such that an orientation of the selected target disposed in the second pattern matches, when the selected target disposed in the second pattern is turned over, an orientation of the selected target disposed in the first pattern.
when it describes the disposal pattern of S1 to be mirrored to the disposal pattern of S2.
Regarding Claim 7, Oda in view of Ito discloses all of the limitations of claim 3, and Ito further discloses in figure 1 and 0040
wherein the controller is configured to control the robot such that the holder disposes the selected target in a first area when a front surface of the selected target is shown and
wherein the controller is configured to control the robot such that the holder disposes the selected target in a second area when a back surface of the selected target is shown.
as described in the rejections of claims 3-6 above.
Regarding Claim 8, Oda discloses all of the limitations of claim 1 and further discloses in figure 7B and 0070 based on an attitude of the selected target in the second image as described in the rejection of claim 3 above. Oda does not teach how to dispose of the selected target. However, Ito teaches in 0040 and figure 1
The controller is configured to determine an amount of correction in a rotational angle based on an attitude of the selected target in the second image at a time when the holder disposes the selected target after holding the selected target.
when detailing how the work article W is disposed – specifically parallel to a previously placed article in front of a line, and mirrored to previously placed articles in a neighboring line. It is understood that in order to achieve this, the rotational angle of the selected work article would be corrected. Thus, it would have been known to one of ordinary skill in the art to employ the method of Oda that determines the attitude of the selected target with a second image with the device of Ito that corrects the rotational angle when disposing the selected target in order to dispose selected targets in a preferred pattern.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Nathan Daniel Neckel whose telephone number is (571)272-9537. The examiner can normally be reached M-F, 7-3.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Wade Miles can be reached at 571-270-7777. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NATHAN DANIEL NECKEL/Examiner, Art Unit 3656
/WADE MILES/Supervisory Patent Examiner, Art Unit 3656