DETAILED ACTION
This is a Final Office Action on the Merits in response to communications filed by applicant on February 27th, 2026. Claims 1-29 are currently pending and examined below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendments to the Claims, filed on February 27th, 2026, have been entered. Claims 1, 7, 11, 19, 21-24, 27, and 28 are currently amended and pending, and claims 2-6, 8-10, 12-18, 20, 25-26, and 29 are original, unamended, and pending. The amendments to the Claims have overcome each and every objection and 35 U.S.C. § 112(d) rejection set forth in the previous Non-Final Office Action mailed December 1st, 2025. The amendments to the Drawings, filed on February 27th, 2026, have overcome each and every objection set forth in the previous Non-Final Office Action mailed December 1st, 2025.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-13, 18-19, 23-26, and 28-29 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 11148295 B2 ("Correll") in view of US 11679508 B2 ("Koga") in further view of JP 2016120545 A ("Miike").
Regarding claim 1, Correll teaches an end effector comprising (Correll: Abstract, “Disclosed are various embodiments of a three-dimensional perception and object manipulation robot gripper configured for connection to and operation in conjunction with a robot arm. In some embodiments, the gripper comprises a palm, a plurality of motors or actuators operably connected to the palm, a mechanical manipulation system operably connected to the palm, a plurality of fingers operably connected to the motors or actuators and configured to manipulate one or more objects located within a workspace or target volume that can be accessed by the fingers. A depth camera system is also operably connected to the palm. One or more computing devices are operably connected to the depth camera and are configured and programmed to process images provided by the depth camera system to determine the location and orientation of the one or more objects within a workspace, and in accordance therewith, provide as outputs therefrom control signals or instructions configured to be employed by the motors or actuators to control movement and operation of the plurality of fingers so as to permit the fingers to manipulate the one or more objects located within the workspace or target volume. The gripper can also be configured to vary controllably at least one of a force, a torque, a stiffness, and a compliance applied by one or more of the plurality of fingers to the one or more objects.”):
a base portion (Correll: Figure 5 upper plate 24 and lower plate 26, Column 10 lines 6-31, “In the embodiment shown in FIG. 5, palm 22 also comprises upper plate 24, to which are mounted or attached, or disposed upon or to, various com ponents of gripper 20 such as upper enclosure 38a, depth camera system 30, third computing device 54, motors 61a and 61b, and motor shafts 62a and 62b. in one embodiment, sandwiched between upper plate 24 and lower plate 26 of palm 22 are at least some of the components comprising mechanical manipulation system 40.”. One of ordinary skill in the art would recognize that the plates function as a base for the end effector.);
an imaging apparatus supported by the base portion (Correll: Figure 5 camera system 30, Column 10 lines 6-31, “In the embodiment shown in FIG. 5, palm 22 also comprises upper plate 24, to which are mounted or attached, or disposed upon or to, various com ponents of gripper 20 such as upper enclosure 38a, depth camera system 30, third computing device 54, motors 61a and 61b, and motor shafts 62a and 62b. in one embodiment, sandwiched between upper plate 24 and lower plate 26 of palm 22 are at least some of the components comprising mechanical manipulation system 40.”. The cited passage clearly shows an imaging apparatus supported by the base.); and
a tool movably supported by the base portion, the tool being configured to support a workpiece and perform work with respect to the workpiece (Correll: Column 9 lines 8-30, “In FIG. 3, fingers 41a and 41b of mechanical manipulation system 40a and 40b are shown disposed in an open position. In some embodiments, fingers 41a and 41b comprise surfaces or pads 42a and 42b, which can be removed and replaced with pads or surfaces 42a and 42b of different stiffnesses or compliance. In FIG. 3, fingers 41a and 41 are configured to engage and grasp objects therebetween, and to permit the manipulation of objects 2 by gripper 20.”),
wherein the base portion includes a driving mechanism (Correll: Figure 5, Column 10 lines 6-31, “In the embodiment shown in FIG. 5, palm 22 also comprises upper plate 24, to which are mounted or attached, or disposed upon or to, various com ponents of gripper 20 such as upper enclosure 38a, depth camera system 30, third computing device 54, motors 61a and 61b, and motor shafts 62a and 62b. in one embodiment, sandwiched between upper plate 24 and lower plate 26 of palm 22 are at least some of the components comprising mechanical manipulation system 40.”. The cited passage clearly shows a driving mechanism supported by the base.).
Correll does not teach wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus.
Koga, in the same field of endeavor, teaches wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus (Koga: Figures 6 and 7, Column 9 lines 45-50, “FIG. 6 is a second flowchart of the first control in the present embodiment. Referring to FIGS. 3 and 6, after the robot controller 4 performs a control operation for moving the workpiece 91 closer to the workpiece 81, at step 115, the operation control unit 43 causes the camera 25 to capture images of the workpieces 81 and 91.”, Column 9 lines 51-60, “FIG. 7 shows an image captured by the camera in order to adjust the position of the second workpiece with respect to the first workpiece. The image 62 includes an image of the upper surface of the projection part 82 which is the first characteristic portion and an image of the upper surface of the projection part 92 which is the second characteristic portion. In the image 62, the second workpiece 91 is displaced from the first workpiece 81 toward the positive side of the u-axis of the screen coordinate system 52 as designated by arrow 101.”. The cited figures and passage clearly shows that the drive device causes the robot gripping the workpiece to move to a position such that the work piece is not overlapping with the imagining object. One of ordinary skill in the art would recognize that the workpiece refers to workpiece 91 in the cited paragraphs and the imaging object is the hole 82a of workpiece 81.).
Correll teaches an end effector comprising: a base portion; an imaging apparatus supported by the base portion; and a tool movably supported by the base portion, the tool being configured to support a workpiece and perform work with respect to the workpiece, wherein the base portion includes a driving mechanism. Correll does not teach wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus. Koga teaches wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities required to have modified the end effector taught in Correll with wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus taught in Koga. Furthermore, the end effector taught in Correll is already configured to use visual servoing methods in the control of the end effector and is also configured to convey and manipulate a workpiece. As such the visual servoing methods taught in Koga could easily be added to Correll using methods known to one of ordinary skill in the art. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of an end effector comprising: wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the end-effector taught in Correll with wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus taught in Koga with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Correll in view of Koga does not teach wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus.
Miike, in the same field of endeavor, teaches wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus (Miike: Abstract, “PROBLEM TO BE SOLVED: To provide a novel handling device that does not need lateral movement of a robot arm after an imaging portion images a workpiece, when the device holds the workpiece with fingers. SOLUTION: A handling device 1 according to the present invention includes: an imaging portion 2 for imaging a workpiece H to obtain geometric information of the workpiece H; a holding portion 6 for securely holding the imaging portion 2, and having a mounting portion 8 to be mounted on a robot arm R; and multiple fingers 10 provided movably with respect to the holding portion 6 through a drive portion 9, for holding the workpiece H. When the device images the workpiece H, the drive portion 9 moves the fingers 10 to the outside of an imaging area A1 of the imaging portion 2. Additionally, when the device holds the workpiece H, the drive portion can move the fingers 10 from the outside of the imaging area A1 of the imaging portion 2 to the inside of the imaging area A1, on the basis of the geometric information of the workpiece H”, ¶ 0010, “The handling device of the present invention moves the fingers outside the imaging area of the imaging unit when imaging the workpiece, but can move the fingers from outside the imaging area to inside the imaging area when holding the workpiece.”, ¶ 0021, “A drive unit 9 and a plurality of fingers 10 movably provided via the drive unit 9 are provided on the lower surface (the surface on the opposite side where the mounting unit 8 is located) of the holding unit 6. The finger 10 of the present embodiment is composed of four of the first finger 11 to the fourth finger 14. Further, claws 11 a to 14 a extending in a direction orthogonal to the axial direction of the fingers are provided at the tips of the first finger 11 to the fourth finger 14. The claws 11 a to 14 a of the present embodiment are provided to extend on both sides of the axis of each finger.”, ¶ 0023, “The first to fourth fingers 11 to 14 are held by the movable elements 20, 21, 23, and 24, respectively, and the first to fourth fingers 11 to 14 are movable in the X and Y directions from the inside to the outside of the imaging area A1 shown in FIG.”. The cited passages clearly shows that the fingers of the robot are configured to move in the X and Y directions, both when gripping the object and when not. As is clearly stated this allows the robot to move the gripped object in and out of the imaging range or the imaging apparatus. This clearly teaches that the robot is configured to move the object in and out of the imaging apparatus without moving the base or the imaging apparatus.).
Correll in view of Koga teaches an end effector comprising: a base portion; an imaging apparatus supported by the base portion; and a tool movably supported by the base portion, the tool being configured to support a workpiece and perform work with respect to the workpiece, wherein the base portion includes a driving mechanism, and wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus. Correll in view of Koga does not teach wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus. Miike teaches wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities to modify the end effector taught in Correll in view of Koga with wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus taught in Miike. Furthermore, the fingers of the end effector taught in Correll in view of Koga are already configured to move inside and outside of the field of view of the imaging device without causing the base or the imaging device to move. Additionally, one of ordinary skill in the art would recognize that the fingers are configured to move and object within the field of view of the imaging device and even partially outside of the field of view of the imaging device. As such, the end effector is readily modifiable to move the workpiece outside of the field of view in addition to the fingers, without moving the base or imaging device, as taught in Miike using known methods. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of an end effector comprising: wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the end effector taught in Correll in view of Koga with wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus taught in Miike with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Regarding claim 2, Correll in view of Koga in further view of Miike teaches wherein, from a state in which at least part of the tool or the supported workpiece is located inside of the range, before imaging the imaging object by the imaging apparatus, the driving mechanism is configured to move the tool supporting the workpiece outside of the range (Koga: Column 9 lines 45-50, “FIG. 6 is a second flowchart of the first control in the present embodiment. Referring to FIGS. 3 and 6, after the robot controller 4 performs a control operation for moving the workpiece 91 closer to the workpiece 81, at step 115, the operation control unit 43 causes the camera 25 to capture images of the workpieces 81 and 91.”, Column 9 lines 51-60, “FIG. 7 shows an image captured by the camera in order to adjust the position of the second workpiece with respect to the first workpiece. The image 62 includes an image of the upper surface of the projection part 82 which is the first characteristic portion and an image of the upper surface of the projection part 92 which is the second characteristic portion. In the image 62, the second workpiece 91 is displaced from the first workpiece 81 toward the positive side of the u-axis of the screen coordinate system 52 as designated by arrow 101.”. The cited passage shows that the robot is configured to move the tool outside of the range prior to capturing an image.).
Regarding claim 3, Correll in view of Koga in further view of Miike teaches wherein the imaging object is a work object on which the work is performed by the tool (Koga: Column 4 lines 27-34, “In the first robot device 5 of the present embodiment, the robot 1 attaches the workpiece 91 to the workpiece 81 while the conveyor 75 continues to convey the workpiece 81. In other words, the workpiece 81 is moved by the conveyor 75 during an operation for attaching the workpiece 91. The robot 1 attaches the workpiece 91 to the workpiece 81 while changing its position and the orientation so as to follow the workpiece 81.”), and
wherein, from a state in which the tool supporting the workpiece is located outside of the range, before performing the work on the work object by the tool, the driving mechanism is configured to move the tool supporting the workpiece such that at least part of the workpiece overlaps with the work object in the field of view of the imaging apparatus (Koga: Column 4 line 55 – Column 5 line 2, “In the first control of the present embodiment, the pins 92a and 93a are controlled so as to be aligned with the holes 82a and 83a. The orientation of the workpiece 91 with respect to the workpiece 81 is adjusted before the first control is performed. In other words, the orientation of the robot 1 is adjusted so that the pins 92a and 93a of the workpiece 91 are arranged on a straight line extending in the vertical direction. Thus, in the first control, the control for arranging the pin 92a directly above the hole 82a is performed so that the pins 92a and 93a can be aligned with the holes 82a and 83a. After the first control of the present embodiment, a downward movement of the workpiece 91 in the vertical direction indicated by arrow 103 causes the pins 92a, 93a to be inserted into the holes 82a, 83a, and causes the workpiece 91 to be attached to the workpiece 81.”, Column 7 lines 57-67, “At step 111, a reference image for performing the first control is generated. FIG. 5 shows a reference image for performing the first control of the present embodiment. The reference image 61 corresponds to an image captured by the camera 25 when the second workpiece 91 is arranged at the target position with respect to the first workpiece 81. In the present embodiment, the reference image 61 is an image captured by the camera 25 when the pins 92a and 93a of the workpiece 91 are arranged directly above the holes 82a and 83a of the workpiece 81. The reference image 61 can be prepared by an operator and stored in the storage part 42.”, Column 10 lines 48-58, “Subsequently, the command generation unit 34 of the image processing unit 31 generates a movement command for the robot 1 so that the second workpiece 91 is arranged at the target position with respect to the first workpiece 81, based on the relative position amount in the image 62 captured by the camera 25 and the relative position amount in the reference image 61. The command generation unit 34 of the present embodiment generates a movement command for operating the robot 1 so that the relative position amount in the image 62 captured by the camera 25 approaches the relative position amount in the reference image 61.”. The cited passages clearly show that the system is configured to align the workpieces before performing an attachment operation on said workpieces.).
Regarding claim 4, Correll in view of Koga in further view of Miike teaches wherein, in a case of capturing the image of the imaging object by the imaging apparatus, the driving mechanism is configured to position the tool and the supported workpiece outside of the field of view of the imaging apparatus (Correll: Column 18 line 59 – Column 19 line 28, “FIGS. 6a and 6b show top views of the embodiment of gripper 20 shown in FIGS. 3 through 5, where mechanical manipulation system 40 is shown in two different configurations next to workspace or target volume 74. In FIG. 6a, mechanical manipulation system 40 is shown with fingers 41a and 41b in a closed position, and fingers 41a and 41b located adjacent to and just behind target volume 74. In FIG. 6b, mechanical manipulation system 40 is shown with fingers 41a and 41b in an open position, with fingers 41a and 41b located just outside the perimeter of field of view 71.”, Miike: Abstract, “PROBLEM TO BE SOLVED: To provide a novel handling device that does not need lateral movement of a robot arm after an imaging portion images a workpiece, when the device holds the workpiece with fingers. SOLUTION: A handling device 1 according to the present invention includes: an imaging portion 2 for imaging a workpiece H to obtain geometric information of the workpiece H; a holding portion 6 for securely holding the imaging portion 2, and having a mounting portion 8 to be mounted on a robot arm R; and multiple fingers 10 provided movably with respect to the holding portion 6 through a drive portion 9, for holding the workpiece H. When the device images the workpiece H, the drive portion 9 moves the fingers 10 to the outside of an imaging area A1 of the imaging portion 2. Additionally, when the device holds the workpiece H, the drive portion can move the fingers 10 from the outside of the imaging area A1 of the imaging portion 2 to the inside of the imaging area A1, on the basis of the geometric information of the workpiece H”, ¶ 0010, “The handling device of the present invention moves the fingers outside the imaging area of the imaging unit when imaging the workpiece, but can move the fingers from outside the imaging area to inside the imaging area when holding the workpiece.”, ¶ 0023, “The first to fourth fingers 11 to 14 are held by the movable elements 20, 21, 23, and 24, respectively, and the first to fourth fingers 11 to 14 are movable in the X and Y directions from the inside to the outside of the imaging area A1 shown in FIG.”. The cited passages clearly teaches that the robot can move the tool and workpiece inside and outside of the imagine area (i.e. the field of view of the imaging device).).
Regarding claim 5, Correll in view of Koga in further view of Miike teaches wherein the imaging object is a contact position at which the workpiece supported by the tool is brought into contact in the work (Koga: Column 4 lines 27-34, “In the first robot device 5 of the present embodiment, the robot 1 attaches the workpiece 91 to the workpiece 81 while the conveyor 75 continues to convey the workpiece 81. In other words, the workpiece 81 is moved by the conveyor 75 during an operation for attaching the workpiece 91. The robot 1 attaches the workpiece 91 to the workpiece 81 while changing its position and the orientation so as to follow the workpiece 81.”, Column 9 lines 51-60, “FIG. 7 shows an image captured by the camera in order to adjust the position of the second workpiece with respect to the first workpiece. The image 62 includes an image of the upper surface of the projection part 82 which is the first characteristic portion and an image of the upper surface of the projection part 92 which is the second characteristic portion. In the image 62, the second workpiece 91 is displaced from the first workpiece 81 toward the positive side of the u-axis of the screen coordinate system 52 as designated by arrow 101.”. The cited passages clearly shows that the imagining object is projection part 82 with hole 82a. One of ordinary skill in the art would recognize that, because the operation being performed is an attachment operation in which pin 92a is inserted into hole 82a of the projection part 82, the imaging object is a contact position at which the workpiece supported by the tool is brought into contact.).
Regarding claim 6, Correll in view of Koga in further view of Miike teaches wherein the tool includes a first finger portion and a second finger portion that are configured to hold and support the workpiece (Correll: Column 9 lines 8-30, “In FIG. 3, fingers 41a and 41b of mechanical manipulation system 40a and 40b are shown disposed in an open position. In some embodiments, fingers 41a and 41b comprise surfaces or pads 42a and 42b, which can be removed and replaced with pads or surfaces 42a and 42b of different stiffnesses or compliance. In FIG. 3, fingers 41a and 41 are configured to engage and grasp objects therebetween, and to permit the manipulation of objects 2 by gripper 20.”).
Regarding claim 7, Correll in view of Koga in further view of Miike teaches wherein the first finger portion includes a first base portion that is movably supported with respect to the base portion (Correll: Figure 3 and 5, Column 9 lines 8-30, “In FIG. 3, fingers 41a and 41 are configured to engage and grasp objects therebetween, and to permit the manipulation of objects 2 by gripper 20. Links 42a and 42b (not shown in FIG. 3), 43a and 44b, and 45a and 45b of mechanical manipulation system 40 are configured to open and close fingers 41a and 41b in conjunction with motors or actuators 6la and 61b (not shown in FIG. 3).”, Column 10 lines 46-61, “In the embodiment of gripper 20 shown in FIGS. 3-5, mechanical manipulation system 40 comprises four-bar link ages 43, 44 and 45 connecting fingers 41 to internal actuators or motors 61 and motor or actuator shafts 62 housed on and between plates 24 and 26, leaving free space or volume 28 therebetween for ventilation.”. The cited passage and image clearly show that the first finger has base portion.),
and a first fingertip portion that is arranged further distal than the first base portion and is configured to hold the workpiece (Correll: Figure 3, Column 9 lines 8-30, “In FIG. 3, fingers 41a and 41b of mechanical manipulation system 40a and 40b are shown disposed in an open position. In some embodiments, fingers 41a and 41b comprise surfaces or pads 42a and 42b, which can be removed and replaced with pads or surfaces 42a and 42b of different stiffnesses or compliance. In FIG. 3, fingers 41a and 41 are configured to engage and grasp objects therebetween, and to permit the manipulation of objects 2 by gripper 20.”, The cited passage and image clearly show that the first finger has a fingertip portion further distal than the base portion.), and
wherein the second finger portion includes a second base portion that is movably supported with respect to the base portion (Correll: Figure 3 and 5, Column 9 lines 8-30, “In FIG. 3, fingers 41a and 41 are configured to engage and grasp objects therebetween, and to permit the manipulation of objects 2 by gripper 20. Links 42a and 42b (not shown in FIG. 3), 43a and 44b, and 45a and 45b of mechanical manipulation system 40 are configured to open and close fingers 41a and 41b in conjunction with motors or actuators 6la and 61b (not shown in FIG. 3).”, Column 10 lines 46-61, “In the embodiment of gripper 20 shown in FIGS. 3-5, mechanical manipulation system 40 comprises four-bar link ages 43, 44 and 45 connecting fingers 41 to internal actuators or motors 61 and motor or actuator shafts 62 housed on and between plates 24 and 26, leaving free space or volume 28 therebetween for ventilation.”. The cited passage and image clearly show that the first finger has base portion.),
and a second fingertip portion that is arranged further distal than the second base portion and is configured to hold the workpiece (Correll: Figure 3, Column 9 lines 8-30, “In FIG. 3, fingers 41a and 41b of mechanical manipulation system 40a and 40b are shown disposed in an open position. In some embodiments, fingers 41a and 41b comprise surfaces or pads 42a and 42b, which can be removed and replaced with pads or surfaces 42a and 42b of different stiffnesses or compliance. In FIG. 3, fingers 41a and 41 are configured to engage and grasp objects therebetween, and to permit the manipulation of objects 2 by gripper 20.”, The cited passage and image clearly show that the second finger has a fingertip portion further distal than the base portion.).
Regarding claim 8, Correll in view of Koga in further view of Miike teaches wherein the first finger portion and the second finger portion are formed such that, in a case of being moved with respect to the base portion by the driving mechanism, the first fingertip portion and the second fingertip portion are configured to move inside and outside of the range (Correll: Figure 4, 6a-b, 9C, Column 9 lines 8-30, “Upper and lower housings or enclosures 38a and 38b protect and shield various internally-disposed components of gripper 20, such as palm 22, motors 61a and 61b, and other components of gripper 20 that are not visible in FIG. 3. With respect to FIG. 3, FIG. 4 shows gripper 20 with finger 41b having moved independently of Finger 41a into a new position.”, Column 18 line 59 – Column 19 line 28, “FIGS. 6a and 6b show top views of the embodiment of gripper 20 shown in FIGS. 3 through 5, where mechanical manipulation system 40 is shown in two different configurations next to workspace or target volume 74. In FIG. 6a, mechanical manipulation system 40 is shown with fingers 41a and 41b in a closed position, and fingers 41a and 41b located adjacent to and just behind target volume 74. In FIG. 6b, mechanical manipulation system 40 is shown with fingers 41a and 41b in an open position, with fingers 41a and 41b located just outside the perimeter of field of view 71.”, Column 22 lines 11-34, “Gripper 20 is shown approaching object or strawberry 2 in FIG. 9a, which has been located and detected by gripper 20's depth camera system 30. One finger 41a runs into obstacle 9, however, as shown in FIG. 9b. As fingers 41a and 41b are independently controlled, one of the fingers, 41b, keeps moving, eventually pushing strawberry 2 until it touches the other finger, 41a (FIG. 9c).”. One of ordinary skill in the art would see from the cited passages that, because the fingers are independently driven, are able to move in and out of the range.)
and, even in a case where the first fingertip portion and the second fingertip portion are located inside of the range, the first base portion and the second base portion are located outside of the range (Correll: Figure 3 and 5, Column 9 lines 8-30, “In FIG. 3, fingers 41a and 41 are configured to engage and grasp objects therebetween, and to permit the manipulation of objects 2 by gripper 20. Links 42a and 42b (not shown in FIG. 3), 43a and 44b, and 45a and 45b of mechanical manipulation system 40 are configured to open and close fingers 41a and 41b in conjunction with motors or actuators 6la and 61b (not shown in FIG. 3).”, Column 10 lines 46-61, “In the embodiment of gripper 20 shown in FIGS. 3-5, mechanical manipulation system 40 comprises four-bar link ages 43, 44 and 45 connecting fingers 41 to internal actuators or motors 61 and motor or actuator shafts 62 housed on and between plates 24 and 26, leaving free space or volume 28 therebetween for ventilation.”. One of ordinary skill in the art would recognize based on the position of the base of the fingers and the cameras, that the base will always be outside of the range.).
Regarding claim 9, Correll in view of Koga in further view of Miike teaches wherein, in a state of being located inside of the range, the first fingertip portion and the second fingertip portion are located between the imaging apparatus and a focal point of the imaging apparatus (Correll: Figures 3 and 6a-b, Column 9 lines 8-30, “FIG. 3 shows one embodiment of gripper 20 operably attached to robotic wrist 14. In one embodiment, and in conjunction with light projector 36, camera sensors 32 and 34 are configured to provide stereo-vision capabilities for gripper 20, more about which is said below. In FIG. 3, fingers 41a and 41b of mechanical manipulation system 40a and 40b are shown disposed in an open position. In some embodiments, fingers 41a and 41b comprise surfaces or pads 42a and 42b, which can be removed and replaced with pads or surfaces 42a and 42b of different stiffnesses or compliance. In FIG. 3, fingers 41a and 41 are configured to engage and grasp objects therebetween, and to permit the manipulation of objects 2 by gripper 20.”, Column 18 line 59 – Column 19 line 28, “FIGS. 6a and 6b show top views of the embodiment of gripper 20 shown in FIGS. 3 through 5, where mechanical manipulation system 40 is shown in two different configurations next to workspace or target volume 74. In FIG. 6a, mechanical manipulation system 40 is shown with fingers 41a and 41b in a closed position, and fingers 41a and 41b located adjacent to and just behind target volume 74. In FIG. 6b, mechanical manipulation system 40 is shown with fingers 41a and 41b in an open position, with fingers 41a and 41b located just outside the perimeter of field of view 71.”. One of ordinary skill in the art would see from the cited passages and figures, that the fingertips of the end effector are located between the imaging apparatus and a focal point of the imaging apparatus.).
Regarding claim 10, Correll in view of Koga in further view of Miike in further view of Miike teaches wherein the driving mechanism includes a slide portion that is configured to support the first base portion and the second base portion in a slidingly movable manner (Miike: Figure 3b, ¶ 0023, “Specifically, as shown in FIG. 3(b), two stators 15, 16 extending along the X direction are provided on both outer sides of the opening 7 on the underside of the holding part 6, and two movers 17, 18 are provided so as to be slidable relative to the stators 15, 16, with the stators 15, 16 and the movers 17 constituting a first X-direction drive unit, and the stators 15, 16 and the movers 18 constituting a second X-direction drive unit. The movable element 17 is provided with one stator 19 extending along the Y direction and two movable elements 20 and 21 that are slidably arranged relative to the stator 19, with the stator 19 and movable element 20 constituting a first Y direction drive unit and the stator 19 and movable element 21 constituting a second Y direction drive unit. The movable element 18 is provided with one stator 22 extending along the Y direction and two movable elements 23 and 24 that are slidably arranged relative to the stator 22, with the stator 22 and movable element 23 constituting a third Y-direction drive unit and the stator 22 and movable element 24 constituting a fourth Y-direction drive unit.”. The cited passage and figure clearly shows that the end effector has a slide portion for the fingers on the left side and a slide portion for the fingers on the right side. Furthermore, the base of each finger is clearly coupled to the slide portions.),
a first drive unit that is configured to drive the first base portion to slidingly move in the slide portion, and a second drive unit that is configured to be driven independently from the first drive unit and is configured to drive the second base portion to slidingly move in the slide portion (Figure 3b, ¶ 0023, “Specifically, as shown in FIG. 3(b), two stators 15, 16 extending along the X direction are provided on both outer sides of the opening 7 on the underside of the holding part 6, and two movers 17, 18 are provided so as to be slidable relative to the stators 15, 16, with the stators 15, 16 and the movers 17 constituting a first X-direction drive unit, and the stators 15, 16 and the movers 18 constituting a second X-direction drive unit. The movable element 17 is provided with one stator 19 extending along the Y direction and two movable elements 20 and 21 that are slidably arranged relative to the stator 19, with the stator 19 and movable element 20 constituting a first Y direction drive unit and the stator 19 and movable element 21 constituting a second Y direction drive unit. The movable element 18 is provided with one stator 22 extending along the Y direction and two movable elements 23 and 24 that are slidably arranged relative to the stator 22, with the stator 22 and movable element 23 constituting a third Y-direction drive unit and the stator 22 and movable element 24 constituting a fourth Y-direction drive unit.”. The cited passage and figure clearly shows that the end effector has a separate drive unit for each side configured to drive the fingers in a sliding motion.).
Regarding claim 11, Correll teaches a robot system comprising (Correll: Abstract, “Disclosed are various embodiments of a three-dimensional perception and object manipulation robot gripper configured for connection to and operation in conjunction with a robot arm. In some embodiments, the gripper comprises a palm, a plurality of motors or actuators operably connected to the palm, a mechanical manipulation system operably connected to the palm, a plurality of fingers operably connected to the motors or actuators and configured to manipulate one or more objects located within a workspace or target volume that can be accessed by the fingers. A depth camera system is also operably connected to the palm. One or more computing devices are operably connected to the depth camera and are configured and programmed to process images provided by the depth camera system to determine the location and orientation of the one or more objects within a workspace, and in accordance therewith, provide as outputs therefrom control signals or instructions configured to be employed by the motors or actuators to control movement and operation of the plurality of fingers so as to permit the fingers to manipulate the one or more objects located within the workspace or target volume. The gripper can also be configured to vary controllably at least one of a force, a torque, a stiffness, and a compliance applied by one or more of the plurality of fingers to the one or more objects.”):
an end effector including a base portion (Correll: Figure 5 upper plate 24 and lower plate 26, Column 10 lines 6-31, “In the embodiment shown in FIG. 5, palm 22 also comprises upper plate 24, to which are mounted or attached, or disposed upon or to, various com ponents of gripper 20 such as upper enclosure 38a, depth camera system 30, third computing device 54, motors 61a and 61b, and motor shafts 62a and 62b. in one embodiment, sandwiched between upper plate 24 and lower plate 26 of palm 22 are at least some of the components comprising mechanical manipulation system 40.”. One of ordinary skill in the art would recognize that the plates function as a base for the end effector.);
an imaging apparatus that is supported by the base portion and is configured to capture an image (Correll: Figure 5 camera system 30, Column 10 lines 6-31, “In the embodiment shown in FIG. 5, palm 22 also comprises upper plate 24, to which are mounted or attached, or disposed upon or to, various com ponents of gripper 20 such as upper enclosure 38a, depth camera system 30, third computing device 54, motors 61a and 61b, and motor shafts 62a and 62b. in one embodiment, sandwiched between upper plate 24 and lower plate 26 of palm 22 are at least some of the components comprising mechanical manipulation system 40.”. The cited passage clearly shows an imaging apparatus supported by the base.),
and a tool that is movably supported by the base portion and is configured to perform work with respect to a workpiece (Correll: Column 9 lines 8-30, “In FIG. 3, fingers 41a and 41b of mechanical manipulation system 40a and 40b are shown disposed in an open position. In some embodiments, fingers 41a and 41b comprise surfaces or pads 42a and 42b, which can be removed and replaced with pads or surfaces 42a and 42b of different stiffnesses or compliance. In FIG. 3, fingers 41a and 41 are configured to engage and grasp objects therebetween, and to permit the manipulation of objects 2 by gripper 20.”),
a robot to which the end effector is attached, the robot being configured to move a position and posture of the end effector (Correll: Column 8 lines 4-23, “FIG. 1 shows robot 10 engaged in a "bin picking" task, which involves retrieving items such as screws from bin 25. Here, robot 10 is shown in a configuration that puts gripper 20 comprising cameras 32 and 34 and a light illumination device 36 (not shown in FIG. 1) above bin 4, which in the illustrative embodiment of FIG. 1 features a workspace or target volume within which object(s) are situated or located, 10 where the workspace or target volume has an extent or distance associated therewith which ranges between about 11 cm and about 30 cm from the forward or sensing portion of gripper 20.”, Column 27 lines 52-58, “FIG. 14 shows one embodiment of a method 200 that may be employed using gripper 20. According to one embodiment, method 200 comprises step 201 of determining the position and characteristics of object 2, step 203 of actuating and controlling one or more of any of motors 41, mechanical manipulation system 40, wrist 14, arm 12, and robot 10, and step 205 of grasping, manipulating, and/or moving object 2.”. The cited passages clearly show that the end effector is attached to a robot and wherein the robot is configured to move the end effector.); and
a control unit configured to control the robot and the end effector (Correll: Column 3 line 25 – Column 4 line 17, “… a second computing device mounted to, on or in, or connected operably to, the palm, the second computing device being operably connected to the first computing device and to the plurality of at least one of motors and actuators, the second computing device being configured to receive the output images provided by the first computing device, and further being configured and programmed to process the output images to determine the location and orientation of the one or more objects within the workspace, and in accordance therewith provide as outputs therefrom control signals or instructions configured to be employed by the motors or actuators to control movement and operation of the plurality of fingers so as to permit the fingers to manipulate the one or more objects located within the workspace or target volume,”, Column 9 line 48 – Column 10 line 5, “Further as shown in FIG. 5, in one embodiment, second computing device 52 comprises a processor, controller, microcontroller, and/or CPU (such as a system-on-a-module) attached to the underside of palm 22, while peripheral break-out device 51 comprising, for example, Ethernet and USB connections, is likewise attached to the underside of palm 22.”).
Correll does not teach wherein, in a case of capturing the image of an imaging object by the imaging apparatus, the control unit is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus.
Koga, in the same field of endeavor, teaches wherein, in a case of capturing the image of an imaging object by the imaging apparatus, the control unit is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus (Koga: Figures 6 and 7, Column 9 lines 45-50, “FIG. 6 is a second flowchart of the first control in the present embodiment. Referring to FIGS. 3 and 6, after the robot controller 4 performs a control operation for moving the workpiece 91 closer to the workpiece 81, at step 115, the operation control unit 43 causes the camera 25 to capture images of the workpieces 81 and 91.”, Column 9 lines 51-60, “FIG. 7 shows an image captured by the camera in order to adjust the position of the second workpiece with respect to the first workpiece. The image 62 includes an image of the upper surface of the projection part 82 which is the first characteristic portion and an image of the upper surface of the projection part 92 which is the second characteristic portion. In the image 62, the second workpiece 91 is displaced from the first workpiece 81 toward the positive side of the u-axis of the screen coordinate system 52 as designated by arrow 101.”. The cited figures and passage clearly shows that the drive device causes the robot gripping the workpiece to move to a position such that the work piece is not overlapping with the imagining object. One of ordinary skill in the art would recognize that the workpiece refers to workpiece 91 in the cited paragraphs and the imaging object is the hole 82a of workpiece 81.).
Correll teaches a robot system comprising: an end effector including a base portion, an imaging apparatus that is supported by the base portion and is configured to capture an image, and a tool that is movably supported by the base portion and is configured to perform work with respect to a workpiece; a robot to which the end effector is attached, the robot being configured to move a position and posture of the end effector; and a control unit configured to control the robot and the end effector. Correll does not teach wherein, in a case of capturing the image of an imaging object by the imaging apparatus, the control unit is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus. Koga teaches wherein, in a case of capturing the image of an imaging object by the imaging apparatus, the control unit is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities required to have modified the robot system taught in Correll with wherein, in a case of capturing the image of an imaging object by the imaging apparatus, the control unit is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus taught in Koga. Furthermore, the robot system taught in Correll is already configured to use visual servoing methods in the control of the end effector and is also configured to convey and manipulate a workpiece. As such the visual servoing methods taught in Koga could easily be added to Correll using methods known to one of ordinary skill in the art. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of an robot system comprising: wherein, in a case of capturing the image of an imaging object by the imaging apparatus, the control unit is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot system taught in Correll with wherein, in a case of capturing the image of an imaging object by the imaging apparatus, the control unit is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus taught in Koga with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Correll in view of Koga does not teach wherein, in a case of capturing the image of an imaging object by the imaging apparatus, the control unit is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus.
Correll in view of Koga does not teach wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus.
Miike, in the same field of endeavor, teaches wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus (Miike: Abstract, “PROBLEM TO BE SOLVED: To provide a novel handling device that does not need lateral movement of a robot arm after an imaging portion images a workpiece, when the device holds the workpiece with fingers. SOLUTION: A handling device 1 according to the present invention includes: an imaging portion 2 for imaging a workpiece H to obtain geometric information of the workpiece H; a holding portion 6 for securely holding the imaging portion 2, and having a mounting portion 8 to be mounted on a robot arm R; and multiple fingers 10 provided movably with respect to the holding portion 6 through a drive portion 9, for holding the workpiece H. When the device images the workpiece H, the drive portion 9 moves the fingers 10 to the outside of an imaging area A1 of the imaging portion 2. Additionally, when the device holds the workpiece H, the drive portion can move the fingers 10 from the outside of the imaging area A1 of the imaging portion 2 to the inside of the imaging area A1, on the basis of the geometric information of the workpiece H”, ¶ 0010, “The handling device of the present invention moves the fingers outside the imaging area of the imaging unit when imaging the workpiece, but can move the fingers from outside the imaging area to inside the imaging area when holding the workpiece.”, ¶ 0021, “A drive unit 9 and a plurality of fingers 10 movably provided via the drive unit 9 are provided on the lower surface (the surface on the opposite side where the mounting unit 8 is located) of the holding unit 6. The finger 10 of the present embodiment is composed of four of the first finger 11 to the fourth finger 14. Further, claws 11 a to 14 a extending in a direction orthogonal to the axial direction of the fingers are provided at the tips of the first finger 11 to the fourth finger 14. The claws 11 a to 14 a of the present embodiment are provided to extend on both sides of the axis of each finger.”, ¶ 0023, “The first to fourth fingers 11 to 14 are held by the movable elements 20, 21, 23, and 24, respectively, and the first to fourth fingers 11 to 14 are movable in the X and Y directions from the inside to the outside of the imaging area A1 shown in FIG.”. The cited passages clearly shows that the fingers of the robot are configured to move in the X and Y directions, both when gripping the object and when not. As is clearly stated this allows the robot to move the gripped object in and out of the imaging range or the imaging apparatus. This clearly teaches that the robot is configured to move the object in and out of the imaging apparatus without moving the base or the imaging apparatus.).
Correll in view of Koga a robot system comprising: an end effector including a base portion, an imaging apparatus that is supported by the base portion and is configured to capture an image, and a tool that is movably supported by the base portion and is configured to perform work with respect to a workpiece; a robot to which the end effector is attached, the robot being configured to move a position and posture of the end effector; and a control unit configured to control the robot and the end effector, and wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus. Correll in view of Koga does not teach wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus. Miike teaches wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities to modify the end effector taught in Correll in view of Koga with wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus taught in Miike. Furthermore, the fingers of the end effector taught in Correll in view of Koga are already configured to move inside and outside of the field of view of the imaging device without causing the base or the imaging device to move. Additionally, one of ordinary skill in the art would recognize that the fingers are configured to move and object within the field of view of the imaging device and even partially outside of the field of view of the imaging device. As such, the end effector is readily modifiable to move the workpiece outside of the field of view in addition to the fingers, without moving the base or imaging device, as taught in Miike using known methods. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of an robot system comprising: wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot system taught in Correll in view of Koga with wherein, in a case of capturing an image of an imaging object by the imaging apparatus, the driving mechanism is configured to position the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus taught in Miike with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Regarding claim 12, Correll in view of Koga in further view of Miike teaches wherein the control unit is configured to perform alignment control to align the end effector with respect to the imaging object based on the image captured by the imaging apparatus (Koga: Column 4 line 55 – Column 5 line 2, “In the first control of the present embodiment, the pins 92a and 93a are controlled so as to be aligned with the holes 82a and 83a. The orientation of the workpiece 91 with respect to the workpiece 81 is adjusted before the first control is performed. In other words, the orientation of the robot 1 is adjusted so that the pins 92a and 93a of the workpiece 91 are arranged on a straight line extending in the vertical direction. Thus, in the first control, the control for arranging the pin 92a directly above the hole 82a is performed so that the pins 92a and 93a can be aligned with the holes 82a and 83a. After the first control of the present embodiment, a downward movement of the workpiece 91 in the vertical direction indicated by arrow 103 causes the pins 92a, 93a to be inserted into the holes 82a, 83a, and causes the workpiece 91 to be attached to the workpiece 81.”, Column 7 lines 57-67, “At step 111, a reference image for performing the first control is generated. FIG. 5 shows a reference image for performing the first control of the present embodiment. The reference image 61 corresponds to an image captured by the camera 25 when the second workpiece 91 is arranged at the target position with respect to the first workpiece 81. In the present embodiment, the reference image 61 is an image captured by the camera 25 when the pins 92a and 93a of the workpiece 91 are arranged directly above the holes 82a and 83a of the workpiece 81. The reference image 61 can be prepared by an operator and stored in the storage part 42.”, Column 10 lines 48-58, “Subsequently, the command generation unit 34 of the image processing unit 31 generates a movement command for the robot 1 so that the second workpiece 91 is arranged at the target position with respect to the first workpiece 81, based on the relative position amount in the image 62 captured by the camera 25 and the relative position amount in the reference image 61. The command generation unit 34 of the present embodiment generates a movement command for operating the robot 1 so that the relative position amount in the image 62 captured by the camera 25 approaches the relative position amount in the reference image 61.”. The cited passages clearly show that the system is configured to align the workpieces before performing an attachment operation on said workpieces.).
Regarding claim 13, Correll in view of Koga in further view of Miike teaches wherein, as the alignment control, by calculating a misalignment amount between the end effector and the imaging object based on the image captured by the imaging apparatus and a target image that serves as a target (Column 4 line 55 – Column 5 line 2, “In the first control of the present embodiment, the pins 92a and 93a are controlled so as to be aligned with the holes 82a and 83a. The orientation of the workpiece 91 with respect to the workpiece 81 is adjusted before the first control is performed. In other words, the orientation of the robot 1 is adjusted so that the pins 92a and 93a of the workpiece 91 are arranged on a straight line extending in the vertical direction. Thus, in the first control, the control for arranging the pin 92a directly above the hole 82a is performed so that the pins 92a and 93a can be aligned with the holes 82a and 83a. After the first control of the present embodiment, a downward movement of the workpiece 91 in the vertical direction indicated by arrow 103 causes the pins 92a, 93a to be inserted into the holes 82a, 83a, and causes the workpiece 91 to be attached to the workpiece 81.”, Column 7 lines 57-67, “At step 111, a reference image for performing the first control is generated. FIG. 5 shows a reference image for performing the first control of the present embodiment. The reference image 61 corresponds to an image captured by the camera 25 when the second workpiece 91 is arranged at the target position with respect to the first workpiece 81. In the present embodiment, the reference image 61 is an image captured by the camera 25 when the pins 92a and 93a of the workpiece 91 are arranged directly above the holes 82a and 83a of the workpiece 81. The reference image 61 can be prepared by an operator and stored in the storage part 42.”),
the control unit is configured to perform correction control to correct a position of the end effector to eliminate the misalignment amount (Koga: Column 10 line 59 – Column 11 line 7, “At step 119, the command generation unit 34 calculates the difference between the relative position amounts, which is the difference between the relative position amount in the image 62 captured by the camera 25 and the relative position amount in the reference image 61. In the present embodiment, the command generation unit 34 calculates the difference between the relative position amounts by subtracting the relative position amount in the reference image 61 from the relative position amount in the image 62 captured by the camera 25. The difference between the relative position amounts can be represented by [(ulm-u2m)-(ulb-u2b), (vlm-v2m)-(vlb-v2b)] as a value for each of the u-axis and the v-axis. As described above, in the present embodiment, the difference between the relative position amounts related to the u-axis and the difference between the relative position amounts related to the v-axis are calculated.”, Column 11 line 34-38, “At step 120, when the difference between the relative position amounts deviates from the determination range, the control proceeds to step 121.”, Column 7 lines 39-59, “At step 121, the command generation unit 34 sets a driving method for the robot 1 based on the difference between the relative position amounts. The command generation unit 34 sets a movement direction and a movement amount of the position of the robot 1 in the reference coordinate system 51.”, Column 7 line 60 – Column 8 line 7, “Furthermore, a method for calculating the movement amount of the position of the robot 1 with respect to the difference between the relative position amounts is previously determined. For example, as the movement amount of the position of the robot 1 in a direction corresponding to the u-axis, a value obtained by multiplying a value related to the u-axis ((ulm-u2m)-(ulb-u2b)) by a predetermined coefficient can be adopted. Further, as the movement amount of the position of the robot 1 in a direction corresponding to the v-axis, a value obtained by multiplying a value related to the v-axis ((vlm-v2m)-(vl b-v2b )) by a predetermined coefficient can be adopted. In this way, the movement amount of the position of the robot 1 can be calculated in the direction corresponding to each axis of the screen coordinate system 52.”, Column 12 lines 27-35, “Subsequently, at step 122, the robot 1 is driven based on the movement direction and movement amount of the position of the robot 1. The command generation unit 34 generates a movement command for driving the robot 1 based on the movement direction and the movement amount of the position of the robot 1. The command generation unit 34 transmits a movement command to the operation control unit 43. The operation control unit 43 causes the robot 1 to change its position based on the movement command.”).
Regarding claim 18, Correll in view of Koga in further view of Miike teaches wherein the imaging object is a work position at which the tool performs the work with respect to the workpiece (Koga: Column 4 lines 27-34, “In the first robot device 5 of the present embodiment, the robot 1 attaches the workpiece 91 to the workpiece 81 while the conveyor 75 continues to convey the workpiece 81. In other words, the workpiece 81 is moved by the conveyor 75 during an operation for attaching the workpiece 91. The robot 1 attaches the workpiece 91 to the workpiece 81 while changing its position and the orientation so as to follow the workpiece 81.”, Column 4 line 55 – Column 5 line 2, “In the first control of the present embodiment, the pins 92a and 93a are controlled so as to be aligned with the holes 82a and 83a. The orientation of the workpiece 91 with respect to the workpiece 81 is adjusted before the first control is performed. In other words, the orientation of the robot 1 is adjusted so that the pins 92a and 93a of the workpiece 91 are arranged on a straight line extending in the vertical direction. Thus, in the first control, the control for arranging the pin 92a directly above the hole 82a is performed so that the pins 92a and 93a can be aligned with the holes 82a and 83a. After the first control of the present embodiment, a downward movement of the workpiece 91 in the vertical direction indicated by arrow 103 causes the pins 92a, 93a to be inserted into the holes 82a, 83a, and causes the workpiece 91 to be attached to the workpiece 81.”. The cited passages clearly teach that the imaging object is a work position (i.e. the location of the holes 82a and 83a at which the tool performs work (i.e. an insertion of the pins 92a and 93a of the workpiece into holes 82a and 83a.).), and
wherein, in a case where there are a plurality of work positions, a same target image is used in the alignment control performed with respect to each of the plurality of work positions (Koga: Column 7 lines 57-67, “At step 111, a reference image for performing the first control is generated. FIG. 5 shows a reference image for performing the first control of the present embodiment. The reference image 61 corresponds to an image captured by the camera 25 when the second workpiece 91 is arranged at the target position with respect to the first workpiece 81. In the present embodiment, the reference image 61 is an image captured by the camera 25 when the pins 92a and 93a of the workpiece 91 are arranged directly above the holes 82a and 83a of the workpiece 81. The reference image 61 can be prepared by an operator and stored in the storage part 42.”. The cited passage teaches that a reference image is generated using an image of only one of the holes in the insertion task, and this reference image is used in the control of the insertion for all holes).
Regarding claim 19, Correll in view of Koga in further view of Miike teaches wherein the work performed at each of the plurality of work positions includes a first operation to cause the tool to support the workpiece that is located at least one of the plurality of work positions (Koga: Column 4 line 55 – Column 5 line 2, “In the first control of the present embodiment, the pins 92a and 93a are controlled so as to be aligned with the holes 82a and 83a. The orientation of the workpiece 91 with respect to the workpiece 81 is adjusted before the first control is performed. In other words, the orientation of the robot 1 is adjusted so that the pins 92a and 93a of the workpiece 91 are arranged on a straight line extending in the vertical direction. Thus, in the first control, the control for arranging the pin 92a directly above the hole 82a is performed so that the pins 92a and 93a can be aligned with the holes 82a and 83a. After the first control of the present embodiment, a downward movement of the workpiece 91 in the vertical direction indicated by arrow 103 causes the pins 92a, 93a to be inserted into the holes 82a, 83a, and causes the workpiece 91 to be attached to the workpiece 81.”, Column 7 lines 57-67, “At step 111, a reference image for performing the first control is generated. FIG. 5 shows a reference image for performing the first control of the present embodiment. The reference image 61 corresponds to an image captured by the camera 25 when the second workpiece 91 is arranged at the target position with respect to the first workpiece 81. In the present embodiment, the reference image 61 is an image captured by the camera 25 when the pins 92a and 93a of the workpiece 91 are arranged directly above the holes 82a and 83a of the workpiece 81. The reference image 61 can be prepared by an operator and stored in the storage part 42.”, Column 10 lines 48-58, “Subsequently, the command generation unit 34 of the image processing unit 31 generates a movement command for the robot 1 so that the second workpiece 91 is arranged at the target position with respect to the first workpiece 81, based on the relative position amount in the image 62 captured by the camera 25 and the relative position amount in the reference image 61. The command generation unit 34 of the present embodiment generates a movement command for operating the robot 1 so that the relative position amount in the image 62 captured by the camera 25 approaches the relative position amount in the reference image 61.”. The cited passages clearly show that the end effector of the robot is configured to support and move the workpiece to the wok position (i.e. the holes 82a and 83s.)),
and a second operation to bring the workpiece supported by the tool into contact with at least one of the plurality of work positions (Koga: Column 4 line 55 – Column 5 line 2, “In the first control of the present embodiment, the pins 92a and 93a are controlled so as to be aligned with the holes 82a and 83a. The orientation of the workpiece 91 with respect to the workpiece 81 is adjusted before the first control is performed. In other words, the orientation of the robot 1 is adjusted so that the pins 92a and 93a of the workpiece 91 are arranged on a straight line extending in the vertical direction. Thus, in the first control, the control for arranging the pin 92a directly above the hole 82a is performed so that the pins 92a and 93a can be aligned with the holes 82a and 83a. After the first control of the present embodiment, a downward movement of the workpiece 91 in the vertical direction indicated by arrow 103 causes the pins 92a, 93a to be inserted into the holes 82a, 83a, and causes the workpiece 91 to be attached to the workpiece 81.”), and
wherein the same target image is used in the alignment control performed before the first operation and the alignment control performed before the second operation (Koga: Column 4 line 55 – Column 5 line 2, Column 7 lines 57-67, Column 10 lines 48-58. The cited passages clearly show that the target image is used in the alignment control. Additionally, the cited passages show that the workpiece is aligned before the insertion begins.).
Regarding claim 23, Correll teaches a control method of an end effector including (Correll: Abstract, “Disclosed are various embodiments of a three-dimensional perception and object manipulation robot gripper configured for connection to and operation in conjunction with a robot arm. In some embodiments, the gripper comprises a palm, a plurality of motors or actuators operably connected to the palm, a mechanical manipulation system operably connected to the palm, a plurality of fingers operably connected to the motors or actuators and configured to manipulate one or more objects located within a workspace or target volume that can be accessed by the fingers. A depth camera system is also operably connected to the palm. One or more computing devices are operably connected to the depth camera and are configured and programmed to process images provided by the depth camera system to determine the location and orientation of the one or more objects within a workspace, and in accordance therewith, provide as outputs therefrom control signals or instructions configured to be employed by the motors or actuators to control movement and operation of the plurality of fingers so as to permit the fingers to manipulate the one or more objects located within the workspace or target volume. The gripper can also be configured to vary controllably at least one of a force, a torque, a stiffness, and a compliance applied by one or more of the plurality of fingers to the one or more objects.”)
a base portion (Correll: Figure 5 upper plate 24 and lower plate 26, Column 10 lines 6-31, “In the embodiment shown in FIG. 5, palm 22 also comprises upper plate 24, to which are mounted or attached, or disposed upon or to, various com ponents of gripper 20 such as upper enclosure 38a, depth camera system 30, third computing device 54, motors 61a and 61b, and motor shafts 62a and 62b. in one embodiment, sandwiched between upper plate 24 and lower plate 26 of palm 22 are at least some of the components comprising mechanical manipulation system 40.”. One of ordinary skill in the art would recognize that the plates function as a base for the end effector.),
an imaging apparatus supported by the base portion (Correll: Figure 5 camera system 30, Column 10 lines 6-31, “In the embodiment shown in FIG. 5, palm 22 also comprises upper plate 24, to which are mounted or attached, or disposed upon or to, various com ponents of gripper 20 such as upper enclosure 38a, depth camera system 30, third computing device 54, motors 61a and 61b, and motor shafts 62a and 62b. in one embodiment, sandwiched between upper plate 24 and lower plate 26 of palm 22 are at least some of the components comprising mechanical manipulation system 40.”. The cited passage clearly shows an imaging apparatus supported by the base.), and
a tool configured to perform work with respect to the workpiece (Correll: Column 9 lines 8-30, “In FIG. 3, fingers 41a and 41b of mechanical manipulation system 40a and 40b are shown disposed in an open position. In some embodiments, fingers 41a and 41b comprise surfaces or pads 42a and 42b, which can be removed and replaced with pads or surfaces 42a and 42b of different stiffnesses or compliance. In FIG. 3, fingers 41a and 41 are configured to engage and grasp objects therebetween, and to permit the manipulation of objects 2 by gripper 20.”).
Correll does not teach the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus.
Koga, in the same field of endeavor, teaches the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus (Koga: Figures 6 and 7, Column 9 lines 45-50, “FIG. 6 is a second flowchart of the first control in the present embodiment. Referring to FIGS. 3 and 6, after the robot controller 4 performs a control operation for moving the workpiece 91 closer to the workpiece 81, at step 115, the operation control unit 43 causes the camera 25 to capture images of the workpieces 81 and 91.”, Column 9 lines 51-60, “FIG. 7 shows an image captured by the camera in order to adjust the position of the second workpiece with respect to the first workpiece. The image 62 includes an image of the upper surface of the projection part 82 which is the first characteristic portion and an image of the upper surface of the projection part 92 which is the second characteristic portion. In the image 62, the second workpiece 91 is displaced from the first workpiece 81 toward the positive side of the u-axis of the screen coordinate system 52 as designated by arrow 101.”. The cited figures and passage clearly shows that the drive device causes the robot gripping the workpiece to move to a position such that the work piece is not overlapping with the imagining object. One of ordinary skill in the art would recognize that the workpiece refers to workpiece 91 in the cited paragraphs and the imaging object is the hole 82a of workpiece 81.).
Correll teaches a control method of an end effector including a base portion, an imaging apparatus supported by the base portion, and a tool configured to perform work with respect to the workpiece. Correll does not teach the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus. Koga teaches the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities required to have modified the method taught in Correll with the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus taught in Koga. Furthermore, the method taught in Correll is already configured to use visual servoing methods in the control of the end effector and is also configured to convey and manipulate a workpiece. As such the visual servoing methods taught in Koga could easily be added to Correll using methods known to one of ordinary skill in the art. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of an method comprising: the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method taught in Correll with the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus taught in Koga with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Correll in view of Koga does not each the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus.
Miike, in the same field of endeavor, teaches the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus (Miike: Abstract, “PROBLEM TO BE SOLVED: To provide a novel handling device that does not need lateral movement of a robot arm after an imaging portion images a workpiece, when the device holds the workpiece with fingers. SOLUTION: A handling device 1 according to the present invention includes: an imaging portion 2 for imaging a workpiece H to obtain geometric information of the workpiece H; a holding portion 6 for securely holding the imaging portion 2, and having a mounting portion 8 to be mounted on a robot arm R; and multiple fingers 10 provided movably with respect to the holding portion 6 through a drive portion 9, for holding the workpiece H. When the device images the workpiece H, the drive portion 9 moves the fingers 10 to the outside of an imaging area A1 of the imaging portion 2. Additionally, when the device holds the workpiece H, the drive portion can move the fingers 10 from the outside of the imaging area A1 of the imaging portion 2 to the inside of the imaging area A1, on the basis of the geometric information of the workpiece H”, ¶ 0010, “The handling device of the present invention moves the fingers outside the imaging area of the imaging unit when imaging the workpiece, but can move the fingers from outside the imaging area to inside the imaging area when holding the workpiece.”, ¶ 0021, “A drive unit 9 and a plurality of fingers 10 movably provided via the drive unit 9 are provided on the lower surface (the surface on the opposite side where the mounting unit 8 is located) of the holding unit 6. The finger 10 of the present embodiment is composed of four of the first finger 11 to the fourth finger 14. Further, claws 11 a to 14 a extending in a direction orthogonal to the axial direction of the fingers are provided at the tips of the first finger 11 to the fourth finger 14. The claws 11 a to 14 a of the present embodiment are provided to extend on both sides of the axis of each finger.”, ¶ 0023, “The first to fourth fingers 11 to 14 are held by the movable elements 20, 21, 23, and 24, respectively, and the first to fourth fingers 11 to 14 are movable in the X and Y directions from the inside to the outside of the imaging area A1 shown in FIG.”. The cited passages clearly shows that the fingers of the robot are configured to move in the X and Y directions, both when gripping the object and when not. As is clearly stated this allows the robot to move the gripped object in and out of the imaging range or the imaging apparatus. This clearly teaches that the robot is configured to move the object in and out of the imaging apparatus without moving the base or the imaging apparatus.).
Correll in view of Koga teaches a control method of an end effector including a base portion, an imaging apparatus supported by the base portion, and a tool configured to perform work with respect to a workpiece, the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus. Correll in view of Koga does not teach the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus. Miike teaches the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities to modify the control method of an end effector taught in Correll in view of Koga with the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus taught in Miike. Furthermore, the fingers of the end effector taught in Correll in view of Koga are already configured to move inside and outside of the field of view of the imaging device without causing the base or the imaging device to move. Additionally, one of ordinary skill in the art would recognize that the fingers are configured to move and object within the field of view of the imaging device and even partially outside of the field of view of the imaging device. As such, the end effector is readily modifiable to move the workpiece outside of the field of view in addition to the fingers, without moving the base or imaging device, as taught in Miike using known methods. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a control method of an end effector comprising: the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the control method of an end effector taught in Correll in view of Koga with the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the base portion or the imaging apparatus taught in Miike with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Regarding claim 24, Correll teaches a control method of a robot system including (Correll: Abstract, “Disclosed are various embodiments of a three-dimensional perception and object manipulation robot gripper configured for connection to and operation in conjunction with a robot arm. In some embodiments, the gripper comprises a palm, a plurality of motors or actuators operably connected to the palm, a mechanical manipulation system operably connected to the palm, a plurality of fingers operably connected to the motors or actuators and configured to manipulate one or more objects located within a workspace or target volume that can be accessed by the fingers. A depth camera system is also operably connected to the palm. One or more computing devices are operably connected to the depth camera and are configured and programmed to process images provided by the depth camera system to determine the location and orientation of the one or more objects within a workspace, and in accordance therewith, provide as outputs therefrom control signals or instructions configured to be employed by the motors or actuators to control movement and operation of the plurality of fingers so as to permit the fingers to manipulate the one or more objects located within the workspace or target volume. The gripper can also be configured to vary controllably at least one of a force, a torque, a stiffness, and a compliance applied by one or more of the plurality of fingers to the one or more objects.”)
an end effector including a base portion (Correll: Figure 5 upper plate 24 and lower plate 26, Column 10 lines 6-31, “In the embodiment shown in FIG. 5, palm 22 also comprises upper plate 24, to which are mounted or attached, or disposed upon or to, various com ponents of gripper 20 such as upper enclosure 38a, depth camera system 30, third computing device 54, motors 61a and 61b, and motor shafts 62a and 62b. in one embodiment, sandwiched between upper plate 24 and lower plate 26 of palm 22 are at least some of the components comprising mechanical manipulation system 40.”. One of ordinary skill in the art would recognize that the plates function as a base for the end effector.),
a base portion (Correll: Figure 5 upper plate 24 and lower plate 26, Column 10 lines 6-31, “In the embodiment shown in FIG. 5, palm 22 also comprises upper plate 24, to which are mounted or attached, or disposed upon or to, various com ponents of gripper 20 such as upper enclosure 38a, depth camera system 30, third computing device 54, motors 61a and 61b, and motor shafts 62a and 62b. in one embodiment, sandwiched between upper plate 24 and lower plate 26 of palm 22 are at least some of the components comprising mechanical manipulation system 40.”. One of ordinary skill in the art would recognize that the plates function as a base for the end effector.),
an imaging apparatus supported by the base portion (Correll: Figure 5 camera system 30, Column 10 lines 6-31, “In the embodiment shown in FIG. 5, palm 22 also comprises upper plate 24, to which are mounted or attached, or disposed upon or to, various com ponents of gripper 20 such as upper enclosure 38a, depth camera system 30, third computing device 54, motors 61a and 61b, and motor shafts 62a and 62b. in one embodiment, sandwiched between upper plate 24 and lower plate 26 of palm 22 are at least some of the components comprising mechanical manipulation system 40.”. The cited passage clearly shows an imaging apparatus supported by the base.), and
a tool configured to perform work with respect to the workpiece (Correll: Column 9 lines 8-30, “In FIG. 3, fingers 41a and 41b of mechanical manipulation system 40a and 40b are shown disposed in an open position. In some embodiments, fingers 41a and 41b comprise surfaces or pads 42a and 42b, which can be removed and replaced with pads or surfaces 42a and 42b of different stiffnesses or compliance. In FIG. 3, fingers 41a and 41 are configured to engage and grasp objects therebetween, and to permit the manipulation of objects 2 by gripper 20.”),
a robot to which the end effector is attached, the robot being configured to move a position and posture of the end effector (Correll: Column 8 lines 4-23, “FIG. 1 shows robot 10 engaged in a "bin picking" task, which involves retrieving items such as screws from bin 25. Here, robot 10 is shown in a configuration that puts gripper 20 comprising cameras 32 and 34 and a light illumination device 36 (not shown in FIG. 1) above bin 4, which in the illustrative embodiment of FIG. 1 features a workspace or target volume within which object(s) are situated or located, 10 where the workspace or target volume has an extent or distance associated therewith which ranges between about 11 cm and about 30 cm from the forward or sensing portion of gripper 20.”, Column 27 lines 52-58, “FIG. 14 shows one embodiment of a method 200 that may be employed using gripper 20. According to one embodiment, method 200 comprises step 201 of determining the position and characteristics of object 2, step 203 of actuating and controlling one or more of any of motors 41, mechanical manipulation system 40, wrist 14, arm 12, and robot 10, and step 205 of grasping, manipulating, and/or moving object 2.”. The cited passages clearly show that the end effector is attached to a robot and wherein the robot is configured to move the end effector.); and
a control unit configured to control the robot and the end effector (Correll: Column 3 line 25 – Column 4 line 17, “… a second computing device mounted to, on or in, or connected operably to, the palm, the second computing device being operably connected to the first computing device and to the plurality of at least one of motors and actuators, the second computing device being configured to receive the output images provided by the first computing device, and further being configured and programmed to process the output images to determine the location and orientation of the one or more objects within the workspace, and in accordance therewith provide as outputs therefrom control signals or instructions configured to be employed by the motors or actuators to control movement and operation of the plurality of fingers so as to permit the fingers to manipulate the one or more objects located within the workspace or target volume,”, Column 9 line 48 – Column 10 line 5, “Further as shown in FIG. 5, in one embodiment, second computing device 52 comprises a processor, controller, microcontroller, and/or CPU (such as a system-on-a-module) attached to the underside of palm 22, while peripheral break-out device 51 comprising, for example, Ethernet and USB connections, is likewise attached to the underside of palm 22.”).
Correll does not teach the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the robot or the imaging apparatus.
Koga, in the same field of endeavor, teaches the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus (Koga: Figures 6 and 7, Column 9 lines 45-50, “FIG. 6 is a second flowchart of the first control in the present embodiment. Referring to FIGS. 3 and 6, after the robot controller 4 performs a control operation for moving the workpiece 91 closer to the workpiece 81, at step 115, the operation control unit 43 causes the camera 25 to capture images of the workpieces 81 and 91.”, Column 9 lines 51-60, “FIG. 7 shows an image captured by the camera in order to adjust the position of the second workpiece with respect to the first workpiece. The image 62 includes an image of the upper surface of the projection part 82 which is the first characteristic portion and an image of the upper surface of the projection part 92 which is the second characteristic portion. In the image 62, the second workpiece 91 is displaced from the first workpiece 81 toward the positive side of the u-axis of the screen coordinate system 52 as designated by arrow 101.”. The cited figures and passage clearly shows that the drive device causes the robot gripping the workpiece to move to a position such that the work piece is not overlapping with the imagining object. One of ordinary skill in the art would recognize that the workpiece refers to workpiece 91 in the cited paragraphs and the imaging object is the hole 82a of workpiece 81.).
Correll teaches a control method of a robot system including an end effector including a base portion, an imaging apparatus supported by the base portion, and a tool configured to perform work with respect to the workpiece, a robot to which the end effector is attached, the robot being configured to move a position and posture of the end effector, and a control unit configured to control the robot and the end effector. Correll does not teach the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus. Koga teaches the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities required to have modified the method taught in Correll with the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus taught in Koga. Furthermore, the method taught in Correll is already configured to use visual servoing methods in the control of the end effector and is also configured to convey and manipulate a workpiece. As such the visual servoing methods taught in Koga could easily be added to Correll using methods known to one of ordinary skill in the art. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of an method comprising: the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method taught in Correll with the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus taught in Koga with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Correll in view of Koga does not teach the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the robot or the imaging apparatus.
Miike, in the same field of endeavor, teaches the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the robot or the imaging apparatus (Miike: Abstract, “PROBLEM TO BE SOLVED: To provide a novel handling device that does not need lateral movement of a robot arm after an imaging portion images a workpiece, when the device holds the workpiece with fingers. SOLUTION: A handling device 1 according to the present invention includes: an imaging portion 2 for imaging a workpiece H to obtain geometric information of the workpiece H; a holding portion 6 for securely holding the imaging portion 2, and having a mounting portion 8 to be mounted on a robot arm R; and multiple fingers 10 provided movably with respect to the holding portion 6 through a drive portion 9, for holding the workpiece H. When the device images the workpiece H, the drive portion 9 moves the fingers 10 to the outside of an imaging area A1 of the imaging portion 2. Additionally, when the device holds the workpiece H, the drive portion can move the fingers 10 from the outside of the imaging area A1 of the imaging portion 2 to the inside of the imaging area A1, on the basis of the geometric information of the workpiece H”, ¶ 0010, “The handling device of the present invention moves the fingers outside the imaging area of the imaging unit when imaging the workpiece, but can move the fingers from outside the imaging area to inside the imaging area when holding the workpiece.”, ¶ 0021, “A drive unit 9 and a plurality of fingers 10 movably provided via the drive unit 9 are provided on the lower surface (the surface on the opposite side where the mounting unit 8 is located) of the holding unit 6. The finger 10 of the present embodiment is composed of four of the first finger 11 to the fourth finger 14. Further, claws 11 a to 14 a extending in a direction orthogonal to the axial direction of the fingers are provided at the tips of the first finger 11 to the fourth finger 14. The claws 11 a to 14 a of the present embodiment are provided to extend on both sides of the axis of each finger.”, ¶ 0023, “The first to fourth fingers 11 to 14 are held by the movable elements 20, 21, 23, and 24, respectively, and the first to fourth fingers 11 to 14 are movable in the X and Y directions from the inside to the outside of the imaging area A1 shown in FIG.”. The cited passages clearly shows that the fingers of the robot are configured to move in the X and Y directions, both when gripping the object and when not. As is clearly stated this allows the robot to move the gripped object in and out of the imaging range or the imaging apparatus. This clearly teaches that the robot is configured to move the object in and out of the imaging apparatus without moving the base or the imaging apparatus.).
Correll in view of Koga a control method of a robot system including an end effector including a base portion, an imaging apparatus supported by the base portion, and a tool configured to perform work with respect to the workpiece, a robot to which the end effector is attached, the robot being configured to move a position and posture of the end effector, and a control unit configured to control the robot and the end effector, the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus. Correll in view of Koga does not teach the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the robot or the imaging apparatus. Miike teaches the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the robot or the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities to modify the control method of a robot system taught in Correll in view of Koga with the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the robot or the imaging apparatus taught in Miike. Furthermore, the fingers of the end effector taught in Correll in view of Koga are already configured to move inside and outside of the field of view of the imaging device without causing the base or the imaging device to move. Additionally, one of ordinary skill in the art would recognize that the fingers are configured to move and object within the field of view of the imaging device and even partially outside of the field of view of the imaging device. As such, the end effector is readily modifiable to move the workpiece outside of the field of view in addition to the fingers, without moving the base or imaging device, as taught in Miike using known methods. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a control method of a robot system comprising: the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the robot or the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the control method of a robot system taught in Correll in view of Koga with the method comprising: a positioning step in which, in a case of capturing an image of an imaging object by the imaging apparatus, the control unit positions the tool supporting the workpiece outside of a range in which at least part of the tool or the workpiece supported by the tool overlaps with the imaging object in a field of view of the imaging apparatus without moving either the robot or the imaging apparatus taught in Miike with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Regarding claim 25, Correll in view of Koga in further view of Miike teaches wherein, at the positioning step of positioning the tool, from a state in which at least part of the tool and the supported workpiece is located inside of the range, before capturing the image of the imaging object by the imaging apparatus, the control unit moves the tool supporting the workpiece outside of the range (Koga: Column 9 lines 45-50, “FIG. 6 is a second flowchart of the first control in the present embodiment. Referring to FIGS. 3 and 6, after the robot controller 4 performs a control operation for moving the workpiece 91 closer to the workpiece 81, at step 115, the operation control unit 43 causes the camera 25 to capture images of the workpieces 81 and 91.”, Column 9 lines 51-60, “FIG. 7 shows an image captured by the camera in order to adjust the position of the second workpiece with respect to the first workpiece. The image 62 includes an image of the upper surface of the projection part 82 which is the first characteristic portion and an image of the upper surface of the projection part 92 which is the second characteristic portion. In the image 62, the second workpiece 91 is displaced from the first workpiece 81 toward the positive side of the u-axis of the screen coordinate system 52 as designated by arrow 101.”. The cited passage shows that the robot is configured to move the tool outside of the range prior to capturing an image.).
Regarding claim 26, Correll in view of Koga in further view of Miike teaches further comprising: an imaging step in which the control unit captures the image of the imaging object by the imaging apparatus in a state in which the tool supporting the workpiece is located outside of the range (Koga: Figures 6 and 7, Column 9 lines 45-50, “FIG. 6 is a second flowchart of the first control in the present embodiment. Referring to FIGS. 3 and 6, after the robot controller 4 performs a control operation for moving the workpiece 91 closer to the workpiece 81, at step 115, the operation control unit 43 causes the camera 25 to capture images of the workpieces 81 and 91.”, Column 9 lines 51-60, “FIG. 7 shows an image captured by the camera in order to adjust the position of the second workpiece with respect to the first workpiece. The image 62 includes an image of the upper surface of the projection part 82 which is the first characteristic portion and an image of the upper surface of the projection part 92 which is the second characteristic portion. In the image 62, the second workpiece 91 is displaced from the first workpiece 81 toward the positive side of the u-axis of the screen coordinate system 52 as designated by arrow 101.”. The cited passages clearly teach a step of capturing an image where the tool is located outside of the range.); and
a restoring step in which, after the imaging step, from the state in which the tool supporting the workpiece is located outside of the range, before performing the work by the tool, the control unit moves the tool supporting the workpiece inside of the range such that at least part of the workpiece overlaps with the imaging object in the field of view of the imaging apparatus (Koga: Column 4 line 55 – Column 5 line 2, “In the first control of the present embodiment, the pins 92a and 93a are controlled so as to be aligned with the holes 82a and 83a. The orientation of the workpiece 91 with respect to the workpiece 81 is adjusted before the first control is performed. In other words, the orientation of the robot 1 is adjusted so that the pins 92a and 93a of the workpiece 91 are arranged on a straight line extending in the vertical direction. Thus, in the first control, the control for arranging the pin 92a directly above the hole 82a is performed so that the pins 92a and 93a can be aligned with the holes 82a and 83a. After the first control of the present embodiment, a downward movement of the workpiece 91 in the vertical direction indicated by arrow 103 causes the pins 92a, 93a to be inserted into the holes 82a, 83a, and causes the workpiece 91 to be attached to the workpiece 81.”, Column 7 lines 57-67, “At step 111, a reference image for performing the first control is generated. FIG. 5 shows a reference image for performing the first control of the present embodiment. The reference image 61 corresponds to an image captured by the camera 25 when the second workpiece 91 is arranged at the target position with respect to the first workpiece 81. In the present embodiment, the reference image 61 is an image captured by the camera 25 when the pins 92a and 93a of the workpiece 91 are arranged directly above the holes 82a and 83a of the workpiece 81. The reference image 61 can be prepared by an operator and stored in the storage part 42.”, Column 10 lines 48-58, “Subsequently, the command generation unit 34 of the image processing unit 31 generates a movement command for the robot 1 so that the second workpiece 91 is arranged at the target position with respect to the first workpiece 81, based on the relative position amount in the image 62 captured by the camera 25 and the relative position amount in the reference image 61. The command generation unit 34 of the present embodiment generates a movement command for operating the robot 1 so that the relative position amount in the image 62 captured by the camera 25 approaches the relative position amount in the reference image 61.”. The cited passages clearly show that the system is configured to align the workpieces before performing an attachment operation on said workpieces.).
Regarding claim 28, Correll in view of Koga in further view of Miike teaches a method for manufacturing an article by using the robot system according to claim 11, the method comprising: obtaining a position of the imaging object by capturing an image of the imaging object with the imaging apparatus; and assembling the workpiece to the imaging object based on the position of the imaging object to manufacture the article (Correll: Column 16 line 60 – Column 17 line 10, “Another typical application is the "peg-in-hole" problem, which requires inserting a peg-like object 2, for example a bolt or a screw, into a hole. This task requires locating the hole as well as the object 2 that needs to be inserted. Object 2 must be grasped so that its centerline aligns with that of gripper 20, or-at the very least-so that its exact pose within gripper 20 can be expected to be known. While accurate localization and placement is not sufficient for this task ( see below), there is agreement among practitioners that accurate localization and placement are beneficial to quickly and reliably complete a peg-in-hole task. Assuming the task is to insert a bolt of 5 mm diameter into a hole having an appropriate size of 5.2 mm, both the bolt and the hole (assuming a top-down view) span across approximately 14 pixels for a total area of about 152 pixels, allowing gripper 20 to localize both the screw and the hole with submillimeter accuracy and permitting both an accurate grasp of the object and subsequent placement above the hole.” Koga: Column 4 line 55 – Column 5 line 2, “In the first control of the present embodiment, the pins 92a and 93a are controlled so as to be aligned with the holes 82a and 83a. The orientation of the workpiece 91 with respect to the workpiece 81 is adjusted before the first control is performed. In other words, the orientation of the robot 1 is adjusted so that the pins 92a and 93a of the workpiece 91 are arranged on a straight line extending in the vertical direction. Thus, in the first control, the control for arranging the pin 92a directly above the hole 82a is performed so that the pins 92a and 93a can be aligned with the holes 82a and 83a. After the first control of the present embodiment, a downward movement of the workpiece 91 in the vertical direction indicated by arrow 103 causes the pins 92a, 93a to be inserted into the holes 82a, 83a, and causes the workpiece 91 to be attached to the workpiece 81.”).
Regarding claim 29, Correll in view of Koga in further view of Miike teaches a non-transitory computer readable medium storing a program that enables a computer to execute the control method of the robot system according to claim 24 (Correll: Column 29 lines 36-55, “Computer system 300 can include a hard disk drive 303, a magnetic disk drive 308 (e.g., to read from or write to removable disk 309), or an optical disk drive 310 (e.g., for reading CD-ROM disk 311 or to read from or write to other optical media). Hard disk drive 303, magnetic disk drive 308, and optical disk drive 310 are connected to system bus 303 by a hard disk drive interface 312, a magnetic disk drive interface 313, and an optical drive interface 314, respectively. The drives and their associated computer-readable media are configured to provide nonvolatile storage of data, data structures, and computer-executable instructions for computer system 300.”).
Claim(s) 14 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 11148295 B2 ("Correll") in view of US 11679508 B2 ("Koga") in further view of JP 2016120545 A ("Miike") in further view of US 2023/0086122 A1 ("Gombolay").
Regarding claim 14, Correll in view of Koga in further view of Miike wherein the end effector includes an irradiation unit irradiating light in an optical axis direction (Correll: Figures 3 and 6a-b, Column 9 lines 8-30, “FIG. 3 shows one embodiment of gripper 20 operably attached to robotic wrist 14. In one embodiment, and in conjunction with light projector 36, camera sensors 32 and 34 are configured to provide stereo-vision capabilities for gripper 20, more about which is said below.”, Column 14 lines, 24-37, “In one embodiment, gripper 20 includes a depth camera 30 that comprises appropriate infrared camera sensors 32 and 34 along with s suitable infrared projector 36, where depth camera 30 is mounted on palm 22 of gripper 20. The stereo camera sensors 32 and 34 can be configured to be sensitive to the visible light spectrum (monochrome), to the infrared light spectrum, or to both spectra.”, Column 19 lines 1-28, “Among other things, workspace or target volume 74 of FIGS. 6a and 6b is defined by the capabilities of depth camera 30, which depends upon angle 33 over which light from projector 36 can be reflected or backscattered from an object 2 back towards camera sensors 32 and 34 and detected thereby, and the distance between projector 36 and camera sensors 32 and 34 on the one hand, and object 2 (not shown in FIGS. 6a and 6b).”. The cited passages clearly show a irradiation unit irradiating light in an optical axis direction.),
wherein the control unit is configured to capture the image by the imaging apparatus in a manner irradiating the imaging object by the irradiation unit (Correll: Column 14 lines 3-23, “Referring now to FIGS. 3 through 5, in one embodiment robotic gripper 20 is equipped with two infrared camera sensors 32 and 34 and infrared light projector 36 mounted on palm 22 of gripper 20, which together serve as a stereo pair of cameras and an infrared illumination source that provides substantial depth perception capabilities. … . In such an embodiment, camera sensors 32 and 34 are configured to operate in the infrared light spectrum and record up to 1280x720 pixels at 90 Hertz. Infrared light projector 36 provides an illumination pattern that facilitates matching of stereo-pairs in environments with few graphical features.”, Column 19 lines 1-28, “Among other things, workspace or target volume 74 of FIGS. 6a and 6b is defined by the capabilities of depth camera 30, which depends upon angle 33 over which light from projector 36 can be reflected or backscattered from an object 2 back towards camera sensors 32 and 34 and detected thereby, and the distance between projector 36 and camera sensors 32 and 34 on the one hand, and object 2 (not shown in FIGS. 6a and 6b).”. The cited passages clearly show that the imagining device is configured to capture an image of the illuminated objected.)
Correll in view of Koga in further view of Miike does not teach wherein the imaging object includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion,
wherein the control unit is configured to capture the image by the imaging apparatus in a manner irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus.
Gombolay, in the same field of endeavor, teaches wherein the imaging object includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion (Gombolay: Figure 11 countersunk classification test 1102, ¶ 0092, “The study provided an approach to applying supervised and reinforcement learning techniques with a collaborative robot to complete drilling and fastening tasks in final aerospace assembly. The system tasks included inserting fasteners into chamfered and unchamfered holes on a custom-built section of an aircraft fuselage.”, ¶ 0093, “Example System Layout. For this task, the study used a custom-built section of an aircraft fuselage, known as a "coupon", as a test bed. The coupon was 0.9 m by 0.9 m and contained over one-hundred drilled holes. Each hole was either a "countersunk" hole with a chamfered rim or a "buttonhead" hole without a rim.”, ¶ 0127, “Using flat metal sheets lined with countersunk and buttonhead holes, the study tested the classifier to label the video feed in FIG. 11 real-time. shows the results of the classification. 1102, 1104 Diagrams show the classification labels "c" and "b" superimposed over the countersunk holes and the buttonhead holes. The number shows the estimated size of the 1106 hole (in microns). Plot shows a confusion matrix from 1000 one-shot classification tasks completed by the SNN trained on 30 samples.”. The cited passages clearly teach that the imagining object includes a flat surface and an inclined portion with respect to the flat portion (i.e. the chamfer).),
wherein the control unit is configured to capture the image by the imaging apparatus in a manner irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus (Gombolay: Figure 11 countersunk classification test 1102, ¶ 0092, “The study provided an approach to applying supervised and reinforcement learning techniques with a collaborative robot to complete drilling and fastening tasks in final aerospace assembly. The system tasks included inserting fasteners into chamfered and unchamfered holes on a custom-built section of an aircraft fuselage.”, ¶ 0093, “Example System Layout. For this task, the study used a custom-built section of an aircraft fuselage, known as a "coupon", as a test bed. The coupon was 0.9 m by 0.9 m and contained over one-hundred drilled holes. Each hole was either a "countersunk" hole with a chamfered rim or a "buttonhead" hole without a rim.”, ¶ 0127, “Using flat metal sheets lined with countersunk and buttonhead holes, the study tested the classifier to label the video feed in FIG. 11 real-time. shows the results of the classification. 1102, 1104 Diagrams show the classification labels "c" and "b" superimposed over the countersunk holes and the buttonhead holes. The number shows the estimated size of the 1106 hole (in microns). Plot shows a confusion matrix from 1000 one-shot classification tasks completed by the SNN trained on 30 samples.”. One of ordinary skill in the art would have recognized from the cited passages, that as long as the camera is at a position that is roughly parallel to the center axis of the holes, that the light reflected from the flat portions would be reflected to towards the camera and the light reflected from the inclined portion (i.e. the chamfer) would be reflected away from the camera.).
Correll in view of Koga in further view of Miike teaches a robot system comprising: wherein the end effector includes an irradiation unit irradiating light in an optical axis direction, and wherein the control unit is configured to capture the image by the imaging apparatus in a manner irradiating the imaging object by the irradiation unit. Correll in view of Koga in further view of Miike does not teach wherein the imaging object includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, wherein the control unit is configured to capture the image by the imaging apparatus in a manner irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus. Gombolay teaches wherein the imaging object includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, wherein the control unit is configured to capture the image by the imaging apparatus in a manner irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities required to have modified the robot system taught in Correll in view of Koga in further view of Miike with wherein the imaging object includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, wherein the control unit is configured to capture the image by the imaging apparatus in a manner irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus taught in Gombolay. Furthermore, the system taught in Correll in view of Koga in further view of Miike is already configured to perform a peg-and-hole type insertion task (Correll: Column 16 line 60 – Column 17 line 10, “Another typical application is the "peg-in-hole" problem, which requires inserting a peg-like object 2, for example a bolt or a screw, into a hole. This task requires locating the hole as well as the object 2 that needs to be inserted. Object 2 must be grasped so that its centerline aligns with that of gripper 20, or-at the very least-so that its exact pose within gripper 20 can be expected to be known. While accurate localization and placement is not sufficient for this task ( see below), there is agreement among practitioners that accurate localization and placement are beneficial to quickly and reliably complete a peg-in-hole task. Assuming the task is to insert a bolt of 5 mm diameter into a hole having an appropriate size of 5.2 mm, both the bolt and the hole (assuming a top-down view) span across approximately 14 pixels for a total area of about 152 pixels, allowing gripper 20 to localize both the screw and the hole with submillimeter accuracy and permitting both an accurate grasp of the object and subsequent placement above the hole.”). As such, modifying the imagining object to include a flat portion and an inclined portion as taught in Gombolay would only require the simple modification of a chamfer to the hole already taught in Correll in view of Koga in further view of Miike. This modification would be well within the technological capabilities of a person of ordinary skill in the art. Modifying the hole to include a chamfer would not change or introduce new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a robot system comprising: wherein the imaging object includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, wherein the control unit is configured to capture the image by the imaging apparatus in a manner irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot system taught in Correll in view of Koga in further view of Miike with wherein the imaging object includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, wherein the control unit is configured to capture the image by the imaging apparatus in a manner irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus taught in Gombolay with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Regarding claim 15, Correll in view of Koga in further view of Miike in further view of Gombolay teaches wherein the imaging object includes a hole portion formed in a hole-like shape with respect to the flat surface portion, and wherein the inclined portion is a chamfered portion chamfered at an outer edge of an opening of the hole portion (Gombolay: Figure 11 countersunk classification test 1102, ¶ 0092, “The study provided an approach to applying supervised and reinforcement learning techniques with a collaborative robot to complete drilling and fastening tasks in final aerospace assembly. The system tasks included inserting fasteners into chamfered and unchamfered holes on a custom-built section of an aircraft fuselage.”, ¶ 0093, “Example System Layout. For this task, the study used a custom-built section of an aircraft fuselage, known as a "coupon", as a test bed. The coupon was 0.9 m by 0.9 m and contained over one-hundred drilled holes. Each hole was either a "countersunk" hole with a chamfered rim or a "buttonhead" hole without a rim.”, ¶ 0127, “Using flat metal sheets lined with countersunk and buttonhead holes, the study tested the classifier to label the video feed in FIG. 11 real-time. shows the results of the classification. 1102, 1104 Diagrams show the classification labels "c" and "b" superimposed over the countersunk holes and the buttonhead holes. The number shows the estimated size of the 1106 hole (in microns). Plot shows a confusion matrix from 1000 one-shot classification tasks completed by the SNN trained on 30 samples.”).
Claim(s) 16-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 11148295 B2 ("Correll") in view of US 11679508 B2 ("Koga") in further view of JP 2016120545 A ("Miike") in further view of US 2023/0086122 A1 ("Gombolay") in further view of US 2018/0272537 A1 ("Ishigaki").
Regarding claim 16, Correll in view of Koga in further view of Miike in further view of Gombolay does not teach wherein the imaging apparatus includes a telecentric optical system, and
wherein the control unit is configured to capture the image by the imaging apparatus in a state in which an optical axis of the telecentric optical system is maintained to be perpendicular to the flat surface portion.
Ishigaki, in the same field of endeavor, teaches wherein the imaging apparatus includes a telecentric optical system (Ishigaki: ¶ 0066, “For example, the second imaging unit C2 is a camera including the CCD or the CMOS which is an imaging element for converting condensed light into an electric signal. The second imaging unit C2 is a camera having a telecentric lens.”), and
wherein the control unit is configured to capture the image by the imaging apparatus in a state in which an optical axis of the telecentric optical system is maintained to be perpendicular to the flat surface portion (Ishigaki: ¶ 0111, “Hereinafter, referring to FIGS. 4 to 9, the adjustment of the posture of the first imaging unit Cl and the posture of the second imaging unit C2 will be described within the adjustments performed as preparations before the robot control device 30 causes both the first robot 21 and the second robot 22 to carry out the predetermined work. In the adjustment, the posture of the first imaging unit Cl is adjusted so that an optical axis of the first imaging unit Cl and the upper surface of the object are orthogonal to each other. In this manner, the robot control device 30 can more accurately detect the position and the posture of the object included in the first image captured by the first imaging unit Cl, compared to the detection before the adjustment is performed. In adjusting the posture of the first imaging unit Cl and the posture of the second imaging unit C2, the posture of the second imaging unit C2 is adjusted so that an optical axis of the second imaging unit C2 and the upper surface of the object O are orthogonal to each other. In this manner, the robot control device 30 can more accurately detect the position and the posture of the object included in the second image captured by the second imaging unit C2, compared to the detection before the adjustment is performed.”. The cited passages teach that the system is configured to keep the cameras orthogonal to the surface of the object, which is the same as keeping the cameras perpendicular to the surface.).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot system taught in Correll in view of Koga in further view of Miike in further view of Gombolay with wherein the imaging apparatus includes a telecentric optical system, and wherein the control unit is configured to capture the image by the imaging apparatus in a state in which an optical axis of the telecentric optical system is maintained to be perpendicular to the flat surface portion taught in Ishigaki with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have required the simple substitution of one known camera for another. The robot system taught in Correll in view of Koga in further view of Miike in further view of Gombolay already teaches an imagining apparatus comprising a camera. Additionally changing one type of camera for a different type of camera would have been well within the technological capabilities of a person of ordinary skill in the art. Such a modification would not have changed or introduced new functionality to either. No inventive effort would have been required.
Regarding claim 17, Correll in view of Koga in further view of Miike in further view of Gombolay in further view of Ishigaki teaches wherein the irradiation unit is a coaxial incident illumination unit configured to irradiate the light coaxially with the optical axis of the telecentric optical system (Correll: Figures 3 and 6a-b, Column 9 lines 8-30, “FIG. 3 shows one embodiment of gripper 20 operably attached to robotic wrist 14. In one embodiment, and in conjunction with light projector 36, camera sensors 32 and 34 are configured to provide stereo-vision capabilities for gripper 20, more about which is said below.”, Column 14 lines, 24-37, “In one embodiment, gripper 20 includes a depth camera 30 that comprises appropriate infrared camera sensors 32 and 34 along with s suitable infrared projector 36, where depth camera 30 is mounted on palm 22 of gripper 20. The stereo camera sensors 32 and 34 can be configured to be sensitive to the visible light spectrum (monochrome), to the infrared light spectrum, or to both spectra.”, Column 19 lines 1-28, “Among other things, workspace or target volume 74 of FIGS. 6a and 6b is defined by the capabilities of depth camera 30, which depends upon angle 33 over which light from projector 36 can be reflected or backscattered from an object 2 back towards camera sensors 32 and 34 and detected thereby, and the distance between projector 36 and camera sensors 32 and 34 on the one hand, and object 2 (not shown in FIGS. 6a and 6b).”. The cited passages clearly show a irradiation unit irradiating light in an optical axis direction.), and
wherein, while irradiating the flat surface portion by the coaxial incident illumination unit in a state in which the optical axis of the telecentric optical system is maintained to be perpendicular to the flat surface portion, the control unit is configured to capture the image by the imaging apparatus (Correll: Column 14 lines 3-23, “Referring now to FIGS. 3 through 5, in one embodiment robotic gripper 20 is equipped with two infrared camera sensors 32 and 34 and infrared light projector 36 mounted on palm 22 of gripper 20, which together serve as a stereo pair of cameras and an infrared illumination source that provides substantial depth perception capabilities. … . In such an embodiment, camera sensors 32 and 34 are configured to operate in the infrared light spectrum and record up to 1280x720 pixels at 90 Hertz. Infrared light projector 36 provides an illumination pattern that facilitates matching of stereo-pairs in environments with few graphical features.”, Column 19 lines 1-28, “Among other things, workspace or target volume 74 of FIGS. 6a and 6b is defined by the capabilities of depth camera 30, which depends upon angle 33 over which light from projector 36 can be reflected or backscattered from an object 2 back towards camera sensors 32 and 34 and detected thereby, and the distance between projector 36 and camera sensors 32 and 34 on the one hand, and object 2 (not shown in FIGS. 6a and 6b).”. The cited passages clearly show that the imagining device is configured to capture an image of the illuminated objected.).
Correll in view of Koga in further view of Miike in view of Gombolay teaches an irradiation unit that irradiates light coaxially with the imaging device and capturing an image when the imaging object is irradiated by the irradiating unit. Ishigaki teaches a telecentric camera that is configured to be maintained perpendicular to a flat surface. One of ordinary skill in the art would recognize that when the camera of the system taught in Correll in view of Koga in further view of Miike in view of Gombolay is modified with the telecentric camera taught in Ishigaki, the irradiating unit would still irradiate light coaxially with the imaging device and the system would still be configured to capture an image when the irradiating unit irradiates light. As such, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed, that the combination of Correll in view of Koga in further view of Miike in further view of Gombolay in further view of Ishigaki teaches the limitations of claim 17.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 11148295 B2 ("Correll") in view of US 11679508 B2 ("Koga") in further view of US 2023/0086122 A1 ("Gombolay") in further view of JP 2016120545 A ("Miike") in further view of KR 20190079326 A ("Chung").
Regarding claim 20, Correll in view of Koga in further view of Miike in further view of Gombolay does not teach wherein the control unit is configured to calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus while changing the posture of the end effector, and
perform posture correction control to correct the posture of the end effector such that the calculated brightness is maximized.
Chung, in the same field of endeavor, teaches wherein the control unit is configured to calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus while changing the posture of the end effector (Chung: ¶ 0032, “The optical unit 120 includes a laser light generator 121 that emits a laser light LB and a light receiver 122 that receives reflected light RB of the laser light LB reflected from the correction plate 110 can do.”, ¶ 0034, “he light receiver 122 receives the reflected light RB of the laser light LB reflected from the correction plate 110 and can measure the light amount. The photodetector 122 can receive the reflected light RB reflected by at least one of the correction groove 111 and the inclined surface 112 of the correction plate 110. [ When the robot MAR is operated at the normal position, the light receiver 122 can receive the reflected light RB reflected by the correction groove 111. On the other hand, when the robot MAR is operated at a non-normal position, that is, when a position error of the robot MAR occurs, reflected light RB reflected from the correction groove 111, (RB) can also be received.”. The cited passages clearly teach that the system is configured to measure the amount of light reflected. Furthermore, as can be clearly see from figures 2 and 3, the correction groove which reflects the maximum amount of light is a flat surface.), and
perform posture correction control to correct the posture of the end effector such that the calculated brightness is maximized (Chung: ¶ 0059, “Next, the light amount of the reflected light RB reflected by at least one of the correction groove 111 and the inclined surface 112 of the correction plate 110 is measured by the light receiver 122 (S503). 2, the light amount of the reflected light RB measured by the light receiver 122 is a maximum value when the reflected light RB is reflected so that the reflected light RB fills all the inside of the critical range R Quot; reference light amount & quot;), indicating that the robot MAR is operating at the normal position.”, ¶ 0061, “3, when the reflected light RB does not fill the inside of the critical range R, that is, when a part of the laser light LB is reflected on the inclined surface 112 of the correction plate 110 , The light amount of the reflected light RB measured by the light receiver 122 becomes smaller than the reference light amount (S503).”, ¶ 0062, “If the light amount of the reflected light RB measured by the light receiver 122 is lower than the reference light amount, it is regarded that the position error of the robot MAR is sensed (S504).”, ¶ 0063, “f the position error of the robot MAR is sensed at step S504, the signal processor 141 of the controller 140 receives the image signal generated from the image sensing unit 130 and outputs the reflected light RB based on the image signal. The position error L is calculated (S505).”, ¶ 0064, “Next, the robot position corrector 142 corrects the position of the robot MAR based on the position error L calculated in the signal processor 141 (S506).”, ¶ 0065, “hereafter, the step S503 of measuring the light amount of the reflected light RB to the light receiver 122 from the step S502 of irradiating the laser light LB to the correction groove 111 of the correction plate 110, If the light amount of the reflected light RB measured by the light receiver 122 satisfies the reference light amount through the step S504 of determining whether or not the position error is detected, the position accuracy correction of the robot can be completed (S507).”. The cited passages clearly teach that the system is configured to correct the position of the robot until such a time that the maximum amount of light is detected.).
Correll in view of Koga in further view of Miike in further view of Gombolay teaches a robot system comprising an imaging device and an illumination device. Correll in view of Koga in further view of Miike in further view of Gombolay does not teach wherein the control unit is configured to calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus while changing the posture of the end effector, and perform posture correction control to correct the posture of the end effector such that the calculated brightness is maximized. Chung teaches wherein the control unit is configured to calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus while changing the posture of the end effector, and perform posture correction control to correct the posture of the end effector such that the calculated brightness is maximized. A person of ordinary skill in the art would have had the technological capabilities required to have modified the robot system taught in Correll in view of Koga in further view of Miike in further view of Gombolay with wherein the control unit is configured to calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus while changing the posture of the end effector, and perform posture correction control to correct the posture of the end effector such that the calculated brightness is maximized taught in Chung. Furthermore, the system taught in Correll in view of Koga in further view of Miike in further view of Gombolay is already configured to control the robot by illuminating the object using the illumination device and determining the pattern of illuminated light using an imaging device. As such, the system taught in Correll in view of Koga in further view of Miike in further view of Gombolay is readily configurable for the method taught in Chung. A person of ordinary skill in the art would have been easily able to modify the system taught in Correll in view of Koga in further view of Miike in further view of Gombolay with the methods used to detect the amount of light reflected and control the robot based on this taught in Chung. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a robot system comprising: wherein the control unit is configured to calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus while changing the posture of the end effector, and perform posture correction control to correct the posture of the end effector such that the calculated brightness is maximized.
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot system taught in Correll in view of Koga in further view of Miike in further view of Gombolay with wherein the control unit is configured to calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus while changing the posture of the end effector, and perform posture correction control to correct the posture of the end effector such that the calculated brightness is maximized taught in Chung with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because it would have yielded predictable results.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 11148295 B2 ("Correll") in view of US 11679508 B2 ("Koga") in further view of US 2023/0086122 A1 ("Gombolay") in further view of JP 2016120545 A ("Miike") in further view of US 11059169 B2 ("Suzuki") in further view of KR 20190079326 A ("Chung").
Regarding claim 21, Correll in view of Koga in further view of Miike in further view of Gombolay teaches a display apparatus that is configured to display information (Correll: Column 29 lines 56-62, “A number of program modules may be stored in drives and RAM 303, including operating system 315, one or more application programs 316, other program modules 313, and program data 318. The application programs and program data can include functions and methods programmed to acquire, process and display data from one or more sensors, such as is shown and described herein.”, Column 29 line 63 - Column 30 line 11, “A user may enter commands and information into computer system 300 through one or more input devices 320, such as a pointing device ( e.g., a mouse, a touch screen, etc.), a keyboard, a microphone, a joystick, a game pad, a scanner, and the like. For example, the user can employ input device 320 to edit or modify the data being input into a data processing algorithm or method. These and other input devices 320 may be connected to processing unit 301 through a corresponding input device interface or port 322 that is operably coupled to the system bus, but may be connected by other interfaces or ports, such as a parallel port, a serial port, or a universal serial bus (USB). One or more output devices 324 (e.g., display, a monitor, a printer, a projector, or other type of display device) may also be operably connected to system bus 303 via interface 326, such as through a video adapter.”. The cited passages clearly show that the system includes a display apparatus configured to display information.).
Correll in view of Koga in further view of Miike in further view of Gombolay does not teach wherein the control unit is configured to be connected to a teaching apparatus that is configured to perform teaching by manipulating the posture of the end effector
in a case where the posture of the end effector has been changed, calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus and display the brightness on the display apparatus.
Suzuki, in the same field of endeavor, teaches wherein the control unit is configured to be connected to a teaching apparatus that is configured to perform teaching by manipulating the posture of the end effector (Suzuki: Column 4 lines 9-18, “FIG. 1 is an explanatory diagram of a robot system of a first embodiment. A robot system 100 includes a robot 200, a stereo camera 300 which is one example of vision sensors, a workpiece holding jig 400, a robot control device 500, a sensor control device 600, a display 700, a teaching pendant 800, and an input device 850. The robot 200 includes a robot arm 201, and a robot hand 202 which is one example of end effectors. The robot arm 201 is a vertically articulated robot arm. The base end of the robot 200, that is, the base end of the robot arm 201 is fixed to a stand 150.”, Column 4 lines 39-53, “The teaching pendant 800 is used to send a command to the robot control device 500 through the operation by a user. Upon receiving the command, the robot control device 500 causes the robot 200 to move, depending on the command. That is, by operating the teaching pendant 800, the user can move the robot arm 201, and perform a jog feed on the robot hand 202 in any direction and at any speed. Also, by operating the teaching pendant 800, the user allows the robot 200 to open and close the fingers 211 and 212 of the robot hand 202.”, Column 20 lines 57-67, “Thus, the process sets the posture of the robot 200, taken at the time, as the teach point (S211: setting step, setting process). That is, the robot control unit 521 causes the storage unit 522 to store the current posture of the robot arm 201 as the teach point information 511.”. The cited passages clearly teach a teaching apparatus used to teach a robot by manipulating the posture of the robot.).
Correll in view of Koga in further view of Miike in further view of Gombolay teaches a robot system comprising a display apparatus configured to display information. Correll in view of Koga in further view of Miike in further view of Gombolay does not teach wherein the control unit is configured to be connected to a teaching apparatus that is configured to perform teaching by manipulating the posture of the end effector. Suzuki teaches wherein the control unit is configured to be connected to a teaching apparatus, that is configured to perform teaching by manipulating the posture of the end effector. A person of ordinary skill in the art would have had the technological capabilities required to have modified the robot system taught in Correll in view of Koga in further view of Miike in further view of Gombolay with wherein the control unit is configured to be connected to a teaching apparatus that is configured to perform teaching by manipulating the posture of the end effector taught in Suzuki. Furthermore, the system taught in Correll in view of Koga in further view of Miike in further view of Gombolay already teaches an input device that allows to provide inputs to a robot. A person of ordinary skill in the art would have been able to modify the system taught in Correll in view of Koga in further view of Miike in further view of Gombolay with the ability to input instruction used to teach a robot by manipulating the posture of the robot as taught in Suzuki according to known methods. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a robot system comprising: wherein the control unit is configured to be connected to a teaching apparatus that is configured to perform teaching by manipulating the posture of the end effector.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot system taught in Correll in view of Koga in further view of Miike in further view of Gombolay with wherein the control unit is configured to be connected to a teaching apparatus that is configured to perform teaching by manipulating the posture of the end effector taught in Suzuki with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded the predictable results.
Correll in view of Koga in further view of Miike in further view of Gombolay in further view of Suzuki does not teach in a case where the posture of the end effector has been changed, calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus and display the brightness on the display apparatus.
Chung, in the same field of endeavor, teaches in a case where the posture of the end effector has been changed, calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus and display the brightness on the display apparatus (Chung: ¶ 0032, “The optical unit 120 includes a laser light generator 121 that emits a laser light LB and a light receiver 122 that receives reflected light RB of the laser light LB reflected from the correction plate 110 can do.”, ¶ 0034, “he light receiver 122 receives the reflected light RB of the laser light LB reflected from the correction plate 110 and can measure the light amount. The photodetector 122 can receive the reflected light RB reflected by at least one of the correction groove 111 and the inclined surface 112 of the correction plate 110. [ When the robot MAR is operated at the normal position, the light receiver 122 can receive the reflected light RB reflected by the correction groove 111. On the other hand, when the robot MAR is operated at a non-normal position, that is, when a position error of the robot MAR occurs, reflected light RB reflected from the correction groove 111, (RB) can also be received.”, ¶ 0042, “Here, the area of interest IA means a part of the area of the photoreceptor 122 which is photographed by the image pickup unit 130, and may mean an area displayed on the image generated by the image pickup unit 130 . That is, the image generated by the image pickup unit 130 may be an image displayed on a display screen that can be visually confirmed by the user through digital conversion. In this case, a part of the light receiver 122 The region may refer to the region of interest (IA).”. The cited passages clearly teach that the system is configured to measure the amount of light reflected.).
Correll in view of Koga in further view of Miike in further view of Gombolay in further view of Suzuki teaches a robot system comprising: a display apparatus configured to display information. Correll in view of Koga in further view of Miike in further view of Gombolay in further view of Suzuki does not teach in a case where the posture of the end effector has been changed, calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus and display the brightness on the display apparatus. Chung teaches in a case where the posture of the end effector has been changed, calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus and display the brightness on the display apparatus. A person of ordinary skill in the art would have had the technological capabilities required to have modified the robot system taught in Correll in view of Koga in further view of Miike in further view of Gombolay in further view of Suzuki with in a case where the posture of the end effector has been changed, calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus and display the brightness on the display apparatus taught in Chung. Furthermore, the system taught in Correll in view of Koga in further view of Miike in further view of Gombolay in further view of Suzuki is already configured to control the robot by illuminating the object using the illumination device and determining the pattern of illuminated light using an imaging device. As such, the system taught in Correll in view of Koga in further view of Miike in further view of Gombolay in further view of Suzuki is readily configurable for the method taught in Chung. A person of ordinary skill in the art would have been easily able to modify the system taught in Correll in view of Koga in further view of Miike in further view of Gombolay in further view of Suzuki with the methods used to detect the amount of light reflected and control the robot based on this taught in Chung. Additionally, the system taught in Correll in view of Koga in further view of Miike in further view of Gombolay in further view of Suzuki is configured to display data to a display device. A person of ordinary skill in the art would have been easily able to modify the device to display the calculated brightness taught in Chung according to known methods. Such modifications would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a robot system comprising: in a case where the posture of the end effector has been changed, calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus and display the brightness on the display apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot system taught in Correll in view of Koga in further view of Miike in further view of Gombolay in further view of Suzuki with in a case where the posture of the end effector has been changed, calculate brightness of the light reflected at the flat surface portion from the image captured by the imaging apparatus and display the brightness on the display apparatus taught in Chung with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Claim(s) 14-15, 22 and 27 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 11148295 B2 ("Correll") in view of US 11679508 B2 ("Koga") in further view of US 2023/0086122 A1 ("Gombolay") in further view of JP 2016120545 A ("Miike").
Regarding claim 22, Correll teaches a robot system comprising (Correll: Abstract, “Disclosed are various embodiments of a three-dimensional perception and object manipulation robot gripper configured for connection to and operation in conjunction with a robot arm. In some embodiments, the gripper comprises a palm, a plurality of motors or actuators operably connected to the palm, a mechanical manipulation system operably connected to the palm, a plurality of fingers operably connected to the motors or actuators and configured to manipulate one or more objects located within a workspace or target volume that can be accessed by the fingers. A depth camera system is also operably connected to the palm. One or more computing devices are operably connected to the depth camera and are configured and programmed to process images provided by the depth camera system to determine the location and orientation of the one or more objects within a workspace, and in accordance therewith, provide as outputs therefrom control signals or instructions configured to be employed by the motors or actuators to control movement and operation of the plurality of fingers so as to permit the fingers to manipulate the one or more objects located within the workspace or target volume. The gripper can also be configured to vary controllably at least one of a force, a torque, a stiffness, and a compliance applied by one or more of the plurality of fingers to the one or more objects.”):
an end effector including an imaging apparatus (Correll: Figure 3 camera sensors 32 and 34, Column 9 lines 8-30, “FIG. 3 shows one embodiment of gripper 20 operably attached to robotic wrist 14. In one embodiment, and in conjunction with light projector 36, camera sensors 32 and 34 are configured to provide stereo-vision capabilities for gripper 20, more about which is said below.”, Column 10 lines 6-31, “In the embodiment shown in FIG. 5, palm 22 also comprises upper plate 24, to which are mounted or attached, or disposed upon or to, various com ponents of gripper 20 such as upper enclosure 38a, depth camera system 30, third computing device 54, motors 61a and 61b, and motor shafts 62a and 62b. in one embodiment, sandwiched between upper plate 24 and lower plate 26 of palm 22 are at least some of the components comprising mechanical manipulation system 40.”)
and an irradiation unit configured to irradiate light in an optical axis direction (Correll: Figures 3 and 6a-b, Column 9 lines 8-30, “FIG. 3 shows one embodiment of gripper 20 operably attached to robotic wrist 14. In one embodiment, and in conjunction with light projector 36, camera sensors 32 and 34 are configured to provide stereo-vision capabilities for gripper 20, more about which is said below.”, Column 14 lines, 24-37, “In one embodiment, gripper 20 includes a depth camera 30 that comprises appropriate infrared camera sensors 32 and 34 along with s suitable infrared projector 36, where depth camera 30 is mounted on palm 22 of gripper 20. The stereo camera sensors 32 and 34 can be configured to be sensitive to the visible light spectrum (monochrome), to the infrared light spectrum, or to both spectra.”, Column 19 lines 1-28, “Among other things, workspace or target volume 74 of FIGS. 6a and 6b is defined by the capabilities of depth camera 30, which depends upon angle 33 over which light from projector 36 can be reflected or backscattered from an object 2 back towards camera sensors 32 and 34 and detected thereby, and the distance between projector 36 and camera sensors 32 and 34 on the one hand, and object 2 (not shown in FIGS. 6a and 6b).”. The cited passages clearly show a irradiation unit irradiating light in an optical axis direction.);
a robot to which the end effector is attached, the robot being configured to move a position and posture of the end effector (Correll: Column 8 lines 4-23, “FIG. 1 shows robot 10 engaged in a "bin picking" task, which involves retrieving items such as screws from bin 25. Here, robot 10 is shown in a configuration that puts gripper 20 comprising cameras 32 and 34 and a light illumination device 36 (not shown in FIG. 1) above bin 4, which in the illustrative embodiment of FIG. 1 features a workspace or target volume within which object(s) are situated or located, 10 where the workspace or target volume has an extent or distance associated therewith which ranges between about 11 cm and about 30 cm from the forward or sensing portion of gripper 20.”, Column 27 lines 52-58, “FIG. 14 shows one embodiment of a method 200 that may be employed using gripper 20. According to one embodiment, method 200 comprises step 201 of determining the position and characteristics of object 2, step 203 of actuating and controlling one or more of any of motors 41, mechanical manipulation system 40, wrist 14, arm 12, and robot 10, and step 205 of grasping, manipulating, and/or moving object 2.”. The cited passages clearly show that the end effector is attached to a robot and wherein the robot is configured to move the end effector.); and
a control unit configured to control the robot and the end effector (Correll: Column 3 line 25 – Column 4 line 17, “… a second computing device mounted to, on or in, or connected operably to, the palm, the second computing device being operably connected to the first computing device and to the plurality of at least one of motors and actuators, the second computing device being configured to receive the output images provided by the first computing device, and further being configured and programmed to process the output images to determine the location and orientation of the one or more objects within the workspace, and in accordance therewith provide as outputs therefrom control signals or instructions configured to be employed by the motors or actuators to control movement and operation of the plurality of fingers so as to permit the fingers to manipulate the one or more objects located within the workspace or target volume,”, Column 9 line 48 – Column 10 line 5, “Further as shown in FIG. 5, in one embodiment, second computing device 52 comprises a processor, controller, microcontroller, and/or CPU (such as a system-on-a-module) attached to the underside of palm 22, while peripheral break-out device 51 comprising, for example, Ethernet and USB connections, is likewise attached to the underside of palm 22.”).
Correll does not teach wherein, when performing alignment control to align the end effector with respect to an imaging object based on an image captured by the imaging apparatus,
the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus.
Koga, in the same field of endeavor, teaches wherein, when performing alignment control to align the end effector with respect to an imaging object based on an image captured by the imaging apparatus (Koga: Column 4 line 55 – Column 5 line 2, “In the first control of the present embodiment, the pins 92a and 93a are controlled so as to be aligned with the holes 82a and 83a. The orientation of the workpiece 91 with respect to the workpiece 81 is adjusted before the first control is performed. In other words, the orientation of the robot 1 is adjusted so that the pins 92a and 93a of the workpiece 91 are arranged on a straight line extending in the vertical direction. Thus, in the first control, the control for arranging the pin 92a directly above the hole 82a is performed so that the pins 92a and 93a can be aligned with the holes 82a and 83a. After the first control of the present embodiment, a downward movement of the workpiece 91 in the vertical direction indicated by arrow 103 causes the pins 92a, 93a to be inserted into the holes 82a, 83a, and causes the workpiece 91 to be attached to the workpiece 81.”, Column 7 lines 57-67, “At step 111, a reference image for performing the first control is generated. FIG. 5 shows a reference image for performing the first control of the present embodiment. The reference image 61 corresponds to an image captured by the camera 25 when the second workpiece 91 is arranged at the target position with respect to the first workpiece 81. In the present embodiment, the reference image 61 is an image captured by the camera 25 when the pins 92a and 93a of the workpiece 91 are arranged directly above the holes 82a and 83a of the workpiece 81. The reference image 61 can be prepared by an operator and stored in the storage part 42.”, Column 10 lines 48-58, “Subsequently, the command generation unit 34 of the image processing unit 31 generates a movement command for the robot 1 so that the second workpiece 91 is arranged at the target position with respect to the first workpiece 81, based on the relative position amount in the image 62 captured by the camera 25 and the relative position amount in the reference image 61. The command generation unit 34 of the present embodiment generates a movement command for operating the robot 1 so that the relative position amount in the image 62 captured by the camera 25 approaches the relative position amount in the reference image 61.”. The cited passages clearly show that the system is configured to align the workpieces before performing an attachment operation on said workpieces.).
Correll teaches a robot system comprising: an end effector including an imaging apparatus and an irradiation unit configured to irradiate light in an optical axis direction; a robot to which the end effector is attached, the robot being configured to move a position and posture of the end effector; and a control unit configured to control the robot and the end effector. Correll does not teach wherein, when performing alignment control to align the end effector with respect to an imaging object based on an image captured by the imaging apparatus. Koga teaches wherein, when performing alignment control to align the end effector with respect to an imaging object based on an image captured by the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities required to have modified the robot system taught in Correll with wherein, when performing alignment control to align the end effector with respect to an imaging object based on an image captured by the imaging apparatus taught in Koga. Furthermore, the robot system taught in Correll is already configured to use visual servoing methods in the control of the end effector and is also configured to convey and manipulate a workpiece. As such the visual servoing methods taught in Koga could easily be added to Correll using methods known to one of ordinary skill in the art. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of an robot system comprising: wherein, when performing alignment control to align the end effector with respect to an imaging object based on an image captured by the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot system taught in Correll with wherein, when performing alignment control to align the end effector with respect to an imaging object based on an image captured by the imaging apparatus taught in Koga with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Correll in view of Koga does not teach the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus.
Gombolay, in the same field of endeavor, teaches the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus (Gombolay: Figure 11 countersunk classification test 1102, ¶ 0092, “The study provided an approach to applying supervised and reinforcement learning techniques with a collaborative robot to complete drilling and fastening tasks in final aerospace assembly. The system tasks included inserting fasteners into chamfered and unchamfered holes on a custom-built section of an aircraft fuselage.”, ¶ 0093, “Example System Layout. For this task, the study used a custom-built section of an aircraft fuselage, known as a "coupon", as a test bed. The coupon was 0.9 m by 0.9 m and contained over one-hundred drilled holes. Each hole was either a "countersunk" hole with a chamfered rim or a "buttonhead" hole without a rim.”, ¶ 0127, “Using flat metal sheets lined with countersunk and buttonhead holes, the study tested the classifier to label the video feed in FIG. 11 real-time. shows the results of the classification. 1102, 1104 Diagrams show the classification labels "c" and "b" superimposed over the countersunk holes and the buttonhead holes. The number shows the estimated size of the 1106 hole (in microns). Plot shows a confusion matrix from 1000 one-shot classification tasks completed by the SNN trained on 30 samples.”. One of ordinary skill in the art would have recognized from the cited passages, that as long as the camera is at a position that is roughly parallel to the center axis of the holes, that the light reflected from the flat portions would be reflected to towards the camera and the light reflected from the inclined portion (i.e. the chamfer) would be reflected away from the camera.).
Correll in view of Koga teaches a robot system configured to perform an alignment operation and illuminate an object using an irradiation unit. Correll in view of Koga does not teach the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus. Gombolay teaches the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities required to have modified the robot system taught in Correll in view of Koga with the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus taught in Gombolay. Furthermore, the system taught in Correll in view of Koga is already configured to perform a peg-and-hole type insertion task (Correll: Column 16 line 60 – Column 17 line 10, “Another typical application is the "peg-in-hole" problem, which requires inserting a peg-like object 2, for example a bolt or a screw, into a hole. This task requires locating the hole as well as the object 2 that needs to be inserted. Object 2 must be grasped so that its centerline aligns with that of gripper 20, or-at the very least-so that its exact pose within gripper 20 can be expected to be known. While accurate localization and placement is not sufficient for this task ( see below), there is agreement among practitioners that accurate localization and placement are beneficial to quickly and reliably complete a peg-in-hole task. Assuming the task is to insert a bolt of 5 mm diameter into a hole having an appropriate size of 5.2 mm, both the bolt and the hole (assuming a top-down view) span across approximately 14 pixels for a total area of about 152 pixels, allowing gripper 20 to localize both the screw and the hole with submillimeter accuracy and permitting both an accurate grasp of the object and subsequent placement above the hole.”). As such, modifying the imagining object to include a flat portion and an inclined portion as taught in Gombolay would only require the simple modification of a chamfer to the hole already taught in Correll in view of Koga. This modification would be well within the technological capabilities of a person of ordinary skill in the art. Modifying the hole to include a chamfer would not change or introduce new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a robot system comprising: with the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot system taught in Correll in view of Koga with the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus taught in Gombolay with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Correll in view of Koga in further view of Gombolay does not teach the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus.
Miike, in the same field of endeavor, teaches the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus (Miike: Abstract, “PROBLEM TO BE SOLVED: To provide a novel handling device that does not need lateral movement of a robot arm after an imaging portion images a workpiece, when the device holds the workpiece with fingers. SOLUTION: A handling device 1 according to the present invention includes: an imaging portion 2 for imaging a workpiece H to obtain geometric information of the workpiece H; a holding portion 6 for securely holding the imaging portion 2, and having a mounting portion 8 to be mounted on a robot arm R; and multiple fingers 10 provided movably with respect to the holding portion 6 through a drive portion 9, for holding the workpiece H. When the device images the workpiece H, the drive portion 9 moves the fingers 10 to the outside of an imaging area A1 of the imaging portion 2. Additionally, when the device holds the workpiece H, the drive portion can move the fingers 10 from the outside of the imaging area A1 of the imaging portion 2 to the inside of the imaging area A1, on the basis of the geometric information of the workpiece H”, ¶ 0010, “The handling device of the present invention moves the fingers outside the imaging area of the imaging unit when imaging the workpiece, but can move the fingers from outside the imaging area to inside the imaging area when holding the workpiece.”, ¶ 0021, “A drive unit 9 and a plurality of fingers 10 movably provided via the drive unit 9 are provided on the lower surface (the surface on the opposite side where the mounting unit 8 is located) of the holding unit 6. The finger 10 of the present embodiment is composed of four of the first finger 11 to the fourth finger 14. Further, claws 11 a to 14 a extending in a direction orthogonal to the axial direction of the fingers are provided at the tips of the first finger 11 to the fourth finger 14. The claws 11 a to 14 a of the present embodiment are provided to extend on both sides of the axis of each finger.”, ¶ 0023, “The first to fourth fingers 11 to 14 are held by the movable elements 20, 21, 23, and 24, respectively, and the first to fourth fingers 11 to 14 are movable in the X and Y directions from the inside to the outside of the imaging area A1 shown in FIG.”. The cited passages clearly shows that the fingers of the robot are configured to move in the X and Y directions, both when gripping the object and when not. As is clearly stated this allows the robot to move the gripped object in and out of the imaging range or the imaging apparatus. This clearly teaches that the robot is configured to move the object in and out of the imaging apparatus without moving the base or the imaging apparatus.).
Correll in view of Koga in further view of Gombolay teaches a robot system configured to perform an alignment operation and illuminate an object using an irradiation unit, the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus. Correll in view of Koga in further view of Gombolay does not teach the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus. Miike teaches the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities to modify the robot system taught in Correll in view of Koga in further view of Gombolay with the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus taught in Miike. Furthermore, the fingers of the end effector taught in Correll in view of Koga in further view of Gombolay are already configured to move inside and outside of the field of view of the imaging device without causing the base or the imaging device to move. Additionally, one of ordinary skill in the art would recognize that the fingers are configured to move and object within the field of view of the imaging device and even partially outside of the field of view of the imaging device. As such, the end effector is readily modifiable to move the workpiece outside of the field of view in addition to the fingers, without moving the base or imaging device, as taught in Miike using known methods. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a robot system comprising: the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the robot system taught in Correll in view of Koga in further view of Gombolay with the control unit is configured to capture the image of an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the imaging apparatus in a manner of irradiating the imaging object by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus taught in Miike with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Regarding claim 27, Correll teaches a control method of a robot system including (Correll: Abstract, “Disclosed are various embodiments of a three-dimensional perception and object manipulation robot gripper configured for connection to and operation in conjunction with a robot arm. In some embodiments, the gripper comprises a palm, a plurality of motors or actuators operably connected to the palm, a mechanical manipulation system operably connected to the palm, a plurality of fingers operably connected to the motors or actuators and configured to manipulate one or more objects located within a workspace or target volume that can be accessed by the fingers. A depth camera system is also operably connected to the palm. One or more computing devices are operably connected to the depth camera and are configured and programmed to process images provided by the depth camera system to determine the location and orientation of the one or more objects within a workspace, and in accordance therewith, provide as outputs therefrom control signals or instructions configured to be employed by the motors or actuators to control movement and operation of the plurality of fingers so as to permit the fingers to manipulate the one or more objects located within the workspace or target volume. The gripper can also be configured to vary controllably at least one of a force, a torque, a stiffness, and a compliance applied by one or more of the plurality of fingers to the one or more objects.”)
an end effector including an imaging apparatus configured to perform imaging (Correll: Figure 3 camera sensors 32 and 34, Column 9 lines 8-30, “FIG. 3 shows one embodiment of gripper 20 operably attached to robotic wrist 14. In one embodiment, and in conjunction with light projector 36, camera sensors 32 and 34 are configured to provide stereo-vision capabilities for gripper 20, more about which is said below.”, Column 10 lines 6-31, “In the embodiment shown in FIG. 5, palm 22 also comprises upper plate 24, to which are mounted or attached, or disposed upon or to, various com ponents of gripper 20 such as upper enclosure 38a, depth camera system 30, third computing device 54, motors 61a and 61b, and motor shafts 62a and 62b. in one embodiment, sandwiched between upper plate 24 and lower plate 26 of palm 22 are at least some of the components comprising mechanical manipulation system 40.”)
and an irradiation unit configured to irradiate light in an optical axis direction (Correll: Figures 3 and 6a-b, Column 9 lines 8-30, “FIG. 3 shows one embodiment of gripper 20 operably attached to robotic wrist 14. In one embodiment, and in conjunction with light projector 36, camera sensors 32 and 34 are configured to provide stereo-vision capabilities for gripper 20, more about which is said below.”, Column 14 lines, 24-37, “In one embodiment, gripper 20 includes a depth camera 30 that comprises appropriate infrared camera sensors 32 and 34 along with s suitable infrared projector 36, where depth camera 30 is mounted on palm 22 of gripper 20. The stereo camera sensors 32 and 34 can be configured to be sensitive to the visible light spectrum (monochrome), to the infrared light spectrum, or to both spectra.”, Column 19 lines 1-28, “Among other things, workspace or target volume 74 of FIGS. 6a and 6b is defined by the capabilities of depth camera 30, which depends upon angle 33 over which light from projector 36 can be reflected or backscattered from an object 2 back towards camera sensors 32 and 34 and detected thereby, and the distance between projector 36 and camera sensors 32 and 34 on the one hand, and object 2 (not shown in FIGS. 6a and 6b).”. The cited passages clearly show a irradiation unit irradiating light in an optical axis direction.),
a robot to which the end effector is attached, the robot being configured to move a position and posture of the end effector (Correll: Column 8 lines 4-23, “FIG. 1 shows robot 10 engaged in a "bin picking" task, which involves retrieving items such as screws from bin 25. Here, robot 10 is shown in a configuration that puts gripper 20 comprising cameras 32 and 34 and a light illumination device 36 (not shown in FIG. 1) above bin 4, which in the illustrative embodiment of FIG. 1 features a workspace or target volume within which object(s) are situated or located, 10 where the workspace or target volume has an extent or distance associated therewith which ranges between about 11 cm and about 30 cm from the forward or sensing portion of gripper 20.”, Column 27 lines 52-58, “FIG. 14 shows one embodiment of a method 200 that may be employed using gripper 20. According to one embodiment, method 200 comprises step 201 of determining the position and characteristics of object 2, step 203 of actuating and controlling one or more of any of motors 41, mechanical manipulation system 40, wrist 14, arm 12, and robot 10, and step 205 of grasping, manipulating, and/or moving object 2.”. The cited passages clearly show that the end effector is attached to a robot and wherein the robot is configured to move the end effector.), and
a control unit configured to control the robot and the end effector, the method comprising (Correll: Column 3 line 25 – Column 4 line 17, “… a second computing device mounted to, on or in, or connected operably to, the palm, the second computing device being operably connected to the first computing device and to the plurality of at least one of motors and actuators, the second computing device being configured to receive the output images provided by the first computing device, and further being configured and programmed to process the output images to determine the location and orientation of the one or more objects within the workspace, and in accordance therewith provide as outputs therefrom control signals or instructions configured to be employed by the motors or actuators to control movement and operation of the plurality of fingers so as to permit the fingers to manipulate the one or more objects located within the workspace or target volume,”, Column 9 line 48 – Column 10 line 5, “Further as shown in FIG. 5, in one embodiment, second computing device 52 comprises a processor, controller, microcontroller, and/or CPU (such as a system-on-a-module) attached to the underside of palm 22, while peripheral break-out device 51 comprising, for example, Ethernet and USB connections, is likewise attached to the underside of palm 22.”):
an imaging step in which the control unit captures the image of the imaging object, that is irradiated with the light at the irradiating step, by the imaging apparatus (Correll: Column 14 lines 3-23, “Referring now to FIGS. 3 through 5, in one embodiment robotic gripper 20 is equipped with two infrared camera sensors 32 and 34 and infrared light projector 36 mounted on palm 22 of gripper 20, which together serve as a stereo pair of cameras and an infrared illumination source that provides substantial depth perception capabilities. … . In such an embodiment, camera sensors 32 and 34 are configured to operate in the infrared light spectrum and record up to 1280x720 pixels at 90 Hertz. Infrared light projector 36 provides an illumination pattern that facilitates matching of stereo-pairs in environments with few graphical features.”, Column 19 lines 1-28, “Among other things, workspace or target volume 74 of FIGS. 6a and 6b is defined by the capabilities of depth camera 30, which depends upon angle 33 over which light from projector 36 can be reflected or backscattered from an object 2 back towards camera sensors 32 and 34 and detected thereby, and the distance between projector 36 and camera sensors 32 and 34 on the one hand, and object 2 (not shown in FIGS. 6a and 6b).”. The cited passages clearly show that the imagining device is configured to capture an image of the illuminated objected.).
Correll does not teach an irradiating step in which the control unit irradiates an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus;
a calculating step in which the control unit calculates a positional relationship between the end effector and the imaging object from the image captured by the imaging apparatus.
Koga, in the same field of endeavor, teaches a calculating step in which the control unit calculates a positional relationship between the end effector and the imaging object from the image captured by the imaging apparatus (Koga: Column 10 line 59 – Column 11 line 7, “At step 119, the command generation unit 34 calculates the difference between the relative position amounts, which is the difference between the relative position amount in the image 62 captured by the camera 25 and the relative position amount in the reference image 61. In the present embodiment, the command generation unit 34 calculates the difference between the relative position amounts by subtracting the relative position amount in the reference image 61 from the relative position amount in the image 62 captured by the camera 25. The difference between the relative position amounts can be represented by [(ulm-u2m)-(ulb-u2b), (vlm-v2m)-(vlb-v2b)] as a value for each of the u-axis and the v-axis. As described above, in the present embodiment, the difference between the relative position amounts related to the u-axis and the difference between the relative position amounts related to the v-axis are calculated.”).
Correll teaches a control method of a robot system including an end effector including an imaging apparatus configure to perform imaging and an irradiation unit configured to irradiate light in an optical axis direction, a robot to which the end effector is attached, the robot being configured to move a position and posture of the end effector, and a control unit configured to control the robot and the end effector, the method comprising: an imaging step in which the control unit captures the image of the imaging object, that is irradiated with the light at the irradiating step, by the imaging apparatus. Correll does not teach a calculating step in which the control unit calculates a positional relationship between the end effector and the imaging object from the image captured by the imaging apparatus. Koga teaches a calculating step in which the control unit calculates a positional relationship between the end effector and the imaging object from the image captured by the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities required to have modified the method taught in Correll with a calculating step in which the control unit calculates a positional relationship between the end effector and the imaging object from the image captured by the imaging apparatus taught in Koga. Furthermore, the method taught in Correll is already configured to use visual servoing methods in the control of the end effector and is also configured to convey and manipulate a workpiece. As such the visual servoing methods taught in Koga could easily be added to Correll using methods known to one of ordinary skill in the art. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a method comprising: a calculating step in which the control unit calculates a positional relationship between the end effector and the imaging object from the image captured by the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method taught in Correll with a calculating step in which the control unit calculates a positional relationship between the end effector and the imaging object from the image captured by the imaging apparatus taught in Koga with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Correll in view of Koga does not teach an irradiating step in which the control unit irradiates an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus.
Gombolay, in the same field of endeavor, teaches an irradiating step in which the control unit irradiates an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus (Gombolay: Figure 11 countersunk classification test 1102, ¶ 0092, “The study provided an approach to applying supervised and reinforcement learning techniques with a collaborative robot to complete drilling and fastening tasks in final aerospace assembly. The system tasks included inserting fasteners into chamfered and unchamfered holes on a custom-built section of an aircraft fuselage.”, ¶ 0093, “Example System Layout. For this task, the study used a custom-built section of an aircraft fuselage, known as a "coupon", as a test bed. The coupon was 0.9 m by 0.9 m and contained over one-hundred drilled holes. Each hole was either a "countersunk" hole with a chamfered rim or a "buttonhead" hole without a rim.”, ¶ 0127, “Using flat metal sheets lined with countersunk and buttonhead holes, the study tested the classifier to label the video feed in FIG. 11 real-time. shows the results of the classification. 1102, 1104 Diagrams show the classification labels "c" and "b" superimposed over the countersunk holes and the buttonhead holes. The number shows the estimated size of the 1106 hole (in microns). Plot shows a confusion matrix from 1000 one-shot classification tasks completed by the SNN trained on 30 samples.”. One of ordinary skill in the art would have recognized from the cited passages, that as long as the camera is at a position that is roughly parallel to the center axis of the holes, that the light reflected from the flat portions would be reflected to towards the camera and the light reflected from the inclined portion (i.e. the chamfer) would be reflected away from the camera.).
Correll in view of Koga teaches a method configured to determine a positional relationship between the end effector and imaging object. and illuminate the object using an irradiation unit. Correll in view of Koga does not an irradiating step in which the control unit irradiates an imaging object, that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus. Gombolay teaches an irradiating step in which the control unit irradiates an imaging object, that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities required to have modified the method taught in Correll in view of Koga with an irradiating step in which the control unit irradiates an imaging object, that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus taught in Gombolay. Furthermore, the method taught in Correll in view of Koga is already configured to perform a peg-and-hole type insertion task (Correll: Column 16 line 60 – Column 17 line 10, “Another typical application is the "peg-in-hole" problem, which requires inserting a peg-like object 2, for example a bolt or a screw, into a hole. This task requires locating the hole as well as the object 2 that needs to be inserted. Object 2 must be grasped so that its centerline aligns with that of gripper 20, or-at the very least-so that its exact pose within gripper 20 can be expected to be known. While accurate localization and placement is not sufficient for this task ( see below), there is agreement among practitioners that accurate localization and placement are beneficial to quickly and reliably complete a peg-in-hole task. Assuming the task is to insert a bolt of 5 mm diameter into a hole having an appropriate size of 5.2 mm, both the bolt and the hole (assuming a top-down view) span across approximately 14 pixels for a total area of about 152 pixels, allowing gripper 20 to localize both the screw and the hole with submillimeter accuracy and permitting both an accurate grasp of the object and subsequent placement above the hole.”). As such, modifying the imagining object to include a flat portion and an inclined portion as taught in Gombolay would only require the simple modification of a chamfer to the hole already taught in Correll in view of Koga. This modification would be well within the technological capabilities of a person of ordinary skill in the art. Modifying the hole to include a chamfer would not change or introduce new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a method comprising: an irradiating step in which the control unit irradiates an imaging object, that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the method taught in Correll in view of Koga an irradiating step in which the control unit irradiates an imaging object, that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus taught in Gombolay with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Correll in view of Koga in further view of Gombolay does not teach an irradiating step in which the control unit irradiates an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus.
Miike, in the same field of endeavor, teaches an irradiating step in which the control unit irradiates an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus (Miike: Abstract, “PROBLEM TO BE SOLVED: To provide a novel handling device that does not need lateral movement of a robot arm after an imaging portion images a workpiece, when the device holds the workpiece with fingers. SOLUTION: A handling device 1 according to the present invention includes: an imaging portion 2 for imaging a workpiece H to obtain geometric information of the workpiece H; a holding portion 6 for securely holding the imaging portion 2, and having a mounting portion 8 to be mounted on a robot arm R; and multiple fingers 10 provided movably with respect to the holding portion 6 through a drive portion 9, for holding the workpiece H. When the device images the workpiece H, the drive portion 9 moves the fingers 10 to the outside of an imaging area A1 of the imaging portion 2. Additionally, when the device holds the workpiece H, the drive portion can move the fingers 10 from the outside of the imaging area A1 of the imaging portion 2 to the inside of the imaging area A1, on the basis of the geometric information of the workpiece H”, ¶ 0010, “The handling device of the present invention moves the fingers outside the imaging area of the imaging unit when imaging the workpiece, but can move the fingers from outside the imaging area to inside the imaging area when holding the workpiece.”, ¶ 0021, “A drive unit 9 and a plurality of fingers 10 movably provided via the drive unit 9 are provided on the lower surface (the surface on the opposite side where the mounting unit 8 is located) of the holding unit 6. The finger 10 of the present embodiment is composed of four of the first finger 11 to the fourth finger 14. Further, claws 11 a to 14 a extending in a direction orthogonal to the axial direction of the fingers are provided at the tips of the first finger 11 to the fourth finger 14. The claws 11 a to 14 a of the present embodiment are provided to extend on both sides of the axis of each finger.”, ¶ 0023, “The first to fourth fingers 11 to 14 are held by the movable elements 20, 21, 23, and 24, respectively, and the first to fourth fingers 11 to 14 are movable in the X and Y directions from the inside to the outside of the imaging area A1 shown in FIG.”. The cited passages clearly shows that the fingers of the robot are configured to move in the X and Y directions, both when gripping the object and when not. As is clearly stated this allows the robot to move the gripped object in and out of the imaging range or the imaging apparatus. This clearly teaches that the robot is configured to move the object in and out of the imaging apparatus without moving the base or the imaging apparatus.).
Correll in view of Koga in further view of Gombolay teaches a method configured to determine a positional relationship between the end effector and imaging object. and illuminate the object using an irradiation unit, an irradiating step in which the control unit irradiates an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus. Correll in view of Koga in further view of Gombolay does not teach an irradiating step in which the control unit irradiates an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus. Miike teaches an irradiating step in which the control unit irradiates an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus. A person of ordinary skill in the art would have had the technological capabilities to modify the control method of a robot system taught in Correll in view of Koga in further view of Gombolay with an irradiating step in which the control unit irradiates an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus taught in Miike. Furthermore, the fingers of the end effector taught in Correll in view of Koga in further view of Gombolay are already configured to move inside and outside of the field of view of the imaging device without causing the base or the imaging device to move. Additionally, one of ordinary skill in the art would recognize that the fingers are configured to move and object within the field of view of the imaging device and even partially outside of the field of view of the imaging device. As such, the end effector is readily modifiable to move the workpiece outside of the field of view in addition to the fingers, without moving the base or imaging device, as taught in Miike using known methods. Such a modification would not have changed or introduced new functionality. No inventive effort would have been required. The combination would have yielded the predictable result of a control methods of a robot system comprising: an irradiating step in which the control unit irradiates an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to have combine the control method of a robot system taught in Correll in view of Koga in further view of Gombolay with an irradiating step in which the control unit irradiates an imaging object that includes a flat surface portion formed in a planar shape and an inclined portion inclined with respect to the flat surface portion, by the irradiation unit, without moving either the robot or the imaging apparatus, such that an optical axis of reflected light reflected at the flat surface portion is directed toward the imaging apparatus and an optical axis of reflected light reflected at the inclined portion is directed toward a direction away from the imaging apparatus taught in Miike with a reasonable expectation of success. One of ordinary skill in the art would have been motivated to make this modification because the combination would have yielded predictable results.
Response to Arguments
Applicant’s arguments with respect to the amended independent claim(s) 1, 11, 22-24, and 27 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Noah W Stiebritz whose telephone number is (571)272-3414. The examiner can normally be reached Monday thru Friday 7-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached at (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.W.S./ Examiner, Art Unit 3658
/Ramon A. Mercado/Supervisory Patent Examiner, Art Unit 3658