Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed 8/21/2025 has been entered. Claims 1-10 remain pending in the application, and new claims 11-16 have been added. Applicant’s amendments to the claims no longer invoke interpretation under 35 U.S.C. 112(f), and have overcome rejection under 35 U.S.C. 101 previously set forth in the Non-Final Office Action mailed 05/21/2025.
Response to Arguments
Applicant argues that the cited references do not disclose the newly amended features involving “virtually moving” the end effector toward the appropriate position along a candidate harvesting direction.
Indeed, in view of the amendment, Applicant' s arguments with respect to independent claim(s) 1, 9, and 10 have been considered but are moot because the arguments do not apply to the combination of references and/or rationale being used in the current rejection.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-16 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The independent claims 1, 9, and 10 recite language for “determining whether or not one or more obstacles virtually interfere with an end effector”. One of ordinary skill in the art would not be able to determine the metes and bounds of this language in light of the specification. The terms “virtual” or “virtually” do not, under BRI, inherently describe whether a movement of an end effector is real or within a computer simulation. Moreover, the new language for “virtually interfere” and “virtually moved” do not appear in the specification at all.
The specification details steps for simulating movement of the end effector in P [0045]-[0046], as well as a process for changing the candidate harvesting direction if there is interference. Assuming that the concept of “virtually moved” refers to simulation, there appears to be a contradiction within the use of the term. The first step of claim 1 recites “determining whether or not the one or more obstacles virtually interfere with the end effector when the end effector is virtually moved toward the appropriate position along a candidate harvesting direction” and the second step recites “determining whether or not the one or more obstacles virtually interfere with the arm when the end effector is moved toward the appropriate position when it is determined that a virtual interference does not occur in the first step”.
It is clear that the first step is checking interference with the end effector directly and the second step is checking interference with the arm that moves the end effector. However, the second step recites checking for a “virtual interference” when the end effector is simply “moved toward the appropriate position”, implying a real movement of the arm to affect movement of the end effector. Therefore the determination of whether or not the obstacles “virtually interfere” cannot be a simulated result if the arm is being physically moved. This also introduces an antecedent basis issue within the dependent claims, as the dependent claims use terminology such as “the interference” or “interfering” but do not make it clear whether it is the same “virtual interference” of claim 1. As an example, dependent claim 2 introduces a fourth step of “causing the travel mechanism to move the main body to avoid the interference”, but the metes and bounds of the claim cannot be ascertained since it is not clear whether the interference in this step is an interference in simulation or in the physical environment.
Finally, the third step involves causing the end effector to perform harvesting of the object when virtual interference does not occur in the second step, but as shown above it is unclear in which steps the arm has actually moved the end effector physically versus simulating the movement “virtually”.
In the interest of compact prosecution, the claims will be interpreted in accordance with the Specification’s description of simulating movement of an arm before performing real movement as shown in the steps of FIG. 4. Although there is not currently a rejection applied, Applicant is additionally cautioned against the addition of new mater in light of the lack of the terminology involving a “virtual movement” or “virtual interference”.
Finally, dependent claims 2-8 and 11-16 are rejected for their dependency on a rejected base claim.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1, 9, and 10-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Schroll et al., hereinafter Schroll (Document ID: US 11202409 B1) in view of Moridaira (Document ID: US 20160158936 A1).
Regarding claims 1, 9, and 10, Schroll teaches a harvesting method, and a recording medium recording a program executed by a computer that controls a harvesting device, comprising:
an end effector that harvests an object to be harvested (end effectors 132, 134);
an arm that moves the end effector to an appropriate position for harvesting the object (robotic arms 122, 124);
a main body on which the end effector and the arm are provided (base 102);
a travel mechanism that moves the main body (rotational components 106); and
a controller (control system 108),
Schroll further teaches in at least FIG. 4 and FIG. 6 in step 606 a process for determining a cost for harvesting each object. The cost is based on “position of an object with respect to other objects to be harvested, approach angle needed to harvest object, the presence of any obstructions, ripeness of object, etc.” (Col 13, Line 5) But Schroll does not explicitly teach that
the controller executes:
a first step of determining whether or not the one or more obstacles virtually interfere with the end effector when the end effector is virtually moved toward the appropriate position along a candidate harvesting direction,
a second step of determining whether or not the one or more obstacles virtually interfere with the arm when the end effector is moved toward the appropriate position when it is determined that a virtual interference does not occur in the first step, and
Instead, Moridaira, whose invention pertains to avoiding collision with a robotic arm intersecting with an obstacle, teaches in P [0069]-[0070] a process including steps S104 and S106 for determining the target hand position and a determination whether collision occurs. The test is undertaken “by expressing the hand part 126 and the torso 110 by polygons and using an Oriented Bounding Box”, that is to say the check is performed virtually in simulation. P [0073] additionally specifies that step S104 for determining the target hand position involves a directionality of the hand. Additionally, Moridaira teaches in P [0072]: “when the collision of the hand part 126 does not occur (NO in S106), the control device 2 calculates the movable areas of the robot arm 120 in Step S108.” The movable areas are just kinematic limits, so lastly in the step S112 “the control device 2 determines the range in which the robot arm 120 does not collide against the torso 110 in the movable areas”. Note that P [0173] makes it clear that the obstacle detection and avoidance process applies for an obstacle that is not just the torso of the robot itself. The obstacle could be any object in range of the robot and plural in quantity.
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to have modified the cost analysis of arm movement for harvesting of Schroll with the coordinate based collision avoidance procedure of Moridaira in order to generate a collision free, planned operation of a robotic arm that navigates around obstacles in an environment and considers its own movement constraints. Although the system of Moridaira is not explicitly used in a harvesting scenario, one of ordinary skill in the art would understand that Schroll teaches the particulars of robot harvesting, and one of ordinary skill in the art would be motivated to include the particulars of collision avoidance and path planning taught by Moridaira to avoid damaging other objects to be harvested, as established in Col 2 Line 58 of Schroll.
Finally, in view of the modification, Schroll teaches
a third step of causing the end effector to harvest the object when it is determined that the virtual interference does not occur in the second step (see at least step 608 in FIG. 6 of Schroll wherein the object is harvested once they are obstruction free and ideal for harvesting).
Regarding claims 11, 13, and 15, modified Schroll teaches the harvesting device according to claim 1, the harvesting method according to claim 9, and the recording medium according to claim 10, and Schroll further teaches in Col 10 Line 40 that each fruit is assigned with an identifier based on its physical characteristics, and that each obstacle is similarly identified and categorized in Col 10 Line 54. The computer vision therefore allows for modeling and cost based determination about collision avoidance and optimal harvesting procedure. But Schroll does not explicitly teach that
in the first step, the controller executes:
simulating whether or not a mapped obstacle, mapped based on the one or more obstacles captured in an image, virtually interferes with the end effector when the end effector is virtually moved toward the appropriate position along the candidate harvesting direction;
updating the candidate harvesting direction when the mapped obstacle virtually interferes with the end effector; and
determining the candidate harvesting direction as a harvesting direction of the end effector when the mapped obstacle does not virtually interfere with the end effector.
Instead Moridaira teaches in P [0069]-[0070] a first step for simulating a target hand position through instruction of virtual coordinate data and checking for interference. The test for interference is undertaken “by expressing the hand part 126 and the torso 110 by polygons and using an Oriented Bounding Box”, that is to say the check is performed virtually in simulation. P [0073] additionally specifies that step S104 for determining the target hand position involves a directionality of the hand. In P [0071] “the control device 2 performs processing for correcting the target hand position in Step S140” when the obstacle virtually interferes with the hand, and otherwise the target hand position and direction are taken to be the hand position and direction to be used.
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to have modified the cost analysis of arm movement for harvesting of Schroll with the target hand positioning and correction of Moridaira in order to generate a collision free, planned operation of a robotic hand that navigates around obstacles in an environment and considers its own movement constraints.
Regarding claims 12, 14, and 16, modified Schroll teaches the harvesting device according to claim 11, the harvesting method according to claim 13, and the recording medium according to claim 15, and Schroll further teaches in Col 10 Line 40 that each fruit is assigned with an identifier based on its physical characteristics, and that each obstacle is similarly identified and categorized in Col 10 Line 54. The computer vision therefore allows for modeling and cost based determination about collision avoidance and optimal harvesting procedure. But Schroll does not explicitly teach that
in the second step, the controller determines whether or not the one or more obstacles virtually interfere with the arm by:
determining a harvesting pose based on the harvesting direction and a dimension of the arm; and
simulating, based on the harvesting pose, whether or not the mapped obstacle virtually interferes with the arm when the end effector is virtually moved toward the appropriate position along the harvesting direction.
Instead, Moridaira teaches in P [0072]: “when the collision of the hand part 126 does not occur (NO in S106), the control device 2 calculates the movable areas of the robot arm 120 in Step S108.” The movable areas are just kinematic limits, so lastly in the step S112 “the control device 2 determines the range in which the robot arm 120 does not collide against the torso 110 in the movable areas”. Note that P [0173] makes it clear that the obstacle detection and avoidance process applies for an obstacle that is not just the torso of the robot itself. The obstacle could be any object in range of the robot and plural in quantity. Additionally, P [0125] shows that “the joint angle calculation unit 22 virtually sets the position of the robot arm 120” during the procedure for determining the movable area, and P [0177] further establishes the use of simulation for virtual movements before application to a real world scenario.
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to have modified the cost analysis of arm movement for harvesting of Schroll with the arm dimension and movement considerations of Moridaira in order to generate a collision free, planned operation of a robotic hand that navigates around obstacles in an environment and considers its own movement constraints.
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Schroll in view of Moridaira, and further in view of Feng et al., hereinafter Feng (Document ID: CN111216125A).
Regarding claim 2, modified Schroll teaches the harvesting device according to Claim 1, and Schroll further teaches that the base may be enabled to move with a plurality of rotational components in Col 5, Line 24, but Schroll and Moridaira do not explicitly teach that
when it is determined that the interference occurs in the second step, the controller further executes a fourth step of causing the travel mechanism to move the main body to avoid the interference between the one or more obstacles and the arm.
Instead, Feng, whose invention pertains to navigating a mobile robot with a mechanical arm in a narrow space, teaches in at least P [0023] and FIG. 1 a feedback control loop for detecting whether a robot main body or robot arm cause collision, and the ability to reorganize the robot arm or the mobile platform to avoid the collision.
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to have modified the mobile harvesting robot with obstacle avoidance of Schroll and Moridaira with the collision detection and mobile platform movement of Feng in order to navigate a narrow region of operation with minimal interference. The robot of Feng is not necessarily dedicated to harvesting an object, but the harvesting problem is essentially akin to the problem solved by Feng of navigating a narrow space. The many constraints posed by a group of objects that may be targets for harvesting is similar to the navigation of a narrow passage, wherein avoiding collision and interference is key. The modification would thus produce expected results in the field of the invention.
Claim(s) 3-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Schroll in view of Moridaira, and further in view of Hannya et al., hereinafter Hannya (Document ID: US 20180250824 A1).
Regarding claim 3, modified Schroll teaches the harvesting device according to Claim 1, and Schroll further teaches
an image imaging unit that captures an image (see at least Col 6, Line 3: “Sensor system 114 may be comprised of one or more cameras and/or the one or more other sensors and utilize computer vision algorithms for maneuvering and obstacle avoidance.”),
Schroll teaches obstacle avoidance and a base 102 that is designed to “move between the rows without contacting the existing infrastructure” in Col 5, Line 15. But Schroll and Moridaira do not explicitly teach specifying a specific obstacle interfering with the arm and determining at least one of a moving direction of the main body and a moving amount of the main body necessary for the arm to avoid the interference with the specific obstacle based on a position of the specific obstacle and a position of the arm.
Instead, Hannya, whose invention pertains to avoiding an incidental damage to an object based on robot arm movement limits, teaches a determination of a specific obstacle in at least P [0030] that is too close or interfering with a first arm 7 or second arm 8, and then performs a movement restriction in step S4 of FIG. 4 by identifying a moving direction of the body that allows for avoiding the interference.
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to have modified the collision avoidance for a harvesting robot of Schroll and Moridaira with the multiple arm link and robot body consideration for collision avoidance of Hannya in order to prevent the object from being sandwiched between the links as in P [0057] of Hannya.
Regarding claim 4, modified Schroll teaches the harvesting device according to Claim 3, and Schroll further teaches calculating the coordinates of a robotic arm and the stem in Col. 11 Line 11, keeping awareness of the area that the arm occupies. But Schroll does not explicitly teach that the controller generates second coordinate data indicating a second region occupied by the specific obstacle in the space-in the second step.
Instead, Moridaira teaches in P [0067] the generation of coordinate data for the torso as an “obstacle area” and in P [0072]-[0073] the step S108 calculates the movable area of the arm using coordinate data of the arm and the hand.
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to have modified the coordinate aware and obstacle measuring system of Schroll, Moridaira, and Hannya with the three dimensional coordinate measurement and shape recognition for an object to be grasped of Moridaira in order to avoid interference between a robotic grasping arm and obstacles in the environment by directly comparing coordinate information in a common reference frame.
Regarding claim 5, modified Schroll teaches the harvesting device according to Claim 4, and in view of the modification Schroll further teaches
wherein the first coordinate data and the second coordinate data are constituted by a plurality of first axes along a first direction in which the main body is movable and a plurality of second axes along a second direction intersecting the first direction (Col 5, Line 25 establishes a plurality of rotational components for moving the base 102 “in different directions, such as forward, backwards, left, right, and/or any combination thereof.” FIGs. 1 and 5 also clarify the ability to move in a 2D plane when viewed from above).
Regarding claim 6, modified Schroll teaches the harvesting device according to Claim 5, and in view of the modification Schroll further teaches
wherein the first coordinate data and the second coordinate data are further constituted by a plurality of third axes along a height direction of the harvesting device (see at least FIG. 3 which demonstrates the arm moveable in 3D space along a height direction in a picking environment. The items to be harvested will also necessarily occupy different locations in the vertical axis by nature of how they grow).
Regarding claim 7, modified Schroll teaches the harvesting device according to Claim 5, and Moridaira teaches the ability to consider each robotic arm segment in the interference analysis, but Schroll and Moridaira do not explicitly teach that
the controller, when the mapped obstacle virtually interferes with the arm in the second step:
specifies a first partial region occupied by a first portion of the arm in the space and a second partial region occupied by a second portion of the arm in the space,
calculates, by overlapping the first coordinate data and the second coordinate data, a first degree of overlapping that is a degree of overlapping between the first partial region and the second region,
calculates, by overlapping the first coordinate data and the second coordinate data, a second degree of overlapping that is a degree of overlapping between the second partial region and the second region,
determines a direction from the first partial region toward the second partial region as the moving direction of the main body when the first degree of overlapping is larger than the second degree of overlapping,
determines a direction from the second partial region toward the first partial region as the moving direction of the main body when the second degree of overlapping is larger than the first degree of overlapping,
Instead, Hannya teaches in at least FIGs. 2 and 4-6 as well as P [0028] monitoring the robotic arm as two partial regions made up of the first arm 7 and the second arm 8 and performing an interference avoidance operation that allows the system to move in a way that reduces the overlap. FIG. 9 additionally discloses monitoring each arm individually and acting to avoid damage to the object or the robotic arm when one arm has a greater degree of impending overlap or collision with the object.
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to have modified the collision avoidance for a harvesting robot with 3D coordinate detection of Schroll and Moridaira with the multiple arm link and robot body consideration for collision avoidance of Hannya in order to prevent the object from being sandwiched between the links as in P [0057] of Hannya.
Finally, in view of the modification, Schroll teaches
causes the travel mechanism to move the main body toward the moving direction. (see at least step 608 in FIG. 6 of Schroll wherein the object is harvested once they are obstruction free and ideal for harvesting).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Schroll in view of Moridaira and Hannya, and further in view of Watanabe et al., hereinafter Watanabe (Document ID: US 20200230821 A1), as well as Feng.
Regarding claim 8, modified Schroll teaches the harvesting device according to Claim 5, and Moridaira teaches in at least P [0017] the use of a cone shaped region for determining overlap between the arm and an obstacle, but Schroll, Moridaira, and Hannya do not explicitly teach
the controller, when the mapped obstacle virtually interferes with the arm in the second step:
overlaps the first coordinate data and the second coordinate data, measures a size of an overlapping region in which the first region and the second region are overlapped,
determines the moving amount of the main body based on the size of the overlapping region in the fifth step, and
Instead, Watanabe, whose invention pertains to avoiding interference when grasping a target object in a cluttered space, teaches in P [0059] and step S206 “A three-dimensional spatial overlap between each polygon and the sphere is detected to perform an interference determination.” The model representations for the objects and the robotic arm respectively each occupy a region in space for checking interference.
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to have modified the coordinate aware and obstacle measuring system of Schroll, Moridaira, and Hannya with the three dimensional coordinate measurement and shape recognition for an object to be grasped and overlap evaluation of Watanabe in order to avoid interference with a robotic grasping arm when picking items in a cluttered environment as in P [0021] of Watanabe.
Schroll, Moridaira, Hannya, and Watanabe do not explicitly teach determining the moving amount of the main body.
Instead, Feng teaches in P [0023] a process for determining collision when the robotic platform “moves one step from the current position” and then subsequently retreating that one step to replan when there is interference.
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to have modified the mobile harvesting robot with obstacle avoidance of Schroll, Moridaira, Hannya, and Watanabe with the collision detection and mobile platform movement of Feng in order to navigate a cluttered harvesting area with minimal interference.
Finally, in view of the modification, Schroll teaches
causes the travel mechanism to move the main body toward the moving direction. (see at least step 608 in FIG. 6 of Schroll wherein the object is harvested once they are obstruction free and ideal for harvesting).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Additional art made of record and not relied upon is considered pertinent to applicant's disclosure.
NPL Reference: Robotic Aubergine Harvesting Using Dual-Arm Manipulation
Work pertains to cooperative arm harvesting using vision technology to plan grasping.
NPL Reference: Smoothing obstacle avoidance path planning based on C-space for harvesting robot
Work pertains to modeling and planning for an apple harvesting mechanical arm with obstacle avoidance procedures.
NPL Reference: Graph-Based Cooperative Robot Path Planning in Agricultural Environments
Work pertains to utilizing a search arm in tandem with a grasping arm for finding and planning clear paths for harvesting fruit.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Dairon Estevez whose telephone number is (703)756-4552. The examiner can normally be reached M-R 6:30AM - 4:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Khoi Tran can be reached at (571) 272-6919. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.E./Examiner, Art Unit 3656
/KHOI H TRAN/Supervisory Patent Examiner, Art Unit 3656