DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
This application claims benefit of foreign priority under 35 U.S.C. 119(a)-(d) of KR10-2023-0161117, filed in Korea on 11/20/2023.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 7/5/2024 and 4/20/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered and attached by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
an extraction module in claims 1, 9, and 13 described in ¶31.
a space characteristic extraction module in claims 1, 9, and 13 described in ¶31.
a 3D model reconstruction module in claims 1, 9, and 13 described in ¶31.
an inference module in claims 1, 9, and 13-17 described in ¶31.
an operation control module in claim 9 described in ¶31.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3, 5-7, 9, 11-13, and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Hwang (KR20230069569A) in view of Zhang (U.S. Patent Pub. No. 2021/0056685).
Regarding Claim 1, Hwang teaches a method of extracting characteristics of smart farm corps in an edge device equipped in a robot, the method comprising (¶17 The fruit and vegetable segmentation and posture estimation method according to an embodiment of the present invention may further include transmitting information about the position and posture of the fruit and vegetable to the fruit and vegetable harvesting robot:)
a step of extracting depth information about a crop object by using an extraction module, based on a depth image and an RGB image, and extracting object information about the crop object, based on the RGB image (¶18 the present invention includes a camera that acquires a color image and a depth image by photographing the fruit and vegetable; a segmentation model generating a color image of fruit and vegetable by segmenting the fruit and vegetable from the acquired color image; a masking unit generating a fruit and vegetable depth image by masking the fruit and vegetable color image with the depth image;)
a step of extracting space characteristic information representing a shape, a size, and a direction of the crop object in a three-dimensional (3D) space by using a space characteristic extraction module, based on the depth information and the object information (¶40 The 3D data restoration unit 150 uses the fruit and vegetable color image generated by the segmentation model 130 and the fruit and vegetable depth image generated by the masking unit 140 to generate point cloud data on 3D space, which is 3D data of fruits and vegetables;)
a step of inferring volume information and pose information about the crop object by using an inference module, based on the reconstructed 3D model (¶46 The location information of fruits and vegetables calculated by the position calculation unit 160 and the posture information of fruits and vegetables estimated by the posture estimation unit 170 are transferred to a fruit and vegetable harvesting robot (not shown) and used to control the fruit and vegetable harvesting robot.)
Hwang does not explicitly disclose a step of reconstructing a 3D model of the crop object in the 3D space by using a 3D model reconstruction module, based on the space characteristic information; and a step of inferring volume information and pose information about the crop object by using an inference module, based on the reconstructed 3D model.
Zhang is in the same field of art of image analysis. Further, Zhang teaches a step of reconstructing a 3D model of the crop object in the 3D space by using a 3D model reconstruction module, based on the space characteristic information; and (¶129 The 3D point cloud is converted by encapsulation into a curved surface model formed by triangles, and hole parts in the surface of lettuce are filled; ¶130 Smoothing on the lettuce model is finally performed; ¶131 The biomass, leaf area, plant height, and stem diameter of lettuce are modeled.)
a step of inferring volume information and pose information about the crop object by using an inference module, based on the reconstructed 3D model (Fig. 4; ¶132 Volume Calculation; ¶134 Point cloud data of each layer of lettuce is projected onto an X-Y plane perpendicular to the plant height direction, and the data is segmented at equal intervals with a step length of a at the same time respectively in the X axis direction and the Y axis direction, to generate i×j pixel cell. Equation 1 and 2 are the formulas for the volume)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Hwang by reconstructing a 3d model and inferring the volume and pose that is taught by Zhang; thus, one of ordinary skilled in the art would be motivated to combine the references to accurately monitor the growth condition, nutrient information, and growth process of crop (Zhang ¶3).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 3, Hwang in view of Zhang discloses the method of claim 1, wherein the step of inferring the volume information and the pose information comprises a step of inferring the volume information about the crop object, based on the point cloud configuring the reconstructed 3D model (Zhang, Fig. 4; ¶132 Volume Calculation; ¶134 Point cloud data of each layer of lettuce is projected onto an X-Y plane perpendicular to the plant height direction, and the data is segmented at equal intervals with a step length of a at the same time respectively in the X axis direction and the Y axis direction, to generate i×j pixel cell. Equation 1 and 2 are the formulas for the volume)
The reasons for combining Hwang and Zhang are similar to that stated in the rejection of claim 1. In addition, this same reasoning is pertinent and applicable to the rejections of claim 6 below.
Regarding Claim 5, Hwang in view of Zhang discloses the method of claim 1, wherein the step of inferring the volume information and the pose information comprises:
a step of extracting characteristics of each point included in the point cloud configuring the reconstructed 3D model; and a step of predicting the pose information about the crop object, based on the extracted characteristics of each point (Hwang, ¶45 The posture estimator 170 calculates the posture of fruits and vegetables from the direction of the central axis of the created virtual cylinder. Since the central axis of the virtual cylinder and the central axis of the fruit/vegetable (the axis passing through the top and the base of the fruit/vegetable) coincide, the posture (direction) of the fruit/vegetable can be estimated from the direction of the central axis
Regarding Claim 6, Hwang in view of Zhang discloses the method of claim 5, wherein the pose information comprises position information including X, Y, and Z coordinates of the crop object with respect to the robot corresponding to a reference point in the 3D space (Zhang, ¶141 is assumed that the coordinates of any point in the point cloud data are f(x, y, z). It is only necessary to calculate the maximum value z.sub.max and the minimum value z.sub.min of the lettuce model in the Z axis direction. It is labeled that the coordinate point of the maximum value z.sub.max in this case is f(x.sub.1, y.sub.1, z.sub.1) and the coordinate point of the minimum value z.sub.min is f(x.sub.2, y.sub.2, z.sub.2).) and direction information including an X-axis rotation angle (Roll), a Y-axis rotation angle (Pitch), and a Z-axis rotation angle (Yaw) each representing a direction in which the reconstructed 3D model is inclined with respect to the robot corresponding to the reference point in the 3D space (Hwang ¶45 The posture estimator 170 calculates the posture of fruits and vegetables from the direction of the central axis of the created virtual cylinder. Since the central axis of the virtual cylinder and the central axis of the fruit/vegetable (the axis passing through the top and the base of the fruit/vegetable) coincide, the posture (direction) of the fruit/vegetable can be estimated from the direction of the central axis of the virtual cylinder; ¶46 The location information of fruits and vegetables calculated by the position calculation unit 160 and the posture information of fruits and vegetables estimated by the posture estimation unit 170 are transferred to a fruit and vegetable harvesting robot (not shown) and used to control the fruit and vegetable harvesting robot.)
Regarding Claim 7, Hwang in view of Zhang discloses the method of claim 1, wherein the step of inferring the volume information and the pose information further comprises a step of inferring a semantic image of the crop object, based on the point cloud configuring the reconstructed 3D model (Hwang, ¶50 The segmentation model 130 performing the segmentation in step S230 is a deep learning model learned from synthesized images generated by synthesizing foreground images and background images of fruits and vegetables in a variety of different ways.)
Regarding claim 9, claim 9 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Hwang further teaching on: A control method of a robot, the control method comprising: a step of generating an operation control instruction by using an operation control module, based on the volume information and the pose information; and a step of controlling an operation of a robotic arm according to the operation control instruction by using a robot actuator (Hwang, ¶45 The location information of fruits and vegetables calculated by the position calculation unit 160 and the posture information of fruits and vegetables estimated by the posture estimation unit 170 are transferred to a fruit and vegetable harvesting robot (not shown) and used to control the fruit and vegetable harvesting robot . Since accurate position and posture information is provided, damage to fruits and vegetables caused by the robot arm of the harvesting robot is minimized.)
Claim 11 recites limitations similar to claim 5 and is rejected under the same rationale and reasoning.
Regarding Claim 12, Hwang in view of Zhang discloses the control method of claim 9, wherein the operation of the robotic arm is an operation of harvesting the crop object (Hwang, ¶45 The location information of fruits and vegetables calculated by the position calculation unit 160 and the posture information of fruits and vegetables estimated by the posture estimation unit 170 are transferred to a fruit and vegetable harvesting robot (not shown) and used to control the fruit and vegetable harvesting robot . Since accurate position and posture information is provided, damage to fruits and vegetables caused by the robot arm of the harvesting robot is minimized.)
Regarding claim 13, claim 13 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Hwang further teaching on: An edge device equipped in a robot, the edge device comprising (Hwang, ¶45 The location information of fruits and vegetables calculated by the position calculation unit 160 and the posture information of fruits and vegetables estimated by the posture estimation unit 170 are transferred to a fruit and vegetable harvesting robot (not shown) and used to control the fruit and vegetable harvesting robot . Since accurate position and posture information is provided, damage to fruits and vegetables caused by the robot arm of the harvesting robot is minimized.)
Claim 15 recites limitations similar to claim 5 and is rejected under the same rationale and reasoning.
Claim 16 recites limitations similar to claim 7 and is rejected under the same rationale and reasoning.
Claims 2, 4, 8, 10, 14, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Hwang (KR20230069569A) in view of Zhang (U.S. Patent Pub. No. 2021/0056685) in view of Porth (U.S. Patent Pub. No. 2022/0346303).
Regarding Claim 2, Hwang in view of Zhang teaches the method of claim 1.
Hwang in view of Zhang does not explicitly disclose wherein the step of reconstructing the 3D model comprises a step of reconstructing, by using an artificial neural network, the 3D model where the crop object is configured with a point cloud, based on the space characteristic information.
Porth is in the same field of art of image analysis. Further, Porth teaches wherein the step of reconstructing the 3D model comprises a step of reconstructing, by using an artificial neural network, the 3D model where the crop object is configured with a point cloud, based on the space characteristic information (¶92 The single colorized point cloud may be pre-processed by applying a noise reduction process, a filtering process, and/or an outlier removal process; ¶93 The preprocessed data may be used by a Convolutional Neural Network (CNN) process 1506. The CNN model 1506 has previously been trained to segment one or more plants from each other.)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Hwang in view of Zhang by using a neural network to construct a model that is taught by Porth; thus, one of ordinary skilled in the art would be motivated to combine the references to create a highly accurate 3D model (Porth ¶49).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 4, Hwang in view of Zhang in view of Porth discloses the method of claim 1, wherein the step of inferring the volume information and the pose information comprises:
a step of calculating a convex hull corresponding to the point cloud configuring the reconstructed 3D model; and a step of inferring the volume information about the crop object, based on the calculated convex hull (Porth, Fig. 15; ¶93 The ML model 1508 may determine at least one of: concave hull volume 1510, leaf area index 1512, bush size 1514, bush height 1516, and/or convex hull volume 1518 for each of the one or more segmented plants 608.)
The reasons for combining Hwang, Zhang, and Porth are similar to that stated in the rejection of claim 2. In addition, this same reasoning is pertinent and applicable to the rejections of claims 8, 10, 14, and 17 below.
Regarding Claim 8, Hwang in view of Zhang in view of Porth discloses the method of claim 7, wherein the step of inferring the volume information and the pose information comprises:
a step of clustering points of the point cloud into a plurality of clusters and segmenting the reconstructed 3D model into detailed models (Porth, ¶93 The preprocessed data may be used by a Convolutional Neural Network (CNN) process 1506. The CNN model 1506 has previously been trained to segment (cluster) one or more plants from each other. Each of the segmented plants 608 may be processed by a mathematical model and/or a machine learning (ML) model 1508 in order to determine one or more plant characteristics;)
a step of allocating a class to each cluster to classify the detailed models, based on the class; and a step of inferring a semantic image including the detailed models classified based on the class (Porth, ¶93 The ML model 1508 may determine at least one of: concave hull volume 1510, leaf area index 1512, bush size 1514, bush height 1516, and/or convex hull volume 1518 for each of the one or more segmented plants 608. In some aspects, shown particularly in FIG. 16, the ML model 1508 may place the plant characteristics for each plant on one or more maps to produce a volume map 706, a leaf area index (LAI) map 708, a Normalized Difference Vegetation Index (NDVI) map 710, and/or a biomass map 712.)
Claim 10 recites limitations similar to claim 4 and is rejected under the same rationale and reasoning.
Claim 14 recites limitations similar to claim 4 and is rejected under the same rationale and reasoning.
Claim 17 recites limitations similar to claim 8 and is rejected under the same rationale and reasoning.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Anderson (U.S. Patent Pub. No 2022/0138987) teaches clustering analysis of point clouds ¶44.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUSTIN BILODEAU whose telephone number is (571)272-1032. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUSTIN BILODEAU/Examiner, Art Unit 2664
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664
Prosecution Insights