DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/05/2026 has been entered.
Status of Claims
This action is in reply to the RCE filed on 03/05/2026.
Claims 1-20 are currently pending and have been examined.
Claims 1, 11, and 16 are amended.
Claims 1-20 are currently rejected.
This action is made NON-FINAL.
Response to Arguments
Applicant’s arguments filed 03/05/2026 have been fully considered but they are not persuasive.
Applicant’s arguments with regards to the art rejections have been considered and appear to be directed solely to the instant amendments to the claims. Accordingly, the claims are addressed in the body of the updated rejections below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-8, 11-13, and 16-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fu et. al. (US 2022/0101554), herein Fu in view of Wu et. al. (US 2019/0150357), herein Wu, Redden (US 2020/0187406), herein Redden, and Lin et. al. (CN 116152562), herein Lin.
Regarding claim 1:
Fu teaches:
A method (methods of identifying and treating plants [0063]) comprising:
Detecting (The farming machine 100, illustrated in FIGS. 1A-1E, includes a detection mechanism 110 [0079]), by an [autonomous] farming machine (fig. 1c, farming machine 100b), [a brightness level] at each of a plurality of points of a ground surface in front of a boom (The farming machine 100 can additionally include a mounting mechanism 140 [0079]) of the [autonomous] farming machine (a farming machine may utilize depth information obtained by a depth sensor coupled to identify and treat plants. For example, a farming machine may employ a light detection and ranging system (LIDAR) to identify and treat plants that are too tall, too short, too close, etc. [0070]), the boom comprising one or more lights (a light detection and ranging system (LIDAR)) configured to illuminate the ground surface in front of the boom when the autonomous farming machine operates (see at least fig. 3a showing the detecting means capturing the surface ahead of the machine.) [at night] (although not explicitly taught LIDAR would be able to capture data and work at night.);
Identifying, by the [autonomous] farming machine, a region of interest (The field of view 315, herein, is the angular extent of an area captured by a camera 310 [0104])
Detecting (The control system 130 may identify a plant in the image based on the depth information using a plant identification module (e.g., plant identification module 232) [0112]), by the [autonomous] farming machine, a plant within the identified region of interest (The field of view 315a includes several plants: crops 302a, 302b, 302c, and weed 350 [0106]);
Classifying (control system 130 can determine whether a plant is a weed or a crop [0112]), by the autonomous farming machine, the detected plant (The control system 130 may identify a plant in the image based on the depth information using a plant identification module (e.g., plant identification module 232) [0112])
Selecting, by the [autonomous] farming machine, an action based on the classified plant (the control system 130 may be configured to generate and take a treatment action for the identified plant based on the extracted depth information. For example, the control system 130 can determine whether a plant is a weed or a crop based on the height of the plant, and treat the plant accordingly [0112]); and
Performing, by the [autonomous] farming machine, the selected action (Operating parameters may include, for example, speed of the farming machine 200, direction of the farming machine 200, etc. Treatment parameters may include, for example, height of the treatment mechanism 120 (e.g., distance between the treatment mechanism 120 and the ground), type of treatment (e.g., spray, mechanical manipulation, etc.), time of treatment (e.g., at selected times, periodic intervals, length of treatment, a time delay between an image capture and a treatment, etc.), location of treatment (e.g., near the stem, treatment area 122), and/or other parameters related to treatment of the one or more plants [0200]) at a delayed time based on a location of the region of interest (a time delay between an image capture and a treatment [0200]).
While Fu teaches the ability to use lidar as a means to identify plants, Fu does not explicitly teach, however Wu teaches:
Detecting (If the differential value is past a threshold value or, for example, four sigma deviation, this is characterized as an anomaly and a spray is applied to treat the anomalous area [0105]),by an autonomous farming machine (an autonomous agricultural vehicle [0005]), a brightness level (triggering off of distant objects in the image versus nearby objects in the image is equalized by scaling the threshold for each pixel or grid member to constitute a “hit” [0089]) at each of a plurality of points of a ground surface in front of a boom of the autonomous farming machine (For night spraying, a pre-run calibration is performed to eliminate the shadow effect of the different lights shining on the ground (e.g. the headlights of the vehicle and boom lights illuminating the ground). The pre-run calibration also sets a threshold of what is considered “green” during the nighttime conditions. The night-time color that is associated with “green” is contrasted against the expected background color. When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [0115]), the boom comprising one or more lights configured to illuminate the ground surface in front of the boom when the autonomous farming machine operates (boom lights illuminating the ground [0115]) at night (For night spraying [0115]);
Identifying, by the autonomous farming machine, a region of interest (In some embodiments, to reduce the likelihood of false triggers, multiple pixels or at least a selected number of pixels are each required to satisfy one or more of the conditions stored or programmed into the computer processor in order to fully trigger the spray nozzle to actually release herbicide to kill the weeds. As an example, the selected number of pixels include adjacent pixels within a region of interest or distance among the pixels (e.g. all pixels within 10 pixel distance), and a peak value can be taken as a maximum average value among these pixels within such region of interest [0098]) corresponding to a portion of the ground surface in front of the boom corresponding to an above-threshold detected brightness level (When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [0115]; the magnitude of contrast instead of absolute color is another procedure or an additional procedure. The background color (e.g. ground or residue) is contrasted with the candidate signal color. When the contrast passes a threshold, herbicide is released in the area where the candidate object is found. [0150]; examiner notes that contrast is the difference between brightness levels which would therefore require some threshold brightness level in order to satisfy a contrast threshold.);
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the claimed invention to have modified Fu to include the teachings as taught by Wu with a reasonable expectation of success. Fu teaches a machine able to identify and treat weeds in a field but does not teach the ability to perform this task at night. Wu teaches the ability to perform this task at night and provides the benefits of “For night spraying, a pre-run calibration is performed to eliminate the shadow effect of the different lights shining on the ground (e.g. the headlights of the vehicle and boom lights illuminating the ground). The pre-run calibration also sets a threshold of what is considered “green” during the nighttime conditions. The night-time color that is associated with “green” is contrasted against the expected background color. When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [Wu, 0115].”
Fu and Wu do not explicitly teach, however Redden teaches:
classifying, by the autonomous farming machine, the detected plant by comparing the modified plant classification threshold (Each extracted point of interest preferably includes one or more features that a plant center is expected to exhibit. The points of interest can be dark regions surrounded by one or more colors associated with a plant (e.g. particularly when low incidence lighting is used to capture the image) [0020]) to a classification confidence value representing a likelihood that the detected plant belongs to a particular plant class (The point of interest is preferably classified as a plant center when the associated confidence level exceeds a predetermined threshold [0022])
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the claimed invention to have modified Fu and Wu to include the teachings as taught by Redden with a reasonable expectation of success. Redden teaches the ability to improve over the state of the art of automated plant removal by identifying that “these systems fail to offer the plant removal flexibility in plant selection and removal that human labor offers. In one example, a conventional crop thinning system removes plants at fixed intervals, whether or not the plant removal was necessary. In another example, a conventional crop thinning system removes plants using system vision, but fails to identify multiple close-packed plants as individual plants and treats the close-packed plants as a single plant. [Redden, 0003]”.
Fu, Wu, and Redden do not explicitly teach, however Lin teaches:
modifying, [by the autonomous farming machine], a plant classification threshold for the detected plant (dynamically setting threshold value, forming adaptive adjustment of threshold value, and performing colour classification according to the adjusted threshold value [page 6]) based on a brightness level of one or more image pixels depicting the detected plant (the preset dynamic threshold value is dynamically set according to the central point gray value range of the color stripe. It should be noted that the preset dynamic threshold value set based on the color corresponding to the angle of the color ring based on the actual color structure to be classified color structure light image in the grey value of the central point, namely the brightness of the setting, not a fixed threshold [page 6]);
comparing the modified plant classification threshold to a classification confidence value (performing colour classification according to the adjusted threshold value [page 6])
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the claimed invention to have modified Fu, Wu, and Redden to include the teachings as taught by Lin with a reasonable expectation of success. Lin teaches the benefit of “performing colour classification according to the adjusted threshold value, which is helpful for improving the accuracy of the classification. In practical application, the larger the brightness is easier to identify color, the smaller the influence is, the threshold value can be set more wide; the lower the brightness, the more it is affected by the environment light and object surface color, the threshold range is set to be smaller [Lin, page 6]”.
Regarding claim 2:
Fu, Wu, Redden, and Lin teach all the limitations of claim 1, upon which this claim is dependent.
Wu further teaches:
wherein detecting the brightness level at each of the plurality of points of the ground surface in front of the boom (If the differential value is past a threshold value or, for example, four sigma deviation, this is characterized as an anomaly and a spray is applied to treat the anomalous area [0105]) comprises:
for a plurality of partitions of an image of the ground surface along a first dimension of the image (see at least fig. 4 showing grid 100 extending in a horizontal direction.):
determining the brightness level of a plurality of image pixels along a second dimension of the image (If the coarse grid or data indicates a “hit” or a possible activity (threshold detect), then a fine grid's data centered on the coarse grid may be stored or analyzed instantaneously. [0087]; the difference in values among the grid elements provides a trigger whether an anomaly is observed or detected [0105]), wherein the plurality of points correspond to the plurality of image pixels, and wherein the first dimension is orthogonal to the second dimension (see at least fig. 4 showing grid 100 extending in a vertical direction direction.).
Regarding claim 3:
Fu, Wu, Redden, and Lin teach all the limitations of claim 2, upon which this claim is dependent.
Wu further teaches:
wherein identifying the region of interest comprises determining a region within the image comprising pixels having the above-threshold detected brightness level (When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [0115]; the magnitude of contrast instead of absolute color is another procedure or an additional procedure. The background color (e.g. ground or residue) is contrasted with the candidate signal color. When the contrast passes a threshold, herbicide is released in the area where the candidate object is found. [0150]).
Regarding claim 4:
Fu, Wu, Redden, and Lin teach all the limitations of claim 3, upon which this claim is dependent.
Wu further teaches:
determining, along the second dimension, two or more boundary lines (fig. 4, crop rows 12), wherein the region comprises pixels having values in the second dimension that are bounded (see at least fig. 4 showing anomalies bounded by the crop rows.) by the two or more boundary lines (the image grid 100 is overlaid on or be associated with the soil, but can also be overlaid on the crop rows 12, or whatever image is captured by the image sensor units 50 [0096]).
Regarding claim 5:
Fu, Wu, Redden, and Lin teach all the limitations of claim 1, upon which this claim is dependent.
Fu further teaches:
determining the location of the region of interest (the control system 130 determines a treatment direction for the farming machine that allows the farming machine to correctly actuate a treat mechanism 120 to treat the weed 250 as it travels past the weed 250 in the field. Accordingly, the control system 130 actuates the systems of the farming machine such that it travels at the treatment velocity. In other examples, the farming machine may modify a sprayer timing or sprayer height to correctly actuate a treatment mechanism to treat the plant. Similarly, the control system 130 may determine a distance between each plant (e.g., a proximity) based on the feature values. The control system 130 can then determine an inter-treatment velocity that allows the farming machine to efficiently treat plants based on the proximity between plants (e.g., spray less if the plants are proximate). The control system 130 may then actuate systems of the farming machine 200 to apply the treatments as needed. [0232]); and
determining the delayed time (a time delay between an image capture and a treatment [0200]) based on a speed of the autonomous farming machine (speed of the farming machine 200 [0200]) and the location of the region of interest (location of treatment [0200]).
Regarding claim 6:
Fu, Wu, Redden, and Lin teach all the limitations of claim 1, upon which this claim is dependent.
Wu further teaches:
wherein region of interest comprises a fixed area (three weeds happen to be in the area between crop rows 12 where ideally they would not be expected. In some programs, the found unexpected objects are sprayed with herbicide, represented by a dashed circle [0097]).
Regarding claim 7:
Fu, Wu, Redden, and Lin teach all the limitations of claim 1, upon which this claim is dependent.
Fu further teaches:
for each of a plurality of heights at which the boom is configurable (a height of the treatment mechanism relative to the ground [0006]):
Wu further teaches:
wherein the region of interest is a first region of interest associated with the boom configured at a first height (the sensor units facing the corn cobs are tilted downwards by 10 to 40 degrees depending on the height of the back horizontal bar and depending on whether broader forward image capture is desired. [0134]), further comprising:
for each of a plurality of heights at which the boom is configurable (to adjust the height of the boom during field operation [0090]):
identifying a respective region of interest associated with the boom configured at the height (Equalize or normalize the elements (farther image elements are multiplied by some factor based on height at which the sensor is mounted, pointing angle, distance to the point of view [0185]), wherein the respective region of interest corresponds to a respective portion of the ground surface in front of the boom, the respective portion corresponding to the above-threshold detected brightness level (When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [0115]; the magnitude of contrast instead of absolute color is another procedure or an additional procedure. The background color (e.g. ground or residue) is contrasted with the candidate signal color. When the contrast passes a threshold, herbicide is released in the area where the candidate object is found. [0150]).
Regarding claim 8:
Fu, Wu, Redden, and Lin teach all the limitations of claim 1, upon which this claim is dependent.
Wu further teaches:
in response to determining, based on a first image, a first brightness level of a first point of the ground surface, determining an accuracy threshold for classifying the plant (For night spraying, a pre-run calibration is performed to eliminate the shadow effect of the different lights shining on the ground (e.g. the headlights of the vehicle and boom lights illuminating the ground). The pre-run calibration also sets a threshold of what is considered “green” during the nighttime conditions. The night-time color that is associated with “green” is contrasted against the expected background color. [0115]), wherein the first brightness level satisfies the above-threshold detected brightness level (When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [0115]); and
in response to determining, based on a second image captured subsequent to the first image (Adjust real-time pattern recognition and calibration when major discrepancies are found [0185]), a second brightness level of a second point of the ground surface, modifying the accuracy threshold for classifying the plant, wherein the second brightness level falls below the above-threshold detected brightness level (there is automatic calibration for in situ conditions such as color to adjust for detected lighting or time of day, to adjust the height of the boom during field operation based on calibrated crop height, and so on. [0090]).
Regarding claim 11:
Fu teaches:
A farming machine (A farming machine [abstract]) comprising:
a sensor configured for capturing images of plants in a field as the farming machine travels through the field (The detection mechanism 110 is configured to identify a plant for treatment. As such, the detection mechanism 110 can include one or more sensors for identifying a plant. For example, the detection mechanism 110 can include a multispectral camera, a stereo camera, a CCD camera, a single lens camera, a CMOS camera, hyperspectral imaging system, LIDAR system (light detection and ranging system), a depth sensing system, dynamometer, IR camera, thermal camera, humidity sensor, light sensor, temperature sensor, or any other suitable sensor. In one embodiment, and described in greater detail below, the detection mechanism 110 includes an array of image sensors configured to capture an image of a plant [0083]);
a treatment array configured to treat plants (The treatment mechanism 120 functions to apply a treatment to an identified plant 102 [0084]);
a boom (fig. 1c mounting mechanism 140) comprising one or more lights configured to illuminate a ground surface in front of the boom when the farming machine (a farming machine may utilize depth information obtained by a depth sensor coupled to identify and treat plants. For example, a farming machine may employ a light detection and ranging system (LIDAR) to identify and treat plants that are too tall, too short, too close, etc. [0070])
a processor (The processor 1602 is, for example, a central processing unit (CPU) [0241]);
a non-transitory computer readable storage medium (The computer system 1600 also includes a main memory 1604. The computer system may include a storage unit 1616 [0241]) storing computer program instructions (a computer program may be stored in a computer readable storage medium [0248]), the computer program instructions, when executed by the processor, causing the processor (The instructions 1624 may also reside, completely or at least partially, within the main memory 1604 or within the processor 1602 (e.g., within a processor's cache memory) during execution thereof by the computer system 1600 [0243]) to:
detect (The farming machine 100, illustrated in FIGS. 1A-1E, includes a detection mechanism 110 [0079]) a [brightness level] at each of a plurality of points of the ground surface in front of a boom of the farming machine (The farming machine 100 can additionally include a mounting mechanism 140 [0079]);
identify a region of interest corresponding to a portion of the ground surface in front of the boom (The field of view 315, herein, is the angular extent of an area captured by a camera 310 [0104])
detect (The control system 130 may identify a plant in the image based on the depth information using a plant identification module (e.g., plant identification module 232) [0112]) a plant within the identified region of interest (The field of view 315a includes several plants: crops 302a, 302b, 302c, and weed 350 [0106]);
select an action based on the classified plant (the control system 130 may be configured to generate and take a treatment action for the identified plant based on the extracted depth information. For example, the control system 130 can determine whether a plant is a weed or a crop based on the height of the plant, and treat the plant accordingly [0112]); and
perform the selected action (Operating parameters may include, for example, speed of the farming machine 200, direction of the farming machine 200, etc. Treatment parameters may include, for example, height of the treatment mechanism 120 (e.g., distance between the treatment mechanism 120 and the ground), type of treatment (e.g., spray, mechanical manipulation, etc.), time of treatment (e.g., at selected times, periodic intervals, length of treatment, a time delay between an image capture and a treatment, etc.), location of treatment (e.g., near the stem, treatment area 122), and/or other parameters related to treatment of the one or more plants [0200]) at a delayed time based on a location of the region of interest (a time delay between an image capture and a treatment [0200]).
While Fu teaches the ability to use lidar as a means to identify plants, Fu does not explicitly teach, however Wu teaches:
a boom comprising one or more lights (boom lights illuminating the ground [0115]) configured to illuminate a ground surface in front of the boom when the farming machine operates at night (For night spraying, a pre-run calibration is performed to eliminate the shadow effect of the different lights shining on the ground (e.g. the headlights of the vehicle and boom lights illuminating the ground). The pre-run calibration also sets a threshold of what is considered “green” during the nighttime conditions. The night-time color that is associated with “green” is contrasted against the expected background color. When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [0115]);
detect (If the differential value is past a threshold value or, for example, four sigma deviation, this is characterized as an anomaly and a spray is applied to treat the anomalous area [0105]) a brightness level (triggering off of distant objects in the image versus nearby objects in the image is equalized by scaling the threshold for each pixel or grid member to constitute a “hit” [0089]) at each of a plurality of points of the ground surface in front of a boom of the farming machine (an autonomous agricultural vehicle [0005]);
identify a region of interest (In some embodiments, to reduce the likelihood of false triggers, multiple pixels or at least a selected number of pixels are each required to satisfy one or more of the conditions stored or programmed into the computer processor in order to fully trigger the spray nozzle to actually release herbicide to kill the weeds. As an example, the selected number of pixels include adjacent pixels within a region of interest or distance among the pixels (e.g. all pixels within 10 pixel distance), and a peak value can be taken as a maximum average value among these pixels within such region of interest [0098]) corresponding to a portion of the ground surface in front of the boom corresponding to an above-threshold detected brightness level (When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [0115]; the magnitude of contrast instead of absolute color is another procedure or an additional procedure. The background color (e.g. ground or residue) is contrasted with the candidate signal color. When the contrast passes a threshold, herbicide is released in the area where the candidate object is found. [0150]; examiner notes that contrast is the difference between brightness levels which would therefore require some threshold brightness level in order to satisfy a contrast threshold.);
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the claimed invention to have modified Fu to include the teachings as taught by Wu with a reasonable expectation of success. Fu teaches a machine able to identify and treat weeds in a field but does not teach the ability to perform this task at night. Wu teaches the ability to perform this task at night and provides the benefits of “For night spraying, a pre-run calibration is performed to eliminate the shadow effect of the different lights shining on the ground (e.g. the headlights of the vehicle and boom lights illuminating the ground). The pre-run calibration also sets a threshold of what is considered “green” during the nighttime conditions. The night-time color that is associated with “green” is contrasted against the expected background color. When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [Wu, 0115].”
Fu and Wu do not explicitly teach, however Redden teaches:
classifying the detected plant by comparing the modified plant classification threshold (Each extracted point of interest preferably includes one or more features that a plant center is expected to exhibit. The points of interest can be dark regions surrounded by one or more colors associated with a plant (e.g. particularly when low incidence lighting is used to capture the image) [0020]) to a classification confidence value representing a likelihood that the detected plant belongs to a particular plant class (The point of interest is preferably classified as a plant center when the associated confidence level exceeds a predetermined threshold [0022])
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the claimed invention to have modified Fu and Wu to include the teachings as taught by Redden with a reasonable expectation of success. Redden teaches the ability to improve over the state of the art of automated plant removal by identifying that “these systems fail to offer the plant removal flexibility in plant selection and removal that human labor offers. In one example, a conventional crop thinning system removes plants at fixed intervals, whether or not the plant removal was necessary. In another example, a conventional crop thinning system removes plants using system vision, but fails to identify multiple close-packed plants as individual plants and treats the close-packed plants as a single plant. [Redden, 0003]”.
Fu, Wu, and Redden do not explicitly teach, however Lin teaches:
modifying a plant classification threshold for the detected plant (dynamically setting threshold value, forming adaptive adjustment of threshold value, and performing colour classification according to the adjusted threshold value [page 6]) based on a brightness level of one or more image pixels depicting the detected plant (the preset dynamic threshold value is dynamically set according to the central point gray value range of the color stripe. It should be noted that the preset dynamic threshold value set based on the color corresponding to the angle of the color ring based on the actual color structure to be classified color structure light image in the grey value of the central point, namely the brightness of the setting, not a fixed threshold [page 6]);
comparing the modified plant classification threshold to a classification confidence value (performing colour classification according to the adjusted threshold value [page 6])
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the claimed invention to have modified Fu, Wu, and Redden to include the teachings as taught by Lin with a reasonable expectation of success. Lin teaches the benefit of “performing colour classification according to the adjusted threshold value, which is helpful for improving the accuracy of the classification. In practical application, the larger the brightness is easier to identify color, the smaller the influence is, the threshold value can be set more wide; the lower the brightness, the more it is affected by the environment light and object surface color, the threshold range is set to be smaller [Lin, page 6]”.
Regarding claim 12:
Fu, Wu, Redden, and Lin teach all the limitations of claim 11, upon which this claim is dependent.
Fu further teaches:
for each of a plurality of heights at which the boom is configurable (a height of the treatment mechanism relative to the ground [0006]):
Wu further teaches:
wherein the region of interest is a first region of interest associated with the boom configured at a first height (the sensor units facing the corn cobs are tilted downwards by 10 to 40 degrees depending on the height of the back horizontal bar and depending on whether broader forward image capture is desired. [0134]),
for each of a plurality of heights at which the boom is configurable (to adjust the height of the boom during field operation [0090]):
identifying a respective region of interest associated with the boom configured at the height (Equalize or normalize the elements (farther image elements are multiplied by some factor based on height at which the sensor is mounted, pointing angle, distance to the point of view [0185]), wherein the respective region of interest corresponds to a respective portion of the ground surface in front of the boom, the respective portion corresponding to the above-threshold detected brightness level (When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [0115]; the magnitude of contrast instead of absolute color is another procedure or an additional procedure. The background color (e.g. ground or residue) is contrasted with the candidate signal color. When the contrast passes a threshold, herbicide is released in the area where the candidate object is found. [0150]).
Regarding claim 13:
Fu, Wu, Redden, and Lin teach all the limitations of claim 11, upon which this claim is dependent.
Wu further teaches:
in response to determining, based on a first image, a first brightness level of a first point of the ground surface, determining an accuracy threshold for classifying the plant (For night spraying, a pre-run calibration is performed to eliminate the shadow effect of the different lights shining on the ground (e.g. the headlights of the vehicle and boom lights illuminating the ground). The pre-run calibration also sets a threshold of what is considered “green” during the nighttime conditions. The night-time color that is associated with “green” is contrasted against the expected background color. [0115]), wherein the first brightness level satisfies the above-threshold detected brightness level (When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [0115]); and
in response to determining, based on a second image captured subsequent to the first image (Adjust real-time pattern recognition and calibration when major discrepancies are found [0185]), a second brightness level of a second point of the ground surface, modifying the accuracy threshold for classifying the plant, wherein the second brightness level falls below the above-threshold detected brightness level (there is automatic calibration for in situ conditions such as color to adjust for detected lighting or time of day, to adjust the height of the boom during field operation based on calibrated crop height, and so on. [0090]).
Regarding claim 16:
Fu teaches:
A non-transitory computer readable storage medium (The computer system 1600 also includes a main memory 1604. The computer system may include a storage unit 1616 [0241]) storing computer program storing instructions (a computer program may be stored in a computer readable storage medium [0248]), the computer program instructions, when executed by a processor (The instructions 1624 may also reside, completely or at least partially, within the main memory 1604 or within the processor 1602 (e.g., within a processor's cache memory) during execution thereof by the computer system 1600 [0243]), causing the processor to:
detect (The farming machine 100, illustrated in FIGS. 1A-1E, includes a detection mechanism 110 [0079]) a [brightness level] at each of a plurality of points of the ground surface in front of a boom of the farming machine (The farming machine 100 can additionally include a mounting mechanism 140 [0079]);
identify a region of interest corresponding to a portion of the ground surface in front of the boom (The field of view 315, herein, is the angular extent of an area captured by a camera 310 [0104])
detect (The control system 130 may identify a plant in the image based on the depth information using a plant identification module (e.g., plant identification module 232) [0112]) a plant within the identified region of interest (The field of view 315a includes several plants: crops 302a, 302b, 302c, and weed 350 [0106]);
select an action based on the classified plant (the control system 130 may be configured to generate and take a treatment action for the identified plant based on the extracted depth information. For example, the control system 130 can determine whether a plant is a weed or a crop based on the height of the plant, and treat the plant accordingly [0112]); and
perform the selected action (Operating parameters may include, for example, speed of the farming machine 200, direction of the farming machine 200, etc. Treatment parameters may include, for example, height of the treatment mechanism 120 (e.g., distance between the treatment mechanism 120 and the ground), type of treatment (e.g., spray, mechanical manipulation, etc.), time of treatment (e.g., at selected times, periodic intervals, length of treatment, a time delay between an image capture and a treatment, etc.), location of treatment (e.g., near the stem, treatment area 122), and/or other parameters related to treatment of the one or more plants [0200]) at a delayed time based on a location of the region of interest (a time delay between an image capture and a treatment [0200]).
While Fu teaches the ability to use lidar as a means to identify plants, Fu does not explicitly teach, however Wu teaches:
a boom comprising one or more lights (boom lights illuminating the ground [0115]) configured to illuminate a ground surface in front of the boom when the farming machine operates at night (For night spraying, a pre-run calibration is performed to eliminate the shadow effect of the different lights shining on the ground (e.g. the headlights of the vehicle and boom lights illuminating the ground). The pre-run calibration also sets a threshold of what is considered “green” during the nighttime conditions. The night-time color that is associated with “green” is contrasted against the expected background color. When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [0115]);
detect (If the differential value is past a threshold value or, for example, four sigma deviation, this is characterized as an anomaly and a spray is applied to treat the anomalous area [0105]) a brightness level (triggering off of distant objects in the image versus nearby objects in the image is equalized by scaling the threshold for each pixel or grid member to constitute a “hit” [0089]) at each of a plurality of points of the ground surface in front of a boom of the farming machine (an autonomous agricultural vehicle [0005]);
identify a region of interest (In some embodiments, to reduce the likelihood of false triggers, multiple pixels or at least a selected number of pixels are each required to satisfy one or more of the conditions stored or programmed into the computer processor in order to fully trigger the spray nozzle to actually release herbicide to kill the weeds. As an example, the selected number of pixels include adjacent pixels within a region of interest or distance among the pixels (e.g. all pixels within 10 pixel distance), and a peak value can be taken as a maximum average value among these pixels within such region of interest [0098]) corresponding to a portion of the ground surface in front of the boom corresponding to an above-threshold detected brightness level (When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [0115]; the magnitude of contrast instead of absolute color is another procedure or an additional procedure. The background color (e.g. ground or residue) is contrasted with the candidate signal color. When the contrast passes a threshold, herbicide is released in the area where the candidate object is found. [0150]; examiner notes that contrast is the difference between brightness levels which would therefore require some threshold brightness level in order to satisfy a contrast threshold.);
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the claimed invention to have modified Fu to include the teachings as taught by Wu with a reasonable expectation of success. Fu teaches a machine able to identify and treat weeds in a field but does not teach the ability to perform this task at night. Wu teaches the ability to perform this task at night and provides the benefits of “For night spraying, a pre-run calibration is performed to eliminate the shadow effect of the different lights shining on the ground (e.g. the headlights of the vehicle and boom lights illuminating the ground). The pre-run calibration also sets a threshold of what is considered “green” during the nighttime conditions. The night-time color that is associated with “green” is contrasted against the expected background color. When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [Wu, 0115].”
Fu and Wu do not explicitly teach, however Redden teaches:
classifying the detected plant by comparing the modified plant classification threshold (Each extracted point of interest preferably includes one or more features that a plant center is expected to exhibit. The points of interest can be dark regions surrounded by one or more colors associated with a plant (e.g. particularly when low incidence lighting is used to capture the image) [0020]) to a classification confidence value representing a likelihood that the detected plant belongs to a particular plant class (The point of interest is preferably classified as a plant center when the associated confidence level exceeds a predetermined threshold [0022])
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the claimed invention to have modified Fu and Wu to include the teachings as taught by Redden with a reasonable expectation of success. Redden teaches the ability to improve over the state of the art of automated plant removal by identifying that “these systems fail to offer the plant removal flexibility in plant selection and removal that human labor offers. In one example, a conventional crop thinning system removes plants at fixed intervals, whether or not the plant removal was necessary. In another example, a conventional crop thinning system removes plants using system vision, but fails to identify multiple close-packed plants as individual plants and treats the close-packed plants as a single plant. [Redden, 0003]”.
Fu, Wu, and Redden do not explicitly teach, however Lin teaches:
modifying a plant classification threshold for the detected plant (dynamically setting threshold value, forming adaptive adjustment of threshold value, and performing colour classification according to the adjusted threshold value [page 6]) based on a brightness level of one or more image pixels depicting the detected plant (the preset dynamic threshold value is dynamically set according to the central point gray value range of the color stripe. It should be noted that the preset dynamic threshold value set based on the color corresponding to the angle of the color ring based on the actual color structure to be classified color structure light image in the grey value of the central point, namely the brightness of the setting, not a fixed threshold [page 6]);
comparing the modified plant classification threshold to a classification confidence value (performing colour classification according to the adjusted threshold value [page 6])
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the claimed invention to have modified Fu, Wu, and Redden to include the teachings as taught by Lin with a reasonable expectation of success. Lin teaches the benefit of “performing colour classification according to the adjusted threshold value, which is helpful for improving the accuracy of the classification. In practical application, the larger the brightness is easier to identify color, the smaller the influence is, the threshold value can be set more wide; the lower the brightness, the more it is affected by the environment light and object surface color, the threshold range is set to be smaller [Lin, page 6]”.
Regarding claim 17:
Fu, Wu, Redden, and Lin teach all the limitations of claim 16, upon which this claim is dependent.
Fu further teaches:
for each of a plurality of heights at which the boom is configurable (a height of the treatment mechanism relative to the ground [0006]):
Wu further teaches:
wherein the region of interest is a first region of interest associated with the boom configured at a first height (the sensor units facing the corn cobs are tilted downwards by 10 to 40 degrees depending on the height of the back horizontal bar and depending on whether broader forward image capture is desired. [0134]), further comprising:
for each of a plurality of heights at which the boom is configurable (to adjust the height of the boom during field operation [0090]):
identifying a respective region of interest associated with the boom configured at the height (Equalize or normalize the elements (farther image elements are multiplied by some factor based on height at which the sensor is mounted, pointing angle, distance to the point of view [0185]), wherein the respective region of interest corresponds to a respective portion of the ground surface in front of the boom, the respective portion corresponding to the above-threshold detected brightness level (When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [0115]; the magnitude of contrast instead of absolute color is another procedure or an additional procedure. The background color (e.g. ground or residue) is contrasted with the candidate signal color. When the contrast passes a threshold, herbicide is released in the area where the candidate object is found. [0150]).
Regarding claim 18:
Fu, Wu, Redden, and Lin teach all the limitations of claim 16, upon which this claim is dependent.
Wu further teaches:
in response to determining, based on a first image, a first brightness level of a first point of the ground surface, determining an accuracy threshold for classifying the plant (For night spraying, a pre-run calibration is performed to eliminate the shadow effect of the different lights shining on the ground (e.g. the headlights of the vehicle and boom lights illuminating the ground). The pre-run calibration also sets a threshold of what is considered “green” during the nighttime conditions. The night-time color that is associated with “green” is contrasted against the expected background color. [0115]), wherein the first brightness level satisfies the above-threshold detected brightness level (When there is a large enough contrast signal past a pre-determined threshold value, a candidate object would trigger the herbicide to be released in the area of the candidate object [0115]); and
in response to determining, based on a second image captured subsequent to the first image (Adjust real-time pattern recognition and calibration when major discrepancies are found [0185]), a second brightness level of a second point of the ground surface, modifying the accuracy threshold for classifying the plant, wherein the second brightness level falls below the above-threshold detected brightness level (there is automatic calibration for in situ conditions such as color to adjust for detected lighting or time of day, to adjust the height of the boom during field operation based on calibrated crop height, and so on. [0090]).
Claim(s) 9-10, 14-15, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fu et. al. (US 2022/0101554), herein Fu in view of Wu et. al. (US 2019/0150357), herein Wu, Redden (US 2020/0187406), herein Redden, and Lin et. al. (CN 116152562), herein Lin in further view of Hu et. al. (CN 115278099), herein Hu.
Regarding claim 9:
Fu, Wu, Redden, and Lin teach all the limitations of claim 1, upon which this claim is dependent.
Wu further teaches:
determining a number of points of the ground surface in front of the boom having a brightness level at the above-threshold detected brightness level (fig. 4, anomalies 11);
Fu, Wu, Redden, and Lin do not explicitly teach, however Hu teaches:
determining a number of points of the ground surface in front of the boom having a brightness level at the above-threshold detected brightness level (determining the brightness value of each pixel point in the initial image [Hu]);
determining, based on the number of points, whether the one or more lights is illuminating the ground surface in front of the boom at the above-threshold detected brightness level (determining the area composed of the pixel points less than the preset brightness threshold in all the brightness values as the dark area; determining the external rectangle of the dark area; and determining the area included in the external rectangle as the target area. [Hu]).
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the claimed invention to have modified Fu, Wu, Redden, and Lin to include the teachings as taught by Hu with a reasonable expectation of success. Hu teaches “when the user uses the rear camera of the terminal device to shoot the content of the book in parallel, if the whole environment light is dark, then the image quality of shooting can be improved turn on the light supplementing lamp. However, if the whole ambient light is not dark when shooting, but the part of the light above the book is shielded to form a local shadow, if not turn on the shooting effect of the book shadow area will not be good. Therefore, how to intelligently and accurately control the fill light of the camera to fill light into the problem urgently needs to be solved [Hu, background]”.
Regarding claim 10:
Fu, Wu, Redden, Lin and Hu teach all the limitations of claim 9, upon which this claim is dependent.
Hu further teaches:
in response to determining that the one or more lights is illuminating the ground surface in front of the boom below the above-threshold detected brightness level (when detecting that the target image in the brightness value is less than the preset brightness threshold value of the area, prompt message,),
generating a notification that the one or more lights are mounted at an incorrect angle (the prompt message is used for prompting the user to adjust the angle of the terminal device, so that the light to the brightness value is less than the preset brightness threshold value of the area by the fill light.).
Regarding claim 14:
Fu, Wu, Redden, and Lin teach all the limitations of claim 11, upon which this claim is dependent.
Wu further teaches:
determine a number of points of the ground surface in front of the boom having a brightness level at the above-threshold detected brightness level (fig. 4, anomalies 11);
Fu, Wu, Redden, and Lin do not explicitly teach, however Hu teaches:
determine a number of points of the ground surface in front of the boom having a brightness level at the above-threshold detected brightness level (determining the brightness value of each pixel point in the initial image [Hu]);
determine, based on the number of points, whether the one or more lights is illuminating the ground surface in front of the boom at the above-threshold detected brightness level (determining the area composed of the pixel points less than the preset brightness threshold in all the brightness values as the dark area; determining the external rectangle of the dark area; and determining the area included in the external rectangle as the target area. [Hu]).
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the claimed invention to have modified Fu, Wu, Redden, and Lin to include the teachings as taught by Hu with a reasonable expectation of success. Hu teaches “when the user uses the rear camera of the terminal device to shoot the content of the book in parallel, if the whole environment light is dark, then the image quality of shooting can be improved turn on the light supplementing lamp. However, if the whole ambient light is not dark when shooting, but the part of the light above the book is shielded to form a local shadow, if not turn on the shooting effect of the book shadow area will not be good. Therefore, how to intelligently and accurately control the fill light of the camera to fill light into the problem urgently needs to be solved [Hu, background]”.
Regarding claim 15:
Fu, Wu, Redden, Lin and Hu teach all the limitations of claim 14, upon which this claim is dependent.
Hu further teaches:
in response to determining that the one or more lights is illuminating the ground surface in front of the boom below the above-threshold detected brightness level (when detecting that the target image in the brightness value is less than the preset brightness threshold value of the area, prompt message,),
generate a notification that the one or more lights are mounted at an incorrect angle (the prompt message is used for prompting the user to adjust the angle of the terminal device, so that the light to the brightness value is less than the preset brightness threshold value of the area by the fill light.).
Regarding claim 19:
Fu, Wu, Redden, and Lin teach all the limitations of claim 16, upon which this claim is dependent.
Wu further teaches:
determine a number of points of the ground surface in front of the boom having a brightness level at the above-threshold detected brightness level (fig. 4, anomalies 11);
Fu, Wu, Redden, and Lin do not explicitly teach, however Hu teaches:
determine a number of points of the ground surface in front of the boom having a brightness level at the above-threshold detected brightness level (determining the brightness value of each pixel point in the initial image [Hu]);
determine, based on the number of points, whether the one or more lights is illuminating the ground surface in front of the boom at the above-threshold detected brightness level (determining the area composed of the pixel points less than the preset brightness threshold in all the brightness values as the dark area; determining the external rectangle of the dark area; and determining the area included in the external rectangle as the target area. [Hu]).
It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the claimed invention to have modified Fu, Wu, Redden, and Lin to include the teachings as taught by Hu with a reasonable expectation of success. Hu teaches “when the user uses the rear camera of the terminal device to shoot the content of the book in parallel, if the whole environment light is dark, then the image quality of shooting can be improved turn on the light supplementing lamp. However, if the whole ambient light is not dark when shooting, but the part of the light above the book is shielded to form a local shadow, if not turn on the shooting effect of the book shadow area will not be good. Therefore, how to intelligently and accurately control the fill light of the camera to fill light into the problem urgently needs to be solved [Hu, background]”.
Regarding claim 20:
Fu, Wu, Redden, Lin and Hu teach all the limitations of claim 19, upon which this claim is dependent.
Hu further teaches:
in response to determining that the one or more lights is illuminating the ground surface in front of the boom below the above-threshold detected brightness level (when detecting that the target image in the brightness value is less than the preset brightness threshold value of the area, prompt message,),
generate a notification that the one or more lights are mounted at an incorrect angle (the prompt message is used for prompting the user to adjust the angle of the terminal device, so that the light to the brightness value is less than the preset brightness threshold value of the area by the fill light.).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Marin (US 2022/0245949) discloses The self-calibration module 230 determines appropriate machine-learned detection calibration actions such as modifying classification thresholds, color space transformations, brightness/contrast/sharpness corrections. The calibrations are made to maximize the ability of the system to detect a set of special calibration objects, placed at different positions along the rail track 102, which are known to be observable/detectable from a given rail vehicle position. This self-calibration functionality allows the system to adapt to environmental and operational conditions that may affect the sensor measurements or generally degrade the detection performance such as low visibility due to weather conditions.
Guo (CN111523457) discloses a weed identification method and a weed processing device; it can collect the image of the ground through the image collecting device to obtain the image to be identified; using the preset brightness threshold to binarize the image to be identified to obtain the binarized image; determining the image area of each plant in the binarized image; inputting the each image area into the pre-trained plant identification model, obtaining the identification result of each plant in each image area; identifying the weeds in the image to be identified according to each of the identification results. The solution provided by the embodiment of the invention improves the precision of the automatic weeding by automatic identification of the weed, and replaces the artificial to process the weed, improves the processing efficiency.
Sun (CN107145879) discloses a plant type automatic identification method and system, the method comprises: uploading the plant image; image quality the plant image server identification, judging image quality of plant image is qualified; When qualified, performing image processing to plant the uploaded image, obtaining a segmented image with plant; the obtained comprises a plant divided image for encoding so as to generate divided image contained in the coding vector of the plant parameters and the plant of each part of the coding vector parameter; the obtained coding vector parameters of the plant and plant of each part of the coding vector parameter with the pre-stored standard vector plant parameter to compare, determine the species of plant, the determined plant type back to the user terminal, and to display on the user terminal, to automatically identify the plant species, the recognition rate is high, and the recognition speed is fast, the better experience to the user.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Scott R Jagolinzer whose telephone number is (571)272-4180. The examiner can normally be reached M-Th 8AM - 4PM Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Christian Chace can be reached at (571)272-4190. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Scott R. Jagolinzer
Examiner
Art Unit 3665
/S.R.J./Examiner, Art Unit 3665 /CHRISTIAN CHACE/Supervisory Patent Examiner, Art Unit 3665
Prosecution Insights