DETAILED ACTION
Claims 29 and 30 have been added.
Claims 1-30 are currently pending.
Response to Arguments
Applicant's arguments filed 7/17/26 have been fully considered but they are not persuasive.
The Applicant argues on pages 17-19 of the response in essence that: Wu's paragraph [0170] contains no disclosure of identifying a group of pixels that represent the emitted fluid projectile itself, as distinct from pixels representing the resulting pattern on the ground.
Wu discloses that the procedure includes capturing the image, associating or overlaying a virtual grid 100 over the captured image, then check whether (dyed) spray went past a boundary “Boundary” associated with one of the Y-direction lines of the grid 100 (paragraph 112). Wu further specifies that the image of the spray includes shining a light on the spray boom to help improve the contrast of the spray droplets against the air or against the ground (paragraph 112).
The Applicant argues on pages 19 and 20 of the response in essence that: Indexing, by contrast, as recited in claim 1 and as described in the specification (see, e.g., paragraphs [0103]-[0105], [0125] and [0128] and FIGs. 9A and 9B), requires indexing the identified first spray object and the identified spray projectile - that is, the specific objects that were identified from the first and second groups of pixels. Bulk logging of unbiased raw data is the antithesis of indexing specifically identified spray objects and spray projectiles.
Wu discloses identifying a first group of pixels that represent a first spray object (paragraph 170, In yet another embodiment, the calculated predicted spray drift are compared with the detected spray drift based on a color image (e.g. spray dye of the end units). If the predicted and detected spray drift pattern on the ground agree to within a predetermined threshold (e.g. 85 to 95%), and over-or-under spraying or drift is found, the spray nozzles or agricultural vehicle take corrective actions and/or alert the operator) and identifying a second group of pixels that represent a spray projectile of the emitted first fluid projectile in an image of the second set of images (paragraph 112, The procedure includes capturing the image, associating or overlaying a virtual grid 100 over the captured image, then check whether (dyed) spray went past a boundary “Boundary” associated with one of the Y-direction lines of the grid 100). Wu discloses that the identified first spray object and the identified spray projectile are subsequently indexed (paragraph 156, In some embodiments, some percentage of all of the data (every sensor, vehicle speed and position, and every image, pixel or array member, i.e. completely unbiased data) is logged so that a secondary or tertiary analysis aids in improving or corroborating (e.g. a cross check of) the initial on-vehicle, real time trigger analysis used to decide the real time action). The ”bulk logging” of Wu results in indexing of the first spray object and the spray projectile that were previously identified. Therefore, it is irrelevant whether “bulk logging” is the antithesis of indexing specifically identified spray objects and spray projectiles.
The Applicant argues on pages 20 and 21 of the response in essence that: While paragraph [0052] of Wu also describes stitching images together to map terrain or a crop row to form a 3D image, a 3-dimensional image of terrain is not a 3-dimensional vector representing an identified spray projectile, nor a model of a full 3-dimensional profile of an identified spray projectile, and Wu does not describe indexing any detected spray as such a vector or model mapped into a virtual scene.
Wu discloses that the spray cone envelope calculations are performed in a reference frame where the effective central axis is determined based on instantaneous wind and travel directions (vector sum of the vehicle travel velocity and wind velocity) with respect to the ground (paragraph 168).
The Applicant argues on page 22 of the response in essence that: Claims 7 and 21, as amended, recite predetermining a predicted spray path of the first fluid projectile and identifying the line of the spray object by performing spray segmentation only in a portion of an image of the second set of images contained in a region defined by the predicted spray path. None of Wu, Wiltshire or Polzounov teaches or suggests confining spray segmentation to only a portion of an image contained in a region defined by a predicted spray path predetermined for the emitted fluid projectile. Wu's paragraph [0170] describes comparing a predicted spray drift with a detected spray drift pattern on the ground; it does not describe using a predicted spray path to define a region of an image within which spray segmentation is performed to identify a line of the spray object.
Wu discloses that the procedure includes capturing the image, associating or overlaying a virtual grid 100 over the captured image, then check whether (dyed) spray went past a boundary “Boundary” associated with one of the Y-direction lines of the grid 100. Drift detection can be used to determine spray overlap, as well, based on the image captured (paragraph 112).
The Applicant argues on pages 22 and 23 of the response in essence that: Claim 4, as amended, recites determining, based on the inaccurate application, an adjustment accounting for at least one of a wind that moved the emitted first fluid projectile, a speed of a vehicle supporting the agricultural treatment system, or a mechanical defect causing a misalignment between the first target agricultural object and a line of sight of a treatment head of the agricultural treatment system. Wu's paragraph [0170], at most, describes that spray nozzles or the agricultural vehicle "take corrective actions and/or alert the operator" when over-spraying, under-spraying or drift is found. A generic reference to unspecified corrective actions or an operator alert does not teach or suggest determining an adjustment that accounts for the specifically recited causes of inaccurate application - wind that moved the projectile, the speed of the vehicle supporting the treatment system, or a mechanical defect causing a misalignment between the target and the line of sight of the treatment head.
Wu discloses that that travel path of the droplets is corrected for factors such as wind direction, local eddies, vehicle travel direction, speed of vehicle, and humidity to determine or to correct for spray drift cone versus expected or ideal spray travel cone, and is also correlated with droplet size (where larger droplets do not veer as much) and the chemical composition (where additives in the spray fluid can also reduce spray drift (paragraph 112).
The Applicant argues on page 23 of the response in essence that: Claims 12 and 26, as amended, recite generating and superimposing 3-dimensional models of the first target agricultural object, the spray projectile, and a splash pattern on each other to reconstruct a spray action from targeting of the first target agricultural object, to spraying of the first target agricultural object, to a splash made and a splat detected. As discussed in Section IV.B above, the cited portions of Wu describe only time-stamp-based frame selection and bulk logging of raw, unbiased data,
Wu discloses generating and superimposing 3-dimensional models of the first target agricultural object, the spray projectile, and a splash pattern on each other to reconstruct a spray action from targeting of the first target agricultural object, to spraying of the first target agricultural object, to a splash made and a splat detected (paragraph 126, If two or more lenses are used together to collect images and videos or if stereo image sensor are used, the upcoming terrain is reconstructed in 3 dimensions).
The Applicant argues on pages 23 and 24 of the response in essence that: Wu's cited passages (paragraphs [0052], [0156] and [0170]) describe frame synchronization, bulk data logging and drift-pattern comparison, and nowhere describe separating spray-projectile pixels from spray-impact pixels and cataloguing the isolated impact as a treatment pattern. Wiltshire generically classifies changed image patches and likewise does not distinguish projectile pixels from impact pixels.
Wu discloses that to avoid needless spray (e.g. crop residue, twig), such objects are calibrated out by void pattern instructions performed on an image captured (paragraph 103).
The Applicant argues on page 24 of the response in essence that: The cited portion of Wu (paragraph [0066]) describes detecting and assessing an amount of residue on the ground SO that a cultivator or planter can adjust the pressure on its implements - not a per-object treatment history. Polzounov's verification compares an executed treatment map derived from a post-image against a stored treatment map for a single treatment pass; it does not maintain a treatment history for an individual agricultural object that is consulted to decide whether to re-treat that same object when it is detected again at a later time or at a later phenological (growth) stage. Wiltshire performs no treatment at all.
Wu discloses the algorithm analyzes the captured image elements to verify if the spread statistics are correct, and spread uniformly within, say, less than 10% difference among the captured images either in the other crop regions or the crop rows that have already been traveled by the vehicle and spread with fertilizer. Corrective action and or alarm levels are raised based on a threshold of undesirable spread (paragraph 113).
The Applicant argues on pages 24 and 25 of the response in essence that: Claim 24, as amended, recites that each pixel of the change in pixels has an r value, a g value and a b value in an RGB color space, and a d value indicating a distance from a focal plane of the one or more image sensors. The cited portion of Wu (paragraph [0170]) describes comparing spray drift patterns based on a color image (e.g., spray dye); it does not describe a per-pixel depth (d) value, indicating a distance from a focal plane of the image sensor, as part of the determined change in pixels.
Wu discloses wherein each pixel of the change in pixels has an r value, a g value, and a b value in an RGB color space (paragraph 115, During the daytime, calibration is optional, green is reasonably identifiable by checking for the intensity of the green (e.g. in RGB wiring, or digitized image signals) component of the pixels), and a d value indicating a distance from a focal plane of the one or more image sensors (paragraph 125, FIG. 10 is an example method to generate 3D images obtained from two or more image sensor units 50, to determine distance, depth and/or height values).
The Applicant argues on pages 26 and 27 of the response in essence that:
Calibrating an image sensor to map image pixel points to real-world coordinates when the sensor's position changes is a sensor-calibration function; it is not estimating an amount of movement in space from a pre-spray first image to a post-spray second image caused by motion of the sensors on a moving vehicle, where both images depict a common plane of equal distance from the sensors, and it is not applying the homography matrix when performing image differencing between those two images. The Office Action's stated motivation - "to accommodate using sensors of different heights to detect an object in both images" - likewise concerns sensor calibration, not compensating for frame-to-frame vehicle motion during image differencing.
Humpal discloses that homography is used to calibrate the image sensors 122 to accommodate the fact that the image pixel points and real-world coordinates are different when image sensor position changes (paragraph 130). Accommodating for different positions of the sensors will inherently accommodate the amount of movement caused by motion of the vehicle.
The Applicant argues on pages 27 and 28 of the response in essence that: Takatsu does not teach or suggest using a failure to fit a line as a basis for determining that a spray did not happen or happened but not at a target agricultural object. Nor do Wu, Wiltshire or Polzounov.
Wu discloses determining that a spray of the first fluid projectile did not happen or happened but not at the first target agricultural object (paragraph 112, the procedure has instructions check the observed results from the captured image against the computed expected trajectory in X direction (transverse to travel direction, or in both X and Y directions, where the Y direction is parallel to travel direction). Takatsu discloses when a line cannot be fitted to the identified pixels (col. 4, lines 35-39, I Moreover, there are shown selectors 71 to 7n for selecting data, which serves as an addend, from among data (located by x and y coordinates) read from the associated latch circuits 21 to 2n according to a result of decoding (selection signal) provided by the decoder 6). Not selecting an addend will result in no line be fitted.
The Applicant argues on pages 29 and 30 of the response in essence that: Wu's paragraph [0173], cited for the temporal ordering of the time periods, describes only that sensors capture information periodically or continuously and that specific frames are grabbed during data acquisition; it likewise does not disclose capturing an emitted fluid projectile from frame entry through full splash while the supporting vehicle moves laterally.
Wu discloses wherein the second set of images is obtained while a vehicle supporting the agricultural treatment system moves in a lateral direction (paragraph 46, In some embodiments, the image sensing system includes rotatable or pivotable cameras or video devices), the second set of images capturing the emitted first fluid projectile from when the emitted first fluid projectile comes into a frame until the emitted first fluid projectile is fully splashed onto a surface of the first target agricultural object or the ground area (paragraph 109, The accessed image 210 also includes information representing a treated area where certain spray areas 1040 have been sprayed by the spray nozzles. In this case, the information indicating a plant treatment is illustrated as dampened soil 1210).
Allowable Subject Matter
Claims 9, 23, 29 and 30 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4-7, 10, 12-16, 18-21, 24 and 26-28 are rejected under 35 U.S.C. 103 as being unpatentable over Wu US Publication 2019/0150357 (hereafter “Wu”), Wiltshire US Publication 2020/0104719 (hereafter “Wiltshire”) and Polzounov et al. US Publication 2019/0362146 (hereafter “Polzounov”).
Referring to claim 1, Wu discloses a method of evaluating a treatment of an agricultural object, the method comprising:
obtaining with one or more image sensors coupled to an agricultural treatment system, at a first time period, a first set of images each comprising a plurality of pixels depicting a ground area and a first target agricultural object positioned in the ground area (paragraph 112, The procedure includes capturing the image, associating or overlaying a virtual grid 100 over the captured image, then check whether (dyed) spray went past a boundary “Boundary” associated with one of the Y-direction lines of the grid 100);
emitting a first fluid projectile of a first fluid at the first target agricultural object (paragraph 99, the planned crop row 12 is first or subsequently sprayed/spread with fertilizer after the herbicide spot spraying);
obtaining with the one or more image sensors at a second time period, a second set of images comprising a plurality of pixels depicting the emitted first fluid projectile and the ground area and the agricultural object (paragraph 52, In some embodiments, the images are from videos. For example, when the vehicle is traveling, the video is constantly filming. At very specific intervals or upon a command to send images, a particular image from the video is extracted for analysis);
identifying a first group of pixels that represent a first spray object (paragraph 170, In yet another embodiment, the calculated predicted spray drift are compared with the detected spray drift based on a color image (e.g. spray dye of the end units). If the predicted and detected spray drift pattern on the ground agree to within a predetermined threshold (e.g. 85 to 95%), and over-or-under spraying or drift is found, the spray nozzles or agricultural vehicle take corrective actions and/or alert the operator);
identifying a second group of pixels that represent a spray projectile of the emitted first fluid projectile in an image of the second set of images (paragraph 112, The procedure includes capturing the image, associating or overlaying a virtual grid 100 over the captured image, then check whether (dyed) spray went past a boundary “Boundary” associated with one of the Y-direction lines of the grid 100); and
indexing the identified first spray object and the identified spray projectile, wherein the identified spray projectile is indexed as a 3-dimensional vector, or a 2-dimensional or 3- dimensional model of a full 3-dimensional profile of the spray projectile, having a shape and an orientation mapped into a virtual scene (paragraph 168, The spray cone envelope calculations are performed in a reference frame where the effective central axis is determined based on instantaneous wind and travel directions (vector sum of the vehicle travel velocity and wind velocity) with respect to the ground) (paragraph 156, In some embodiments, some percentage of all of the data (every sensor, vehicle speed and position, and every image, pixel or array member, i.e. completely unbiased data) is logged so that a secondary or tertiary analysis aids in improving or corroborating (e.g. a cross check of) the initial on-vehicle, real time trigger analysis used to decide the real time action).
While Wu discloses identifying a first group of pixels that represent a first spray object, Wu does not disclose expressly the identification is based on a determined change in pixels between the first and second images.
Wiltshire discloses comparing a first image of the first set of images with a second image of the second set of images to determine a change in pixel values between at least the first image and at least the second image (paragraph 41, At block 312, the method 300 includes a classification module to classify each pair of patches according to the updated feature map and the resultant probability that is greater. For example, if the matching probability is greater than 50%, the non-match probability is less than 50%, and the classifier classifies the pair of adjacent patches as a match. Conversely, if the matching probability is less than 50%, the non-matching probability is higher than 50%, and the classifier classifies the pair of adjacent patches as a non-match); and
based on the determined change in pixel values as between the first image and second image, identifying a first group of pixels that represent a first object (paragraph 43, At block 314, the method 300 includes an identification module for identifying dissimilar patches based on the classification probability from block 312).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to determine a change in pixels between the first and second images. The motivation for doing so would have been to effectively identify differences between images to obtain additional information that is not easily detected from analyzing each image separately.
Wu and Wiltshire do not disclose expressly wherein the determined change in pixel values indicates a spray impact on a ground area, a spray impact on an agricultural object and/or a spray splat.
Polzounov discloses wherein the determined change in pixel values indicates a spray impact on a ground area, a spray impact on an agricultural object and/or a spray splat (paragraph 108-110, In this example, the verification mechanism 150 is mounted to the rear of the farming machine 100 such that the area of the field is imaged after the spray nozzles pass over the area and treat the cotton plants. The accessed image 210 also includes information representing a treated area where certain spray areas 1040 have been sprayed by the spray nozzles. In this case, the information indicating a plant treatment is illustrated as dampened soil 1210. Returning to FIG. 11, the control system applies 1130 a model 800 to identify pixels in the accessed post-image 212 representing a treated area).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to analyze a spray area. The motivation for doing so would have been to verify that intended areas have been sprayed. Therefore, it would have been obvious to combine Wiltshire and Polzounov with Wu to obtain the invention as specified in claim 1.
Referring to claims 2 and 16, Wu discloses Polzounov discloses determining a quantity of fluid that impacted the ground area and/or that impacted the agricultural object by identifying a group of pixels that comprise the agricultural object, and another group of pixels that depict the spray impact on the ground and the spray impact on the agricultural object (paragraph 109, The accessed image 210 also includes information representing a treated area where certain spray areas 1040 have been sprayed by the spray nozzles. In this case, the information indicating a plant treatment is illustrated as dampened soil 1210 [determining that an area is not damped is determining a quantity of fluid that impacted the ground i.e. zero fluid]).
Referring to claim 4, Wu discloses determining an inaccurate application of the emitted first fluid projectile to the first target agricultural object by determining whether the spray object lines up with the first target agricultural object and applying a second corrected fluid projectile to the first target agricultural object (paragraph 170, In yet another embodiment, the calculated predicted spray drift are compared with the detected spray drift based on a color image (e.g. spray dye of the end units). If the predicted and detected spray drift pattern on the ground agree to within a predetermined threshold (e.g. 85 to 95%), and over-or-under spraying or drift is found, the spray nozzles or agricultural vehicle take corrective actions and/or alert the operator), and
determining, based on the inaccurate application, an adjustment accounting for at least one of: a wind that moved the emitted first fluid projectile, a speed of a vehicle supporting the agricultural treatment system, or a mechanical defect causing a misalignment between the first target agricultural object and a line of sight of a treatment head of the agricultural treatment system (paragraph 112, The travel path of the droplets is corrected for factors such as wind direction, local eddies, vehicle travel direction, speed of vehicle, and humidity to determine or to correct for spray drift cone versus expected or ideal spray travel cone, and is also correlated with droplet size (where larger droplets do not veer as much) and the chemical composition (where additives in the spray fluid can also reduce spray drift)).
Referring to claims 5 and 19, Wu discloses determining the first group of pixels of the first spray object is the spray impact on the agricultural object (paragraph 168, The image of the spray cones that are dyed or illuminated is analyzed (e.g. pattern color and intensity) to determine the average spray envelope of the cone, then correlated with a calibrated envelope threshold value with a known spray envelope (e.g. 90-95% droplets within an envelope) with respect to droplets, using the color/intensity image captured for each particular angle of view of the spray cone);
determining a second group of pixels is a second spray object of a spray impact on a ground area about the agricultural object (paragraph 168, Yet another embodiment project the vectors and perform calculations along the two axes parallel and perpendicular to the geometrical boundaries of the field in order to assess if the spray has gone beyond the boundary or if the spray pattern on the ground indicate uneven coverage (“skipping”)); and
based on the group of pixels representing the spray impact on the agricultural object and the second group of pixels of the spray impact on the ground area, determining a quantity of the first fluid projectile that likely comprises the spray object (paragraph 170, In yet another embodiment, the calculated predicted spray drift are compared with the detected spray drift based on a color image (e.g. spray dye of the end units). If the predicted and detected spray drift pattern on the ground agree to within a predetermined threshold (e.g. 85 to 95%), and over-or-under spraying or drift is found, the spray nozzles or agricultural vehicle take corrective actions and/or alert the operator).
Referring to claims 6 and 20, Wu discloses based on the group of pixels representing the spray object, determining a spray coverage percentage based on an average of the multiple fluid projectiles identified to have actually been sprayed upon their intended target object (paragraph 170, In yet another embodiment, the calculated predicted spray drift are compared with the detected spray drift based on a color image (e.g. spray dye of the end units). If the predicted and detected spray drift pattern on the ground agree to within a predetermined threshold (e.g. 85 to 95%), and over-or-under spraying or drift is found, the spray nozzles or agricultural vehicle take corrective actions and/or alert the operator).
Referring to claims 7 and 21, Wu discloses predetermining a predicted spray path of the first fluid projectile; and
based on a group of pixels representing a spray object, identifying a line of the spray object by performing spray segmentation only in a portion of an image of the second set of images contained in a region defined by the predicted spray path, wherein the spray object is the spray projectile of the emitted first fluid projectile (paragraph 112, the procedure includes capturing the image, associating or overlaying a virtual grid 100 over the captured image, then check whether (dyed) spray went past a boundary “Boundary” associated with one of the Y-direction lines of the grid 100).
Referring to claim 10, Wu discloses determining a change in pixels (colors/luminosity/etc.) that are above a threshold value of a ground area and/or the target agricultural object in the first image as compared to the second image (paragraph 170, In yet another embodiment, the calculated predicted spray drift are compared with the detected spray drift based on a color image (e.g. spray dye of the end units). If the predicted and detected spray drift pattern on the ground agree to within a predetermined threshold (e.g. 85 to 95%), and over-or-under spraying or drift is found, the spray nozzles or agricultural vehicle take corrective actions and/or alert the operator), wherein each pixel of the change in pixels has an r value, a g value, and a b value in an RGB color space (paragraph 115, During the daytime, calibration is optional, green is reasonably identifiable by checking for the intensity of the green (e.g. in RGB wiring, or digitized image signals) component of the pixels), and a d value indicating a distance from a focal plane of the one or more image sensors (paragraph 125, FIG. 10 is an example method to generate 3D images obtained from two or more image sensor units 50, to determine distance, depth and/or height values).
Referring to claims 12 and 26, Wu discloses based on the first group of pixels representing the first spray object, determining a first emission pattern of the first fluid projectile (paragraph 170, In yet another embodiment, the calculated predicted spray drift are compared with the detected spray drift based on a color image (e.g. spray dye of the end units). If the predicted and detected spray drift pattern on the ground agree to within a predetermined threshold (e.g. 85 to 95%), and over-or-under spraying or drift is found, the spray nozzles or agricultural vehicle take corrective actions and/or alert the operator); and
indexing the first spray object as the first emission pattern (paragraph 52, To synchronize different image sensor (e.g. cameras), a single image from the different videos are picked off based upon a particular time stamp on the image) (paragraph 156, In some embodiments, some percentage of all of the data (every sensor, vehicle speed and position, and every image, pixel or array member, i.e. completely unbiased data) is logged so that a secondary or tertiary analysis aids in improving or corroborating (e.g. a cross check of) the initial on-vehicle, real time trigger analysis used to decide the real time action);
generating and superimposing 3-dimensional models of the first target agricultural object, the spray projectile, and a splash pattern on each other to reconstruct a spray action from targeting of the first target agricultural object, to spraying of the first target agricultural object, to a splash made and a splat detected (paragraph 126, If two or more lenses are used together to collect images and videos or if stereo image sensor are used, the upcoming terrain is reconstructed in 3 dimensions).
Referring to claims 13 and 27, Wu discloses based on the first group of pixels representing the first spray object, determining a first treatment pattern (paragraph 170, In yet another embodiment, the calculated predicted spray drift are compared with the detected spray drift based on a color image (e.g. spray dye of the end units). If the predicted and detected spray drift pattern on the ground agree to within a predetermined threshold (e.g. 85 to 95%), and over-or-under spraying or drift is found, the spray nozzles or agricultural vehicle take corrective actions and/or alert the operator); and
indexing the first spray object as the first treatment pattern (paragraph 52, To synchronize different image sensor (e.g. cameras), a single image from the different videos are picked off based upon a particular time stamp on the image) (paragraph 156, In some embodiments, some percentage of all of the data (every sensor, vehicle speed and position, and every image, pixel or array member, i.e. completely unbiased data) is logged so that a secondary or tertiary analysis aids in improving or corroborating (e.g. a cross check of) the initial on-vehicle, real time trigger analysis used to decide the real time action), wherein determining the first treatment pattern comprises removing pixels associated with the spray projectile and storing and cataloguing remaining pixels representing a spot of fluid, a splat of fluid, and/or a splash of fluid as the first treatment pattern (paragraph 103, To avoid needless spray (e.g. crop residue, twig), such objects are calibrated out by void pattern instructions performed on an image captured).
Referring to claims 14 and 28, Wu discloses based on the first group of pixels representing a first spray object, indexing the agricultural object as being treated (paragraph 66, Detect and assess the amount of residue on the ground in order for a cultivator or planter to adjust the pressure on its implements) (paragraph 156, In some embodiments, some percentage of all of the data (every sensor, vehicle speed and position, and every image, pixel or array member, i.e. completely unbiased data) is logged so that a secondary or tertiary analysis aids in improving or corroborating (e.g. a cross check of) the initial on-vehicle, real time trigger analysis used to decide the real time action), wherein indexing the agricultural object as being treated comprises adding a verification of a treatment of the agricultural object to a treatment history of the agricultural object, wherein the treatment history is used to determine whether to perform a subsequent treatment of the agricultural object upon detection of the agricultural object at a later time or a later phenological stage of the agricultural object (paragraph 113, The algorithm analyzes the captured image elements to verify if the spread statistics are correct, and spread uniformly within, say, less than 10% difference among the captured images either in the other crop regions or the crop rows that have already been traveled by the vehicle and spread with fertilizer. Corrective action and or alarm levels are raised based on a threshold of undesirable spread).
Referring to claim 15, Wu discloses a system for treating agricultural objects, the system comprising:
a first treatment unit, first treatment unit having at least one spraying head configured to emit a fluid projectile, the spraying head moveable about an Θ position and a Ψ position (paragraph 169, Example spray nozzles have adjustable partitioning doors, valves, or “blinders” that are remotely controlled to close or open to adjust the spray fan angle in the side-side and/or fore-aft direction);
one or more image sensors configured to obtain 3-dimensional image data (paragraph 52, Images from adjacent or near adjacent image sensor units 50 are stitched together to map the terrain or crop row 12 to form a 3D image); and
one or more processors, the one more processors configured to:
obtain with one or more image sensors at a first time period, a first set of images each comprising a plurality of pixels depicting a ground area and a first target agricultural object positioned in the ground area (paragraph 112, The procedure includes capturing the image, associating or overlaying a virtual grid 100 over the captured image, then check whether (dyed) spray went past a boundary “Boundary” associated with one of the Y-direction lines of the grid 100);
instruct, via a controller, the emitting a first fluid projectile of a first fluid at the first target agricultural object (paragraph 99, he planned crop row 12 is first or subsequently sprayed/spread with fertilizer after the herbicide spot spraying);
obtain with the one or more image sensors at a second time period, a second set of images comprising a plurality of pixels depicting the emitted first fluid projectile and the ground area and the agricultural object paragraph 52, In some embodiments, the images are from videos. For example, when the vehicle is traveling, the video is constantly filming. At very specific intervals or upon a command to send images, a particular image from the video is extracted for analysis);
identify first group of pixels that represent a first spray object (paragraph 112, The procedure includes capturing the image, associating or overlaying a virtual grid 100 over the captured image, then check whether (dyed) spray went past a boundary “Boundary” associated with one of the Y-direction lines of the grid 100);
identify a second group of pixels that represent a spray projectile of the emitted first fluid projectile in an image of the second set of images (paragraph 170, In yet another embodiment, the calculated predicted spray drift are compared with the detected spray drift based on a color image (e.g. spray dye of the end units). If the predicted and detected spray drift pattern on the ground agree to within a predetermined threshold (e.g. 85 to 95%), and over-or-under spraying or drift is found, the spray nozzles or agricultural vehicle take corrective actions and/or alert the operator); and
index the identified first spray object and the identified spray projectile, wherein the identified spray projectile is indexed as a 3-dimensional vector, or a 2-dimensional or 3- dimensional model of a full 3-dimensional profile of the spray projectile, having a shape and an orientation mapped into a virtual scene (paragraph 168, The spray cone envelope calculations are performed in a reference frame where the effective central axis is determined based on instantaneous wind and travel directions (vector sum of the vehicle travel velocity and wind velocity) with respect to the ground) (paragraph 156, In some embodiments, some percentage of all of the data (every sensor, vehicle speed and position, and every image, pixel or array member, i.e. completely unbiased data) is logged so that a secondary or tertiary analysis aids in improving or corroborating (e.g. a cross check of) the initial on-vehicle, real time trigger analysis used to decide the real time action).
While Wu discloses identifying a first group of pixels that represent a first spray object, Wu does not disclose expressly the identification is based on a determined change in pixels between the first and second images.
Wiltshire discloses compare the first image with the second image to determine a change in pixels between at least a first image of the first set of images and at least a second image of the second set of images (paragraph 41, At block 312, the method 300 includes a classification module to classify each pair of patches according to the updated feature map and the resultant probability that is greater. For example, if the matching probability is greater than 50%, the non-match probability is less than 50%, and the classifier classifies the pair of adjacent patches as a match. Conversely, if the matching probability is less than 50%, the non-matching probability is higher than 50%, and the classifier classifies the pair of adjacent patches as a non-match); and
based on the determined change in pixel values as between the first and second images, identify first group of pixels that represent a first object (paragraph 43, At block 314, the method 300 includes an identification module for identifying dissimilar patches based on the classification probability from block 312).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to determine a change in pixels between the first and second images. The motivation for doing so would have been to effectively identify differences between images to obtain additional information that is not easily detected from analyzing each image separately.
Wu and Wiltshire do not disclose expressly wherein the determined change in pixel values indicates a spray impact on a ground area, a spray impact on an agricultural object and/or a spray splat.
Polzounov discloses wherein the determined change in pixel values indicates a spray impact on a ground area, a spray impact on an agricultural object and/or a spray splat (paragraph 108-110, In this example, the verification mechanism 150 is mounted to the rear of the farming machine 100 such that the area of the field is imaged after the spray nozzles pass over the area and treat the cotton plants. The accessed image 210 also includes information representing a treated area where certain spray areas 1040 have been sprayed by the spray nozzles. In this case, the information indicating a plant treatment is illustrated as dampened soil 1210. Returning to FIG. 11, the control system applies 1130 a model 800 to identify pixels in the accessed post-image 212 representing a treated area).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to analyze a spray area. The motivation for doing so would have been to verify that intended areas have been sprayed. Therefore, it would have been obvious to combine Wiltshire and Polzounov with Wu to obtain the invention as specified in claim 15.
Referring to claim 18, Wiltshire discloses wherein the performing image segmentation comprises:
aligning the first image and the second image using features or common pixel patterns in the images (paragraph 37, As the UAV flies over a scene, a plurality of digital images of the scene having the same resolution is captured by the imaging subsystem 12, namely the optics 16, and are aligned according with the imagery in the field of view 11 by an alignment module, block 304); and
generating a pixel mask of the first spray object (paragraph 43, The mask 602 identifies a change in the corresponding patch locations and a degree of change between the input images 400 and 500. The degree of change can include information such as color mapping based on a corresponding classification confidence level associated with the corresponding patch. The mask 602 can then be overlaid on top of the latter of two input images, e.g. image 500, as shown in reference image 600 of FIG. 6).
Referring to claim 24, Wu discloses determining a change in pixels (colors/luminosity/etc.) that are above a threshold value (paragraph 170, In yet another embodiment, the calculated predicted spray drift are compared with the detected spray drift based on a color image (e.g. spray dye of the end units). If the predicted and detected spray drift pattern on the ground agree to within a predetermined threshold (e.g. 85 to 95%), and over-or-under spraying or drift is found, the spray nozzles or agricultural vehicle take corrective actions and/or alert the operator).
Claims 3 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Wu US Publication 2019/0150357, Wiltshire US Publication 2020/0104719 and Polzounov et al. US Publication 2019/0362146 as applied to claims 1 and 15 above, and further in view of Humpal et al. US Publication 2022/0192174 (hereafter “Humpal”).
Referring to claims 3 and 17, Wu discloses when the agricultural treatment system obtained the first set and/or second set of images by the one or more image sensors, but does not disclose expressly creating a homography matrix.
Humpal discloses creating a homography matrix to account for motion when the agricultural treatment system obtained the first set and/or second set of images by the one or more image sensors; and
applying the homography matrix to account for a point of view change between the first image and the second image and using the homography matrix to account for an image misalignment of the first image and the second image, wherein the homography matrix estimates an amount of movement in space from the first image to the second image caused by motion of the one or more image sensors on a moving vehicle, the first image and the second image depicting a common plane of equal distance from the one or more image sensors, and wherein the homography matrix is applied when performing image differencing between the first image and the second image (paragraph 130, Homography is used to calibrate the image sensors 122 to accommodate the fact that the image pixel points and real-world coordinates are different when image sensor position changes).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to apply a homography matrix. The motivation for doing so would have been to accommodate using sensors of different heights to detect an object in both images. Therefore, it would have been obvious to combine Humpal with Wu to obtain the invention as specified in claims 3 and 17.
Claims 8, 9, 22 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Wu US Publication 2019/0150357, Wiltshire US Publication 2020/0104719 and Polzounov et al. US Publication 2019/0362146 as applied to claims 7 and 21 above, and further in view of Takatsu US Patent 6,430,319 (hereafter “Takatsu”).
Referring to claims 8 and 22, Wu discloses identifying pixels in the second image depicting the emitted first fluid projectile, wherein the second image depicts both the emitted first fluid projectile and the identified first spray object (paragraph 112, FIG. 8 depicts another example Master Application, which is used to detect spray drift and the spray pattern landing on the ground of a crop field);
determining that a spray of the first fluid projectile did not happen or happened but not at the first target agricultural object (paragraph 112, the procedure has instructions check the observed results from the captured image against the computed expected trajectory in X direction (transverse to travel direction, or in both X and Y directions, where the Y direction is parallel to travel direction)
Wu does not disclose expressly line fitting the identified pixels to determine the spray line of the spray projectile.
Takatsu discloses line fitting the identified pixels to determine the spray line of the spray projectile (col. 1, lines 48-59, Based on the foregoing principle, a trajectory linking N points in an image is drawn in the ρ-θ space, and the cross point of the trajectory is obtained in order to define a straight line fitted to the set of points in the image); and
when a line cannot be fitted to the identified pixels (col. 4, lines 35-39, I Moreover, there are shown selectors 71 to 7n for selecting data, which serves as an addend, from among data (located by x and y coordinates) read from the associated latch circuits 21 to 2n according to a result of decoding (selection signal) provided by the decoder 6 [not selecting an addend will result in no line be fitted]).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to fit a line to determine a trajectory. The motivation for doing so would have been to increase the accuracy and to simply the calculations required for determining the direction of an object. Therefore, it would have been obvious to combine Takatsu with Wu to obtain the invention as specified in claims 8 and 22.
Claims 11 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Wu US Publication 2019/0150357, Wiltshire US Publication 2020/0104719 and Polzounov et al. US Publication 2019/0362146 as applied to claims 1 and 17 above, and further in view of Nevenka et al. US Publication 2003/0108334 (hereafter “Nevenka”).
Referring to claims 11 and 25, Wu discloses wherein the second time period is temporally later than the first time period (paragraph 173, The example sensors capture information periodically (e.g. camera, one image at a time) and/or capture information continuously (e.g. video, continuous images). Even in some embodiments where the sensors are operated continuously, specific frames are grabbed during the data acquisition of the sensor data),
wherein the second set of images is obtained while a vehicle supporting the agricultural treatment system moves in a lateral direction (paragraph 46, In some embodiments, the image sensing system includes rotatable or pivotable cameras or video devices), the second set of images capturing the emitted first fluid projectile from when the emitted first fluid projectile comes into a frame until the emitted first fluid projectile is fully splashed onto a surface of the first target agricultural object or the ground area (paragraph 109, The accessed image 210 also includes information representing a treated area where certain spray areas 1040 have been sprayed by the spray nozzles. In this case, the information indicating a plant treatment is illustrated as dampened soil 1210).
Wu does not disclose expressly that the duration between the first time period and second time period is less than 60 seconds.
Nevenka discloses that the duration between the first time period and second time period is less than 60 seconds (paragraph 68, It should be noted that the process of grabbing and analyzing frames is preferably performed at pre-defined intervals for each recording device. For instance, when a recording device begins recording data, keyframes can be grabbed every 30 seconds).
Before the effective filing date of the claimed invention, it would have obvious to a person of ordinary skill in the art to grab an image every 30 seconds. The motivation for doing so would have been to efficiently perform detection while also reducing the memory and processing involved. Furthermore, selecting the interval of uploading images is a matter of design choice. Therefore, it would have been obvious to combine Nevenka with Wu to obtain the invention as specified in claims 11 and 25.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PETER K HUNTSINGER whose telephone number is (571)272-7435. The examiner can normally be reached Monday - Friday 8:30 - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Benny Q Tieu can be reached at 571-272-7490. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PETER K HUNTSINGER/Primary Examiner, Art Unit 2682
Prosecution Insights