DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 2/02/2026 has been entered.
Drawings
The drawings were received on 06/03/2021. These drawings are acceptable.
Information Disclosure Statement
The information disclosure statements (IDSs) submitted 03/20/2022 and 06/08/2021 have been considered by the examiner.
Response to Arguments
Applicant's arguments filed 2/02/2026 have been fully considered.
Regarding applicant’s remarks directed to the 35 USC 101 rejection, examiner notes that the newly amended limitations have been reviewed and the rejection made in the previous office action has been withdrawn.
Regarding remarks directed to the rejections under 35 USC 102 and 103, are directed to elements not previous rejected by the examiner. See rejection below, that addresses the filed claim amendments.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7 are rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US 20210089771, hereinafter ‘Fu’) in view of Kim et al. (US 10970520, hereinafter ‘Kim’).
Regarding independent claim 1, Fu teaches a method implemented by one or more processors, the method comprising: (in [0137] The storage unit 1416 includes a machine-readable medium 1422 on which is stored instructions 1424 (e.g., software) embodying any one or more of the methodologies or functions described herein. For example, the instructions 1424 may include the functionalities of modules of the system 130 described in FIG. 2. The instructions 1424 may also reside, completely or at least partially, within the main memory 1404 or within the processor 1402 [a method implemented by one or more processors, the method comprising] (e.g., within a processor's cache memory) during execution thereof by the computer system 1400, the main memory 1404 and the processor 1402 also constituting machine-readable media. …)
generating a plurality of simulated images, each simulated image depicting one or more simulated instances of a plant in a plurality of plants, (in [0100] There are several methods of simulating non-ideal operating conditions [generating a plurality of simulated images, each simulated image depicting one or more simulated instances of a plant] in an accessed image obtained by a farming machine 100 operating in an ideal operating condition. In one example, a control system 130 simulates airborne particulates in a labelled image by employing a particulate augmentation model... In this manner, one or more previously accessed images may be used to train a plant identification model to identify plants in non-ideal conditions [simulated image depicting one or more simulated instances of a plant in a plurality of plants]. That is, the particulate augmentation model generates an array of particulate images from a previously labelled image [in a plurality of plants] such that plant matter generated images are correctly labelled despite being obscured by (simulated) particulates [generating a plurality of simulated images, each simulated image depicting one or more simulated instances of a plant in a plurality of plants]… [0106] FIGS. 9C-9F illustrate particulate images [in a plurality of plants] generated according to various particulate levels [generating a plurality of simulated images, each simulated image depicting one or more simulated instances of a plant in a plurality of plants]….; And in [0112] The array of particulate images [generating a plurality of simulated images, each simulated image depicting one or more simulated instances of a plant in a plurality of plants] may be used to train a plant identification model (e.g., model 500) to identify plants in non-ideal operating conditions…)
the generating comprising: extracting, from a plurality of ground truth images, attributes for the plant based on the plurality of ground truth images, ([0112] The array of particulate images [the generating comprising: extracting, from a plurality of ground truth images, attributes for the plant based on the plurality of ground truth images] may be used to train a plant identification model (e.g., model 500) to identify plants in non-ideal operating conditions. Arrays of particulate images are informative because each particulate image corresponds to a previously labelled image […based on the plurality of ground truth images]. That is, even if the particulate image includes simulated particulates that wholly obscure plant matter (i.e., an obscuring pixel), the obscuring pixel is still labelled as plant matter. In this way, a model can be trained to identify latent information in an image to identify plants when one or more of the pixels representing the plant are obscured pixels [extracting, from a plurality of ground truth images, attributes for the plant based on the plurality of ground truth images, as pixels associated with obscure plant matter]. For example, referring to FIGS. 9F and 9A, a second type of plant is represented by a group of pixels in the bottom left of the labelled image 910. In the particulate image 960, a portion of the plant is obscured by the simulated particulates. However, the obscuring pixels are still labeled as pixels representing the second type of plant. Because the particulate image is used to train a model, the model identifies latent information in an image representing the plant despite the presence of airborne particulates that obscure all or some portion of the plant.)
generating, using the extracted plant attributes, a three dimensional model of the plant; generating the plurality of simulated images by rendering the three- dimensional model as the plurality of simulated images; (in [0068] The control system 130 applies the model 500 to relate accessed images 210 in the convolutional layer 510 to plant and particulate identification information in the identification layer 530 [generating, using the extracted plant attributes, a three dimensional model of the plant; generating the plurality of simulated images by rendering the three- dimensional model as the plurality of simulated images]. The control system 130 retrieves relevance information between these elements can by applying a set of transformations (e.g., W.sub.1, W.sub.2, etc.) between the corresponding layers. Continuing with the example from FIG. 5, the convolutional layer 510 of the model 500 represents an encoded accessed image 210, and identification layer 530 of the model 500 represents plant and particulate identification information. The control system 130 identifies plants and particulates in an accessed image 210 by applying the transformations W.sub.1 and W.sub.2 to the pixel values of the accessed image 210 in the space of convolutional layer 510. The weights and parameters for the transformations may indicate relationships between information contained in the accessed image and the identification of a plant and/or particulates.)
for each of the plurality of simulated images, labeling the simulated image with at least one ground truth label that identifies an attribute of the one or more simulated instances of the plant depicted in the simulated image, wherein the attribute describes both a visible portion and an occluded portion of the one or more simulated instances of the plant depicted in the simulated image; (in [0111] The approach of modelling various particulate levels for one image, as illustrated in FIGS. 9C-9F, is superior than collecting images with varying levels of obscuration [for each of the plurality of simulated images, labeling the simulated image with at least one ground truth label that identifies an attribute of the one or more simulated instances of the plant depicted in the simulated image,] because human labelling of acquired images is costly, time-consuming, and error prone. Of course, while FIGS. 9C-9F only showed 4 examples of generated particulate levels (e.g., 0.0, 0.34, 0.66, and 1.0), many other possible particulate levels are also possible. More generally, the stochastic nature of the technique described herein allows a control system 130 (or some other system) to generate a vast number of labelled particulate images [wherein the attribute describes both a visible portion and an occluded portion of the one or more simulated instances of the plant depicted in the simulated image]…. [0112] The array of particulate images may be used to train a plant identification model (e.g., model 500) to identify plants in non-ideal operating conditions. Arrays of particulate images are informative because each particulate image corresponds to a previously labelled image. That is, even if the particulate image includes simulated particulates that wholly obscure plant matter (i.e., an obscuring pixel), the obscuring pixel is still labelled as plant matter [for each of the plurality of simulated images, labeling the simulated image with at least one ground truth label that identifies an attribute of the one or more simulated instances of the plant depicted in the simulated image]. In this way, a model can be trained to identify latent information in an image to identify plants when one or more of the pixels representing the plant are obscured pixels… However, the obscuring pixels are still labeled as pixels representing the second type of plant… [0113] There are various methods for generating arrays of particulate image for training a plant identification model... Thus, in another example, the control system 130 determines a particulate probability [wherein the attribute describes both a visible portion and an occluded portion of the one or more simulated instances of the plant depicted in the simulated image] for a labelled image and generates particulate images based on the probability [for each of the plurality of simulated images, labeling the simulated image with at least one ground truth label that identifies an attribute of the one or more simulated instances of the plant depicted in the simulated image]. The particulate probability is a quantification of a likelihood that a labelled image includes airborne particulates [wherein the attribute describes both a visible portion and an occluded portion of the one or more simulated instances of the plant depicted in the simulated image]… [0111] The approach of modelling various particulate levels for one image [for each of the plurality of simulated images, labeling the simulated image with at least one ground truth label that identifies an attribute of the one or more simulated instances of the plant depicted in the simulated image], as illustrated in FIGS. 9C-9F, is superior than collecting images with varying levels of obscuration because human labelling of acquired images is costly, time-consuming, and error prone. Of course, while FIGS. 9C-9F only showed 4 examples of generated particulate levels (e.g., 0.0, 0.34, 0.66, and 1.0), many other possible particulate levels are also possible [for each of the plurality of simulated images, labeling the simulated image with at least one ground truth label that identifies an attribute of the one or more simulated instances of the plant depicted in the simulated image, wherein the attribute describes both a visible portion and an occluded portion of the one or more simulated instances of the plant depicted in the simulated image as the collection of generated particulate images labelled with a generated particle level as claimed attribute]…)
and training a machine learning model to make an agricultural prediction using the labeled plurality of simulated images. (in [0114] A control system 130 employing plant identification model trained using particulate images [and training a machine learning model to make an agricultural prediction using the labeled plurality of simulated images] is more precise and accurate at identifying plants in non-ideal operating conditions. For example, FIGS. 10A-11C compare the identification capabilities of a plant identification model that is not trained using particulate images (“normal model”) and a plant identification model trained using particulate images (“augmented model”) [training a machine learning model to make an agricultural prediction using the labeled plurality of simulated images]. Fu teaches the identify model as a convolutional neural network model, in [0065] In the illustrated embodiment, referred to throughout the remainder of the specification, the plant identification model 500 is a convolutional neural network model with layers of nodes, in which values at nodes of a current layer are a transformation of values at nodes of a previous layer.)
and as an agricultural robot moves through a field, capturing, using an image system of the agricultural robot, an image of a plant in the field; applying, using a processor of the agricultural robot, the trained machine learning model to the image to determine an agricultural prediction for the plant; and performing, using the agricultural robot, a remedial action on the plant based on the agricultural prediction. (in [0121] A control system 130 employing an augmented model is more precise and accurate than a normal model which also leads to improved treatment of identified plants (e.g., less overspray, fewer spray misses, etc.). For example, FIGS. 12A-13B compare the treatment capabilities of a farming machine employing a normal model (“normal machine”) vs. a farming machine employing an augmented model (“augmented machine”) [applying, using a processor of the agricultural robot, the trained machine learning model to the image to determine an agricultural prediction for the plant as farming machine employing a machine learning algorithm] to treat plants in a field [and performing, using the agricultural robot, a remedial action on the plant based on the agricultural prediction].; And in [0083] The farming machine 100 images an area of the field using the detection mechanism 110 [a plant in an image captured by an agricultural robot in a field]. The image includes information representing cotton plants, weed plants, airborne particulates, and soil in the field. In this example, the detection mechanism 110 is mounted to the front of the farming machine 100 such that the area of the field is imaged before the front end of the farming machine 100 passes over the area [and as an agricultural robot moves through a field, capturing, using an image system of the agricultural robot, an image of a plant in the field]. The detection mechanism 110 transmits the image to the control system 130 of the farming machine 100; And as depicted in Fog. 5 and Fig. 6, And in [0064] Semantic segmentation may be implemented by a control system 130 using a plant identification model. A farming machine 100 [and as an agricultural robot moves through a field, capturing, using an image system of the agricultural robot, an image of a plant in the field; applying, using a processor of the agricultural robot, the trained machine learning model to the image to determine an agricultural prediction for the plant] can execute the plant identification model to identify features (e.g., plants, particulates, soil, etc.) in an accessed image (e.g., accessed image 210) and quickly generate an accurate treatment map. FIG. 5 is a representation of a plant identification model based on accessed images and previously identified plants, according to one example embodiment. As described in greater detail below, the plant identification model can identify plants in both ideal and non-ideal operating conditions […applying, using a processor of the agricultural robot, the trained machine learning model to the image to determine an agricultural prediction for the plant; …]. The previously identified plants may have been identified by another plant identification model or a human identifier. [0065] In the illustrated embodiment, referred to throughout the remainder of the specification, the plant identification model 500 is a convolutional neural network model with layers of nodes, in which values at nodes of a current layer are a transformation of values at nodes of a previous layer…[0081] FIGS. 6 and 7 illustrate a specific example of a farming machine 100 identifying and treating plants using plant treatment mechanisms 120 as the farming machine 100 travels through the field [and as an agricultural robot moves through a field, capturing, using an image system of the agricultural robot, an image of a plant in the field; applying, using a processor of the agricultural robot, the trained machine learning model to the image to determine an agricultural prediction for the plant]. The farming machine 100 is operating in non-ideal operating conditions and airborne particulates are present in the field. The farming machine 100 determines a particulate level for the field and generates a notification based on a determined particulate level. In this example, the farming machine 100 is a crop sprayer operating in a field planted with cotton (e.g., first plant type 220). The farming machine is configured to identify weeds (e.g. second plant type 230) in the field and treat the identified weeds by spraying them with an herbicide [and performing, using the agricultural robot, a remedial action on the plant based on the agricultural prediction]. The farming machine 100 is configured with a single row of eight spray nozzles that serve as treatment mechanisms 120. That is, the spray nozzles spray an herbicide when actuated by the farming machine 100. The farming machine 100 includes a detection mechanism 110 that captures images of plants and airborne particulates in the field as the farming machine 100 travels down the cotton crop rows. Further, the farming machine 100 includes a control system 130 that identifies plants and particulates in the field and controls the spray nozzles [and performing, using the agricultural robot, a remedial action on the plant based on the agricultural prediction]...)
While Fu teaches the use of the convolutional neural network to process image attributes as pixel features for modeling three-dimensional rendering of the image plant data for processing and making predictions using deep learning models such as a neural network, as noted above.
One of ordinary skill in the art would know that image processing tasks can include processing units to learn three dimensional data using convolutional neural networks.
Kim expressly teaches image processing tasks can include processing units to learn three dimensional data using convolutional neural networks, in 4:60-5:5: The three-dimensional image generation unit 120 generates a plurality of three-dimensional data using the two-dimensional images received from the image acquisition unit 110 [generating, using the extracted plant attributes, a three dimensional model of ]. As a simple example, the three-dimensional image generation unit 120 may convert the two-dimensional images into a first three-dimensional data by stacking the two-dimensional images, and reconstruct the three-dimensional data into a plurality of pieces to generate a second three-dimensional data by rotating the converted first three-dimensional data at a predetermined angle in a three-dimensional space, preferably, rotating any one axis among three axes (x, y, z) to the position of another axis in a three-dimensional space… 5:47-53: The deep learning algorithm analysis unit 130 analyzes a three-dimensional image by applying a two-dimensional convolutional neural network (CNN) to each of the plurality of reconstructed three-dimensional data [generating, using the extracted plant attributes, a three dimensional model of ] and combining the results of applying the two-dimensional convolutional neural network to each three-dimensional data…
PNG
media_image1.png
438
850
media_image1.png
Greyscale
Kim and Fu are analogous art because both involve developing image processing and analysis techniques using deep learning machine models.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for developing computer-implemented image analysis techniques using deep learning based image reconstruction techniques, as disclosed by Kim with the method of using artificial intelligence to identify and treat plants in a field use deep learning models and techniques, as disclosed by Fu.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Kim and Fu as note above; Doing so allows an advantage to perform effective learning and image analysis on three-dimensional image data, while solving the problem of a three-dimensional convolutional neural network model occupying a lot of memory since the number of parameters is large, taking a long time in learning, and having a long calculation time when using a learned model, (Kim, 4:.4-12)
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US 20210089771, hereinafter ‘Fu’) in view of Kim et al. (US 10970520, hereinafter ‘Kim’) in further view of Sibley et al. (US 20220121847, hereinafter ‘Si’)
Regarding claim 2, the rejection of claim 1 is incorporated and Fu in combination with Kim further teaches the method according to claim 1, wherein the at least one ground truth label identifies a number of leaves, fruits, flowers, or pods on the one or more simulated instances of the plant depicted in the simulated image. (in [0054] FIG. 2B is an illustration of a labelled image, according to one example embodiment. A labelled image is an accessed image after it has been processed by the control system to label features and objects. In this example, the labelled image 250 is the accessed image 210 of FIG. 2A after the control system 130 labels its pixels using a machine learning model. As illustrated, the labelled image 250 includes pixels the control system 130 identifies as representing the first type of plant 260, the second type of plant 270, [wherein the at least one ground truth label identifies a number of leaves, fruits, flowers as a plant type] and the soil 280. Pixels labelled as the first type of plant 260 are illustrated with a dotted fill, pixels labelled as the second type of plant 270 are illustrated with a hatched fill, and pixels labelled as soil 280 are illustrated with no fill… [0043] The plants 102 within each plant field, plant row, or plant field subdivision generally includes the same type of crop (e.g. same genus, same species, etc.), but can alternatively include multiple crops (e.g., a first and a second crop), both of which are to be treated. And in [0068] The control system 130 applies the model 500 to relate accessed images 210 in the convolutional layer 510 to plant and particulate identification information in the identification layer 530. The control system 130 retrieves relevance information between these elements can by applying a set of transformations (e.g., W.sub.1, W.sub.2, etc.) between the corresponding layers. Continuing with the example from FIG. 5, the convolutional layer 510 of the model 500 represents an encoded accessed image 210, and identification layer 530 of the model 500 represents plant [wherein the at least one ground truth label identifies a number of leaves, fruits, flowers as a plant information in observation space] and particulate identification information…)
Additionally, Si teaches wherein the at least one ground truth label identifies a number of leaves, fruits, flowers, or pods on the one or more simulated instances of the plant depicted in the simulated image. (in [0084] In one example, the agricultural treatment system 400 includes a camera module 450 having one or more cameras, sensing module 451 having other sensing devices, or both, for receiving image data or other sensing data of a ground, terrain, orchard, crops, trees, plants, or a combination thereof, for identifying agricultural objects [wherein the at least one ground truth label identifies a number of leaves, fruits, flowers, or pods on the one or more simulated instances of the plant depicted in the simulated image], such as flowers, fruits, fruitlets, buds, branches, plant petals and leaves, plant pistils and stigma, plant roots, or other subcomponent of a plant, and the location, position, and pose of the agricultural objects relative to a treatment unit 470, camera module 450, or both, and its position on the ground or terrain.…; And in [0134] For example, as illustrated in FIGS. 14A and 14B, an image depicting an agricultural environment including a fruit tree having one or more spurs, one or more branches and stems, one or more laterals, and one or more potential crops growing on the one or more laterals. At the moment the agricultural treatment system 400 has observed and labelled each identifiable feature of the image [wherein the at least one ground truth label identifies a number of leaves, fruits, flowers, or pods on the one or more simulated instances of the plant depicted in the simulated image], including detecting agricultural objects and labelling its growth stage, detecting and labelling landmarks including orientations of portions of the tree growing including configurations of leaves, branches, physical manmade materials that can be detected in the image, or other objects and sights of interest in the image that is not a potential crop, the agricultural treatment system 400 can detect that not all identified objects in the image include agricultural objects of the same growth stage. For example, some agricultural objects detected are labelled as buds, some as blossoms, and some as fruitlets [wherein the at least one ground truth label identifies a number of leaves, fruits, flowers, or pods on the one or more simulated instances of the plant depicted in the simulated image]. Each of these labels are of agricultural objects of interest to observe and potentially treat, but not necessarily treated the same way depending on the growth stage...)
Si, Kim and Fu are analogous art because both involve developing image processing and analysis techniques using deep learning machine models.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for developing computer-implemented image analysis techniques to develop an agricultural observation and treatment system using imaging processing algorithms, as disclosed by Si with the method of using artificial intelligence to identify and treat plants in a field use deep learning models and techniques, as collectively disclosed by Kim and Fu.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Si, Kim and Fu as note above; Doing so allows an advantage of developing automated image analysis techniques and performing agricultural service that more effectively and efficiently use labor, use tools and machinery, and reduce the amount of chemicals used on plants and cultivated land, (Si, Abstract & 0002-0003)
Regarding claim 3, the rejection of claim 1 is incorporated and Fu in combination with Kim further teaches the method according to claim 1, wherein the at least one ground truth label identifies a weight or volume yield associated with the one or more simulated instances of the plant depicted in the simulated image (in [0054] FIG. 2B is an illustration of a labelled image, according to one example embodiment. A labelled image is an accessed image after it has been processed by the control system to label features and objects. In this example, the labelled image 250 is the accessed image 210 of FIG. 2A after the control system 130 labels its pixels using a machine learning model. As illustrated, the labelled image 250 includes pixels the control system 130 identifies as representing the first type of plant 260, the second type of plant 270, and the soil 280. Pixels labelled as the first type of plant 260 are illustrated with a dotted fill, pixels labelled as the second type of plant 270 are illustrated with a hatched fill, and pixels labelled as soil 280 are illustrated with no fill. In some examples, the control system 130 may not label pixels representing soil and only label pixels representing plant matter (“plant pixels”) [the at least one ground truth label identifies … volume yield associated with the one or more simulated instances of the plant depicted in the simulated image] And in [0068] The control system 130 applies the model 500 to relate accessed images 210 in the convolutional layer 510 to plant and particulate identification information in the identification layer 530. The control system 130 retrieves relevance information between these elements can by applying a set of transformations (e.g., W.sub.1, W.sub.2, etc.) between the corresponding layers [the at least one ground truth label identifies a weight … associated with the one or more simulated instances of the plant depicted in the simulated image]. Continuing with the example from FIG. 5, the convolutional layer 510 of the model 500 represents an encoded accessed image 210, and identification layer 530 of the model 500 represents plant [the at least one ground truth label identifies … volume yield associated with the one or more simulated instances of the plant depicted in the simulated image as plant matter in observation space] and particulate identification information. )
Additionally, Si teaches: wherein the at least one ground truth label identifies a weight or volume yield associated with the one or more simulated instances of the plant depicted in the simulated image. (in in [0134] For example, as illustrated in FIGS. 14A and 14B, an image depicting an agricultural environment including a fruit tree having one or more spurs, one or more branches and stems, one or more laterals, and one or more potential crops growing on the one or more laterals. At the moment the agricultural treatment system 400 has observed and labelled each identifiable feature of the image [wherein the at least one ground truth label identifies… volume yield associated with the one or more simulated instances of the plant depicted in the simulated image], including detecting agricultural objects and labelling its growth stage, detecting and labelling landmarks including orientations of portions of the tree growing including configurations of leaves, branches, physical manmade materials that can be detected in the image, or other objects and sights of interest in the image that is not a potential crop, the agricultural treatment system 400 can detect that not all identified objects in the image include agricultural objects of the same growth stage. For example, some agricultural objects detected are labelled as buds, some as blossoms, and some as fruitlets [wherein the at least one ground truth label identifies… volume yield associated with the one or more simulated instances of the plant depicted in the simulated image]. Each of these labels are of agricultural objects of interest to observe and potentially treat, but not necessarily treated the same way depending on the growth stage...)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Si, Kim and Fu for the same reasons disclosed above in the claim 2 rejection.
Regarding claim 4, the rejection of claim 1 is incorporated and Fu in combination with Kim further teaches the method according to claim 1, wherein the plurality of simulated images comprises images having different instances of camera occlusion. (in [0068] The control system 130 applies the model 500 to relate accessed images 210 in the convolutional layer 510 to plant and particulate identification information in the identification layer 530. The control system 130 retrieves relevance information between these elements can by applying a set of transformations (e.g., W.sub.1, W.sub.2, etc.) between the corresponding layers. Continuing with the example from FIG. 5, the convolutional layer 510 of the model 500 represents an encoded accessed image 210, and identification layer 530 of the model 500 represents plant and particulate identification information. The control system 130 identifies plants and particulates in an accessed image 210 by applying the transformations W.sub.1 and W.sub.2 to the pixel values of the accessed image 210 in the space of convolutional layer 510. The weights and parameters for the transformations may indicate relationships between information contained in the accessed image and the identification of a plant and/or particulates [wherein the plurality of simulated images comprises images having different instances of camera occlusion as particulates]…; And in [0056] To demonstrate, FIG. 3A and FIG. 3B illustrate an example process of a control system 130 labelling pixels of an accessed image obtained in non-ideal operating conditions. FIG. 3A is an illustration of an accessed image, according to one example embodiment. The accessed image 310 is obtained by a detection mechanism 110 coupled to a farming machine 100 as the farming machine 100 travels through a field. The accessed image 310 includes the same area of field as the accessed image 210 and, under ideal conditions, should include the same information. However, in this example, the farming machine 100 is operating in non-ideal operating conditions (e.g., windy) and the accessed image 310 includes information representing airborne particulates 350. The airborne particulates 350 are illustrated as small cloud like shapes [wherein the plurality of simulated images comprises images having different instances of camera occlusion as particulates], however, in other accessed images, airborne particulates may not take a defined shape. The accessed image 310 also includes information representing…, and an entire plant of the second type (referring to accessed image 210) are obscured from view by the airborne particulates 350. In other examples, the airborne particulates may cause objects (e.g., plants) in an accessed image to appear fuzzy, out of focus, darker, striated, etc)
Additionally, Si teaches: wherein the plurality of simulated images comprises images having different instances of camera occlusion. (in 0062: … The image 320 can also include specific patches within captured full frame images. The patches can be identified by the agricultural system 400 detecting, classifying, identifying features, and labelling specific portions of a full image frame, including labelling agricultural objects [wherein the plurality of simulated images comprises images having different instances of camera occlusion] and specific stages of growth of agricultural objects… And in [0134] …At the moment the agricultural treatment system 400 has observed and labelled each identifiable feature of the image, including detecting agricultural objects and labelling its growth stage, detecting and labelling landmarks including orientations of portions of the tree growing including configurations of leaves, branches [wherein the plurality of simulated images comprises images having different instances of camera occlusion], physical manmade materials that can be detected in the image, or other objects and sights of interest in the image that is not a potential crop [wherein the plurality of simulated images comprises images having different instances of camera occlusion], the agricultural treatment system 400 can detect that not all identified objects in the image include agricultural objects of the same growth stage. For example, some agricultural objects detected are labelled as buds, some as blossoms, and some as fruitlets… Instead of refraining from treating one type of agricultural object at a certain growth stage while treating other agricultural objects having the desired growth stage for a particular trial, the agricultural treatment system 400 can treat multiple types of growth stages of agricultural objects growing on the same tree simultaneously by selecting and receiving a desired chemical mixture for treatment in real time.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Si, Kim and Fu for the same reasons disclosed above in the claim 2 rejection.
Regarding claim 5, the rejection of claim 4 is incorporated and Fu in combination with Kim further teaches the method according to claim 4, wherein a distribution of the different instances of camera occlusion is determined using real-life yield data. (in [0056] To demonstrate, FIG. 3A and FIG. 3B illustrate an example process of a control system 130 labelling pixels of an accessed image obtained in non-ideal operating conditions. FIG. 3A is an illustration of an accessed image, according to one example embodiment. The accessed image 310 is obtained by a detection mechanism 110 coupled to a farming machine 100 as the farming machine 100 travels through a field. The accessed image 310 includes the same area of field as the accessed image 210 and, under ideal conditions, should include the same information. However, in this example, the farming machine 100 is operating in non-ideal operating conditions (e.g., windy) and the accessed image 310 includes information representing airborne particulates 350. The airborne particulates 350 are illustrated as small cloud like shapes, however, in other accessed images, airborne particulates may not take a defined shape. The accessed image 310 also includes information representing a single plant of a first type 320, two plants a second type 330, and soil 340 in the field. Notably, portions of the plant of the first type 320, a portion of a plant of the second type 330, and an entire plant of the second type (referring to accessed image 210) are obscured from view by the airborne particulates 350. In other examples, the airborne particulates may cause objects (e.g., plants) in an accessed image to appear fuzzy, out of focus, darker, striated, etc) [wherein a distribution of the different instances of camera occlusion is determined using real-life yield data as real-life plants captured by a camera effected by airborne particulates that may impact image quality of the captured plant yield] And in [0058] A control system 130 uses labelled images to treat plants in the field. To do so, the control system 130 generates a mapped image from the labelled image, and creates a treatment map based on the mapped image. A mapped image maps a labelled image to a real-world area of a field where the image was obtained. A treatment map is a mapped image in which regions (i.e., pixel groups) in the mapped image correspond to treatment areas 122 of treatment mechanisms 120 of a farming machine 100…)
Additionally, Si teaches wherein a distribution of the different instances of camera occlusion is determined using real-life yield data. (in [0067] In one example, the simulated geographic boundary, at each position in the virtual world where there is a representation of an agricultural object, can have multiple images of the same agricultural object based on the system 100 capturing multiple images at different angles, positions, or orientations, with different poses, as the system 100 scans across the geographic boundary and captures images of the agricultural object from one trial of capturing images as another. In another example, because some of the one or more agricultural objects detected will grow into crops [wherein a distribution of the different instances of camera occlusion is determined using real-life yield data] or other stages of growth for the portion of the particular agricultural objects, each being detected, identified, and assigned a label by the system 100, a visual or other representation of each agricultural object represented in the simulated geographic boundary, can have images of the same location or images of the same agricultural object taken at a progressing period of time, … 0062: … The image 320 can also include specific patches within captured full frame images. The patches can be identified by the agricultural system 400 detecting, classifying, identifying features, and labelling specific portions of a full image frame, including labelling agricultural objects and specific stages of growth of agricultural objects [wherein a distribution of the different instances of camera occlusion is determined using real-life yield data]… And in [0134] …At the moment the agricultural treatment system 400 has observed and labelled each identifiable feature of the image, including detecting agricultural objects and labelling its growth stage, detecting and labelling landmarks including orientations of portions of the tree growing including configurations of leaves, branches, physical manmade materials that can be detected in the image, or other objects and sights of interest in the image that is not a potential crop, the agricultural treatment system 400 can detect that not all identified objects in the image include agricultural objects of the same growth stage. For example, some agricultural objects detected are labelled as buds, some as blossoms, and some as fruitlets [wherein a distribution of the different instances of camera occlusion is determined using real-life yield data]… Instead of refraining from treating one type of agricultural object at a certain growth stage while treating other agricultural objects having the desired growth stage for a particular trial, the agricultural treatment system 400 can treat multiple types of growth stages of agricultural objects growing on the same tree [wherein a distribution of the different instances of camera occlusion is determined using real-life yield data] simultaneously by selecting and receiving a desired chemical mixture for treatment in real time.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Si, Kim and Fu for the same reasons disclosed above in the claim 2 rejection.
Regarding claim 6, the rejection of claim 1 is incorporated and Fu in combination with Kim further teaches the method according to claim 1, wherein the plurality of simulated images include images simulating a plurality of camera angles. (in [0040] FIG. 1A is a side view [wherein the plurality of simulated images include images simulating a plurality of camera angles used for modeling treatment space] illustration of a system for applying a treatment fluid to plants in a field and FIG. 1B is a front view [wherein the plurality of simulated images include images simulating a plurality of camera angles] illustration of the same system, according to one example embodiment. The farming machine 100 for plant treatment includes a detection mechanism 110, a treatment mechanism 120, and a control system 130. The farming machine 100 can additionally include a mounting mechanism 140, a verification mechanism 150, a power source, digital memory, communication apparatus, or any other suitable component)
Additionally Kim teaches wherein the plurality of simulated images include images simulating a plurality of camera angles in 4:55-63: The image acquisition unit 110 prepares two-dimensional images stacked in order of a photographing angle [wherein the plurality of simulated images include images simulating a plurality of camera angles] or time of the two-dimensional images. The image acquisition unit 110 may be connected to a camera [wherein the plurality of simulated images include images simulating a plurality of camera angles], a control unit, a communication unit and the like. The three-dimensional image generation unit 120 generates a plurality of three-dimensional data using the two-dimensional images received from the image acquisition unit 110 [wherein the plurality of simulated images include images simulating a plurality of camera angles]….
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Kim and Fu for the same reasons disclosed above in the claim 1 rejection.
Additionally, Si teaches (in [0067] In one example, the simulated geographic boundary, at each position in the virtual world where there is a representation of an agricultural object, can have multiple images of the same agricultural object based on the system 100 capturing multiple images at different angles [wherein the plurality of simulated images include images simulating a plurality of camera angles], positions, or orientations, with different poses, as the system 100 scans across the geographic boundary and captures images of the agricultural object from one trial of capturing images as another…)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Si, Kim and Fu for the same reasons disclosed above in the claim 2 rejection.
Regarding claim 7, the rejection of claim 1 is incorporated and Fu in combination with Kim further teaches the method according to claim 1, wherein the plurality of simulated images include images of simulated instances of plants grown in a plurality of different configurations. in [0058] A control system 130 uses labelled images to treat plants in the field. To do so, the control system 130 generates a mapped image from the labelled image, and creates a treatment map based on the mapped image. A mapped image maps a labelled image to a real-world area of a field where the image was obtained [wherein the plurality of simulated images include images of simulated instances of plants grown in a plurality of different configurations]. A treatment map is a mapped image in which regions (i.e., pixel groups) in the mapped image correspond to treatment areas 122 of treatment mechanisms 120 of a farming machine 100…
Additionally, Si teaches: wherein the plurality of simulated images include images of simulated instances of plants grown in a plurality of different configurations. (in [0062] In this example, the vehicle 310 may have an onboard object determination and object treatment engine. The vehicle 310 may travel along a route proximate to the external agricultural objects of a geographic scene. The object determination and object treatment engine captures images of the agricultural objects via onboard cameras. For example, as the vehicle 310 passes by a particular agricultural object, the object determination and object treatment engine capture an image(s). As will be further described below, the agricultural observation and agricultural treatment system 400 may use the captured image of an agricultural object and determine which agricultural objects are to be emitted with a fluid projectile. The agricultural treatment system 400 may emit an amount of fluid along a trajectory such that the fluid comes into contact with a particular portion of a targeted agricultural object [wherein the plurality of simulated images include images of simulated instances of plants grown in a plurality of different configurations]. The diagram 300 indicates a plurality of mapped images 320, or images patches, that may have been obtained by the system 400. Each of the images 320 may have an associated geo-graphic data associated to the image, including position data, orientation [wherein the plurality of simulated images include images of simulated instances of plants grown in a plurality of different configurations] and pose estimation, relative to the geographic boundary view, relative to physical components of the agricultural treatment system 400, including image sensors, or treatment engines, or relative to other agricultural objects…)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Si, Kim and Fu for the same reasons disclosed above in the claim 2 rejection.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Fu, Kim and Si in further view of Peshov et al. (US 20200193589, hereinafter ‘Pesh”).
Regarding claim 8, the rejection of claim 1 is incorporated and Si in combination with Ja further teaches the method according to claim 1, wherein the plurality of simulated images include images of simulated instances of plants that are lodged. (in [0044] The treatment mechanism 120 of the farming machine 100 functions to apply a treatment to the identified plant 102. The treatment mechanism 120 includes a treatment area 122 to which the treatment mechanism 120 applies the treatment. The effect of the treatment can include plant necrosis, plant growth stimulation, plant portion necrosis or removal, plant portion growth stimulation, or any other suitable treatment effect. The treatment can include plant 102 dislodgement from the substrate 106 [wherein the plurality of simulated images include images of simulated instances of plants that are lodged. as determined space in simulated by the deep learning model], severing the plant (e.g., cutting), plant incineration, electrical stimulation of the plant, fertilizer or growth hormone application to the plant, watering the plant, light or other radiation application to the plant, injecting one or more working fluids into the substrate 106 adjacent the plant (e.g., within a threshold distance from the plant), or otherwise treating the plant...)
Additionally, Si teaches wherein the plurality of simulated images include images of simulated instances of plants that are lodged. (in [0073] The agricultural objects can be any number of objects and features detected in the image by an agricultural treatment system including different varieties of plants, different stages of different varieties of plants, target plants to treat including treating plants to turn into a crop or treating plats, plants for plant removal or stopping or controlling the growth rate of a plant, that are considered crops and can be treated with different treatment parameters. Other objects detected and observed by a treatment system can include landmarks in the scene including trees and portions of trees including spurs, stems, shoots, laterals, specific portions of the terrain including dirt, soil, water, mud, etc., trellises, wires [wherein the plurality of simulated images include images of simulated instances of plants that are lodged], and other farming materials used for agriculture. In this example, an agricultural object of interest can be a target plant for growing into a harvestable crop…)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Si, Kim and Fu for the same reasons disclosed above in the claim 2 rejection.
Additionally, Pesh does expressly use the term lodge. (in 0183: In the machine learning stage, in an embodiment, programmed deep (transfer) learning models based on the ImageNet pretrained convolutional neural networks model (Inception v3) are programmed to classify digital images in multiple categories. The first category, in one embodiment, is intact rows of crop, such as corn or the like. The second category is non-intact corn rows occurring due to lodging [wherein the plurality of simulated images include images of simulated instances of plants that are lodged], weeds, and/or bare soil. The output of the model is used to generate a map of the imaged areas of the field where each image is classified as intact corn, lodging [wherein the plurality of simulated images include images of simulated instances of plants that are lodged], weeds, and bare soil. While lodging or crop damage, weeds and bare soil are identified herein for purposes of providing a clear example, other embodiments may operate to classify images for other anomalies, such as burning, animal damage, heat damage and so forth, based upon one or more training datasets that have been selected and used to train the CNN to address those anomalies.)
Pesh, Si, Kim and Fu are analogous art because both involve developing image processing and machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for developing machine learning algorithms for classifying digital images and generating pot anomaly maps using machine learning algorithms as disclosed by Pesh with the method of using artificial intelligence to develop an agricultural observation and treatment system using imaging processing algorithms as collectively disclosed by Si, Kim and Fu.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Pesh, Si, Kim and Fu, as noted above, in order to improve accuracy in developing computer-implemented methods for determining anomalies in agricultural fields based on digital images, (Pesh, 0005-0007).
Claims 9-10 are rejected under 35 U.S.C. 103 as being unpatentable over Fu in view of Kim in further view of Guo et al. (US 20200126232 hereinafter ‘Guo”).
Regarding claim 9, the rejection of claim 1 is incorporated and Fu in combination with Kim further teaches the method according to claim 1, wherein the plurality of simulated images include simulated … images. (in [0068] The control system 130 applies the model 500 to relate accessed images 210 in the convolutional layer 510 to plant and particulate identification information in the identification layer 530 [wherein the plurality of simulated images include simulated … images]. The control system 130 retrieves relevance information between these elements can by applying a set of transformations (e.g., W.sub.1, W.sub.2, etc.) between the corresponding layers. Continuing with the example from FIG. 5, the convolutional layer 510 of the model 500 represents an encoded accessed image 210, and identification layer 530 of the model 500 represents plant and particulate identification information. The control system 130 identifies plants and particulates in an accessed image 210 by applying the transformations W.sub.1 and W.sub.2 to the pixel values of the accessed image 210 in the space of convolutional layer 510. The weights and parameters for the transformations may indicate relationships between information contained in the accessed image and the identification of a plant and/or particulates.)
Additionally Kim teaches wherein the plurality of simulated images include simulated … images. (in 4:55-63: The image acquisition unit 110 prepares two-dimensional images stacked in order of a photographing angle or time of the two-dimensional images. The image acquisition unit 110 may be connected to a camera [wherein the plurality of simulated images include simulated … images.], a control unit, a communication unit and the like. The three-dimensional image generation unit 120 generates a plurality of three-dimensional data [wherein the plurality of simulated images include simulated … images.] using the two-dimensional images received from the image acquisition unit 110 [wherein the plurality of simulated images include images simulating a plurality of camera angles]….
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Kim and Fu for the same reasons disclosed above in the claim 1 rejection.
Fu and Kim does not expressly teach the camera associated with the generated digital images as thermal images.
Guo does expressly teach the camera associated with the generated digital images as thermal images. (in 0084: At block 506, the system, e.g., by way of terrain classification engine 128, may determine, across pixels of a corpus of digital images that align spatially with the one or more obscured geographic units, one or more spectral-temporal data fingerprints of the one or more obscured geographic units… Each row of the 3D array may represent a particular pixel (and spatially corresponding geographic unit). Each column of the array may correspond to, for instance, a different digital image captured at a different time. Each unit in the third dimension of the 3D array may correspond to different spectral frequencies that are available in the digital images, such as red, green, blue, near infrared (“IR”), mid-IR, far-IR, thermal IR [camera associated with the generated digital images as thermal images], microwave, and/or radar. In various implementations, this 3D array structure may be used at block 306 to determine domain fingerprints, such as spectral-temporal fingerprints, of individual geographic units.)
Guo, Kim and Fu are analogous art because both involve developing image processing and machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for machine learning techniques for modeling terrain using generated digital images that correspond to different spectral frequencies as disclosed by Guo with the method of using artificial intelligence to develop an agricultural observation and treatment system using imaging processing algorithms as collectively disclosed by Kim and Fu.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Guo, Kim and Fu, as noted above, in order to implement digital images to obtain diagnosis of crop yield predictions and/or crop yields at the field- and pixel-level, (Guo, Abstract).
Regarding claim 10, the rejection of claim 1 is incorporated and Fu in combination with Kim further teaches the method according to claim 1, wherein the plurality of simulated images include simulated … images. (in [0068] The control system 130 applies the model 500 to relate accessed images 210 in the convolutional layer 510 to plant and particulate identification information in the identification layer 530 [wherein the plurality of simulated images include simulated … images]. The control system 130 retrieves relevance information between these elements can by applying a set of transformations (e.g., W.sub.1, W.sub.2, etc.) between the corresponding layers. Continuing with the example from FIG. 5, the convolutional layer 510 of the model 500 represents an encoded accessed image 210, and identification layer 530 of the model 500 represents plant and particulate identification information. The control system 130 identifies plants and particulates in an accessed image 210 by applying the transformations W.sub.1 and W.sub.2 to the pixel values of the accessed image 210 in the space of convolutional layer 510. The weights and parameters for the transformations may indicate relationships between information contained in the accessed image and the identification of a plant and/or particulates.)
Additionally Kim teaches wherein the plurality of simulated images include simulated … images. (in 4:55-63: The image acquisition unit 110 prepares two-dimensional images stacked in order of a photographing angle or time of the two-dimensional images. The image acquisition unit 110 may be connected to a camera [wherein the plurality of simulated images include simulated … images.], a control unit, a communication unit and the like. The three-dimensional image generation unit 120 generates a plurality of three-dimensional data [wherein the plurality of simulated images include simulated … images.] using the two-dimensional images received from the image acquisition unit 110 [wherein the plurality of simulated images include images simulating a plurality of camera angles]….
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Kim and Fu for the same reasons disclosed above in the claim 1 rejection.
Fu and Kim not expressly teach the camera associated with the generated digital images as near-infrared images.
Guo does expressly teach the camera associated with the generated digital images as near-infrared images. (in 0084: At block 506, the system, e.g., by way of terrain classification engine 128, may determine, across pixels of a corpus of digital images that align spatially with the one or more obscured geographic units, one or more spectral-temporal data fingerprints of the one or more obscured geographic units… Each row of the 3D array may represent a particular pixel (and spatially corresponding geographic unit). Each column of the array may correspond to, for instance, a different digital image captured at a different time. Each unit in the third dimension of the 3D array may correspond to different spectral frequencies that are available in the digital images, such as red, green, blue, near infrared (“IR”) [camera associated with the generated digital images as near-infrared images], mid-IR, far-IR, thermal IR, microwave, and/or radar. In various implementations, this 3D array structure may be used at block 306 to determine domain fingerprints, such as spectral-temporal fingerprints, of individual geographic units.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Guo, Kim and Fu for the same reasons disclosed above in claim 9.
Claims 11-17 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable Fu in view of Kim in further view of Si and in further view of Shriver et al. (US 20160224703 hereinafter ‘Shriver”).
Regarding independent claim 11, Fu teaches a computer program product comprising one or more non-transitory computer- readable storage media having program instructions collectively stored on the one or more computer-readable storage media, the program instructions executable to: (in [0133] FIG. 14 is a block diagram illustrating components of an example machine for reading and executing instructions from a machine-readable medium. Specifically, FIG. 14 shows a diagrammatic representation of control system 130 in the example form of a computer system 1400. The computer system 1400 can be used to execute instructions 1424 (e.g., program code or software) for causing the machine to perform any one or more of the methodologies (or processes) described herein. In alternative embodiments, the machine operates as a standalone device or a connected (e.g., networked) device that connects to other machines. In a networked deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.)
to: generate a plurality of three-dimensional simulated instances of a plant; generate training data comprising a plurality of simulated images, each simulated image being a two-dimensional projection of one or more of the simulated instances of the plant in a plurality of plants, (in As depicted in Fig 5 and in [0068] The control system 130 applies the model 500 to relate accessed images 210 in the convolutional layer 510 to plant and particulate identification information in the identification layer 530 [generate a plurality of three-dimensional simulated instances of a plant; generate training data comprising a plurality of simulated image]. The control system 130 retrieves relevance information between these elements can by applying a set of transformations (e.g., W.sub.1, W.sub.2, etc.) between the corresponding layers. Continuing with the example from FIG. 5, the convolutional layer 510 of the model 500 represents an encoded accessed image 210, and identification layer 530 of the model 500 represents plant and particulate identification information. The control system 130 identifies plants and particulates in an accessed image 210 by applying the transformations W.sub.1 and W.sub.2 to the pixel values of the accessed image 210 in the space of convolutional layer 510. The weights and parameters for the transformations may indicate relationships between information contained in the accessed image and the identification of a plant and/or particulates [generate a plurality of three-dimensional simulated instances of a plant; generate training data comprising a plurality of simulated image]. And in [0054] FIG. 2B is an illustration of a labelled image, according to one example embodiment. A labelled image is an accessed image [generate a plurality of three-dimensional simulated instances of a plant; generate training data comprising a plurality of simulated images, each simulated image being a two-dimensional projection of one or more of the simulated instances of the plant in a plurality of plants] after it has been processed by the control system to label features and objects. In this example, the labelled image 250 is the accessed image 210 of FIG. 2A after the control system 130 labels its pixels using a machine learning model. As illustrated, the labelled image 250 includes pixels the control system 130 identifies as representing the first type of plant 260, the second type of plant 270, and the soil 280 [….each simulated image being a two-dimensional projection of one or more of the simulated instances of the plant in a plurality of plants]. Pixels labelled as the first type of plant 260 are illustrated with a dotted fill, pixels labelled as the second type of plant 270 are illustrated with a hatched fill, and pixels labelled as soil 280 are illustrated with no fill. In some examples, the control system 130 may not label pixels representing soil and only label pixels representing plant matter (“plant pixels”). And in [0112] The array of particulate images may be used to train a plant identification model (e.g., model 500) to identify plants in non-ideal operating conditions. Arrays of particulate images [.. each simulated image being a two-dimensional projection of one or more of the simulated instances of the plant in a plurality of plants]] are informative because each particulate image corresponds to a previously labelled image…)
and train a regression model to make an agricultural prediction using the training data. ([0127] [0126] As described herein, a control system 130 may employ a plant identification model configured to determine a particulate level in an accessed image. Throughout the embodiments, the plant identification models are trained using alpha-blended augmented images as described above [and train a regression model to make an agricultural prediction using the training data.]… In various embodiments, the plant identification model may employ one or more approaches to determine a particulate level in the accessed image. Broadly, these methods may be grouped into, for example, two groups: (i) image level identification, and (ii) pixel level identification. Other groups are also possible. Image level identification determines a particulate level for an accessed image based on an aggregate classification of pixels in the image. Pixel level identification determines a particulate level based on a classification of individual pixels in an accessed image and subsequent analysis of the classified pixels. Both types of determination may employ classification and/or regression analysis techniques [and train a regression model to make an agricultural prediction using the training data as part of the trained identification models used to analyze and classify pixels].)
Additionally, Si teaches: generate a plurality of three-dimensional simulated instances of a plant; (in 0043] Additionally, the systems, robots, computer software and systems, applications using computer vision and automation, or a combination thereof, … Generally, the computer system provides computer vision functionality using stereoscopic digital cameras and performs object detection and classification and apply a chemical treatment to target objects that are potential crops via an integrated onboard observation and treatment system. The system utilizes one or more image sensors, including stereoscopic cameras to obtain digital imagery, including 3D imagery of an agricultural scene [generate a plurality of three-dimensional simulated instances of a plant] such as a tree in an orchard or a row of plants on a farm while the system moves along a path near the crops.…)
generate training data comprising a plurality of simulated images, (in 0002: Implementations disclosed herein are directed towards automatically generating synthetic training images that simulate plants in an area [generate training data comprising a plurality of simulated images], and automatically and uniquely labeling constituent parts of individual plants in the synthetic training image…; And in 0003: …In some implementations, one or more generated aspects of each synthetic plant [generate training data comprising a plurality of simulated images], such as each generated leaf in a synthetic plant, can be annotated, e.g., collectively and or on an individual basis …)
each simulated image being a two-dimensional projection of one or more of the simulated instances of the plant; (in [0064] In one example, the agricultural treatment system 400, or similar system 100, can build one or more graphical visualizations and constructing an animation of a virtual geographic boundary based on each individual images captures [each simulated image being a two-dimensional projection of one or more of the simulated instances of the plant], for example a simulated virtual farm or orchard having each agricultural object detected in space based on the images and location data of each object detected in a real-world geographic boundary, with each agricultural object, or other objects in the geographic boundary, animated and imposed in the simulated virtual geographic boundary.; And in [0066] In one example, one or more agricultural object detected in the real-world will change characteristics such that the system 100 can detect a new feature of the agricultural object and assign a label or identifier to the agricultural object that had a different label or identifier previously assigned to the same agricultural object having the same or similar position detected in the geographic boundary. This is due to a portion of a potential crop growing on a plant [each simulated image being a two-dimensional projection of one or more of the simulated instances of the plant;], for example a lateral, changing characteristics due to the growth stage of the plant…)
and train a … model to make an agricultural prediction using the training data. (in [0079] For example, the communications module 426 can communicate signals, through a network 1520 such as a wired network, wireless network, Bluetooth network, wireless network under 5G wireless standards technology, radio, cellular, etc. to edge and cloud computing devices including a mobile device 1540, a device for remote computing of data including remote computing 1530, databases storing image and other sensor data of crops such as crop plot repository 1570, or other databases storing information related to agricultural objects, scenes, environments, images and videos related to agricultural objects and terrain [and train a … model to make an agricultural prediction using the training data], training data for machine learning algorithms [and train a … model to make an agricultural prediction using the training data s], raw data captured by image capture devices or other sensing devices, processed data such as a repository of indexed images of agricultural objects […using the training data]…)
Si, Kim and Fu are analogous art because both involve developing image processing and analysis techniques using deep learning machine models.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for developing computer-implemented image analysis techniques to develop an agricultural observation and treatment system using imaging processing algorithms, as disclosed by Si with the method of using artificial intelligence to identify and treat plants in a field use deep learning models and techniques, as collectively disclosed by Kim and Fu.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Si, Kim and Fu as note above; Doing so allows an advantage of developing automated image analysis techniques and performing agricultural service that more effectively and efficiently use labor, use tools and machinery, and reduce the amount of chemicals used on plants and cultivated land, (Si, Abstract & 0002-0003)
The remaining limitations are similar with claim 1 limitations and are rejected under the same rationale.
While Si teaches the use of machine learning algorithms to make agriculture predictions as noted above. But does not expressly teach the use of a regression model as in the limitation …a regression model to make an agricultural prediction using … training data.
Shriver expressly teach the use of a regression model as in the limitation and train a regression model to make an agricultural prediction using the training data. In 0027: Each module of the plurality can utilize one or more of: supervised learning (e.g., using logistic regression, using back propagation neural networks, using random forests, decision trees, etc.), unsupervised learning (e.g., using an Apriori algorithm, using K-means clustering), semi-supervised learning, reinforcement learning (e.g., using a Q-learning algorithm, using temporal difference learning), and any other suitable learning style. Each module of the plurality can implement any one or more of: a regression algorithm […a regression model to make an agricultural prediction using … training data] (e.g., ordinary least squares, logistic regression, stepwise regression, multivariate adaptive regression splines, locally estimated scatterplot smoothing, etc.), an instance-based method (e.g., k-nearest neighbor, learning vector quantization, self-organizing map, etc.), a regularization method (e.g., ridge regression, least absolute shrinkage and selection operator, elastic net, etc.), a decision tree learning method (e.g., classification and regression tree, iterative dichotomiser 3, C.sub.4.5, chi-squared automatic interaction detection, decision stump, random forest, multivariate adaptive regression splines, gradient boosting machines, etc.), a Bayesian method (e.g., naive Bayes, averaged one-dependence estimators, Bayesian belief network, etc.), a kernel method (e.g., a support vector machine, a radial basis function, a linear discriminate analysis, etc.), a clustering method (e.g., k-means clustering, expectation maximization, etc.), an associated rule learning algorithm (e.g., an Apriori algorithm, an Eclat algorithm, etc.), an artificial neural network model (e.g., a Perceptron method, a back-propagation method, a Hopfield network method, a self-organizing map method, a learning vector quantization method, etc.), a deep learning algorithm (e.g., a restricted Boltzmann machine, a deep belief network method, a convolution network method, a stacked auto-encoder method, etc.), a dimensionality reduction method (e.g., principal component analysis, partial lest squares regression, Sammon mapping, multidimensional scaling, projection pursuit, etc.), an ensemble method (e.g., boosting, boostrapped aggregation, AdaBoost, stacked generalization, gradient boosting machine method, random forest method, etc.), and any suitable form of machine learning algorithm. And in 0057: he transition date can subsequently be used (e.g., by the user or automatically) to plan future treatments, schedule future operations, forecast harvest parameters (e.g., yield), or used in any other suitable manner.
Shriver, Si, Kim and Fu are analogous art because both involve developing image processing and machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for using machine learning retrieving and processing geographic and environment data of crops to be analyzed by a computer system as disclosed by Shriver with the method of using artificial intelligence to develop an agricultural observation and treatment system using imaging processing algorithms as collectively disclosed by Si, Kim and Fu.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Shriver, Si, Kim and Fu, as noted above, in order to generated methods that estimate crop growth stages based on up-to-date parameter values exhibited by the crops themselves using machine learning models, (Shriver, 0021-0023).
Regarding claims 12-17 the rejection of claim 11 is incorporated. The limitations of claims 12-17 are similar to claim 2-7 limitations and are thus rejected under the same rationale.
Regarding independent claim 19, Fu teaches system comprising: a processor, a computer-readable memory, one or more computer-readable storage media, and program instructions collectively stored on the one or more computer-readable storage media, the program instructions executable to: (in [0042] Some embodiments are implemented by a computer system. A computer system may include a processor, a memory, and a non-transitory computer-readable medium. The memory and non-transitory medium may store instructions for performing methods and steps described herein...)
Regarding the remaining limitations of claim 19, the limitations are similar to claim 11 limitations and are thus rejected under the same rationale.
Regarding claim 20, the claim is similar to claim 2 limitations and such rejected under the same rationale.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Fu, Kim, Si in further view of Shriver and Peshov et al. (US 20200193589, hereinafter ‘Pesh”).
Regarding claim 18, the limitation is similar to claim 8, and thus rejected under the same rationale.
Additionally, Pesh, Shriver, Si, Kim and Fu are analogous art because both involve developing image processing and machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for developing machine learning algorithms for classifying digital images and generating pot anomaly maps using machine learning algorithms as disclosed by Pesh with the method of using artificial intelligence to develop an agricultural observation and treatment system using imaging processing algorithms as collectively disclosed by Shriver, Si, Kim and Fu.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Pesh, Shriver, Si, Kim and Fu as noted above, in order to improve accuracy in developing computer-implemented methods for determining anomalies in agricultural fields based on digital images, (Pesh, 0005-0007).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zhang et al. (NPL: Three-dimensional convolutional neural network model for tree species classification using airborne hyperspectral images, hereinafter ‘Zhang’): s that convolutional neural network render 3-dimensional images, as feature maps for processing image attributes and making predictions. (As depicted in Fig. 3)
PNG
media_image2.png
740
1398
media_image2.png
Greyscale
Sec. 3.2: In this paper, a spatial-spectral joint CNN classification framework is constructed to integrate the spectral and spatial information in the hyperspectral images…Through the research on 2D-CNN we can find that small receptive domains with deeper architectures of 3 × 3 convolution kernels generally yield better results. Compared with the image level classification model, the input data space size used in the remote sensing image classification model is relatively small, and the space size of the feature map is further reduced after the convolution operation, so the spatially smaller convolution kernel is generally used to avoid excessive loss of input information. Tran et al. (2015) demonstrated that a small 3 × 3 ×3kernel is the best choice for 3D CNN in spatial-temporal features learning. Li et al. (2017) also used a spatially 3 × 3 3D con volution kernel to study 3D-CNN for common hyperspectral data sets and achieved good results. Inspired by these researches, in this paper, the space size of the 3D convolution kernel is fixed to 3 × 3, on the basis of which the spectral depth of the kernel is set to 7.
Gu et al. (NPL: Path Tracking Control of Field Information-Collecting Robot Based on Improved Convolutional Neural Network Algorithm) teaches in Sec. 2.2: The convolutional neural network is a deep learning model that specializes in processing grid-like data. It consists of the layers for input, convolution, pooling, full connection, and output. The convolutional layer is composed of multiple feature surfaces, each feature surface made up of multiple neurons, each of which is connected to a local area of the previous feature surface through a convolution kernel. The convolutional neural network takes the original image as an input and performs a convolution operation on the convolution layer with the feature map of the previous layer and a The convolutional neural network is a deep learning model that specializes in processing grid like data. It consists of the layers for input, convolution, pooling, full connection, and output. The convolutional layer is composed of multiple feature surfaces, each feature surface made up of multiple neurons, each of which is connected to a local area of the previous feature surface through a convolution kernel. The convolutional neural network takes the original image as an input and performs a convolution operation on the convolution layer with the feature map of the previous layer and a convolution kernel. The convolution result is mapped by the activation function to form the feature map of the next layer. The pooling layer mainly reduces the dimension of the feature map between consecutive convolutional layers, maintains the translation invariance of the data to a certain extent, and decreases parameters and calculations in the network. The fully connected layer is located at the end of the structure of the convolutional neural network model, with each neuron in it fully connected to all the neurons in the previous layer. The layer can integrate the class-specific local convolution kernel. The convolution result is mapped by the activation function to form the feature map of the next layer. The pooling layer mainly reduces the dimension of the feature map between consecutive convolutional layers, maintains the translation invariance of the data to a certain extent, and decreases parameters and calculations in the network. The fully connected layer is located at the end of the structure of the convolutional neural network model, with each neuron in it fully connected to all the neurons in the previous layer. The layer can integrate the class-specific local information in the convolutional or the pooling layer. As a classifier output image, the number of neurons in the fully connected network structure is the same as the output of the convolutional layer.
Ha et al. (NPL: Machine Learning-Enabled Smart Sensor Systems): teaches well-known ML algorithms used to create smart models in an ML-enabled smart sensor system for practical sensing applications. Under this section, the ML algorithms are divided into two categories: classic non-neural network (non-NN) algorithms (e.g., Principal Component Analysis [PCA], Support Vector Machine [SVM], Random Forest [RF]) and the state-of-the-art neural network (NN) algorithms (e.g., Backpropagation Neural Network [BP-NN], Recurrent Neural Network [RNN], Convolution Neural Network [CNN]). Furthermore, the general overview process of training smart model for each ML algorithm is discussed with application examples. Subsequent sections summarize the novelty of ML-enabled smart sensor systems in practical sensing applications under two major categories that are based on their sensing principals… As shown in Section 1, the foundation of a smart sensor system is the ML-based smart models that are developed to solve classification or regression problems for specific sensing applications…
Nagasubramanian et al. (Plant disease identification using explainable 3D deep learning on hyperspectral images): teaches “deep convolutional neural networks (DCNN) have been successfully used in diverse applications such as object recognition, speech recognition, document read ing and sentiment analysis [29–32]. The standard convo lutional filter is tailored to extract spatial features (and correlations) in 2D and is naturally suited to RGB images. In contrast, hyperspectral images can be considered as a stack of 2D images, exhibiting correlations both in space as well as in the spectral directions. To extend DCNN’s applicability to hyperspectral images, a 3D analogue of the convolutional filter was proposed and such 3D-CNN models have been used in classification of hyperspectral images for some interesting engineering applications [33–35].” And the use of 3-D image rendering in processing image data with convolutional neural networks.
PNG
media_image3.png
528
1334
media_image3.png
Greyscale
Gillberg et al. (US 20210319363): teaches generating a synthetic data set comprising of images and annotations. In other example the third set of parameters is used to control the point of view and the ambient lighting conditions when creating images of the entities of the object and the entities of the background in the task environment model. Outputted images can be used as a synthetic dataset. The third parameter (first and/or second as well) can be used as annotations for the created images (i.e. the annotated synthetic dataset).
Regan et al. (US 20190200535): teaches digitized images as synthetic images that can be processed and displayed to allow a user to prescribe a box around each leaf of the plant (e.g., labels 122), prescribe a box around each fruit of the plant (e.g., labels 124), and enter a value for the measured height of the plant (e.g., label 126). Manually adding ground truth labels based on direct observation of the raw data or processed raw data (e.g., viewing image(s) of the plant collected as part of raw data) and/or direct observation or measurements of the actual plant.
Grant et al. (US 20220391752): teaches a method and system for automatically generating labeled synthetic images that are usable as training data for training machine learning models to make an agricultural prediction based on digital images. A method includes: generating a plurality of simulated images, each simulated image depicting one or more simulated instances of a plant; for each of the plurality of simulated images, labeling the simulated image with at least one ground truth label that identifies an attribute of the one or more simulated instances of the plant depicted in the simulated image, the attribute describing both a visible portion and an occluded portion of the one or more simulated instances of the plant depicted in the simulated image; and training a machine learning model to make an agricultural prediction using the labeled plurality of simulated image.
Riley et al. (US 20190108631): teaches in 0025: Modules and/or other components of the system can include and/or otherwise apply models associated with one or more of: classification, decision-trees, regression, neural networks (e.g., convolutional neural networks), heuristics, equations (e.g., weighted equations, etc.), selection (e.g., from a library), instance-based methods (e.g., nearest neighbor), regularization methods (e.g., ridge regression), Bayesian methods, kernel methods, supervised learning, semi-supervised learning, unsupervised learning, reinforcement learning, and/or any other approach.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OLUWATOSIN ALABI whose telephone number is (571)272-0516. The examiner can normally be reached Monday-Friday, 8:00am-5:00pm EST..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached on (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OLUWATOSIN ALABI/ Primary Examiner, Art Unit 2129
Prosecution Insights