DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to the claim(s) have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. The Examiner points out that previous prior art Khait (Khait et al., US 2022/0092705 A1) teaches wherein the computing device and/or image and/or treatment arrangement includes and/or is in communication with one or more physical user interfaces ([0064]) that includes a mechanism for user interaction such as to view data ([0064]). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that Khait teaches sending the results of the first class and the second class, including the first portion of the first image for further review, since the first class (first specific weed species probability percentage using the detector) ([0146]), second class (second specific weed species probability percentage using the classifier) ([0146]), and portion of the image (bounding box part of the image) ([0146]) are all data and that can be viewed by the user in the physical user interface in Khait ([0064]). However, to further advance prosecution, and because the claim language has been changed from the previous iteration, prior art Huval, US 2018/0373980 A1 (Huval) has been newly added to assist in teaching the newly added claim amendments and claim structure. Prior art Polzounov et al., US 2019/0362146 A1, used within the previous office action, is considered pertinent to applicant's disclosure; however, is no longer used within the current rejection.
The 35 USC 101 rejection made to claim 20 has been withdrawn due to Applicant’s amendments.
Claims 1-15 and 18-22 are pending; claims 16 and 17 have been canceled; claims 1, 18, and 20 have been newly amended; and claims 21 and 22 have been newly added.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Independent claims 1 and 20 (and thus dependent claims 2-15, 18, 19, 21, and 22) are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1 and 20 state accessing a first image from the sensor input; applying a first image processing scheme on the first image, wherein the first image processing scheme implements a first machine learning (ML) algorithm configured to detect plant objects or target objects on a portion of the first image; detecting a first target object in a first portion of the first image, the detection comprising identifying at least a first class associated with the first target object. It is unclear, the way the claim is written, if the “detecting a first target object in a first portion of the first image, the detection comprising identifying at least a first class associated with the first target object”, as claimed, is based on the applying of the “first image processing scheme” since the claim doesn’t say (something to the effect of) “detecting a first target object in a first portion of the first image, the detection comprising identifying at least a first class associated with the first target object using the first image processing scheme (or the first ML algorithm)”. Appropriate correction and/or reasoning/remarks is required. This also goes with the part of the claims that state “accessing the first portion of the first image for further processing; applying a second image processing scheme on the first portion of the first image, wherein the second image processing scheme implements a second ML algorithm configured to classify a received image, the received image including the first portion of the first image; identifying at least a second class associated with the first portion of the first image”. Again, it is unclear how the second class is identified (is it using the second ML or something else). An appropriate amendment would be something to the effect of “identifying at least a second class associated with the first portion of the first image using the second image processing scheme (or the second ML algorithm)”. Appropriate correction and/or reasoning/remarks is required. The proposed amendments to clarify the claim language have support in Applicant’s PGPUB in paragraph [0087] and paragraph [0090] in the Specification as filed.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 5-15, 18, and 20-22 are rejected under 35 U.S.C. 103 as being unpatentable over Khait et al., US 2022/0092705 A1 (Khait) and further in view of Huval, US 2018/0373980 A1 (Huval).
Regarding claim 1, Khait teaches a computer-implemented method (computing device 104) (Fig. 1; [0051]) of sensor input processing (processing from each imaging and treatment arrangement 108 that includes one or more sensors) (Fig. 1; [0055]), implemented by an agricultural system (implemented on a system 100, with agricultural machine 110) (Fig. 1; [0053]) comprising a processor (processor(s) 102) (Fig. 1; [0052]) and one or more sensors (imaging and treatment arrangement 108 that includes one or more sensors) (Fig. 1; [0055]), comprising:
receiving a sensor input from the one or more sensors (wherein computing device 104 receives the image(s) from image sensor(s) 112) (Fig. 1; [0055-0056]);
accessing a first image from the sensor input (an input image depicting a portion of the agricultural field is obtained) ([0141]);
applying a first image processing scheme on the first image, wherein the first image processing scheme implements a first machine learning (ML) algorithm (wherein the input image is fed into an object detector component (e.g. a neural network) that generates bounding boxes) ([0143]) configured to detect plant objects or target objects on a portion of the first image (configured to detect weeds on a portion (bounding box within the first image) ([0143]);
detecting a first target object in a first portion of the first image (detecting an object, such as a potential weed, within the bounding box, of the image, with a probability of the weed being there) ([0143-0144]), the detection comprising identifying at least a first class associated with the first target object (wherein the first class can represent a suspicious object proposal; i.e., not a clear if an object is depicted therein or not) ([0144-0145]);
accessing the first portion of the first image for further processing (wherein when the class is a suspicious object proposal, it’s fed into a classifier component (e.g., another neural network) for further processing) ([0145]);
applying a second image processing scheme on the first portion of the first image, wherein the second image processing scheme implements a second ML algorithm (wherein the bounding box is fed into a classifier component (e.g. a neural network) that generates another probability for the bounding box portion) ([0145]) configured to classify a received image (wherein the portion of the bounding box is classified as a species of weed) ([0145-0146]), the received image including the first portion of the first image (wherein the received input includes the bounding box portion of the image) ([0145]);
identifying at least a second class associated with the first portion of the first image (wherein the classifier (second neural network) classifies the bounding box as a weed species) ([0145-0146]);
comparing results of the first class with the second class (wherein a result for the first class of a suspicious object proposal is compared with the classifier result; which if the classifier has a high probability the bounding box is reclassified) ([0146]); and
performing a subsequent action based on the comparing (performing further processing on the bounding box to determine a final probability that is above the threshold for a specific weed species, and then when a weed species is identified a herbicide is selected for spraying) ([0146-0153]);
wherein the subsequent action is one of two actions (wherein the subsequent action is confirming and spraying a herbicide or re-classifying the specific weed species) ([0146], [0152-0154], and [0160]) comprising:
when results of the first class and the second class are in agreement (when the specific weed species from the detector is a probability of a specific percentage (such as 30%) and the classifier identified the same specific weed species with a specific probability percentage (such as 92%); i.e. the detector and classifier are in agreement that the bounding box is confirmed to include the specific weed species) ([0146]), treating the first target object (when the weeds of a specific species are identified a herbicide is selected for spraying) ([0152-0154] and [0160]); and
when results of the first class and the second class are in disagreement (when the first result is a weed species of a specific probability percentage (such as 50%) and the second result is a second/different weed species with a specific probability percentage (such as 90%), then re-classifying to the second weed species) ([0146]).
Although Khait does not explicitly state a “first class” and “second class” it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that the object detector component ([0143]) can output that a bounding box represents “suspicious object proposal” ([0143-0145]) or a specific “weed species” ([0143-0144]) and that the classifier component ([0145]) outputs a “weed species”, which can be different then the first weed species, that these are each types of classes (suspicious object, first weed species, second weed species).
Khait teaches wherein the computing device and/or image and/or treatment arrangement includes and/or is in communication with one or more physical user interfaces ([0064]) that includes a mechanism for user interaction such as to view data ([0064]). However, Khait does not explicitly teach “sending the results of the first class and the second class, including the first portion of the first image for further review”.
Huval teaches a method for training and refining an artificial intelligence, such as a neural network (machine learning techniques) (Abstract and [0023]); and wherein when results of the first class and the second class are in disagreement (wherein a first confidence score that the object is a first object type and a second confidence score that the object is a second object type are the results; wherein the confidence scores being high and similar) ([0034]), sending the results of the first class and the second class, including the first portion of the first image for further review (wherein the remote computer system can serve both the first automated label (first object type/class) and the second automated label (second object type/class) with the optical image to a human annotator for confirmation of one of these two labels) ([0034] and [0053]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Khait to include that if the detector and classifier both had high and similar probability percentages, but to two different weed species, that the classes and image should be sent to the user interface of Khait for review as in Huval since it allows for selecting the more accurate of the two labels (Huval; [0080]) while also handling conflicting labels collected from human annotators and generated by the neural network over time in order to develop a larger, more accurate training set that may yield a more effective, accurate neural network (Huval; [0014]).
Regarding claim 5, Khait teaches wherein detecting a first target object in the first portion of the first image comprises generating a bounding box associated with the first portion of the first image (wherein the input image is fed into the object detector (e.g., neural network), and detecting an object, such as a potential weed, within a generated bounding box, with a probability of the weed being there) ([0143-0144]).
Regarding claim 6, Khait teaches wherein the first portion of the first image is an image patch (wherein the first portion of the first image is an image patch in the form of a bounding box) ([0143-0145]).
Regarding claim 7, Khait teaches wherein identifying at least a second class comprises classifying the image patch (wherein the second neural network classifies the bounding box image patch) ([0145-0146]).
Regrading claim 8, Khait teaches wherein the first class and the second class can be classifications from a plurality classifications including a crop class, a weed class, a background class, or a soil class (wherein the classifications can be of different weed species) ([0019] and [0140]).
Regarding claim 9, Khait teaches wherein the first ML algorithm (object detector component; e.g. neural network) ([0143]) is configured to assign more than one class of a plurality of classifications to one or more detected objects, including plant objects and target objects (wherein the object detector component can classify the object, such as a weed, into one of a plurality of classifications; i.e. from a plurality of weed species and/or a suspicious object) ([0143-0144] and [0146]).
Regarding claim 10, Khait teaches wherein each class, including the first class and second class, has a confidence level associated with the class (wherein the output from the first and second neural networks includes a probability of the class) ([0144-0146]).
Regarding claim 11, Khait teaches wherein the second ML algorithm (classifier component; e.g. another neural network) ([0145]) is configured to assign more than one class of a plurality of classifications to one or more detected objects, including plant objects and target objects (wherein the classifier can classify the object, such as a weed, into one of a plurality of classifications; i.e. from a plurality of weed species) ([0145-0146]).
Regarding claim 12, Khait teaches wherein each class, including the first class and second class, has a confidence level associated with the class (wherein the output from the first and second neural networks includes a probability of the class) ([0144-0146]).
Regarding claim 13, Khait teaches wherein each of the first class and second class has a confidence level (wherein the output from the first and second neural networks includes a probability of the class) ([0144-0146]).
Regarding claim 14, Khait teaches wherein comparing results of the first class with the second class (results from the first and second neural networks) ([0143-0145]) comprises analyzing and comparing each confidence level of the first class and the second class with each other (comparing the probability score of the second classification with the probability score of the first classification) ([0143-0146]) .
Regarding claim 15, Khait teaches wherein performing a subsequent action includes performing a treatment action on a real-world object associated with the detected first target object (wherein based on the final probability a herbicide is selected for a real-world weed) ([0153]) (and applying a treatment to the weeds, such as a herbicide) ([0020], [0060], and [0161]).
Regarding claim 18, Huval teaches wherein a selection by a user of a user interface (selection by a human annotator using a local computer system) ([0012] and [0034]) to verify a correct classification of the detected first target object (wherein the human annotator can confirm the correct label/classification for the detected object) ([0034]) is indexed for training of a first ML model associated with the first ML algorithm (retraining the neural network using the optical image based on confirmation received from the human annotator) ([0008] and [0010]).
Regarding claim 20, see the rejection made to claim 1, as well as Khait for a computer-readable medium (computer readable medium) ([0044]) having code stored thereon (wherein computer readable program instructions are stored on the computer readable storage medium) ([0045]), the code (computer readable program instructions) ([0046]), upon execution by a processor (executed by a processor) ([0043] and [0048]), causing the processor to implement a method (the processor to execute the instructions to carry out aspects of the present invention) ([0043] and [0048]) of sensor input processing (processing from each imaging and treatment arrangement 108 that includes one or more sensors) (Fig. 1; [0055]), for they teach all the limitations within this claim.
Regarding claim 21, Huval teaches wherein, the sending includes sending to a user interface (sending to a human annotator using a local computer system) ([0012]), and the method further includes: receiving a user input comprising a correct classification of the first target object, or receiving a user input comprising a corrected bounding box for the first target object (receiving an input from the human annotator for selecting the correct classification/type label for the object) ([0034] and [0053]).
Regarding claim 22, Huval teaches wherein, the sending includes sending to an ML model (neural network) ([0010] and [0014]) that is trained to resolve conflicts (conflict resolution; by sending the data to a neural network to be retrained to resolve conflicts) ([0010] and [0014]).
Claim(s) 2-4 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Khait et al., US 2022/0092705 A1 (Khait), Huval, US 2018/0373980 A1 (Huval), and further in view of Redden et al., US 2018/0330166 A1 (Redden).
Regarding claim 2, Khait teaches the detected first target object in the first portion of the first image (detecting an object, such as a potential weed, within the bounding box, of the image, with a probability of the weed being there) ([0143-0144]) is a smaller portion of the first portion (wherein the bounding box is a smaller portion; with the weed being even smaller inside the bounding box) ([0143]), the smaller portion represented by a bounding box associated with pixels of the first image (the bounding box associated with pixels in the first image that indicate a likelihood of a weed and/or suspicious object) ([0143-0145]). Huval teaches a method for training and refining an artificial intelligence, such as a neural network (machine learning techniques) (Abstract and [0023]).
However, neither explicitly teaches “wherein the first portion of the first image is a tile”.
Redden teaches a plant treatment platform that uses a plant detection model to detect plants as the plant treatment platform travels through a field (Abstract); and wherein the first portion of the first image is a tile (wherein the image data can be split into multiple images with respect to the physical region of the field captured within those images; a technique referred to as “tiling”) ([0051-0052]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of prior arts to include tiling since it helps ensure consistency between the images used to train the model and the images captured by the camera for use in performing the task at hand (Redden; [0052])
Regarding claim 3, Khait teaches wherein the second ML algorithm (second neural network) ([0145]) classifies the smaller portion of the first portion of the first image (classifies the smaller weed portion within the bounding box) ([0145-0146]).
Regarding claim 4, Khait teaches wherein the second ML algorithm (second neural network) ([0145]) classifies the smaller portion represented by the bounding box (classifies the smaller weed portion within the bounding box based on a probability) ([0145-0146]).
Regarding claim 19, Redden teaches wherein the first class is a weed species (wherein model tasks from the plant detection model can be trained to perform one or more tasks using one or more submodels; such as to identify bounding boxes that specify where plants/crops/weeds/species are physically located) ([0053-0055]) and the second class is a soil pattern (another task is to detect with a spray box detection model similar to the plant detection model, bounding boxes regarding soil and/or plants) ([0056]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Previously used prior art Polzounov et al., US 2019/0362146 A1: teaches a system for customized application of herbicides (Abstract); wherein a selection by a user of the user interface (wherein the computer system can include a graphics or liquid crystal display with input devices) ([0126]) to verify a correct classification of the detected first target object (sending the labeled image to users for verification) ([0077]) is indexed for training of a first ML model associated with the first ML algorithm (wherein users verify the labels for using as pre-identification for training model 800; a plant identification model) (Fig. 8; [0072-0073] and [0077]); wherein the model can be configured to identify a category of a plant (e.g. weed or a crop); and the identification layer of the model can identify soil (Fig. 8; [0085]); and wherein the identification layer is configured to identify weeds, cotton, and soil ([0096]).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL J VANCHY JR whose telephone number is (571)270-1193. The examiner can normally be reached Monday - Friday 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL J VANCHY JR/Primary Examiner, Art Unit 2666 Michael.Vanchy@uspto.gov
Prosecution Insights