DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see page 6 paragraph 2, filed 3/20/2026, with respect to claims 9-14 and 16-20 have been fully considered and are persuasive. The objection of claims 9-14 and 16-20 has been withdrawn.
Applicant’s arguments, see page 6 paragraph 3 and 4, filed 3/20/2026, with respect to claims 2, 9, and 16 have been fully considered and are persuasive. The 35 U.S.C 112(b) rejection of claims 2, 9, and 16 has been withdrawn.
Applicant’s arguments, see page 6 paragraph 5 and 6, filed 3/20/2026, with respect to claim 15 have been fully considered and are persuasive. The 35 U.S.C 101 rejection of claim 15 has been withdrawn.
Applicant's arguments, see page 6 paragraph 5 and 6, filed 3/20/2026 have been fully considered but they are not persuasive. The applicant only amended claim 15 to recite "non-transitory computer-readable medium", claims 16-20 were not amended thus claims 16-20 are objected to because it is not clear if the readable medium in claims 16-20 are referring to the non-transitory medium of claim 15.
Applicant's arguments, see page 7 and 8 in regards of claim 1 and 8, filed 3/20/2026 have been fully considered but they are not persuasive. Solovyev teaches combining outputs from multiple object-detection models. Specifically, Solovyev discloses that “the proposed WBF method uses confidence scores of all proposed bounding boxes to construct the average boxes” and further teaches “combining predictions from different models.” The outputs include predicted object locations/bounding boxes, class labels, and confidence scores. Solovyev further compares boxes based on overlap metrics (IoU) to determine correspondence between detections generated by different models. Thus, Solovyev teaches generating combined or super-imposed labeled outputs from multiple ML algorithms and comparing those outputs based on overlap and confidence.
Horowitz teaches displaying sensed image data and machine-learning output to users within a graphical user interface for review and interaction. Horowitz discloses that sensed data, including image data, is captured, processed using machine learning techniques to identify target objects, and used within a system in which operators review object-recognition results. Horowitz further teaches that object trajectories are determined from “a series of images” and that machine-learning outputs are used in facility operations and model evaluation.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to display Solovyev’s combined labeled detections on Horowitz’s graphical user interface because doing so merely presents known object-detection outputs in a known user-interface environment for the predictable purpose of user review, comparison, validation, and model improvement. Displaying multiple model outputs overlaid on a common sensor image would allow a user to visually compare where the models agree or differ with respect to detected object location, labels, or confidence.
Applicant’s argument that Solovyev does not expressly disclose a user interface is not persuasive because the rejection does not rely on Solovyev alone for that feature. Horowitz is relied upon for the user-interface/display limitation. Solovyev provides the multi-model outputs and fused detection results, while Horowitz provides the display and user-interaction environment. The rejection is based on the combined teachings of Solovyev in view of Horowitz.
Applicant further argues that the cited references do not teach the limitation "such that a user can compare the similarities or differences in the outputs of the ML algorithms" is describing the intended use of displaying the interactive graphical representation. The claims do not require the user to compare the outputs of the ML algorithms. The claim limitation is met if the prior art teaches displaying. This argument is not persuasive because Horowitz teaches a graphical screen showing images, masks, or bounding boxes.
Applicant additionally argues that the cited references do not teach improving an operational characteristic using the comparison results. This argument is not persuasive. Solovyev expressly teaches that combining predictions from multiple models “usually yields more accurate results compared to a single model,” thereby improving detection performance. Horowitz further teaches using machine-learning output generated from sensed data to improve operational performance within the sorting facility. Accordingly, using comparison results from multiple ML outputs to improve an operational characteristic would have been an obvious and predictable application of the cited teachings.
Accordingly, the applicant argument is not persuasive, and the rejection of claims 1, 4-8, and 11-14 is maintained.
Applicant's arguments, see page 9 in regards of claim 2, and 9, filed 3/20/2026 have been fully considered but they are not persuasive. Applicant argues that Casado-Garcia does not cure the alleged deficiencies of Solovyev and Horowitz with respect to independent claims 1 and 8. This argument is not persuasive because, as set forth above, Solovyev in view of Horowitz teaches or suggests the limitations of independent claims 1 and 8, including displaying the combined or super-imposed labeled sensor input frame on a user interface such that a user can compare similarities or differences in the outputs of the ML algorithms.
Applicant further argues that claims 2 and 9 are patentable merely because they depend from claims 1 and 8. This argument is not persuasive because the base claims are not allowable for the reasons previously stated.
Accordingly, the applicant argument is not persuasive, and the rejection of claims 2 and 9 is maintained.
Applicant's arguments, see page 9 in regards of claims 3 and 10, filed 3/20/2026 have been fully considered but they are not persuasive. Applicant argues that Lee does not cure the alleged deficiencies of Solovyev and Horowitz with respect to independent claims 1 and 8. This argument is not persuasive because Solovyev in view of Horowitz teaches or suggests the limitations of independent claims 1 and 8 for the reasons previously stated. Lee is not relied upon to replace Horowitz’s user-interface teachings. Rather, Lee is relied upon for the additional limitations recited in dependent claims 3 and 10.
Applicant also argues that Lee does not disclose generating or displaying a graphical representation of the super-imposed labeled sensor input frame to a user. This argument is not persuasive because the rejection does not rely on Lee for that limitation. Solovyev teaches the multi-model object-detection and fusion aspects, including combining bounding-box predictions from different object-detection models. Horowitz teaches the user-interface/display environment. Lee is additionally relied upon for comparing and combining region proposals generated by multiple CNN/object-detection models.
Applicant’s argument that claims 3 and 10 are patentable merely because they depend from claims 1 and 8 is not persuasive because claims 1 and 8 remain unpatentable over Solovyev in view of Horowitz. Applicant has not identified any separate limitation of claims 3 and 10 that is not taught or suggested by the cited combination.
Accordingly, the applicant argument is not persuasive, and the rejection of claims 3 and 10 is maintained.
Applicant's arguments, see page 10 in regards of claims 15, and 18-20, filed 3/20/2026 have been fully considered but they are not persuasive. As discussed above with respect to claim 1, Solovyev in view of Horowitz teaches or suggests the limitations of amended claim 15. Solovyev teaches processing sensor/image input using multiple machine-learning algorithms and combining outputs from those algorithms, including bounding boxes, labels, and confidence scores, to generate fused detection outputs. Horowitz teaches receiving sensed data, including image data from sensors, processing the sensed data using machine-learning techniques to identify target objects, and presenting the resulting outputs within a system that supports operator review and interaction. Thus, the limitations directed to sensor-input processing are taught by Solovyev and Horowitz.
Cavender-Bares is not relied upon to teach the base sensor-input processing limitations of claim 15. Rather, Cavender-Bares is relied upon only for the additional limitations recited in claims 15 and 18–20, particularly with respect to the agricultural platform including sensors operating within a physical environment. The rejection does not rely on Cavender-Bares as disclosing the entirety of the claimed sensor-input processing pipeline.
Applicant’s argument that Cavender-Bares does not disclose generating or displaying the super-imposed labeled sensor input frame is likewise not persuasive because Cavender-Bares is not cited for that limitation. Solovyev and Horowitz collectively teach those features, while Cavender-Bares provides additional teachings relevant to the dependent claim subject matter.
Applicant further argues that claims 18–20 are patentable because they depend from claim 15. This argument is not persuasive because claim 15 remains unpatentable for the reasons set forth above. Applicant has not separately identified any limitation in claims 18–20 that is not taught or suggested by the applied combination.
Accordingly, the applicant argument is not persuasive, and the rejection of claims 15 and 18-20 is maintained.
Applicant's arguments, see page 11 in regards of claims 16, filed 3/20/2026 have been fully considered but they are not persuasive. As discussed above, Solovyev in view of Horowitz teaches or suggests the substantive sensor-input processing, multiple-ML-algorithm processing, combined/super-imposed labeled detection output, and user-interface display limitations of claim 15. Cavender-Bares is not relied upon to teach those base processing and display limitations; rather, Cavender-Bares is relied upon for the additional robotic/platform environment limitations. Casado-Garcia is likewise not relied upon to teach the entirety of claim 15, but instead is relied upon for the additional limitation of claim 16.
With respect to claim 16, Casado-Garcia teaches using ensemble object-detection methods, including voting strategies and model/data distillation, to improve object-detection models. Thus, it would have been obvious to use the combined or super-imposed labeled detection output generated by Solovyev, displayed and reviewed in Horowitz, and applied in the sensor-equipped platform environment of Cavender-Bares, as training data or feedback for further training one or more ML algorithms, consistent with Casado-Garcia’s teaching of ensemble-based model improvement.
Applicant’s argument that claim 16 is patentable merely because it depends from claim 15 is not persuasive because claim 15 remains unpatentable for the reasons previously stated. Applicant has not identified any separate limitation of claim 16 that is not taught or suggested by the applied combination.
Accordingly, the applicant argument is not persuasive, and the rejection of claim 16 is maintained.
Applicant's arguments, see page 11 in regards of claims 17, filed 3/20/2026 have been fully considered but they are not persuasive. As discussed above, Solovyev in view of Horowitz teaches or suggests the substantive limitations of independent claim 15, including the multiple-ML-algorithm processing, combined/super-imposed labeled output, and user-interface display for comparison. Cavender-Bares is relied upon for the additional platform/sensor environment limitations, not for the full sensor-input processing pipeline. Lee is not relied upon to cure the alleged deficiencies of claim 15; Lee is relied upon for the additional limitation of dependent claim 17.
With respect to claim 17, Lee teaches comparing and combining region proposals generated by multiple ML/object-detection models to improve object-detection output. This teaching further supports the obviousness of using multiple ML algorithm outputs, comparing their detected regions, and combining the outputs to generate an improved detection result.
Applicant’s argument that Lee does not disclose all limitations of claim 15 is not persuasive because the rejection relies on the combined teachings of the references, not Lee alone. Solovyev and Horowitz provide the claim 15 processing and display teachings, Cavender-Bares provides the applicable sensor-equipped platform environment, and Lee provides the additional region-proposal comparison/combination teaching for claim 17.
Applicant’s argument that claim 17 is patentable merely because it depends from claim 15 is not persuasive because claim 15 remains unpatentable for the reasons previously stated. Applicant has not identified any separate limitation of claim 17 that is not taught or suggested by the applied combination.
Accordingly, the applicant argument is not persuasive, and the rejection of claim 17 is maintained.
Claim Objections
Claim 16-20 objected to because of the following informalities: It is not clear if the "computer-readable medium" in claims 16-20 are referring to the " A non-transitory computer-readable medium " of claim 15. Appropriate correction is required. For the purpose of examination, the examiner interpreted it as “A non-transitory computer-readable medium”
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 1, 8, and 15 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Claims 1, 8, and 15, as amended, each recite "displaying an interactive graphical representation of the super-imposed labeled sensor input frame to a user on a user interface such that the user can compare the similarities or differences in the outputs of the ML algorithms." The originally filed specification does not provide written description support for this limitation.
The specification describes the super-imposed labeled sensor input frame in the context of FIGS. 6–7. Per paragraph [0067], labeled image outputs from multiple ML models "may be provided to a verification stage 1610 as a super-labeled image." Per paragraph [0072], the resulting labeled images "may be combined (1740) into a super-imposed labeled image," and the super-imposed labeled images "may be used for further training (1760) with or without additional user feedback." In each instance, the super-imposed labeled image is described as being provided to an internal verification stage (1610) or a training stage (1612/1760) not as being displayed to a user, much less displayed as an interactive graphical representation. Therefore, the claims are rejected under 35 USC 112(a) as failing to comply with written description requirement.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4, 6-8, 11, 13-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Solovyev et al. (Weighted boxes fusion: Ensembling boxes from different object detection models, 3 Feb 2021) in view of Horowitz et al. (US 12340287 B2).
As per claim 1, Solovyev as modified by Horowitz teaches A computer-implemented method of sensor input processing, implemented by an agricultural platform comprising a processor and a sensor, comprising:
processing the sensor input by multiple machine learning (ML) algorithms, each using a corresponding ML model for generating labels for objects identified in the sensor input; (see Solovyev [Introduction] “detecting instances of semantic objects of a particular class in images and videos … Object detection models typically return proposed locations of the objects of a given class, class labels, and confidence scores.” And see [6.4. An ensemble of many different models] “we combine predictions from several different models”)
combining labels generated by each ML algorithm to generate a super-imposed labeled sensor input frame; (see Solovyev [3. Weighted boxes fusion] “we have bounding boxes predictions for the same image from N different models. Alternatively, we have N predictions of the same model for original and augmented versions of the same image.”, and see fig. 2 it represents that Both NMS and soft-NMS exclude some boxes however WBF will fuse it using all predicted boxes.)
PNG
media_image1.png
494
544
media_image1.png
Greyscale
comparing outputs of the ML algorithms to determine similarities or differences; and (see Solovyev [3. Weighted boxes fusion] “3. Iterate through predicted boxes in B in a cycle and try to find a matching box in the list F. The match is defined as a box with a large overlap with the box under question (IoU > THR). Note: in our experiments, THR= 0.55 was close to an optimal threshold.”, and see “4. If the match is not found, add the box from the list B to the end of lists L and F as new entries; proceed to the next box in the list B.”, and see “5. If the match is found, add this box to the list L at the position pos corresponding to the matching box in the list F.”)
displaying an interactive graphical representation of the super-imposed labeled sensor input frame to a user on a user interface such that the user can compare the similarities or differences in the outputs of the ML algorithms (see Solovyev [1. Introduction] “the proposed WBF method uses confidence scores of all proposed bounding boxes to constructs average boxes.”, and see Solovyev [3. Weighted boxes fusion] “3. Iterate through predicted boxes in B in a cycle and try to find a matching box in the list F. The match is defined as a box with a large overlap with the box under question (IoU > THR). Note: in our experiments, THR= 0.55 was close to an optimal threshold.”, and see “4. If the match is not found, add the box from the list B to the end of lists L and F as new entries; proceed to the next box in the list B.”, and see “5. If the match is found, add this box to the list L at the position pos corresponding to the matching box in the list F.”)
using results of the comparing for improving an operational characteristic of the sensor input processing. (See Solovyev [3. Weighted boxes fusion] “6. Recalculate box coordinates and its confidence score in F[pos], using all T boxes accumulated in the cluster L[pos], with the following fusion formulas ….” And see “7. After all boxes in B are processed, re-scale confidence scores in F list: multiply it by a number of boxes in a cluster and divide by a number of models N. If the number of boxes in the cluster is low, it could mean that only a small number of models predict it. Thus, we need to decrease confidence scores for such cases ….”
Solovyev does not teach “receiving sensor input from the sensor;”, and “displaying an interactive graphical representation”
However, Horowitz teaches:
receiving sensor input from the sensor;(see Horowitz [Col 3 L 6] “imaging sensors can be used to rapidly recognize objects within sensed data based on image training and subsequent machine learning techniques.”)
displaying an interactive graphical representation (see Horowitz [Col 22 L 12] “may include one or more of the following: a command-line interface, graphical screens showing images, masks, bounding boxes, and links to external tools. The annotation user interface can support both the manual labeling of data objects (e.g., the assignment of the label “PET” to an image of a plastic bottle) as well as automated labeling tools.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the invention of Solovyev to include an imaging sensor and a user interface to rapidly recognize objects within sensed data and present the output in a user interface to the user for the predictable purpose of user review, comparison, validation, and model improvement. [Col 3 L6]
As per claim 4, Solovyev teaches wherein the ML models include ML models that are different versions of a same baseline ML model that has undergone different training. (See Solovyev [6.4. An ensemble of many different models] “6.4.1. Ensemble of models for COCO dataset … we combine predictions from several different models. We used models trained … different backbones”, and see “6.4.2. Ensemble of RetinaNet models for open images dataset the results obtained on Open Images Dataset [18] for combining predictions from RetinaNet models with different backbones ….”, and see “6.4.3. Ensemble of fairly different models for open images dataset in this experiment, performed on the Open Images dataset, we used a range of different models.”)
As per claim 6, Solovyev as modified by Horowitz teaches generating an ML performance metrics based on the comparison of outputs. (See Horowitz [Col 22 L 48] “In some embodiments, model evaluation logic 210 is configured to implement software to analyze and compare the performance across multiple training sessions. This analysis is provided both as numerical or statistical metrics and uses graphical representations of performance metrics (e.g., such as model convergence time, comparison of model accuracy against real data, etc.).”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the invention of Solovyev to include a model evaluation logic capable of generating numerical or statistical performance metrics to compare the performance of multiple machine learning models run against various training datasets. [Col 22 L55]
As per claim 7, Horowitz teaches further comprising: presenting the ML performance metrics on a user interface. (See Horowitz [Col 22 L 48] “In some embodiments, model evaluation logic 210 is configured to implement software to analyze and compare the performance across multiple training sessions. This analysis is provided both as numerical or statistical metrics and uses graphical representations of performance metrics (e.g., such as model convergence time, comparison of model accuracy against real data, etc.).”, and see [ Col 23 L 36] “These annotated videos are then used as an evaluation test set by model evaluation logic 210 to provide detailed metrics on model performance.”, and see [ Col 23 L 40] “) Report generation logic 212 is configured to provide the operational data and reports/visualization for one or more sorting facilities. … report generation logic 212 … incorporates a user interface”).
As per claim 8, this is directed to a system or a computing device claim that corresponds to
method claim 1. See the rejection for claim 1 above, which also applies to claim 8. In addition, Horowitz teaches an apparatus comprising a processor and a sensor (see [ Col 2 L 42] “such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor.” And see [ Col 3 L 6] “Artificial intelligence (AI) systems coupled to imaging sensors”).
As per claim 11, this is directed to a system or a computing device claim that corresponds to
method claim 4. See the rejection for claim 4 above, which also applies to claim 11.
As per claim 13, this is directed to a system or a computing device claim that corresponds to method claim 6. See the rejection for claim 6 above, which also applies to claim 13.
As per claim 14, this is directed to a system or a computing device claim that corresponds to method claim 7. See the rejection for claim 7 above, which also applies to claim 14.
Claims 2, 5, 9, 12 are rejected under 35 U.S.C. 103 as being unpatentable over Solovyev et al. (Weighted boxes fusion: Ensembling boxes from different object detection models, 3 Feb 2021) in view of Horowitz et al. (US 12340287 B2) and Casado-García et al. (Ensemble Methods for Object Detection, 2020).
As per claim 2, Solovyev as modified by Horowitz teaches the limitations of claim 1
Solovyev does not teach “wherein the operation characteristic is improved by:
performing further training of one or more ML models using the super-imposed labeled sensor input frame and/or the similarities of differences in outputs of the ML algorithms.”
However, Casado-García teaches wherein the operation characteristic is improved by:
performing further training of one or more ML models using the super-imposed labeled sensor input frame and/or the similarities of differences in outputs of the ML algorithms. (See Casado-García [3.3 Data and model distillation] “Data distillation [34] applies a trained model on manually labelled data to multiple transformations of unlabelled data, ensembles the multiple predictions, and, finally, retrains the model on the union of manually and automatically labelled data. Similarly, model distillation [5] obtains multiple predictions of unlabelled data using several models, ensembles the result, and retrains the models with the combination of manually and automatically annotated data.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the invention of Solovyev to perform further training to improve the accuracy of object detection models, as it is a well-known and routine practice in machine learning.
As per claim 5, Solovyev as modified by Horowitz teaches wherein the further training is performed based on user feedback on the super-labeled sensor input frame. (See Horowitz [Col 22 L 33] “In some embodiments, such machine learning model output labels are first confirmed by data scientists or operators before being stored as annotations corresponding to the training data. As such, an operator of the system can easily support the ingestion or creation of new object models to be utilized by the machine learning system.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the invention of Solovyev to incorporate human-in-the-loop feedback or active learning into a machine learning training model to improve the model. [Col 22 L24]
As per claim 9, this is directed to a system or a computing device claim that corresponds to method claim 2. See the rejection for claim 2 above, which also applies to claim 9.
As per claim 12, this is directed to a system or a computing device claim that corresponds to method claim 5. See the rejection for claim 5 above, which also applies to claim 12.
Claims 3, 10 are rejected under 35 U.S.C. 103 as being unpatentable over Solovyev et al. (Weighted boxes fusion: Ensembling boxes from different object detection models, 3 Feb 2021) in view of Horowitz et al. (US 12340287 B2) and Lee et al. (An Ensemble Method of CNN Models for Object Detection, 2018).
As per claim 3, Solovyev as modified by Horowitz teaches the limitations of claim 1
Solovyev does not teach “wherein the ML models include ML models that are based on different sets of hyperparameters.”
However, Lee teaches wherein the ML models include ML models that are based on different sets of hyperparameters. (See Lee [IV. EXPERIMENTS AND RESULTS] “The proposed ensemble method is evaluated with experiment on the comparison with the original ensemble method … with different hyperparameters.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the invention of Solovyev to include machine learning models that are based on different sets of hyperparameters to improve accuracy in object detection.
Note: it is widely recognized before the effective filing date of the invention that tuning or using machine learning models with different Hyperparameters is a fundamental technique for improving model accuracy.
As per claim 10, this is directed to a system or a computing device claim that corresponds to method claim 3. See the rejection for claim 3 above, which also applies to claim 10.
Claims 15, 18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Solovyev et al. (Weighted boxes fusion: Ensembling boxes from different object detection models, 3 Feb 2021) in view of Horowitz et al. (US 12340287 B2) and Cavender-Bares et al. (US 20160361949 A1).
As per claim 15, Solovyev as modified by Horowitz and Cavender-Bares teaches a non-transitory computer-readable medium having code stored thereon, the code, upon execution by a processor, causing the processor to implement a method of sensor input processing, comprising:
processing the sensor input by multiple machine learning (ML) algorithms, each using a corresponding ML model for generating labels for objects identified in the sensor input; (see Solovyev [Introduction] “detecting instances of semantic objects of a particular class in images and videos … Object detection models typically return proposed locations of the objects of a given class, class labels, and confidence scores.” And see [6.4. An ensemble of many different models] “we combine predictions from several different models”)
combining labels generated by each ML algorithm to generate a super-imposed labeled sensor input frame; (see Solovyev [3. Weighted boxes fusion] “we have bounding boxes predictions for the same image from N different models. Alternatively, we have N predictions of the same model for original and augmented versions of the same image.”, and see fig. 2 it represents that Both NMS and soft-NMS exclude some boxes however WBF will fuse it using all predicted boxes.)
PNG
media_image1.png
494
544
media_image1.png
Greyscale
comparing outputs of the ML algorithms to determine similarities or differences; and (see Solovyev [3. Weighted boxes fusion] “3. Iterate through predicted boxes in B in a cycle and try to find a matching box in the list F. The match is defined as a box with a large overlap with the box under question (IoU > THR). Note: in our experiments, THR= 0.55 was close to an optimal threshold.”, and see “4. If the match is not found, add the box from the list B to the end of lists L and F as new entries; proceed to the next box in the list B.”, and see “5. If the match is found, add this box to the list L at the position pos corresponding to the matching box in the list F.”)
displaying an interactive graphical representation of the super-imposed labeled sensor input frame to a user on a user interface such that the user can compare the similarities or differences in the outputs of the ML algorithms (see Solovyev [1. Introduction] “the proposed WBF method uses confidence scores of all proposed bounding boxes to constructs average boxes.”, and see Solovyev [3. Weighted boxes fusion] “3. Iterate through predicted boxes in B in a cycle and try to find a matching box in the list F. The match is defined as a box with a large overlap with the box under question (IoU > THR). Note: in our experiments, THR= 0.55 was close to an optimal threshold.”, and see “4. If the match is not found, add the box from the list B to the end of lists L and F as new entries; proceed to the next box in the list B.”, and see “5. If the match is found, add this box to the list L at the position pos corresponding to the matching box in the list F.”)
using results of the comparing for improving an operational characteristic of the sensor input processing. (See Solovyev [3. Weighted boxes fusion] “6. Recalculate box coordinates and its confidence score in F[pos], using all T boxes accumulated in the cluster L[pos], with the following fusion formulas ….” And see “7. After all boxes in B are processed, re-scale confidence scores in F list: multiply it by a number of boxes in a cluster and divide by a number of models N. If the number of boxes in the cluster is low, it could mean that only a small number of models predict it. Thus, we need to decrease confidence scores for such cases ….”
Solovyev does not teach “receiving sensor input from a sensor of an agricultural platform;”, “displaying an interactive graphical representation”
However, Horowitz teaches:
receiving sensor input from the sensor; (see Horowitz [Col 3 L 6] “imaging sensors can be used to rapidly recognize objects within sensed data based on image training and subsequent machine learning techniques.”)
displaying an interactive graphical representation (see Horowitz [Col 22 L 12] “may include one or more of the following: a command-line interface, graphical screens showing images, masks, bounding boxes, and links to external tools. The annotation user interface can support both the manual labeling of data objects (e.g., the assignment of the label “PET” to an image of a plastic bottle) as well as automated labeling tools.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the invention of Solovyev to include an imaging sensor and a user interface to rapidly recognize objects within sensed data and present the output in a user interface to the user for the predictable purpose of user review, comparison, validation, and model improvement. [Col 3 L6]
Solovyev-Horowitz does not teach “a sensor of an agricultural platform”
However, Cavender-Bares teaches receiving sensor input from a sensor of an agricultural platform; (see Cavender-Bares paragraph [0040] “FIG. 11A depicts an individual planted crop as viewed from above by a forward-facing sensor on an agricultural platform traversing substantially parallel to a planted row in accordance with an embodiment of the disclosure.”, and see Paragraph [0041] “FIG. 11B depicts an individual planted crop when viewed from the side by a forward-facing sensor on a platform traversing generally traverse to a planted row in accordance with an embodiment of the disclosure.”, and see paragraph [0061] “a control system … that uses one or more sensor inputs to locate the position of crop rows … relative to the moving agricultural platform”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the invention of Solovyev-Horowitz to include an imaging sensor on an agricultural platform for traversal of an agricultural field without crushing the individual plants. [0047]
As per claim 18, Solovyev teaches the computer-readable medium of claim 1, wherein the ML models include ML models that are different versions of a same baseline ML model that has undergone different training. (See Solovyev [6.4. An ensemble of many different models] “6.4.1. Ensemble of models for COCO dataset … we combine predictions from several different models. We used models trained … different backbones”, and see “6.4.2. Ensemble of RetinaNet models for open images dataset the results obtained on Open Images Dataset [18] for combining predictions from RetinaNet models with different backbones ….”, and see “6.4.3. Ensemble of fairly different models for open images dataset in this experiment, performed on the Open Images dataset, we used a range of different models.”)
As per claim 20, Solovyev as modified by Horowitz teaches the computer-readable medium of claim 1, wherein the method further includes:
generating an ML performance metrics based on the comparison of outputs, or (See Horowitz [Col 22 L 48] “In some embodiments, model evaluation logic 210 is configured to implement software to analyze and compare the performance across multiple training sessions. This analysis is provided both as numerical or statistical metrics and uses graphical representations of performance metrics (e.g., such as model convergence time, comparison of model accuracy against real data, etc.).”)
presenting the ML performance metrics on a user interface. (See Horowitz [Col 22 L 48] “In some embodiments, model evaluation logic 210 is configured to implement software to analyze and compare the performance across multiple training sessions. This analysis is provided both as numerical or statistical metrics and uses graphical representations of performance metrics (e.g., such as model convergence time, comparison of model accuracy against real data, etc.).”, and see [ Col 23 L 36]” These annotated videos are then used as an evaluation test set by model evaluation logic 210 to provide detailed metrics on model performance.”, and see [Col 23 L 40] “) Report generation logic 212 is configured to provide the operational data and reports/visualization for one or more sorting facilities. … report generation logic 212 … incorporates a user interface”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the invention of Solovyev to include a model evaluation logic capable of generating numerical or statistical performance metrics to compare the performance of multiple machine learning models run against various training datasets. [Col 22 L55]
Claims 16, 19 are rejected under 35 U.S.C. 103 as being unpatentable over Solovyev et al. (Weighted boxes fusion: Ensembling boxes from different object detection models, 3 Feb 2021) in view of Horowitz et al. (US 12340287 B2) and Cavender-Bares et al. (US 20160361949 A1) and further in view of Casado-García et al. (Ensemble Methods for Object Detection, 2020)..
As per claim 16, The computer-readable medium of claim 1, wherein the operation characteristic is improved by:
Solovyev-Horowitz as modified by Cavender-Bares teaches the limitations of claim 15
Solovyev-Horowitz-Cavender does not teach “wherein the operation characteristic is improved by:
performing further training of one or more ML models using the super-imposed labeled sensor input frame and/or the similarities of differences in outputs of the ML algorithms.”
However, Casado-García teaches wherein the operation characteristic is improved by:
performing further training of one or more ML models using the super-imposed labeled sensor input frame and/or the similarities of differences in outputs of the ML algorithms. (See Casado-García [3.3 Data and model distillation] “Data distillation [34] applies a trained model on manually labelled data to multiple transformations of unlabelled data, ensembles the multiple predictions, and, finally, retrains the model on the union of manually and automatically labelled data. Similarly, model distillation [5] obtains multiple predictions of unlabelled data using several models, ensembles the result, and retrains the models with the combination of manually and automatically annotated data.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the invention of Solovyev to perform further training to improve the accuracy of object detection models, as it is a well-known and routine practice in machine learning.
As per claim 19, Solovyev as modified by Horowitz teaches the computer-readable medium of claim 1, wherein the further training is performed based on user feedback on the super-labeled sensor input frame. (See Horowitz [Col 22 L 33] “In some embodiments, such machine learning model output labels are first confirmed by data scientists or operators before being stored as annotations corresponding to the training data. As such, an operator of the system can easily support the ingestion or creation of new object models to be utilized by the machine learning system.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the invention of Solovyev to incorporate human-in-the-loop feedback or active learning into a machine learning training model to improve the model. [Col 22 L24]
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Solovyev et al. (Weighted boxes fusion: Ensembling boxes from different object detection models, 3 Feb 2021) in view of Horowitz et al. (US 12340287 B2) and Cavender-Bares et al. (US 20160361949 A1) and further in view of Lee et al. (An Ensemble Method of CNN Models for Object Detection, 2018).
As per claim 17, The computer-readable medium of claim 1, wherein the ML models include ML models that are based on different sets of hyperparameters.
Solovyev-Horowitz as modified by Cavender-Bares teaches the limitations of claim 15
Solovyev-Horowitz-Cavender does not teach “wherein the ML models include ML models that are based on different sets of hyperparameters.”
However, Lee teaches wherein the ML models include ML models that are based on different sets of hyperparameters. (See Lee [IV. EXPERIMENTS AND RESULTS] “The proposed ensemble method is evaluated with experiment on the comparison with the original ensemble method … with different hyperparameters.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the invention of Solovyev to include machine learning models that are based on different sets of hyperparameters to improve accuracy in object detection.
Note: it is widely recognized before the effective filing date of the invention that tuning or using machine learning models with different Hyperparameters is a fundamental technique for improving model accuracy.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABDULLAH K ABOUD whose telephone number is (571)272-0025. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen, can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ABDULLAH KHALED ABOUD/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights