DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment to the claims filed on 11/7/2025 does not comply with the requirements of 37 CFR 1.121(c) because the amendments did not provide a marked-up version of the amended claims. Amendments to the claims filed on or after July 30, 2003 must comply with 37 CFR 1.121(c) which states:
(c) Claims. Amendments to a claim must be made by rewriting the entire claim with all changes (e.g., additions and deletions) as indicated in this subsection, except when the claim is being canceled. Each amendment document that includes a change to an existing claim, cancellation of an existing claim or addition of a new claim, must include a complete listing of all claims ever presented, including the text of all pending and withdrawn claims, in the application. The claim listing, including the text of the claims, in the amendment document will serve to replace all prior versions of the claims, in the application. In the claim listing, the status of every claim must be indicated after its claim number by using one of the following identifiers in a parenthetical expression: (Original), (Currently amended), (Canceled), (Withdrawn), (Previously presented), (New), and (Not entered).
(1) Claim listing. All of the claims presented in a claim listing shall be presented in ascending numerical order. Consecutive claims having the same status of “canceled” or “not entered” may be aggregated into one statement (e.g., Claims 1–5 (canceled)). The claim listing shall commence on a separate sheet of the amendment document and the sheet(s) that contain the text of any part of the claims shall not contain any other part of the amendment.
(2) When claim text with markings is required. All claims being currently amended in an amendment paper shall be presented in the claim listing, indicate a status of “currently amended,” and be submitted with markings to indicate the changes that have been made relative to the immediate prior version of the claims. The text of any added subject matter must be shown by underlining the added text. The text of any deleted matter must be shown by strike-through except that double brackets placed before and after the deleted characters may be used to show deletion of five or fewer consecutive characters. The text of any deleted subject matter must be shown by being placed within double brackets if strike-through cannot be easily perceived. Only claims having the status of “currently amended,” or “withdrawn” if also being amended, shall include markings. If a withdrawn claim is currently amended, its status in the claim listing may be identified as “withdrawn—currently amended.”
(3) When claim text in clean version is required. The text of all pending claims not being currently amended shall be presented in the claim listing in clean version, i.e., without any markings in the presentation of text. The presentation of a clean version of any claim having the status of “original,” “withdrawn” or “previously presented” will constitute an assertion that it has not been changed relative to the immediate prior version, except to omit markings that may have been present in the immediate prior version of the claims of the status of “withdrawn” or “previously presented.” Any claim added by amendment must be indicated with the status of “new” and presented in clean version, i.e., without any underlining.
(4) When claim text shall not be presented; canceling a claim.
(i) No claim text shall be presented for any claim in the claim listing with the status of “canceled” or “not entered.”
(ii) Cancellation of a claim shall be effected by an instruction to cancel a particular claim number. Identifying the status of a claim in the claim listing as “canceled” will constitute an instruction to cancel the claim.
(5) Reinstatement of previously canceled claim. A claim which was previously canceled may be reinstated only by adding the claim as a “new” claim with a new claim number.
In the interest of advancing prosecution, Examiner has nevertheless examined the non-compliant claims. Applicant should, however, ensure that a properly marked-up copy of the claims, in compliance with 37 CFR 1.121(c), is submitted on the record.
Response to Arguments
Applicant’s arguments, see remarks, filed 11/07/2025, with respect to the objection to the claims, rejection of the claims under 35 U.S.C. 112(b), and interpretation of the claims under 35 U.S.C. 112(f) have been fully considered and are persuasive, and therefore have all been withdrawn.
Applicant's arguments filed 11/07/2025 have been fully considered but they are not persuasive regarding the rejections of the claims under 35 U.S.C. 101. Applicant argues, on page 10 of the remarks, that “
PNG
media_image1.png
638
814
media_image1.png
Greyscale
”.
Examiner disagrees. There are no examples of specialized equipment that can be shown by Applicant in the claim. The sensory system, machine learning model, separator unit, separate accessible compartment, and an incremental version of the machine learning model, are all generic components that are not specialized; Applicant cannot argue sensors such as X-Ray, high-speed cameras, etc. are used because they are not recited in the claims. As Examiner previously argued in the previous Office Action, the human moves the trash themselves as a “separator” into a separate accessible compartment using their own vision (scanning/scanner) and body, and a machine learning model being trained is a generic computer component that equates to a human learning/remembering using their own human vision over time. Applicant arguments are only valid if there was an example of specialized hardware in the claim that Applicant can point to that makes the claim an “inherently technological process” as they argue. The claimed method does not contain capturing data at the “speed, volume, specificity, and spectral range” that the human eye cannot capture. Applicant further argues on page 11 of the remarks, that “
PNG
media_image2.png
444
806
media_image2.png
Greyscale
”
without 1) pointing to where in the claims there is “specialized sensing equipment”, 2) proving how a human cannot reasonably function as a “separator unit” themselves to identify and isolate trash on a moving conveyor belt through their own human vision and body, and 3) how a human cannot reasonably do a “chemical analysis” such as burning the piece of trash with a lighter to see, learn, and record their properties. Applicant must add extra details in the claims regarding the separation/isolation step, chemical analysis step, or the type of sensing equipment to make the argument that these steps cannot be reasonably accomplished by a human being with generic equipment assistance. Applicant further argues, on page 12 of the remarks, that “
PNG
media_image3.png
546
802
media_image3.png
Greyscale
”.
Examiner disagrees. Applicant cannot point to what exactly the specific technological solution is that differentiates the claims from human trash sorters isolating items they cannot identify properly on a conveyor belt and then learning about the unknown item by doing generic chemical analysis for future trash sorting; Applicant argues that the solution is training a machine learning model; however, Applicant is silent regarding Examiner’s arguments in the previous Office Action, that simply training a machine learning model to do a process that a human easily accomplishes is not itself a technological improvement, but rather technological automation of human endeavors. Examiner argues that all additional elements recited in the claims generally links the judicial exception to a particular technological environment or field of use, rather than actually improve the technology itself, as discussed in the 2024 Guidance Update on Patent Subject Matter Eligibility, Including on Artificial Intelligence.
Therefore, the rejection of the claims under 35 U.S.C. 101 is maintained.
Applicant’s arguments, see remarks, filed 11/07/2025, with respect to the rejection of the claims 1 and 26 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of non-patent literature "Zenrobotics recycler–robotic sorting using machine learning"; Proceedings of the international conference on sensor-based sorting (SBS); No. 1; Citeseer, 2014 (Lukka et al.) (hereinafter Lukka). Applicant amended the details of dependent claim 11 into independent claims 1 and 26 and addressed the Lukka reference cited by Examiner in the previous Office Action for those aspects in the amendment. Applicant argues, on page 15 of the remarks, that “
PNG
media_image4.png
448
802
media_image4.png
Greyscale
”.
Examiner Disagrees. Lukka was not cited by Examiner to teach or suggest “using the difficulty of isolation as a criterion to select which unknown component should be analyzed for ground truth labeling to train the model” as argued by Applicant. Rather, Lukka was cited only for the aspect of determining a value associated with each unknown component that is indicative of a difficulty for performing physical isolation of the unknown component from the material stream by the separator unit and selecting at least one unknown component from a plurality of unknown components in the material stream based at least partially on the value indicative of the difficulty for performing physical isolation, which Applicant has conceded to as shown above; Lukka determines how easily/difficult an object can be grabbed by an automated robot in a material stream and getting a human inspector to do it if the system deems it too difficult for the automated robot to grab; it is immaterial to the rejection of the claims whether Lukka is used to optimize for finding valuable objects in the material stream or not because that is not the section of Lukka Examiner cited in the 103 combination rejection. Wu, in view of Salman, when modified by Lukka, teaches selecting at least one unknown component from the plurality of unknown components in the material stream based at least partially on the training reward associated with the unknown components and the value of the difficulty for performing physical isolation (Salman teaches using computer vision and machine learning for generating scores (training rewards) that indicate how accurate the labeling of an object is from a plurality of objects recognized and ranks the objects based on which ones have the highest uncertainty that would benefit from user-annotated ground truth labeling for more efficient re-training of the ML model; one of ordinary skill in the art, when combining Wu and Salman applies this concept of “active learning” and ranking relevant items for re-labeling with ground truth labels, from Salman, to the waste material stream in Wu, to decide which waste items were most likely to be mis-labeled by the CNN and requires further analysis by a user; one of ordinary skill in the art, when combining Lukka with Wu and Salman, applies the concept of indicating the difficulty of physical isolation of waste stream items and only selecting items the separator/grabber accurately has the ability to grab to isolate (Lukka), together with waste stream items indicated as highly uncertain/mis-labeled (Salman and Wu), so waste items indicated as very difficult to isolate by a separation unit, but are indicated as potentially mis-labeled/identified, are ignored so the automation of the system operation continues without isolating that unknown item because it will jeopardize the separator unit from working properly and potentially damage the system).
Therefore, the rejection of the claims under 35 U.S.C. 103 is maintained.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 2, 4, 6-7, 9-10, 12, 16-17, 20-21, 23, and 26 are rejected are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., an abstract idea) without integration into a practical application or recitation of significantly more.
In the analysis below, the method of independent claim 1 is considered representative of independent claim 26 since all of the independent claims recite identical steps despite being directed to different statutory matter. Furthermore, independent claims 1 and 26 are directed to one of the four statutory categories of eligible subject matter (a process for independent claim 1 and an apparatus for independent claim 26); thus, the claims pass Step 1 of the Subject Matter Eligibility Test (See flowchart in MPEP 2106).
Step 2A, prong 1 analysis:
The independent claims are directed to scanning the material stream to perform imaging of the material stream with the plurality of unknown components; predicting one or more prediction labels and associated label prediction probabilities for each of the unknown components in the material stream by analyzing the imaging of the material stream and/or one or more features of the unknown components extracted from the imaging of the material stream; determining a training reward associated with each unknown component within the plurality of unknown components in the material stream; determining a value associated with each unknown component that is indicative of a difficulty for performing physical isolation of the unknown component from the material stream; selecting at least one unknown component from the plurality of unknown components in the material stream based at least partially on the training reward associated with the unknown components and the value indicative of the difficulty for performing physical isolation, wherein the selected at least one unknown component is physically isolated from the material stream by moving the selected unknown component to a separate accessible compartment; analyzing the isolated at least one unknown component for determining a ground truth label thereof, wherein the determined ground truth label of the isolated at least one unknown component is added to a training database; and training, using the determined ground truth label of the physically isolated at least one unknown component; wherein the at least one unknown component which is isolated from the material stream is subjected to chemical analysis for determining the ground truth label at least partially based thereon.
Each of the above steps can be performed mentally. In particular, at a landfill that processes waste/trash/garbage, the trash is collected and put onto a conveyor belt for sorting by humans; human vision is used to analyze the trash to identify and label/classify each piece moving along the conveyor belt; If there are waste items that are unidentifiable or the humans have low confidence in their labeling/prediction of what the waste item is, then the waste item is sorted/separated manually into another compartment off of the conveyor so that it is physically isolated for further evaluation, unless it is determined (done for each item before isolation) that it is too difficult to physically isolate because of its size, or lack of ability to separate the item from other items on the belt; an additional human who uses technology such as chemical analyzers analyzes the waste with more detail then the conveyor human analyzers looking at numerous waste items in the heterogenous mix quickly, and determines the classification of the waste item that has been sorted away from the others; the additional human analyzer who analyzes the isolated waste item then takes an image of the waste item and notes in a database what the item actually is and trains the other human analyzer who sort the waste items on the conveyor to remember and recognize this specific waste item so it does not have to be sorted for further expert labeling in the future; this improves the efficiency of waste sorting; therefore, this process can all be done mentally.
As such, the description in independent claims 1 and 26 is an abstract idea – namely, a mental process. Accordingly, the analysis under prong one of step 2A of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
Additional elements:
The additional element recited in independent claims 1 and 26 are a sensory system, a machine learning model, a separator unit, a separate accessible compartment, and an incremental version of the machine learning model.
Step 2A, prong 2 analysis:
The above-identified additional elements do not integrate the judicial exception into a practical application. The human moves the trash themselves as a “separator” into a separate accessible compartment using their own vision, and body; a machine learning model being trained is a generic computer component that equates to a human learning/remembering using their own human vision over time.
Each of the other additional elements (a sensory system, a machine learning model, a separator unit, a separate accessible compartment, and an incremental version of the machine learning model) amounts to merely using different devices as tools to perform the claimed mental process. Implementing an abstract idea on a computer or using known generic devices does not integrate a judicial exception into a practical application (See MPEP 2106.05(f)).
Moreover, the additional elements of the claims do not recite an improvement in the functioning of a computer or other technology or technical field, the claimed steps are not performed using a particular machine, the claimed steps do not effect a transformation, and the claims do not apply the judicial exception in any meaningful way beyond generically linking the use of the judicial exception to a particular technological environment (See MPEP 2106.04(d)). Therefore, the analysis under prong two of step 2A of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
Step 2B:
Finally, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Each of the other additional elements (a sensory system, a machine learning model, a separator unit, a separate accessible compartment, and an incremental version of the machine learning model) are generic computer features which perform generic computer functions that are well-understood, routine, and conventional and do not amount to more than implementing the abstract idea with a computerized system. Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea).
Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation, and mere implementation on a generic computer does not add significantly more to the claims. Accordingly, the analysis under step 2B of the Subject Matter Eligibility Test does not result in a conclusion of eligibility (See flowchart in MPEP 2106).
For all of the foregoing reasons, independent claims 1 and 26 do not recite eligible subject matter under 35 USC 101.
Claims 2, 4, 6-7, 9-10, 12, 16-17, 20-21, and 23 all teach different aspects of separating waste items, visually analyzing the waste items for identification by observing features such as size, shape, etc., ranking waste items depending on how mixed up with other items they are, manually moving the waste items, doing destructive measurements such as burning the waste item, and using an X-ray fluorescence machine to analyze the waste items, which are all steps easily accomplished by human landfill workers; a human has the ability to use a fluid jet to blow waste items off a conveyor (controlling the force based on the mass of the waste item) when they conclude further analysis of the waste item is needed or lack of confidence/high uncertainty in their labeling; the fluid jet and X-Ray fluorescence machine are extra-solution components; therefore, these processes can all be done mentally.
Therefore, dependent claims 2, 4, 6-7, 9-10, 11-12, 16-17, 20-21, and 23 recite the same abstract idea of a mental process which can be performed in the mind with the aid of pen and paper, and are therefore also rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 12, 16, 23, and 26 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Application Publication No.: 2020/0050922 (Wu et al.) (hereinafter Wu), in view of U.S. Patent Application Publication No.: 2022/0262104 (Salman et al.) (hereinafter Salman), in view of non-patent literature "Zenrobotics recycler–robotic sorting using machine learning"; Proceedings of the international conference on sensor-based sorting (SBS); No. 1; Citeseer, 2014 (Lukka et al.) (hereinafter Lukka), and in view of Chinese Patent Application Publication No.: CN 104299315 A (Zhang et al.) (hereinafter Zhang).
Regarding claim 1, Wu teaches A method for training a machine learning model configured to perform characterization of components in a heterogeneous material stream with a plurality of unknown components, the method comprising (Wu, abstract: “a recycling system and a method based on deep-learning and computer vision technology are disclosed. The system includes a trash sorting device and a trash sorting algorithm. The trash sorting device includes a trash arraying mechanism, trash sensors, a trash transfer mechanism and a controller. The trash arraying mechanism is configured to process trash in a batch manner. The controller drives the trash arraying mechanism according to the signals of trash sensors and controls the sorting gates of the trash sorting mechanism to rotate. The trash sorting algorithm makes use of the images of trash, wherein the images are taken by cameras in different directions. The trash sorting algorithm includes a dynamic object detection algorithm, an image pre-processing algorithm, an identification module and a voting and selecting algorithm. The identification module is based on the convolutional neural networks (CNNs) and may at least identify four kinds of trash.”; the CNN’s are trained; see para. [0041] of Wu below for further details on the training)
scanning the material stream by means of a sensory system configured to perform imaging of the material stream with the plurality of unknown components (Wu, para. [0031]: “The identification unit 660 contains at least the first video camera 71, the second video camera 72, and the metal sensor 73. The identification unit 660 is disposed over the trash transfer mechanism 2 and under the cover 61. The first video camera 71 is disposed to vertically and downward photograph the trash objects on the garbage transfer mechanism 2. The second camera 72 obliquely photographs the trash objects on the trash transfer mechanism 2.”;
PNG
media_image5.png
678
666
media_image5.png
Greyscale
);
predicting one or more prediction labels and associated label prediction probabilities for each of the unknown components in the material stream by means of a machine learning model which is configured to receive as input the imaging of the material stream and/or one or more features of the unknown components extracted from the imaging of the material stream (Wu, para. [0041]-[0044]; para. [0054]; FIG. 6: “The identification module 630 includes at least two convolutional neural network models. The convolutional neural network models may be, for example, a VGG 16 convolutional network module. Each convolutional neural network model is used to identify images taken by different cameras. The identification module 630 includes two convolutional neural network models. The first convolutional neural network model 631 is trained by images taken by the first camera 71 configured vertically relative to the trash transfer mechanism 2 … The second convolutional neural network model 632 is trained by images taken by the second camera 72 … According to an embodiment of the present disclosure, the identification module 630 can identify four different types of trash and generate the identification results … The types of trash which the identification module 630 can identify are arbitrary, depending on the number of trash types that train the first convolutional neural network model 631 and the second convolutional neural network model 632. The voting and selecting algorithm 640 receives the identification results of the identification module 630. After many identification results are accumulated, the final image identification result 650 is further determined by statistics. An example of voting and selecting according to an embodiment of the present invention, please refer to Table 1:
PNG
media_image6.png
194
430
media_image6.png
Greyscale
.
The identification results of each convolutional neural network model are counted by the voting and selecting algorithm 640 to determine the final image identification result 650 (voting and selecting result). In Table 1, according to the identification of the first convolutional neural networks 631, from the 10 images taken from the first camera 71, one of which is identified as the first type of trash, another two are identified as the second type of trash, yet another two are identified as the third type of trash, and the remaining five are identified as the fourth type of trash; according to the identification of the second convolutional neural networks 631, from the 10 images taken from the second camera 72, two of which are identified as the first type of trash, another two are identified as the second type of trash, yet another two are identified as the third type of trash, and the remaining four are identified as the fourth type of trash. The results of the above identification module are statistically calculated by the voting and selecting algorithm 640, and the comprehensive results are shown in Table 1; “Referring to Table 2, which shows the identification accuracy according to an embodiment of present invention for four types of the trash objects (Types 1˜4 in Table 2 respectively represent PET bottle, Tetra Paka, iron or aluminum can, and trash of types different from the above types) …
PNG
media_image7.png
140
436
media_image7.png
Greyscale
”;
Examiner has mapped Wu to teaching a machine learning model which is configured to receive as input the imaging of the material stream and not to receive as input one or more features of the unknown components extracted from the imaging of the material stream, because the CNN machine learning model already does the step of extracting one or more features from the imaging of the material stream after receiving the image of the material stream; this is appropriate because Applicant used the claim term “and/or”;
PNG
media_image8.png
931
604
media_image8.png
Greyscale
); and
selecting at least one unknown component from the plurality of unknown components in the material stream based on the prediction label one or more prediction labels and associated label prediction probabilities for each of the unknown components in the material stream, wherein the selected at least one unknown component is physically isolated from the material stream by means of a separator unit, wherein the separator unit is configured to move the selected unknown component to a separate accessible compartment (Wu, para. [0045]; para. [0032]; FIG. 1; FIG. 4-5: “According to an embodiment of the present disclosure, the image identification result 650 (camera detecting signal) obtained by the trash sorting algorithm module shown in FIG. 6 is further combined with the metal detecting signal measured by the metal sensor 73 through the identification unit 660 to generate the fourth signal. The fourth signal decides the rotating action of the sorting gate 82 of the trash sorting mechanism 8, and opens the entrance of the trash object passage corresponding to the type of the trash object, and closes the entrances of the other trash object passages. The trash object is transported to the outlet of the trash object passage by the trash object transfer mechanism 2 for temporary storage to a corresponding position in the trash storage bin 9.”; “Referring to FIG. 4 and FIG. 5, FIG. 4 illustrates the structure of the trash sorting mechanism according to an embodiment of the present invention; FIG. 5 illustrates the operation of the trash sorting mechanism according to an embodiment of the present invention. The trash sorting mechanism 8 is disposed on the trash transfer mechanism 2 to provide a function of sorting trash. The trash sorting mechanism 8 includes at least two sorting gate drive motors 81, at least two sorting gates 82 and multiple parallel trash channels 83. The sorting gates 82 stably connected to the sorting gate drive motors 81 are configured on one end of each trash tunnel 83, which is closer to the identifying unit 660 than the other end thereof. These two sorting gates 82 decide the open or closed state of the entrances of these three trash tunnels 83 by rotating, wherein each of the trash tunnels 83 is used for passing (or sorting) different types of trash.”; see components 73, 81-83, and 9 in FIG. 1 above for trash sorting
PNG
media_image9.png
488
538
media_image9.png
Greyscale
;
PNG
media_image10.png
260
478
media_image10.png
Greyscale
).
Wu fails to teach
determining a training reward associated with each unknown component within the plurality of unknown components in the material stream; selecting at least one unknown component from the plurality of unknown components in the material stream based at least partially on the training reward associated with the unknown components; analyzing the isolated at least one unknown component for determining a ground truth label thereof, wherein determining the ground truth for said at least one unknown component requires analysis in physical isolation, wherein the determined ground truth label of the isolated at least one unknown component is added to a training database; and training an incremental version of the machine learning model using the determined ground truth label of the physically isolated at least one unknown component.
Salman teaches
determining a training reward associated with each unknown component within the plurality of unknown components (Salman, para. [0119]; para. [0142]-[0143]; FIG. 5-6: “As shown in the example of FIG. 6, the method 600 continues from the train block 620 to a decision block 630 for deciding whether one or more targeted goals as to training of the one or more machine learning tools are met. Where the decision block 630 decides that such one or more goals have been met, the method 600 continues to the train block 680 for training an inspection learner (e.g., an object detector, etc.); otherwise, where the decision block 630 decides that such one or more goals have not been met, the method 600 continues to a prediction block 642 for predicting using a pool of unlabeled data (see, e.g., the unlabeled data block 524 of FIG. 5), which proceeds to a computation block 644 for computing one or more scores as to one or more of typicality, density, uncertainty and relevance (see, e.g., the QC block 540 of FIG. 5) and then a rank block 646 for ranking unlabeled observations using weighted scores.”; “As an example, a QC component can provide for uncertainty computations. For example, consider computations that a quality control component can perform during active learning where such computations indicate how confident an inspection learner is about its predictions. Depending on an inspection learner architecture, one or more of various uncertainty metrics can be implemented. As an example, one or more uncertainty scores can be computed based upon output of an inspection learner. FIG. 10 shows an example of a quality control component or quality control block 1040 that can receive prediction via a prediction block 1042, generate one or more metrics 1044-1, 1044-2, to 1044-N, and determine one or more scores 1046.”;
PNG
media_image11.png
692
716
media_image11.png
Greyscale
;
PNG
media_image12.png
886
583
media_image12.png
Greyscale
);
analyzing the at least one unknown component for determining a ground truth label thereof, wherein the determined ground truth label of the at least one unknown component is added to a training database (Salman, para. [0137]; FIG. 9; para. [0153]; para. [0168]: “FIG. 9 shows an example of a machine learning component or machine learning block 930 that can provide for training and prediction. As shown, an input block 931 can provide input to a feed learner block 932 for making a prediction per a prediction block 933, which can provide a predicted output per an output block 939. Such a stream may be a prediction stream. FIG. 9 also shows a ground truth block 934 that includes acceptably labeled observation data that can be utilized for making comparisons to one or more predictions of the prediction block 933. In such an example, a computation block 935 can compute one or more error metrics that can be utilized to guide an update block 936 for updating a learner (e.g., re-training, further training, etc.).”;
PNG
media_image13.png
673
728
media_image13.png
Greyscale
;
“As an example, a framework can include a labeling component, which may be part of a healing and labeling component. As an example, an active labeling process can utilize an oracle to establish a so-called “ground truth” for observations. As mentioned, an oracle may be a human annotator or, for example, consider a framework that is adaptable for simulated data where labels may be already available and/or generated via simulation. As to the latter, a framework may provide for selecting out of a very large quantity of observations, particular observations to be utilized for training.”; “As explained, an annotator may be utilized to establish one or more ground truths for training data (e.g., segmentation, classification, human activity recognition, etc.) … As an example, in the case of asset integrity inspection one type of inspection task involves detecting objects (e.g., anomalies) in a video stream. As an example, a framework may utilize one or more processes for typicality and uncertainty in the context of object detection.”); and training an incremental version of the machine learning model using the determined ground truth label of the at least one unknown component (Salman, para. [0062]: “As an example, a framework can utilize active learning where a trained machine learning model is utilized for one or more purposes in selecting data for labeling. In such an example, a selection process can involve assessing output from the trained machine learning model where less confident inferences (e.g., predictions) on unlabeled data can be utilized as an indicator that such unlabeled data is to be labeled such that, once labeled (e.g., by an oracle) to provide labeled data, the labeled data can be utilized to improve the trained machine learning model through further training. Such a process can be iterative where, for example, once the trained machine learning model achieves an acceptable level of confidence as to detection of features (e.g., objects, etc.), oracle-based labeling may be foregone and the trained machine learning model output, for example, for deployment in one or more machine vision systems (e.g., one or more inspection tools, etc.) for purposes of one or more inspection tasks.”).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the method for training a machine learning model configured to perform characterization of components in a heterogeneous material stream with a plurality of unknown components, as taught by Wu, to include the steps of determining a training reward associated with each unknown component within the plurality of unknown components, analyzing the at least one unknown component for determining a ground truth label thereof, wherein the determined ground truth label of the at least one unknown component is added to a training database, and training an incremental version of the machine learning model using the determined ground truth label of the at least one unknown component, as taught by Salman.
The suggestion/motivation for doing so would have been that “active learning may aim to reduce demands on a human oracle for annotation (e.g., labeling). As an example, an active learning workflow can include sampling and selecting such that data selected for labeling by an oracle have some assurances as to ability to improve training of a machine learning tool” (Salman, para. [0061]); additionally, “to improve efficiency where resources may be constrained (e.g., limited, etc.), a framework may aim to make best use of each available minute of each available annotator; in the case of multi-labeled inspection tasks (e.g., multiple labels for a single observation), a framework can provide for speeding up the labeling task by partially labeling observations with most confident labels so that an annotator can focus on the least confident labels; additionally, a framework can offer a mechanism for an annotator to review one or more predicted labels, as it may be faster to validate or reject a predicted label than create a label from scratch (e.g., it may be an utterly tedious task for object detection and even more for segmentation).” (Salman, para. [0093]).
Wu, in view of Salman, teaches determining a training reward associated with each unknown component within the plurality of unknown components in the material stream (the method of Salman of determining which objects imaged is most uncertain in terms of its label/classification that then is taken to be labeled with better ground truth label for more efficient re-training of the machine learning model is applied to each component of the waste stream on a conveyor belt taught in Wu to determine which waste component is most likely to be improperly labeled/classified);
selecting at least one unknown component from the plurality of unknown components in the material stream based at least partially on the training reward associated with the unknown components (Salman teaches using computer vision and machine learning for generating scores (training rewards) that indicate how accurate the labeling of an object is from a plurality of objects recognized and ranks the objects based on which ones have the highest uncertainty that would benefit from use-annotated ground truth labeling for more efficient re-training of the ML model; one of ordinary skill in the art, when combining Wu and Salman applies this concept of “active learning” and ranking relevant items for re-labeling with ground truth labels from Salman, to the waste material stream in Wu, to decide which waste items were most likely to be mis-labeled by the CNN and requires further analysis by a user); analyzing the isolated at least one unknown component for determining a ground truth label thereof, , wherein the determined ground truth label of the isolated at least one unknown component is added to a training database; and training an incremental version of the machine learning model using the determined ground truth label of the physically isolated at least one unknown component (Salman teaches using computer vision and machine learning for generating scores (training rewards) that indicate how accurate the labeling of an object is from a plurality of objects recognized and ranks the objects based on which ones have the highest uncertainty that would benefit from user-annotated ground truth labeling for more efficient re-training of the ML model; one of ordinary skill in the art, when combining Wu and Salman applies this concept of “active learning” and ranking relevant items for re-labeling with ground truth labels from Salman, to the segmented physically isolated waste item in the separate bins 9 shown in WU in FIG. 1 that are isolated from the rest of waste stream, based on the scores (training reward); so instead of doing what Wu teaches which is physically isolate the waste items based on the prediction labels and prediction probabilities, one of ordinary skill in the art combines Wu and Salman to sort the specific waste items based on the which waste item has the highest level of uncertainty; Wu teaches the ML process of active learning applied to the mechanical process of Wu’s waste sorting).
Wu, in view of Salman, fails to teach
determining a value associated with each unknown component that is indicative of a difficulty for performing physical isolation of the unknown component from the material stream by a separator unit; and selecting at least one unknown component from a plurality of unknown components in the material stream based at least partially on the value indicative of the difficulty for performing physical isolation.
Lukka teaches
determining a value associated with each unknown component that is indicative of a difficulty for performing physical isolation of the unknown component from the material stream by a separator unit; and selecting at least one unknown component from a plurality of unknown components in the material stream based at least partially on the value indicative of the difficulty for performing physical isolation (Lukka, page 6; FIG. 5: “Figure 5 shows a block diagram of the overall data processing steps in the ZenRobotics Recycler. Two important machine learning problems involved in these steps are the material classification of the identified objects and choosing how to grasp irregular objects. Material classification is a supervised learning problem and the grasping task is a reinforcement learning problem [3, p. 3], while both involve sensor fusion and computer vision. We continuously enhance the performance of our system by manually annotating mistakes done by the system. These include (1) classification errors causing contaminants in the recovered fractions or missing valuable objects, (2) failed grasping causing either missing an object or causing contaminants, and (3) collisions while throwing the grasped objects. Most often, the manual annotation can be done based on collected offline data, but in some cases the materials are either unrecognizable based on the sensor images or are too rare that sampling the waste stream is insufficient. In those cases, we also involve humans sorting actual waste into desired fractions which are fed one at a time through the system to provide training samples of known materials”;
PNG
media_image14.png
324
674
media_image14.png
Greyscale
).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the method for training a machine learning model configured to perform characterization of components in a heterogeneous material stream with a plurality of unknown components, as taught by Wu, in view of Salman, to include determining a value associated with each unknown component that is indicative of a difficulty for performing physical isolation of the unknown component from the material stream by a separator unit; and selecting at least one unknown component from a plurality of unknown components in the material stream based at least partially on the value indicative of the difficulty for performing physical isolation, as taught by Lukka.
The suggestion/motivation for doing so would have been to more accurately and properly classify and isolate waste items that are unknown from a stream of a mix of waste items on a conveyor belt; only isolating the waste items that have the ability to be isolated properly allows for the system to not try and isolate waste items that are too mixed up with other waste items that make them too unidentifiable to either humans and machine learning models using computer vision for labeling.
Wu, in view of Salman, and in view of Lukka, teaches selecting at least one unknown component from a plurality of unknown components in the material stream based at least partially on the training reward associated with the unknown components and the value indicative of the difficulty for performing physical isolation (one of ordinary skill in the art, when combining Lukka with Wu and Salman, applies the concept of indicating the difficulty of physical isolation of waste stream items and only selecting items the separator/grabber accurately has the ability to grab to isolate (Lukka), together with waste stream items indicated as highly uncertain/mis-labeled (Salman and Wu), so waste items indicated as very difficult to isolate by a separation unit, but are indicated as potentially mis-labeled/identified, are ignored so the automation of the system operation continues without isolating that unknown item because it will jeopardize the separator unit from working properly and potentially damage the system).
Wu, in view of Salman, and in view of Lukka, fails to teach
wherein the at least one unknown component which is isolated from the material stream is subjected to chemical analysis for determining the ground truth label at least partially based thereon.
Zhang teaches
wherein the at least one unknown component is subjected to chemical analysis for determining the ground truth label at least partially based thereon (Zhang, para. [0054]: “If the invention cannot find any related automatic identification ID (M = 0) around the object, then the object will not be automatically sorted. The invention stores the storage area cannot be identified region, this invention can use conventional methods (e.g. chemical evaluation, manual) for classification (5). For example, the article can determine its waste other agents after chemical analysis. If the waste cannot be processed by chemical analysis conclusion, then manual classification is needed. finally, by the relative characteristics of the article updated Internet of things”; the claim limitation “for determining the ground truth label” is intended use has no patentable weight; the concept of determining the ground truth label for a physically isolated waste item in a separate bin from a stream of waste items on a conveyor belt is taught by the combination of Wu and Salman above; therefore, Zhang teaches the concept of analyzing a waste item with chemical analysis for labeling/classification/identification).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the method for training a machine learning model configured to perform characterization of components in a heterogeneous material stream with a plurality of unknown components, as taught by Wu, in view of Salman, and in view of Lukka, to include the step of subjecting the at least one unknown component to chemical analysis, as taught by Zhang.
The suggestion/motivation for doing so would have been to reliably label a waste item that a human cannot reliably identify or label with their own human vision; chemical analysis provides more accurate evidence for identification when human labeling is unreliable for unknown waste items; this “greatly improves the efficiency of the garbage classification” (Zhang, para. [0022]).
Wu, in view of Salman, in view of Lukka, and in view of Zhang, teaches wherein the at least one unknown component which is isolated from the material stream is subjected to chemical analysis for determining the ground truth label at least partially based thereon (by combining Wu, Salman, and Zhang, a stream of waste items on a conveyor are analyzed by convolutional neural networks (CNNs) that identify prediction labels and prediction probabilities (Wu) that are then sorted to be physically isolated from the stream (mechanically by Wu) based on the “active learning” process of determining which items have the highest level of uncertainty with their labeling for targeted ground-truth labeling by a user that is then used to re-train the machine learning mode (Salman); in the case where the highest ranked uncertainty waste item in terms of labeling sorted into its own waste bin cannot be identified by a human for ground truth labeling, the isolated waste item is subjected to chemical analysis (Zhang) to determine a more accurate identification/label/classification).
Therefore, it would have been obvious to combine Wu with Salman, Lukka, and Zhang, to obtain the invention as specified in claim 1.
Regarding claim 4, Wu, in view of Salman, in view of Lukka, and in view of Zhang, teaches the method according to claim 1, wherein the separation unit has at least a first separation device and a second separation device, wherein one of the first or second separation device is selected for physical isolation of the selected at least one unknown component based on one or more features of the unknown components extracted from the imaging of the material stream (Wu, para. [0045]; para. [0032]; FIG. 1; FIG. 4-5; see rejection of claim 1 above; FIG. 4 and FIG. 5 show the two subunits that open and close to sort waste items into a multi-chamber waste basket based on the type of waste item identified by the convolutional neural network (CNN) that extracts and analyzes features of the waste item to generate a prediction label for the item; the label is used to determine which part of the waste bin is chosen for the waste item; the claim term “one or more features” refers back to the claim limitation in claim 1 reciting “machine learning model which is configured to receive as input the imaging of the material stream and/or one or more features of the unknown components extracted from the imaging of the material stream” which Examiner has interpreted “and/or” as “or” and therefore chose the input of the ML model as the images which extracts features using a CNN and not inputting features into the ML model itself).
Regarding claim 12, Wu, in view of Salman, in view of Lukka, in view of Zhang, teaches the method according to claim 1, wherein a top number of unknown components are selected from the plurality of unknown components in the material stream based on the training reward associated with the unknown components (Salman, para. [0119]; para. [0142]-[0143]; FIG. 5-6; see rejection of claim 1 above; Salman teaches ranking the objects based on highest uncertainty to lowest in terms of the label/classification given to it by the machine learning model (training reward)), wherein a subset of the top number of unknown components is selected for physical isolation based on the value indicative of the difficulty for performing physical isolation by means of the separation unit (Lukka, page 6; FIG. 5; see rejection of claim 1 above; Wu, in view of Salman, in view of Lukka, and in view of Zhang, teaches ranking heterogenous waste items scanned by the computer vision machine learning system moving on a conveyor and ranking the items recognized and labeled by the ML model by level of uncertainty for potential manual ground-truth labeling by a human to more efficiently train the ML model; Lukka teaches taking into account error on selecting waste items for separation that are potentially totally unrecognizable or not able to be physically isolated from the waste stream on the conveyor).
Regarding claim 16, Wu, in view of Salman, in view of Lukka, and in view of Zhang teaches the method according to claim 1, wherein the separate accessible compartment enables a manual removal of the isolated unknown component, wherein the analysis of the at least one selected unknown component is performed at least partially by human annotation (Salman, para. [0137]; FIG. 9; para. [0153]; para. [0168; Wu, FIG. 1, trash storage bin 9; see rejection of claim 1 above; the sorted trash bin 9 shown in FIG. 1 of Wu enables a human to manually pick up the sorted piece of trash for further evaluation if necessary because it has been separated from the heterogenous waste stream on the conveyor belt; Wu, in view of Salman teaches “active learning” wherein after the waste item that has the highest uncertainty (training reward) in its labeling from the machine learning model is sorted from the rest of the stream, human annotators/oracle are then used to manually label the isolated waste item with a proper ground truth label that will help then re-train the machine learning model optimally which provides relief from traditional machine learning training for computer vision wherein human labelers must label each image to train the model).
Wu, in view of Salman, in view of Lukka, and in view of Zhang, fails to teach
wherein an indication of an internal reference of the machine learning model is provided for the isolated unknown component within the separate accessible compartment.
Salman further teaches
wherein an indication of an internal reference of the machine learning model is provided for the unknown component (Salman, para. [0277]: “As an example, a label can be a machine generated label that depends on receipt of a signal from a human input device (HID) (e.g., a mouse, a touchscreen, a stylus, a trackball, a microphone, etc.). For example, consider a workstation computing system that includes a display where a graphical user interface (GUI) can be rendered to the display with an image that may include one or more labels and/or one or more other types of indicia. In such an example, one or more features in the image may be unlabeled and identified by a human through visual inspection (e.g., optionally aided by one or more image processing techniques). In such an example, one identified, the human may utilize a HID to assign a label or labels to the one or more features. Such a process can be referred to as human labeling or human annotation where a human is in the loop (HITL). The workstation computing system may be instructed to save the label or labels with the image such that the image can be a labeled image suitable for use in training (e.g., further training of an inspection learner).”; the GUI is an indication of an internal reference to the machine learning model that labels the waste items).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the method for training a machine learning model configured to perform characterization of components in a heterogeneous material stream with a plurality of unknown components, as taught by Wu, in view of Salman, in view of Lukka, and in view of Zhang, to include providing an indication of an internal reference of the machine learning model for the unknown component, as further taught by Salman.
The suggestion/motivation for doing so would have been to allow the human to view machine learning results to determine using their own human vision whether or not the waste item sorted away from the waste stream can be accurately labeled or not for further re-training of the machine learning model.
Wu, in view of Salman, in view of Lukka, and in view of Zhang, teaches
wherein an indication of an internal reference of the machine learning model is provided for the isolated unknown component within the separate accessible compartment (The GUI taught by Salman for viewing the results of the machine learning analysis to determine if human ground truth labeling is needed for re-training the machine learning model is applied/attached to the trash storage bin 9 shown in FIG. 1 of Wu).
Therefore, it would have been obvious to combine Wu, Salman, Lukka, and Zhang, with Salman further, to obtain the invention as specified in claim 16.
Regarding claim 23, Wu, in view of Salman, in view of Lukka, and in view of Zhang, teaches the method according to claim1, wherein the one or more features relate to at least one of a volume, dimension, diameter, shape, texture, color, or eccentricity (the claim term “the one or more features” is reference to the claim term “machine learning model which is configured to receive as input the imaging of the material stream and/or one or more features of the unknown components extracted from the imaging of the material stream” by antecedent basis from independent claim 1; Examiner, in the rejection of claim 1 above has interpreted “an/or” as “or” and therefore chose to map to receiving as input the imaging of the material stream to the machine learning model; Examiner made this decision because Wu teaches using a convolutional neural network to take as input the images of the material stream and already extracts the one or more features based on the input images; therefore this claim is moot due to antecedent basis and the choice of Examiner mapping).
Regarding claim 26, Wu teaches a system for training a machine learning model which is configured to perform characterization of components in a heterogeneous material stream with a plurality of unknown components, the system including a processor, a computer readable storage medium, a sensory system, and a separator unit, wherein the computer readable storage medium has instructions stored which, when executed by the processor, result in the processor performing operations comprising: (Wu, para. [0063]; para. [0066]; para. [0031]; para. [0033]: “The term memory circuit is a subset of the term computer-readable medium. The term computer-readable medium, as used herein, does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium may therefore be considered tangible and non-transitory. Non-limiting examples of a non-transitory, tangible computer-readable medium”; “The computer programs include processor-executable instructions that are stored on at least one non-transitory, tangible computer-readable medium.”; “The identification unit 660 contains at least the first video camera 71, the second video camera 72, and the metal sensor 73. The identification unit 660 is disposed over the trash transfer mechanism 2 and under the cover 61.”; “Referring to FIG. 1, the trash storage bin 9 is disposed underneath the end of the trash tunnel, and functions as a temporary trash storage location.”).
Regarding the remaining limitations of claim 23, they recite the process of claim1, as an apparatus. Thus, the analysis in rejecting claim 1 is equally applicable to the remaining limitations of claim 23.
Claims 2, 6-7, 17, and 20-21 are rejected under 35 U.S.C. 103 as being unpatentable over Wu, in view of Salman, in view of Lukka, in view of Zhang, and in further view of U.S. Patent Application Publication No.: 2018/0243800 (Kumar et al.) (hereinafter Kumar).
Regarding claim 2, Wu, in view of Salman, in view of Lukka, and in view of Zhang, teaches the method according to claim 1.
Wu, in view of Salman, in view of Lukka, and in view of Zhang, fails to teach
wherein the machine learning model is configured to receive as input one or more user-defined features of the unknown components extracted from the imaging of the material stream, and wherein user-generated selection criteria for the selection of components are employed.
Kumar teaches
wherein the machine learning model is configured to receive as input one or more user-defined features of the unknown components extracted from the imaging of the material stream, and wherein user-generated selection criteria for the selection of components are employed (Kumar, para [0060]; para. [0072]; para. [0091]; FIG. 1: “For example, the vision system 110 may be configured (e.g., with a machine learning system) to collect any type of information that can be utilized within the system 100 to selectively sort the scrap pieces 101 as a function of a set of one or more (user-defined) physical characteristics, including, but not limited to, color, size, shape, uniformity, composition, and/or manufacturing type of the scrap pieces 101.”; “Depending upon the variety of classifications of scrap pieces desired, multiple classifications may be mapped to a single sorting device and associated sorting bin. In other words, there need not be a one-to-one correlation between classifications and sorting bins. For example, it may be desired by the user to sort certain classifications of materials (e.g., aluminum alloys, cast materials, wrought materials, paper, plastic, etc.) into the same sorting bin. To accomplish this sort, when a scrap piece 101 is classified as falling into a predetermined grouping of classifications, the same sorting device may be activated to sort these into the same sorting bin. Such combination sorting may be applied to produce any desired combination of sorted scrap pieces. The mapping of classifications may be programmed by the user (e.g., using the sorting algorithm (e.g., see FIGS. 7, 22, and 35) operated by the computer system 107) to produce such desired combinations. Additionally, the classifications of scrap pieces are user-definable, and not limited to any particular known classifications of scrap pieces.”; “In accordance with certain embodiments of the present disclosure, such libraries may be inputted into the vision system and then the user of the system 100, 300 may be able to adjust certain ones of the parameters in order to adjust an operation of the system 100, 300 (for example, adjusting the threshold effectiveness of how well the vision system recognizes a particular material from a heterogeneous mix of materials).”;
PNG
media_image15.png
546
736
media_image15.png
Greyscale
).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the machine learning model, as taught by Wu, in view of Salman, in view of Lukka, and in view of Zhang, to be configured to receive as input one or more user-defined features of the unknown components extracted from the imaging of the material stream, and wherein user-generated selection criteria for the selection of components are employed, as taught by Kumar.
The suggestion/motivation for doing so would have been to allow the user to more narrowly define the categories/labels/classifications of a waste sorting machine learning model so that when a specific type of waste stream exists (ex: metal scraps, wood pulp, etc.) the user can dictate to the machine learning model the broad category of waste material which will lower the chances of inaccuracies of labeling and sorting for the machine learning model.
Therefore, it would have been obvious to combine Wu, Salman, Lukka, and Zhang, with Kumar, to obtain the invention as specified in claim 2.
Regarding claim 6, Wu, in view of Salman, in view of Lukka, and in view of Zhang, teaches the method according to claim 4.
Wu, in view of Salman, in view of Lukka, and in view of Zhang, fails to teach
wherein the first separation device is used for physical isolation of smaller and/or lighter components in the material stream and the second separation device is used for physical isolation of larger and/or heavier components in the material stream.
Kumar teaches
wherein the first separation device is used for physical isolation of smaller and/or lighter components in the material stream and the second separation device is used for physical isolation of larger and/or heavier components in the material stream (Kumar, para. [0125]; para. [0069]-[0070]: “In the process block 710, a sorting device (e.g., air jet, plunger, paint brush type plunger, etc.) positioned along the singulated stream in which the scrap piece is travelling, and associated with the determined material classification (which may be performed via a vision system 110, 310 in combination with the XRF system 120, 320), is identified along with the time period during which the scrap piece will pass by this sorting device. In the process block 711, signals pertaining to the identified time period are sent to the particular sorting device (or to a device controlling the sorting device, (e.g., see the automation control system 108 of FIG. 1)).”; “For example, a sorting device may utilize air jets, with each of the air jets assigned to one or more of the classifications. When one of the air jets (e.g., 127) receives a signal from the automation control system 108, that air jet emits a stream of air that causes a scrap piece 101 to be ejected from the conveyor belt 103 into a sorting bin (e.g., 137) corresponding to that air jet. High speed air valves from Mac Industries may be used, for example, to supply the air jets with an appropriate air pressure configured to eject the scrap pieces 101 from the conveyor belt 103. Although the example illustrated in FIG. 1 uses air jets to eject scrap pieces, other mechanisms may be used to eject the scrap pieces, such as robotically removing the scrap pieces from the conveyor belt, pushing the scrap pieces from the conveyor belt (e.g., with paint brush type plungers), causing an opening (e.g., a trap door) in the conveyor belt 103 from which a scrap piece may drop, or using air jets to separate the scrap pieces into separate bins as they fall from the edge of the conveyor belt. As an example, FIG. 3 shows an exemplary embodiment in which plungers are utilized to eject the scrap pieces from a conveyor belt.”; Kumar teaches using multiple subunits using air jets, which work well for smaller/lighter waste items, as well as robotically sorting the waste items, such as paint brush type plungers, for larger items that an air jet could not move).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify 1) the first separation device, as taught by Wu, in view of Salman, in view of Lukka, and in view of Zhang, to be used for physical isolation of smaller and/or lighter components in the material stream, as taught by Kumar, and 2) the second separation device as taught by Wu, in view of Salman, in view of Lukka, and in view of Zhang, to be used for physical isolation of larger and/or heavier components in the material stream, as taught by Kumar.
The suggestion/motivation for doing so would have been that “if four different types of materials are to be separated, then four sorting devices may be required to push each different material into one of four bins” (Kumar, para. [0087]).
Therefore, it would have been obvious to combine Wu, Salman, Lukka, and Zhang, with Kumar, to obtain the invention as specified in claim 6.
Regarding claim 7, Wu, in view of Salman, in view of Lukka, and in view of Zhang, teaches the method according to claim 4.
Wu, in view of Salman, in view of Lukka, and in view of Zhang, fails to teach
wherein the first separation device is configured to isolate components by directing a fluid jet towards the components in order to blow the components to the separate accessible compartment, and wherein the second separation device is configured to isolate components by means of a mechanical manipulation device (Kumar, para. [0125]; para. [0069]-[0070]: “In the process block 710, a sorting device (e.g., air jet, plunger, paint brush type plunger, etc.) positioned along the singulated stream in which the scrap piece is travelling, and associated with the determined material classification (which may be performed via a vision system 110, 310 in combination with the XRF system 120, 320), is identified along with the time period during which the scrap piece will pass by this sorting device. In the process block 711, signals pertaining to the identified time period are sent to the particular sorting device (or to a device controlling the sorting device, (e.g., see the automation control system 108 of FIG. 1)).”; “For example, a sorting device may utilize air jets, with each of the air jets assigned to one or more of the classifications. When one of the air jets (e.g., 127) receives a signal from the automation control system 108, that air jet emits a stream of air that causes a scrap piece 101 to be ejected from the conveyor belt 103 into a sorting bin (e.g., 137) corresponding to that air jet. High speed air valves from Mac Industries may be used, for example, to supply the air jets with an appropriate air pressure configured to eject the scrap pieces 101 from the conveyor belt 103. Although the example illustrated in FIG. 1 uses air jets to eject scrap pieces, other mechanisms may be used to eject the scrap pieces, such as robotically removing the scrap pieces from the conveyor belt, pushing the scrap pieces from the conveyor belt (e.g., with paint brush type plungers), causing an opening (e.g., a trap door) in the conveyor belt 103 from which a scrap piece may drop, or using air jets to separate the scrap pieces into separate bins as they fall from the edge of the conveyor belt. As an example, FIG. 3 shows an exemplary embodiment in which plungers are utilized to eject the scrap pieces from a conveyor belt.”; Kumar teaches using both the air jets and robotic paint brush type plungers).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify 1) the first separation device, as taught by Wu, in view of Salman, in view of Lukka, and in view of Zhang, to be configured to isolate components by directing a fluid jet towards the components in order to blow the components to the separate accessible compartment, as taught by Kumar, and 2) the second separation device, as taught by Wu, in view of Salman, in view of Lukka, and in view of Zhang, to be configured to isolate components by means of a mechanical manipulation device, as taught by Kumar.
The suggestion/motivation for doing so would have been that “if four different types of materials are to be separated, then four sorting devices may be required to push each different material into one of four bins” (Kumar, para. [0087]).
Therefore, it would have been obvious to combine Wu, Salman, Lukka, and Zhang, with Kumar, to obtain the invention as specified in claim 7.
Regarding claim 17, Wu, in view of Salman, in view of Lukka, and in view of Zhang, teaches the method according to claim 1.
Wu, in view of Salman, in view of Lukka, and in view of Zhang, fails to teach
wherein the isolated unknown component is analyzed by automatically performing a characterization of the isolated unknown component.
Kumar teaches
wherein the isolated unknown component is analyzed by automatically performing a characterization of the isolated unknown component (Kumar, para. [0149]-[0152]; para. [0069]; FIG. 1: “As has been previously explained, x-ray fluorescence (“XRF”) is the emission of characteristic “secondary” (or fluorescent) x-rays from a material that has been excited by irradiating it with x-rays or gamma rays. XRF is based on the principal that individual atoms, when excited by an external energy source, emit x-ray photons of a characteristic energy or wavelength … With XRF, quantitative analysis is possible as the net peak area for an element in an acquired XRF spectrum is directly proportional to the mass of the sample. For example, for an acquired XRF spectrum from a sample (e.g., a scrap piece), if an aluminum peak having an area of 10,000 counts represents 10 grams of aluminum, then a peak of 20,000 counts would represent 20 grams of aluminum, and a peak of 30,000 counts would represent 30 grams of aluminum. This linear methodology can be used to quantitatively determine both the type and quantity of various elements in a sample.”; “As will be described herein with respect to FIGS. 9-13, in accordance with certain embodiments of the present disclosure, the x-ray source may include an in-line x-ray fluorescence (“IL-XRF”) tube. Such an IL-XRF tube may include a separate x-ray source dedicated for one or more of the singulated streams of conveyed scrap pieces. Likewise, one or more XRF detectors may be implemented to detect fluoresced x-rays from scrap pieces within each of the singulated streams.”;
PNG
media_image16.png
548
736
media_image16.png
Greyscale
).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the method for training a machine learning model configured to wherein the isolated unknown component is analyzed by automatically performing a characterization of the isolated unknown component, as taught by Wu, in view of Salman, in view of Lukka, and in view of Zhang, to include wherein the isolated unknown component is analyzed by automatically performing a characterization of the isolated unknown component within the separate accessible compartment for determining the ground truth label based on the characterization, as taught by Kumar.
The suggestion/motivation for doing so would have been that “there is a need for cost-effective sorting platforms that can identify, analyze, and separate mixed industrial or municipal waste streams with high throughput to economically generate higher quality feedstocks (which may also include lower levels of trace contaminants) for subsequent processing; typically, material recovery facilities are either unable to discriminate between many materials, which limits the scrap to lower quality and lower value markets, or too slow, labor intensive, and inefficient, which limits the amount of material that can be economically recycled or recovered” (Kumar para. [0005]).
Wu, in view of Salman, in view of Lukka, in view of Zhang, and in view of Kumar teaches wherein the isolated unknown component is analyzed by automatically performing a characterization of the isolated unknown component within the separate accessible compartment for determining the ground truth label based on the characterization (One of ordinary skill in art takes the x-ray fluorescence system (XRF) (Kumar) and analyzes the isolated waste item in the trash storage bin 9 in Wu after the sorting process based on isolating the highest level of uncertainty labeled waste item indicated by the machine learning model (Salman) for re-training the ML model, including human assisted manual ground truth labeling; the XRF system assists in characterizing waste material; this characterization assists the ground-labeling process taught by Salman or “active learning”).
Therefore, it would have been obvious to combine Wu, Salman, Lukka, and Zhang, with Kumar, to obtain the invention as specified in claim 17.
Regarding claim 20, Wu, in view of Salman, in view of Lukka, in view of Zhang, and in view of Kumar teaches the method according to claim 17.
Wu, in view of Salman, in view of Lukka, in view of Zhang, and in view of Kumar fails to teach
wherein the characterization is by destructive measurements on isolated components for determining the ground truth label at least partially based thereon.
Kumar further teaches
wherein the characterization is by destructive measurements on isolated components for determining the ground truth label at least partially based thereon (Kumar, para. [0013]-[0014]: “While it would therefore be beneficial to be able to sort a mass or body of aluminum scrap containing a heterogeneous mixture of pieces of different alloys, to separate the different alloy compositions or at least different alloy families before re-melting for recycling, scrap pieces of different aluminum alloy compositions are not ordinarily visually distinguishable from each other … Furthermore, the presence of commingled pieces of different alloys in a body of scrap limits the ability of the scrap to be usefully recycled, unless the different alloys (or, at least, alloys belonging to different compositional families such as those designated by the Aluminum Association series 1000, 2000, 3000, etc.) can be separated prior to re-melting. This is because, when commingled scrap of plural different alloy compositions or composition families is re-melted, the resultant molten mixture contains proportions of the principle alloy and elements (or the different compositions) that are too high to satisfy the compositional limitations required in any particular commercial alloy.”; “for determining the ground truth label at least partially based thereon” is intended use and therefore has no patentable weight; the melting procedure allows for better identification of the scrap metals in waste materials).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the step of performing characterization of the isolated unknown component, as taught by Wu, in view of Salman, in view of Lukka, in view of Zhang, and in view of Kumar, to include destructive measurements on isolated components for determining the ground truth label at least partially based thereon, as further taught by Kumar.
The suggestion/motivation for doing so would have been “because, when commingled scrap of plural different alloy compositions or composition families is re-melted, the resultant molten mixture contains proportions of the principal alloy and elements (or the different compositions) that are too high to satisfy the compositional limitations required in any particular commercial alloy” (Kumar, para. [0014]); therefore, melting (destructive measurements) allow for proper recycling of the waste metals.
Therefore, it would have been obvious to combine Wu, Salman, Lukka, Zhang, and Kumar, with Kumar further, to obtain the invention as specified in claim 20.
Regarding claim 21, Wu, in view of Salman, in view of Lukka, in view of Zhang, and in view of Kumar, teaches the method according to claim 17, wherein characterization is by at least one of: an energy or wavelength dispersive X-ray fluorescence spectrometry, fire assay, inductively coupled plasma optical emission spectrometry, inductively coupled plasma atomic emission spectroscopy, inductively coupled plasma mass spectrometry, laser-induced breakdown spectroscopy, infrared spectroscopy, hyperspectral spectroscopy, x-ray diffraction analysis, scanning electron microscopy, nuclear magnetic resonance, Raman spectroscopy (Kumar, para. [0149]-[0152]; para. [0069]; FIG. 9; FIG. 1; see rejection of claim 17 above; Kumar teaches X-ray fluorescence spectrometry).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Wu, in view of Salman, in view of Lukka, in view of Zhang, in view of Kumar, and in further view of U.S. Patent Application Publication No.: 2021/0287357 (Horowitz et al.) (hereinafter Horowitz).
Regarding claim 9, Wu, in view of Salman, in view of Lukka, in view of Zhang, and in view of Kumar, teaches the method according to claim 7.
Wu, in view of Salman, in view of Lukka, in view of Zhang, and in view of Kumar, fails to teach
wherein for each unknown component in the material stream data indicative of a mass is calculated.
Horowitz teaches
wherein for each unknown component in the material stream data indicative of a mass is calculated (Horowitz, para. [0041]: “FIG. 5 is a diagram illustrating generally at 500 an example display of material characterization data at a user interface device for one embodiment of the present disclosure … For example, at 510, statistics showing the cumulative total number of materials detected over a given period, and at 511 the estimated mass of those materials, may calculated by the object characterization processor 160. Estimated mass may be calculated, for example, by correlating the number of objects of each material type observed against a look-up table stored in the memory of object characterization processor 160 that provides a standard mass estimate for an item of that material. Moreover, the memory of object characterization processor 160 may be updated to include current market values for one or more of the materials detected so that the market value of that material collected by the facility and detected by the optical material characterization system 10 may be provided. In some embodiments, the displayed may be filtered to provide such statistics for specific materials … For example, graphical data (shown at 515) may incrementally display the number of times a specific material has been detected over a previous time period. The object characterization processor 160 may also process the material characterization data to display statistics on hazardous material collected (shown at 520) or materials considered to be contaminant materials (shown at 525). In some embodiments, a live video feed of image frames from the imaging device 162 may be displayed (shown at 530).”;
PNG
media_image17.png
430
596
media_image17.png
Greyscale
).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the method for training a machine learning model configured to perform characterization of components in a heterogeneous material stream with a plurality of unknown components, as taught by Wu, in view of Salman, in view of Lukka, in view of Zhang, and in view of Kumar, to calculate data indicative of mass for each unknown component in the material stream data, as taught by Horowitz.
The suggestion/motivation for doing so would have been that “often a conveyor belt will carry an unsorted mixture of various objects and materials; in some instances, like within recycling and waste management facilities for example, some of the objects may be considered desirable (e.g. valuable) materials while others may be considered undesirable contaminants for example, the random and unsorted contents of a collection truck may be unloaded at the facility onto a conveyor belt; at that point the facility operator does not really know specific details about the types of material that have just been received; the facility operator may therefore wish to be able to identify what material is being carried on the conveyor belt in order to gather data about the type of material being conveyed, and/or to identify target material for removal from the conveyor belts such as by a sorting robot” (Horowitz, para. [0002]).
Therefore, it would have been obvious to combine Wu, Salman, Lukka, Zhang, and Kumar, with Horowitz, to obtain the invention as specified in claim 9.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Wu, in view of Salman, in view of Lukka, in view of Zhang, in view of Kumar, and in further view of Japanese Patent Application Publication No.: JPH 11226508 A (Takashi et al.) (hereinafter Takashi).
Regarding claim 10, Wu, in view of Salman, in view of Lukka, in view of Zhang, and in view of Kumar, teaches the method according to claim 7.
Wu, in view of Salman, in view of Lukka, in view of Zhang, and in view of Kumar, fails to teach
wherein a resulting force induced by the fluid jet is adjusted based on the mass of the selected at least one unknown component.
Takashi teaches
wherein a resulting force induced by the fluid jet is adjusted based on the mass of the selected at least one unknown component (Takashi, page 2, para. 11-12: “In the above configuration, when the sorted waste is put into the supply chute 16, it is vibrated by the vibrator 15 and sent to the guide groove 14, and is conveyed to the outlet side along the guide groove 14 while being appropriately dispersed. Is done. During the transportation, the separation air A is injected from the wind box 21 through the air injection holes 26 toward the sorted waste, and a lightweight PET bottle, an aluminum can, or the like, climbs over the inclined surface 14b and the left guide groove. 14 or directly to the light-weight outlet 18, and is separated and discharged from the light-weight outlet 18 to the light-weight tray 32. On the other hand, a heavy glass bottle or the like does not climb over the inclined surface 14a even when the separation air A is hit, and the guide groove 14 and is discharged to the heavy load tray 31 from the heavy load discharge port 17. In the above embodiment, by adjusting the pressure of the separation air, it is possible to separate only the glass bottle, which is the maximum weight, or the glass bottle and the steel can as the heavy weight.”).
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the method for training a machine learning model configured to perform characterization of components in a heterogeneous material stream with a plurality of unknown components, as taught by Wu, in view of Salman, in view of Zhang, and in view of Kumar, to include adjusting a resulting force induced by the fluid jet based on the mass of the selected at least one unknown component, as taught by Takashi.
Therefore, it would have been obvious to combine Wu, Salman, Lukka, Zhang, and Kumar, with Takashi, to obtain the invention as specified in claim 10.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL ADAM SHARIFF whose telephone number is 571-272-9741. The examiner can normally be reached M-F 8:30-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached on 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL ADAM SHARIFF/
Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672