FINAL REJECTION
Applicants arguments filed 7/29/2026 have been fully considered but they are not persuasive for reasons as detailed below.
The prior art rejections are maintained or modified as follows:
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2-7, 10-14 are rejected under 35 U.S.C. 103 as being unpatentable over Havir et al. (US11636602) (“Havir”) in view of Higgins et al (US20210374467) (“Higgins”).
Referring to claims 2-3. Havir discloses a “Pre-labeling For Semantic Segmentation Tasks”. Havir further discloses a method comprising displaying, at an electronic device, an image of a plurality of items and that an image includes pedestrians, roads, skies, buildings, and other objects or entities captured by a camera (¶0041). Likewise, Havir disclose a visual representation of one or more item categories displayed concurrently with the image, comprising user-interface elements presenting “a list of classes”, each element including a name of a corresponding class and a color (¶¶0065-0066). Additionally, Havir discloses receiving an input, at the electronic device, the input selecting a portion of the image of the plurality of items and selecting one of the one or more item categories. Havir discloses a drawing tool that allows the user to select pixels or regions of pixels in the image, which additionally allows the user to assign, change, or remove a label for the selected pixels or regions (¶0067), and further discloses that the user interacts with the class list and display control elements to adjust, confirm, and/or change the labels for pixels in the image (¶0072). Furthermore, Havir discloses that in response to the input, associating an item of the selected portion of the image with the selection of the one or more item categories, in that user input is used to update the labels of the selected pixel regions in the semantic segmentation of the image (¶¶0063-0064, 0067). Havir does not disclose training one or more machine learning models based on the associating of the selected portion of the image with the selection of the one or more categories. Higgins discloses a “Correlated Slice and View Image Annotation For Machine Learning” system (should be noted directed to same field of endeavor as Havir) in which the system is configure for GUI based labeling of image data to generate training data for supervised machine learning (¶¶0012-0014). Higgins further discloses training one or more machine learning models based on the associating of the selected portion of the image with the selection of the one or more categories. Higgins further discloses a training module (216) executable to train the feature determination, correlated determination, and tagging modules based on labeled correlated images (222) and to perform additional training using updated label images (224) generated from user edits (¶¶0053-0054, 0119). It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine Havir’s item/category display and selection GUI with Higgins teachings of training set building and model retraining architecture, as a means of optimizing speed and accuracy, while reducing the burden of manually labeling image data for supervised learning, while also allowing the labeling model to retrain on it’s own corrections.
Referring to claim 4. Havir discloses displaying a suggested one of the one or more item categories for a portion of the image, the suggested category based on one or more machine learning models, likewise, Havir discloses that the pixels in the image are initialized with prelabeled semantic segmentation, and that when a user starts a labeling task, pixels are assigned “tentative” label values representing the highest probability labels from the prelabeled semantic segmentation, displayed as color coded overlays (¶¶0042, 0069-0070).
Referring to claim 5. Havir in view of Higgins discloses the combination as set forth above as applied to claim 2. Havir in view of Higgins further disclose identifying the items as each belonging to one of the item categories to determine an identified item category for each item, and transmitting each identified item category for display in association with the corresponding item. Further, Havir discloses that predicted probabilities of labels are generated for pixels/regions in the image and displayed as color coded overlays, corresponding to those regions (¶¶0056-0057, 0070). Higgins discloses a tagging module that labels individual features and transmits the label information for display in association with the corresponding feature (¶¶0044-0045, 0049).
Referring to claims 6-7. Havir in view of Higgins discloses the combination as set forth above as applied to claim 2. Havir does note explicitly disclose forming labeled images and appending the labeled images to a training set for training one or more machine learning models. Higgins discloses forming labeled images of one of the plurality of items based on the associating, and appending the labeled images to a training data set for training one or more machine learning models having images of items as inputs and corresponding item categories as outputs (¶¶0035-0037, 0053) and further that after the appending, training the models using the labeled images of the training data set (¶¶0053-0054). It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Higgins into the system of Havir as it would improve the accuracy of the labeling model, enable more precise labeling and retraining, and improve the speed of labeling image data.
Referring to claim 10. Havir in view of Higgins discloses the combination as set forth above as applied to claim 2. Havir further discloses designation portions of the image corresponding to items as selectable portions, wherein displaying includes displaying the selectable portions. Likewise, Havir’s drawing tool allows the user to select pixels or regions of pixels in the image independent of predefined object boundaries (¶0067).
Referring to claims 11-12. Havir in view of Higgins discloses the combination as set forth above as applied to claim 2. Havir discloses displaying an overlay over the image to highlight selectable items, wherein the overlay includes an indication of a suggested item category for the highlighted item and further tentative labels displayed as color coded overlays representing various features (¶0069-0070).
Referring to claims 13. Havir in view of Higgins discloses the combination as set forth above as applied to claim 2. Havir further discloses a system comprising one or more processors (CPU 302) and memory (304) communicatively coupled thereto, with software (data labeling 104) stored in memory that includes instructions for displaying, receiving and associating (¶¶0030-0038).
Referring to claim 14. Havir in view of Higgins discloses the combination as set forth above as applied to claim 2. Higgins discloses that computer storage media including RAM, ROM, EEPROM, Flash Memory, CD-ROM and DVD and does not include communications media (non-transitory) (¶0057). It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Havir to store Havir’s labeling application instructions on a non-transitory storage medium of the type described in Higgins, as this would provide an efficient and cost-friendly means of storing executable instructions.
Claim(s) 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over Havir in view of Higgins, and further in view of Decker et al (US12293561) (“Decker”).
Referring to claims 8-9. Havir in view of Higgins discloses the combination as set forth above as applied to claim 2. Havir doesn’t disclose wherein the items comprise one or more of recyclable items or non-recyclable items or wherein the categories comprise glass, plastic, paper, or metal. Decker discloses a “Device and Method For Automatically Identifying and Categorizing Waste, and Directing A Desired User Action”. See Figs. 1-14 and respective portions of the specification. Decker further discloses a method for waste management wherein a machine learning model is trained to classify items depicted in images according to categories including recyclable type, trash type, and compost type (¶0055), and wherein the item categories into which such items are classified include metals, glass, cardboard, paper, food, plastic (¶0068). It should be noted as disclosed by Decker that it is generally known and understood in the sorting arts that glass, paper, plastic, metal are standard categories for classifying recyclable items. It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combined Havir and Higgins item labeling system such that items that comprise recyclable or non-recyclable items, and that item categories including glass, plastic, paper, and/or metal as taught by Decker are part of the recyclable categories, as this would allow for item labeling to this specific application of recyclable and non-recyclable items such that a more precise labeling could occur within more definite/structured categories.
Claim(s) 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Havir in view of Higgins, and further in view of Wellwood et al (US8875901)(“’Wellwood”).
Referring to claims 15, 17, 19. Havir in view of Higgins discloses the combination as set forth above as applied to claim 2. Havir and Higgins combination does not disclose that the plurality of items comprise a plurality of separable and sortable articles. Wellwood discloses “Sorting mined material on the basis of two or more properties of the material”. See Figs. 1-2 and respective portions of the specification. Wellwood further discloses a feed system delivering individual fragments of material, each of which is sensed, classified, and separated from the others on a fragment by fragment basis (See Col. 2 L. 30 – Col. 4 L. 35). It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combined system of Havir and Higgins such that the depicted items correspond to individually, classifiable, and physically separable fragments as taught by Wellwood, because doing so would allow for the image based item classification of Havir/Higgins to a known application in the sorting art, wherein a stream of labeled/identified articles would be subject to downstream separating/processing.
Referring to claim 16, 18, 20. Havir in view of Higgins discloses the combination as set forth above as applied to claim 2. Havir doesn’t disclose causing movement of items based on the selected category, in accordance with the input selecting a portion of the image and selecting one of the item categories. Wellwood discloses that based on classification of a fragment from sensed property data, a processor causes a sorter to separate the fragment, and that fragments are selectively deflected by air or water jets into different collection bins according to their classification (See at least Col. 4 L. 35 – Col. 7 L. 30). It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combined Havir/Higgins system to include a classification triggered sort actuator as taught by Wellwood, so that once an item is associated with a selected category via the labeling system, the system triggers a corresponding physical movement action for that item, that outputs it to its selected/defined classification area, as this would allow an application for a machine learning, labeling system, to discern the item and category and sort the item to its corresponding location.
Response to Arguments
Applicant’s arguments filed 7/29/2026 have been fully considered but they are not persuasive, or are moot in view of the new grounds of rejection set forth above.
Applicant argues that the combination of Higgins and Havir fails to teach or suggest “displaying, at an electronic device, an image of a plurality of items”, contending that Higgins is concerned with “features” of a series of microscopy images of a single semiconductor chip rather than a plurality of distinct items, and that prior art rejection’s interpretation equating “feature” with “items” is unreasonably broad. This argument is moot in view of the rejection set forth above, which no longer relies on Higgins to teach this limitation. Havir independently and explicitly discloses an image depicting a plurality of distinct items “image 402 includes pedestrians, roads, skies, buildings, trees, cars, and/or other objects or entities” (¶0041). Thus, Havir’s disclosure of an image depicting pedestrians, cars, buildings, trees, etc. discloses “an image of a plurality of items” as claimed. Applicant further argues that Havir relates to “semantic labeling within pictures” and would at best add semantic labeling to Higgins image process. However, that is no longer the structure of the rejection, Havir is relied upon simply to supply “the plurality of items” category display, portion selection, and category association limitations, while Higgins is combined to address the correlated image training data set building limitations.
Conclusion
Accordingly, THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TERRELL HOWARD MATTHEWS whose telephone number is (571)272-5929. The examiner can normally be reached Monday thru Friday; 8:00 AM - 4:30 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael McCullough can be reached at (571)272-7805. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TERRELL H MATTHEWS/Primary Examiner, Art Unit 3653
