DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“a control and evaluation unit that is configured to…” in claim 1.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-7 and 15-18 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by USPN 2024/0390946 to Potter et al.
With regard to claim 1, Potter discloses an image capturing device for associating a target class with an object (Fig. 2, system 200),
wherein the image capturing device has an image sensor for recording image data having the object (Fig. 2, camera 204 imaging object 208) and
a control and evaluation unit (Fig. 2, MCS 202) that is configured to evaluate and classify the image data using a method of machine learning, and to associate a target class with the image data (paragraph [0028], The MCS using machine learning models to classify each object imaged),
wherein the control and evaluation unit is further configured to use as a method of machine learning a multiclass classifier for the classification into a plurality of intermediate classes that determines respective confidence values for the association of the image data with a respective intermediate class (paragraphs [0050]-[0051] and [0062], and Figs. 6A and 6B, Objects are recognized and classified using a neural network. In this case intermediate classes are interpreted as the classifications and confidence levels shown I Fig. 6A which are described as occurring in images captured in a base sorting line before the objects are separated into 2D and 3D objects) and
subsequently to determine the target class by applying a map of confidence values in target classes (paragraphs [0050]-[0051] and [0062], and Figs. 6A and 6B, Objects recognized in Fig. 6B are also shown with corresponding confidence values and are imaged after an initial step of sorting the objects into 2D and 3D objects in order to perform a target classification).
With regard to claim 2, Potter discloses the image capturing device in accordance with claim 1, wherein the method of machine learning comprises a neural network (paragraph [0062], Neural network is listed as a machine learning algorithm).
With regard to claim 3, Potter discloses the image capturing device in accordance with claim 1, wherein none of the target classes is an intermediate class (class (paragraphs [0050]-[0051] and [0062], and Figs. 6A and 6B, The target class is considered determined after the intermediate class in a sorting process and is therefore interpreted as not the same as an intermediate class).
With regard to claim 4, Potter discloses the image capturing device in accordance with claim 1, wherein the intermediate classes are defined by at least one of the following properties of the recorded object: material; strength; and shape (paragraph [0035], A classification of the object includes “a material type, a location, a shape, a size, dimensions, a mass, a density, a priority, a condition, a form factor, a color, a polymer, and/or a brand.”).
With regard to claim 5, Potter discloses the image capturing device in accordance with claim 4, wherein the material is one of plastic, polystyrene, wood, and metal (paragraphs [0035], [0050], [0051], [0072], materials are listed as plastic, polymers and metal among others).
With regard to claim 6, Potter discloses the image capturing device in accordance with claim 4, wherein the strength is one of rigid and flexible (paragraphs [0026], [0039] and [0049], Potter discloses recognizing several different types of material which inherently have properties of rigid or flexible. It should be noted that claim 4 lists properties in the alternative: wherein the intermediate classes are defined by at least one of the following properties of the recorded object: material; strength; and shape. Accordingly claim 4 does not explicitly require strength.
With regard to claim 7, Potter discloses the image capturing device in accordance with claim 4, wherein the shape is one of parallelepiped, cylinder, torus, and irregular (paragraphs [0035]-[0036], [0062], [0064], [0073] and [0078], Potter discloses that shape is a criteria for identifying the objects to be recognized and categorized. Potter gives the example of cans and bottles (paragraphs [0049]-[0050]), which are considered cylinders as well as other objects that would be considered irregular shapes such as cardboard and plastic. It should be noted that claim 4 lists properties in the alternative: wherein the intermediate classes are defined by at least one of the following properties of the recorded object: material; strength; and shape. Accordingly claim 4 does not explicitly require shape.
With regard to claim 15, Potter discloses the image capturing device in accordance with claim 1, that is installed at a conveying device on which objects to be classified are conveyed through the field of view of the image sensor (Figs. 1 and 2, Objects are imaged as they pass through the field of view on a conveyor).
With regard to claim 16, Potter discloses the image capturing device in accordance with claim 15, wherein a plurality of camera heads are provided and the control and evaluation unit is configured to merge the recordings of the camera heads in the image data to one common image (paragraph [0072], One or more vision sensors or cameras can be configured to image the conveyor).
With regard to claim 17, the discussion of claim 1 applies. The method of claim 17 is disclosed in the operation of the device of claim 1.
Potter discloses a method of associating a target class with an object, wherein image data having the object are evaluated and classified using a method of machine learning, and a target class is associated with the image data (paragraph [0028], The MCS using machine learning models to classify each object imaged),
wherein, as a method of machine learning, a multiclass classifier for the classification into a plurality of intermediate classes is used that determines respective confidence values for the association of the image data with a respective intermediate class (paragraphs [0050]-[0051] and [0062], and Figs. 6A and 6B, Objects are recognized and classified using a neural network. In this case intermediate classes are interpreted as the classifications and confidence levels shown I Fig. 6A which are described as occurring in images captured in a base sorting line before the objects are separated into 2D and 3D objects) and
subsequently to determine the target class by applying a map of confidence values in target classes (paragraphs [0050]-[0051] and [0062], and Figs. 6A and 6B, Objects recognized in Fig. 6B are also shown with corresponding confidence values and are imaged after an initial step of sorting the objects into 2D and 3D objects in order to perform a target classification).
With regard to claim 18, the discussion of claim 2 applies.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over USPN 2024/03909046 to Potter et al.
With regard to claim 8, Potter discloses many different classifications for the trash objects detected and categorized by the machine vision system. However, Potter does not explicitly disclose wherein exactly two target classes are provided.
One of ordinary skill in the art will readily recognize that the number of target classes is set as a matter of design choice depending on the desired outcome of what is being imaged and categorized. Potter teaches for example that 2D and 3D objects are recognized and categorized in two classes. Therefore it would have been obvious to one of ordinary skill in the art before time of filing to designate exactly two target classes as a matter of design choice in order to distinguish between two classes of objects.
With regard to claim 9, the discussion of claim 8 applies. Potter does not explicitly disclose wherein the exactly two target classes comprise cardboard or not cardboard. Potter does disclose that cardboard is a class of object that is recognized and categorized (paragraphs [0026], [0030], [0039], [0049], [0054]). One of ordinary skill in the art will readily recognize that determining cardboard and non-cardboard is a useful distinction in sorting trash. Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the cardboard recognition and classification shown in Potter in order to sort cardboard.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over USPN 2024/03909046 to Potter et al. and 2007/0019865 to Owechko.
With regard to claim 10, Potter discloses the image capturing device in accordance with claim 1, but does not disclose wherein the multiclass classifier has an attention mechanism.
Owechko discloses an object recognition and classification system that uses neural networks similar to that of Potter and further teaches the use of an attention mechanism to detect objects by focusing on regions most likely to contain objects (paragraphs [0008], [0013] and [0045]). Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use an attention mechanism as taught by Owechko in combination with the object detection of Potter in order to focus on regions most likely to include objects in order to produce more efficient and accurate object recognition.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over USPN 2024/03909046 to Potter et al. and 11,693,901 to Gorban.
With regard to claim 11, Potter discloses the image capturing device in accordance with claim 1, but does not disclose wherein the multiclass classifier has a first stage that generates an embedding from the image data and a second stage that determines the intermediate classes from the features of the embedding.
Gorban teaches a CNN for performing object classification and teaches that embeddings are used to aid in the classification of the recognized objects (column 4, lines 60-67, column 5, lines 1-6 and 43-60). Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the embeddings as taught by Gorban to aid in the identification and classification of objects in the system of Potter.
Claims 12-13 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of USPNs 2024/0390946 to Potter et al. and 2020/0222949 to Murad et al.
With regard to claim 12, Potter discloses the image capturing device in accordance with claim1, but does not explicitly disclose wherein the map evaluates the intermediate classes individually with a threshold value. Murad discloses a trash sorting system that uses a neural network similar to that of Potter and teaches that confidence values are compared to thresholds to determine if the trash item identification was successful or not (paragraphs [0124]). Therefore it would have been obvious to one of ordinary skill in the art before time of filing to use the confidence scores for identifying trash items as taught by Potter in combination with the threshold comparison for the confidence values taught by Murad in order to decide on the success of a trash item identification.
With regard to claim 13, Potter discloses the image capturing device in accordance with claim 1, but does not disclose wherein the map is taught in that the multiclass classifier determines confidence values for a plurality of example images annotated by a desired target class and that map is determined in an optimization that best reproduces the associated annotated target class with a predetermination of the respective confidence values found with respect to an example image.
Murad teaches that when the image recognition confidence values do not meet the threshold, that “flagged images are then re-annotated and retrained allowing for the network 130 to learn from its low confidence predictions and improve its overall accuracy.” (paragraph [0124]). Therefore it would have been obvious to one of ordinary skill in the art before time of filing to re-annotate and retrain for identified image objects that don’t meet the confidence threshold in order to best reproduce the associated annotated class with confidence values as taught by Murad in combination with the confidence value determinations of Potter in order to improve the recognition confidence scores of the system.
With regard to claim 19, the discussion of claim 13 applies.
Allowable Subject Matter
Claim 14 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WESLEY J TUCKER whose telephone number is (571)272-7427. The examiner can normally be reached 9AM-5PM Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JOHN VILLECCO can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WESLEY J TUCKER/Primary Examiner, Art Unit 2661
