DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 3 and 8 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim(s) 3 and 8 recite “a threshold” twice, which causes a lack of antecedent basis. It is unclear if the claim is referring to the same or different thresholds. It is suggested to amend the claims to recite "a first threshold" and "a second threshold" or clarify if same threshold (i.e. “a threshold” and “the threshold”).
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2, 4-7, and 9-11 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more. Claims 1, 6, and 11 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to an abstract idea without significantly more. The claims recite generating reference source data by associating product attributes with hierarchies based on a variance relationship, which constitutes a mathematical concept (variance calculations) and mental process (organizing data into categories). The additional elements of a storage medium, processor/memory, and setting data for a zero-shot image classifier are recited at a high level of generality and amount to no more than mere instructions to apply the exception using generic computer components. The claims do not include additional elements sufficient to amount to significantly more than the judicial exception because the additional elements, when considered individually and in combination, do not add anything that is not already well-understood, routine, and conventional.
STEP 1: Statutory Category? YES.
Claims 1-5 recite(s) a CRM (manufacture).
Claims 6-10 recite(s) a process.
Claim 11 recite(s) a machine.
STEP 2A, PRONG 1: Judicial Exception? YES.
The independent claims recite:
"acquiring product data" - data gathering
"generating reference source data in which attributes of products are associated with each of a plurality of hierarchies based on a variance relationship" - Mathematical concept (variance calculation) and Mental process (organizing/associating data)"
“setting the generated reference source data as data to be referred to by a zero-shot image classifier" - data assignment/configuration
Abstract Ideas Identified:
Mathematical concept: "variance relationship" involves mathematical calculations
Mental process: Organizing product attributes into hierarchical categories based on relationships could be performed mentally with pen and paper
Certain methods of organizing human activity: Organizing commercial product data
STEP 2A, PRONG 2: Practical Application? NO.
This judicial exception is not integrated into a practical application because the additional elements:
"non-transitory computer-readable storage medium" - generic computer component
"one or more memories" and "one or more processors" - generic computer components
"zero-shot image classifier" - generic recitation of Al tool without specific integration
Analysis under MPEP 2106.05 (a)-(h) considerations:
Improvements to technology? No - the claim does not improve the functioning of the zero-shot classifier itself; it merely provides input data. MPEP 2106.05(a).
Particular machine? No - generic storage medium.
Transformation? No - data manipulation only.
Other meaningful limitations? No.
Mere instructions to apply? Yes - using generic computer to organize data.
Insignificant extra-solution activity? Yes - acquiring data is mere data gathering. MPEP 2106.05(g).
Field of use? Yes - limiting to "zero-shot image classifier" is field of use limitation. MPEP 2106.05(h).
Conclusion Prong 2: Additional elements do not integrate the abstract idea into a practical
application.
STEP 2B: Inventive Concept? NO.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the following elements are well-understood, routine, conventional) WURC:
Generic computer storage medium – WURC
Acquiring product data -WURC data gathering
Setting data for classifier reference - WURC computer function
No inventive concept beyond the abstract idea.
Dependent claims analysis:
Claims 2, 7:
Additional Limitations: Adding hierarchy based on price attributes, assigning similar products.
Abstract idea? Yes, mental attributes process/math.
Practical application? No.
Inventive Concept? Ineligible – Adds abstract mental steps.
Claims 3,8:
Additional Limitations: Determining terminus based on variance threshold.
Abstract idea? Yes, mathematical calculation
Practical application? Yes, MPEP 2106.05(a), “improvements to the functioning of a computer” determines if an abstract idea is integrated into a practical application. Claims 3 and 8 recite a specific rule applied to the data structure that dictates how the computer processes the query. By cutting off branches of the hierarchy, the invention reduces the number of inference steps the zero-shot classifier must perform. See the specification’s explanation in ¶¶127-128 describing reduction in processing cost.
Inventive Concept? Eligible – improvements to the functioning of a computer.
Claims 4,9:
Additional Limitations: Setting to class caption for text encoder.
Abstract idea? No, but mere data output.
Practical application? No, insignificant extra-solution activity.
Inventive Concept? Ineligible – insignificant extra-solution activity.
Claims 5, 10:
Additional Limitations: Narrowing down attributes using classifier.
Abstract idea? Describes intended use.
Practical application? Field of use limitation.
Inventive Concept? Ineligible – field of use limitation.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 5-6, 10-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Al-Halah et al. ("How to Transfer? Zero-Shot Object Recognition via Hierarchical Transfer of Semantic Attributes" – hereinafter “Al-Halah”) in view of Tong et al. (Hierarchical Disentanglement of Discriminative Latent Features for Zero-shot Learning – hereinafter “Tong”) and Klasson et al. (A Hierarchical Grocery Store Image Dataset with Visual and Semantic Labels – hereinafter “Klasson”).
Claims 1, 6, and 11.
Al-Halah discloses a non-transitory computer-readable storage medium storing a data generation program that causes at least one computer to execute a process (), the process comprising:
acquiring (Al-Halah teaches acquiring data for object recognition , utilizing datasets like " aPascal, Animals with Attributes and CUB-200-2011 Birds" [Al-Halah, Abstract, § 4.1]. Klassen discloses acquiring "product data" specifically in a retail environment [§ 3].);
generating reference source data in which attributes of products are associated with each of a plurality of hierarchies (Al-Halah teaches generating a hierarchical model where attributes are associated with different levels (hierarchies) to capture intra-attribute variations. “Such information provides important cues on the intra-attribute variations. We propose to capture these variations in a hierarchical model” [Al-Halah, Abstract, § 3.1].)
setting the generated reference source data as data to be referred to by a zero-shot image classifier (Al-Halah teaches using the hierarchical attribute data as reference data for a "Zero-Shot Object Recognition" classifier to recognize unseen classes [Al-Halah, § 4].).
Al-Halah discloses all of the subject matter as described above except for specifically teaching “product.” However, Klasson in the same field of endeavor teaches “product” data (Klasson describes a "Grocery Store" dataset containing "natural images” of grocery items such as fruits, vegetables, and packages [Klasson, Abstract].).
It would be obvious to apply the zero-shot recognition framework of Al-Halah to the retail "product data" of Klasson to create a scalable automated checkout or inventory system that can recognize newly introduced products without requiring retraining for every new stock keeping unit (SKU).
Al-Halah discloses all of the subject matter as described above except for specifically teaching “based on a variance relationship of attributes of products included in the acquired product data.” However, Tong in the same field of endeavor teaches “based on a variance relationship of attributes of products included in the acquired product data” (Tong teaches calculating the variance relationship to determine discriminative power, noting that low variance features hurt discrimination. “Figure 1(c) shows … We observed that not all dimensions of features contribute to the discrimination … Features with low variances even hurt the discrimination.” [Tong, §1, p. 11467, col. 2] “The variance σi is calculated for the i-th column” [Tong, p. 11468, Figure 1’s description]).
It would have been obvious at the time of filling for one of ordinary skill in the art to use Tong’s variance analysis to automate the hierarchy generation in Al-Halah, ensuring that attributes are only assigned to hierarchy level where the show high variance and discriminative value, rather than relying on static, potentially insufficient manual taxonomy.
Claims 5 and 10.
The combination of Al-Halah, Klasson, and Tong discloses the non-transitory computer-readable storage medium according to claim 1, wherein the zero-shot image classifier narrows down, by inputting an acquired video (Klasson p. 2, col. 2, discloses “video recordings”) to the zero-shot image classifier that refers to reference source data in which attributes of objects are associated with each of a plurality of hierarchies, attributes of an object included in the video among attributes of objects of a first hierarchy (Al-Halah notes that "predictions towards
the leaf level (lower hierarchy) ... have higher precision" but depend on the parent nodes [Al-Halah, §§ 3.2-3.3].), identifies attributes of objects of a second hierarchy under the first hierarchy by using the attributes of the object obtained by the narrowing down (Al-Halah teaches this hierarchical inference flow: identifying broad attributes (root/first hierarchy) allows the system to narrow down and identify specific attributes in the "second hierarchy" (leaves/children) [Al-Halah, §§ 3.2-3.3].), and specifies, by inputting the acquired video to the zero-shot image classifier, an attribute of the object included in the video among the attributes of the objects of the second hierarchy (Al-Halah discloses specifying the final attributes at the specific level (second hierarchy) to achieve zero-shot recognition of the object [Al-Halah,
Section 4]. “HAT [Hierarchical Attribute Transfer model] enables us to carry out zero-shot recognition even if the attribute description of the novel class is unknown. To do that, we again leverage the hierarchy and transfer the attribute description of the parent node to the
novel class.”).
Claims 2-4 and 7-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Al-Halah, Tong, Klasson, and further in view of Fisher et al. (US 2023/0079018 A1 – hereinafter “Fisher”).
Claims 2 and 7.
The combination of Al-Halah, Klasson, and Tong discloses the non-transitory computer-readable storage medium according to claim 1, wherein the generating includes: adding a second hierarchy that has attributes related to (Klasson discloses that the products are grocery items [Klasson, Title & Abstract].); and assigning items of products similar to elements of the second hierarchy among items of products to be sold in a store under the elements of the second hierarchy (Klasson disclose at least two levels of hierarchy in at least Fig. 1 which shows with "coarse-grained" classes (e.g., Fruit) and "fine-grained" classes (e.g., Granny Smith Apple) where the “Apple” class comes between “Fruit” and “Granny Smith Apple.”).
Al-Halah discloses all of the subject matter as described above except for specifically teaching “prices.” However, Fisher in the same field of endeavor teaches “prices” (¶197 “The system can add heuristic methods or other signals like shelf location, product category, price, and/or item count (most of which are available in catalog data) to achieve the desired item detection and classification accuracy”).
It would be obvious to one of ordinary skill in the art at the time of filling integrate Al-Halah’s zero-shot model into Fisher’s cashier-less checkout architecture to utilize the “Camogram Data Structure 235” as a source for hierarchical semantic attributes. Leveraging pre-existing metadata such as “List Price” allow the system to capture the differences between visually similar products at specific nodes without requiring new-manually labeled training images for each new inventory item..
Claims 3 and 8.
The combination of Al-Halah, Klasson, Tong, and discloses the non-transitory computer-readable storage medium according to claim 2, wherein the generating includes: determining, as a terminus of the hierarchical structure, an element in which a variance of prices of products that belong to the elements of the first hierarchy is a threshold or less among the elements of the first hierarchy, and determining, as a terminus of the hierarchical structure, an element in which a variance of prices of products that belong to the elements of the second hierarchy is a threshold or less among the elements of the second hierarchy (Tong teaches that features with low variance "hurt discrimination" and implies they should not be used. This reads on determining a "terminus" (stopping point) for that attribute branch when variance is low (below a threshold). Modifying Klasson to stop creating finer-grained sub-classes when the attribute variance is minimal is a direct application of Tong's principle.).
Claims 4 and 9.
The non-transitory computer-readable storage medium according to claim 1, wherein the setting includes setting to a class caption to be input to a text encoder included in the zero-shot image classifier (Fisher Fig. 10 discloses a “Visual Tag Encoder 1070” as part of its “Camogram Generation Engine 192”. That can be used as “additional inputs to detect and classify items” [Fisher ¶236] within an autonomous checkout system. Tong discloses an encoder in an auto-encoder framework for zero-shot learning. “the encoder learns a mapping from an image feature to a latent factor z.” [Tong § 3, p. 11469, col. 2]. Al-Halah discloses “semantic attributes” that describe the classes [Al-Halah § 4]. Klasson discloses a “text description” [Klasson § 6].).
Conclusion
The prior art made of record but not relied, yet considered pertinent to the applicant’s disclosure, is listed on the PTO-892 form.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ross Varndell whose telephone number is (571)270-1922. The examiner can normally be reached M-F, 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached at (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ross Varndell/Primary Examiner, Art Unit 2674