Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of claims 1 – 5, 9 and 10 in the reply filed on 18 June, 2026 is acknowledged.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 14 June, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 1, 9, and 10 are objected to because of the following informalities:
Claim 1 states “acquire class area relationship information which is information indicating a relationship of a plurality of classes set in advance…”, the claim does not define what this relationship is meant to be. This could be a relationship between different class labels, a relationship of the plurality of classes to something else unrelated to the class labels, it is unclear. As such this claim is extremely broad and applicant should consider revising and making clear what this relationship is. Claims 9 and 10 are objected to for the same reasoning.
Claim 4 is objected to because of the following informalities:
Claim 4 states “the object name relationship information indicating whether or not the name of the object assumed to actually belong to one of the plurality of classes”. This limitation is confusing as the following limitation of “the name of the object recognized by the object recognition process” seemingly refers to the claim 1 limitation of “acquire recognition results corresponding to each of the plurality of objects included in the image by performing object recognition processing on the image”, however there is no step where a name of the object is identified or recognized. While the claim 1 limitation acquires recognition results, it is not indicated these pertain to an object name.
There is nothing that gives context to the limitation of “the name of the object assumed to actually belong to one of the plurality of classes”. There is no step in any of the claims which relates to assuming an object belongs to one of the plurality of classes. The examiner understands the closest step of this to be that of the claim 1 limitation “acquire recognition results corresponding to each of the plurality of objects”, however this is seemingly being compared with the recognition results as determined from the object recognition process.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 4, 9 and 10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "the class to which each of the plurality of objects belongs" in line 24. There is insufficient antecedent basis for this limitation in the claim. Claim 1 states the limitation “a plurality of classes set in advance”, however “the class” is singular and there is no step which defines specifically “a class” for one or more objects.
Claim 9 and 10 are rejected for the same reasoning.
Claim 4 recites the limitation "the name of the object" in line 4 and 5. There is insufficient antecedent basis for this limitation in the claim. “the name of the object” of line 4 also appears to refer to a different object than “the name of the object” of line 5, however neither of these names of objects has antecedent basis.
Claim 4 and 5 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being incomplete for omitting essential steps, such omission amounting to a gap between the steps. See MPEP § 2172.01.
Claim 4 states the following: “the name of the object recognized by the object recognition process”, however there is no step in any of claims 1 – 3 which references identifying or determining a name for any of the plurality of objects during the object recognition process. The object recognition process, as defined in claim 1, consists of “acquire recognition results corresponding to each of the plurality of objects included in the image by performing object recognition processing on the image”. While the recognition results could comprise name information, there is nothing indicating a name is identified for the object in the step.
Claim 5, depending from claim 4 is also rejected under 35 U.S.C. 112(b) for the same reasoning.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of pre-AIA 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a) the invention was known or used by others in this country, or patented or described in a printed publication in this or a foreign country, before the invention thereof by the applicant for a patent.
(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on sale in this country, more than one year prior to the date of application for patent in the United States.
Claims 1 - 4, 9, and 10 are rejected under pre-AIA 35 U.S.C. 102(a)(2) as being anticipated by Akatsuka et al (U.S. Patent Publication No. 2020/0394599 A1, hereinafter “Akatsuka”).
Regarding claim 1, Akatsuka teaches an object recognition device comprising:
A memory configured to store instructions; and
A processor configured to execute the instructions to:
acquire an image including a plurality of objects (¶ 0045: More specifically, when product display shelves are imaged by the camera 21, the image acquiring unit 11 acquires an image that is imaged by the camera 21 as an image of product display shelves.);
acquire recognition results corresponding to each of the plurality of objects included in the image by performing object recognition processing on the image (¶ 0049: The detection unit 12 detects a product area image representing products from an image of product display shelves acquired by the image acquiring unit 11. More specifically, the detection unit 12, for example, recognizes each object extracted using a technique such as a known edge detection technique or the like for an image of product display shelves as a product area image representing products.);
perform processing for specifying whether or not there is a connection relationship of a plurality of object areas corresponding to the plurality of objects based on the recognition results obtained by the object recognition processing (Figure 4; ¶ 0046: The product display shelves and the products arranged on the product display shelves, as illustrated in FIG. 3 as an example, have features as described below. In other words, there is a feature in that the same products are arranged adjacently or closely (emphasis added)… The number of units arranged for each product is represented by the word "face." In other words, the number of faces of the same product is large, and there are a sufficient arrangement space in a large store);
acquire area relationship information that is information related to a relationship of each object areas identified to have the connection relationship (Figure 7 and 8; ¶ 0046: The product display shelves and the products arranged on the product display shelves, as illustrated in FIG. 3 as an example, have features as described below. In other words, there is a feature in that the same products are arranged adjacently or closely (emphasis added)… The number of units arranged for each product is represented by the word "face." In other words, the number of faces of the same product is large, and there are a sufficient arrangement space in a large store.; ¶ 0059: FIG. 7 is a diagram illustrating an example of planogram data acquired by the planogram analyzing unit 14. As illustrated in FIG. 7, the planogram data includes information of a serial number, a shelf board number, a shelf position, a product ID, the number of faces, and the number of stacking stages in association with each other.; In this embodiment, for example, the determination unit 15 generates a feature quantity relating to one product area image mp13 on the basis of the relevancy information me illustrated in FIG. 8. More specifically, the determination unit 15 generates a feature quantity ie1 as below by using values representing same/difference of product names, series/brands, and manufacturers represented in the relevancy information me as values (feature) of each item.);
acquire class relationship information that is information indicating a relationship of a plurality of classes set in advance in order to obtain the recognition results by the object recognition processing (¶ 0052: As illustrated in FIG. 5, the product image data 31 stores a plurality of pieces of product image data mb1 to mb8 representing outer views of products in association with product IDs used for identifying products.; ¶ 0053: The product recognizing unit 13 collates product image data mb stored in the product data storing unit 30 with each product area image detected by the detection unit 12 using a known collation technology and accordingly can recognize a product represented by the product area image.; Examiner’s note: As the claim does not specify what the “relationship of a plurality of classes” pertains to but is used to obtain recognition results, the examiner is interpreting this claim limitation to mean any class information from which a detected object is identified from.);
acquire a plurality of corrected recognition results by performing recognition result correction processing for correcting the recognition results obtained by the object recognition processing based on the area relationship information and the class relationship information (Figure 10; ¶ 0048: In consideration of the features of product display shelves and arranged products described above, the planogram information generating device 10 according this embodiment determines the validity of a result of recognition of products relating to one product area image using results of recognition of other product area images positioned on the vicinity of the one product area image and products relating to the other product area images.; ¶ 0068: Subsequently, the determination unit 15 generates a feature quantity relating to one product area image on the basis of relevancy information and determines validity of a result of recognition of a product relating to one product area image on the basis of the generated feature quantity.); and
acquire a final recognition result relating to the class to which each of the plurality of objects belongs, by evaluating the recognition results obtained by the object recognition processing using the plurality of the corrected recognition result (Figure 10; ¶ 0071: As illustrated in FIG. 10, the determination unit 15 inputs the generated feature quantity ie1 to a validity determiner CM1 and acquires a result r1 (validity score) of determination of validity of recognition of a product relating to one product area image. The validity determiner CM1 is a machine-learned determiner relating to determination of validity based on a predetermined feature quantity.).
Regarding claim 2, Akatsuka teaches the object recognition device according to claim 1.
Additionally, Akatsuka teaches wherein the object recognition means acquires a recognition score as the recognition result by performing the object recognition processing on the image (¶ 0092: For example, as illustrated in FIG. 15, the product recognizing unit 13 outputs three products (product names: C21, C22, and C33) in order of highest to lowest score representing the reliability of product recognition as a result of recognition of one product area image mp41. Here, it is assumed that scores representing reliability of image recognition are higher in order of a product having the product name C22, a product having the product name C33, and a product having the product name C21.)., the recognition score being a value indicating a probability of each class when each of the plurality of objects is classified into one of the plurality of classes (¶ 0092: For example, as illustrated in FIG. 15, the product recognizing unit 13 outputs three products (product names: C21, C22, and C33) in order of highest to lowest score representing the reliability of product recognition as a result of recognition of one product area image mp41.; Examiner’s note: By ordering the score of highest to lowest for representing a reliability of product recognition, the examiner understands this as ranking the scores based on a probability that the detected object matches the associated recognition result.).
Regarding claim 3, Akatsuka teaches the object recognition device according to claim 2.
Additionally, Akatsuka teaches wherein the area relationship acquisition means acquires appearance similarity information as the area relationship information corresponding to the two object areas having the connection relationship, the appearance similarity information being information relating to similarity of appearances between objects included in two object areas (¶ 0063: More specifically, between products that are recognized as products represented by the product area image mp12 of the shelf position 2 and the product area image mp13 of the shelf position 3, the product names are different, and the series/brands and the manufacturers are the same, and the determination unit 15 generates relevancy information me of which data representing such same/difference is "0, 1, 1".).
Regarding claim 4, Akatsuka teaches the object recognition device according to claim 3.
Akatsuka does not explicitly teach wherein the class relationship acquisition means acquires object name relationship information as the class relationship information, the object name relationship information indicating whether or not a name of the object assumed to actually belong to one of the plurality of classes, and the name of the object recognized by the object recognition process agree with each other (Figure 15; ¶ 0061: For example, a product recognized in relation to a product area image of a product arranged at the shelf position 1 has attributes of a product name: N11, a series/brand: B1, and a manufacturer C1.; ¶ 0063: More specifically, between products that are recognized as products represented by the product area image mp12 of the shelf position 2 and the product area image mp13 of the shelf position 3, the product names are different, and the series/brands and the manufacturers are the same, and the determination unit 15 generates relevancy information me of which data representing such same/difference is "0, 1, 1".; Examiner’s note: In light of the objection of claim 4, the examiner is interpreting this claim under broadest reasonable interpretation to mean a comparison between two detected objects (the object assumed to belong to one of the classes, and the object recognized by the object recognition process). Akatsuka teaches a comparison between the names of products that are within an close proximity to one another to determine if the products match using metrics such as name, series, and brand.).
The rejection of device claim 1 above applies mutatis mutandis to the corresponding limitations of method claim 9 while noting that the rejection above cites to both device and method disclosures.
The rejection of device claim 1 above applies mutatis mutandis to the corresponding limitations of manufacture claim 10 while noting that the rejection above cites to both device and manufacture disclosures.
For the manufacture limitations of claim 10 see Akatsuka’s teaching on:
A non-transitory computer-readable recording medium recording a program, the program causing a computer to execute (¶ 0036: device 10 is realized by the processor 1001 performing an arithmetic operation and controlling communication using the communication device 1004 and data reading and/or writing for the memory 1002 and the storage 1003 by causing the processor 1001 to read predetermined software (a program) onto hardware such as the memory 1002 or the like.; ¶ 0039: The memory 1002 is a computer-readable recording medium and, for example, may be configured by at least one of a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), and the like.)…
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
George et al (M. George "Fine-Grained Product Class Recognition for Assisted Shopping," 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), Santiago, Chile, 2015, pp. 546-554) teaches a system for product class inference from user input and performing class recognition from images of a product shelf.
Takayanagi et al (M. Takayanagi "Vision-based Scene Recognition for Product Search," 2021 7th International Conference on Electrical, Electronics and Information Engineering (ICEEIE), Malang, Indonesia, 2021, pp. 1-5, doi: 10.1109/ICEEIE52663.2021.9616880.) teaches a method for object recognition from scene images that individually identifies products, product proximity, and product labels.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW JONES whose telephone number is (703)756-4573. The examiner can normally be reached Monday - Friday 8:00-5:00 EST, off Every Other Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW B. JONES/Examiner, Art Unit 2667 /MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights