DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This action is responsive to the amendments and remarks received 30 December 2025. Claims 1, 3 - 7, 11 - 13, 15, 18 - 20, 22, 24 and 25 are currently pending.
The amendment to the claims filed on 30 December 2025 does not comply with the requirements of 37 CFR 1.121(c) because each claim has not been provided with the proper status identifier, for example, claim 11 does not include the proper status identifier. Amendments to the claims filed on or after July 30, 2003 must comply with 37 CFR 1.121(c) which states:
(c) Claims. Amendments to a claim must be made by rewriting the entire claim with all changes (e.g., additions and deletions) as indicated in this subsection, except when the claim is being canceled. Each amendment document that includes a change to an existing claim, cancellation of an existing claim or addition of a new claim, must include a complete listing of all claims ever presented, including the text of all pending and withdrawn claims, in the application. The claim listing, including the text of the claims, in the amendment document will serve to replace all prior versions of the claims, in the application. In the claim listing, the status of every claim must be indicated after its claim number by using one of the following identifiers in a parenthetical expression: (Original), (Currently amended), (Canceled), (Withdrawn), (Previously presented), (New), and (Not entered).
(1) Claim listing. All of the claims presented in a claim listing shall be presented in ascending numerical order. Consecutive claims having the same status of “canceled” or “not entered” may be aggregated into one statement (e.g., Claims 1–5 (canceled)). The claim listing shall commence on a separate sheet of the amendment document and the sheet(s) that contain the text of any part of the claims shall not contain any other part of the amendment.
(2) When claim text with markings is required. All claims being currently amended in an amendment paper shall be presented in the claim listing, indicate a status of “currently amended,” and be submitted with markings to indicate the changes that have been made relative to the immediate prior version of the claims. The text of any added subject matter must be shown by underlining the added text. The text of any deleted matter must be shown by strike-through except that double brackets placed before and after the deleted characters may be used to show deletion of five or fewer consecutive characters. The text of any deleted subject matter must be shown by being placed within double brackets if strike-through cannot be easily perceived. Only claims having the status of “currently amended,” or “withdrawn” if also being amended, shall include markings. If a withdrawn claim is currently amended, its status in the claim listing may be identified as “withdrawn—currently amended.”
(3) When claim text in clean version is required. The text of all pending claims not being currently amended shall be presented in the claim listing in clean version, i.e., without any markings in the presentation of text. The presentation of a clean version of any claim having the status of “original,” “withdrawn” or “previously presented” will constitute an assertion that it has not been changed relative to the immediate prior version, except to omit markings that may have been present in the immediate prior version of the claims of the status of “withdrawn” or “previously presented.” Any claim added by amendment must be indicated with the status of “new” and presented in clean version, i.e., without any underlining.
(4) When claim text shall not be presented; canceling a claim.
(i) No claim text shall be presented for any claim in the claim listing with the status of “canceled” or “not entered.”
(ii) Cancellation of a claim shall be effected by an instruction to cancel a particular claim number. Identifying the status of a claim in the claim listing as “canceled” will constitute an instruction to cancel the claim.
(5) Reinstatement of previously canceled claim. A claim which was previously canceled may be reinstated only by adding the claim as a “new” claim with a new claim number.
Claim Objections
Claim 1 is objected to because of the following informalities: Line 7 of claim 1 recites, in part, “each of the detected products; and” which appears to contain inconsistent claim terminology and/or a minor informality. The Examiner suggests amending the claim to --each of the detected images of products; and-- in order to maintain consistency with line 5 of claim 1 and to improve the clarity and precision of the claim. Appropriate correction is required.
Claim 1 is objected to because of the following informalities: Line 8 of claim 1 recites, in part, “trained to classify the detected products” which appears to contain inconsistent claim terminology and/or a minor informality. The Examiner suggests amending the claim to --trained to classify the detected images of products-- in order to maintain consistency with line 5 of claim 1 and to improve the clarity and precision of the claim. Appropriate correction is required.
Claim 1 is objected to because of the following informalities: Line 10 of claim 1 recites, in part, “the detected produce within the” which appears to contain a typographical error and/or a minor informality. The Examiner suggests amending the claim to --the detected product within the-- in order to improve the clarity and precision of the claim. Appropriate correction is required.
Claim 4 is objected to because of the following informalities: Lines 1 - 2 of claim 4 recite, in part, “wherein capturing the images of two or more portions of the aisle” which appears to contain a minor informality. The Examiner suggests amending the claim to --wherein capturing the images of the two or more portions of the aisle-- in order to maintain a clear link back to line 12 of claim 1 and to improve the clarity and precision of the claims. Appropriate correction is required.
The objections to claims 15, 18, 22 and 25, due to minor informalities, are hereby withdrawn in view of the amendments and remarks received 30 December 2025.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 3 - 7, 11 - 13, 15, 18 - 20, 22, 24 and 25 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "the image of the detected produce [sic] within the bounding box;" in line 10. There is insufficient antecedent basis for this limitation in the claim.
Claim 1 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention because it is unclear as to which detected product “the detected produce” [sic] recited on line 10 is referencing since line 5 of claim 1, which recites, in part, “detect images of products”, and line 8 of claim 1, which recites, in part, “the detected products”, make it clear that there are a plurality of detected products. Thus, the Examiner asserts that it is unclear as to which detected product of the detected products, if any, “the detected produce” [sic] recited on line 10 of claim 1 is referencing. Clarification and appropriate correction are required. For purposes of examination, the Examiner will treat “the detected produce” [sic] recited on line 10 of claim 1 as referencing any detected product.
Claim 11 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention because it is unclear as to which detected product “the detected product” recited on line 9 is referencing. Is it referring to “the detected produce” [sic] recited on line 10 of claim 1 or the “detected product” recited on line 4 of claim 11? Clarification and appropriate correction are required. For purposes of examination, the Examiner will treat “the detected product” recited on line 9 of claim 11 as referencing the “detected product” recited on line 4 of claim 11 and suggests amending line 9 of claim 11 to --the detected product typed as a shelf-ready package is empty or not empty.--.
Claim 13 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention because it is unclear as to which detected products “the detected products” recited on line 3 are referencing. Are they referring to the “detected products” recited on line 7 of claim 1 or the “detected products” recited on line 16 of claim 1? Additionally, it is unclear as to whether the “detected products” recited on line 7 of claim 1 and the “detected products” recited on line 16 of claim 1 are the same detected products or are different detected products. Clarification and appropriate correction are required. For purposes of examination, the Examiner will treat “the detected products” recited on line 3 of claim 13 as referencing the “detected products” recited on line 16 of claim 1.
Claim 22 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention because it is unclear as to which detected product “the detected product” recited on line 6, along with subsequent recitations of “the detected product”, are referencing. Are they referring to “the detected produce” [sic] recited on line 10 of claim 1 or the “detected product” recited on line 4 of claim 22? Clarification and appropriate correction are required. For purposes of examination, the Examiner will treat “the detected product” recited on line 6 of claim 22, along with subsequent recitations of “the detected product”, as referencing the “detected product” recited on line 4 of claim 22.
Claim 22 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention because it is unclear as to which image of the detected product “the image of the detected product” recited on line 7 is referencing. Is it referring to “the image of the detected produce” [sic] recited on line 10 of claim 1 or the “image of a detected product” recited on line 4 of claim 22? Clarification and appropriate correction are required. For purposes of examination, the Examiner will treat “the image of the detected product” recited on line 7 of claim 22 as referencing the “image of a detected product” recited on line 4 of claim 22.
Claim 24 recites the limitation "the two or more images of the aisle" in lines 3 - 4. There is insufficient antecedent basis for this limitation in the claim.
Claims 3 - 7, 12, 15, 18 - 20 and 25 are also rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, due to being dependent upon a rejected base claim(s) but would be withdrawn from the rejection if their base claim(s) overcomes the rejection.
Response to Arguments
Applicant's arguments filed 30 December 2025 have been fully considered but they are not persuasive.
On pages 9 - 10 of the remarks the Applicant’s Representative argues that the previously cited prior art, particularly Skaff et al., “fails to teach a product type classifier that identifies product type based solely on an image of the detected product.” The Applicant’s Representative argues that Skaff et al. fail to teach the aforementioned disputed claim limitation at least because Skaff et al. teach “that one can obtain the product type by first identifying the product and then looking up the associated product in a database, where the record for that product will have the product type.” Therefore, the Applicant’s Representative argues that Skaff et al. fail to teach the aforementioned disputed claim limitation because of their “need for a database to obtain the product type”.
The Examiner respectfully disagrees.
Initially, the Examiner asserts that claim 1 is/was rejected under 35 U.S.C. 103 as being unpatentable over Skaff et al. in view of Perrella et al.
In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Furthermore, the Examiner asserts that, at least, Skaff et al. disclose a product type classifier trained to classify the detected products as one of a shelf product, a grill product or a shelf-ready package based solely on the image of the detected product within the bounding box, see at least the abstract, page 1 paragraphs 0005 - 0007, page 3 paragraphs 0027 and 0032, page 4 paragraphs 0038 - 0042, page 6 paragraphs 0090 - 0091, 0093 and 0103 - 0104, page 7 paragraphs 0106 - 0114, page 8 paragraphs 0133 - 0136 and 0142 - 0143 and page 9 paragraphs 0151 - 0158 of Skaff et al. wherein it is disclosed that “Bounding boxes around possible products in a panoramic image can be taken with at least one camera associated with the autonomous robot. Products in the bounding boxes are automatically identified” [abstract], “defining a product bounding box; associating the bounding box to a shelf label to build a training data set; and using the training data set to train a product classifier” [0005], that “Inventory data 314 can include but is not limited to an inventory database capable of storing data on a plurality of products, each product associated with a product type, product dimensions, a product 3D model, a product image and a current product price, shelf location, shelf inventory count and number of facings” [0032], that “Products within product bounding boxes can be… automatically identified using various image classifiers discussed herein” [0038], that an “alternative to extracting product descriptors is to use the bounding boxes as labeled categories and train classifiers on the images contained in the bounding boxes” [0104], that the “final layer of a convolutional layer network outputs the confidence values of the product being in one of the designated image categories” [0104], that for “those embodiments utilizing deep learning based image recognition, the input image is classified belonging to one of the product categories using convolutional neural network which outputs the confidence level” [0114], that in addition to “identifying products on the shelves by a first method that segments images to extract product images and identifiers and associate ones to the others; or performing a second method involving template matching without product segmentation; a third method using vision-based product recognition by extracting features descriptors can be used alone or in addition to other methods improve the overall accuracy of product identification. In situations where the first method or second methods fail to return results that meet a minimum level of confidence, the third method can utilize vision-based product recognition to extract features descriptors from image scans, and match them with sets of descriptors stored in the library's product objects. The matching can be performed through an exhaustive search, or preferably by restricting the search field to the product objects whose location is noted or established to be proximal to the image scan. The location can be inferred from metric or topological position estimates of the image scan” [0136] and that a “fourth method utilizing deep learning can be used. For example, product classifiers may include those based on deep structured learning, hierarchical learning, deep machine learning, or other suitable deep learning algorithms associated with convolutional, feedforward, recurrent, or another suitable neural network. A deep learning based classifier can automatically learn and update the product recognition system” [0142].
The Examiner asserts that, as shown herein above and in the cited portions, Skaff et al. disclose a product classifier(s) that recognizes identities of detected products based solely on images of the detected products within bounding boxes and that each product is associated with a product type, shelf location and product name. The Examiner asserts that the product identities recognized by the product classifier(s) of Skaff et al. classify each of the detected products as one of a shelf product, a grill product or a shelf-ready package at least because Skaff et al. disclose that each product is associated with a product type, shelf location and product name, because the broadest reasonable interpretation, in view of the instant specification, of a grill product appears to encompass any type of product, such as any product that “may be placed in a bin”, and because the broadest reasonable interpretation, in view of the instant specification, of a shelf-ready package appears to encompass any product that is placed on a shelf, such as a box, a container, a tray, or equivalent thereof containing individual products, for example the box, container, tray or equivalent thereof may be the product, see at least page 5 paragraph 0033 and page 14 paragraph 0060 of the instant specification. Moreover, the Examiner asserts that at least because Skaff et al. disclose that each product is associated with a product type, shelf location and product name that the act of recognizing/identifying a detected product as being a particular product by the product classifier(s) of Skaff et al. classifies the detected product as one of a particular product type since, in the system of Skaff et al., product type and shelf location are attributes intrinsically linked to each product to be identified. Thus, the Examiner asserts that at least Skaff et al. disclose a product type classifier trained to classify the detected products as one of a shelf product, a grill product or a shelf-ready package based solely on the image of the detected product within the bounding box.
In addition, the Examiner asserts that Perrella et al. disclose “a product type classifier trained to classify the detected products as one of a shelf product, a peg product, a grill product or a shelf-ready package based solely on the image of the detected produce [product] within the bounding box”, see at least figures 3, 4 and 6 - 9, page 1 paragraph 0018 - page 2 paragraph 0020, page 2 paragraph 0026, page 5 paragraphs 0046 - 0050 page 6 paragraphs 0053 - 0058 of Perrella et al. wherein it is disclosed that “products may be arranged in stocking areas: products may be supported on shelves for selection and purchase by customers or for picking by employees, vendors, or contractors; products may be placed in a particular floor zone; set on a pallet; or collected in a bin” [0018], that their “system 100 is deployed, in the illustrated example, in a retail environment including a plurality of shelf modules 110-1, 110-2, 110-3 and so on (collectively referred to as shelves 110, and generically referred to as a shelf 110—this nomenclature is also employed for other elements discussed herein). Each shelf module 110 supports a plurality of products 112... The shelf modules may also, in some examples, including other support structures, such as pegs, commercial racking, hangers and the like” [0026], that “primary object generator 304 is configured to generate a plurality of primary objects based on the captured data received at block 415. The primary object generator 304 may include a number of sub-components (not shown) each configured to detect a given type of primary object in the captured data. The objects detected in the captured data, for which primary data objects are generated at block 420, include gaps between products 112 (e.g. generated by a gap detection sub-component), and product indicators corresponding to the products 112 themselves (e.g. generated by a product/item recognition engine). The primary data objects can also include shelf edges (e.g. generated by a shelf structure detection subcomponent), labels (e.g. generated by a label detection subcomponent), and the like” [0047], that “Each primary data object generated at block 420 includes a location, which may be indicated by a bounding box overlaid on the relevant image, as well as one or more object attributes… The object attributes include an object type, defining the type of the primary data object” [0048], that “various primary objects are illustrated in connection with the images 600, following the performance of block 420. In particular, a shelf edge object 700 is shown in connection with each image 600… The primary data objects also include… product indicators 712, each indicating the presence of a product 112 in the underlying image data… The product indicators 712 include location and type attributes, as well as product identifier attributes (e.g. corresponding to a SKU or other identifier of the product depicted in the image)” [0049], that “secondary data objects may be generated by the above-mentioned sub-components configured to generate the primary data objects; in other words, in some examples the primary object generator 304 and the secondary object generator 308 are integrated with each other” [0050] and that “the secondary data objects are stored in a data structure associated with the image data (instead of the overlay shown in FIG. 8A), such as a tree structure in which the secondary data objects are assigned to hierarchical nodes. FIG. 9 illustrates an example tree data structure 900... The data structure 900 further includes a first layer of child nodes each corresponding to a shelf module 110. Each shelf module 110, in turn has child nodes corresponding to the shelf structure (e.g. shelf edge secondary data objects), which in turn have child nodes corresponding to products, gaps, labels and the like” [0054].
The Examiner asserts that, as shown herein above and in the cited portions, Perrella et al. disclose that products can be arranged on shelves, collected in a bin, supported on a shelf module, such as pegs, and/or placed in a particular floor zone, that primary and secondary object generators may generate a plurality of primary and secondary objects based on captured image data, that the primary and secondary object generators may include a number of sub-components each configured to detect a given type of object, such as the products themselves and shelf structures, that the primary and secondary object generators may be integrated with each other, and that the objects generated by the primary and secondary object generators may be stored in a tree data structure, wherein the tree data structure has a first layer of nodes that each correspond to a shelf module, each shelf module has child nodes corresponding to a shelf structure and each shelf structure has child nodes corresponding to each product. Hence, the Examiner asserts that Perrella et al. disclose using an integrated primary and secondary object generator, corresponding to the claimed product type classifier, to classify each detected product as being associated with a particular shelf module and structure, which Perrella et al. disclose as including shelves, bins, pegs, etc., based solely on an image of each detected product. Thus, the Examiner asserts that detected products classified as being associated with a shelf structure that is identified as being, for example, a peg would be classified as peg products. Additionally, the Examiner asserts that the product identities recognized by the integrated primary/secondary object generator/ product/item recognition engine of Perrella et al. classify the detected products as one of a shelf product, a peg product, a grill product or a shelf-ready package at least because Perrella et al. disclose that each product may be associated with a product type, shelf location/structure, such as pegs, and product name.
Therefore, the Examiner asserts that Skaff et al. in view of Perrella et al. disclose the aforementioned disputed claim limitation.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1, 3, 5 - 7, 11 - 13, 15, 18 - 20, 24 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Skaff et al. U.S. Publication No. 2017/0286773 A1 in view of Perrella et al. U.S. Publication No. 2020/0118064 A1.
- With regards to claim 1, Skaff et al. disclose a system (Skaff et al., Abstract, Figs. 1 - 7, Pg. 1 ¶ 0004 - 0007, Pg. 1 ¶ 0009 - Pg. 2 ¶ 0010, Pg. 2 ¶ 0018 - 0023, Pg. 3 ¶ 0026, Pg. 3 ¶ 0033 - Pg. 4 ¶ 0034) comprising: one or more processors; (Skaff et al., Figs. 1 - 3, 5A & 5B, Pg. 1 ¶ 0004 and 0006 - 0007, Pg. 2 ¶ 0022 - 0023, Pg. 3 ¶ 0027, Pg. 3 ¶ 0033 - Pg. 4 ¶ 0034, Pg. 4 ¶ 0036 - 0037, Pg. 7 ¶ 0108) one or more image sensors coupled to the one or more processors; (Skaff et al., Abstract, Figs. 1 - 5B, Pg. 1 ¶ 0004 - 0007, Pg. 1 ¶ 0009 - Pg. 2 ¶ 0010, Pg. 2 ¶ 0019 - Pg. 3 ¶ 0028, Pg. 3 ¶ 0033 - Pg. 4 ¶ 0034, Pg. 4 ¶ 0036 - 0039) software, executing on the one or more processors; (Skaff et al., Figs. 1 - 3, 4, 6 & 7, Pg. 1 ¶ 0004 and 0007, Pg. 2 ¶ 0010, 0019 and 0022 - 0023, Pg. 3 ¶ 0027, Pg. 3 ¶ 0033 - Pg. 4 ¶ 0038) a product detection neural network trained to detect images of products in images captured by the one or more image sensors and produce a bounding box containing each of the detected products; (Skaff et al., Abstract, Fig. 6, Pg. 1 ¶ 0005 - 0007, Pg. 2 ¶ 0019, Pg. 4 ¶ 0038 - 0041, Pg. 5 ¶ 0073 and 0084, Pg. 6 ¶ 0091 - 0093, Pg. 6 ¶ 0104 - Pg. 7 ¶ 0107, Pg. 7 ¶ 0114 - 0116, Pg. 8 ¶ 0130 - 0133 and 0142) a product type classifier trained to classify the detected products as one of a shelf product, a grill product or a shelf-ready package based solely on the image of the detected produce [product] within the bounding box; (Skaff et al., Abstract, Pg. 1 ¶ 0005 - 0007, Pg. 2 ¶ 0020, Pg. 3 ¶ 0027 - 0028 and 0032, Pg. 4 ¶ 0038 - 0042, Pg. 6 ¶ 0090 - 0091, 0093 and 0103 - 0104, Pg. 7 ¶ 0106 - 0114, Pg. 8 ¶ 0133 - 0136 and 0142 - 0143, Pg. 9 ¶ 0151 - 0158 [The Examiner asserts that the product classifier(s) of Skaff et al. recognizes identities of detected products and that the recognized product identities classify the recognized detected products as one of a shelf product, a grill product or a shelf-ready package at least because Skaff et al. disclose that each product is associated with a product type, shelf location and product name, because the broadest reasonable interpretation, in view of the instant specification, of a grill product appears to encompass any type of product, such as any product that “may be placed in a bin”, and because the broadest reasonable interpretation, in view of the instant specification, of a shelf-ready package appears to encompass any product that is placed on a shelf, such as a box, a container, a tray, or equivalent thereof containing individual products, for example the box, container, tray or equivalent thereof may be the product, see at least page 5 paragraph 0033 and page 14 paragraph 0060 of the instant specification. The Examiner asserts that since, in the system of Skaff et al., product type and shelf location are attributes intrinsically linked to each product to be identified that the act of recognizing/identifying a detected product as being a particular product by the product classifier(s) of Skaff et al. classifies the detected product as one of a particular product type, i.e., as one of a shelf product, a grill product or a shelf-ready package.]) wherein the software causes the system to: (Skaff et al., Figs. 3, 4, 6 & 7, Pg. 2 ¶ 0010, 0019 and 0022 - 0023, Pg. 3 ¶ 0027, Pg. 3 ¶ 0033 - Pg. 4 ¶ 0038) capture images of two or more portions of an aisle; (Skaff et al., Fig. 4, Pg. 1 ¶ 0007 and 0009, Pg. 2 ¶ 0019 - 0020, Pg. 3 ¶ 0024 - 0027, Pg. 4 ¶ 0036 and 0038) stitch the images for the two or more portions together to create a unified image; (Skaff et al., Fig. 4, Pg. 1 ¶ 0007 and 0009, Pg. 2 ¶ 0019 - 0020, Pg. 3 ¶ 0024 - 0027, Pg. 4 ¶ 0036 and 0038) use the product detection neural network to obtain bounding boxes of detected products in the unified image; (Skaff et al., Abstract, Figs. 2, 4 & 6, Pg. 1 ¶ 0005 - 0007, Pg. 1 ¶ 0009 - Pg. 2 ¶ 0010, Pg. 2 ¶ 0018 - 0019, Pg. 3 ¶ 0024 - 0025, Pg. 4 ¶ 0038 - 0041, Pg. 5 ¶ 0073 and 0084, Pg. 6 ¶ 0093, Pg. 6 ¶ 0104 - Pg. 7 ¶ 0107, Pg. 8 ¶ 0130 - 0131) and use the product type classifier to classify products in each bounding box as containing one of a shelf product, a grill product or a shelf-ready package. (Skaff et al., Abstract, Pg. 1 ¶ 0005 - 0007, Pg. 2 ¶ 0020, Pg. 3 ¶ 0027 - 0028 and 0032, Pg. 4 ¶ 0038 - 0042, Pg. 6 ¶ 0090 - 0091, 0093 and 0103 - 0104, Pg. 7 ¶ 0106 - 0114, Pg. 8 ¶ 0133 - 0136 and 0142 - 0143, Pg. 9 ¶ 0151 - 0158 [The Examiner asserts that the product classifier(s) of Skaff et al. recognizes identities of detected products and that the recognized product identities classify the recognized detected products as one of a shelf product, a grill product or a shelf-ready package at least because Skaff et al. disclose that each product is associated with a product type, shelf location and product name, because the broadest reasonable interpretation, in view of the instant specification, of a grill product appears to encompass any type of product, such as any product that “may be placed in a bin”, and because the broadest reasonable interpretation, in view of the instant specification, of a shelf-ready package appears to encompass any product that is placed on a shelf, such as a box, a container, a tray, or equivalent thereof containing individual products, for example the box, container, tray or equivalent thereof may be the product, see at least page 5 paragraph 0033 and page 14 paragraph 0060 of the instant specification. The Examiner asserts that since, in the system of Skaff et al., product type and shelf location are attributes intrinsically linked to each product to be identified that the act of recognizing/identifying a detected product as being a particular product by the product classifier(s) of Skaff et al. classifies the detected product as one of a particular product type, i.e., as one of a shelf product, a grill product or a shelf-ready package.]) Skaff et al. fail to disclose explicitly classifying a peg product. Pertaining to analogous art, Perrella et al. disclose a system (Perrella et al., Abstract, Figs. 1 - 3, Pg. 2 ¶ 0024 - Pg. 3 ¶ 0034, Pg. 9 ¶ 0098 - 0099) comprising: one or more processors; (Perrella et al., Pg. 2 ¶ 0027 - Pg. 3 ¶ 0034, Pg. 9 ¶ 0098 - 0099) one or more image sensors coupled to the one or more processors; (Perrella et al., Figs. 1 - 3, Pg. 2 ¶ 0023 - Pg. 3 ¶ 0034, Pg. 9 ¶ 0098 - 0099) software, executing on the one or more processors; (Perrella et al., Abstract, Figs. 1 - 3, Pg. 2 ¶ 0024 - 0025, Pg. 2 ¶ 0027 - Pg. 3 ¶ 0032, Pg. 9 ¶ 0098 - 0099) a product type classifier trained to classify the detected products as one of a shelf product, a peg product, a grill product or a shelf-ready package based solely on the image of the detected produce [product] within the bounding box; (Perrella et al., Figs. 3, 4 & 6 - 9, Pg. 1 ¶ 0018 - Pg. 2 ¶ 0020, Pg. 2 ¶ 0026, Pg. 5 ¶ 0046 - 0050, Pg. 6 ¶ 0053 - 0058, Pg. 8 ¶ 0071, 0077 - 0078 and 0081 - 0083 [“Certain environments may assign attributes to products or to SKUs to better manage the products, such as a cost, price, batch, date code, expiration date, components and/or ingredients, packaging, tracking number, order number, shipping number, storage requirements, or the like. Within the environment, products may be arranged in stocking areas: products may be supported on shelves for selection and purchase by customers or for picking by employees, vendors, or contractors; products may be placed in a particular floor zone; set on a pallet; or collected in a bin”, “products in such environments are often associated with labels placed on a shelf edge, a peg, racking, displays or the products themselves”, “the primary object generator 304 is configured to generate a plurality of primary objects based on the captured data received at block 415. The primary object generator 304 may include a number of sub-components (not shown) each configured to detect a given type of primary object in the captured data. The objects detected in the captured data, for which primary data objects are generated at block 420, include gaps between products 112 (e.g. generated by a gap detection sub-component), and product indicators corresponding to the products 112 themselves (e.g. generated by a product/item recognition engine). The primary data objects can also include shelf edges (e.g. generated by a shelf structure detection subcomponent)”, “Each primary data object generated at block 420 includes a location, which may be indicated by a bounding box overlaid on the relevant image, as well as one or more object attributes… The object attributes include an object type, defining the type of the primary data object. Additional attributes may also be included…”, “primary data objects also include label objects 708, each indicating the presence of a label in the underlying image data, as well as product indicators 712, each indicating the presence of a product 112 in the underlying image data... The product indicators 712 include location and type attributes, as well as product identifier attributes (e.g. corresponding to a SKU or other identifier of the product depicted in the image)”, “secondary data objects may be generated by the above-mentioned sub-components configured to generate the primary data objects; in other words, in some examples the primary object generator 304 and the secondary object generator 308 are integrated with each other” and “the secondary data objects are stored in a data structure associated with the image data (instead of the overlay shown in FIG. 8A), such as a tree structure in which the secondary data objects are assigned to hierarchical nodes. FIG. 9 illustrates an example tree data structure 900... The data structure 900 further includes a first layer of child nodes each corresponding to a shelf module 110. Each shelf module 110, in turn has child nodes corresponding to the shelf structure (e.g. shelf edge secondary data objects), which in turn have child nodes corresponding to products, gaps, labels and the like.” The Examiner asserts that the integrated primary/secondary object generator/ product/item recognition engine of Perrella et al. recognizes identities of each detected product and classifies each detected product as being associated with a particular shelf module and structure, which Perrella et al. disclose as including shelves, bins, pegs, etc. Thus, detected products classified as being associated with a shelf structure that is identified as being, for example, a peg would be classified as peg products. In addition, the Examiner asserts that the recognized product identities of Perrella et al. additionally classify the detected products as one of a shelf product, a peg product, a grill product or a shelf-ready package at least because Perrella et al. disclose that each product may be associated with a product type, shelf location/structure, such as pegs, and product name.]) wherein the software causes the system to: (Perrella et al., Abstract, Figs. 1 - 3, Pg. 2 ¶ 0024 - 0025, Pg. 2 ¶ 0027 - Pg. 3 ¶ 0032, Pg. 9 ¶ 0098 - 0099) use the product type classifier to classify products in each bounding box as containing one of a shelf product, a peg product, a grill product or a shelf-ready package. (Perrella et al., Figs. 3, 4 & 9, Pg. 1 ¶ 0018 - Pg. 2 ¶ 0020, Pg. 2 ¶ 0026, Pg. 5 ¶ 0046 - 0050, Pg. 6 ¶ 0054 - 0058, Pg. 8 ¶ 0071, 0077 - 0078 and 0081 - 0083) Skaff et al. and Perrella et al. are combinable because they are both directed towards image processing systems and methods that detect and identify products stocked on a shelf. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Skaff et al. with the teachings of Perrella et al. This modification would have been prompted in order to enhance the base device of Skaff et al. with the well-known and applicable technique Perrella et al. applied to a comparable device. Classifying products as peg products, as taught by Perrella et al., would enhance the base device of Skaff et al. by improving its ability to accurately and reliably manage product inventory since product inventory data associated with pegs would also be detected and analyzed during inventory management thereby ensuring that each and every product, product type and product stocking area is examined and analyzed during inventory management processing. Furthermore, this modification would have been prompted by the teachings and suggestions of Skaff et al. to detect a label type and that each product may be associated with a product type, see at least page 1 paragraph 0007 and page 3 paragraph 0032 of Skaff et al. This combination could be completed according to well-known techniques in the art and would likely yield predictable results, in that peg products would also be detected and identified as such so as to improve the ability of the base device of Skaff et al. to accurately and reliably manage product inventory by allowing for peg type products to be additionally included and analyzed during inventory management processing. Therefore, it would have been obvious to combine Skaff et al. with Perrella et al. to obtain the invention as specified in claim 1.
- With regards to claim 3, Skaff et al. in view of Perrella et al. disclose the system of claim 1, the unified image being a panoramic image showing the entirety of the aisle. (Skaff et al., Fig. 4, Pg. 1 ¶ 0007, Pg. 2 ¶ 0010 and 0019 - 0020, Pg. 3 ¶ 0025, Pg. 4 ¶ 0035 - 0036)
- With regards to claim 5, Skaff et al. in view of Perrella et al. disclose the system of claim 1 wherein the bounding boxes containing each detected product are expressed as tuples, (Skaff et al., Abstract, Pg. 1 ¶ 0005 - 0007, Pg. 4 ¶ 0038 - 0040, Pg. 5 ¶ 0089, Pg. 6 ¶ 0093 - 0095 and 0100 - 0103) each of the tuples comprising: coordinates of one corner of the bounding box with respect to a coordinate system superimposed on the unified image, a width of the bounding box; and a height of the bounding box. (Skaff et al., Abstract, Pg. 1 ¶ 0005 - 0007, Pg. 3 ¶ 0025, 0028 and 0032, Pg. 4 ¶ 0038 - 0040, Pg. 5 ¶ 0089, Pg. 6 ¶ 0093 - 0095 and 0100 - 0103)
- With regards to claim 6, Skaff et al. in view of Perrella et al. disclose the system of claim 5 wherein units of the coordinate system correspond to pixels in the unified image. (Skaff et al., Pg. 0025, Pg. 8 ¶ 0130 - 0131)
- With regards to claim 7, Skaff et al. in view of Perrella et al. disclose the system of claim 5 further comprising: submitting the unified image to a zone detection model that can detect fixture zones containing one or more fixtures, (Skaff et al., Pg. 1 ¶ 0007, Pg. 3 ¶ 0025, Pg. 4 ¶ 0039 and 0046 - 0049, Pg. 5 ¶ 0055 - 0065 and 0089, Pg. 6 ¶ 0093 - 0094 and 0099, Pg. 9 ¶ 0163 - 0166) wherein fixtures comprise shelves; (Skaff et al., Pg. 1 ¶ 0007, Pg. 3 ¶ 0025, Pg. 4 ¶ 0039 and 0046 - 0049, Pg. 5 ¶ 0055 - 0065 and 0089, Pg. 6 ¶ 0093 - 0094 and 0099, Pg. 9 ¶ 0163 - 0166) and receiving, from the zone detection model, a list comprising locations and sizes of the detected fixture zones. (Skaff et al., Pg. 1 ¶ 0007, Pg. 3 ¶ 0025, Pg. 4 ¶ 0039 and 0046 - 0049, Pg. 5 ¶ 0055 - 0065 and 0089, Pg. 6 ¶ 0093 - 0094 and 0099, Pg. 9 ¶ 0163 - 0166) Skaff et al. fail to disclose explicitly wherein fixtures comprise pegs. Pertaining to analogous art, Perrella et al. disclose submitting the unified image to a zone detection model that can detect fixture zones containing one or more fixtures, (Perrella et al., 1 & 6 - 9, Pg. 1 ¶ 0018 - Pg. 2 ¶ 0020, Pg. 2 ¶ 0026, Pg. 4 ¶ 0041, Pg. 5 ¶ 0046 - 0047 and 0049, Pg. 6 ¶ 0054 - 0056 and 0058 [“Each mismatch record generated at block 430 identifies a product, label, shelf structure or the like…”]) wherein fixtures comprise shelves and pegs; (Perrella et al., Pg. 1 ¶ 0018 - Pg. 2 ¶ 0020, Pg. 2 ¶ 0026, Pg. 5 ¶ 0046 - 0049, Pg. 6 ¶ 0054 - 0055 and 0058) and receiving, from the zone detection model, a list comprising locations and sizes of the detected fixture zones. (Perrella et al., Figs. 1 & 6 - 9, Pg. 2 ¶ 0026, Pg. 4 ¶ 0041, Pg. 5 ¶ 0046 - 0049, Pg. 6 ¶ 0052 - 0054, 0056 and 0058)
- With regards to claim 11, Skaff et al. in view of Perrella et al. disclose the system of claim 1 further comprising: a shelf ready package classifier for determining whether a shelf-ready package is empty or not empty; (Skaff et al., Pg. 1 ¶ 0007, Pg. 3 ¶ 0032, Pg. 4 ¶ 0036, Pg. 8 ¶ 0134, Pg. 9 ¶ 0162 - 0169) wherein, when a detected product is typed as a shelf-ready package, (Skaff et al., Pg. 1 ¶ 0007, Pg. 3 ¶ 0032, Pg. 4 ¶ 0036, Pg. 8 ¶ 0134, Pg. 9 ¶ 0162 - 0169) the software causes the system to perform (Skaff et al., Figs. 3, 4, 6 & 7, Pg. 2 ¶ 0010, 0019 and 0022 - 0023, Pg. 3 ¶ 0027, Pg. 3 ¶ 0033 - Pg. 4 ¶ 0038) the further functions of: submitting the detected product typed as a shelf-ready package to the shelf-ready package classifier; (Skaff et al., Pg. 1 ¶ 0007, Pg. 3 ¶ 0032, Pg. 4 ¶ 0036, Pg. 8 ¶ 0134, Pg. 9 ¶ 0162 - 0169) and receiving, from the shelf-ready package classifier, an indication of whether the detected product is empty or not empty. (Skaff et al., Pg. 1 ¶ 0005 - 0007, Pg. 3 ¶ 0032, Pg. 4 ¶ 0036, Pg. 8 ¶ 0134 - 0136 and 0143 - 0144, Pg. 9 ¶ 0147 - 0148, 0150, 0153 - 0154 and 0158 - 0169) In addition, analogous art Perrella et al. disclose a shelf ready package classifier for determining whether a shelf-ready package is empty or not empty; (Perrella et al., Pg. 1 ¶ 0018 - 0019, Pg. 6 ¶ 0056 - 0058, Pg. 7 ¶ 0064 - 0065) wherein, when a detected product is typed as a shelf-ready package, (Perrella et al., Pg. 1 ¶ 0018 - 0019, Pg. 6 ¶ 0056 - 0058, Pg. 7 ¶ 0064 - 0065) the software causes the system to perform (Perrella et al., Abstract, Figs. 1 - 3, Pg. 2 ¶ 0024 - 0025, Pg. 2 ¶ 0027 - Pg. 3 ¶ 0032, Pg. 9 ¶ 0098 - 0099) the further functions of: submitting the detected product typed as a shelf-ready package to the shelf-ready package classifier; (Perrella et al., Pg. 1 ¶ 0018 - 0019, Pg. 6 ¶ 0056 - 0058, Pg. 7 ¶ 0064 - 0065) and receiving, from the shelf-ready package classifier, an indication of whether the detected product is empty or not empty. (Perrella et al., Pg. 1 ¶ 0018 - 0019, Pg. 6 ¶ 0056 - 0058, Pg. 7 ¶ 0064 - 0065)
- With regards to claim 12, Skaff et al. in view of Perrella et al. disclose the system of claim 1, further comprising: a ghosted product detection model for detecting out-of-focus or blurry images of products in the unified image; (Skaff et al., Pg. 1 ¶ 0007, Pg. 2 ¶ 0019 - 0021, Pg. 3 ¶ 0025 - 0028 and 0031 - 0032, Pg. 4 ¶ 0036 - 0040, Pg. 8 ¶ 0134 - 0140 and 0143, Pg. 9 ¶ 0148, 0150 - 0151, 0153 - 0154 and 0158 - 0163) wherein the software causes the system to perform (Skaff et al., Figs. 3, 4, 6 & 7, Pg. 2 ¶ 0010, 0019 and 0022 - 0023, Pg. 3 ¶ 0027, Pg. 3 ¶ 0033 - Pg. 4 ¶ 0038) the further functions of: submitting the unified image to the ghosted product detection model; (Skaff et al., Pg. 1 ¶ 0007, Pg. 2 ¶ 0019 - 0021, Pg. 3 ¶ 0025 - 0028 and 0031 - 0032, Pg. 4 ¶ 0036 - 0040, Pg. 8 ¶ 0134 - 0140 and 0143, Pg. 9 ¶ 0148, 0150 - 0151, 0153 - 0154 and 0158 - 0163) and receiving, from the ghosted product detection model, a list comprising detected ghosted products. (Skaff et al., Pg. 1 ¶ 0007, Pg. 2 ¶ 0019 - 0021, Pg. 3 ¶ 0025 - 0028 and 0031 - 0032, Pg. 4 ¶ 0036 - 0040, Pg. 8 ¶ 0134 - 0140 and 0143, Pg. 9 ¶ 0148, 0150 - 0151, 0153 - 0154 and 0158 - 0163)
- With regards to claim 13, Skaff et al. in view of Perrella et al. disclose the system of claim 12, the software causes the system to perform (Skaff et al., Figs. 3, 4, 6 & 7, Pg. 2 ¶ 0010, 0019 and 0022 - 0023, Pg. 3 ¶ 0027, Pg. 3 ¶ 0033 - Pg. 4 ¶ 0038) the further function of: combining the detected products and the list of detected ghosted products. (Skaff et al., Pg. 1 ¶ 0006 - 0007 and 0009, Pg. 2 ¶ 0020, Pg. 3 ¶ 0028 and 0032, Pg. 4 ¶ 0036 - 0040, Pg. 6 ¶ 0093 and 0103, Pg. 7 ¶ 0106 - 0109, Pg. 8 ¶ 0133 - 0136 and 0142 - 0144, Pg. 9 ¶ 0148, 0150 - 0151 and 0153 - 0163)
- With regards to claim 15, Skaff et al. in view of Perrella et al. disclose the system of claim 1, further comprising: a label detection neural network trained to detect labels in one or more images captured by the one or more image sensors; (Skaff et al., Abstract, Fig. 6, Pg. 1 ¶ 0005 - 0007, Pg. 2 ¶ 0020, Pg. 3 ¶ 0028, Pg. 4 ¶ 0036, Pg. 5 ¶ 0073 and 0084, Pg. 6 ¶ 0091 [“in a first step 610, shelf labels are detected either in individual shelf images, or in a stitched panorama. Classification algorithms such as convolution neural networks or other deep learning methods, template matching or HAAR cascades can be used to aid in detection of each shelf label”]) and a label type classifier for identifying detected labels as shelf labels; (Skaff et al., Abstract, Fig. 6, Pg. 1 ¶ 0005 - 0007, Pg. 2 ¶ 0020, Pg. 3 ¶ 0028, Pg. 4 ¶ 0036 - 0040, Pg. 5 ¶ 0073, Pg. 6 ¶ 0090 - 0094 and 0104, Pg. 8 ¶ 0133) wherein the software causes the system to perform (Skaff et al., Figs. 3, 4, 6 & 7, Pg. 2 ¶ 0010, 0019 and 0022 - 0023, Pg. 3 ¶ 0027, Pg. 3 ¶ 0033 - Pg. 4 ¶ 0038) the further functions of: using the label detection neural network to detect labels in the one or more images; (Skaff et al., Abstract, Figs. 2, 4 & 6, Pg. 1 ¶ 0005 - 0007, Pg. 1 ¶ 0009 - Pg. 2 ¶ 0010, Pg. 2 ¶ 0018 - 0020, Pg. 3 ¶ 0024 - 0025 and 0028, Pg. 4 ¶ 0035 - 0036, Pg. 5 ¶ 0073 and 0084, Pg. 6 ¶ 0091) and using the label type classifier to identify detected labels as shelf labels. (Skaff et al., Pg. 1 ¶ 0005 - 0007, Pg. 2 ¶ 0020, Pg. 3 ¶ 0028, Pg. 4 ¶ 0036 and 0038 - 0040, Pg. 5 ¶ 0073, Pg. 6 ¶ 0090 - 0093, Pg. 7 ¶ 0107 - 0109 and 0114, Pg. 8 ¶ 0134 - 0136 [“detect label types”]) Skaff et al. fail to disclose explicitly identifying detected labels as peg labels. Pertaining to analogous art, Perrella et al. disclose a label type classifier for identifying detected labels as shelf labels or peg labels; (Perrella et al., Pg. 1 ¶ 0018 - Pg. 2 ¶ 0020, Pg. 2 ¶ 0026, Pg. 5 ¶ 0046 - 0050, Pg. 6 ¶ 0054 - 0058, Pg. 8 ¶ 0071, 0077 - 0078 and 0081 - 0083 [“products in such environments are often associated with labels placed on a shelf edge, a peg, racking, displays or the products themselves”]) and using the label type classifier to identify detected labels as shelf labels or peg labels. (Perrella et al., Pg. 1 ¶ 0018 - Pg. 2 ¶ 0020, Pg. 2 ¶ 0026, Pg. 5 ¶ 0046 - 0050, Pg. 6 ¶ 0054 - 0058, Pg. 8 ¶ 0071, 0077 - 0078 and 0081 - 0083 [“products in such environments are often associated with labels placed on a shelf edge, a peg, racking, displays or the products themselves”])
- With regards to claim 18, Skaff et al. in view of Perrella et al. disclose the system of claim 15, further comprising: a shelf detection classifier for determining if pixels in the unified image are part of a shelf or are not part of a shelf; (Skaff et al., Fig. 6, Pg. 1 ¶ 0007, Pg. 3 ¶ 0025, Pg. 4 ¶ 0039 - 0041 and 0046 - 0049, Pg. 5 ¶ 0055 - 0065, 0073 and 0088 - 0089, Pg. 6 ¶ 0091 - 0094 and 0099 - 0104, Pg. 8 ¶ 0136 and 0142 - 0145, Pg. 9 ¶ 0147 and 0163 - 0166) wherein the software causes the system to perform (Skaff et al., Figs. 3, 4, 6 & 7, Pg. 2 ¶ 0010, 0019 and 0022 - 0023, Pg. 3 ¶ 0027, Pg. 3 ¶ 0033 - Pg. 4 ¶ 0038) the further function of: using the shelf detection classifier to detect shelves; (Skaff et al., Fig. 6, Pg. 1 ¶ 0007, Pg. 3 ¶ 0025, Pg. 4 ¶ 0039 - 0041 and 0046 - 0049, Pg. 5 ¶ 0055 - 0065, 0073 and 0088 - 0089, Pg. 6 ¶ 0091 - 0094 and 0099 - 0104, Pg. 8 ¶ 0136 and 0142 - 0145, Pg. 9 ¶ 0147 and 0163 - 0166) wherein the detected shelves are used by the label type classifier to determine which detected labels are shelf labels. (Skaff et al., Abstract, Fig. 6, Pg. 1 ¶ 0005 - 0007, Pg. 2 ¶ 0020, Pg. 3 ¶ 0025 and 0028, Pg. 4 ¶ 0036 - 0041, Pg. 5 ¶ 0073, Pg. 5 ¶ 0084 - Pg. 6 ¶ 0094, Pg. 6 ¶ 0099 and 0103 - 0104, Pg. 8 ¶ 0133) In addition, analogous art, Perrella et al. disclose a shelf detection classifier for determining if pixels in the unified image are part of a shelf or are not part of a shelf; (Perrella et al., Figs. 1 & 6 - 9, Pg. 1 ¶ 0018 - Pg. 2 ¶ 0020, Pg. 2 ¶ 0026, Pg. 4 ¶ 0041, Pg. 5 ¶ 0046 - 0049, Pg. 6 ¶ 0054 - 0056 and 0058) wherein the software causes the system to perform (Perrella et al., Abstract, Figs. 1 - 3, Pg. 2 ¶ 0024 - 0025, Pg. 2 ¶ 0027 - Pg. 3 ¶ 0032, Pg. 9 ¶ 0098 - 0099) the further function of: using the shelf detection classifier to detect shelves; (Perrella et al., 1 & 6 - 9, Pg. 1 ¶ 0018 - Pg. 2 ¶ 0020, Pg. 2 ¶ 0026, Pg. 4 ¶ 0041, Pg. 5 ¶ 0046 - 0049, Pg. 6 ¶ 0054 - 0056 and 0058 [“Each mismatch record generated at block 430 identifies a product, label, shelf structure or the like…”]) wherein the detected shelves are used by the label type classifier to determine which detected labels are shelf labels. (Perrella et al., Pg. 1 ¶ 0018 - Pg. 2 ¶ 0020, Pg. 2 ¶ 0026, Pg. 5 ¶ 0046 - 0049, Pg. 6 ¶ 0054 - 0056 and 0058 [“products in such environments are often associated with labels placed on a shelf edge, a peg, racking, displays or the products themselves”])
- With regards to claim 19, Skaff et al. in view of Perrella et al. disclose the system of claim 15, the software causing the system to perform (Skaff et al., Figs. 3, 4, 6 & 7, Pg. 2 ¶ 0010, 0019 and 0022 - 0023, Pg. 3 ¶ 0027, Pg. 3 ¶ 0033 - Pg. 4 ¶ 0038) the further function of: inferring a location of shelves from locations of shelf labels which appear in a straight horizontal line in the unified image. (Skaff et al., Abstract, Fig. 6, Pg. 1 ¶ 0005 - 0007, Pg. 3 ¶ 0028, Pg. 4 ¶ 0036, Pg. 5 ¶ 0073 and 0086, Pg. 6 ¶ 0091 - 0097 and 0103)
- With regards to claim 20, Skaff et al. in view of Perrella et al. disclose the system of claim 19, the software causing the system to perform (Skaff et al., Figs. 3, 4, 6 & 7, Pg. 2 ¶ 0010, 0019 and 0022 - 0023, Pg. 3 ¶ 0027, Pg. 3 ¶ 0033 - Pg. 4 ¶ 0038) the further function of: inferring a type of each detected shelf based on a bounding box of the shelf. (Skaff et al., Pg. 1 ¶ 0007, Pg. 3 ¶ 0025, Pg. 4 ¶ 0038 - 0039, Pg. 5 ¶ 0087 - 0089, Pg. 6 ¶ 0093 - 0101, Pg. 9 ¶ 0163 - 0166)
- With regards to claim 24, Skaff et al. in view of Perrella et al. disclose the system of claim 1, further comprising: a mobile inventory robot upon which the one or more image sensors are mounted, (Skaff et al., Abstract, Figs. 1 - 5B, Pg. 1 ¶ 0004, Pg. 1 ¶ 0008 - Pg. 2 ¶ 0010, Pg. 2 ¶ 0018 - 0020, Pg. 2 ¶ 0022 - Pg. 3 ¶ 0028, Pg. 4 ¶ 0035 - 0038) that autonomously traverses the aisle to capture the two or more images of the aisle. (Skaff et al., Abstract, Figs. 1 - 5B, Pg. 1 ¶ 0006 - Pg. 2 ¶ 0010, Pg. 2 ¶ 0018 - 0020, Pg. 2 ¶ 0022 - Pg. 3 ¶ 0027, Pg. 4 ¶ 0035 - 0038)
- With regards to claim 25, Skaff et al. in view of Perrella et al. disclose the system of claim 15, the software causing the system to perform the further function of: associating a detected product with a detected label (Skaff et al., Fig. 6, Pg. 1 ¶ 0005 - 0007, Pg. 5 ¶ 0073 and 0085 - 0088, Pg. 6 ¶ 0090 - 0093 and 0103 - 0104, Pg. 8 ¶ 0134 - 0139) based on a type of product as determined by the product type classifier, a type of label, as identified by the label type classifier and proximity of the detected product and the detected label. (Skaff et al., Fig. 6, Pg. 1 ¶ 0005 - 0007, Pg. 2 ¶ 0020, Pg. 3 ¶ 0025, 0028 and 0032, Pg. 4 ¶ 0036 and 0038 - 0039, Pg. 5 ¶ 0073, Pg. 5 ¶ 0084 - Pg. 6 ¶ 0091, Pg. 6 ¶ 0093 - 0095 and 0103 - 0104, Pg. 8 ¶ 0133 - 0136) In addition, analogous art Perrella et al. disclose associating a detected product with a detected label (Perrella et al., Figs. 1, 4 & 6 - 9, Pg. 1 ¶ 0018, Pg. 2 ¶ 0020, Pg. 5 ¶ 0046 - 0049, Pg. 6 ¶ 0053 - 0058) based on a type of product as determined by the product type classifier, a type of label, as identified by the label type classifier and proximity of the detected product and the detected label. (Perrella et al., Figs. 1 & 6 - 9, Pg. 1 ¶ 0018, Pg. 2 ¶ 0020 and 0026, Pg. 5 ¶ 0046 - 0049, Pg. 6 ¶ 0053 - 0058 [The Examiner asserts that at least figures 7 - 9 of Perrella et al. illustrate that shelf edge object 700/812, shelf product 804-3 and shelf label 808-3 are associated with each other.])
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Skaff et al. U.S. Publication No. 2017/0286773 A1 in view of Perrella et al. U.S. Publication No. 2020/0118064 A1 as applied to claim 1 above, and further in view of Chou et al. U.S. Publication No. 2013/0250041 A1.
- With regards to claim 4, Skaff et al. in view of Perrella et al. disclose the system of claim 1, wherein capturing the images of two or more portions of the aisle comprises: obtaining multiple images of the two or more portions at varying focus depths. (Skaff et al., Pg. 2 ¶ 0019 - 0020, Pg. 3 ¶ 0028, Pg. 5 ¶ 0073 and 0078) Skaff et al. fail to disclose explicitly selecting one or more of the multiple images for each portion of the two or more portions at any of the varying focus depths for inclusion in the unified image. Pertaining to analogous art, Chou et al. disclose obtaining multiple images of the two or more portions at varying focus depths; (Chou et al., Abstract, Figs. 1 - 3 & 5, Pg. 1 ¶ 0010 and 0017, Pg. 2 ¶ 0038 - 0040 and 0042, Pg. 4 ¶ 0045 - 0048) and selecting one or more of the multiple images for each portion of the two or more portions at any of the varying focus depths for inclusion in the unified image. (Chou et al., Abstract, Figs. 1 - 3 & 5, Pg. 1 ¶ 0010 and 0017, Pg. 2 ¶ 0038 - 0040 and 0042, Pg. 4 ¶ 0045 - 0048) Skaff et al. in view of Perrella et al. and Chou et al. are combinable because they are all directed towards image processing systems and methods that stitch a plurality of captured images together into a single composite image. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combined teachings of Skaff et al. in view of Perrella et al. with the teachings of Chou et al. This modification would have been prompted in order to enhance the combined base device of Skaff et al. in view of Perrella et al. with the well-known and applicable technique Chou et al. applied to a similar device. Selecting one or more of the multiple images obtained at varying focus depths for each portion of the two or more portions for inclusion in the unified image, as taught by Chou et al., would enhance the combined base device by improving the quality of the stitched together panoramic image in order to help ensure that each product and label is clearly represented in the panoramic image and thereby improving the ability of the combined base device to accurately and reliably detect and identify each product and label in the panoramic image. This combination could be completed according to well-known techniques in the art and would likely yield predictable results, in that one or more of multiple images obtained at varying focus depths would be selected for each portion of the two or more portions for inclusion in the unified image so as to improve the quality of the stitched together panoramic image and help ensure that each product and label is clearly represented in the panoramic image thereby improving the ability of the combined base device to accurately and reliably detect and identify each product and label in the panoramic image. Therefore, it would have been obvious to combine Skaff et al. in view of Perrella et al. with Chou et al. to obtain the invention as specified in claim 4.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Skaff et al. U.S. Publication No. 2017/0286773 A1 in view of Perrella et al. U.S. Publication No. 2020/0118064 A1 as applied to claim 1 above, and further in view of Medina et al. U.S. Publication No. 2019/0087772 A1.
- With regards to claim 22, Skaff et al. in view of Perrella et al. disclose the system of claim 1, further comprising: a product identification classifier; (Skaff et al., Abstract, Fig. 7, Pg. 1 ¶ 0005 - 0007, Pg. 2 ¶ 0020, Pg. 3 ¶ 0027 - 0028 and 0032, Pg. 4 ¶ 0038 - 0042, Pg. 6 ¶ 0090 - 0091 and 0104, Pg. 7 ¶ 0106 - 0114, Pg. 8 ¶ 0133 - 0136 and 0142) wherein the software causes the system perform (Skaff et al., Figs. 3, 4, 6 & 7, Pg. 2 ¶ 0010, 0019 and 0022 - 0023, Pg. 3 ¶ 0027, Pg. 3 ¶ 0033 - Pg. 4 ¶ 0038) the further functions of: obtaining an image of a detected product from an image of a portion of the aisle in which the detected product appears; (Skaff et al., Abstract, Pg. 1 ¶ 0007, Pg. 2 ¶ 0020, Pg. 3 ¶ 0028, Pg. 4 ¶ 0036 - 0040, Pg. 6 ¶ 0091 - 0093 and 0104, Pg. 7 ¶ 0114, Pg. 8 ¶ 0129 - 0130, 0134 - 0136 and 0142) and using the product identification classifier to identify the detected product. (Skaff et al., Abstract, Fig. 7, Pg. 1 ¶ 0005 - 0007, Pg. 2 ¶ 0020, Pg. 3 ¶ 0027 - 0028 and 0032, Pg. 4 ¶ 0035 - 0041, Pg. 6 ¶ 0090 - 0091 and 0104, Pg. 7 ¶ 0106 - 0114, Pg. 8 ¶ 0133 - 0136 and 0142) Skaff et al. fail to disclose explicitly wherein the image of the detected product is of a higher resolution than an image of the detected product obtained from the unified image. Pertaining to analogous art, Medina et al. disclose obtaining an image of a detected product from an image of a portion of the aisle in which the detected product appears; (Medina et al., Pg. 5 ¶ 0063, Pg. 6 ¶ 0071 - 0076, Pg. 6 ¶ 0078 - Pg. 7 ¶ 0079, Pg. 8 ¶ 0085 - 0086) wherein the image of the detected product is of a higher resolution than an image of the detected product obtained from the unified image. (Medina et al., Pg. 5 ¶ 0063, Pg. 6 ¶ 0071 - 0076, Pg. 6 ¶ 0078 - Pg. 7 ¶ 0079, Pg. 8 ¶ 0085 - 0086) Skaff et al. in view of Perrella et al. and Medina et al. are combinable because they are all directed towards image processing systems and methods that detect and identify products stocked on a shelf. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combined teachings of Skaff et al. in view of Perrella et al. with the teachings of Medina et al. This modification would have been prompted in order to enhance the combined base device of Skaff et al. in view of Perrella et al. with the well-known and applicable technique Medina et al. applied to a comparable device. Obtaining an image of the detected product that is of a higher resolution than an image of the detected product obtained from the unified image, as taught by Medina et al., would enhance the combined base device by helping ensure that the image of the product is of the highest quality possible thereby improving the ability of the combined base device to accurately and reliably identify the product. This combination could be completed according to well-known techniques in the art and would likely yield predictable results, in that an image of the product that is of a higher resolution than an image of the product obtained from the unified image would be obtained so as to help ensure that the image of the product is clearly represented and of the highest quality possible thereby improving the ability of the combined base device to accurately and reliably identify the product. Therefore, it would have been obvious to combine Skaff et al. in view of Perrella et al. with Medina et al. to obtain the invention as specified in claim 22.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERIC RUSH whose telephone number is (571) 270-3017. The examiner can normally be reached 9am - 5pm Monday - Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270 - 5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ERIC RUSH/Primary Examiner, Art Unit 2677
Prosecution Insights