DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 1/7/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered and attached by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 5-6, 8-10, 12-13, 15-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Vanapalli (U.S. Patent Pub. No. 2021/0304123) in view of Balasubramaniam (U.S. Patent Pub. No. 2021/0097567).
Regarding Claim 1, Vanapalli teaches a system configured to execute an attribute recognition workflow, comprising: a computer, comprising a processor and memory, and configured to (¶12-13 database server executed by one or more storage devices for storing image data comprising real-time inventory images and historical images of each product type; a data analytics system executed by one or more computers)
receive one or more labelled images (¶70 For example, the semantic category of pixels in the training images can be obtained by user labeling;)
train a deep learning model using the one or more labelled images to recognize one or more products and one or more product attributes, wherein the deep learning model comprises (¶70 the image semantic segmentation is achieved based on convolutional neural networks (CNNs). To this end, the data analytics system 110 is trained using the training data stored in the database server 120 or from outside database. The training data includes a plurality of training images. The semantic category of each pixel in each training image is determined in advance:)
a feature extraction network configured to be applied to the received one or more labelled images to extract the one or more product attributes (¶71 FIG. 2 shows a block diagram of the convolutional neural networks in accordance with example implementations of the subject matter described herein; ¶72 The convolution filters in the first convolutional layer apply convolution filtering on the input image to be segmented. The results are referred to “convolutional feature maps” or “feature maps.” The feature maps record the strength and spatial locations of the responses of the filters on the respective channels.)
a region proposal network configured to generate one or more regions of interest containing the one or more products; and (¶81 the segmentation module includes a pooling layer 211. The pooling layer 211 receives and pools the segment features generated by the CFM layers. As known, the spatial pooling combines the responses of features obtained at nearby locations into some statistic that summarizes the joint distribution of the feature over the region of interests;)
generate one or more bounding boxes (¶77 The binary mask may the foreground mask and enclosing bounding box. The binary mask is defined by the bounding box of the candidate segment;)
receive media containing one or more additional products; apply the trained deep learning model to identify the one or more additional products and one or more additional product attributes; and (¶89 The object classification component can query or compare the object against a convolutional neural network (CNN), recurrent neural network (RNN), other artificial neural network (ANN), or against a spatio-temporal memory network (that can be collectively referred to as a deep neural network (DNN)) that has been previously trained, for example to recognize different products)
transmit the identified one or more additional products and the identified one or more additional product attributes to a retail planning system (¶113 The present invention addresses these challenges through application of graphical representation of image processing tasks. FIG. 5 illustrates an exemplary workflow for retail shelf analytics.)
Vanapalli does not explicitly disclose wherein the deep learning model comprises a detection network configured to receive input from the feature extraction network and the region proposal network and to generate one or more bounding boxes for the one or more products.
Balasubramaniam is in the same field of art of image analysis. Further, Balasubramaniam teaches wherein the deep learning model comprises a detection network configured to receive input from the feature extraction network and the region proposal network and to generate one or more bounding boxes for the one or more products (¶144 The CNN generates convolutional feature maps. Regions of proposals are extracted using the convolutional feature maps. A region of interest (ROI) pooling layer is used to reshape the proposed regions into a fixed size so that it can be fed into a fully connected network. Each region is passed on to a fully connected network. A softmax layer is used on top of the fully connected network to output classes. Along with the softmax layer, a linear regression layer is also used in parallel to output bounding box coordinates for predicted classes;)
receive media containing one or more additional products; apply the trained deep learning model to identify the one or more additional products and one or more additional product attributes; (¶305 The shopping container image processor 358 may implement a machine learning model trained using an item dataset. The trained model can detect an object and generate detected item data 312 (e.g. an item identity) for the detected object)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Vanapalli by generating bounding boxes for the products that is taught by Balasubramaniam; thus, one of ordinary skilled in the art would be motivated to combine the references to improve performance (Balasubramaniam ¶47).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 2, Vanapalli in view of Balasubramaniam discloses the system of Claim 1, wherein the one or more labelled images received for training each comprise a bounding box (Vanapalli, ¶110 The bounding-box may be provided when the recurrent neural network is trained with an action class label but not necessarily.)
Regarding Claim 3, Vanapalli in view of Balasubramaniam discloses the system of Claim 1, wherein the computer is further configured to: convert, using the feature extraction network, the received media into one or more feature maps, wherein the one or more feature maps are each at a lower resolution than the received media (Vanapalli, ¶72 The convolution filters in the first convolutional layer apply convolution filtering on the input image to be segmented. The results are referred to “convolutional feature maps” or “feature maps.” The feature maps record the strength and spatial locations of the responses of the filters on the respective channels; the act of convolution lowers the resolution of the input.)
Regarding Claim 5, Vanapalli in view of Balasubramaniam discloses the system of Claim 1, wherein the received media comprises one or more of: one or more images, one or more videos and one or more social media feeds (Vanapalli, ¶8 The present invention provides a method and system to automatically analyze image or video data to identify all the products caught by the image or video and output product attributes.)
Regarding Claim 6, Vanapalli in view of Balasubramaniam discloses the system of Claim 1, wherein the detection network comprises a regression layer and a classification layer (Vanapalli, ¶85 The output of the second convolutional layer 213 is fed to into a perceptron layer 214. The perceptron layer 214 is suitable for classification prediction problems where inputs are assigned a class or label. The perceptron layer 214 are also suitable for regression prediction problems where a real-valued quantity is predicted given a set of inputs.); (Balasubramaniam, ¶144 A softmax layer is used on top of the fully connected network to output classes. Along with the softmax layer, a linear regression layer is also used in parallel to output bounding box coordinates for predicted classes. Fast CNN uses a single model which extracts features from the regions, divides them into different classes, and returns the boundary boxes for the identified classes simultaneously.)
Regarding claim 8, claim 8 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Vanapalli further teaching on: A method executed by an attribute recognition workflow (Vanapalli, Claim 9: A method for identifying a product in an image)
Claim 9 recites limitations similar to claim 2 and is rejected under the same rationale and reasoning.
Claim 10 recites limitations similar to claim 3 and is rejected under the same rationale and reasoning.
Claim 12 recites limitations similar to claim 5 and is rejected under the same rationale and reasoning.
Claim 13 recites limitations similar to claim 6 and is rejected under the same rationale and reasoning.
Regarding claim 15, claim 15 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Balasubramaniam further teaching on: A non-transitory computer-readable medium embodied with software to execute an attribute recognition workflow (Balasubramaniam ¶56 The applications, computer readable instructions or programs may be stored in memory or in secondary storage or may be received from the Internet or other network 20.)
Claim 16 recites limitations similar to claim 2 and is rejected under the same rationale and reasoning.
Claim 17 recites limitations similar to claim 3 and is rejected under the same rationale and reasoning.
Claim 19 recites limitations similar to claim 5 and is rejected under the same rationale and reasoning.
Claim 20 recites limitations similar to claim 6 and is rejected under the same rationale and reasoning.
Claims 4, 11, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Vanapalli (U.S. Patent Pub. No. 2021/0304123) in view of Balasubramaniam (U.S. Patent Pub. No. 2021/0097567) in view of Roh (U.S. Patent Pub. No. 2018/0165551).
Regarding Claim 4, Vanapalli in view of Balasubramaniam teaches the system of Claim 1.
Vanapalli in view of Balasubramaniam does not explicitly disclose wherein the one or more bounding boxes each comprise spatial coordinates and a probability score indicating a presence of a product
Roh is in the same field of art of image analysis. Further, Roh teaches wherein the one or more bounding boxes each comprise spatial coordinates and a probability score indicating a presence of a product (¶32 the computing device 100 executes the multi-scale RPN 212 to generate a region proposal from each RPN layer 214. In some embodiments, in block 414 the computing device 100 may generate 2k objectness scores and 4k proposed object coordinates for each sliding window position of each RPN layer 214. Each of the objectness scores estimates the probability of object or not object for each region proposal, and each of the proposed object coordinates encodes the coordinates of the bounding box of the proposed object (i.e., coordinates of the four corners).)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Vanapalli in view of Balasubramaniam by using bounding boxes with coordinates and a probability score that is taught by Roh; thus, one of ordinary skilled in the art would be motivated to combine the references to boost object detection accuracy for the same computational cost required for previous approaches (Roh ¶15).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 11 recites limitations similar to claim 4 and is rejected under the same rationale and reasoning.
Claim 18 recites limitations similar to claim 4 and is rejected under the same rationale and reasoning.
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Vanapalli (U.S. Patent Pub. No. 2021/0304123) in view of Balasubramaniam (U.S. Patent Pub. No. 2021/0097567) in view of Polania Cabrera (U.S. Patent Pub. No. 2021/0110457).
Regarding Claim 7, Vanapalli in view of Balasubramaniam teaches the system of Claim 1.
Vanapalli in view of Balasubramaniam does not explicitly disclose wherein the computer is further configured to: train the deep learning model for a quantity of epochs.
Polania Cabrera is in the same field of art of image analysis. Further, Polania Cabrera teaches wherein the computer is further configured to: train the deep learning model for a quantity of epochs (¶63 Hyperparameters of the optimization process may be predetermined, including a number of epochs (e.g., a number of complete passes through a training dataset such as the first dataset 402 or the second dataset 404) and a batch size (e.g., a number of training samples to process before the internal parameters of the machine-learning model 200 are updated).)
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Vanapalli in view of Balasubramaniam by training for a quantity of epochs that is taught by Polania Cabrera ; thus, one of ordinary skilled in the art would be motivated to combine the references for an optimized model (Polania Cabrera ¶63).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 14 recites limitations similar to claim 7 and is rejected under the same rationale and reasoning.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUSTIN BILODEAU whose telephone number is (571)272-1032. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUSTIN BILODEAU/Examiner, Art Unit 2664
Prosecution Insights