DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 8-10, and 15-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ramenahalli (PGPUB: 20210166028 A1) in view of LEE (KR 20210092552 A).
Regarding claims 1, 8, and 15. Ramenahalli teaches a system, comprising: a memory; and a processor coupled the memory and configured to:
receive at least one image, wherein the at least one image includes one or more objects (see Fig. 2, paragraph 57, receiving an image. The received image may be a target object image or a cluttered environment image);
process the received at least one image to detect the one or more objects (see Fig. 2, paragraph 69, using marker detection and extraction model to detect and extract markers and labels used by regulatory authorities such as safety marks, quality certifications, and dietary marks from the received image, and output marker descriptors and annotate or assign marker identity and associated product information).
However, Ramenahalli does not expressly teach to display the one or more labels in the received at least one image using the detected one or more objects.
LEE teaches that in order to recognize the ingredients of such products, various object detection techniques may be used. The object detection technique refers to a technique for identifying where (x, y) and in which size (w, h) an object (label) to be detected exists in an image. A general object detection technique detects an object in an image and displays which object it is in the form of a bounding box (see page 4, lines 15-19).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Ramenahalli by LEE to obtain The object detection technique refers to a technique for identifying where (x, y) and in which size (w, h) an object (label) to be detected exists in an image, in order to provide to display the one or more labels in the received at least one image using the detected one or more objects. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
Regarding claims 2, 9, and 16. The combination teaches a method as claimed in claim 1, wherein processing the received at least one image comprises:
assigning a class to one or more objects in the received at least one image;
predicting a bounding box for each of the one or more objects (see Ramenahalli, paragraph 49, classifying or assigning the semantic features to appropriate semantic meaningful categories. Each semantic category is assigned semantic meaning. Semantic feature categorization or classification may be single-label classification into mutually exclusive semantic meaningful categories, or multi-label classification where each semantic feature may be classified or categorized into multiple semantic categories. Semantic classification or categorization may also be hierarchical); and
segmenting the received at least one image based on the bounding box (see Ramenahalli, paragraph 54, Recognizing instances of the target object from a cluttered environment image may include outputting target object recognition data for each instance of the target object recognized in the clutter environment image. The target object recognition data for an instance of the target object recognized may include, but are not limited to, target object information (e.g., object identification, category, description), location, segmentation information (e.g., bounding box enclosing the object), size, orientation, and time stamp for that instance of the target object).
Regarding claims 3, 10, and 17. The method as claimed in claim 2, further comprises:
detecting the one or more objects from the segmented at least one image by inputting the segmented at least one image to a plurality of models (see Ramenahalli, paragraph 100, 107, and 64, generating a bounding box to segment each of the identified proposed instances of the target object in the cluttered environment image; detecting and extracting semantic features from the target object image, an example of which is explained in detail above in reference to FIG. 2. Detecting and extracting semantic features from the target object image does not require the prior performance of generating bounding box described at step 410; the extracted product details may be matched with or assigned to loosely defined super-categories of target objects or products (e.g., any men's shampoo, women's shampoo, 16 oz sized shampoo, shampoo for oily hair, shampoo for normal hair, and shampoo for damaged hair, etc., —will be assigned to the super-category Hair Care Products) stored in a database (e.g., product details database). This allows for the selection of right type of image classification model later, used during the deep CNN model's extraction of features for matching purpose. In some implementations, one or more CNN neural network-based image classification models may be trained or fine-tuned for each of the super-categories); and
generating a word confidence score for the detected one or more objects (see Ramenahalli, Fig. 4, paragraph 113, detecting and extracting perceptual features from the target object image at multiple scales comprising detecting and extracting perceptual features of the target object only from the target object image at multiple scales; step 411 of detecting and extracting semantic features from the target object image comprising detecting and extracting semantic features of only the target object extracted from the target object image; and 4) the above step 413 of for each semantic feature type, matching the feature descriptors of the target object image for that semantic feature type to that of the clutter environment image within each bounding box, and outputting an individual matching score for that semantic feature type comprising for each semantic feature type, matching the feature descriptors of only the target object extracted from the target object image for that semantic feature type to that of the clutter environment image within each bounding box, and outputting an individual matching score for that semantic feature type).
Claim(s) 5-6, 12-13, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ramenahalli (PGPUB: 20210166028 A1) in view of LEE (KR 20210092552 A), and further in view of Ekambaram (20210117628 A1).
Regarding claims 5, 12, and 19. The combination does not expressly teach the method as claimed in claim 3, further comprises:
determining whether the generated word confidence score is above a threshold; and confirming the detected one or more objects as the one or more labels when the generated word confidence score is above the threshold.
Ekambaram teaches that It is noted that a word-embedding representation (e.g., word embedding representation 120) is a representation of a word that shows the relationship between the word and a corpus of other words. For example, the representation may be a word2vec representation, wherein the word is represented as a vector within a vector space, where vectors are positioned in the vector space such that words sharing common contexts in the corpus are located in close proximity within the vector space. At step 408, the belief score is calculated and assigned to the correctness of the user's answer. The belief score is calculated based on the conflict impact between a wordweb edge score and a word-embedding score across words. For example, an annotator may manually test at least part of the wordweb to determine how complete the wordweb is, and then assign a confidence score indicating how complete the wordweb is. According to at least one example embodiment, the wordweb confidence score is a value between 0 and 1 (see paragraph 44); At step 410, an aggregated belief score is determined by aggregating the belief score from step 406 with previously determined belief scores. At step 412, the aggregated belief score is compared to a threshold value. At step 414, if the belief score is below the threshold value then additional dependent questions may be generated and presented to the user, such as to validate the user's answer through other parameters (see paragraph 45).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination by xxx to obtain At step 408, the belief score is calculated and assigned to the correctness of the user's answer. The belief score is calculated based on the conflict impact between a wordweb edge score and a word-embedding score across words. For example, an annotator may manually test at least part of the wordweb to determine how complete the wordweb is, and then assign a confidence score indicating how complete the wordweb is. According to at least one example embodiment, the wordweb confidence score is a value between 0 and 1 and At step 410, an aggregated belief score is determined by aggregating the belief score from step 406 with previously determined belief scores. At step 412, the aggregated belief score is compared to a threshold value. At step 414, if the belief score is below the threshold value then additional dependent questions may be generated and presented to the user, such as to validate the user's answer through other parameters, in order to provide determining whether the generated word confidence score is above a threshold; and confirming the detected one or more objects as the one or more labels when the generated word confidence score is above the threshold. Therefore, combining the elements from prior arts according to known methods and technique would yield predictable results.
Regarding claims 6, 13, and 20. The method as claimed in claim 3, further comprises:
extracting the one or more characters from the detected one or more objects (see Ramenahalli, paragraph 61, the OCR detection and extraction model records the width and height of each character to perform character detection and recognition in the wild, at character, word and sentence level);
identifying the one or more objects by combining the extracted one or more characters (see Ramenahalli, paragraph 61, the OCR detection and extraction model supports multi-lingual character detection and recognition and is capable of detecting characters in any orientation, color, font type, and size); and
generating a character level confidence score for the identified one or more objects (see Ramenahalli, paragraph 63, from the outputted characters, words, and sentences of the OCR detection and extraction model, a product detail detection and extraction model extracts product details. Product details define detailed parameters of products; see Ekambaram, Fig. 4, paragraph 45, At step 410, an aggregated belief score is determined by aggregating the belief score from step 406 with previously determined belief scores. At step 412, the aggregated belief score is compared to a threshold value. At step 414, if the belief score is below the threshold value then additional dependent questions may be generated and presented to the user, such as to validate the user's answer through other parameters).
Allowable Subject Matter
Claims 4, 7, 11, 14, 18, and 21 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIN JIA whose telephone number is (571)270-5536. The examiner can normally be reached 9:00 am-7:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIN JIA/Primary Examiner, Art Unit 2663
Prosecution Insights