Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 7-17, and 19-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Greenberger (US 11282133 B2), and further in view of Ravichandran (US 10776417 B1).
Regarding claim 1, Greenberger teaches a processor-implemented method comprising:
obtaining, by a processor associated with an augmented reality device, attribute data associated with a current object in a view of the augmented reality device (col. 10, lines 20-28: “Embodiments of the visual recognition system 124 may record or stream visual images and video data from an input device of the visual recognition system 124 (such as a camera) to the visual recognition module 108. The visual recognition module 108 may process the recorded or streamed visual images and/or video data to identify one or more objects 401 that may be the focus of the user 301 physically controlling the positioning of the visual recognition system's input.”); and
generating, by the processor, an augmented reality graphical user interface displaying a virtual representation of the attribute of the comparison object in the view of the current object (col. 12, lines 16-20: “Referring to the drawings, FIG. 4a depicts an embodiment of a HUD being overlaid onto a display device 110 by providing a GUI capable or presenting product information 405 dynamically to the user or in a manner controlled by the user 301.”).
Greenberger fails to teach, responsive to receiving an instruction indicating an attribute for a comparison of the current object with a comparison object,
extracting, by the processor, a first feature vector for the current object based upon the attribute indicated by the instruction;
generating, by the processor, a similarity score for a potential comparison object based upon comparing the first feature vector extracted using the attribute data of the current object against a second feature vector of the potential comparison object for the attribute indicated by the instruction; and
identifying, by the processor, the potential comparison object as the comparison object based upon the similarity score of the potential comparison object for the attribute indicated by the instruction.
Ravichandran teaches a processor-implemented method comprising:
responsive to receiving an instruction indicating an attribute for a comparison of the current object with a comparison object,
extracting, by the processor, a first feature vector for the current object based upon the attribute indicated by the instruction (col. 2, lines 52-56: “The patches of the image that correspond to the respective visual attributes can then be processed using various feature extraction techniques to determine the feature values (e.g., feature vector) of the respective visual attributes.”);
generating, by the processor, a similarity score for a potential comparison object based upon comparing the first feature vector extracted using the attribute data of the current object against a second feature vector of the potential comparison object for the attribute indicated by the instruction (col. 14, lines 6-16: “Second feature vectors corresponding to the second visual attribute are then determined 706b for the plurality of items. Thus, second attribute similarity scores can be determined 708b for the items by comparing the second feature vectors of the items to the second query feature vector. After the first attribute similarity scores and the second attribute similarity scores are determined for the items, overall similarity scores can be determined 710 for the items based on the first and second attribute similarity scores.”); and
identifying, by the processor, the potential comparison object as the comparison object based upon the similarity score of the potential comparison object for the attribute indicated by the instruction (col. 14, lines 16-26: “In some embodiments, the first and second similarity scores of an individual item are aggregated to produce the overall similarity score for the item. In some embodiments, the aggregation may utilize a weighting function. The weighting function may define [how much] the first and second similarity scores contribute, respectively, to the overall similarity score. Items from the plurality of items can then be selected 712 to be displayed as search results based on the over similarity scores. In some embodiments, the items may be ranked based on their overall similarity scores.”).
It would have been obvious to one familiar in the art prior to the effective filing date of the claimed invention to integrate the parts-based visual similarity search of Ravichandran into the augmented reality product comparison of Greenberger, as both are in the same field of endeavor of product comparison. The software of Ravichandran could easily be integrated into the hardware of Greenberger, and doing so would prove obviously beneficial, as it would enable a user of Greenberger’s technology to search for similar products to that which they were able to observe in person for comparison purposes.
Regarding claim 2, Greenberger and Ravichandran teach the method of claim 1. Greenberger further teaches wherein generating the augmented reality graphical user interface includes generating, by the processor, an augmented reality overlay for the virtual representation of the attribute to be displayed in the augmented reality graphical user interface, wherein the processor generates the augmented reality overlay based upon the attribute indicated by the instruction (col. 4, lines 19-24: “The HUD may display each of the features of the identified objects in some embodiments, while in other embodiments, the user may selectively view individual features one or more at a time, wherein the augmented display highlights and describes each individual features within in the augmented display.”).
Regarding claim 3, Greenberger and Ravichandran teach the method of claim 2. Greenberger further teaches generating, by the processor, a second augmented reality overlay for the virtual representation of a second attribute of at least one of the comparison object or the current object (col. 4, lines 19-24, as above); and
updating, by the processor, the augmented reality graphical user interface to display a second virtual representation of the second attribute using the second augmented reality overlay for the second attribute (col. 4, lines 19-24, as above).
Regarding claim 4, Greenberger and Ravichandran teach the method of claim 1. Ravichandran further teaches wherein the attribute data includes one or more attributes, including at least one of: a dimension attribute, a text attribute, or an object type (col. 5, lines 34-43: “FIG. 2A illustrates example representations 200 of parts-based visual similarity search, in accordance with various embodiments. A query image 202 of an item 204, such as a dress, may be displayed to a user along with a plurality of visual attributes 206a-e associated with the item 204. In this example, the dress has a sleeve style attribute 206a, a neckline attribute 206b, a cut attribute 206c, hemline attribute 206d, and a color attribute 206e. In other embodiments, the item 204 may have more, fewer, and/or different visual attributes.” NOTE: These are all dimensional/type attributes. However, the phrase “more, fewer, and/or different visual attributes” implies that text attributes are also possible. It is well-known in the art, for example, that many articles of clothing may have text on them. It would be obvious to one familiar in the art to consider a text attribute as a potential vector for comparison in the method taught by Ravichandran.).
Regarding claim 5, Greenberger and Ravichandran teach the method of claim 1. Greenberger further teaches wherein the instruction comprises a user gesture indicating the current object for the comparison against the comparison object (col. 4, lines 63-67: “As the user view toggles through the product information via computer input devices, peripherals, voice commands or hand gestures, the HUD may highlight and feature information directed toward the specific features selected by the user, in some embodiments.”; col. 9, lines 40-49: “Embodiments of the augmented display system 103 may include an AR comparison module 105 which may be responsible for recognizing objects 401 as products, retrieving product information, displaying the product information within a HUD overlaying the product being viewed in real time, comparing the product to similar products previously viewed and making recommendations to a user 301 of the augmented display device 103 based on the comparison between the current product and previously viewed products.”).
Regarding claim 7, Greenberger and Ravichandran teach the method of claim 1. Ravichandran further teaches extracting, by the processor, one or more attributes of the attribute data for the current object by applying an object recognition engine on image data for the current object (col. 11, lines 53-61: “As mentioned, for CNN-based approaches there can be pairs of images submitted that are classified by a type of attribute, while for GAN-based approaches a series of images may be submitted for training that may include metadata or other information useful in classifying one or more aspects of each image. For example, a CNN may be trained to perform object recognition using images of different types of objects, then learn how the attributes relate to those objects using the provided training data.”).
Regarding claim 8, Greenberger and Ravichandran teach the method of claim 1. Ravichandran further teaches:
identifying, by the processor, one or more attributes of the attribute data of the current object (col. 2, lines 36-39: “In order to create an electronic catalog of items that is searchable by parts-based visual attributes, the visual attributes are identified and extracted from the image data of each item.”); and
determining, by the processor, the comparison object having the one or more attributes in the attribute data of the comparison object (col. 14, lines 7-12: “Second feature vectors corresponding to the second visual attribute are then determined 706b for the plurality of items. Thus, second attribute similarity scores can be determined 708b for the items by comparing the second feature vectors of the items to the second query feature vector.”).
Regarding claim 9, Greenberger and Ravichandran teach the method of claim 8. Ravichandran further teaches wherein the processor compares the one or more attributes of the current object against the one or more attributes of the comparison object in response to identifying the one or more attributes of the current object (col. 14, lines 7-12, as above).
Regarding claim 10, Greenberger and Ravichandran teach the method of claim 1. Greenberger further teaches wherein identifying the comparison object includes applying, by the processor, an object recognition engine on image data of the current object to extract one or more attributes of the attribute data for the current object (col. 10, lines 54-59: “Some object recognition techniques may implement a variety of learning models including feature extraction, machine learning models, deep learning models, Bag-of-words models such as SURF and MSER, derivative based matching approaches, the Viola-Jones algorithm, template matching, image segmentation and/or blob analysis.”).
Ravichandran further teaches wherein identifying the comparison object includes applying, by the processor, a selection engine on the attribute of the current object to extract the first feature vector representing the attribute data for the current object satisfying a threshold similarity score to the second feature vector representing the attribute data for the comparison object (col. 9, line 65 – col. 10, line 4: “In some embodiments, the similarity score is the rank position, the distance, or another measure derived from the rank or distance. A similarity score can be determined using the above-described technique for each item with respect to each of the selected attributes. In some embodiments, the items can be ranked for each selected attribute based on the respective similarity scores for the individual attributes.”).
Regarding claim 11, Greenberger and Ravichandran teach the method of claim 1. Greenberger further teaches querying, by the processor, a database of comparison objects previously-viewed by the augmented reality device (col. 11, line 48 – col. 12, line 2: “The product information 405 retrieved by the product module 113 may also be cataloged and registered in a product database or other data structure for quick retrieval of the product information 405 at a later point in time if needed. The entry of the product information 405 may be tagged with keywords, Meta tags, identifiers and a retrieval date which may allow for quick searches and retrievals of desired product information 405. The product module 113 may in some instances receive a system call from the visual recognition module 108 requesting the product module 113 to search the database of product information 405 for product information pertaining to similar products to the objects 401 currently being viewed by the user 301 of the augmented display system 103. The product module 113 may query the database of stored product information 405 for similar keywords, categories and meta tags, identify similar products (which may be sorted by retrieval date, i.e. the date the original object 401 associated with the product information 405 was viewed) and return the product information 405 to the HUD module 115 for augmented display on the display device 110 over the images and/or video data provided to the display device 110 by the visual recognition system 124.”).
Regarding claim 12, Greenberger and Ravichandran teach the method of claim 1. Greenberger further teaches wherein the processor continually captures image data for a plurality of current objects and continually applies an object recognition engine on the image data for the plurality of current objects (col. 11, lines 10-27: “A display controller 109 may be an electrical circuit that may perform the task or function of actuating the display device 110 and deliver the stream of images or video data recorded by the visual recognition system 124 to the display device 110. Upon initiating the visual recognition system 124, the image and video data may be transmitted from the visual recognition system's camera(s) to the visual recognition module 108 and/or the display controller 109. The display controller 109 may deliver the frames content of the images and video being recorded by the visual recognition system 124 to the display device 110, allowing for the user 301 of the augmented display system 103 to view the images and/or video data as the data is being inputted into the camera system of the visual recognition system 124. Thus, allowing for a real-time viewing of the image and video data of objects 401 being viewed in real life or as visual representations 501 on the display device 110 of the augmented display system 103.”).
Regarding claim 13, Greenberger and Ravichandran teach the method of claim 1. Greenberger further teaches wherein the augmented reality graphical user interface includes an augmented reality overlay of the virtual representation of the attribute of the comparison object in proximity to the attribute of the current object (col. 8, lines 50-67: “The augmented display system 103 may be responsible for detecting the presence of an object 401 being viewed, identifying the object 401 as a product that may be available for sale, retrieving product information associated with the object 104, overlaying the product information onto a HUD depicting the product information and features of the currently viewed product while simultaneously viewing the object 401 in real life, logging the viewing of the product for subsequent reference at a later date to other similarly viewed products, querying previously viewed objects 401, comparing and contrasting the currently product features of a viewed object 401 with product features of other similar object 401 that may have been previously viewed, displaying the compared and contrasted features onto the HUD of the augmented display system 103 and providing recommendations to the user 301 of the augmented display system 103.”).
Regarding claim 14, Greenberger and Ravichandran teach the method of claim 1. Greenberger further teaches wherein the instruction indicating the attribute for the comparison includes a verbal instruction (col. 14, lines 23-28: “In some embodiments of the augmented display system 103, the AR comparison module 105 may comprise a speech recognition module 117. Embodiments of the speech recognition module 117 may access, control, receive and process voice commands in the form of audio data recorded by the audio recording system 126.”).
Claim 15 is substantially similar to claim 1, save that it teaches a system rather than a method. As such, it is rejected on a similar basis as claim 1.
Claim 16 is substantially similar to claims 2 and 3, save that it depends on claim 15 rather than claim 1. As such, it is rejected on a similar basis as claims 2 and 3.
Claim 17 is substantially similar to claim 5, save that it depends on claim 15 rather than claim 1. As such, it is rejected on a similar basis as claim 5.
Claim 19 is substantially similar to claim 8, save that it depends on claim 15 rather than claim 1. As such, it is rejected on a similar basis as claim 8.
Claim 20 is substantially similar to claim 9, save that it depends on claim 15 rather than claim 1. As such, it is rejected on a similar basis as claim 9.
Claim 21 is substantially similar to claim 1, save that it teaches a non-transitory machine-readable storage medium rather than a method. As such, it is rejected on a similar basis as claim 1.
Claim(s) 6 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Greenberger (US 11282133 B2), and Ravichandran (US 10776417 B1) as applied to claim 1 above, and further in view of Weston (US 20200349249 A1).
Regarding claim 6, Greenberger and Ravichandran teach the method of claim 1. Greenberger further teaches an object recognition engine (col. 10, lines 42-46: “In some embodiments, the visual recognition module 108 may include object recognition software that may implement algorithms and mechanisms that allow for the augmented display system 103 to identify the objects 401 being viewed by the visual recognition system 124.”), but fails to teach:
identifying, by the processor, a text attribute of the current object in image data for the current object by applying an object recognition engine; and
recognizing, by the processor, the text in the text attribute of the current object.
Weston teaches identifying, by the processor, a text attribute of the current object in image data for the current object by applying an object recognition engine (par. 0084: “The recognition component 304A can include object recognition component 308A to perform object recognition analysis on the received image data 102 to identify and/or characterize (e.g., describe features thereof) one or more objects included in received image data 102. The recognition component 304A can include text recognition component 310A to perform text recognition analysis on the received image data 102 to determine text (e.g., words, phrases, sentences, etc.) included in the image data.”); and
recognizing, by the processor, the text in the text attribute of the current object (par. 0084, as above).
It would have been obvious to one familiar in the art to incorporate the text attribute identification of Weston into the product comparison method of Greenberger, as both are in the same field of endeavor of augmented-reality object recognition. Doing so would be obviously beneficial to Greenberger’s invention, since text frequently provides valuable details about an object that can be used to inform one’s purchasing decisions.
Claim 18 is substantially similar to claim 6, save that it depends on claim 15 rather than claim 1. As such, it is rejected on a similar basis as claim 6.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1 and 15 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN A BARHAM whose telephone number is (571)272-4338. The examiner can normally be reached Mon-Fri, 8:30am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu, can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN ALLEN BARHAM/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights