DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 3/9/2026 has been entered.
Claims Status
Claims 1-30 and 42 have been cancelled.
Claim 51 is newly added.
Claims 31-41 and 43-51 remain pending and stand rejected.
Claim Interpretation – Claim Terms
The following terms have be interpreted as set forth below:
Overlay element: Though the term overlay is not recited within the specification, it is understood as an AR element that is superimposed on a display of an image (see: Specification, 0010-0011).
Computer vision: using machines to interpret, analyze, and pull meaningful data from images and videos (see https://azure.microsoft.com/en-us/resources/cloud-computing-dictionary/what-is-computer-vision).
Extracted attributes of the relevant object: attributes (e.g., price, brand, etc.) of an object determined from its image (see: 0020-0021).
Extracted from structured content: retrieving data from a database, the data having some structure (e.g. table) (see: 0026, Fig. 1D).
Extracted from historical interaction data: retrieving data from historical data (e.g., within a database) (see: 0027-0028, Fig. 1D).
Timestamp: a date (e.g., when the transaction event occurred) (see: 0027, Fig. 1D).
Computing resources: resources (e.g., processing resources and memory resources) that are needed to determine a content of the AR elements, generate the AR elements, and/or generate the AR user interface. (see: 0011)
Proposed Amendment
The Examiner proposes the amendment below to place the application in condition for allowance. The combination of features proposed are necessary to maintain eligibility under 35 USC 101, overcome the rejection under 35 USC 112(a), and render the claims unobvious over the prior art. Claim 31 is exemplary, and similar amendments should be applied to independent claims 40 and 45. The proposed amendment is as follows:
31. (Currently Amended) A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of an augmented reality (AR) device comprising an image sensor and a display, cause the AR device to:
determine an environment type, based on one or more factors, and whether to proceed with image analysis and perform generation of an AR user interface based on the environment type;
obtain, via the image sensor, real-time image data depicting a physical environment comprising a plurality of physical objects;
for each of the physical objects depicted in the real-time image data, determine an identity and a displayed amount using a combination of computer vision, optical character recognition, and association of the displayed amount based on a spatial relationship between a corresponding one of the physical objects and an information item indicating the displayed amount;
determine, for one or more of the plurality of physical objects, whether a discount is available for each of the one or more of the plurality of physical objects;
process the plurality of physical objects for which an identity and displayed amount have been determined using an AR rendering module to generate a set of candidate AR overlay elements, each candidate AR overlay element corresponding to one of the physical objects;
filter the candidate AR overlay elements by:
(i) applying a relevance determination model trained on historical transaction data or historical web browsing data associated with a user of the AR device, and
(ii) for each candidate AR overlay element associated with a relevant object, comparing one or more extracted attributes of the relevant object to corresponding reference data retrieved from at least one external data source, and discarding each candidate AR overlay element for which the comparison does not satisfy a selection criterion, wherein the selection criteria comprises whether a discount is available for the relevant object;
generate, for each of the remaining candidate AR overlay elements, a graphical overlay comprising:
an indicator of a reference value associated with [[the]] a corresponding physical object,
an indicator of the discount associated with the corresponding physical object, based on determining that the discount is available for the corresponding physical object, and
an interactive element comprising a hyperlink to a webpage for the object, the webpage enabling execution of a transaction for the object;
cause the AR device to render the graphical overlay comprising the indicator of the reference value, the indicator of the discount associated with the corresponding physical object, and the interactive element comprising the hyperlink in a corresponding position in proximity to the corresponding physical object.
51. (Cancelled).
Claim Rejections - 35 USC § 112(a)
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 31-41 and 43-51 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Regarding claim 31, claim 31 has been amended to recite “allocate computing resources to render the remaining candidate AR overlay elements each in a corresponding position in proximity to a corresponding physical object”.
The specification as originally filed does not disclose allocation computing resources to perform rendering. Instead, the invention merely renders the overlays sans any specific method of allocating computing resources. Paragraph 0011 and 0041 indicate the following:
0011: However, if numerous objects are identified in the image, the AR elements may overwhelm the AR user interface, which creates a poor user experience and consumes excessive computing resources (e.g., processing resources and memory resources) that are needed for the user device and/or a server device communicating with the user device to determine a content of the AR elements, generate the AR elements, and/or generate the AR user interface.
Although discussing consumption of resources in relation to use experience, paragraph 0011 does not describe the manner by which “allocate computing resources” is accomplished. The specification merely acts to not include discarded elements in the user interface sans any relation to allocated resources such as processing resources or memory resources.
The subject matter of allocate computing resources is thereby considered new matter that is not adequately described by the original disclosure such that one of ordinary skill in the art would have understood that the claimed invention was possessed at the time of filing.
Dependent claims 32-39 and 51 depend either directly or indirectly from independent claim 31. These claims thereby inherit the deficiencies of claim 31 and are rejected therewith.
Regarding claims 40-41 and 43-44 (method), claims 40-41 and 43-44 recite at least similar limitations and scope as recited in claims 31-39 such that similar analysis would be readily apparent to one of ordinary skill in the art. Namely, claim 40 recites a parallel method to that of CRM claim 31, and fails the written description requirement (lack of possession) for at least similar reasons as discussed above. Claims 41 and 43-44 depend from claim 40 and are rejected therewith.
Regarding claims 45-50, claims 45-50 recite at least similar limitations and scope as recited in claims 31-39 such that similar analysis would be readily apparent to one of ordinary skill in the art. Namely, claim 45 recites a parallel method to that of CRM claim 31, and fails the written description requirement (lack of possession) for at least similar reasons as discussed above. Claims 45-50 depend from claim 40 and are rejected therewith.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 31, 34-39, 40-41, 43-44, 45-47 and 50 is/are rejected under 35 U.S.C. 103 as being unpatentable over Furlan (US 2021/0295047 A1) in view of Aravamudan (US 2012/0176509 A1), Melamed (US 2022/0019912 A1) and Chachek (US 2020/0302510).
Regarding claim 31, Furlan discloses non-transitory computer-readable medium storing instructions that, when executed by one or more processors of an augmented reality (AR) device comprising an image sensor and a display (see: 0048-0049, 0051, Fig. 1-6), cause the AR device to:
determine an environment type, based on one or more factors, and whether to proceed with image analysis and perform generation of an AR user interface based on the environment type (see: 0124-0125, Fig. 7 (702-704), 0207, Fig. 13 (1302));
obtain, via the image sensor, real-time image data depicting a physical environment comprising a plurality of physical objects (see: 0134-0135, Fig. 8 (802-804), 0156, Fig. 10 (1002), Fig. 21);
for each of the physical objects depicted in the real-time image data, determine an identity using a combination of computer vision and other techniques (see: 0171, Fig. 10 (1016), 0177-0178, Fig. 11 (1102-1104));
process the plurality of physical objects for which an identity determined using an AR rendering module to generate a set of candidate AR overlay elements, each candidate AR overlay element corresponding to one of the physical objects (see: 0171, Fig. 10 (1016), 0177-0178, Fig. 11 (1102-1104));
filter the candidate AR overlay elements by:
(i) applying a relevance determination model trained on user data associated with a user of the AR device, and (ii) for each candidate AR overlay element associated with a relevant object, comparing one or more extracted attributes of the relevant object to corresponding reference data retrieved from at least one external data source, and discarding each candidate AR overlay element for which the comparison does not satisfy a selection criterion (see: (see: 0036 (highlight items based on user preferences), 0042 (generate dynamic GUIs that specifically display elements that are relevant for a user search while removing irrelevant elements), 0179, 0181, 0208, 0232, Fig. 11 #1108-1110, Fig. 22, Fig. 24);
Note: From the identified items in the field of view, items of interest are selected based on user preferences or qualification criteria. The items of interest/qualified items are selected and the video feed is modified (e.g., highlighting the selected items based on user preferences). This also includes leaving items that are not of interest/unqualified items unmodified in the feed, or alternatively modifies the unqualified items to a non-highlight color (e.g., graying out) (i.e., discard one or more objects). Note that paragraph 0013 of Applicant’s specification defines “discard” as “not select”.
allocate computing resources to render the remaining candidate AR overlay elements each in a corresponding position in proximity to a corresponding physical object (see: 0036 (highlight items based on user preferences), 0042 (generate dynamic GUIs that specifically display elements that are relevant for a user search while removing irrelevant elements), 0179, 0181, 0208, 0232, Fig. 11 #1118-1120, Fig. 22, Fig. 24);
Note: as understood in light of paragraphs 0033 and 0041 of Applicant’s specification, “allocate computing resources” is analogous to allocating the corresponding position within a display of the AR device to a particular AR overlay element. As discussed above, Furlan discloses selecting and discarding items through a filtering process. Furlan subsequently modifies the display such that corresponding AR overlay elements that have been selected are displayed within the AR device display and in proximity to a respective item. No overlay elements are displayed for the discarded items. In this manner, Furlan discloses allocating computing resources (e.g., display resources) as claimed.
generate, for each of the remaining candidate AR overlay elements, a graphical overlay comprising: an indicator of a reference value associated with the corresponding physical object (see: Fig. 22 (2206A)-C, 0268, 0281), and, an interactive element configured to enable further interaction with the remaining candidate AR overlay element (see: Fig. 22 (2215), 0283, Fig. 24 (2412, 2413, 2415), 0283).
Though disclosing the above including recognizing landmarks and parking lot number within the frame of an AR interface, Furlan does not disclose:
a displayed amount, using optical character recognition, and association of the displayed amount based on a spatial relationship between a corresponding one of the physical objects and an information item indicating the displayed amount.
To this accord, Aravamudan discloses providing comparison data including comparative pricing information from a plurality of entities for the at least one object including determining a displayed amount using optical character recognition and association of the displayed amount based on a spatial relationship between a corresponding one of the physical objects and an information item indicating the displayed amount (see: 0076 (OCR; text labels are text stuck on or imprinted on an object, or external labels adjacent to the object), 0044-0045).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Furlan in view of Melamed to have utilized the known techniques as taught by Aravamudan in order to have facilitated using a camera-based centralized input interface to augment text/speech interfaces and improve efficacy while making such interfaces more useful for acting on objects in the user's immediate surrounding visual environment (see: Aravamudan: 0040, 0076).
Lastly, Furlan discloses the use of user data in determining relevancy but does not expressly disclose historical transaction data or historical web browsing.
To this accord, Melamed discloses a recommendation system that uses a model, such as machine learning functionality, to predict user preferences of users. The recommendation system may take into account various user attributes of a user, such as age, gender, location, browsing history, past purchases, movies and music the user has consumed, and/or prior activities of the user (see: 0036). Notably the model is a trained machine learning model (see: 0037). As such, Melamed teaches analyzing historical transaction data or historical web browsing data.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Furlan to have utilized historical web browsing data and/or pas purchases as taught by Melamed in order to have generated more accurate user preference predictions (see: Melamed: 0041), thereby improving the ability of Furlan to determine preferred items in the field of view.
Lastly, Furlan does not discloses determine, for one or more of the plurality of physical objects, whether a discount is available for each of the one or more of the plurality of physical objects, and,
an indicator of the discount associated with the corresponding physical object, based on determining that the discount is available for the corresponding physical object.
To this accord, Chachek discloses an AR device used during shopping configured to determine, for one or more of the plurality of physical objects, whether a discount is available for each of the one or more of the plurality of physical objects and generate an indicator of the discount associated with the corresponding physical object, based on determining that the discount is available for the corresponding physical object (see: 0093, 0096, 0126, Fig. 7, Fig. 15).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Furlan to have utilized the known technique for indicating availability of discounts via an AR device as taught by Chachek in order to have drawn a user’s attention to the fact that a product is on sale or is available at a discount (see: Chachek: 0093), thereby making the user aware of the discount and increasing the likelihood of sales.
34. The non-transitory computer-readable medium of claim 31, wherein determining whether an object is relevant to the user includes comparing one or more attributes associated with the object to attributes extracted from historical interaction data associated with the user (see: Furlan: 0179, 0181, 0208, 0232, Fig. 11 #1108-1110; Melamed: 0036).
35. The non-transitory computer-readable medium of claim 31, wherein the spatial relationship includes at least one of: a threshold distance between a corresponding one of the physical objects and an information item indicating the displayed amount, or based on a relative position of the information item with respect to the corresponding one of the physical objects (see: Aravamudan: 0076).
36. The non-transitory computer-readable medium of claim 31, wherein each of the remaining candidate AR overlay elements includes a visual indicator representing a comparison between multiple reference values associated with the corresponding physical object (see: Furlan: Fig. 22 (2206A-C: Price & Monthly Payment), 0245).
37. The non-transitory computer-readable medium of claim 31, wherein the AR device is configured to proceed with generation of the AR user interface if the environment type is a commercial environment (see: Furlan: 0124-0125, Fig. 7 (702-704), 0207, Fig. 13 (1302)).
38. The non-transitory computer-readable medium of claim 31, wherein the AR device uses optical character recognition to extract alphanumeric content from the image data and maps the extracted content to physical object profiles (see: Furlan: 0171, Fig. 10 (1016), 0177-0178, Fig. 11 (1102-1104)); Aravamudan: 0044, 0072, 0076).
39. The non-transitory computer-readable medium of claim 31, wherein the AR device utilizes location data to identify a geographic location of the AR device, and to identify an entity associated with the geographic location of the AR device (see: Furlan: 0124-0125, Fig. 7 (702-704), 0207, Fig. 13 (1302); Aravamudan: 0004 (user’s location), 0033 (location of the mobile device), 0038).
Regarding claim 40, claim 40 recites substantially similar limitations and scope as recited in claim 1 such that similar analysis would be apparent to one of ordinary skill in the art. Accordingly, claim 40 is rejected under at least similar rationale.
41. The method of claim 40, wherein generating the graphical overlay further comprises generating an input element for display in association with the corresponding object as identified in the image data (see: Furlan: Fig. 22 (2206A)-C, 0268, 0281), and, an interactive element configured to enable further interaction with the remaining candidate AR overlay element (see: Furlan: Fig. 22 (2215), 0283, Fig. 24 (2412, 2413, 2415)).
43. The method of claim 40, wherein the predefined selection criterion for discarding candidate AR overlay elements is dynamically adjusted based on contextual data derived from the historical transaction data or web browsing data (see: Melamed: 0036-0037, Fig. 4 (406, 408)).
44. The method of claim 40, wherein the overlay position is in proximity to a corresponding object (see: Furlan: Fig. 22 (2215), Fig. 24 (2412, 2413, 2415), 0281, 0283; Aravamudan: Fig. 7 (704a-c)).
Regarding claim 45, claim 45 recites substantially similar limitations and scope as recited in claim 1 (as well as claim 40) such that similar analysis would be apparent to one of ordinary skill in the art. Accordingly, claim 45 is rejected under at least similar rationale.
Regarding claims 46-47, claims 46-47 recites substantially similar limitations and scope as recited in claims 41 and 43 such that similar analysis would be apparent to one of ordinary skill in the art. Accordingly, claims 46-47 are rejected under at least similar rationale.
50. The apparatus of claim 45, wherein the spatial relationship includes at least one of: a threshold distance between a corresponding one of the objects and an information item indicating the displayed amount, or based on a relative position of the information item with respect to the corresponding one of the objects (see: Aravamudan: 0076 (text labels that are stuck on or imprinted on an object, or external labels adjacent to the object).
Claim(s) 32 and 48 is/are rejected under 35 U.S.C. 103 as being unpatentable over Furlan in view of Aravamudan, Melamed and Chachek as applied to claims 31 and 45 above, and further in view of He (US 2014/0351091).
Regarding claim 32 and parallel claim 48, Furlan in view of Aravamudan, Melamed and Chachek teaches all of the above including wherein the reference data retrieved from the at least one external data source is obtained from a plurality of data sources (see: Furlan: 0047, 0111, 0125, Fig. 1 #140; see also: Aravamudan: Fig. 7 (704s-c), 0072), but does not disclose wherein the reference data is explicitly scraped from one or more web pages. Such techniques were notoriously well know in the art before the effective filing date of the invention.
For example, He teaches the known technique of scraping from one or more web pages (see: 0024, 0037).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Furlan in view of Aravamudan, Melamed and Chachek to have utilized the known technique for scraping data as taught by He in order to have provided more accurate data for comparing items by using information collected for the same item from different websites at substantially the same time (see: He: 0003).
Claim(s) 33 and 49 is/are rejected under 35 U.S.C. 103 as being unpatentable over Furlan in view of Aravamudan, Melamed and Chachek as applied to claims 31 and 45 above, and further in view of Tavernier (US 10,706,450).
Regarding claim 33 and parallel claim 49, Furlan in view of Aravamudan, Melamed and Chachek teaches all of the above as noted but does not teach wherein the historical transaction data or historical web browsing data includes a timestamp associated with each event, and determining relevance is based at least in part on temporal proximity of events. Such techniques were notoriously well known in the art before the effective filing date of the invention.
For example, teaches historical transaction data or historical web browsing data includes a timestamp associated with each event, and determining relevance is based at least in part on temporal proximity of events (see: col. 16 line 48-col. 17 line 16, ), such as by storing time data (including timestamp data) in a user interaction data repository and utilizing the user interaction data to determine user intent, and filtering/generating recommendations based on the intent.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Furlan in view of Aravamudan, Melamed and Chachek to have utilized the known technique for utilizing time data (timestamp data) as taught by Tavernier in order to have provided recommendations that would have better aligned with a user's intent and thus have been more likely to be relevant to the user (see: Tavernier: col. 4 lines 19-34).
Claim Objection – Subject Matter Over the Prior Art
Claim 51 is rejected on other grounds but objected to as being dependent upon a rejected base claim, and would be allowable over the prior art if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Response to Amendment/Arguments
I. Applicant’s arguments made with respect to the rejection under 35 USC 112(a) have been fully considered in view of the accompanying amendment, but are not persuasive.
Although the amendments remove bypassing, the specification lacks adequate support for allocate computing resources as discussed above. Accordingly, the rejection has been maintained for the reasons explained above.
II. Applicant’s arguments made with respect to the rejection under 35 USC 103 have been fully considered in view of the accompanying amendment, but are moot in view of new grounds of rejection. Applicant’s amendment necessitated the new grounds of rejection (above).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Abhishek (US 20190332864) discloses surfacing coupon alerts during in-store shopping (see: 0052, 0035, Fig. 1D (128), Fig. 1F (132))
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM J ALLEN whose telephone number is (571)272-1443. The examiner can normally be reached Monday-Friday, 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anita Coupe can be reached at 571-270-3614. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM J ALLEN/Primary Examiner, Art Unit 3619
WILLIAM J. ALLEN
Primary Examiner
Art Unit 3625
Prosecution Insights