DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed March 5, 2026 has been entered. Claims 1-18 and 20-21 remain pending in the application.
Response to Arguments
Applicant’s arguments, see Pages 9-11 of Remarks, filed March 5, 2026, with respect to the rejection(s) of claim(s) 1-18 and 20-21 under 35 USC 103 have been fully considered and are persuasive. The examiner agrees with the applicant that the references Spivack and Chojnacka does not teach the claim limitation: “the one or more listed items listed for sale by the one or more listing servers”. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Champavannarth et al. (US 20220405813 A1), which was used to reject claim 10 in the non-final office action filed November 5, 2025. Champavannarth teaches the one or more listed items listed for sale by the one or more listing servers (Paragraph 0043, 0051 – “The price comparison result screen 500 also includes, for the second entity (e.g., “Retailer B”), an item number 520, a current (or sale) price 522, an entity identifier 524, store number 526, in stock quantity 528, and store location 530…The price comparison client application or server application can extract (or “scrape”) price, quantity, and/or other information from the website of the second entity and provide the information to the price comparison application for display to the user”; Note: the entity/item is listed for sale on a website that is accessed by the application. The server that hosts the website listing the entities is equivalent to the listing server. It is implied that there is a server hosting the website because a public website cannot exist without a server). See further details below in the 103 rejection.
Drawings
The drawings are objected to because in Fig. 4, “224” should read “214”. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: “516” in Fig. 5. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities: In paragraph 0039 line 2, perhaps “national language model” should read “natural language model”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-6, 8, 10-11, 13-16, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Spivack et al. (US 20210074068 A1) in view of Chojnacka et al. (US 20200342668 A1) and Champavannarth et al. (US 20220405813 A1), hereinafter Spivack, Chojnacka, and Champavannarth respectively.
Regarding claim 1, Spivack teaches a method (Paragraph 0098 – “Embodiments of the present disclosure further include systems, methods and apparatuses of seamless integration of augmented, alternate, virtual, and/or mixed realities with physical realities for enhancement of web, mobile and/or other digital experiences”) comprising:
capturing, by a computing device, image data of a physical environment for displaying an augmented reality (AR) of the physical environment as an AR environment (Fig. 2C, Paragraph 0035, 0227 – “client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) having AR/VR software agent/hardware module having sensors (e.g., the camera and mic and other sensors) that are always on, and the system is always aware of what is happening…a streaming video camera at the physical location of each device or apparatus controlled by AR/VR window manager 380 so there is always a feed of the AR activity from various perspectives of the physical location”; Note: image data of a physical space is captured by the client device, which is equivalent to the computing device. Fig. 2C shows how the AR of the physical environment is displayed; see screenshot of Fig. 2C below);
PNG
media_image1.png
392
550
media_image1.png
Greyscale
Screenshot of Fig. 2C (taken from Spivack)
rendering a virtual object in the AR environment as an AR assistant (Paragraph 0132, 0145, 0186 – “The digital assistant avatar repository 124 can store virtual content or digital assistant avatars that can be retrieved for consumption or deployment in a target environment, where the virtual content or digital assistant avatars are contextually relevant, required, or requested.… the content associated with the virtual object which may be an avatar can include a text message, a post, a chat, a conversation or a graphical message…The object or virtual object is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment”; Note: the virtual object, which is a digital assistant, is rendered into the AR environment) for accessing one or more servers (Paragraph 0173-0174, 0208-0209 – “The host server 300 includes a network interface 302, a virtual object (VOB) sharing/publication engine 310, a communications manager 340, a digital assistant manager 350, a virtual billboard engine 360…The host server 300 is also coupled to a billboard object repository 322…The digital assistant manager 350 can send a VOB message with a 3D emoji…the digital assistant manager 350 is also able to represent knowledge to the user and is able to perceive the real world environment and/or the AR environment as its user”; Note: the digital assistant is able to send VOB messages, which implies that the digital assistant can be used to access the host server since the host server contains the VOB sharing engine, virtual billboard engine, and is coupled to the billboard object repository);
and responsive to detecting a user interaction with the AR assistant in the AR environment, generating an output using information associated with one or more items (Paragraph 0204, 0214-0215 – “the digital assistant is able to receive a command (e.g., by the command processing engine 352), either explicitly or implicitly, from a user… The virtual billboard can include one or more of, a note, a review, an offer, an ad, a promotion. A virtual billboard object can be a seed for threaded discussions or interactions. In general, virtual billboards can be placed in space, pinned to a user or a moveable object…the virtual billboard engine 360 can create the virtual billboard responsive to a request of a creator user”; Note: a user can request a virtual billboard, through a command. The virtual billboard is an output that shows information related to an item).
Spivack does not teach identifying one or more physical items in the physical environment based at least in part on the image data; and communicating with one or more listing servers over a network to identify one or more listed items related to the one or more physical items, the one or more listed items listed for sale by the one or more listing servers. Spivack also does not teach the “one or more listing servers” in the limitation: “rendering a virtual object in the AR environment as an AR assistant for accessing the one or more listing servers”, nor the “one or more listed items” in the limitation: “responsive to detecting a user interaction with the AR assistant in the AR environment, generating an output using information associated with the one or more listed items”. However, Chojnacka teaches identifying one or more physical items in the physical environment based at least in part on the image data (Paragraph 0025 – “As the images of the physical environment 100A are streamed, and the image frames are fed into the recognition algorithm or model, 3D pose and location information related to real objects in the physical environment 100A may be detected. Images of the real objects may be correlated against images stored in an image database, to identify the detected real objects”); and communicating with one or more listing servers over a network (Paragraph 0028, 0046, 0066-0067 – “[items] may be identified or recognized based on comparison and/or matching with images of previously identified items in a database…The communication module 695 may provide for communication between the electronic device 600 and other, external devices, external data sources, databases, and the like, through a network…The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server)…The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network”; Note: the data server is equivalent to the listing server, as it comprises a database storing items) to identify one or more listed items related to the one or more physical items (Paragraph 0040 – “the system may recognize and/or identify these physical objects, and apply labels to these physical objects, based on image recognition and comparison with an extensive collection of labeled, annotated images stored in a database that is accessible to the application running on the electronic device 110. In some implementations, findings may be scored, or ranked, to reflect a degree of correlation between a detected physical object and a possible identified match from the database. In some implementations, similar scoring, or ranking, may be taken into consideration when determining connections, or relationships between objects, based on the semantic understanding of the physical environment, and selecting suggested objects to add to the camera view of the physical environment”; Note: the objects from the database are equivalent to listed items). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chojnacka to identify physical items in the physical world using image data for the benefit of being able to gain more information about the environment and being able to analyze the objects, which may help the user navigate the environment or understand the objects better. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chojnacka to communicate with a server to identify items related to the physical items for the benefit of providing relevant and useful information to the user. Additionally, having the listed items be in a server makes it easier to access information corresponding to the items.
Furthermore, Spivack modified by Chojnacka still does not teach that the one or more listed items are listed for sale by the one or more listing servers. However, Champavannarth teaches that the one or more listed items are listed for sale by the one or more listing servers (Paragraph 0043, 0051 – “The price comparison result screen 500 also includes, for the second entity (e.g., “Retailer B”), an item number 520, a current (or sale) price 522, an entity identifier 524, store number 526, in stock quantity 528, and store location 530…The price comparison client application or server application can extract (or “scrape”) price, quantity, and/or other information from the website of the second entity and provide the information to the price comparison application for display to the user”; Note: the entity/item is listed for sale on a website that is accessed by the application. The server that hosts the website listing the entities is equivalent to the listing server. It is implied that there is a server hosting the website because a public website cannot exist without a server). Since Spivack already supports a commerce platform for selling items (Paragraph 0092, 0109 – “further embodiments of the present disclosure include an augmented reality commerce platform administer a marketplace which supports multiple seller entities via an augmented reality environment…One example of an AR environment deployed by the host (e.g., the host server 100 as depicted in the example of FIG. 1) enables users to interact with virtual objects (VOBs) or applications related to shopping and retail in the physical world or online/e-commerce or mobile commerce. Retailers, merchants, commerce/e-commerce platforms, classified ad systems, and other advertisers will be able to pay to promote virtual objects representing coupons and gift cards in physical locations near or within their stores”), it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Champavannarth to list the items for sale on a listing server for the benefit of being able to access all of the items that retailers are trying to advertise and sell so that they can be viewed by users for shopping. Retailers would be able to promote their products, and users would be able to find desired products easier. Additionally, the listing server can help manage and organize all the items that are listed.
Regarding claim 3, Spivack in view of Chojnacka and Champavannarth teaches the method of claim 1. Spivack further teaches wherein generating the output includes rendering the information for display in the AR environment (Paragraph 0191, 0203 – “The content associated with the virtual object can also include one or more of, works of art, animation, games, scenes, limericks, jokes, a performance, an action, a routine which can be rendered or played in 360 degrees or 3D…An object or virtual object in the AR environment is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment and have human perceptible properties to be human discernible or detectable. The object (e.g., VOB) can include a message, a chat, an email, a note, a post, a news item, or any other types of VOBs as disclosed herein”; Note: a virtual object and content related to the virtual object is rendered for display).
Regarding claim 4, Spivack in view of Chojnacka and Champavannarth teaches the method of claim 1. Spivack further teaches wherein rendering the information includes rendering at least one of the one or more listed items in the AR environment (Paragraph 0203 – “An object or virtual object in the AR environment is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment and have human perceptible properties to be human discernible or detectable”; Note: the listed items were previously taught by Chojnacka in the rejection of claim 1. In this case, the virtual object corresponds to listed items).
Regarding claim 5, Spivack in view of Chojnacka and Champavannarth teaches the method of claim 3. Spivack further teaches positioning the rendered information in the AR environment according to a location of at least one of the one or more physical items in the physical environment (Paragraph 0214 – “virtual billboards can be placed in space, pinned to a user or a moveable object, or can be pinned to a physical surface at a physical location”; Note: the virtual billboard, which is the rendered information, is pinned to an object, meaning it is at the location of the object).
Regarding claim 6, Spivack in view of Chojnacka and Champavannarth teaches the method of claim 1. Spivack does not teach detecting, using one or more sensors, that a user of the computing device is viewing an item of the one or more physical items; and selecting the information from data associated with the one or more listed items based on the information corresponding to a listed item that is related to the viewed item. However, Chojnacka teaches detecting, using one or more sensors, that a user of the computing device is viewing an item of the one or more physical items (Paragraph 0025-0026 – “a user in the physical environment 100A may use a sensor, for example, a camera of the electronic device 110 to stream images of the physical environment 100A. As the images of the physical environment 100A are streamed, and the image frames are fed into the recognition algorithm or model, 3D pose and location information related to real objects in the physical environment 100A may be detected…FIGS. 3A-3D illustrate sequential camera views 300, or scenes 300, of the physical environment 100A shown in FIG. 2A, as viewed by the user on, for example, a display of the electronic device, as the user moves the electronic device 110 through the physical environment 100A shown in FIG. 2A”; Note: the camera view is based on the user view, and the objects in the view are detected); and selecting the information from data associated with the one or more listed items based on the information corresponding to a listed item that is related to the viewed item (Paragraph 0040 – “the system may recognize and/or identify these physical objects, and apply labels to these physical objects, based on image recognition and comparison with an extensive collection of labeled, annotated images stored in a database that is accessible to the application running on the electronic device 110. In some implementations, findings may be scored, or ranked, to reflect a degree of correlation between a detected physical object and a possible identified match from the database. In some implementations, similar scoring, or ranking, may be taken into consideration when determining connections, or relationships between objects, based on the semantic understanding of the physical environment, and selecting suggested objects to add to the camera view of the physical environment”; Note: The items in the database are equivalent to the listed items. It is implied that information associated with the listed items is selected based on the viewed item since the listed items are suggested based on the relationship between the items). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chojnacka to detect that a user is viewing an object for the benefit of being able to gain more information about the environment and being able to analyze the object, which may help the user navigate the environment or understand the object better. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chojnacka to select information on listed items related to the viewed item for the benefit of providing relevant and useful information to the user.
Regarding claim 8, Spivack in view of Chojnacka and Champavannarth teaches the method of claim 1. Spivack further teaches detecting the user interaction by detecting a hand gesture using one or more sensors of the computing device (Paragraph 0236, 0248, 0268 – “The client device 402 includes a network interface 404, a timing module 406, an RF sensor 407, a location sensor 408, an image sensor 409, a location calibration engine 412, a user view manager 414, a user stimulus sensor 416, a motion/gesture sensor 418… the motion sensor includes a first sensor configured to detect user motion of a first human user in the first physical location of the real world environment on the first side of the transparent display, where the first human user can access the transparent display from the first side of the transparent display…the command to instruct the digital assistant can be initiated by voice, touch, eye, gaze, gesture (body, hand, head, arms, legs, limbs, eyes, torso, etc.), text input and/or other command submitted by a user”).
Regarding claim 10, Spivack in view of Chojnacka and Champavannarth teaches the method of claim 1. Spivack does not teach determining, by the computing device, that a physical item of the one or more physical item corresponds to a listed item that is listed in the one or more listing servers for sale at a lower price than the physical item; and responsive to the determining, displaying a notification to the user in the AR environment that includes an indication of the price of the physical item and a price of the listed item. However, Champavannarth teaches determining, by the computing device, that a physical item of the one or more physical item corresponds to a listed item that is listed in the one or more listing servers (Paragraph 0049-0051, 0054 – “At 606, item information is received…This item number or barcode number is the number associated with the item at the first entity… the item number or barcode number of the product is automatically correlated to a non-identically identified identical product at the second entity. For example, the user device executing the price comparison application can communicate with a price comparison server to request matching information…price and other information (e.g., quantity on hand or sale price) are retrieved from the second entity using the matching information correlated… a price adjustment recommendation is optionally automatically generated as part of the price comparison process”; Note: the physical item and listed items were previously taught by Spivack in the rejection of claim 1. In this case, the first entity corresponds to the physical item, and the second entity corresponds to the listed item) for sale at a lower price than the physical item (Paragraph 0044 – “if the quantity of the item at the second entity is low or zero and the price is lower than the price of the item at the first entity, the automatically generated recommendation may be to keep the price the same or increase the price of the item at the first entity because of item scarcity in the area”); and responsive to the determining, displaying a notification to the user in the AR environment that includes an indication of the price of the physical item and a price of the listed item (Paragraph 0044, 0053 – “if the quantity of the item at the second entity is low or zero and the price is lower than the price of the item at the first entity, the automatically generated recommendation may be to keep the price the same or increase the price of the item at the first entity because of item scarcity in the area…price comparison information including price, quantity and other information for one or more products from the first and second entities is displayed. For example, the price comparison information can be displayed in a user interface screen such as that shown in FIG. 5”; Note: the AR environment was previously taught by Spivack in the rejection of claim 1). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Champavannarth to determine if a listed item price is less than the physical item and display the prices because “pricing recommendations can help ensure that a given retailer's pricing on such items in a store remains competitive with competitors in the geographic area of the store” (Champavannarth: Paragraph 0002). In other words, showing the prices keeps the user informed and may assist them in making decisions for purchasing or selling items.
Regarding claim 11, Spivack teaches a system comprising (Paragraph 0235-0236 – “FIG. 4A depicts an example functional block diagram of a client device 402 (e.g., a portable device or a mobile device)…The client device 402 may be any electronic device such as the devices described in conjunction with the client devices 102A-N in the example of FIG. 1 including but not limited to portable devices, a computer, a server, location-aware devices, mobile phones, PDAs, laptops, palmtops, iPhones, cover headsets, heads-up displays, helmet mounted display, head-mounted display, scanned-beam display”; Note: the client device is equivalent to the system):
at least one memory (Paragraph 0257 – “the client device 402 further includes a memory unit 436 and a storage unit 438”);
and at least one processor coupled with the at least one memory, the at least one memory storing computer-readable instructions that, when executed by the at least one processor, cause the system to perform operations comprising (Paragraph 0255-0258 – “the client device 402 further includes a processing unit 434…The processing unit 434 can include one or more processors, CPUs… The memory unit 436 and a storage unit 438 are, in some embodiments, coupled to the processing unit 434… any portion of or all of the functions described of the various example modules in the client device 402 of the example of FIG. 4A can be performed by the processing unit 434”):
capturing image data of a physical environment for displaying an augmented reality (AR) of the physical environment as an AR environment (Fig. 2C, Paragraph 0035, 0227 – “client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) having AR/VR software agent/hardware module having sensors (e.g., the camera and mic and other sensors) that are always on, and the system is always aware of what is happening…a streaming video camera at the physical location of each device or apparatus controlled by AR/VR window manager 380 so there is always a feed of the AR activity from various perspectives of the physical location”; Note: image data of a physical space is captured by the client device. Fig. 2C shows how the AR of the physical environment is displayed; see screenshot of Fig. 2C above);
rendering a virtual object as an AR assistant for display in the AR environment (Paragraph 0132, 0145, 0186 – “The digital assistant avatar repository 124 can store virtual content or digital assistant avatars that can be retrieved for consumption or deployment in a target environment, where the virtual content or digital assistant avatars are contextually relevant, required, or requested.… the content associated with the virtual object which may be an avatar can include a text message, a post, a chat, a conversation or a graphical message…The object or virtual object is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment”; Note: the virtual object, which is a digital assistant, is rendered into the AR environment)
and responsive to detecting a user interaction with the AR assistant in the AR environment, generating an output using information associated with one or more items (Paragraph 0204, 0214-0215 – “the digital assistant is able to receive a command (e.g., by the command processing engine 352), either explicitly or implicitly, from a user… The virtual billboard can include one or more of, a note, a review, an offer, an ad, a promotion. A virtual billboard object can be a seed for threaded discussions or interactions. In general, virtual billboards can be placed in space, pinned to a user or a moveable object…the virtual billboard engine 360 can create the virtual billboard responsive to a request of a creator user”; Note: a user can request a virtual billboard, through a command. The virtual billboard is an output that shows information related to an item).
Spivack does not teach identifying one or more physical items in the physical environment based at least in part on the image data; and communicating with one or more listing servers over a network to identify one or more listed items related to the one or more physical items, the one or more listed items listed for sale by the one or more listing servers. Spivack also does not teach the “one or more listed items” in the limitation: “responsive to detecting a user interaction with the AR assistant in the AR environment, generating an output using information associated with the one or more listed items”. However, Chojnacka teaches identifying one or more physical items in the physical environment based at least in part on the image data (Paragraph 0025 – “As the images of the physical environment 100A are streamed, and the image frames are fed into the recognition algorithm or model, 3D pose and location information related to real objects in the physical environment 100A may be detected. Images of the real objects may be correlated against images stored in an image database, to identify the detected real objects”); and communicating with one or more listing servers over a network (Paragraph 0028, 0046, 0066-0067 – “[items] may be identified or recognized based on comparison and/or matching with images of previously identified items in a database…The communication module 695 may provide for communication between the electronic device 600 and other, external devices, external data sources, databases, and the like, through a network…The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server)…The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network”; Note: the data server is equivalent to the listing server, as it comprises a database storing items) to identify one or more listed items related to the one or more physical items (Paragraph 0040 – “the system may recognize and/or identify these physical objects, and apply labels to these physical objects, based on image recognition and comparison with an extensive collection of labeled, annotated images stored in a database that is accessible to the application running on the electronic device 110. In some implementations, findings may be scored, or ranked, to reflect a degree of correlation between a detected physical object and a possible identified match from the database. In some implementations, similar scoring, or ranking, may be taken into consideration when determining connections, or relationships between objects, based on the semantic understanding of the physical environment, and selecting suggested objects to add to the camera view of the physical environment”; Note: the objects from the database are equivalent to listed items). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chojnacka to identify physical items in the physical world using image data for the benefit of being able to gain more information about the environment and being able to analyze the objects, which may help the user navigate the environment or understand the objects better. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chojnacka to communicate with a server to identify items related to the physical items for the benefit of providing relevant and useful information to the user. Additionally, having the listed items be in a server makes it easier to access information corresponding to the items.
Furthermore, Spivack modified by Chojnacka still does not teach that the one or more listed items are listed for sale by the one or more listing servers. However, Champavannarth teaches that the one or more listed items are listed for sale by the one or more listing servers (Paragraph 0043, 0051 – “The price comparison result screen 500 also includes, for the second entity (e.g., “Retailer B”), an item number 520, a current (or sale) price 522, an entity identifier 524, store number 526, in stock quantity 528, and store location 530…The price comparison client application or server application can extract (or “scrape”) price, quantity, and/or other information from the website of the second entity and provide the information to the price comparison application for display to the user”; Note: the entity/item is listed for sale on a website that is accessed by the application. The server that hosts the website listing the entities is equivalent to the listing server. It is implied that there is a server hosting the website because a public website cannot exist without a server). Since Spivack already supports a commerce platform for selling items (Paragraph 0092, 0109 – “further embodiments of the present disclosure include an augmented reality commerce platform administer a marketplace which supports multiple seller entities via an augmented reality environment…One example of an AR environment deployed by the host (e.g., the host server 100 as depicted in the example of FIG. 1) enables users to interact with virtual objects (VOBs) or applications related to shopping and retail in the physical world or online/e-commerce or mobile commerce. Retailers, merchants, commerce/e-commerce platforms, classified ad systems, and other advertisers will be able to pay to promote virtual objects representing coupons and gift cards in physical locations near or within their stores”), it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Champavannarth to list the items for sale on a listing server for the benefit of being able to access all of the items that retailers are trying to advertise and sell so that they can be viewed by users for shopping. Retailers would be able to promote their products, and users would be able to find desired products easier. Additionally, the listing server can help manage and organize all the items that are listed.
Regarding claim 13, Spivack in view of Chojnacka and Champavannarth teaches the system of claim 11. Spivack further teaches wherein generating the output includes rendering the information for display in the AR environment (Paragraph 0191, 0203 – “The content associated with the virtual object can also include one or more of, works of art, animation, games, scenes, limericks, jokes, a performance, an action, a routine which can be rendered or played in 360 degrees or 3D…An object or virtual object in the AR environment is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment and have human perceptible properties to be human discernible or detectable. The object (e.g., VOB) can include a message, a chat, an email, a note, a post, a news item, or any other types of VOBs as disclosed herein”; Note: a virtual object and content related to the virtual object is rendered for display).
Regarding claim 14, Spivack in view of Chojnacka and Champavannarth teaches the system of claim 11. Spivack further teaches wherein rendering the information includes rendering at least one of the one or more listed items in the AR environment (Paragraph 0203 – “An object or virtual object in the AR environment is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment and have human perceptible properties to be human discernible or detectable”; Note: the listed items were previously taught by Chojnacka in the rejection of claim 1. In this case, the virtual object corresponds to listed items).
Regarding claim 15, Spivack in view of Chojnacka and Champavannarth teaches the system of claim 13. Spivack further teaches positioning the rendered information in the AR environment according to a location of at least one of the one or more physical items in the physical environment (Paragraph 0214 – “virtual billboards can be placed in space, pinned to a user or a moveable object, or can be pinned to a physical surface at a physical location”; Note: the virtual billboard, which is the rendered information, is pinned to an object, meaning it is at the location of the object).
Regarding claim 16, Spivack in view of Chojnacka and Champavannarth teaches the system of claim 11. Spivack does not teach detecting, using one or more sensors, that a user of the computing device is viewing an item of the one or more physical items; and selecting the information from data associated with the one or more listed items based on the information corresponding to a listed item that is related to the viewed item. However, Chojnacka teaches detecting, using one or more sensors, that a user of the computing device is viewing an item of the one or more physical items (Paragraph 0025-0026 – “a user in the physical environment 100A may use a sensor, for example, a camera of the electronic device 110 to stream images of the physical environment 100A. As the images of the physical environment 100A are streamed, and the image frames are fed into the recognition algorithm or model, 3D pose and location information related to real objects in the physical environment 100A may be detected…FIGS. 3A-3D illustrate sequential camera views 300, or scenes 300, of the physical environment 100A shown in FIG. 2A, as viewed by the user on, for example, a display of the electronic device, as the user moves the electronic device 110 through the physical environment 100A shown in FIG. 2A”; Note: the camera view is based on the user view, and the objects in the view are detected); and selecting the information from data associated with the one or more listed items based on the information corresponding to a listed item that is related to the viewed item (Paragraph 0040 – “the system may recognize and/or identify these physical objects, and apply labels to these physical objects, based on image recognition and comparison with an extensive collection of labeled, annotated images stored in a database that is accessible to the application running on the electronic device 110. In some implementations, findings may be scored, or ranked, to reflect a degree of correlation between a detected physical object and a possible identified match from the database. In some implementations, similar scoring, or ranking, may be taken into consideration when determining connections, or relationships between objects, based on the semantic understanding of the physical environment, and selecting suggested objects to add to the camera view of the physical environment”; Note: The items in the database are equivalent to the listed items. It is implied that information associated with the listed items is selected based on the viewed item since the listed items are suggested based on the relationship between the items). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chojnacka to detect that a user is viewing an object for the benefit of being able to gain more information about the environment and being able to analyze the object, which may help the user navigate the environment or understand the object better. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chojnacka to select information on listed items related to the viewed item for the benefit of providing relevant and useful information to the user.
Regarding claim 18, Spivack in view of Chojnacka and Champavannarth teaches the system of claim 11. Spivack further teaches detecting the user interaction by detecting a hand gesture using one or more sensors of the computing device (Paragraph 0236, 0248, 0268 – “The client device 402 includes a network interface 404, a timing module 406, an RF sensor 407, a location sensor 408, an image sensor 409, a location calibration engine 412, a user view manager 414, a user stimulus sensor 416, a motion/gesture sensor 418… the motion sensor includes a first sensor configured to detect user motion of a first human user in the first physical location of the real world environment on the first side of the transparent display, where the first human user can access the transparent display from the first side of the transparent display…the command to instruct the digital assistant can be initiated by voice, touch, eye, gaze, gesture (body, hand, head, arms, legs, limbs, eyes, torso, etc.), text input and/or other command submitted by a user”).
Regarding claim 20, Spivack teaches a computing device (Paragraph 0235-0236 – “FIG. 4A depicts an example functional block diagram of a client device 402 (e.g., a portable device or a mobile device)… The client device 402 may be any electronic device such as the devices described in conjunction with the client devices 102A-N in the example of FIG. 1 including but not limited to portable devices, a computer, a server, location-aware devices, mobile phones, PDAs, laptops, palmtops, iPhones, cover headsets, heads-up displays, helmet mounted display, head-mounted display, scanned-beam display”; Note: the client device is a computing device), comprising:
a display configured to display an augmented reality (AR) of a physical environment as an AR environment (Paragraph 0235, 0244-0245 – “FIG. 4A depicts an example functional block diagram of a client device 402 (e.g., a portable device or a mobile device) that includes single or multi-directional lens as a portal between a physical world and a digital world component of an augmented reality environment…the client device 402 includes a transparent display having a first side and a second side…When viewed from the first side of the transparent display to the second side, a second set of virtual objects are rendered to appear to be overlaid over second physical elements in a second physical location in the real world in the real world environment on the second side of the transparent display”);
a sensor configured to capture image data of the physical environment (Fig. 2C, Paragraph 0035, 0235-0236 – “client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) having AR/VR software agent/hardware module having sensors (e.g., the camera and mic and other sensors) that are always on, and the system is always aware of what is happening…FIG. 4A depicts an example functional block diagram of a client device 402 (e.g., a portable device or a mobile device) that includes single or multi-directional lens as a portal between a physical world and a digital world component of an augmented reality environment…The client device 402 includes a network interface 404, a timing module 406, an RF sensor 407, a location sensor 408, an image sensor 409”; Note: image data of a physical space is captured. Fig. 2C shows how the physical space is captured and displayed with the AR; see screenshot of Fig. 2C above);
at least one processor (Paragraph 0255-0256 – “the client device 402 further includes a processing unit 434…The processing unit 434 can include one or more processors, CPUs”);
at least one memory storing instructions that, when executed by the at least one processor, cause the computing device to (Paragraph 0257-0258 – “the client device 402 further includes a memory unit 436 and a storage unit 438. The memory unit 436 and a storage unit 438 are, in some embodiments, coupled to the processing unit 434… any portion of or all of the functions described of the various example modules in the client device 402 of the example of FIG. 4A can be performed by the processing unit 434”):
render a virtual object in the AR environment as an AR assistant (Paragraph 0132, 0145, 0186 – “The digital assistant avatar repository 124 can store virtual content or digital assistant avatars that can be retrieved for consumption or deployment in a target environment, where the virtual content or digital assistant avatars are contextually relevant, required, or requested.… the content associated with the virtual object which may be an avatar can include a text message, a post, a chat, a conversation or a graphical message…The object or virtual object is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment”; Note: the virtual object, which is a digital assistant, is rendered into the AR environment) for accessing one or more servers (Paragraph 0173-0174, 0208-0209 – “The host server 300 includes a network interface 302, a virtual object (VOB) sharing/publication engine 310, a communications manager 340, a digital assistant manager 350, a virtual billboard engine 360…The host server 300 is also coupled to a billboard object repository 322…The digital assistant manager 350 can send a VOB message with a 3D emoji…the digital assistant manager 350 is also able to represent knowledge to the user and is able to perceive the real world environment and/or the AR environment as its user”; Note: the digital assistant is able to send VOB messages, which implies that the digital assistant can be used to access the host server since the host server contains the VOB sharing engine, virtual billboard engine, and is coupled to the billboard object repository);
and responsive to detecting a user interaction with the AR assistant in the AR environment, generate an output using information associated with one or more items (Paragraph 0204, 0214-0215 – “the digital assistant is able to receive a command (e.g., by the command processing engine 352), either explicitly or implicitly, from a user… The virtual billboard can include one or more of, a note, a review, an offer, an ad, a promotion. A virtual billboard object can be a seed for threaded discussions or interactions. In general, virtual billboards can be placed in space, pinned to a user or a moveable object…the virtual billboard engine 360 can create the virtual billboard responsive to a request of a creator user”; Note: a user can request a virtual billboard, through a command. The virtual billboard is an output that shows information related to an item).
Spivack does not teach identifying one or more physical items in the physical environment based at least in part on the image data; and communicating with one or more listing servers over a network to identify one or more listed items related to the one or more physical items, the one or more listed items listed for sale by the one or more listing servers. Spivack also does not teach the “one or more listing servers” in the limitation: “render a virtual object in the AR environment as an AR assistant for accessing the one or more listing servers”, nor the “one or more listed items” in the limitation: “responsive to detecting a user interaction with the AR assistant in the AR environment, generate an output using information associated with the one or more listed items”. However, Chojnacka teaches identifying one or more physical items in the physical environment based at least in part on the image data (Paragraph 0025 – “As the images of the physical environment 100A are streamed, and the image frames are fed into the recognition algorithm or model, 3D pose and location information related to real objects in the physical environment 100A may be detected. Images of the real objects may be correlated against images stored in an image database, to identify the detected real objects”); and communicating with one or more listing servers over a network (Paragraph 0028, 0046, 0066-0067 – “[items] may be identified or recognized based on comparison and/or matching with images of previously identified items in a database…The communication module 695 may provide for communication between the electronic device 600 and other, external devices, external data sources, databases, and the like, through a network…The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server)…The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network”; Note: the data server is equivalent to the listing server, as it comprises a database storing items) to identify one or more listed items related to the one or more physical items (Paragraph 0040 – “the system may recognize and/or identify these physical objects, and apply labels to these physical objects, based on image recognition and comparison with an extensive collection of labeled, annotated images stored in a database that is accessible to the application running on the electronic device 110. In some implementations, findings may be scored, or ranked, to reflect a degree of correlation between a detected physical object and a possible identified match from the database. In some implementations, similar scoring, or ranking, may be taken into consideration when determining connections, or relationships between objects, based on the semantic understanding of the physical environment, and selecting suggested objects to add to the camera view of the physical environment”; Note: the objects from the database are equivalent to listed items). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chojnacka to identify physical items in the physical world using image data for the benefit of being able to gain more information about the environment and being able to analyze the objects, which may help the user navigate the environment or understand the objects better. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chojnacka to communicate with a server to identify items related to the physical items for the benefit of providing relevant and useful information to the user. Additionally, having the listed items be in a server makes it easier to access information corresponding to the items.
Furthermore, Spivack modified by Chojnacka still does not teach that the one or more listed items are listed for sale by the one or more listing servers. However, Champavannarth teaches that the one or more listed items are listed for sale by the one or more listing servers (Paragraph 0043, 0051 – “The price comparison result screen 500 also includes, for the second entity (e.g., “Retailer B”), an item number 520, a current (or sale) price 522, an entity identifier 524, store number 526, in stock quantity 528, and store location 530…The price comparison client application or server application can extract (or “scrape”) price, quantity, and/or other information from the website of the second entity and provide the information to the price comparison application for display to the user”; Note: the entity/item is listed for sale on a website that is accessed by the application. The server that hosts the website listing the entities is equivalent to the listing server. It is implied that there is a server hosting the website because a public website cannot exist without a server). Since Spivack already supports a commerce platform for selling items (Paragraph 0092, 0109 – “further embodiments of the present disclosure include an augmented reality commerce platform administer a marketplace which supports multiple seller entities via an augmented reality environment…One example of an AR environment deployed by the host (e.g., the host server 100 as depicted in the example of FIG. 1) enables users to interact with virtual objects (VOBs) or applications related to shopping and retail in the physical world or online/e-commerce or mobile commerce. Retailers, merchants, commerce/e-commerce platforms, classified ad systems, and other advertisers will be able to pay to promote virtual objects representing coupons and gift cards in physical locations near or within their stores”), it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Champavannarth to list the items for sale on a listing server for the benefit of being able to access all of the items that retailers are trying to advertise and sell so that they can be viewed by users for shopping. Retailers would be able to promote their products, and users would be able to find desired products easier. Additionally, the listing server can help manage and organize all the items that are listed.
Claims 2, 9, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Spivack in view of Chojnacka, Champavannarth, and Kharband et al. (US 20250218139 A1), hereinafter Kharband.
Regarding claim 2, Spivack in view of Chojnacka and Champavannarth teaches the method of claim 1. Spivack teaches wherein generating the output comprises generating an audio output indicative of the information (Paragraph 0191 – “The contextual trigger for the VOB can include any one of or any combination of a user parameter, a time parameter, an activity parameter and a location parameter…the content associated with the virtual object can include a text message, a post, a chat, a conversation or a graphical message. The content associated with the virtual object can also include digital stickers, GIFs, digital tattoos, emoticons, animations, videos, clips, games, photos, images, a 3D scene or an animated 3D object, music, sound, tone, ultrasonic, or other audible content”; Note: the output, which is the virtual object, includes audio). Spivack does not teach “using a large language model” in the limitation: “wherein generating the output comprises using a large language model to generate an audio output indicative of the information”. However, Kharband teaches using a large language model to generate an audio output indicative of the information (Paragraph 0063, 0065 – “The augmented-reality system 200 can process the input data 212 and/or the image data 214 with a generative model 216 to generate a model-generated response. The generative model 216 can include a generative language model, which may include a large language model…The model-generated response can include text data (e.g., a natural language response (e.g., a conversational response, a topic summary, text instructions, a story, a poem, etc.)), image data (e.g., model-generated images that outline a response to the query), audio data (e.g., a plurality of audio signals that may include audio instructions, a song, a conversational audio response, etc.)”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Kharband to use a large language model to generate audio output because the large language model would automate the audio generation and help produce natural language responses, which would enhance the user experience.
Regarding claim 9, Spivack in view of Chojnacka and Champavannarth teaches the method of claim 1. Spivack further teaches detecting the user interaction by receiving audio input indicative of a voice command for the AR assistant (Paragraph 0267 – “the command can be received by the digital assistant through a voice interface of the user interface and the command includes a voice command or speech command initiated by the user”). Spivack does not teach “using a large language model to generate text indicative of the voice command” in the limitation: “detecting the user interaction by receiving audio input indicative of a voice command for the AR assistant and using a large language model to generate text indicative of the voice command”. However, Kharband teaches using a large language model to generate text indicative of the voice command (Paragraph 0063, 0075 – “The augmented-reality system 200 can process the input data 212 and/or the image data 214 with a generative model 216 to generate a model-generated response. The generative model 216 can include a generative language model, which may include a large language model…The generative model 216 may include text encoders, image encoders, other data encoders, and/or one or more decoders, which may decode one or more feature representations into text data…The user input can include…an audio input (e.g., a voice command obtained via an audio sensor (e.g., a microphone)”; Note: the audio input is processed using a large language model, which decodes the input into text). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Kharband to use a large language model to generate text corresponding to the voice command because the large language model would be able to quickly recognize speech and perform a speech-to-text operation. The text would be useful for understanding what the user said so that a response could be generated, and it would also be useful for displaying the conversation.
Regarding claim 12, Spivack in view of Chojnacka and Champavannarth teaches the system of claim 11. Spivack teaches wherein generating the output comprises generating an audio output indicative of the information (Paragraph 0191 – “The contextual trigger for the VOB can include any one of or any combination of a user parameter, a time parameter, an activity parameter and a location parameter…the content associated with the virtual object can include a text message, a post, a chat, a conversation or a graphical message. The content associated with the virtual object can also include digital stickers, GIFs, digital tattoos, emoticons, animations, videos, clips, games, photos, images, a 3D scene or an animated 3D object, music, sound, tone, ultrasonic, or other audible content”; Note: the output, which is the virtual object, includes audio). Spivack does not teach “using a large language model” in the limitation: “wherein generating the output comprises using a large language model to generate an audio output indicative of the information”. However, Kharband teaches using a large language model to generate an audio output indicative of the information (Paragraph 0063, 0065 – “The augmented-reality system 200 can process the input data 212 and/or the image data 214 with a generative model 216 to generate a model-generated response. The generative model 216 can include a generative language model, which may include a large language model…The model-generated response can include text data (e.g., a natural language response (e.g., a conversational response, a topic summary, text instructions, a story, a poem, etc.)), image data (e.g., model-generated images that outline a response to the query), audio data (e.g., a plurality of audio signals that may include audio instructions, a song, a conversational audio response, etc.)”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Kharband to use a large language model to generate audio output because the large language model would automate the audio generation and help produce natural language responses, which would enhance the user experience.
Claims 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Spivack in view of Chojnacka, Champavannarth, and Chia et al. (Structural Descriptors for Category Level Object Detection), hereinafter Chia.
Regarding claim 7, Spivack in view of Chojnacka and Champavannarth teaches the method of claim 1. Spivack does not teach identifying the one or more physical objects based on structure data determined from the image data that indicates a geometry of the one or more physical objects. However, Chia teaches identifying the one or more physical objects based on structure data determined from the image data that indicates a geometry of the one or more physical objects (Fig. 17, Paragraph 1 on 2nd Col. of Page 1, Paragraph 1 on 1st Col. of Page 2 – “we introduce a new set of structural descriptors that are constructed from these primitives. The descriptors have a number of properties that make them suitable for object categorization…Specifically, the descriptors harness the geometrical and spatial relationships between each primitive and its neighbors to describe local object structures. Although one can define many geometrical attributes between primitives, we demonstrate that a small set of geometrical attributes when combined with simple spatial relationships yield rich and accurate local structural descriptions of objects…we detect objects present in the image by matching the descriptors with those previously extracted from training images”; Note: objects are detected based on structural descriptions determined from geometric attributes. Fig. 17 shows the identified objects; see screenshot of Fig. 17 below).
PNG
media_image2.png
409
704
media_image2.png
Greyscale
Screenshot of Fig. 17 (taken from Chia)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chia to identify an object based on structure data determined from geometry data because “shape primitives are simple and generic enough to afford great flexibility in modeling object parts and suggests that structural information of an object may be comprehensively described by the primitives…The descriptors have a number of properties that make them suitable for object categorization. Firstly, they are translation and scale invariant. In addition, we use edge pixels to construct the descriptors and hence they are insensitive to moderate illumination changes” (Chia: Paragraph 1 in 2nd Col. of Page 1). In other words, geometry data can easily represent objects and help indicate the objects’ structures enough to determine what the object is.
Regarding claim 17, Spivack in view of Chojnacka and Champavannarth teaches the system of claim 11. Spivack does not teach identifying the one or more physical objects based on structure data determined from the image data that indicates a geometry of the one or more physical objects. However, Chia teaches identifying the one or more physical objects based on structure data determined from the image data that indicates a geometry of the one or more physical objects (Fig. 17, Paragraph 1 on 2nd Col. of Page 1, Paragraph 1 on 1st Col. of Page 2 – “we introduce a new set of structural descriptors that are constructed from these primitives. The descriptors have a number of properties that make them suitable for object categorization…Specifically, the descriptors harness the geometrical and spatial relationships between each primitive and its neighbors to describe local object structures. Although one can define many geometrical attributes between primitives, we demonstrate that a small set of geometrical attributes when combined with simple spatial relationships yield rich and accurate local structural descriptions of objects…we detect objects present in the image by matching the descriptors with those previously extracted from training images”; Note: objects are detected based on structural descriptions determined from geometric attributes. Fig. 17 shows the identified objects; see screenshot of Fig. 17 above). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chia to identify an object based on structure data determined from geometry data because “shape primitives are simple and generic enough to afford great flexibility in modeling object parts and suggests that structural information of an object may be comprehensively described by the primitives…The descriptors have a number of properties that make them suitable for object categorization. Firstly, they are translation and scale invariant. In addition, we use edge pixels to construct the descriptors and hence they are insensitive to moderate illumination changes” (Chia: Paragraph 1 in 2nd Col. of Page 1). In other words, geometry data can easily represent objects and help indicate the objects’ structures enough to determine what the object is.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Spivack in view of Chojnacka, Champavannarth, and Borhan et al. (US 20130218721 A1), hereinafter Borhan.
Regarding claim 21, Spivack in view of Chojnacka and Champavannarth teaches the method of claim 1. Spivack does not teach receiving, via the AR assistant, a query from a user regarding items available from the one or more listing servers; and generating, by the AR assistant, a response to the query that includes information about a listed item of the one or more listed items that is listed for sale. However, Borhan teaches receiving, via the AR assistant, a query from a user regarding items available from the one or more listing servers (Paragraph 0059-0060, 0083 – “the virtual wallet app executing on the user device may provide features as described below to augment the user's in-store shopping experience… the store management server may have access to the user's prior purchasing behavior, the user's real-time in-store behavior (e.g., which items' barcode did the user scan using the user device, how many times did the user scan the barcodes, did the user engage in comparison shopping by scanning barcodes of similar types of items, and/or the like)…the CSR may comprise an shopping assistant avatar instantiated on the CSR device, with which the consumer may interact with, or the consumer may access the CSR shopping avatar within the consumer mobile wallet…the consumer may provide an indication of interests 231a (e.g., see 427a-b in FIG. 4E; tapping an “add to cart” button, etc.) in the CSR provided items/offers… and the CSR may in turn provide detailed information and/or add the item to shopping cart 233a (e.g., see 439 in FIG. 4G) to the consumer per consumer request”; Note: the CSR, which is an AR assistant, receives an indication of interest or request of an item, which is a user query. The store management server is equivalent to the listing server in this case); and generating, by the AR assistant, a response to the query that includes information about a listed item of the one or more listed items that is listed for sale (Paragraph 0083, 0095 – “the CSR may in turn provide detailed information and/or add the item to shopping cart 233a (e.g., see 439 in FIG. 4G) to the consumer per consumer request… if the request indicates a shopping request (e.g., consumer inquiry on shopping items, see 427a-c in FIG. 4E, etc.), the CSR may retrieve shopping item information and add the item to a shopping cart 331, and provide such to the consumer 337 (e.g., see 434d-e in FIG. 4F)”; Note: the CSR/assistant responds to the query by providing information about an item). Since Spivack already teaches an assistant responding to a command (Paragraph 0206 – “The digital assistant manager 350 can also detect, identify, interpret, react to, respond to implicit commands (e.g., by the command processing engine 352 and/or the artificial intelligence (AI) engine 354). For example, the digital assistance manager 350 can detect activities occurring in the real world, in the AR environment (e.g., with the virtual objects), and/or detect user activity/behavior and trigger an action”), it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Borhan to have the AR assistant provide the user with information on a listed item for sale in response to the user’s query for the benefit of helping the user make informed purchases. Spivack supports a commerce platform for selling items (Paragraph 0092, 0109 – “further embodiments of the present disclosure include an augmented reality commerce platform administer a marketplace which supports multiple seller entities via an augmented reality environment…One example of an AR environment deployed by the host (e.g., the host server 100 as depicted in the example of FIG. 1) enables users to interact with virtual objects (VOBs) or applications related to shopping and retail in the physical world or online/e-commerce or mobile commerce. Retailers, merchants, commerce/e-commerce platforms, classified ad systems, and other advertisers will be able to pay to promote virtual objects representing coupons and gift cards in physical locations near or within their stores”). Therefore, having the assistant be a part of the commerce platform would help users better understand the products, enhance the user experience, and allow retailers to promote their items.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Statler et al. (US 20130030915 A1) teaches online shopping assistance tools that perform tasks such as store navigation, customer assistance, and inventory management.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE HAU MA whose telephone number is (571)272-2187. The examiner can normally be reached M-Th 7-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE HAU MA/ Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights