DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on July 23, 2026 has been entered.
Response to Amendment
The amendment filed July 23, 2026 has been entered. Claims 1-18 and 20-21 remain pending in the application. Applicant’s amendments to the Specification and Drawings have overcome each and every objection previously set forth in the Final Office Action mailed April 23, 2026.
Response to Arguments
Applicant’s arguments, see Pages 11-13 of Remarks, filed July 23, 2026, with respect to the rejection(s) of claim(s) 1-18 and 20-21 under 35 USC 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Chachek et al. (US 20230118119 A1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-6, 8, 11, 13-16, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Spivack et al. (US 20210074068 A1) in view of Chachek et al. (US 20230118119 A1), hereinafter Spivack and Chachek respectively.
Regarding claim 1, Spivack teaches a method (Paragraph 0098 – “Embodiments of the present disclosure further include systems, methods and apparatuses of seamless integration of augmented, alternate, virtual, and/or mixed realities with physical realities for enhancement of web, mobile and/or other digital experiences”) comprising:
capturing, by a computing device, image data of a physical environment for displaying an augmented reality (AR) of the physical environment as an AR environment (Fig. 2C, Paragraph 0035, 0227 – “client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) having AR/VR software agent/hardware module having sensors (e.g., the camera and mic and other sensors) that are always on, and the system is always aware of what is happening…a streaming video camera at the physical location of each device or apparatus controlled by AR/VR window manager 380 so there is always a feed of the AR activity from various perspectives of the physical location”; Note: image data of a physical space is captured by the client device, which is equivalent to the computing device. Fig. 2C shows how the AR of the physical environment is displayed; see screenshot of Fig. 2C below);
PNG
media_image1.png
392
550
media_image1.png
Greyscale
Screenshot of Fig. 2C (taken from Spivack)
rendering a virtual object in the AR environment as an AR assistant (Paragraph 0132, 0145, 0186 – “The digital assistant avatar repository 124 can store virtual content or digital assistant avatars that can be retrieved for consumption or deployment in a target environment, where the virtual content or digital assistant avatars are contextually relevant, required, or requested.… the content associated with the virtual object which may be an avatar can include a text message, a post, a chat, a conversation or a graphical message…The object or virtual object is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment”; Note: the virtual object, which is a digital assistant, is rendered into the AR environment);
and responsive to detecting a user interaction with the AR assistant in the AR environment, operating the AR assistant in a second mode by generating an output using information associated with one or more items (Paragraph 0204, 0214-0215 – “the digital assistant is able to receive a command (e.g., by the command processing engine 352), either explicitly or implicitly, from a user… The virtual billboard can include one or more of, a note, a review, an offer, an ad, a promotion. A virtual billboard object can be a seed for threaded discussions or interactions. In general, virtual billboards can be placed in space, pinned to a user or a moveable object…the virtual billboard engine 360 can create the virtual billboard responsive to a request of a creator user”; Note: a user can request a virtual billboard, through a command. The virtual billboard is an output that shows information related to an item).
Spivack does not teach operating the AR assistant in a first mode by exploring the AR environment to identify one or more physical items in the physical environment and proactively searching one or more listing servers over a network for one or more listed items related to the one or more physical items, the one or more listed items listed for sale by the one or more listing servers; nor generating an output using information associated with the one or more listed items, the output based at least in part on knowledge of the one or more physical items identified in the first mode. However, Chachek teaches operating the AR assistant (Paragraph 0050, 0054 – “a visual navigation system is provided, optionally implemented as an Augmented Reality (AR) based navigation system which may assist human users in real-life navigation scenarios…system 100 is configured to facilitate real-life navigation and/or localization and/or mapping of users (as well as products, items, store elements, or the like) within a superstore or megastore or other large-size facility, typically spanning numerous aisles and/or cabinets and/or shelves and/or corridors and/or other structures for storing products and for allowing users to walk therein.”; Note: the system is an AR assistant in this case) in a first mode by exploring the AR environment (Paragraph 0015, 0106 – “an end-user device that is determined to be located in front of the Sprite bottles shelf, would show on his screen an animated promotion or animated avatar that suggests to him to purchase this product … On the screen of his smartphone, the user sees a real-time image of the store, with Augmented Reality (AR) content shown as overlay elements within the image; for example, showing AR-based content of a walking path to a destination product, showing an AR-based avatar or character roaming within the store, showing an AR-based indicator for a sale or promotion (shown as AR-based “40% off” tag or label in this example), shown an AR-based indicator for a particular product or type-of-product that is within the captured field-of-view (e.g., indicated as an AR-based information board about “Donut” in this example), or the like”; Note: an animated avatar, which is part of the system, explores/roams the environment) to identify one or more physical items in the physical environment (Paragraph 0059 – “a Computer Vision unit 104 and/or an Optical Character Recognition (OCR) unit 105, which may be implemented as part of each such participating end-user device and/or as part of a remote server, may analyze an image or a captured frame, and may determine that it shows three boxes of “Corn Flakes”; and may optionally detect a logo or a brand-name which may be compared to or match with a list of product makers or product manufacturing”; Note: the system identifies physical items in the environment, such as the corn flakes) and proactively searching one or more listing servers over a network (Paragraph 0055, 0059 – “client/server architecture may be used, such that each one of devices 151-153 may communicate over wireless link(s) (e.g., over Wi-Fi, over a cellular connection, over an Internet connection) with server 150, which may be located in the retail store and/or may be located remotely (e.g., a cloud-computing server). In other embodiments, distributed architecture or peer-to-peer architecture may be used, such that one or some or all of the devices may operate as nodes or as network elements that may receive and/or transmit data to other units…any of the components or units or modules that are shown in FIG. 1, may be implemented as part of server 150…data may be obtained or fetched from an Inventory Database 106 of the store; for example, by identifying in the image the product name “corn flakes”, then identifying in the image a particular manufacturer logo and a particular product size (e.g., 16 oz), and then obtaining the other data regarding this particular product from the Inventory Database 106”; Note: server 150 is equivalent to the listing server since the inventory database 106 is part of it and lists items/products, and it is accessed over a network, such as the internet) for one or more listed items related to the one or more physical items, the one or more listed items listed for sale by the one or more listing servers (Paragraph 0059, 0063 – “enabling the system to determine which product is located in real life at which locations in the store, and to further associate each such product with other data or meta-data, for example, manufacturer, product real-life weight, product real-life dimensions, nutritional data of the product, allergy information for the product, product price, discounts or promotions or coupons that are currently associated and/or usable in conjunction with this product, or the like. Such data may be obtained or fetched from an Inventory Database 106 of the store; for example, by identifying in the image the product name “corn flakes”, then identifying in the image a particular manufacturer logo and a particular product size (e.g., 16 oz), and then obtaining the other data regarding this particular product from the Inventory Database 106…the system proceeds to perform rapid real-time recognition of the products that are currently imaged in the field-of-view of the imager of the end-user device, to fetch or obtain or lookup or download product data for each recognized product (e.g., nutrition values, ingredients, allergens information, or the like), to determine which sub-set of the imaged products fulfill the constraints”; Note: the system identifies a physical item and searches a corresponding listed item and its data. The items are for sale since they are part of a store); and generating an output using information associated with the one or more listed items, the output based at least in part on knowledge of the one or more physical items identified in the first mode (Paragraph 0063, 0106 – “determine which sub-set of the imaged products fulfill the constraints…generate and displayed AR-based or VR-based emphasis of such sub-set of products, for example, by adding a glowing frame around such particular products as displayed, and/or by adding a textual label or a graphical indicator or an animated content, or the like…On the screen of his smartphone, the user sees a real-time image of the store, with Augmented Reality (AR) content shown as overlay elements within the image; for example, showing AR-based content of a walking path to a destination product, showing an AR-based avatar or character roaming within the store, showing an AR-based indicator for a sale or promotion (shown as AR-based “40% off” tag or label in this example), shown an AR-based indicator for a particular product or type-of-product that is within the captured field-of-view (e.g., indicated as an AR-based information board about ‘Donut’ in this example), or the like”; Note: an output related to the listed item is generated and displayed based on information of the physical item. For instance, a 40%-off label is generated as an output, which provides a sale information related to the item). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chachek to have the AR assistant explore the environment and identify physical items in the physical environment for the benefit of being able to gain more information about the environment and being able to analyze the objects, which may help the user navigate the environment or achieve their shopping goals more efficiently (Chachek: Paragraph 0048 – “utilize Augmented Reality (AR) data or datasets or objects or representations, particularly for the purpose of organizing and/or categorizing and/or classifying real-life items or products or locations, and/or for navigating from a first real-life location”). Since Spivack already supports a commerce platform for selling items (Paragraph 0092, 0109 – “further embodiments of the present disclosure include an augmented reality commerce platform administer a marketplace which supports multiple seller entities via an augmented reality environment…One example of an AR environment deployed by the host (e.g., the host server 100 as depicted in the example of FIG. 1) enables users to interact with virtual objects (VOBs) or applications related to shopping and retail in the physical world or online/e-commerce or mobile commerce. Retailers, merchants, commerce/e-commerce platforms, classified ad systems, and other advertisers will be able to pay to promote virtual objects representing coupons and gift cards in physical locations near or within their stores”), it also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chachek to search for items listed for sale in a listing server for the benefit of providing relevant and useful information to the user. Additionally, having the listed items be in a server makes it easier to access all of the items that retailers are trying to advertise and sell so that they can be viewed by users for shopping. Retailers would be able to promote their products, and users would be able to find desired products easier. Additionally, the listing server can help manage and organize all the items that are listed. Finally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chachek to have the output be based on knowledge of the identified physical items for the benefit of informing the user about the item, which can help them locate it and make a decision on whether or not to buy it. For example, “showing an AR-based indicator for a sale or promotion” (Chachek: Paragraph 0106) may help a user determine whether or not the item is worth buying.
Regarding claim 3, Spivack in view of Chachek teaches the method of claim 1. Spivack further teaches wherein generating the output includes rendering the information for display in the AR environment (Paragraph 0191, 0203 – “The content associated with the virtual object can also include one or more of, works of art, animation, games, scenes, limericks, jokes, a performance, an action, a routine which can be rendered or played in 360 degrees or 3D…An object or virtual object in the AR environment is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment and have human perceptible properties to be human discernible or detectable. The object (e.g., VOB) can include a message, a chat, an email, a note, a post, a news item, or any other types of VOBs as disclosed herein”; Note: a virtual object and content related to the virtual object is rendered for display).
Regarding claim 4, Spivack in view of Chachek teaches the method of claim 3. Spivack further teaches wherein rendering the information includes rendering at least one of the one or more items in the AR environment (Paragraph 0203 – “An object or virtual object in the AR environment is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment and have human perceptible properties to be human discernible or detectable”). Spivack does not teach rendering at least one of the one or more listed items in the AR environment. However, Chachek teaches rendering at least one of the one or more listed items in the AR environment (Paragraph 0110-0111 – “the system then checks the Store Map and the Inventory Database, to determine which products are located on that particular shelf; and the system displays on the user's smartphone the relevant output, for example, an actual real-life photo of that shelf on Aisle 5 (e.g., a pre-defined static photo, from a static database that contains several hundreds of images of shelf-portions of that store…display as Augmented Reality (AR) content, as an overlay on the entire screen of the user device or on the particular screen-portion or image-portion that the user selected or tapped or clicked, a representation of those soda bottles of Shelf 13 (e.g., stock images of the products, or actual real-world images of those exact bottles as captured and shared a few minutes earlier by another customer in that same store”; Note: the listed item in this case is the soda bottles since its data is stored in the database, and it is displayed/rendered in the AR environment). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the object of Spivack could have been substituted for the listed item of Chachek because both the object and listed item serve the purpose of being rendered and displayed. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of a rendered item in the AR environment. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the object of Spivack for the listed item of Chachek according to known methods to yield the predictable result of rendering an item in the AR environment.
Regarding claim 5, Spivack in view of Chachek teaches the method of claim 3. Spivack further teaches positioning the rendered information in the AR environment according to a location of at least one of the one or more physical items in the physical environment (Paragraph 0214 – “virtual billboards can be placed in space, pinned to a user or a moveable object, or can be pinned to a physical surface at a physical location”; Note: the virtual billboard, which is the rendered information, is pinned to an object, meaning it is at the location of the object).
Regarding claim 6, Spivack in view of Chachek teaches the method of claim 1. Spivack does not teach detecting, using one or more sensors, that a user of the computing device is viewing an item of the one or more physical items; and selecting the information from data associated with the one or more listed items based on the information corresponding to a listed item that is related to the viewed item. However, Chachek teaches detecting, using one or more sensors, that a user of the computing device is viewing an item of the one or more physical items (Paragraph 0063 – “the system proceeds to perform rapid real-time recognition of the products that are currently imaged in the field-of-view of the imager of the end-user device, to fetch or obtain or lookup or download product data for each recognized product”; Note: the imager, which is a type of sensor, detects items in the field of view of a user device); and selecting the information from data associated with the one or more listed items based on the information corresponding to a listed item that is related to the viewed item (Paragraph 0059, 0063, 0106 – “enabling the system to determine which product is located in real life at which locations in the store, and to further associate each such product with other data or meta-data, for example, manufacturer, product real-life weight, product real-life dimensions, nutritional data of the product, allergy information for the product, product price, discounts or promotions or coupons that are currently associated and/or usable in conjunction with this product, or the like. Such data may be obtained or fetched from an Inventory Database 106 of the store; for example, by identifying in the image the product name “corn flakes”, then identifying in the image a particular manufacturer logo and a particular product size (e.g., 16 oz), and then obtaining the other data regarding this particular product from the Inventory Database 106…the system proceeds to perform rapid real-time recognition of the products that are currently imaged in the field-of-view of the imager of the end-user device, to fetch or obtain or lookup or download product data for each recognized product (e.g., nutrition values, ingredients, allergens information, or the like), to determine which sub-set of the imaged products fulfill the constraints…showing an AR-based indicator for a sale or promotion (shown as AR-based “40% off” tag or label in this example), shown an AR-based indicator for a particular product or type-of-product that is within the captured field-of-view (e.g., indicated as an AR-based information board about ‘Donut’ in this example), or the like”; Note: the system obtains information of the listed item corresponding to the viewed item and selects specific information to display, such as a promotion related to the item). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chachek to detect that a user is viewing an object for the benefit of being able to gain more information about the environment and being able to analyze the object, which may help the user navigate the environment or understand the object better. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chachek to select information on listed items related to the viewed item for the benefit of providing relevant and useful information to the user.
Regarding claim 8, Spivack in view of Chachek teaches the method of claim 1. Spivack further teaches detecting the user interaction by detecting a hand gesture using one or more sensors of the computing device (Paragraph 0236, 0248, 0268 – “The client device 402 includes a network interface 404, a timing module 406, an RF sensor 407, a location sensor 408, an image sensor 409, a location calibration engine 412, a user view manager 414, a user stimulus sensor 416, a motion/gesture sensor 418… the motion sensor includes a first sensor configured to detect user motion of a first human user in the first physical location of the real world environment on the first side of the transparent display, where the first human user can access the transparent display from the first side of the transparent display…the command to instruct the digital assistant can be initiated by voice, touch, eye, gaze, gesture (body, hand, head, arms, legs, limbs, eyes, torso, etc.), text input and/or other command submitted by a user”).
Regarding claim 11, Spivack teaches a system comprising (Paragraph 0235-0236 – “FIG. 4A depicts an example functional block diagram of a client device 402 (e.g., a portable device or a mobile device)…The client device 402 may be any electronic device such as the devices described in conjunction with the client devices 102A-N in the example of FIG. 1 including but not limited to portable devices, a computer, a server, location-aware devices, mobile phones, PDAs, laptops, palmtops, iPhones, cover headsets, heads-up displays, helmet mounted display, head-mounted display, scanned-beam display”; Note: the client device is equivalent to the system):
at least one memory (Paragraph 0257 – “the client device 402 further includes a memory unit 436 and a storage unit 438”);
and at least one processor coupled with the at least one memory, the at least one memory storing computer-readable instructions that, when executed by the at least one processor, cause the system to perform operations comprising (Paragraph 0255-0258 – “the client device 402 further includes a processing unit 434…The processing unit 434 can include one or more processors, CPUs… The memory unit 436 and a storage unit 438 are, in some embodiments, coupled to the processing unit 434… any portion of or all of the functions described of the various example modules in the client device 402 of the example of FIG. 4A can be performed by the processing unit 434”):
capturing image data of a physical environment for displaying an augmented reality (AR) of the physical environment on a display as an AR environment (Fig. 2C, Paragraph 0035, 0227 – “client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) having AR/VR software agent/hardware module having sensors (e.g., the camera and mic and other sensors) that are always on, and the system is always aware of what is happening…a streaming video camera at the physical location of each device or apparatus controlled by AR/VR window manager 380 so there is always a feed of the AR activity from various perspectives of the physical location”; Note: image data of a physical space is captured by the client device. Fig. 2C shows how the AR of the physical environment is displayed; see screenshot of Fig. 2C above);
rendering a virtual object as an AR assistant for display in the AR environment (Paragraph 0132, 0145, 0186 – “The digital assistant avatar repository 124 can store virtual content or digital assistant avatars that can be retrieved for consumption or deployment in a target environment, where the virtual content or digital assistant avatars are contextually relevant, required, or requested.… the content associated with the virtual object which may be an avatar can include a text message, a post, a chat, a conversation or a graphical message…The object or virtual object is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment”; Note: the virtual object, which is a digital assistant, is rendered into the AR environment)
and responsive to detecting a user interaction with the AR assistant in the AR environment, operating the AR assistant in a second mode by generating an output using information associated with one or more items (Paragraph 0204, 0214-0215 – “the digital assistant is able to receive a command (e.g., by the command processing engine 352), either explicitly or implicitly, from a user… The virtual billboard can include one or more of, a note, a review, an offer, an ad, a promotion. A virtual billboard object can be a seed for threaded discussions or interactions. In general, virtual billboards can be placed in space, pinned to a user or a moveable object…the virtual billboard engine 360 can create the virtual billboard responsive to a request of a creator user”; Note: a user can request a virtual billboard, through a command. The virtual billboard is an output that shows information related to an item).
Spivack does not teach operating the AR assistant in a first mode by exploring the AR environment to identify one or more physical items in the physical environment and proactively searching one or more listing servers over a network for one or more listed items related to the one or more physical items, the one or more listed items listed for sale by the one or more listing servers; nor generating an output using information associated with the one or more listed items, the output based at least in part on knowledge of the one or more physical items identified in the first mode. However, Chachek teaches operating the AR assistant (Paragraph 0050, 0054 – “a visual navigation system is provided, optionally implemented as an Augmented Reality (AR) based navigation system which may assist human users in real-life navigation scenarios…system 100 is configured to facilitate real-life navigation and/or localization and/or mapping of users (as well as products, items, store elements, or the like) within a superstore or megastore or other large-size facility, typically spanning numerous aisles and/or cabinets and/or shelves and/or corridors and/or other structures for storing products and for allowing users to walk therein.”; Note: the system is an AR assistant in this case) in a first mode by exploring the AR environment (Paragraph 0015, 0106 – “an end-user device that is determined to be located in front of the Sprite bottles shelf, would show on his screen an animated promotion or animated avatar that suggests to him to purchase this product … On the screen of his smartphone, the user sees a real-time image of the store, with Augmented Reality (AR) content shown as overlay elements within the image; for example, showing AR-based content of a walking path to a destination product, showing an AR-based avatar or character roaming within the store, showing an AR-based indicator for a sale or promotion (shown as AR-based “40% off” tag or label in this example), shown an AR-based indicator for a particular product or type-of-product that is within the captured field-of-view (e.g., indicated as an AR-based information board about “Donut” in this example), or the like”; Note: an animated avatar, which is part of the system, explores/roams the environment) to identify one or more physical items in the physical environment (Paragraph 0059 – “a Computer Vision unit 104 and/or an Optical Character Recognition (OCR) unit 105, which may be implemented as part of each such participating end-user device and/or as part of a remote server, may analyze an image or a captured frame, and may determine that it shows three boxes of “Corn Flakes”; and may optionally detect a logo or a brand-name which may be compared to or match with a list of product makers or product manufacturing”; Note: the system identifies physical items in the environment, such as the corn flakes) and proactively searching one or more listing servers over a network (Paragraph 0055, 0059 – “client/server architecture may be used, such that each one of devices 151-153 may communicate over wireless link(s) (e.g., over Wi-Fi, over a cellular connection, over an Internet connection) with server 150, which may be located in the retail store and/or may be located remotely (e.g., a cloud-computing server). In other embodiments, distributed architecture or peer-to-peer architecture may be used, such that one or some or all of the devices may operate as nodes or as network elements that may receive and/or transmit data to other units…any of the components or units or modules that are shown in FIG. 1, may be implemented as part of server 150…data may be obtained or fetched from an Inventory Database 106 of the store; for example, by identifying in the image the product name “corn flakes”, then identifying in the image a particular manufacturer logo and a particular product size (e.g., 16 oz), and then obtaining the other data regarding this particular product from the Inventory Database 106”; Note: server 150 is equivalent to the listing server since the inventory database 106 is part of it and lists items/products, and it is accessed over a network, such as the internet) for one or more listed items related to the one or more physical items, the one or more listed items listed for sale by the one or more listing servers (Paragraph 0059, 0063 – “enabling the system to determine which product is located in real life at which locations in the store, and to further associate each such product with other data or meta-data, for example, manufacturer, product real-life weight, product real-life dimensions, nutritional data of the product, allergy information for the product, product price, discounts or promotions or coupons that are currently associated and/or usable in conjunction with this product, or the like. Such data may be obtained or fetched from an Inventory Database 106 of the store; for example, by identifying in the image the product name “corn flakes”, then identifying in the image a particular manufacturer logo and a particular product size (e.g., 16 oz), and then obtaining the other data regarding this particular product from the Inventory Database 106…the system proceeds to perform rapid real-time recognition of the products that are currently imaged in the field-of-view of the imager of the end-user device, to fetch or obtain or lookup or download product data for each recognized product (e.g., nutrition values, ingredients, allergens information, or the like), to determine which sub-set of the imaged products fulfill the constraints”; Note: the system identifies a physical item and searches a corresponding listed item and its data. The items are for sale since they are part of a store); and generating an output using information associated with the one or more listed items, the output based at least in part on knowledge of the one or more physical items identified in the first mode (Paragraph 0063, 0106 – “determine which sub-set of the imaged products fulfill the constraints…generate and displayed AR-based or VR-based emphasis of such sub-set of products, for example, by adding a glowing frame around such particular products as displayed, and/or by adding a textual label or a graphical indicator or an animated content, or the like…On the screen of his smartphone, the user sees a real-time image of the store, with Augmented Reality (AR) content shown as overlay elements within the image; for example, showing AR-based content of a walking path to a destination product, showing an AR-based avatar or character roaming within the store, showing an AR-based indicator for a sale or promotion (shown as AR-based “40% off” tag or label in this example), shown an AR-based indicator for a particular product or type-of-product that is within the captured field-of-view (e.g., indicated as an AR-based information board about ‘Donut’ in this example), or the like”; Note: an output related to the listed item is generated and displayed based on information of the physical item. For instance, a 40%-off label is generated as an output, which provides a sale information related to the item). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chachek to have the AR assistant explore the environment and identify physical items in the physical environment for the benefit of being able to gain more information about the environment and being able to analyze the objects, which may help the user navigate the environment or achieve their shopping goals more efficiently (Chachek: Paragraph 0048 – “utilize Augmented Reality (AR) data or datasets or objects or representations, particularly for the purpose of organizing and/or categorizing and/or classifying real-life items or products or locations, and/or for navigating from a first real-life location”). Since Spivack already supports a commerce platform for selling items (Paragraph 0092, 0109 – “further embodiments of the present disclosure include an augmented reality commerce platform administer a marketplace which supports multiple seller entities via an augmented reality environment…One example of an AR environment deployed by the host (e.g., the host server 100 as depicted in the example of FIG. 1) enables users to interact with virtual objects (VOBs) or applications related to shopping and retail in the physical world or online/e-commerce or mobile commerce. Retailers, merchants, commerce/e-commerce platforms, classified ad systems, and other advertisers will be able to pay to promote virtual objects representing coupons and gift cards in physical locations near or within their stores”), it also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chachek to search for items listed for sale in a listing server for the benefit of providing relevant and useful information to the user. Additionally, having the listed items be in a server makes it easier to access all of the items that retailers are trying to advertise and sell so that they can be viewed by users for shopping. Retailers would be able to promote their products, and users would be able to find desired products easier. Additionally, the listing server can help manage and organize all the items that are listed. Finally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chachek to have the output be based on knowledge of the identified physical items for the benefit of informing the user about the item, which can help them locate it and make a decision on whether or not to buy it. For example, “showing an AR-based indicator for a sale or promotion” (Chachek: Paragraph 0106) may help a user determine whether or not the item is worth buying.
Regarding claim 13, Spivack in view of Chachek teaches the system of claim 11. Spivack further teaches wherein generating the output includes rendering the information for display in the AR environment (Paragraph 0191, 0203 – “The content associated with the virtual object can also include one or more of, works of art, animation, games, scenes, limericks, jokes, a performance, an action, a routine which can be rendered or played in 360 degrees or 3D…An object or virtual object in the AR environment is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment and have human perceptible properties to be human discernible or detectable. The object (e.g., VOB) can include a message, a chat, an email, a note, a post, a news item, or any other types of VOBs as disclosed herein”; Note: a virtual object and content related to the virtual object is rendered for display).
Regarding claim 14, Spivack in view of Chachek teaches the system of claim 13. Spivack further teaches wherein rendering the information includes rendering at least one of the one or more items in the AR environment (Paragraph 0203 – “An object or virtual object in the AR environment is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment and have human perceptible properties to be human discernible or detectable”). Spivack does not teach rendering at least one of the one or more listed items in the AR environment. However, Chachek teaches rendering at least one of the one or more listed items in the AR environment (Paragraph 0110-0111 – “the system then checks the Store Map and the Inventory Database, to determine which products are located on that particular shelf; and the system displays on the user's smartphone the relevant output, for example, an actual real-life photo of that shelf on Aisle 5 (e.g., a pre-defined static photo, from a static database that contains several hundreds of images of shelf-portions of that store…display as Augmented Reality (AR) content, as an overlay on the entire screen of the user device or on the particular screen-portion or image-portion that the user selected or tapped or clicked, a representation of those soda bottles of Shelf 13 (e.g., stock images of the products, or actual real-world images of those exact bottles as captured and shared a few minutes earlier by another customer in that same store”; Note: the listed item in this case is the soda bottles since its data is stored in the database, and it is displayed/rendered in the AR environment). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the object of Spivack could have been substituted for the listed item of Chachek because both the object and listed item serve the purpose of being rendered and displayed. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of a rendered item in the AR environment. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the object of Spivack for the listed item of Chachek according to known methods to yield the predictable result of rendering an item in the AR environment.
Regarding claim 15, Spivack in view of Chachek teaches the system of claim 13. Spivack further teaches positioning the rendered information in the AR environment according to a location of at least one of the one or more physical items in the physical environment (Paragraph 0214 – “virtual billboards can be placed in space, pinned to a user or a moveable object, or can be pinned to a physical surface at a physical location”; Note: the virtual billboard, which is the rendered information, is pinned to an object, meaning it is at the location of the object).
Regarding claim 16, Spivack in view of Chachek teaches the system of claim 11. Spivack does not teach detecting, using one or more sensors, that a user of the computing device is viewing an item of the one or more physical items; and selecting the information from data associated with the one or more listed items based on the information corresponding to a listed item that is related to the viewed item. However, Chachek teaches detecting, using one or more sensors, that a user of the computing device is viewing an item of the one or more physical items (Paragraph 0063 – “the system proceeds to perform rapid real-time recognition of the products that are currently imaged in the field-of-view of the imager of the end-user device, to fetch or obtain or lookup or download product data for each recognized product”; Note: the imager, which is a type of sensor, detects items in the field of view of a user device); and selecting the information from data associated with the one or more listed items based on the information corresponding to a listed item that is related to the viewed item (Paragraph 0059, 0063, 0106 – “enabling the system to determine which product is located in real life at which locations in the store, and to further associate each such product with other data or meta-data, for example, manufacturer, product real-life weight, product real-life dimensions, nutritional data of the product, allergy information for the product, product price, discounts or promotions or coupons that are currently associated and/or usable in conjunction with this product, or the like. Such data may be obtained or fetched from an Inventory Database 106 of the store; for example, by identifying in the image the product name “corn flakes”, then identifying in the image a particular manufacturer logo and a particular product size (e.g., 16 oz), and then obtaining the other data regarding this particular product from the Inventory Database 106…the system proceeds to perform rapid real-time recognition of the products that are currently imaged in the field-of-view of the imager of the end-user device, to fetch or obtain or lookup or download product data for each recognized product (e.g., nutrition values, ingredients, allergens information, or the like), to determine which sub-set of the imaged products fulfill the constraints…showing an AR-based indicator for a sale or promotion (shown as AR-based “40% off” tag or label in this example), shown an AR-based indicator for a particular product or type-of-product that is within the captured field-of-view (e.g., indicated as an AR-based information board about ‘Donut’ in this example), or the like”; Note: the system obtains information of the listed item corresponding to the viewed item and selects specific information to display, such as a promotion related to the item). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chachek to detect that a user is viewing an object for the benefit of being able to gain more information about the environment and being able to analyze the object, which may help the user navigate the environment or understand the object better. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chachek to select information on listed items related to the viewed item for the benefit of providing relevant and useful information to the user.
Regarding claim 18, Spivack in view of Chachek teaches the system of claim 11. Spivack further teaches detecting the user interaction by detecting a hand gesture using one or more sensors of the computing device (Paragraph 0236, 0248, 0268 – “The client device 402 includes a network interface 404, a timing module 406, an RF sensor 407, a location sensor 408, an image sensor 409, a location calibration engine 412, a user view manager 414, a user stimulus sensor 416, a motion/gesture sensor 418… the motion sensor includes a first sensor configured to detect user motion of a first human user in the first physical location of the real world environment on the first side of the transparent display, where the first human user can access the transparent display from the first side of the transparent display…the command to instruct the digital assistant can be initiated by voice, touch, eye, gaze, gesture (body, hand, head, arms, legs, limbs, eyes, torso, etc.), text input and/or other command submitted by a user”).
Regarding claim 20, Spivack teaches a computing device (Paragraph 0235-0236 – “FIG. 4A depicts an example functional block diagram of a client device 402 (e.g., a portable device or a mobile device)… The client device 402 may be any electronic device such as the devices described in conjunction with the client devices 102A-N in the example of FIG. 1 including but not limited to portable devices, a computer, a server, location-aware devices, mobile phones, PDAs, laptops, palmtops, iPhones, cover headsets, heads-up displays, helmet mounted display, head-mounted display, scanned-beam display”; Note: the client device is a computing device), comprising:
a display configured to display an augmented reality (AR) of a physical environment as an AR environment (Paragraph 0235, 0244-0245 – “FIG. 4A depicts an example functional block diagram of a client device 402 (e.g., a portable device or a mobile device) that includes single or multi-directional lens as a portal between a physical world and a digital world component of an augmented reality environment…the client device 402 includes a transparent display having a first side and a second side…When viewed from the first side of the transparent display to the second side, a second set of virtual objects are rendered to appear to be overlaid over second physical elements in a second physical location in the real world in the real world environment on the second side of the transparent display”);
a sensor configured to capture image data of the physical environment (Fig. 2C, Paragraph 0035, 0227, 0235-0236 – “client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) having AR/VR software agent/hardware module having sensors (e.g., the camera and mic and other sensors) that are always on, and the system is always aware of what is happening… a streaming video camera at the physical location of each device or apparatus controlled by AR/VR window manager 380 so there is always a feed of the AR activity from various perspectives of the physical location…FIG. 4A depicts an example functional block diagram of a client device 402 (e.g., a portable device or a mobile device) that includes single or multi-directional lens as a portal between a physical world and a digital world component of an augmented reality environment…The client device 402 includes a network interface 404, a timing module 406, an RF sensor 407, a location sensor 408, an image sensor 409”; Note: image data of a physical space is captured. Fig. 2C shows how the physical space is captured and displayed with the AR; see screenshot of Fig. 2C above);
at least one processor (Paragraph 0255-0256 – “the client device 402 further includes a processing unit 434…The processing unit 434 can include one or more processors, CPUs”);
at least one memory storing instructions that, when executed by the at least one processor, cause the computing device to (Paragraph 0257-0258 – “the client device 402 further includes a memory unit 436 and a storage unit 438. The memory unit 436 and a storage unit 438 are, in some embodiments, coupled to the processing unit 434… any portion of or all of the functions described of the various example modules in the client device 402 of the example of FIG. 4A can be performed by the processing unit 434”):
render a virtual object in the AR environment as an AR assistant (Paragraph 0132, 0145, 0186 – “The digital assistant avatar repository 124 can store virtual content or digital assistant avatars that can be retrieved for consumption or deployment in a target environment, where the virtual content or digital assistant avatars are contextually relevant, required, or requested.… the content associated with the virtual object which may be an avatar can include a text message, a post, a chat, a conversation or a graphical message…The object or virtual object is generally digitally rendered or synthesized by a machine (e.g., a machine can be one or more of, client device 102 of FIG. 1, client device 402 of FIG. 4A or server 100 of FIG. 1, server 300 of FIG. 3A) to be presented in the AR environment”; Note: the virtual object, which is a digital assistant, is rendered into the AR environment);
and responsive to detecting a user interaction with the AR assistant in the AR environment, operate the AR assistant in a second mode by generating an output using information associated with one or more items (Paragraph 0204, 0214-0215 – “the digital assistant is able to receive a command (e.g., by the command processing engine 352), either explicitly or implicitly, from a user… The virtual billboard can include one or more of, a note, a review, an offer, an ad, a promotion. A virtual billboard object can be a seed for threaded discussions or interactions. In general, virtual billboards can be placed in space, pinned to a user or a moveable object…the virtual billboard engine 360 can create the virtual billboard responsive to a request of a creator user”; Note: a user can request a virtual billboard, through a command. The virtual billboard is an output that shows information related to an item).
Spivack does not teach operating the AR assistant in a first mode by exploring the AR environment to identify one or more physical items in the physical environment based at least in part on the image data and proactively searching one or more listing servers over a network for one or more listed items related to the one or more physical items, the one or more listed items listed for sale by the one or more listing servers; nor generating an output using information associated with the one or more listed items, the output based at least in part on knowledge of the one or more physical items identified in the first mode. However, Chachek teaches operating the AR assistant (Paragraph 0050, 0054 – “a visual navigation system is provided, optionally implemented as an Augmented Reality (AR) based navigation system which may assist human users in real-life navigation scenarios…system 100 is configured to facilitate real-life navigation and/or localization and/or mapping of users (as well as products, items, store elements, or the like) within a superstore or megastore or other large-size facility, typically spanning numerous aisles and/or cabinets and/or shelves and/or corridors and/or other structures for storing products and for allowing users to walk therein.”; Note: the system is an AR assistant in this case) in a first mode by exploring the AR environment (Paragraph 0015, 0106 – “an end-user device that is determined to be located in front of the Sprite bottles shelf, would show on his screen an animated promotion or animated avatar that suggests to him to purchase this product … On the screen of his smartphone, the user sees a real-time image of the store, with Augmented Reality (AR) content shown as overlay elements within the image; for example, showing AR-based content of a walking path to a destination product, showing an AR-based avatar or character roaming within the store, showing an AR-based indicator for a sale or promotion (shown as AR-based “40% off” tag or label in this example), shown an AR-based indicator for a particular product or type-of-product that is within the captured field-of-view (e.g., indicated as an AR-based information board about “Donut” in this example), or the like”; Note: an animated avatar, which is part of the system, explores/roams the environment) to identify one or more physical items in the physical environment based at least in part on the image data (Paragraph 0059 – “a Computer Vision unit 104 and/or an Optical Character Recognition (OCR) unit 105, which may be implemented as part of each such participating end-user device and/or as part of a remote server, may analyze an image or a captured frame, and may determine that it shows three boxes of “Corn Flakes”; and may optionally detect a logo or a brand-name which may be compared to or match with a list of product makers or product manufacturing”; Note: the system identifies physical items in the environment, such as the corn flakes) and proactively searching one or more listing servers over a network (Paragraph 0055, 0059 – “client/server architecture may be used, such that each one of devices 151-153 may communicate over wireless link(s) (e.g., over Wi-Fi, over a cellular connection, over an Internet connection) with server 150, which may be located in the retail store and/or may be located remotely (e.g., a cloud-computing server). In other embodiments, distributed architecture or peer-to-peer architecture may be used, such that one or some or all of the devices may operate as nodes or as network elements that may receive and/or transmit data to other units…any of the components or units or modules that are shown in FIG. 1, may be implemented as part of server 150…data may be obtained or fetched from an Inventory Database 106 of the store; for example, by identifying in the image the product name “corn flakes”, then identifying in the image a particular manufacturer logo and a particular product size (e.g., 16 oz), and then obtaining the other data regarding this particular product from the Inventory Database 106”; Note: server 150 is equivalent to the listing server since the inventory database 106 is part of it and lists items/products, and it is accessed over a network, such as the internet) for one or more listed items related to the one or more physical items, the one or more listed items listed for sale by the one or more listing servers (Paragraph 0059, 0063 – “enabling the system to determine which product is located in real life at which locations in the store, and to further associate each such product with other data or meta-data, for example, manufacturer, product real-life weight, product real-life dimensions, nutritional data of the product, allergy information for the product, product price, discounts or promotions or coupons that are currently associated and/or usable in conjunction with this product, or the like. Such data may be obtained or fetched from an Inventory Database 106 of the store; for example, by identifying in the image the product name “corn flakes”, then identifying in the image a particular manufacturer logo and a particular product size (e.g., 16 oz), and then obtaining the other data regarding this particular product from the Inventory Database 106…the system proceeds to perform rapid real-time recognition of the products that are currently imaged in the field-of-view of the imager of the end-user device, to fetch or obtain or lookup or download product data for each recognized product (e.g., nutrition values, ingredients, allergens information, or the like), to determine which sub-set of the imaged products fulfill the constraints”; Note: the system identifies a physical item and searches a corresponding listed item and its data. The items are for sale since they are part of a store); and generating an output using information associated with the one or more listed items, the output based at least in part on knowledge of the one or more physical items identified in the first mode (Paragraph 0063, 0106 – “determine which sub-set of the imaged products fulfill the constraints…generate and displayed AR-based or VR-based emphasis of such sub-set of products, for example, by adding a glowing frame around such particular products as displayed, and/or by adding a textual label or a graphical indicator or an animated content, or the like…On the screen of his smartphone, the user sees a real-time image of the store, with Augmented Reality (AR) content shown as overlay elements within the image; for example, showing AR-based content of a walking path to a destination product, showing an AR-based avatar or character roaming within the store, showing an AR-based indicator for a sale or promotion (shown as AR-based “40% off” tag or label in this example), shown an AR-based indicator for a particular product or type-of-product that is within the captured field-of-view (e.g., indicated as an AR-based information board about ‘Donut’ in this example), or the like”; Note: an output related to the listed item is generated and displayed based on information of the physical item. For instance, a 40%-off label is generated as an output, which provides a sale information related to the item). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chachek to have the AR assistant explore the environment and identify physical items in the physical environment for the benefit of being able to gain more information about the environment and being able to analyze the objects, which may help the user navigate the environment or achieve their shopping goals more efficiently (Chachek: Paragraph 0048 – “utilize Augmented Reality (AR) data or datasets or objects or representations, particularly for the purpose of organizing and/or categorizing and/or classifying real-life items or products or locations, and/or for navigating from a first real-life location”). Since Spivack already supports a commerce platform for selling items (Paragraph 0092, 0109 – “further embodiments of the present disclosure include an augmented reality commerce platform administer a marketplace which supports multiple seller entities via an augmented reality environment…One example of an AR environment deployed by the host (e.g., the host server 100 as depicted in the example of FIG. 1) enables users to interact with virtual objects (VOBs) or applications related to shopping and retail in the physical world or online/e-commerce or mobile commerce. Retailers, merchants, commerce/e-commerce platforms, classified ad systems, and other advertisers will be able to pay to promote virtual objects representing coupons and gift cards in physical locations near or within their stores”), it also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chachek to search for items listed for sale in a listing server for the benefit of providing relevant and useful information to the user. Additionally, having the listed items be in a server makes it easier to access all of the items that retailers are trying to advertise and sell so that they can be viewed by users for shopping. Retailers would be able to promote their products, and users would be able to find desired products easier. Additionally, the listing server can help manage and organize all the items that are listed. Finally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chachek to have the output be based on knowledge of the identified physical items for the benefit of informing the user about the item, which can help them locate it and make a decision on whether or not to buy it. For example, “showing an AR-based indicator for a sale or promotion” (Chachek: Paragraph 0106) may help a user determine whether or not the item is worth buying.
Claims 2, 9, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Spivack in view of Chakchek and Kharband et al. (US 20250218139 A1), hereinafter Kharband.
Regarding claim 2, Spivack in view of Chachek teaches the method of claim 1. Spivack teaches wherein generating the output comprises generating an audio output indicative of the information (Paragraph 0191 – “The contextual trigger for the VOB can include any one of or any combination of a user parameter, a time parameter, an activity parameter and a location parameter…the content associated with the virtual object can include a text message, a post, a chat, a conversation or a graphical message. The content associated with the virtual object can also include digital stickers, GIFs, digital tattoos, emoticons, animations, videos, clips, games, photos, images, a 3D scene or an animated 3D object, music, sound, tone, ultrasonic, or other audible content”; Note: the output, which is the virtual object, includes audio). Spivack does not teach “using a large language model” in the limitation: “wherein generating the output comprises using a large language model to generate an audio output indicative of the information”. However, Kharband teaches using a large language model to generate an audio output indicative of the information (Paragraph 0063, 0065 – “The augmented-reality system 200 can process the input data 212 and/or the image data 214 with a generative model 216 to generate a model-generated response. The generative model 216 can include a generative language model, which may include a large language model…The model-generated response can include text data (e.g., a natural language response (e.g., a conversational response, a topic summary, text instructions, a story, a poem, etc.)), image data (e.g., model-generated images that outline a response to the query), audio data (e.g., a plurality of audio signals that may include audio instructions, a song, a conversational audio response, etc.)”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Kharband to use a large language model to generate audio output because the large language model would automate the audio generation and help produce natural language responses, which would enhance the user experience.
Regarding claim 9, Spivack in view of Chachek teaches the method of claim 1. Spivack further teaches detecting the user interaction by receiving audio input indicative of a voice command for the AR assistant (Paragraph 0267 – “the command can be received by the digital assistant through a voice interface of the user interface and the command includes a voice command or speech command initiated by the user”). Spivack does not teach “using a large language model to generate text indicative of the voice command” in the limitation: “detecting the user interaction by receiving audio input indicative of a voice command for the AR assistant and using a large language model to generate text indicative of the voice command”. However, Kharband teaches using a large language model to generate text indicative of the voice command (Paragraph 0063, 0075 – “The augmented-reality system 200 can process the input data 212 and/or the image data 214 with a generative model 216 to generate a model-generated response. The generative model 216 can include a generative language model, which may include a large language model…The generative model 216 may include text encoders, image encoders, other data encoders, and/or one or more decoders, which may decode one or more feature representations into text data…The user input can include…an audio input (e.g., a voice command obtained via an audio sensor (e.g., a microphone)”; Note: the audio input is processed using a large language model, which decodes the input into text). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Kharband to use a large language model to generate text corresponding to the voice command because the large language model would be able to quickly recognize speech and perform a speech-to-text operation. The text would be useful for understanding what the user said so that a response could be generated, and it would also be useful for displaying the conversation.
Regarding claim 12, Spivack in view of Chachek teaches the system of claim 11. Spivack teaches wherein generating the output comprises generating an audio output indicative of the information (Paragraph 0191 – “The contextual trigger for the VOB can include any one of or any combination of a user parameter, a time parameter, an activity parameter and a location parameter…the content associated with the virtual object can include a text message, a post, a chat, a conversation or a graphical message. The content associated with the virtual object can also include digital stickers, GIFs, digital tattoos, emoticons, animations, videos, clips, games, photos, images, a 3D scene or an animated 3D object, music, sound, tone, ultrasonic, or other audible content”; Note: the output, which is the virtual object, includes audio). Spivack does not teach “using a large language model” in the limitation: “wherein generating the output comprises using a large language model to generate an audio output indicative of the information”. However, Kharband teaches using a large language model to generate an audio output indicative of the information (Paragraph 0063, 0065 – “The augmented-reality system 200 can process the input data 212 and/or the image data 214 with a generative model 216 to generate a model-generated response. The generative model 216 can include a generative language model, which may include a large language model…The model-generated response can include text data (e.g., a natural language response (e.g., a conversational response, a topic summary, text instructions, a story, a poem, etc.)), image data (e.g., model-generated images that outline a response to the query), audio data (e.g., a plurality of audio signals that may include audio instructions, a song, a conversational audio response, etc.)”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Kharband to use a large language model to generate audio output because the large language model would automate the audio generation and help produce natural language responses, which would enhance the user experience.
Claims 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Spivack in view of Chachek and Chia et al. (Structural Descriptors for Category Level Object Detection), hereinafter Chia.
Regarding claim 7, Spivack in view of Chachek teaches the method of claim 1. Spivack does not teach identifying the one or more physical objects based on structure data determined from the image data that indicates a geometry of the one or more physical objects. However, Chia teaches identifying the one or more physical objects based on structure data determined from the image data that indicates a geometry of the one or more physical objects (Fig. 17, Paragraph 1 on 2nd Col. of Page 1, Paragraph 1 on 1st Col. of Page 2 – “we introduce a new set of structural descriptors that are constructed from these primitives. The descriptors have a number of properties that make them suitable for object categorization…Specifically, the descriptors harness the geometrical and spatial relationships between each primitive and its neighbors to describe local object structures. Although one can define many geometrical attributes between primitives, we demonstrate that a small set of geometrical attributes when combined with simple spatial relationships yield rich and accurate local structural descriptions of objects…we detect objects present in the image by matching the descriptors with those previously extracted from training images”; Note: objects are detected based on structural descriptions determined from geometric attributes. Fig. 17 shows the identified objects; see screenshot of Fig. 17 below).
PNG
media_image2.png
409
704
media_image2.png
Greyscale
Screenshot of Fig. 17 (taken from Chia)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chia to identify an object based on structure data determined from geometry data because “shape primitives are simple and generic enough to afford great flexibility in modeling object parts and suggests that structural information of an object may be comprehensively described by the primitives…The descriptors have a number of properties that make them suitable for object categorization. Firstly, they are translation and scale invariant. In addition, we use edge pixels to construct the descriptors and hence they are insensitive to moderate illumination changes” (Chia: Paragraph 1 in 2nd Col. of Page 1). In other words, geometry data can easily represent objects and help indicate the objects’ structures enough to determine what the object is.
Regarding claim 17, Spivack in view of Chachek teaches the system of claim 11. Spivack does not teach identifying the one or more physical objects based on structure data determined from the image data that indicates a geometry of the one or more physical objects. However, Chia teaches identifying the one or more physical objects based on structure data determined from the image data that indicates a geometry of the one or more physical objects (Fig. 17, Paragraph 1 on 2nd Col. of Page 1, Paragraph 1 on 1st Col. of Page 2 – “we introduce a new set of structural descriptors that are constructed from these primitives. The descriptors have a number of properties that make them suitable for object categorization…Specifically, the descriptors harness the geometrical and spatial relationships between each primitive and its neighbors to describe local object structures. Although one can define many geometrical attributes between primitives, we demonstrate that a small set of geometrical attributes when combined with simple spatial relationships yield rich and accurate local structural descriptions of objects…we detect objects present in the image by matching the descriptors with those previously extracted from training images”; Note: objects are detected based on structural descriptions determined from geometric attributes. Fig. 17 shows the identified objects; see screenshot of Fig. 17 above). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Chia to identify an object based on structure data determined from geometry data because “shape primitives are simple and generic enough to afford great flexibility in modeling object parts and suggests that structural information of an object may be comprehensively described by the primitives…The descriptors have a number of properties that make them suitable for object categorization. Firstly, they are translation and scale invariant. In addition, we use edge pixels to construct the descriptors and hence they are insensitive to moderate illumination changes” (Chia: Paragraph 1 in 2nd Col. of Page 1). In other words, geometry data can easily represent objects and help indicate the objects’ structures enough to determine what the object is.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Spivack in view of Chakchek, and Champavannarth et al. (US 20220405813 A1), hereinafter Champavannarth.
Regarding claim 10, Spivack in view of Chachek teaches the method of claim 1. Spivack does not teach determining, by the computing device, that a physical item of the one or more physical item corresponds to a listed item that is listed in the one or more listing servers for sale at a lower price than the physical item; and responsive to the determining, displaying a notification to the user in the AR environment that includes an indication of the price of the physical item and a price of the listed item. However, Champavannarth teaches determining, by the computing device, that a physical item of the one or more physical item corresponds to a listed item that is listed in the one or more listing servers (Paragraph 0049-0051, 0054 – “At 606, item information is received…This item number or barcode number is the number associated with the item at the first entity… the item number or barcode number of the product is automatically correlated to a non-identically identified identical product at the second entity. For example, the user device executing the price comparison application can communicate with a price comparison server to request matching information…price and other information (e.g., quantity on hand or sale price) are retrieved from the second entity using the matching information correlated… a price adjustment recommendation is optionally automatically generated as part of the price comparison process”; Note: the physical item and listed items were previously taught by Spivack and Chachek respectively in the rejection of claim 1. In this case, the first entity corresponds to the physical item, and the second entity corresponds to the listed item) for sale at a lower price than the physical item (Paragraph 0044 – “if the quantity of the item at the second entity is low or zero and the price is lower than the price of the item at the first entity, the automatically generated recommendation may be to keep the price the same or increase the price of the item at the first entity because of item scarcity in the area”); and responsive to the determining, displaying a notification to the user in the AR environment that includes an indication of the price of the physical item and a price of the listed item (Paragraph 0044, 0053 – “if the quantity of the item at the second entity is low or zero and the price is lower than the price of the item at the first entity, the automatically generated recommendation may be to keep the price the same or increase the price of the item at the first entity because of item scarcity in the area…price comparison information including price, quantity and other information for one or more products from the first and second entities is displayed. For example, the price comparison information can be displayed in a user interface screen such as that shown in FIG. 5”; Note: the AR environment was previously taught by Spivack in the rejection of claim 1). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Champavannarth to determine if a listed item price is less than the physical item and display the prices because “pricing recommendations can help ensure that a given retailer's pricing on such items in a store remains competitive with competitors in the geographic area of the store” (Champavannarth: Paragraph 0002). In other words, showing the prices keeps the user informed and may assist them in making decisions for purchasing or selling items.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Spivack in view of Chachek and Borhan et al. (US 20130218721 A1), hereinafter Borhan.
Regarding claim 21, Spivack in view of Chachek teaches the method of claim 1. Spivack does not teach receiving, via the AR assistant, a query from a user regarding items available from the one or more listing servers; and generating, by the AR assistant, a response to the query that includes information about a listed item of the one or more listed items that is listed for sale. However, Borhan teaches receiving, via the AR assistant, a query from a user regarding items available from the one or more listing servers (Paragraph 0059-0060, 0083 – “the virtual wallet app executing on the user device may provide features as described below to augment the user's in-store shopping experience… the store management server may have access to the user's prior purchasing behavior, the user's real-time in-store behavior (e.g., which items' barcode did the user scan using the user device, how many times did the user scan the barcodes, did the user engage in comparison shopping by scanning barcodes of similar types of items, and/or the like)…the CSR may comprise an shopping assistant avatar instantiated on the CSR device, with which the consumer may interact with, or the consumer may access the CSR shopping avatar within the consumer mobile wallet…the consumer may provide an indication of interests 231a (e.g., see 427a-b in FIG. 4E; tapping an “add to cart” button, etc.) in the CSR provided items/offers… and the CSR may in turn provide detailed information and/or add the item to shopping cart 233a (e.g., see 439 in FIG. 4G) to the consumer per consumer request”; Note: the CSR, which is an AR assistant, receives an indication of interest or request of an item, which is a user query. The store management server is equivalent to the listing server in this case); and generating, by the AR assistant, a response to the query that includes information about a listed item of the one or more listed items that is listed for sale (Paragraph 0083, 0095 – “the CSR may in turn provide detailed information and/or add the item to shopping cart 233a (e.g., see 439 in FIG. 4G) to the consumer per consumer request… if the request indicates a shopping request (e.g., consumer inquiry on shopping items, see 427a-c in FIG. 4E, etc.), the CSR may retrieve shopping item information and add the item to a shopping cart 331, and provide such to the consumer 337 (e.g., see 434d-e in FIG. 4F)”; Note: the CSR/assistant responds to the query by providing information about an item). Since Spivack already teaches an assistant responding to a command (Paragraph 0206 – “The digital assistant manager 350 can also detect, identify, interpret, react to, respond to implicit commands (e.g., by the command processing engine 352 and/or the artificial intelligence (AI) engine 354). For example, the digital assistance manager 350 can detect activities occurring in the real world, in the AR environment (e.g., with the virtual objects), and/or detect user activity/behavior and trigger an action”), it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Spivack to incorporate the teachings of Borhan to have the AR assistant provide the user with information on a listed item for sale in response to the user’s query for the benefit of helping the user make informed purchases. Spivack supports a commerce platform for selling items (Paragraph 0092, 0109 – “further embodiments of the present disclosure include an augmented reality commerce platform administer a marketplace which supports multiple seller entities via an augmented reality environment…One example of an AR environment deployed by the host (e.g., the host server 100 as depicted in the example of FIG. 1) enables users to interact with virtual objects (VOBs) or applications related to shopping and retail in the physical world or online/e-commerce or mobile commerce. Retailers, merchants, commerce/e-commerce platforms, classified ad systems, and other advertisers will be able to pay to promote virtual objects representing coupons and gift cards in physical locations near or within their stores”). Therefore, having the assistant be a part of the commerce platform would help users better understand the products, enhance the user experience, and allow retailers to promote their items.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Beauchamp et al. (US 20240233286 A1) teaches a commerce system that generates AR environments of virtual objects and products for users. Sibai et al. (US 20160048908 A1) teaches a method of generating a virtual marketplace where a buyer and seller can interact as avatars in the virtual marketplace to view and sell products.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE HAU MA whose telephone number is (571)272-2187. The examiner can normally be reached M-Th 7-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE HAU MA/ Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
