DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination - 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17[e], was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17[e] has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR1.114. The applicant’s submission for RCE filed on 22 August 2025 has been entered.
Remarks
This action is in response to the applicant’s RCE filed 22 August 2025, which is in response to the USPTO office action mailed 22 April 2025. Claims 18, 21-24, 29-31, 34 and 35 are amended. Claims 1-17, 25, 27, 28, 32, 33, 36 and 37 are cancelled. Claims 38-42 are added. Claims 18-24, 26, 29-31, 34, 35 and 38-42 are currently pending.
Response to Arguments
With respect to the 35 USC §103 rejections of claims 18-24 and 26-37, the applicant’s arguments are moot in view of a new grounds of rejection, as necessitated by the applicant's amendments.
Claim Rejections - 35 USC § 103
The following is a quotation of 3f5 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 18-24, 29-31, 34, 35 and 39-42 are rejected under 35 U.S.C. 103 as being unpatentable over Ju et al., US 2021/0201029 A1 (hereinafter “Ju”) in view of Nowak-Przygodzki et al., US 2019/0080168 A1 (hereinafter “Nowak-Przygodzki”).
Claim 18: Ju teaches a method comprising: at a device including an image sensor, one or more processors, and non-transitory memory:
capturing, using the image sensor, an image of an environment (Ju, [Fig. 2A], [Fig. 4], [0064] note At step 402 of flowchart 400, visual data of an environment with a real object is captured, where the visual data may correspond to images, video, and/or a scan of a real-world environment having the object. This may therefore be captured by a user's device, such as a mobile phone, wearable computing device, camera, or the like that includes an optical capture device);
detecting a user engagement with an object in the environment based on detecting the object in the image of the environment; and in response to detecting the user engagement with the object (Ju, [0053] note a pointer selection 1014 may allow a user to utilize pointer 1010 to select bike 1006 for object identification):
obtaining information regarding the object (Ju, [0058] note augmented reality objects may correspond to real-world objects, like bike 1006, that includes virtual data for an augmented reality experience. This may include offer A details 1112 that may display additional information for the offer associated with visual indicator 1105 and virtual graphic 1106);
obtaining contextual information of the device at a time at which the image of the environment was captured (Ju, [0054] note Geolocation A 1018 may be used to add identifying data and characteristics to bike 1006 so that bike 1006 is associated with a geolocation. Similarly, a time 1020 may also be added to bike 1006 and/or the corresponding visual or augmented reality data so that time 1020 may be associated with geolocation A 1018 for bike 1006); and
storing, in a database, an entry including the information regarding the object in association with the contextual information (Ju, [0039] note database 116 may store real-world images and/or virtual graphics or indicators for an augmented reality, as well as offer data for the virtual data of the augmented reality experience);
Ju does not explicitly teach receiving a query, wherein the query includes a first portion relating to information regarding objects in entries of the database and a second portion relating to contextual information in the entries of the database; selecting one or more of the entries in the database based on the query; and generating a response to the query based on the one or more of the entries in the database.
However, Nowak-Przygodzki teaches this (Nowak-Przygodzki, [Fig. 3A] note 306, [0055] note the image shortcut setting allows a user to receive data stored at their computing device 302 or otherwise accessible to their computing device 302 in response to directing their camera at one or more objects, [0055] note a user can provide a verbal command… The command can provide the object, conditions, and/or the action for the image shortcut setting… data to be input into the function can include “when I point,” “bike lock,” and “code.”, [0058] note the user can provide the command, “Assistant, when I point to the bike lock, please provide the bike lock code 2-7-1-8.” Subsequently, when the user is wearing the wearable device and points to their bike lock 308 with their finger, the automated assistant can audibly provide the bike lock code through the wearable device or cause the bike lock code to be presented at the display of the wearable device, [Fig. 3B], [0059] note the view 314 shows how the assistant application can provide an output 316 in response to a camera of the computing device 302 being directed at an object (e.g., the bike lock 308), [0017] note context identifier can identify at least one time and/or at least one location, and causing, according to the image shortcut setting, the one or more computer actions to be performed can be further in response to the subsequent image data being provided at a time that matches the at least one time and/or or at a location that matches the at least one location).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the object tracking of Ju with the image shortcuts of Nowak-Przygodzki according to known methods (i.e. providing image shortcuts for an assistant application). Motivation for doing so is that this provides ways of providing more elaborate commands, providing commands with less arduous inputs, providing commands that protect the privacy of a corresponding user, and/or providing commands with additional or alternative benefits (Nowak- Przygodzki, [0001]).
Claim 19: Ju and Nowak-Przygodzki teach the method of claim 18, wherein the information regarding the object includes machine-readable content associated with the object (Ju, [0058] note augmented reality objects may correspond to real-world objects, like bike 1006, that includes virtual data for an augmented reality experience. This may include offer A details 1112 that may display additional information for the offer associated with visual indicator 1105 and virtual graphic 1106).
Claim 20: Ju and Nowak-Przygodzki teach the method of claim 18, wherein the information regarding the object includes an object type of the object (Ju, [0016] note a machine learning model generated from one or more past travels of the object or similar object and types).
Claim 21: Ju and Nowak-Przygodzki teach the method of claim 18, wherein the contextual information includes a time at which the image of the environment was captured (Ju, [0054] note a time 1020 may also be added to bike 1006 and/or the corresponding visual or augmented reality data so that time 1020 may be associated with geolocation).
Claim 22: Ju and Nowak-Przygodzki teach the method of claim 18, wherein the contextual information includes a location of the device at a time at which the image of the environment was captured (Ju, [0054] note Geolocation A 1018 may be used to add identifying data and characteristics to bike 1006 so that bike 1006 is associated with a geolocation).
Claim 23: Ju and Nowak-Przygodzki teach the method of claim 18, wherein the contextual information includes an application executing on the device at a time at which the image of the environment was captured (Ju, [0034] note Augmented reality application 120, [0035] note Transaction application 112, [0038] note Other applications 114 may also include email, texting, voice and IM applications that allow a user to send and receive emails, calls, texts, and other notifications through network 150).
Claim 24: Ju and Nowak-Przygodzki teach the method of claim 18, wherein the contextual information includes an activity of a user of the device being performed at a time at which the image of the environment was captured (Ju, [0053] note a pointer selection 1014 may allow a user to utilize pointer 1010 to select bike 1006 for object identification).
Claim 29: Ju and Nowak-Przygodzki teach the method of claim 18, wherein generating the response includes generating a verbal response (Nowak-Przygodzki, [0058] note the automated assistant can cause the wearable device to audibly or visually present the bike lock code).
Claim 30: Ju and Nowak-Przygodzki teach the method of claim 18, wherein generating the response includes displaying a response window including at least a portion of the image of the environment (Nowak-Przygodzki, [Fig. 3B], [0058] note the automated assistant can cause the wearable device to audibly or visually present the bike lock code).
Claim 31: Ju teaches a device comprising: an image sensor; a non-transitory memory; and one or more processors to:
capture, using the image sensor, an image of an environment (Ju, [Fig. 2A], [Fig. 4], [0064] note At step 402 of flowchart 400, visual data of an environment with a real object is captured, where the visual data may correspond to images, video, and/or a scan of a real-world environment having the object. This may therefore be captured by a user's device, such as a mobile phone, wearable computing device, camera, or the like that includes an optical capture device);
detect a user engagement with an object in the environment based on detecting the object in the image of the environment; and in response to detecting the user engagement with the object (Ju, [0053] note a pointer selection 1014 may allow a user to utilize pointer 1010 to select bike 1006 for object identification):
obtain information regarding the object (Ju, [0058] note augmented reality objects may correspond to real-world objects, like bike 1006, that includes virtual data for an augmented reality experience. This may include offer A details 1112 that may display additional information for the offer associated with visual indicator 1105 and virtual graphic 1106);
obtain contextual information of the device at a time at which the image of the environment was captured (Ju, [0054] note Geolocation A 1018 may be used to add identifying data and characteristics to bike 1006 so that bike 1006 is associated with a geolocation. Similarly, a time 1020 may also be added to bike 1006 and/or the corresponding visual or augmented reality data so that time 1020 may be associated with geolocation A 1018 for bike 1006); and
store, in a database, an entry including the information regarding the object in association with the contextual information (Ju, [0039] note database 116 may store real-world images and/or virtual graphics or indicators for an augmented reality, as well as offer data for the virtual data of the augmented reality experience).
Ju does not explicitly teach receive a query, wherein the query includes a first portion relating to information regarding objects in entries of the database and a second portion relating to contextual information in the entries of the database; select one or more of the entries in the database based on the query; and generate a response to the query based on the one or more of the entries in the database.
However, Nowak-Przygodzki teaches this (Nowak-Przygodzki, [Fig. 3A] note 306, [0055] note the image shortcut setting allows a user to receive data stored at their computing device 302 or otherwise accessible to their computing device 302 in response to directing their camera at one or more objects, [0055] note a user can provide a verbal command… The command can provide the object, conditions, and/or the action for the image shortcut setting… data to be input into the function can include “when I point,” “bike lock,” and “code.”, [0058] note the user can provide the command, “Assistant, when I point to the bike lock, please provide the bike lock code 2-7-1-8.” Subsequently, when the user is wearing the wearable device and points to their bike lock 308 with their finger, the automated assistant can audibly provide the bike lock code through the wearable device or cause the bike lock code to be presented at the display of the wearable device, [Fig. 3B], [0059] note the view 314 shows how the assistant application can provide an output 316 in response to a camera of the computing device 302 being directed at an object (e.g., the bike lock 308), [0017] note context identifier can identify at least one time and/or at least one location, and causing, according to the image shortcut setting, the one or more computer actions to be performed can be further in response to the subsequent image data being provided at a time that matches the at least one time and/or or at a location that matches the at least one location).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the object tracking of Ju with the image shortcuts of Nowak-Przygodzki according to known methods (i.e. providing image shortcuts for an assistant application). Motivation for doing so is that this provides ways of providing more elaborate commands, providing commands with less arduous inputs, providing commands that protect the privacy of a corresponding user, and/or providing commands with additional or alternative benefits (Nowak- Przygodzki, [0001]).
Claim 34: Ju and Nowak-Przygodzki teach the device of claim 31, wherein the one or more processors are to generate the response by generating a verbal response (Nowak-Przygodzki, [0058] note the automated assistant can cause the wearable device to audibly or visually present the bike lock code).
Claim 35: Ju teaches a non-transitory computer-readable medium having instructions encoded thereon which, when executed by a device including a processor and an image sensor, causes the device to:
capture, using the image sensor, an image of an environment (Ju, [Fig. 2A], [Fig. 4], [0064] note At step 402 of flowchart 400, visual data of an environment with a real object is captured, where the visual data may correspond to images, video, and/or a scan of a real-world environment having the object. This may therefore be captured by a user's device, such as a mobile phone, wearable computing device, camera, or the like that includes an optical capture device);
detect a user engagement with an object in the environment based on detecting the object in the image of the environment; and in response to detecting the user engagement with the object (Ju, [0053] note a pointer selection 1014 may allow a user to utilize pointer 1010 to select bike 1006 for object identification):
obtain information regarding the object (Ju, [0058] note augmented reality objects may correspond to real-world objects, like bike 1006, that includes virtual data for an augmented reality experience. This may include offer A details 1112 that may display additional information for the offer associated with visual indicator 1105 and virtual graphic 1106);
obtain contextual information of the device at a time at which the image of the environment was captured (Ju, [0054] note Geolocation A 1018 may be used to add identifying data and characteristics to bike 1006 so that bike 1006 is associated with a geolocation. Similarly, a time 1020 may also be added to bike 1006 and/or the corresponding visual or augmented reality data so that time 1020 may be associated with geolocation A 1018 for bike 1006); and
store, in a database, an entry including the information regarding the object in association with the contextual information (Ju, [0039] note database 116 may store real-world images and/or virtual graphics or indicators for an augmented reality, as well as offer data for the virtual data of the augmented reality experience).
Ju does not explicitly teach receive a query, wherein the query includes a first portion relating to information regarding objects in entries of the database and a second portion relating to contextual information in the entries of the database; select one or more of the entries in the database based on the query; and generate a response to the query based on the one or more of the entries in the database.
However, Nowak-Przygodzki teaches this (Nowak-Przygodzki, [Fig. 3A] note 306, [0055] note the image shortcut setting allows a user to receive data stored at their computing device 302 or otherwise accessible to their computing device 302 in response to directing their camera at one or more objects, [0055] note a user can provide a verbal command… The command can provide the object, conditions, and/or the action for the image shortcut setting… data to be input into the function can include “when I point,” “bike lock,” and “code.”, [0058] note the user can provide the command, “Assistant, when I point to the bike lock, please provide the bike lock code 2-7-1-8.” Subsequently, when the user is wearing the wearable device and points to their bike lock 308 with their finger, the automated assistant can audibly provide the bike lock code through the wearable device or cause the bike lock code to be presented at the display of the wearable device, [Fig. 3B], [0059] note the view 314 shows how the assistant application can provide an output 316 in response to a camera of the computing device 302 being directed at an object (e.g., the bike lock 308), [0017] note context identifier can identify at least one time and/or at least one location, and causing, according to the image shortcut setting, the one or more computer actions to be performed can be further in response to the subsequent image data being provided at a time that matches the at least one time and/or or at a location that matches the at least one location).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the object tracking of Ju with the image shortcuts of Nowak-Przygodzki according to known methods (i.e. providing image shortcuts for an assistant application). Motivation for doing so is that this provides ways of providing more elaborate commands, providing commands with less arduous inputs, providing commands that protect the privacy of a corresponding user, and/or providing commands with additional or alternative benefits (Nowak- Przygodzki, [0001]).
Claim 39: Ju and Nowak-Przygodzki teach the method of claim 1, wherein the query requests information regarding an object identified by the first portion detected during a context identified by second portion (Nowak-Przygodzki, [Fig. 3A] note 306, [0055] note the image shortcut setting allows a user to receive data stored at their computing device 302 or otherwise accessible to their computing device 302 in response to directing their camera at one or more objects, [0017] note context identifier can identify at least one time and/or at least one location, and causing, according to the image shortcut setting, the one or more computer actions to be performed can be further in response to the subsequent image data being provided at a time that matches the at least one time and/or or at a location that matches the at least one location).
Claim 40: Ju and Nowak-Przygodzki teach the method of claim 1, wherein the query introduces the first portion with the word 'what' and the second portion with the word 'when' (Nowak-Przygodzki, [0055] note a user can provide a verbal command… The command can provide the object, conditions, and/or the action for the image shortcut setting… data to be input into the function can include “when I point,” “bike lock,” and “code.”).
Claim 41: Ju and Nowak-Przygodzki teach the method of claim 1, wherein selecting the one or more entries includes selecting the entry based on the first portion matching the information regarding the object and the second portion matching the contextual information (Nowak-Przygodzki, [0058] note the user can provide the command, “Assistant, when I point to the bike lock, please provide the bike lock code 2-7-1-8.” Subsequently, when the user is wearing the wearable device and points to their bike lock 308 with their finger, the automated assistant can audibly provide the bike lock code through the wearable device or cause the bike lock code to be presented at the display of the wearable device).
Claim 42: Ju and Nowak-Przygodzki teach the method of claim 1, wherein selecting the one or more entries includes selecting at least one first entry matching the first portion and selecting at least one second entry matching the second portion (Nowak-Przygodzki, [Fig. 3B], [0059] note the view 314 shows how the assistant application can provide an output 316 in response to a camera of the computing device 302 being directed at an object (e.g., the bike lock 308)).
Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over Ju and Nowak-Przygodzki in further view of TOMIZUKA et al., US 2022/0222900 A1 (hereinafter “Tomizuka”).
Claim 26: Ju and Nowak-Przygodzki do not explicitly teach the method of claim 18, wherein detecting the user engagement with the object is further based on detecting that the user has physically interacted with the object.
However, Tomizuka teaches this (Tomizuka, [0023] note the object 112 may be a real object that is located at the first location 116(1) and the XR environment 108 may represent a real-world environment at the first location 116(1). The first user may interact with the object 112 at the first location 116(1) via the first user device 102(1) executing the XR collaboration application 106. For instance, the first user may physically manipulate the object 112 by changing the position of the object 112 in a real-world environment that is represented as the XR environment 108).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the object tracking of Ju and Nowak-Przygodzki with the object interaction of Tomizuka according to known methods (i.e. determining a target object based on a user interaction). Motivation for doing so is that this provides better tracking and may more accurately render work being performed in real-time (Tomizuka, [0002]).
Claim 38 is rejected under 35 U.S.C. 103 as being unpatentable over Ju and Nowak-Przygodzki in further view of Keating et al., US 2014/0028712 A1 (hereinafter “Keating”).
Claim 38: Ju and Nowak-Przygodzki do not explicitly teach method of claim 1, wherein detecting the user engagement with the object is further based on detecting that the user has looked at the object for at least a threshold amount of time.
However, Keating teaches this (Keating, [0005] note determining whether the object has been selected based at least in part on a set of selection criteria, [0074] note augmentation logic can be configured to overlay audiovisual content (referred to herein as "augmentation") over this view into the real world environment to provide a augmented reality view of the real-world environment. The augmentation logic can provide overlays over the background, foreground, and/or one or more tangible objects within the field of view of the ARD 14, [0047] note the control unit 120 of the ARD may be configured to determine whether the user is looking around. It may perform the following functions, including but not limited to… 3) start augmentation if the user initiates interaction with the object by a) stopping abruptly on the object, b) keeping the object in the camera view for a predetermined period of time, or c) any other direct or indirect means unrelated to velocity).
It would have been obvious to one of ordinary skill in the art at the effective filing date of the application to combine the object tracking of Ju and Nowak-Przygodzki with the object selection criteria of Keating according to known methods (i.e. determining a target object based on keeping objects in a camera view for a predetermined period of time). Motivation for doing so is that this can improve the conventional augmented reality applications (Keating, [0004]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Giuseppi Giuliani whose telephone number is (571)270-7128. The examiner can normally be reached Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aleksandr Kerzhner can be reached at (571)270-1760. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GIUSEPPI GIULIANI/Primary Examiner, Art Unit 2165