DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, 4-10, 12, 14-18, 20 and 22-26, is/are rejected under 35 U.S.C. 103 as being unpatentable over Bendale et al. U.S. Patent Publication No. 2018/0341811 (hereinafter Bendale).
Consider claim 1, Bendale teaches an electronic device comprising (Figure 2, 200): a first sensor ([0035], one or more cameras, one or more sensors); a second sensor ([0035], one or more cameras, one or more sensors); one or more processors [0035]; and memory storing instructions configured to be executed by the one or more processors, the instructions for ([0035], storage…supporting various functionalities. [0101-0102] and figure 11, storage 1106, memory 1104, processor 1102): while the electronic device is at a first position, obtaining, using at least the first sensor, first sensor data for a physical environment ([0030-0031], the camera system 202 may transmit the captured media content to the AR engine 204 which processes the media content. The AR Engine 204 may combine the images or videos captured by the cameras with some annotations in order to generate augmented media content (e.g., images with annotations); storing, in the memory, a first data set for the physical environment based at least on the first sensor data ([0048], previous frame); receiving a query regarding the physical environment [0033], an annotation may refer to one or more specific objects within content a user is viewing, user is viewing); and in accordance with receiving the query regarding the physical environment and while the electronic device is within a threshold distance of the first position ([0053], motion (change of distance) is less than the threshold): obtaining second sensor data from the sensor ([0053], previous frame and current frame); and using the second sensor data to reference the first data set ([0053] When the motion is less than the threshold, the system 400 may automatically identify the object by accessing from the previous frames an identification of the object, metadata associated with the object, annotations associated with the objects, and masks associated with the objects. The system may automatically determine an annotation for the object based on the accessed identification, metadata, annotations, and mask from previous frames).
Bendale’s [0053] do not appear to specifically disclose data from the second sensor.
However, Bendale teaches in [0030], camera system may also include one or more sensors including, for example, but not limited to, a microphone, a motion sensor, an accelerometer, a gyroscope, a light sensor, a direction sensor, an inertial measurement unit, etc. In particular embodiments, the camera system 202 may capture one or more frames of a media content including, for example, images (including still images and/or video) and/or audio. Each frame of the media content may correspond to at least part of an image captured by one or more cameras.
Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to provide data based on second sensors as taught by Bendale with the benefit that the AR Engine 204 may combine the images or videos captured by the cameras with some annotations in order to generate augmented media content (e.g., images with annotations) as suggested in [0031].
Consider claim 2, Bendale teaches all the limitations of claim 1. In addition, Bendale teaches wherein the instructions further comprise instructions for: receiving an additional query regarding the physical environment ([0033], queries for different annotations (see also figures 3a-c)); and in accordance with receiving the additional query regarding the physical environment and while the electronic device is outside the threshold distance from the first position ([0053], when the motion is greater than the threshold): obtaining, using at least the first sensor, third sensor data for the physical environment ([0053], the system may automatically identify the object using one or more object detectors (i.e., object classifiers) and identify the frames containing the motion as a key frame); and storing, in the memory, a second data set for the physical environment based at least on the third sensor data ([0052-0053], the system 400 may compare a current frame (i.e., the frame being analyzed) with a preceding frame or a previous frame).
Consider claim 4, Bendale teaches all the limitations of claim 1. In addition, Bendale teaches wherein the first sensor comprises a depth sensor, a camera, or a motion sensor [0030].
Consider claim 5, Bendale teaches all the limitations of claim 1. In addition, Bendale teaches wherein using the second sensor data to reference the first data set comprises using the second sensor data to reference the first data set without using the first sensor ([0051], the system 400 may only perform object detection and object refinement on key frames and may not perform object detection and object refinement for non-key frames, thereby resulting in a significant computation reduction).
Consider claim 6, Bendale teaches all the limitations of claim 1. In addition, Bendale teaches wherein the instructions further comprise instructions for, in accordance with receiving the query regarding the physical environment ([0033], queries for different annotations (see also figures 3a-c))and while the electronic device is within the threshold distance of the first position ([0053], motion (change of distance) is less than the threshold): determining, using a third sensor, a direction of gaze ([0033], if the user is viewing (gaze)), wherein using the second sensor data to reference the first data set comprises identifying a physical object aligned with the direction of gaze using the second sensor data and the first data set without using the first sensor ([0051], the system 400 may only perform object detection and object refinement on key frames and may not perform object detection and object refinement for non-key frames, thereby resulting in a significant computation reduction. [0051], non-key frames may contain one or more changes relative to a preceding frame, but those changes are less than a threshold, and thus information from the previous frame can be used in the current frame.[0033], if the user is viewing).
Consider claim 7, Bendale teaches all the limitations of claim 1. In addition, Bendale teaches wherein the first data set is a scene understanding data set ([0053], current frame (see also figure 3a)), wherein the first data set comprises a spatial mesh that represents the physical environment ([0053], object (see objects in figures 3a-c)), and wherein the first data set comprises identities of physical objects at corresponding positions or directions relative to the first position (Figures 3a-c, objects in front of user. [0053], motion/movement of an object).
Consider claim 8, Bendale teaches all the limitations of claim 1. In addition, Bendale teaches wherein the instructions further comprise instructions for: based on a result from using the second sensor data to reference the first data set [0053], presenting content associated with the query ([0033], annotation), wherein presenting content associated with the query comprises presenting visual content associated with the query using a display (See annotations in figures 3a-c (e.g. 312)) and presenting audio content associated with the query using a speaker [0087], audio).
Consider claim 9, it includes the limitations of claim 1 and thus these limitations are rejected by the same reasoning. In addition, Bendale teaches a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device ([0101-0102] and figure 11, storage 1106, memory 1104, processor 1102).
Consider claim 10, it includes the limitations of claim 2, and thus rejected by the same reasoning.
Consider claim 12, it includes the limitations of claims 4-5, and thus rejected by the same reasoning.
Consider claim 14, it includes the limitations of claim 6, and thus rejected by the same reasoning.
Consider claim 15, it includes the limitations of claim 7, and thus rejected by the same reasoning.
Consider claim 16, it includes the limitations of claim 8, and thus rejected by the same reasoning.
Consider claim 17, it includes the limitations of claim 1, and thus rejected by the same reasoning.
Consider claim 18, it includes the limitations of claim 2, and thus rejected by the same reasoning.
Consider claim 20, it includes the limitations of claims 4-5, and thus rejected by the same reasoning.
Consider claim 22, it includes the limitations of claim 6, and thus rejected by the same reasoning.
Consider claim 23, it includes the limitations of claim 7, and thus rejected by the same reasoning.
Consider claim 24, it includes the limitations of claim 8, and thus rejected by the same reasoning.
Consider claim 25, Bendale teaches all the limitations of claim 1. In addition, Bendale teaches wherein the query regarding the physical environment comprises a verbal query from a user ([0089] and figures 3a-e, voice/audio command may be used to access, modify, and search the available information and the results may be shown via annotation or audio feedback).
Consider claim 26, Bendale teaches all the limitations of claim 1. In addition, Bendale teaches wherein the query is received from an application running on the electronic device ([0093] and [0098], system may incorporate to services (e.g. applications) and further refers to Yelp, Fandangeo…).
Claim(s) 3, 11 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bendale as applied to claims 1, 9 and 17 above, and further in view of Xiong U.S. Patent Publication No. 2024/0078765 (hereinafter Xiong).
Consider claim 3, Bendale teaches all the limitations of claim 1. In addition, Bendale teaches wherein the instructions further comprise instructions for: receiving an additional query regarding the physical environment ([0033], queries for different annotations (see also figures 3a-c)); and in accordance with receiving the additional query regarding the physical environment, while the electronic device is within the threshold distance of the first position [0053].
Bendale does not appear to specifically disclose and in accordance with determining that the first data set is older than a threshold duration of time; obtaining, using at least the first sensor, fourth sensor data for the physical environment; and storing, in the memory, a third data set for the physical environment based at least on the fourth sensor data.
However, in a related field of endeavor, Xiong teaches an augmented reality device (abstract) and further teaches and in accordance with determining that the first data set is older than a threshold duration of time ([0055], oldest); obtaining, using at least the first sensor, fourth sensor data for the physical environment ([0055], depth maps and camera); and storing, in the memory, a third data set for the physical environment based at least on the fourth sensor data ([0055], latest depth map, store and images queue).
Therefore, it would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to store a third data set with the benefit that the image queue can be updated by replacing oldest image with the latest captured image as suggested in [0055].
Consider claim 11, it includes the limitations of claim 3, and thus rejected by the same reasoning.
Consider claim 19, it includes the limitations of claim 3, and thus rejected by the same reasoning.
Response to Arguments
Applicant's arguments filed 09/14/2026 have been fully considered but they are not persuasive.
On page 15, Applicant argues that “Bendale does not describe "receiving a query regarding the physical environment," or "in accordance with receiving the query…”” The Office respectfully disagrees for the following reasons.
Bendale teaches [0033], an annotation may refer to one or more specific objects within content a user is viewing, user is viewing…the user 208 may modify the annotations or add additional annotations to the AR media content. Thus, annotation is considered a query.
On page 16, Applicant argues that “Specifically, paragraph 53 describes that system 400 may determine when the motion of an object is greater than a threshold. In contrast, claim 1 requires that the stated actions are performed "while the electronic device is within a threshold distance of the first position" [where the first sensor data is obtained].”The Office respectfully disagrees for the following reasons.
Bendale teaches in [0053], when the motion is less (and thus considered “within”) than the threshold, the system 400 may automatically identify the object by accessing from the previous frames an identification of the object, metadata associated with the object, annotations associated with the objects, and masks associated with the objects. The system may automatically determine an annotation for the object based on the accessed identification, metadata, annotations, and mask from previous frames. Consequently, these arguments have been considered but they are not persuasive.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERTO W FLORES whose telephone number is (571)272-5512. The examiner can normally be reached Monday-Friday, 7am-4pm, EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, AMR A AWAD can be reached at (571)272-7764. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERTO W FLORES/Primary Examiner, Art Unit 2621
Prosecution Insights