DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims in Consideration
Claims 1-20 are pending in this application.
Response to Arguments
Applicant's arguments filed 3/20/2026, to independent claims have been fully considered but they are not persuasive. The applicant alleges that Nickerson does not teach responsive to the input, obtaining, based on the tracking data, a current estimated position of the augmented reality device; and storing in association with the environment map, a pinned position for the virtual object corresponding to the current estimated position of the augmented reality device.
Specifically the applicant alleges that Nickerson's approach is fundamentally different from the claimed pinned position for the virtual object, which "correspond[s] to the current estimated position of the augmented reality device," such as when the user physically moves the device to the desired location and initiates pinning, and the device's tracked position becomes the pinned position.
The examiner respectfully disagrees with the applicants interpretation. FIG. 5 describes the process of obtaining spatial data, determining physical location, defining a spatial anchor, and adding digital content to the same. This implicitly teaches that the AR environment is relying on current physical data. Otherwise it would no longer be augmented reality--which is drawn over real reality in real time, but, instead, it would virtual reality--depicting fantasy, whether based on real-world images or fantastic images. Nickerson is teaching Augmented reality: The diagram 300 is generally representative of a residential living space, as seen through the viewfinder of a spatial computing device, as described above herein, augmented with digital content to form the AR environment (Nickerson, FIG. 0033).
Claim Rejection Notes
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-21 are rejected under 35 U.S.C. 103 as being unpatentable over Nickerson et al. (US 20220198764 A1, published: 6/23/2022) and Lazarow et al. (US 20180288386 A1, published: 10/4/2018).
Claim 1. (Currently Amended): Nickerson teaches a method for creating virtual objects pinned to respective physical positions in an augmented reality view of a localized physical environment, the method comprising:
obtaining an environment map for the localized physical environment, the environment map associated with an anchor point in the localized physical environment (based on the identified location of the Spatial Anchor ID, the spatial computing device 404, such as via the Live View application, places the digital content included into an AR environment created according to the spatial map file and based on the previously scanned physical environment. The digital content is placed into the AR environment based on data of the spatial map file, at locations, both positionally and rotationally, relative to the identified location of the Spatial Anchor [Nickerson, 0038]);
generating first tracking data to track an estimated position of the augmented reality device relative to the anchor point associated with the environment map for the localized physical environment (the geographic location of the computing device approximating a geographic location of a physical environment, transmitting the geographic location to a server having access to one or more spatial map files, receiving a spatial map file corresponding to the geographic location, determining position and rotation information of a Spatial Anchor in the physical environment as specified in the received spatial map file, and generating an augmented reality environment based on the spatial map file by locating digital content indicated in the spatial map file at locations determined relative to the determined position and rotation of the Spatial Anchor [Nickerson, 0004]);
detecting an input requesting to pin a virtual object in the localized physical environment; responsive to the input, obtaining, based on the tracking data, a current estimated position of the augmented reality device (when an anchor is first created in the physical environment from which the 3D spatial data is captured, an Azure Spatial Anchor Identification (ID) may be created. In some examples, the anchor may be placed by a user. In other examples, the anchor may be automatically placed without specific action of the user to place the anchor. Azure is a software product offered by MICROSOFT. The Azure Spatial Anchor ID may be associated with the anchor (e.g., be a unique ID of the anchor) and may be stored in the spatial map file as a string or any other suitable data type. In addition to the position and location data defining a location of each digital content object added to the spatial map, as discussed above, each digital content object may be further associated with various properties [Nickerson, 0020]); and
storing in association with the environment map, a pinned position for the virtual object corresponding to the current estimated position of the augmented reality device (the Azure Spatial Anchor ID may be associated with the anchor (e.g., be a unique ID of the anchor) and may be stored in the spatial map file as a string or any other suitable data type [Nickerson, 0020]. Referring now to FIG. 3, a diagram 300 of an AR environment is shown. The diagram 300 is generally representative of a residential living space, as seen through the viewfinder of a spatial computing device, as described above herein, augmented with digital content to form the AR environment. The spatial computing hub may store the information related to the digital content 302, 304, and 306 in the spatial map file along with the spatial map generated from the 3D spatial data and the identification of the Spatial Anchor [Nickerson, 0033]; [Nickerson, 0044]. [Nickerson, FIG. 5]; Examiner's Note: the examiner has added in more of 0033 that make clear that environment is a localized position within a small residential space. Further, FIG. 5 describes the process of obtaining spatial data, determining physical location, defining a spatial anchor, and adding digital content to the same. This implicitly teaches that the AR environment is relying on current physical data. Otherwise it would no longer be augmented reality--which is drawn over real reality, but virtual reality--depicting fantasy).
Nickerson does not teach obtaining motion sensor data from an inertial measurement unit of an augmented reality device; generating, based at least in part on the motion sensor data, first tracking data to track an estimated position of the augmented reality device with the environment map for the localized physical environment.
However Lazarow teaches obtaining motion sensor data from an inertial measurement unit of an augmented reality device; generating, based at least in part on the motion sensor data, first tracking data to track an estimated position of the augmented reality device with the environment map for the localized physical environment (the local map data 36 may include the recorded rotational and translational motions of the display device 30 tracked by the visual sensors and/or inertial measurement sensors 18 in the display device 30 [Lazarow, 0033]. As the users roam about the room 306, the sensors 18 within the first display device 30 and the second display device 34 capture visual and/or inertial tracking data and thereby track the rotational and translational motion of the display devices through the sensor devices 18, which observe the three-dimensional rotation and translation of the sensor device 18 to be recorded as poses 62A-G and keyframes 60A-G, which are subsequently stored as local map data 36 in the first display device 30 and local map data in the second device 34 [Lazarow, 0043]. The information for the map data may be generated by at least a sensor device in a plurality of display devices sending sensor data, including the rotational and translational motion tracked by the sensor device, to the computing device 200 in sets that are configured as keyframes 60A-G and a pose graph 80 linking poses 62A-H [Lazarow, 0044]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the augmented reality pinned virtual objects invention of Nickerson to include the motion sensed feature of Lazarow.
One would have been motivated to make this modification to improve augmented reality by using live moving imagery to be inserted into the AR environment. Because the environment is rarely (if ever) still, it would make sense to incorporate real-world motion into augment reality.
Claims 12 and 20, sharing similar deficiencies of claim 1, are likewise rejected.
Claim 2: The combination of Nickerson and Lazarow, teaches the method of claim 1. Nickerson further teaches wherein the environment map comprises one or more feature points described relative to the anchor point, the at least one or more feature points including visual features detectable from image data of the localized physical environment and position data associated with estimated positions of the visual features relative to an anchor point (computing device 100 is configured to implement at least some of the features disclosed herein, for example, the spatially aware computing described herein, including the capturing of 3D spatial data, the creation, storing, and hosting of a spatial map file, presenting a Live View application to a user to create an AR environment, re-localizing a device in an AR environment, etc. In various embodiments, for instance, the features of this disclosure are implemented using hardware, firmware, and/or software (e.g., such as software modules) installed to run on hardware. In some embodiments, the software utilizes one or more software development kits (SDKs) or SDK functions to perform at least some of the features/methods of this disclosure [Nickerson, 0025]. The diagram 200 illustrates a Spatial Anchor 202 defined as a corner point of a structural feature of the physical environment. While only the Spatial Anchor 202 is shown in FIG. 2, any number of Spatial Anchors may be present and that may be located at any user-defined, or automatically selected or assigned, locations [Nickerson, 0032]).
Claim 13, sharing similar deficiencies of claim 2, is likewise rejected.
Claim 3: The combination of Nickerson and Lazarow, teaches the method of claim 2. Lazarow further teaches wherein obtaining the environment map comprises: sensing motion of the augmented reality device and synchronously capturing the visual features of the localized physical environment as the augmented reality device moves relative to the localized physical environment; generating the feature points based on the motion and the visual features; and storing the feature points to the environment map (data captured by the visible light cameras 18, the depth camera 21, and the inertial motion unit 19 can be used to perform simultaneous location and mapping (SLAM) within the physical environment 9, to thereby produce a map of the physical environment including a mesh of reconstructed surfaces, and to locate the computing device 10 within the map of the physical environment 9 [Lazarow, 0023]).
Claim 14, sharing similar deficiencies of claim 3, is likewise rejected.
Claim 4: The combination of Nickerson and Lazarow, teaches the method of claim 2. Nickerson further teaches wherein obtaining the environment map comprises: detecting correspondence between visual features observed by the augmented reality device and the visual features stored to the environment map; and identifying the environment map from a stored set of environment maps based on the detected correspondence (while viewing the spatial map, the user may interact with the spatial computing device to place digital content into the spatial map corresponding to certain locations in the spatial map or modify previously placed digital content, such as by the user previously, by another user, or automatically without user input. The location of the digital content may be transmitted to the spatial computing hub and stored by the spatial computing hub in the spatial map file so that it may be later recalled by the user using the spatial computing device, or another device, or recalled by other users using the spatial computing device, or other devices [Nickerson, 0018]).
Claim 15, sharing similar deficiencies of claim 4, is likewise rejected.
Claim 5: The combination of Nickerson and Lazarow, teaches the method of claim 1. Nickerson further teaches wherein detecting the input requesting to pin the virtual object comprises at least one of: detecting a selection of a user interface control element of the augmented reality device; detecting a predefined voice control input; and detecting a predefined gesture associated with motion of the augmented reality device (while viewing the spatial map, the user may interact with the spatial computing device to place digital content into the spatial map corresponding to certain locations in the spatial map or modify previously placed digital content, such as by the user previously, by another user, or automatically without user input [Nickerson, 0018]).
Claim 16, sharing similar deficiencies of claim 5, is likewise rejected.
Claim 6: The combination of Nickerson and Lazarow, teaches the method of claim 1. Nickerson further teaches further comprising: prior to detecting the input requesting to pin the virtual object, obtaining via a user interface, content associated with the virtual object (while viewing the spatial map, the user may interact with the spatial computing device to place digital content into the spatial map corresponding to certain locations in the spatial map or modify previously placed digital content, such as by the user previously, by another user, or automatically without user input [Nickerson, 0018]).
Claim 17, sharing similar deficiencies of claim 6, is likewise rejected.
Claim 7: The combination of Nickerson and Lazarow, teaches the method of claim 1. Nickerson further teaches further comprising: subsequent to detecting the input, generating a user prompt for a user to input content associated with the virtual object; and obtaining the content associated with the virtual object responsive to the user prompt (while viewing the spatial map, the user may interact with the spatial computing device to place digital content into the spatial map corresponding to certain locations in the spatial map or modify previously placed digital content, such as by the user previously, by another user, or automatically without user input [Nickerson, 0018]).
Claim 18, sharing similar deficiencies of claim 7, is likewise rejected.
Claim 8: The combination of Nickerson and Lazarow, teaches the method of claim 1. Nickerson further teaches wherein the virtual object comprises at least one of: text, an image, an animation, a video, a structured procedure, an interactive form, and a control element associated with accessing media content (signs located in the spatial map and resulting from the data provided by the SLAM capable device may be replaced in the spatial map with interactive digital content [Nickerson, 0018]).
Claim 19, sharing similar deficiencies of claim 8, is likewise rejected.
Claim 9. (Currently Amended): The combination of Nickerson and Lazarow, teaches the method of claim 1. Nickerson further teaches further comprising: subsequent to storing the pinned position for the virtual object, determining that the augmented reality device is in the localized physical environment associated with the environment map; generating second tracking data relative to the anchor point specified in the environment map for the localized physical environment; detecting, based on the second tracking data, that the pinned position is within a field of view of the augmented reality device; and rendering an augmented reality view of the virtual object at the pinned position (the geographic location of the computing device approximating a geographic location of a physical environment, transmitting the geographic location to a server having access to one or more spatial map files, receiving a spatial map file corresponding to the geographic location, determining position and rotation information of a Spatial Anchor in the physical environment as specified in the received spatial map file, and generating an augmented reality environment based on the spatial map file by locating digital content indicated in the spatial map file at locations determined relative to the determined position and rotation of the Spatial Anchor [Nickerson, 0004]. when an anchor is first created in the physical environment from which the 3D spatial data is captured, an Azure Spatial Anchor Identification (ID) may be created. In some examples, the anchor may be placed by a user. In other examples, the anchor may be automatically placed without specific action of the user to place the anchor [Nickerson, 0020]; Examiner's Note: wherein tracking can be done a second, third, or one hundred and third time, since computers can rerun the same program over and over again).
Claim 10: The combination of Nickerson and Lazarow, teaches the method of claim 9. Nickerson further teaches wherein determining that the augmented reality device is in the localized physical environment associated with the environment map comprises: detecting correspondence between visual features observed by the augmented reality device and the visual features associated with feature points stored to the environment map; and identifying the environment map from a stored set of environment maps based on the detected correspondence (while viewing the spatial map, the user may interact with the spatial computing device to place digital content into the spatial map corresponding to certain locations in the spatial map or modify previously placed digital content, such as by the user previously, by another user, or automatically without user input [Nickerson, 0018]. Based on the geographic location of the spatial computing device 404, the spatial computing hub 402 may select a spatial map file corresponding to the geographic location of the spatial computing device 404 [Nickerson, 0037]).
Claim 11: The combination of Nickerson and Lazarow, teaches the method of claim 10. Nickerson further teaches wherein generating the second tracking data comprises: initializing an estimated initial position of the augmented reality device relative to the anchor point based on the visual features observed by the augmented reality device and position data associated with the feature points in the environment map (determining, via a user device, three-dimensional spatial data for a physical environment, determining a geographic location of the physical environment, assigning a Spatial Anchor in the physical environment, and creating a digital element in an augmented reality environment created based on the spatial data of the physical environment, the digital element located in the augmented reality environment relative to a position of the Spatial Anchor [Nickerson, 0003]).
Claim 21. (New): The combination of Nickerson and Lazarow, teaches the method of claim 1, further comprising physically positioning the augmented reality device at the pinned position in the localized physical environment, the input requesting to pin the virtual object being detected at the pinned position ([Nickerson, FIG. 5]; Examiner's Note: as illustrated by the flow chart).
Conclusion
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SETH A SILVERMAN whose telephone number is (571)272-9783. The examiner can normally be reached Mon-Thur, 8AM-4PM MST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Queler can be reached at (571)272-4140. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Seth A Silverman/Primary Examiner, Art Unit 2172
Prosecution Insights