DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims in Consideration
Claim 21 has been canceled.
Claims 22-24, are new.
Claims 1-20, and 22-24, are pending in this application.
Response to Arguments
Applicant’s arguments, filed 8/11/2026, with respect to the rejection(s) of independent claim(s) under 35 USC 103, to the combination of Nickerson and Lazarow, have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of the combination of Nickerson, Lazarow, and Hazas, wherein Hazas has been added to cure the previous rejection deficiencies.
Claim Rejection Notes
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20, and 22-24, are rejected under 35 U.S.C. 103 as being unpatentable over Nickerson et al. (US 20220198764 A1, published: 6/23/2022), in view of Lazarow et al. (US 20180288386 A1, published: 10/4/2018), and in further view of Hazas et al. ("A Relative Positioning System for Co-located Mobile Devices", published: 5/4/2005, Lancaster University, United Kingdom, University of Karlsruhe, Germany, https://www.usenix.org/legacy/publications/library/proceedings/mobisys05/tech/full_papers/hazas/hazas_html/MobiSys2005.html).
Claim 1. (Currently Amended): Nickerson teaches a method for creating virtual objects pinned to respective physical positions in an augmented reality view of a localized physical environment, the method comprising:
obtaining, by an augmented reality device (the processor 130 further comprises processing logic configured to execute a spatial computing computer program product 160 that is configured to perform spatial computing and/or implement the spatial computing environment (e.g., such as capturing of data to form a spatial map file and or re-localizing of a spatial computing device in an AR environment) as described herein [Nickerson, 0028]), an environment map for the localized physical environment, the environment map associated with an anchor point in the localized physical environment (based on the identified location of the Spatial Anchor ID, the spatial computing device 404, such as via the Live View application, places the digital content included into an AR environment created according to the spatial map file and based on the previously scanned physical environment. The digital content is placed into the AR environment based on data of the spatial map file, at locations, both positionally and rotationally, relative to the identified location of the Spatial Anchor [Nickerson, 0038]);
generating first tracking data to track an estimated position of the augmented reality device relative to the anchor point associated with the environment map for the localized physical environment (the geographic location of the computing device approximating a geographic location of a physical environment, transmitting the geographic location to a server having access to one or more spatial map files, receiving a spatial map file corresponding to the geographic location, determining position and rotation information of a Spatial Anchor in the physical environment as specified in the received spatial map file, and generating an augmented reality environment based on the spatial map file by locating digital content indicated in the spatial map file at locations determined relative to the determined position and rotation of the Spatial Anchor [Nickerson, 0004]).
Nickerson does not teach obtaining motion sensor data from an inertial measurement unit of the augmented reality device; generating, based at least in part on the motion sensor data, first tracking data to track an estimated position of the augmented reality device with the environment map for the localized physical environment; detecting, by the augmented reality device, that the augmented reality device is positioned at a current estimated position within the localized physical environment based on the first tracking data.
However Lazarow teaches obtaining motion sensor data from an inertial measurement unit of the augmented reality device; generating, based at least in part on the motion sensor data, first tracking data to track an estimated position of the augmented reality device with the environment map for the localized physical environment; detecting, by the augmented reality device, that the augmented reality device is positioned at a current estimated position within the localized physical environment based on the first tracking data (the local map data 36 may include the recorded rotational and translational motions of the display device 30 tracked by the visual sensors and/or inertial measurement sensors 18 in the display device 30 [Lazarow, 0033]. As the users roam about the room 306, the sensors 18 within the first display device 30 and the second display device 34 capture visual and/or inertial tracking data and thereby track the rotational and translational motion of the display devices through the sensor devices 18, which observe the three-dimensional rotation and translation of the sensor device 18 to be recorded as poses 62A-G and keyframes 60A-G, which are subsequently stored as local map data 36 in the first display device 30 and local map data in the second device 34 [Lazarow, 0043]. The information for the map data may be generated by at least a sensor device in a plurality of display devices sending sensor data, including the rotational and translational motion tracked by the sensor device, to the computing device 200 in sets that are configured as keyframes 60A-G and a pose graph 80 linking poses 62A-H [Lazarow, 0044]).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the augmented reality pinned virtual objects invention of Nickerson to include the motion sensed feature of Lazarow.
One would have been motivated to make this modification to improve augmented reality by using live moving imagery to be inserted into the AR environment. Because the environment is rarely (if ever) still, it would make sense to incorporate real-world motion into augment reality.
The combination of Nickerson and Lazarow, does not teach detecting, by the augmented reality device and while the augmented reality device is positioned at the current estimated position, an input requesting to pin a virtual object at the current estimated position in the localized physical environment; responsive to the input, storing, in association with the environment map, the current estimated position as a pinned position for the virtual object, the pinned position being described relative to the anchor point.
However, Hazas teaches detecting, by the augmented reality device and while the augmented reality device is positioned at the current estimated position, an input requesting to pin a virtual object at the current estimated position in the localized physical environment; responsive to the input, storing, in association with the environment map, the current estimated position as a pinned position for the virtual object, the pinned position being described relative to the anchor point (most of the information maintained in the Relate model is derived from processing of sensor observations in the pipeline described above. Some attributes however need to be configured when a Relate host is first set up. These include host type and dongle orientation with respect to the host device. Relate users can set these attributes through a graphical configuration tool. The tool allows users to describe where the Relate dongle is physically connected on their mobile device. This information is required for correct spatial mapping of sensor observations, but cannot be known in general as different devices have their USB connectors in different places [Hazas, 4.3 Host-Specific configuration, pg. 7 bottom- pg. 8 top]).
PNG
media_image1.png
473
1112
media_image1.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the augmented reality pinned virtual objects invention of the combination of Nickerson and Lazarow, to include the estimated position feature of Hazas.
One would have been motivated to make this modification to accurately display proximate objects based on sensor data, in manner easily viewable by users.
Claims 12 and 20, sharing similar deficiencies of claim 1, are likewise rejected.
Claim 2: The combination of Nickerson, Lazarow, and Hazas, teaches the method of claim 1. Nickerson further teaches wherein the environment map comprises one or more feature points described relative to the anchor point, the at least one or more feature points including visual features detectable from image data of the localized physical environment and position data associated with estimated positions of the visual features relative to an anchor point (computing device 100 is configured to implement at least some of the features disclosed herein, for example, the spatially aware computing described herein, including the capturing of 3D spatial data, the creation, storing, and hosting of a spatial map file, presenting a Live View application to a user to create an AR environment, re-localizing a device in an AR environment, etc. In various embodiments, for instance, the features of this disclosure are implemented using hardware, firmware, and/or software (e.g., such as software modules) installed to run on hardware. In some embodiments, the software utilizes one or more software development kits (SDKs) or SDK functions to perform at least some of the features/methods of this disclosure [Nickerson, 0025]. The diagram 200 illustrates a Spatial Anchor 202 defined as a corner point of a structural feature of the physical environment. While only the Spatial Anchor 202 is shown in FIG. 2, any number of Spatial Anchors may be present and that may be located at any user-defined, or automatically selected or assigned, locations [Nickerson, 0032]).
Claim 13, sharing similar deficiencies of claim 2, is likewise rejected.
Claim 3: The combination of Nickerson, Lazarow, and Hazas, teaches the method of claim 2. Lazarow further teaches wherein obtaining the environment map comprises: sensing motion of the augmented reality device and synchronously capturing the visual features of the localized physical environment as the augmented reality device moves relative to the localized physical environment; generating the feature points based on the motion and the visual features; and storing the feature points to the environment map (data captured by the visible light cameras 18, the depth camera 21, and the inertial motion unit 19 can be used to perform simultaneous location and mapping (SLAM) within the physical environment 9, to thereby produce a map of the physical environment including a mesh of reconstructed surfaces, and to locate the computing device 10 within the map of the physical environment 9 [Lazarow, 0023]).
Claim 14, sharing similar deficiencies of claim 3, is likewise rejected.
Claim 4: The combination of Nickerson, Lazarow, and Hazas, teaches the method of claim 2. Nickerson further teaches wherein obtaining the environment map comprises: detecting correspondence between visual features observed by the augmented reality device and the visual features stored to the environment map; and identifying the environment map from a stored set of environment maps based on the detected correspondence (while viewing the spatial map, the user may interact with the spatial computing device to place digital content into the spatial map corresponding to certain locations in the spatial map or modify previously placed digital content, such as by the user previously, by another user, or automatically without user input. The location of the digital content may be transmitted to the spatial computing hub and stored by the spatial computing hub in the spatial map file so that it may be later recalled by the user using the spatial computing device, or another device, or recalled by other users using the spatial computing device, or other devices [Nickerson, 0018]).
Claim 15, sharing similar deficiencies of claim 4, is likewise rejected.
Claim 5: The combination of Nickerson, Lazarow, and Hazas, teaches the method of claim 1. Nickerson further teaches wherein detecting the input requesting to pin the virtual object comprises at least one of: detecting a selection of a user interface control element of the augmented reality device; detecting a predefined voice control input; and detecting a predefined gesture associated with motion of the augmented reality device (while viewing the spatial map, the user may interact with the spatial computing device to place digital content into the spatial map corresponding to certain locations in the spatial map or modify previously placed digital content, such as by the user previously, by another user, or automatically without user input [Nickerson, 0018]).
Claim 16, sharing similar deficiencies of claim 5, is likewise rejected.
Claim 6: The combination of Nickerson, Lazarow, and Hazas, teaches the method of claim 1. Nickerson further teaches further comprising: prior to detecting the input requesting to pin the virtual object, obtaining via a user interface, content associated with the virtual object (while viewing the spatial map, the user may interact with the spatial computing device to place digital content into the spatial map corresponding to certain locations in the spatial map or modify previously placed digital content, such as by the user previously, by another user, or automatically without user input [Nickerson, 0018]).
Claim 17, sharing similar deficiencies of claim 6, is likewise rejected.
Claim 7: The combination of Nickerson, Lazarow, and Hazas, teaches the method of claim 1. Nickerson further teaches further comprising: subsequent to detecting the input, generating a user prompt for a user to input content associated with the virtual object; and obtaining the content associated with the virtual object responsive to the user prompt (while viewing the spatial map, the user may interact with the spatial computing device to place digital content into the spatial map corresponding to certain locations in the spatial map or modify previously placed digital content, such as by the user previously, by another user, or automatically without user input [Nickerson, 0018]).
Claim 18, sharing similar deficiencies of claim 7, is likewise rejected.
Claim 8: The combination of Nickerson, Lazarow, and Hazas, teaches the method of claim 1. Nickerson further teaches wherein the virtual object comprises at least one of: text, an image, an animation, a video, a structured procedure, an interactive form, and a control element associated with accessing media content (signs located in the spatial map and resulting from the data provided by the SLAM capable device may be replaced in the spatial map with interactive digital content [Nickerson, 0018]).
Claim 19, sharing similar deficiencies of claim 8, is likewise rejected.
Claim 9. (Previously Presented): The combination of Nickerson, Lazarow, and Hazas, teaches the method of claim 1. Nickerson further teaches further comprising: subsequent to storing the pinned position for the virtual object, determining that the augmented reality device is in the localized physical environment associated with the environment map; generating second tracking data relative to the anchor point specified in the environment map for the localized physical environment; detecting, based on the second tracking data, that the pinned position is within a field of view of the augmented reality device; and rendering an augmented reality view of the virtual object at the pinned position (the geographic location of the computing device approximating a geographic location of a physical environment, transmitting the geographic location to a server having access to one or more spatial map files, receiving a spatial map file corresponding to the geographic location, determining position and rotation information of a Spatial Anchor in the physical environment as specified in the received spatial map file, and generating an augmented reality environment based on the spatial map file by locating digital content indicated in the spatial map file at locations determined relative to the determined position and rotation of the Spatial Anchor [Nickerson, 0004]. when an anchor is first created in the physical environment from which the 3D spatial data is captured, an Azure Spatial Anchor Identification (ID) may be created. In some examples, the anchor may be placed by a user. In other examples, the anchor may be automatically placed without specific action of the user to place the anchor [Nickerson, 0020]; Examiner's Note: wherein tracking can be done a second, third, or one hundred and third time, since computers can rerun the same program over and over again).
Claim 10: The combination of Nickerson, Lazarow, and Hazas, teaches the method of claim 9. Nickerson further teaches wherein determining that the augmented reality device is in the localized physical environment associated with the environment map comprises: detecting correspondence between visual features observed by the augmented reality device and the visual features associated with feature points stored to the environment map; and identifying the environment map from a stored set of environment maps based on the detected correspondence (while viewing the spatial map, the user may interact with the spatial computing device to place digital content into the spatial map corresponding to certain locations in the spatial map or modify previously placed digital content, such as by the user previously, by another user, or automatically without user input [Nickerson, 0018]. Based on the geographic location of the spatial computing device 404, the spatial computing hub 402 may select a spatial map file corresponding to the geographic location of the spatial computing device 404 [Nickerson, 0037]).
Claim 11: The combination of Nickerson, Lazarow, and Hazas, teaches the method of claim 10. Nickerson further teaches wherein generating the second tracking data comprises: initializing an estimated initial position of the augmented reality device relative to the anchor point based on the visual features observed by the augmented reality device and position data associated with the feature points in the environment map (determining, via a user device, three-dimensional spatial data for a physical environment, determining a geographic location of the physical environment, assigning a Spatial Anchor in the physical environment, and creating a digital element in an augmented reality environment created based on the spatial data of the physical environment, the digital element located in the augmented reality environment relative to a position of the Spatial Anchor [Nickerson, 0003]).
Claim 22: The combination of Nickerson, Lazarow, and Hazas, teaches the method of claim 1. Hazas further teaches wherein detecting that the augmented reality device is positioned at the current estimated position comprises tracking, based on the first tracking data, movement of the augmented reality device through the localized physical environment to the current estimated position ([Hazas, 4.3 Host-specific configuration, Image of above]).
Claim 23: The combination of Nickerson, Lazarow, and Hazas, teaches the method of claim 1. Hazas further teaches wherein detecting the input requesting to pin the virtual object comprises detecting that the augmented reality device is stationary for a predetermined length of time ([Hazas, 4.3 Host-specific configuration, Image of above]).
Claim 24: The combination of Nickerson, Lazarow, and Hazas, teaches the method of claim 1. Hazas further teaches wherein the pinned position defines a position at which the virtual object is rendered in the augmented reality view of the localized physical environment ([Hazas, 4.3 Host-specific configuration, Image of above]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SETH A SILVERMAN whose telephone number is (571)272-9783. The examiner can normally be reached Mon-Thur, 8AM-4PM MST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Queler can be reached at (571)272-4140. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Seth A Silverman/Primary Examiner, Art Unit 2172
Prosecution Insights