DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-10 are pending under this Office action.
Claim Objections
Claim 1 is objected to because of the following informalities: “a cash memory” may be “a cache memory”. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-10 are rejected under 35 U.S.C. 103 as being unpatentable over Emami, etc. (US 20210350612 A1) in view of Cardenas, etc. (US 20220075591 A1).
Regarding claim 1, Emami teaches that a method for generating an XR scene, executed by at least one processor of at least one mobile user device, the method comprising at least the following steps (See Emami: Figs. 4-5, and [0053], “FIG. 4 is a flow diagram depicting an example method 400 for utilizing previously created spatial anchors. In an example, method 400 is performed by a computing device or computing system, such as a mobile device. In this example, the mobile device is operated by a user as a consumer of spatial anchors”; and [0064], “FIG. 5 schematically shows a non-limiting embodiment of a computing system 500 that can enact one or more of the methods and processes described above. Computing system 500 is shown in simplified form. Computing system 500 may take the form of one or more personal computers, server computers, tablet computers, home-entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phone), head-mounted display (HMD) device, an IoT device with sensors, robotic devices (e.g., autonomous robots), and/or other computing devices”):
identifying the mobile user device (See Emami: Fig. 1, and [0016], “Virtual content may be located at a spatial anchor, or may be offset from a spatial anchor with which it is associated. For example, arbitrary virtual content 116A illustrated as a star is located at spatial anchor 114A with which it is associated. As another example, virtual content 116B (e.g., text-based content “15.sup.th century vase”) is offset from spatial anchor 114B with which it is associated. Additionally or alternatively, audio content may be associated with a spatial anchor that is playable or that plays automatically via a mobile device when the mobile device is within a threshold proximity (e.g., 3 meters or other suitable value) to the spatial anchor. For example, the audio content may include a human voice that offers a description of a physical object to which the spatial anchor is linked, such as “15.sup.th century vase” with respect to spatial anchor 114B. Some spatial anchors may not be associated with virtual content or audio content”);
obtaining data for generating an XR scene and storing the obtained data in a cash memory of the mobile user device (See Emami: Fig. 2, and [0034], “At 210, the method includes capturing, via a camera of a mobile device, first image data imaging a first physical world location. In an example, the camera is a visible light camera (e.g., an RGB camera). In another example, two or more cameras may be used to capture image data, including a visible light camera and a depth camera”; [0038], “At 218, the method includes sending, from the mobile device to a network-accessible service, data representing the first spatial representation, the pose of the first virtual spatial anchor, the pose of the first hologram (if defined), and one or more of the first hologram and/or an identifier of the first hologram (if defined). The network-accessible service stores this data in a relationship graph, which may be made accessible to the mobile device across multiple sessions and to other devices”; and [0040], “At 220, the method includes tracking movement of the mobile device from the first physical world location to or toward the second physical world location to capture tracking data. The tracking data describes the movement of the mobile device between the first and second physical world locations in 6DOF space. Movement of the mobile device may be tracked based on sensor data obtained from one or more sensors, including the camera, inertial sensors, geo-positioning sensors (e.g., GPS, wireless network connectivity), etc. The mobile device may use sensor fusion of multiple different types of sensors to track its movement between locations”);
generating a virtual part of the XR scene (See Emami: Fig. 6, and [0077], “FIG. 6 depicts an example in which a mobile device 610 is imaging a first physical world location via its camera. A camera view captured via the camera is displayed via display device 612. In this example, the first physical world location includes a table 614. Overlaid upon the camera view is virtual content 616 and 618. Virtual content 616 is a holographic model of an architectural design that is associated with a spatial anchor that is within the first physical world location. Virtual content 618 is an example of wayfinding information comprising a directional indicator (e.g., arrow) that directs a user to another spatial anchor located outside of the camera view”)”);
obtaining coordinates of a real-world part of the XR scene (See Emami: Fig. 1, and [0015], “Within FIG. 1, a user 110 uses a mobile device 112 to create spatial anchors 114A, 114B, 114C, 114D, etc. at various physical world locations throughout the environment. As an example, environment 100 may be a museum or gallery containing physical works of art, and user 110 may provide a virtual tour by adding spatial anchors to specific works of art. Virtual content, such as holograms, may be associated with spatial anchors. User 110 and other users 120, 130, and 140 may view this virtual content via their respective mobile devices. In this example, user 110 is operating a handheld mobile device, whereas users 120, 130, and 140 are wearing head mounted display (HMD) devices”; and [0017], “As described in further detail herein, a user may create a spatial anchor by imaging the physical environment via a camera, and defining a location and/or an orientation of the spatial anchor relative to a feature captured by the camera. For example, spatial anchor 114D may be located at a corner of a physical picture frame that is captured within a camera view of mobile device 112. Once the location and/or orientation of the spatial has been defined, a position and/or orientation of virtual content may be defined relative to that spatial anchor. The term “pose” may be used herein to describe the position and/or orientation of spatial anchors or their virtual content relative to a mobile device”);
creating spatial anchors (See Emami: Fig. 1, and [0015], “Within FIG. 1, a user 110 uses a mobile device 112 to create spatial anchors 114A, 114B, 114C, 114D, etc. at various physical world locations throughout the environment. As an example, environment 100 may be a museum or gallery containing physical works of art, and user 110 may provide a virtual tour by adding spatial anchors to specific works of art. Virtual content, such as holograms, may be associated with spatial anchors. User 110 and other users 120, 130, and 140 may view this virtual content via their respective mobile devices. In this example, user 110 is operating a handheld mobile device, whereas users 120, 130, and 140 are wearing head mounted display (HMD) devices”) for anchoring the virtual part of the XR scene to the coordinates of the real-world part thereof (See Emami: Fig. 6, and [0077], “FIG. 6 depicts an example in which a mobile device 610 is imaging a first physical world location via its camera. A camera view captured via the camera is displayed via display device 612. In this example, the first physical world location includes a table 614. Overlaid upon the camera view is virtual content 616 and 618. Virtual content 616 is a holographic model of an architectural design that is associated with a spatial anchor that is within the first physical world location. Virtual content 618 is an example of wayfinding information comprising a directional indicator (e.g., arrow) that directs a user to another spatial anchor located outside of the camera view”);
obtaining coordinates of the mobile user device and its position and/or spatial orientation and displaying the generated XR scene based on the obtained coordinates of the mobile user device and its position and/or spatial orientation (See Emami: Fig. 2, and [0035], “At 212, the method includes creating a first spatial representation of a first physical world location based on the first image data. In some examples, the first spatial representation may take the form of a sparse point cloud determined from image data, such as visible RGB image data. The points of the sparse point cloud may have location and/or orientation values defined in relative to the mobile device or its camera. In other exam, any other suitable image data may be provided, such as first depth image data captured by an on-board depth image sensor. For example, the first spatial representation of the first physical world location may be alternatively or additional based on the first depth image data”; [0027], “As the end user moves between spatial anchors, the end-user application may periodically calculate a new pose between the device and the destination anchor. The app then may refine the guidance hints that help the user arrive at the destination”; and Fig. 6, and [0077], “FIG. 6 depicts an example in which a mobile device 610 is imaging a first physical world location via its camera. A camera view captured via the camera is displayed via display device 612. In this example, the first physical world location includes a table 614. Overlaid upon the camera view is virtual content 616 and 618. Virtual content 616 is a holographic model of an architectural design that is associated with a spatial anchor that is within the first physical world location. Virtual content 618 is an example of wayfinding information comprising a directional indicator (e.g., arrow) that directs a user to another spatial anchor located outside of the camera view”. Note that device pose is tracked relative to the anchors, and the scene (virtual content overlaid on real-world camera captured images) is updated and displayed based on current device pose, and this teaching is mapped to the current cited limitation);
wherein only one API suitable for generating XR scenes is used to generate the XR scene (See Emami: Fig. 1, and [0014], “FIG. 1 depicts an example environment 100 within which user-designated virtual spatial anchors may be used to represent points of interest. Such spatial anchors may empower developers to build spatially aware augmented reality and mixed reality applications. These applications may support a variety of mobile device hardware and/or operating systems, including as an example: Microsoft HoloLens™, iOS-based devices supporting ARKit, and Android-based devices supporting ARCore. Spatial anchors enable developers to work with augmented reality and mixed reality platforms to perceive spaces, designate precise points of interest, and to recall those points of interest from supported devices”; and Fig. 5, and [0072], “The terms “module,” “program,” and “engine” may be used to describe an aspect of computing system 500 implemented to perform a particular function. In some cases, a module, program, or engine may be instantiated via logic machine 510 executing instructions held by storage machine 512. It will be understood that different modules, programs, and/or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and/or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module,” “program,” and “engine” may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.”). Note that ARKit and ARCore are APIs); and
wherein, after the mobile user device has been identified, a program code is provided, if necessary, which, when executed by a processor of the mobile user device (See Emami: Fig. 1, and [0032], “In some examples, a spatial anchor app may comprise a managed service and client software development kits (SDKs) for supported device platforms. When an anchor is created, a client SDK captures environment information around that point and transmits it to the service. Then, when another device looks for the anchor in that same space, the device transmits similar data to the service. The service matches the data against the environment information previously stored. The position of the anchor relative to the device is then sent back for use in the application”; and [0074], “When included, display subsystem 514 may be used to present a visual representation of data held by storage machine 512. This visual representation may take the form of a graphical user interface (GUI). Within the context of mixed reality or augmented reality, the GUI may include a camera feed having virtual content or other graphical elements overlaid upon the camera feed, as an example. As the herein described methods and processes change the data held by the storage machine, and thus transform the state of the storage machine, the state of display subsystem 514 may likewise be transformed to visually represent changes in the underlying data. Display subsystem 514 may include one or more display devices utilizing virtually any type of technology. Computing system 500 may further include one or more audio speakers or may include a wired or wireless audio interface by which audio content may be output via peripheral audio speakers (e.g., headphones). Such display devices and/or audio speakers/audio interfaces may be combined with logic machine 510 and/or storage machine 512 in a shared enclosure, or such display devices may be peripheral display and/or audio devices”. Note that APIs are provided to the multiple users and XR scene is created and shared with cross-platform user devices, and this is mapped to the current cited limitation), at least allows to at least partially use the API suitable for generating XR scenes instead of the only one API suitable for generating XR scenes.
However, Emami fails to explicitly disclose that at least allows to at least partially use the API suitable for generating XR scenes instead of the only one API suitable for generating XR scene.
However, Cardenas teaches that at least allows to at least partially use the API suitable for generating XR scenes instead of the only one API suitable for generating XR scene (See Cardenas: Figs. 5A-B, and [0095], “FIGS. 5A and 5B show an example of a first shared device AR experience 500, according to some examples, in which users take turns holding a client device to share an AR experience. Specifically, as shown in FIG. 5A, multiple users 590 are within a specified proximity of each other (e.g., within the same room or within less than 15 feet of each other). The users 590 are sharing a single client device 592. Namely, as shown in FIG. 5A, the single client device 592 is being passed around physically from one user to another user to share the AR experience”; Fig. 6A-B, and [0104], “As shown in FIG. 6B, an augmented reality element (also referred to as augmented reality content item) option, such as a feet filter can be selected and activated on the client device 102”; and Fig. 1, and [0047], “In order to integrate the functions of the SDK into the web-based resource, the SDK is downloaded by an external resource (apps) server 110 from the messaging server 118 or is otherwise received by the external resource (apps) server 110. Once downloaded or received, the SDK is included as part of the application code of a web-based external resource. The code of the web-based resource can then call or invoke certain functions of the SDK to integrate features of the messaging client 104 into the web-based resource”. Note that after the user proximity is identified and activated, the downloaded SDK code is executed in the specific device for shared experiences between users, and this dynamic code activation and execution is mapped to “use the API suitable for generating XR scenes instead of the only one API suitable for generating XR scene”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Emami to have at least allows to at least partially use the API suitable for generating XR scenes instead of the only one API suitable for generating XR scene as taught by Cardenas in order to enable providing the augmented reality (AR) experience to the user in an effective manner (See Cardenas: Figs. 1-2, and [0026], “Image processing server 122 is used to implement scan functionality of the augmentation system 208. Scan functionality includes activating and providing one or more augmented reality experiences on a client device 102 when an image is captured by the client device 102. Specifically, the messaging application 104 on the client device 102 can be used to activate a camera. The camera displays one or more real-time images or a video to a user along with one or more icons or identifiers of one or more augmented reality experiences. The user can select a given one of the identifiers to launch the corresponding augmented reality experience. Launching the augmented reality experience includes obtaining one or more augmented reality items associated with the augmented reality experience and overlaying the augmented reality items on top of the images or video being presented”). Emami teaches a method and system that may generate and share XR scene with multiple users using anchors to connecting the spatial anchors and AR contents; while Cardenas teaches a system and method that may create and share AR content sessions by identifying the user proximity, activating the user device and executing the downloaded SDK codes in the sharing devices dynamically. Therefore, it is obvious to one of ordinary skill in the art to modify Emami by Cardenas to dynamically provide program codes on the sharing device to have an effective sharing experience. The motivation to modify Emami by Cardenas is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 2, Emami and Cardenas teach all the features with respect to claim 1 as outlined above. Further, Emami teaches that the method of claim 1, characterized in that the program code is provided through an app distribution system (See Emami: Fig. 1, and [0015], “Within FIG. 1, a user 110 uses a mobile device 112 to create spatial anchors 114A, 114B, 114C, 114D, etc. at various physical world locations throughout the environment. As an example, environment 100 may be a museum or gallery containing physical works of art, and user 110 may provide a virtual tour by adding spatial anchors to specific works of art. Virtual content, such as holograms, may be associated with spatial anchors. User 110 and other users 120, 130, and 140 may view this virtual content via their respective mobile devices. In this example, user 110 is operating a handheld mobile device, whereas users 120, 130, and 140 are wearing head mounted display (HMD) devices”; and [0032], “In some examples, a spatial anchor app may comprise a managed service and client software development kits (SDKs) for supported device platforms. When an anchor is created, a client SDK captures environment information around that point and transmits it to the service. Then, when another device looks for the anchor in that same space, the device transmits similar data to the service. The service matches the data against the environment information previously stored. The position of the anchor relative to the device is then sent back for use in the application”. Note that the client-server architecture is a distribution system).
Regarding claim 3, Emami and Cardenas teach all the features with respect to claim 1 as outlined above. Further, Emami teaches that the method of claim 1, characterized in that, after the XR scene has been generated, data are obtained from at least an accelerometer and a gyroscope of the mobile user device in order to determine, in sequence, its position and/or spatial orientation (See Emami: Fig. 5, and [0075], “When included, input subsystem 516 may comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, camera, microphone, or game controller. In some embodiments, the input subsystem may comprise or interface with selected natural user input (NUI) componentry. Such componentry may be integrated or peripheral, and the transduction and/or processing of input actions may be handled on- or off-board. Example NUI componentry may include a microphone for speech and/or voice recognition; an infrared, color, stereoscopic, and/or depth camera for machine vision and/or gesture recognition; a head tracker, eye tracker, accelerometer, and/or gyroscope for motion detection and/or intent recognition; as well as electric-field sensing componentry for assessing brain activity”);
wherein the playback of the virtual part of the XR scene is adjusted, if necessary, based on the data obtained from the accelerometer and the gyroscope of the mobile user device (See Emami: Fig. 4, and [0060], “At 420, the method includes tracking movement of the mobile device from the first virtual spatial anchor to the second virtual spatial anchor to capture tracking data”; and [0061], “At 422, the method includes displaying updated information regarding locations of the first and/or second virtual spatial anchors based on the tracking data. As an example, the camera of the mobile device may be moved to a different location or orientation in which the second virtual spatial anchor is within the camera view and the first virtual spatial anchor is outside of the camera view. In this example, a hologram associated with second virtual spatial anchor (or other suitable information) may be displayed at its pose defined relative to the second virtual spatial anchor (such as may be defined at 230 of FIG. 2), whereas a direction indicator that identifies a direction from the mobile device to the first virtual spatial anchor may be displayed while the first virtual spatial anchor resides outside of the camera view. The direction indicator may assist the user with way-finding with respect to the first virtual spatial anchor”. Note that displaying the updated information based on tracking is mapped to adjust the playback of the virtual part of the XR scene).
Regarding claim 6, Emami and Cardenas teach all the features with respect to claim 1 as outlined above. Further, Emami teaches that the method of claim 1, characterized in that the spatial anchors are used to generate an environment mask (See Emami: Fig. 1, and [0014], “FIG. 1 depicts an example environment 100 within which user-designated virtual spatial anchors may be used to represent points of interest. Such spatial anchors may empower developers to build spatially aware augmented reality and mixed reality applications. These applications may support a variety of mobile device hardware and/or operating systems, including as an example: Microsoft HoloLens™, iOS-based devices supporting ARKit, and Android-based devices supporting ARCore. Spatial anchors enable developers to work with augmented reality and mixed reality platforms to perceive spaces, designate precise points of interest, and to recall those points of interest from supported devices”; [0024], “In some examples, building a way-finding experience involves preparing a space for the experience and developing an app that end users will interact with. Such a building process may include planning the space by determining locations within the space that participate in the way-finding experience. This may be performed, for example, by a museum tour coordinator, factory supervisor, or the like, depending upon the environment in which the way-finding experience will be implemented”; and [0026], “As an example end-user experience, a first step for end users may be to locate one of the anchors using an end-user application running on a mobile device, which can be in any one of the chosen locations. In some examples, determining the locations where end users can enter the experience may be a part of designing the experience. Once the end user has located one anchor, the app can request nearby anchors. This procedure returns a pose between the device and these anchors, wherein the term pose may indicate a position and orientation of the anchors relative to the device. The end-user application may then take advantage of the pose to each of these anchors to render useful guidance hints about their general direction and distance. For instance, the end-user application may display an icon and arrow on a camera feed representing each potential destination”. Note that the anchors are designed and placed in such a way that the users to enter this experience are limited to the designed ways or regions, and this limited region or way is mapped to the environment mask).
Regarding claim 7, Emami and Cardenas teach all the features with respect to claim 2 as outlined above. Further, Emami teaches that the method of claim 2, characterized in that the spatial anchors are used to generate an environment mask (See Emami: Fig. 1, and [0014], “FIG. 1 depicts an example environment 100 within which user-designated virtual spatial anchors may be used to represent points of interest. Such spatial anchors may empower developers to build spatially aware augmented reality and mixed reality applications. These applications may support a variety of mobile device hardware and/or operating systems, including as an example: Microsoft HoloLens™, iOS-based devices supporting ARKit, and Android-based devices supporting ARCore. Spatial anchors enable developers to work with augmented reality and mixed reality platforms to perceive spaces, designate precise points of interest, and to recall those points of interest from supported devices”; [0024], “In some examples, building a way-finding experience involves preparing a space for the experience and developing an app that end users will interact with. Such a building process may include planning the space by determining locations within the space that participate in the way-finding experience. This may be performed, for example, by a museum tour coordinator, factory supervisor, or the like, depending upon the environment in which the way-finding experience will be implemented”; and [0026], “As an example end-user experience, a first step for end users may be to locate one of the anchors using an end-user application running on a mobile device, which can be in any one of the chosen locations. In some examples, determining the locations where end users can enter the experience may be a part of designing the experience. Once the end user has located one anchor, the app can request nearby anchors. This procedure returns a pose between the device and these anchors, wherein the term pose may indicate a position and orientation of the anchors relative to the device. The end-user application may then take advantage of the pose to each of these anchors to render useful guidance hints about their general direction and distance. For instance, the end-user application may display an icon and arrow on a camera feed representing each potential destination”. Note that the anchors are designed and placed in such a way that the users to enter this experience are limited to the designed ways or regions, and this limited region or way is mapped to the environment mask).
Regarding claim 8, Emami and Cardenas teach all the features with respect to claim 3 as outlined above. Further, Emami teaches that the method of claim 3, characterized in that the spatial anchors are used to generate an environment mask (See Emami: Fig. 1, and [0014], “FIG. 1 depicts an example environment 100 within which user-designated virtual spatial anchors may be used to represent points of interest. Such spatial anchors may empower developers to build spatially aware augmented reality and mixed reality applications. These applications may support a variety of mobile device hardware and/or operating systems, including as an example: Microsoft HoloLens™, iOS-based devices supporting ARKit, and Android-based devices supporting ARCore. Spatial anchors enable developers to work with augmented reality and mixed reality platforms to perceive spaces, designate precise points of interest, and to recall those points of interest from supported devices”; [0024], “In some examples, building a way-finding experience involves preparing a space for the experience and developing an app that end users will interact with. Such a building process may include planning the space by determining locations within the space that participate in the way-finding experience. This may be performed, for example, by a museum tour coordinator, factory supervisor, or the like, depending upon the environment in which the way-finding experience will be implemented”; and [0026], “As an example end-user experience, a first step for end users may be to locate one of the anchors using an end-user application running on a mobile device, which can be in any one of the chosen locations. In some examples, determining the locations where end users can enter the experience may be a part of designing the experience. Once the end user has located one anchor, the app can request nearby anchors. This procedure returns a pose between the device and these anchors, wherein the term pose may indicate a position and orientation of the anchors relative to the device. The end-user application may then take advantage of the pose to each of these anchors to render useful guidance hints about their general direction and distance. For instance, the end-user application may display an icon and arrow on a camera feed representing each potential destination”. Note that the anchors are designed and placed in such a way that the users to enter this experience are limited to the designed ways or regions, and this limited region or way is mapped to the environment mask).
Claims 4-10 are rejected under 35 U.S.C. 103 as being unpatentable over Emami, etc. (US 20210350612 A1) in view of Cardenas, etc. (US 20220075591 A1), further in view of Maciocci, etc. (US 20120249741 A1).
Regarding claim 4, Emami and Cardenas teach all the features with respect to claim 3 as outlined above. However, Emami, modified by Cardenas, fails to explicitly disclose that the method of claim 3, characterized in that, after the XR scene has been generated, additional data are obtained from at least a lidar of the mobile user device; wherein the playback of the virtual part of the XR scene is adjusted, if necessary, based on the data obtained from the accelerometer and the gyroscope of the mobile user device and the additional data obtained from the lidar of the mobile user device.
However, Maciocci teaches that the method of claim 3, characterized in that, after the XR scene has been generated, additional data are obtained from at least a lidar of the mobile user device (See Maciocci: Fig. 5, and [0067], “Further, the head mounted device 10 may include orientation sensors, such as accelerometers, gyroscopes, magnetic sensors, optical sensors, mechanical or electronic level sensors, and inertial sensors which alone or in combination can provide data to the device's processor regarding the up/down/level orientation of the device (e.g., by sensing the gravity force orientation) and thus the user's head position/orientation (and from that viewing perspective). Further, the head mounted device may include rotational orientation sensors, such as an electronic compass and accelerometers, that can provide data to the device's processor regarding left/right orientation and movement. Collectively, sensors (including accelerometers, gyroscopes, magnetic sensors, optical sensors, mechanical or electronic level sensors, inertial sensors, and electronic compasses) configured to provide data regarding the up/down and rotational orientation of the head mounted device (and thus the user's viewing perspective) are referred to herein as "orientation sensors."”; and [0120], “In another embodiment, the scene sensor 500 may use other distance measuring technologies (i.e., different types of distance sensors) to capture the distance of the objects within the image, for example, ultrasound echo-location, radar, triangulation of stereoscopic images, etc. As discussed above, in an embodiment, the scene sensor 500 may include a ranging camera, a flash LIDAR camera, a time-of-flight (ToF) camera, and/or a RGB-D camera 503, which may determine distances to objects using at least one of range-gated ToF sensing, RF-modulated ToF sensing, pulsed-light ToF sensing, and projected-light stereo sensing”);
wherein the playback of the virtual part of the XR scene is adjusted, if necessary, based on the data obtained from the accelerometer and the gyroscope of the mobile user device and the additional data obtained from the lidar of the mobile user device (See Maciocci: Figs. 9A-B, and [0166], “If the processor determines that the second user desires to change the anchor surface for the virtual object (e.g., via an input) (i.e., determination block 908="Yes"), the processor may determine distance and orientation parameters of the newly designated anchor surface in block 909. In block 910, the processor may change the rendering of the virtual object based on the parameters of the anchor surface (desktop), and process the virtual object based on distance and orientation of the anchor surface of the second predetermined surface in block 911. In block 912, the second processor may render the image with the anchored virtual object on the anchor surface in the display of the second head mounted device”. Note that after the initial scene generation, additional data from sensors like LiDAR, IMU for depth information refines the poses, anchors, and the virtual playback rendering stability, and this is mapped to the current limitation).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Emami to have the method of claim 3, characterized in that, after the XR scene has been generated, additional data are obtained from at least a lidar of the mobile user device; wherein the playback of the virtual part of the XR scene is adjusted, if necessary, based on the data obtained from the accelerometer and the gyroscope of the mobile user device and the additional data obtained from the lidar of the mobile user device as taught by Maciocci in order to place the virtual object or interface on a selected physical surface using a head mounted display or other mobile devices (See Maciocci: Figs. 1-2, and [0003], “The present application relates to an augmented or virtual reality system using a head mounted display, or other mobile devices such as smartphones or tablets, that can place a virtual object or interface on a selected physical surface so that a single user or multiple users can collaborate to, view and interact with the virtual object on the physical surface”). Emami teaches a method and system that may generate and share XR scene with multiple users using anchors to connecting the spatial anchors and AR contents; while Maciocci teaches a system and method that may communicate and transmit sensor, such as LiDAR/IMU, data and updated data between users to enable collaborate with multiple users. Therefore, it is obvious to one of ordinary skill in the art to modify Emami by Maciocci to have additional data from various sensors, such as LiDAR, accelerometer, IMUs, etc. to refine the pose/anchor information for virtual content generations. The motivation to modify Emami by Maciocci is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 5, Emami, Cardenas, and Maciocci teach all the features with respect to claim 4 as outlined above. Further, Maciocci teaches that the method of claim 4, characterized in that in case the mobile user device, which is a first mobile user device, does not have a lidar, the lidar data are obtained using a second mobile user device that is equipped with a lidar (See Maciocci: Figs. 9A-B, and [0167], “In block 913, updates of the virtual object received from the user wearing the second head mounted display may be transmitted to the first head mounted display for rendering. In this manner, the users of the two head mounted displays may collaboratively interact with the virtual object during a collaboration session. These updates may be transmitted between the two head mounted devices so each device displays the updated virtual image reflecting all changes made by either user. For example, in a word processing application, the virtual object may reflect changes from both users in a collaborative manner. In block 914, an audio link may be established between the second and the first head mounted devices. The users may utilize the audio link to speak to one another, as well as other users in an audio conference format. This conference may occur at the same time as viewing the virtual object on the display in two different geographic locations. In an embodiment, the head mounted device may use video and audio conferencing software”).
Regarding claim 9, Emami, Cardenas, and Maciocci teach all the features with respect to claim 4 as outlined above. Further, Emami teaches that the method of claim 4, characterized in that the spatial anchors are used to generate an environment mask (See Emami: Fig. 1, and [0014], “FIG. 1 depicts an example environment 100 within which user-designated virtual spatial anchors may be used to represent points of interest. Such spatial anchors may empower developers to build spatially aware augmented reality and mixed reality applications. These applications may support a variety of mobile device hardware and/or operating systems, including as an example: Microsoft HoloLens™, iOS-based devices supporting ARKit, and Android-based devices supporting ARCore. Spatial anchors enable developers to work with augmented reality and mixed reality platforms to perceive spaces, designate precise points of interest, and to recall those points of interest from supported devices”; [0024], “In some examples, building a way-finding experience involves preparing a space for the experience and developing an app that end users will interact with. Such a building process may include planning the space by determining locations within the space that participate in the way-finding experience. This may be performed, for example, by a museum tour coordinator, factory supervisor, or the like, depending upon the environment in which the way-finding experience will be implemented”; and [0026], “As an example end-user experience, a first step for end users may be to locate one of the anchors using an end-user application running on a mobile device, which can be in any one of the chosen locations. In some examples, determining the locations where end users can enter the experience may be a part of designing the experience. Once the end user has located one anchor, the app can request nearby anchors. This procedure returns a pose between the device and these anchors, wherein the term pose may indicate a position and orientation of the anchors relative to the device. The end-user application may then take advantage of the pose to each of these anchors to render useful guidance hints about their general direction and distance. For instance, the end-user application may display an icon and arrow on a camera feed representing each potential destination”. Note that the anchors are designed and placed in such a way that the users to enter this experience are limited to the designed ways or regions, and this limited region or way is mapped to the environment mask).
Regarding claim 10, Emami, Cardenas, and Maciocci teach all the features with respect to claim 5 as outlined above. Further, Emami teaches that the method of claim 5, characterized in that the spatial anchors are used to generate an environment mask (See Emami: Fig. 1, and [0014], “FIG. 1 depicts an example environment 100 within which user-designated virtual spatial anchors may be used to represent points of interest. Such spatial anchors may empower developers to build spatially aware augmented reality and mixed reality applications. These applications may support a variety of mobile device hardware and/or operating systems, including as an example: Microsoft HoloLens™, iOS-based devices supporting ARKit, and Android-based devices supporting ARCore. Spatial anchors enable developers to work with augmented reality and mixed reality platforms to perceive spaces, designate precise points of interest, and to recall those points of interest from supported devices”; [0024], “In some examples, building a way-finding experience involves preparing a space for the experience and developing an app that end users will interact with. Such a building process may include planning the space by determining locations within the space that participate in the way-finding experience. This may be performed, for example, by a museum tour coordinator, factory supervisor, or the like, depending upon the environment in which the way-finding experience will be implemented”; and [0026], “As an example end-user experience, a first step for end users may be to locate one of the anchors using an end-user application running on a mobile device, which can be in any one of the chosen locations. In some examples, determining the locations where end users can enter the experience may be a part of designing the experience. Once the end user has located one anchor, the app can request nearby anchors. This procedure returns a pose between the device and these anchors, wherein the term pose may indicate a position and orientation of the anchors relative to the device. The end-user application may then take advantage of the pose to each of these anchors to render useful guidance hints about their general direction and distance. For instance, the end-user application may display an icon and arrow on a camera feed representing each potential destination”. Note that the anchors are designed and placed in such a way that the users to enter this experience are limited to the designed ways or regions, and this limited region or way is mapped to the environment mask).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights