Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5-6, 9-11, 12-17, 18 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 5 claims a second prompt information, but lacks antecedent basis as a first prompt is not previously define, and is not present in claims 1 and 3. Claim 6 is rejected for depending on claim 5 and not correcting the deficiencies of claim 5.
Claim 9 claims a second captured room, but a first captured room is not previously defined, and not present in claims 1 and 8. Claims 10-11 are similarly rejected, as mentioning third and fourth captured rooms, while lacking antecedent basis.
Claim 12 claims a second instruction, but lacks antecedent basis as a first instruction is not previously defined in claim 12 or claim 1. Claims 13-17 are rejected for depending on claim 12 and not rectifying the deficiencies of claim 12.
Claim 18 claims a second management interface, but a first management interface is not previously defined. Furthermore, claim 18 claims a fourth instruction, fifth captured rooms but lacks antecedent basis for a fourth instruction.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter.
Claim 20 describe a non-volatile computer-readable storage medium.
Further, Applicant's specification, at paragraph [0223] & [0229], fails to explicitly define the scope of non-volatile computer-readable storage medium. Thus, in giving the term its plain meaning (see MPEP 2111.01), the claimed non-volatile computer-readable storage medium is considered to include data signals per se. Data signals per se are not statutory as they fail to fall into one of the four statutory categories of invention.
As an additional note, a non-transitory computer readable medium having executable programming instructions stored thereon is considered statutory as non-transitory computer readable media excludes data signals (data signals are transitory computer readable media.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 7, 19-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Noris et al. (US 2023/0127307).
Re claim 1, Noris teaches a multi-room capture method, comprising:
displaying a room capture interface in response to a room capture trigger instruction, wherein the room capture interface comprises information of uncaptured rooms of a target space ([0082] In particular embodiments, the manual scene capture workflow 400 may be initiated in response to a user wearing an artificial-reality system (e.g., artificial-reality system 200) walking/entering into the room and an application (e.g., first-party application or third-party application) on the artificial-reality system determining that a scene description or room definition for the room is not present. The scene capture workflow 400 may begin, at step 402, with the artificial-reality system 200 presenting a welcome screen to the user 202 wearing the artificial-reality system 200 to initiate a screen capture process. FIG. 5A illustrates an example graphical user interface 500a that may be displayed to the user 202 to initiate the screen capture process. The graphical user interface 500a may include an image 501 and a scene-capture-assist window 502. The image 501 may be displayed as a passthrough image to the user 202. The scene-capture-assist window 502 may be displayed as an AR element on top of the image 501 that the user 202 may be currently seeing. As depicted, the screen-capture-assist window 502 may indicate to the user 202 to setup their room in VR and present two options, including a continue option 506 and a cancel option 508. The continue option 506 may initiate the scene capture process, while the cancel option 508 may cancel the process and exit the scene capture workflow 400. The user 202 may select a desired option via a controller (e.g., controller 206) by hovering over and clicking on the desired option. Once the user confirms the continue option 506 (e.g., as shown by reference numeral 510), the artificial-reality system 200 may initiate the scene capture process), ([0067] In some embodiments, a scene model discussed herein is like a scene graph. The scene graph may be a structed spatial logical hierarchy with scene-related information organized at various levels in the hierarchy. FIG. 3 illustrates an example scene graph 300. In particular embodiments, users (e.g., developers) may be able to query or interact with a scene graph (e.g., scene graph 300) via a set of semantic queries. As shown in FIG. 3, the scene graph 300 represents scene-related information of a real world 302, which may include a plurality of objects 304a-304g (individually or collectively herein referred to as 304) and a plurality of groups 306a-306c (individually or collectively herein referred to as 306). Two or more objects 304 may be grouped together to form a group 306 to represent a larger space or part of the world 302. By way of an example and without limitation, the world 302 may be a house of a user, where different groups 306 may represent different rooms of the house and each group 306 (e.g., room) may include one or more of sub-group(s) or object(s) (e.g., bed, tv, lamp, wall art, couch, etc.) to represent things that are part of that room), and ([0148] Assisted Localization Option 2: Named Rooms and Annotated Fixtures—when a user runs a scene capture process or workflow (e.g., scene capture workflow 400) for the first time, the system (e.g., artificial-reality system 200) may ask the user to provide a name for their room and annotate surfaces and objects (volumes) that are permanent fixtures. In the future when a localization failure occurs (e.g., unassisted localization fails), the user may be asked to specify which room they are in (e.g., from a list of available rooms) and point at the permanent fixtures that they had previously annotated. By specifying the room name, a localization algorithm running on the system may pick the correct map. With knowledge of permanent fixtures that were previously annotated, the localization algorithm may have a strong pose prior and a focal point for search. The named rooms and annotated fixtures option may particularly help when relocalization fails due to large scene changes. It may not help when relocalization fails due to lighting changes. However, lighting changes are presumable easier to detect, and may be provided as explicit guidance to user as part of the scene capture workflow. For example, the user may be asked to turn on the light to help with relocalization)
and an environmental map of the target space is associable with anchors of a plurality of captured rooms within the target space ([0004] Embodiments described herein relate to generating, querying, and managing a scene model. The scene model is an objective (e.g., single source of truth), system-managed, comprehensive, and an up-to-date representation of a user's physical or real world that may be easily indexable and queryable. The scene model may describe static geometry and semantics of the real world. In particular embodiments, the scene model may be composed of a plurality of anchors, where each anchor represents a plane, surface, or an object in a user's physical environment (e.g., user's living room). In some embodiments, the scene model discussed herein may be in the form of a scene graph or hierarchical tree structure comprising of the set of anchors, where each anchor corresponds to an entity in the user's physical environment. These anchors may include, for example, (1) a bounded2D and semanticlabels component to represent a plane (e.g., floor, wall, ceiling, etc.) (2) a bounded3D and semanticlabels component to represent an object (e.g., desk, chair, couch), and (3) a roomlayout and container component to represent an overall scene (e.g., room).
in response to a selection operation by a user on an uncaptured target room in the room capture interface, capturing the target room according to a multi-room capture policy; ([0004] Embodiments described herein relate to generating, querying, and managing a scene model. The scene model is an objective (e.g., single source of truth), system-managed, comprehensive, and an up-to-date representation of a user's physical or real world that may be easily indexable and queryable. The scene model may describe static geometry and semantics of the real world. In particular embodiments, the scene model may be composed of a plurality of anchors, where each anchor represents a plane, surface, or an object in a user's physical environment (e.g., user's living room). In some embodiments, the scene model discussed herein may be in the form of a scene graph or hierarchical tree structure comprising of the set of anchors, where each anchor corresponds to an entity in the user's physical environment. These anchors may include, for example, (1) a bounded2D and semanticlabels component to represent a plane (e.g., floor, wall, ceiling, etc.) (2) a bounded3D and semanticlabels component to represent an object (e.g., desk, chair, couch), and (3) a roomlayout and container component to represent an overall scene (e.g., room)), (0082] In particular embodiments, the manual scene capture workflow 400 may be initiated in response to a user wearing an artificial-reality system (e.g., artificial-reality system 200) walking/entering into the room and an application (e.g., first-party application or third-party application) on the artificial-reality system determining that a scene description or room definition for the room is not present. The scene capture workflow 400 may begin, at step 402, with the artificial-reality system 200 presenting a welcome screen to the user 202 wearing the artificial-reality system 200 to initiate a screen capture process. FIG. 5A illustrates an example graphical user interface 500a that may be displayed to the user 202 to initiate the screen capture process. The graphical user interface 500a may include an image 501 and a scene-capture-assist window 502. The image 501 may be displayed as a passthrough image to the user 202. The scene-capture-assist window 502 may be displayed as an AR element on top of the image 501 that the user 202 may be currently seeing. As depicted, the screen-capture-assist window 502 may indicate to the user 202 to setup their room in VR and present two options, including a continue option 506 and a cancel option 508. The continue option 506 may initiate the scene capture process, while the cancel option 508 may cancel the process and exit the scene capture workflow 400. The user 202 may select a desired option via a controller (e.g., controller 206) by hovering over and clicking on the desired option. Once the user confirms the continue option 506 (e.g., as shown by reference numeral 510), the artificial-reality system 200 may initiate the scene capture process.
and updating environmental data of the environmental map according to a capture result of the target room, and associating an anchor of the target room with the environmental map ([0041] In particular embodiments, the system (e.g., artificial-reality system 200) may create an anchor for each of the user-defined planes, surfaces, or objects. By way of an example and without limitation, for the six entities 102a-102f shown in FIG. 1, the system may create an anchor for floor, an anchor for wall, an anchor for ceiling, an anchor for desk, an anchor for chair, and an anchor for couch. In some embodiments, one or more plane anchors may be associated with one or more 2D planes or surfaces (e.g., wall, ceiling, floor, door, window, etc.) and one or more object anchors may be associated with one or more 3D objects (e.g., couch, chair, lamp, desk, etc.). Each anchor associated with a plane and/or an object may include its component type defining geometric representation (e.g., 2D boundary or 3D bounding box) as well as a semantic label or category indicating what that plane/object represents (e.g., floor, ceiling, walls, desk, couch, etc.). In some embodiments, users (e.g., developers) may modify a scene model including the anchors as per their needs. For instance, the developers may create or keep only plane and object anchors without the existence of entire scene model if they just want to detect and track a plane/object in front of the user at runtime. In some embodiments, an anchor may be able to hold or associate multiple elements belonging to the same semantic category. For example, 2 couches in the room may be associated with a single anchor. As another example, 5 walls may be associated with a first anchor and 2 desktops may be associated with a second anchor), ([0074] Scene capture is a process of capturing scene-related information (e.g., information associated with a scene, such as scene 100). In particular embodiments, the scene may be a user's real or physical environment in which a user is located. For example, the user may be in his living room, and therefore the scene may be the user's living room. The scene may be viewed through a display of an artificial-reality system, such as the artificial-reality system 200. For instance, the scene may be presented as a passthrough image to the user 202 wearing the artificial-reality system 200. While viewing the scene through the artificial-reality system, a set of instructions may be provided to the user. These instructions may guide the user to capture various entities present in the scene. For instance, the user may be guided to capture planes or surfaces (e.g., walls, ceiling, windows, door, etc.) and various objects (e.g., desk, couch, table, art, cabinet, plant, lamp, tv, etc.) present in the user's environment (e.g., living room). Based on the captured planes, surfaces, and objects, the system may generate a scene model, as discussed elsewhere herein).
Re claim 7, Noris teaches claim 1. Furthermore, Norris teaches wherein associating the anchor of the target room with the environmental map comprises:
generating anchor information of the target room, and storing the anchor information of the target room by using an identifier of the target room as an index; and establishing an association relationship between the identifier of the target room and an identifier of the environmental map ([0159] In particular embodiments, the manual scene realignment technique addresses the localization failure discussed herein, particularly with respect to the above-discussed example scenarios. The key idea of this mitigation technique is to rely on users to indicate that they are in a space they have already manually tagged and provide enough information in order to align a cache of the room they earlier tagged into a current map. In particular embodiments, a manual scene realignment process relies on an existence of a well-defined room origin (e.g., room corner 1212), and knowing where that corner is located in a new map, such as map 1204. The manual scene realignment is based on an assumption that any room has a well-defined origin, which may be a corner defined by intersection of two walls, such as a corner 1210 defined by the intersection of walls 1206 and 1208, as shown in FIG. 12. The first two walls (e.g., wall 1 and wall 2) may be created or defined by a user as part of the scene capture process. FIG. 12 illustrates an example alignment of a cached or previously-created scene (e.g., map 1202 of a room) to a current map 1204. As discussed above in example scenario 1, there may be multiple maps created for the same place. For example, on day 1, the user may have done the scene capture of their living room, which may be stored as Map1. On day 2 when they return to the living room, the system (e.g., artificial-reality system 200) may not be able to relocalize or load Map1 and instead create a new map, such as Map2. When the user now tries to launch an application that requires a room definition (e.g., scene model of the living room), the system may initiate the manual scene realignment process. In this process, the user may be asked to select or define two walls, as indicated by reference numerals 1206 and 1208. The user may do so by touching the walls or casting a ray to place a point on each of these walls via their controller 206. The system may then determine a point of intersection of these walls 1206 and 1208. The point of intersection may help determine a particular point, such as a room corner 1210. Based on the room corner 1210, the system may align the cached room 1202 (e.g., stored as Map1) with the new or current map 1204 (e.g., stored as Map2). In other words, the system may move or reposition the cached room 1202 until its room corner 1212 aligns with the room corner 1210 of the new map 1204, as indicated by reference numeral 1214. Based on the alignment, the system may determine that the cached room (e.g., Map1) is actually the current map of the user's living room and may load the cached room or a scene model associated with the cached room (e.g., previous map of the room) into the current map. As such, the manual scene realignment technique may be able to relocalize with minimal user inputs (e.g., user selecting or defining walls for a room corner) without having to redraw an entire room or going through the entire scene capture process), ([0148] Assisted Localization Option 2: Named Rooms and Annotated Fixtures—when a user runs a scene capture process or workflow (e.g., scene capture workflow 400) for the first time, the system (e.g., artificial-reality system 200) may ask the user to provide a name for their room and annotate surfaces and objects (volumes) that are permanent fixtures. In the future when a localization failure occurs (e.g., unassisted localization fails), the user may be asked to specify which room they are in (e.g., from a list of available rooms) and point at the permanent fixtures that they had previously annotated. By specifying the room name, a localization algorithm running on the system may pick the correct map. With knowledge of permanent fixtures that were previously annotated, the localization algorithm may have a strong pose prior and a focal point for search. The named rooms and annotated fixtures option may particularly help when relocalization fails due to large scene changes. It may not help when relocalization fails due to lighting changes. However, lighting changes are presumable easier to detect, and may be provided as explicit guidance to user as part of the scene capture workflow. For example, the user may be asked to turn on the light to help with relocalization), ([0160] In some embodiments, a cache definition associated with a cached room may be used during the alignment process, such as the alignment process discussed above in reference to FIG. 12. A room cache or a cached room (e.g., previously-created room) may include a unique identifier (e.g., UUID) of the room entity and a list of pairs, each pair including an anchor ID and a pose of the anchor (e.g., <anchor uuid, pose>). The pose is relative to the origin of the room, where the origin may be the corner of the room (or may be at any place, where a spatial anchor may be defined corresponding to the room entity). When information for a room is needed, the system (e.g., artificial-reality system 200) may use the stored UUIDs (e.g., of the room and the anchors) to query relevant information. The information that cannot be queried (e.g., because the map is not loaded) may be provided by the cache definition. In some embodiments, the cache definition may include, for example, cache, what can be loaded, and corner alignment, each of which can be combined to provide entire room information needed to relocalize).
Claims 19 and 20 claim limitations in scope to claim 1 and is rejected for at least the reasons above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Noris et al. (US 2023/0127307) in view of Zhao et al. (US 20210056751).
Re claim 2, Noris teaches claim 1. Noris does not explicitly teach wherein the multi-room capture policy comprises the following: closed spaces of any two captured rooms of the environmental map do not overlap with each other.
However, Zhao teaches wherein the multi-room capture policy comprises the following: closed spaces of any two captured rooms of the environmental map do not overlap with each other ([0036] Further, the 3D model assembling unit performs a correction on the 3D models of the multiple rooms, including correcting wall line directions of all rooms by using a statistical method, so that wall lines of all rooms are aligned in the same direction if they were parallel within an error range; and when assembling the 3D models of the rooms, the 3D model assembling unit corrects one or more overlapping parts and/or gaps) and ([0048] Further, step S4 includes: (S41) converting local coordinates of a 3D model of a single photo capture point into global coordinates, for example, by using a transformation matrix based on the position and the capture direction of each photo capture point, so as to obtain an overall 3D model of all photo capture points; (S42) performs a correction on the 3D models of multiple photo capture points, including correcting wall line directions of all photo capture points by using a statistical method, so that wall lines of all rooms are aligned in the same direction if they were parallel within a specific error range; and (S43) when assembling the 3D models of the photo capture points, correcting one or more overlapping parts and/or gaps).
Noris and Zhao teaches claim 2. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Noris’s 3d multi-room capture system with capture policies for environment mapping to explicitly include capture policies wherein closed spaces of any two captured rooms of the environment map do not overlap with each other, as taught by Zhao, as the references are in the analogous art of capture policies for environment mapping. An advantage of the modification is that it achieves the result of adding policies for non-overlapping regions to improve correction and alignment of captured data used for environment mapping.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Noris et al. (US 2023/0127307) in view of Zhao et al. (US 20210056751) and Hill (US 20230230326).
Re claim 3, Noris and Zhao teaches claim 2. Noris further teaches wherein the multi-room capture policy further comprises: a capture box of furniture within a room ([0100] At step 470, the system 200 may create and save a first the object (e.g., desk 564). In particular embodiments, the system 200 may save the object as a 3D bounding box or volume. For instance, based on the four points (e.g., point 566, point 568, point 572, point 578) defined by the user 202, the first vertical plane 572, and the two top horizontal planes 576 and 580, the system may create (1) three more vertical edges (not shown) connecting the points, (2) two remaining top horizontal edges (not shown), and (4) four bottom horizontal edges (not shown). The result will be a 3D bounding or volume defining the object, such as the desk 564. In particular embodiments, the system 200 may save this 3D bounding box or 3D volume of the desk as an object anchor, which may be later used for generating a scene model).
Noris and Zhao do not explicitly teach the capture box does not exceed the closed space of the room.
However, Hill teaches capture box of furniture within a room does not exceed the closed space of the room ([0073] The user interface 220 can be used by a user to position a new digital object selected from the database 22 into the digital model 510. For example, multiple new digital objects can be presented in the user interface 220 as available to be added and positioned in the digital model 510. Digital objects may be created as visual object representations of real objects such as, e.g., furniture, appliances, electronics, and items of home décor. These digital objects may be created by the machine learning system 200 using point cloud 41 data, meshed models, or combinations thereof. Each digital object may be dimensionally accurate to scale relative to the actual object it visually represents. The digital objects can be stored on a computing device, e.g., on the remote computing device 20, the user computing device 12, or the database 22, in the form of object files for later retrieval for viewing and editing. The digital objects 514 may be moved around by a user within the digital model 510 for placement in different locations or points within the digital model 510. The collision meshing used to create the digital model 510 prevents the digital objects 514 from passing through or overlapping space within the digital model 510 that is occupied by static structural elements, architectural elements, items of décor and other objects that may be present in the actual room and that are digitally represented by other digital objects in the digital model 510. In some implementations, the digital model 510 and each of the digital objects 514 includes a respective bounding box which can be used in combination or in place of the collisional meshing to define the limits of the digital room 510 and the digital objects 514. In some implementations, the collisional meshing and/or bounding boxes can facilitate latching mechanisms of the digital objects 514 to the structural elements 512 of the digital model 510. For example, if a digital object 514 is being moved (e.g., based on a user input) in a horizontal direction along a floor plane of the digital model 510, the collisional meshing can detect a collision with a wall plane and halt movement of the digital object 514 in that direction. The collisional meshing can then latch the digital object 514 to the wall and may continue movement in a vertical direction up the wall if such movement is considered allowable by the collisional meshing (e.g., based on the type of digital object). The user interface 220 may also allow removal, deletion, or replacement of digital objects corresponding to furniture and other objects in the room that are present when the 3D scan is initially captured and added to the initial digital model 510 created by the machine learning system 200).
Noris, Zhao, and Hill teaches claim 3. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Noris and Zhao’s 3d multi-room capture system with capture policies for objects including furniture, to explicitly include a capture box of furniture within a room does not exceed the closed space of the room, as taught by Hill, as the references are in the analogous art of capture policies for environment mapping. An advantage of the modification is that it achieves the result of adding policies for capture box of furniture within a room so that they don’t exceed certain spaces and cause conflicts in rendering, such as colliding with walls, objects, etc.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Noris et al. (US 2023/0127307) in view of Zhao et al. (US 20210056751) and Chen (US 20200311429).
Re claim 4, Noris and Zhao teaches claim 2. Furthermore, Noris teaches wherein capturing the target room according to the multi-room capture policy comprises:
acquiring an image corresponding to the target room, and capturing, according to the image corresponding to the target room, the target room in an order of a floor, a wall space, a ceiling and furniture, wherein the closed space of the target room is formed by connecting capture lines of the floor, the wall space and the ceiling of the target room ([0028] The scene capture workflow may be either a manual scene capture workflow or an assisted scene capture workflow. In the manual scene capture workflow, a user is provided with guided step-by-step instructions through a manual tagging flow to capture the different entities in the user's physical environment (e.g., user's room). One such example manual scene capture workflow is shown and discussed with respect to at least FIGS. 4A4B. The user may capture these entities using raycast from a controller (e.g., controller 206) of an artificial-reality system (e.g., artificial-reality system 200). For instance, the user may be instructed to put a point at a particular location by casting/shooting a ray using their controller towards that location in order to capture an entity. The user may be able to easily, quickly, safely, and accurately capture planes with 2D surfaces (e.g., walls, floor) and objects with 3D volumes (e.g., desk, couch, table, chair, etc.). These captured planes and objects may be annotated with semantic labels. The user may be able to edit the captured elements if needed), ([0029] In the assistant scene capture workflow, instead of the user defining each and every entity in the room, some of the entities may be automatically detected or recognized by the artificial-reality system. For instance, planes may be detected using a plane detection or understanding technology and objects in the room may be detected using an object recognition technology. Specifically, the user may be instructed to select walls in their environment. Their walls may be automatically detected when the user is within a certain threshold (e.g., approx. 2 meters) of the wall. The user may then be able to point at the wall and add it to their layout with a raycast from a controller (e.g., controller 206) of the artificial-reality system. After each of the user's walls has been added, their room layout may be calculated and revealed) and (see [0080-0081], capturing floors, ceilings, walls, objects).
capturing the closed space of the target room; and in response to detecting that a target capture line of the closed space of the target room overlaps with the closed space of a captured room of the environmental map, outputting first prompt information ([0110] Once the capture walls process 602 is complete, the artificial-reality system 200 may begin capturing ceiling process 603, as shown in FIG. 6B. At step 644, the system 200 may display a pattern and a slider to adjust ceiling height. For instance, a pattern (e.g., similar to the one shown for the floor) may be revealed on the ceiling above the user 202. The user's menu may show a slider to adjust the height of their ceiling. If a ceiling was already detected, the slider and the pattern may be at the detected height. Otherwise, it will be set at a default height (e.g., of about 2.5 meters). At step 646, the system 200 may send instructions to the user 202 to look up and adjust ceiling height if necessary. At step 648, the user may look up and adjust the ceiling height via the slider. The ceiling pattern may move in real time if the user decides to manually adjust. At step 650, the system 200 may detect the ceiling if not already detected or update the detected ceiling at the ceiling height adjusted by the user 202. At step 652, the system 200 may send instructions to the user 202 to confirm the detected or updated ceiling. For example, the user's menu will reveal a “Confirm Ceiling” button. At step 654, the user 202 confirms the ceiling, for example, by pressing or clicking on the “Confirm Ceiling” button via their controller 206. Once the user confirms the ceiling, at step 656, the system 200 may add the ceiling to the user's room layout. In response to the completion of the capture floor process 601, capture walls process 602, and the capture ceiling process 603, at step 658, the artificial-reality system 200 may create or update a scene model by adding the captured entities (e.g., floor, walls, ceiling) as anchors in the scene model along with additional elements or components (e.g., semantic types, component types, room layout, room entity component, room container, etc.), as discussed elsewhere herein), and ([0083] At step 404, the artificial-reality system 200 may receive acknowledgement from the user 202 to start the scene capture process. For instance, the user may acknowledge by hovering over or navigating to the continue option 506 and clicking on it (e.g., as shown by reference numeral 510) via the controller 206. Upon receiving the acknowledgement, at step 406, the artificial-reality system 200 may present a set of instructions to the user 202 to start capturing a first plane, such as a wall, in the user's surrounding environment (e.g., room). FIG. 5B illustrates an example graphical user interface 500b with an updated screen-capture-assist window 512 including a set of instructions 514a-514c (individually or collectively herein referred to as 514) to outline walls of the room. For example, a first instruction 514a may instruct the user 202 to define a base of a wall by putting a point (e.g., by casting a first ray via the controller 206) on a bottom wall corner. A second instruction 51b may instruct the user 202 to define a height of the wall by putting a point (e.g., by casting a second ray via the controller 206) on the top corner of the same wall, the base of which that the user earlier defined based on the first instruction 514a. Once the base and the height of a wall is known, a third instruction 514c may instruct the user 202 to put a point (e.g., by casting a subsequent ray via the controller 206) on top corners of each wall in the room. For example, if there are 4 walls in the room, then the user may be asked to put 4 points, where each point connects with the previous point and the next point to form respective walls).
Noris and Zhao teaches target correct target capture lines, but do not specifically teach wherein the first prompt information is used for at least one of: prompting a location error of the target capture line, or prompting to adjust a location of the target capture line to a corrected location.
However, Chen teaches wherein the first prompt information is used for at least one of: prompting a location error of the target capture line, or prompting to adjust a location of the target capture line to a corrected location ([[0054] On a high-level, an exemplary user-guidance system includes the following parts, as shown in FIG. 1. In the 3D scanning process, the user-guidance system takes the output data of any or all of the sensors 11 (e.g., a camera, depth camera, gyro sensor, GPS sensor, motion sensor, etc.), analyzes the data and determines if the user(s) (either a scannee-person 14 or the scanner-person 12, or both) is performing the intended tasks (e.g., scanee-person having the right pose, standing at the right location in relation to the smartphone, the background being correct, etc.) via computation algorithms 13 stored on a computer-readable medium in communication with a computer processor that processes the algorithms (for example, on a smartphone). The processor then issues instructions to generate appropriate real-time feedback 15 communicated to the user via on-screen guidance, voice guidance, haptic guidance, etc., from the smart device. The user(s) then follows these prompts to adjust his/her pose; move himself/herself to the right location in relation to the smart device; move, tilt or rotate the smart device to the right location with the right camera angle; clean up the background environment; or move to a better background. The system functions as a feedback system that can generate and communicate, in real-time, clear instructions to the user 14 and/or user 12, so the user(s) can easily follow these instructions to comply with the requirements of the 3D scanning process).
Noris in view of Zhao and Chen teaches claim 4.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Noris and Zhao’s 3d multi-room capture system with capture lines for capturing closed spaces of a target room to explicitly include prompts to adjust the location of the target capture line to a correct location, as taught by Chen, as the references are in the analogous art of 3d rendering of captured data. An advantage of the modification is that it achieves the result of explicitly using a prompt for a user to correct alignments of capture data, so that a user can easily following instructions to comply with scanning requirements for 3d data capture.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Noris et al. (US 2023/0127307) in view of Zhao et al. (US 20210056751), Hill (US 20230230326), and Chen (US 20200311429).
Re claim 5, Noris, Zhao, and Hill teach claim 3. Furthermore, Noris teaches wherein capturing the target room according to the multi-room capture policy comprises:
acquiring an image corresponding to the target room, and capturing, according to the image corresponding to the target room, the target room in an order of a floor, a wall space, a ceiling and furniture, wherein the closed space of the target room is formed by connecting capture lines of the floor, the wall space and the ceiling of the target room ([0028] The scene capture workflow may be either a manual scene capture workflow or an assisted scene capture workflow. In the manual scene capture workflow, a user is provided with guided step-by-step instructions through a manual tagging flow to capture the different entities in the user's physical environment (e.g., user's room). One such example manual scene capture workflow is shown and discussed with respect to at least FIGS. 4A4B. The user may capture these entities using raycast from a controller (e.g., controller 206) of an artificial-reality system (e.g., artificial-reality system 200). For instance, the user may be instructed to put a point at a particular location by casting/shooting a ray using their controller towards that location in order to capture an entity. The user may be able to easily, quickly, safely, and accurately capture planes with 2D surfaces (e.g., walls, floor) and objects with 3D volumes (e.g., desk, couch, table, chair, etc.). These captured planes and objects may be annotated with semantic labels. The user may be able to edit the captured elements if needed), ([0029] In the assistant scene capture workflow, instead of the user defining each and every entity in the room, some of the entities may be automatically detected or recognized by the artificial-reality system. For instance, planes may be detected using a plane detection or understanding technology and objects in the room may be detected using an object recognition technology. Specifically, the user may be instructed to select walls in their environment. Their walls may be automatically detected when the user is within a certain threshold (e.g., approx. 2 meters) of the wall. The user may then be able to point at the wall and add it to their layout with a raycast from a controller (e.g., controller 206) of the artificial-reality system. After each of the user's walls has been added, their room layout may be calculated and revealed) and (see [0080-0081], capturing floors, ceilings, walls, objects), ([0110] Once the capture walls process 602 is complete, the artificial-reality system 200 may begin capturing ceiling process 603, as shown in FIG. 6B. At step 644, the system 200 may display a pattern and a slider to adjust ceiling height. For instance, a pattern (e.g., similar to the one shown for the floor) may be revealed on the ceiling above the user 202. The user's menu may show a slider to adjust the height of their ceiling. If a ceiling was already detected, the slider and the pattern may be at the detected height. Otherwise, it will be set at a default height (e.g., of about 2.5 meters). At step 646, the system 200 may send instructions to the user 202 to look up and adjust ceiling height if necessary. At step 648, the user may look up and adjust the ceiling height via the slider. The ceiling pattern may move in real time if the user decides to manually adjust. At step 650, the system 200 may detect the ceiling if not already detected or update the detected ceiling at the ceiling height adjusted by the user 202. At step 652, the system 200 may send instructions to the user 202 to confirm the detected or updated ceiling. For example, the user's menu will reveal a “Confirm Ceiling” button. At step 654, the user 202 confirms the ceiling, for example, by pressing or clicking on the “Confirm Ceiling” button via their controller 206. Once the user confirms the ceiling, at step 656, the system 200 may add the ceiling to the user's room layout. In response to the completion of the capture floor process 601, capture walls process 602, and the capture ceiling process 603, at step 658, the artificial-reality system 200 may create or update a scene model by adding the captured entities (e.g., floor, walls, ceiling) as anchors in the scene model along with additional elements or components (e.g., semantic types, component types, room layout, room entity component, room container, etc.), as discussed elsewhere herein), and ([0083] At step 404, the artificial-reality system 200 may receive acknowledgement from the user 202 to start the scene capture process. For instance, the user may acknowledge by hovering over or navigating to the continue option 506 and clicking on it (e.g., as shown by reference numeral 510) via the controller 206. Upon receiving the acknowledgement, at step 406, the artificial-reality system 200 may present a set of instructions to the user 202 to start capturing a first plane, such as a wall, in the user's surrounding environment (e.g., room). FIG. 5B illustrates an example graphical user interface 500b with an updated screen-capture-assist window 512 including a set of instructions 514a-514c (individually or collectively herein referred to as 514) to outline walls of the room. For example, a first instruction 514a may instruct the user 202 to define a base of a wall by putting a point (e.g., by casting a first ray via the controller 206) on a bottom wall corner. A second instruction 51b may instruct the user 202 to define a height of the wall by putting a point (e.g., by casting a second ray via the controller 206) on the top corner of the same wall, the base of which that the user earlier defined based on the first instruction 514a. Once the base and the height of a wall is known, a third instruction 514c may instruct the user 202 to put a point (e.g., by casting a subsequent ray via the controller 206) on top corners of each wall in the room. For example, if there are 4 walls in the room, then the user may be asked to put 4 points, where each point connects with the previous point and the next point to form respective walls).
Furthermore, Hill teaches in response to detecting that a capture box of target furniture within the target room exceeds the closed space of the target room, adjust the capture box of the target furniture ([0073] The user interface 220 can be used by a user to position a new digital object selected from the database 22 into the digital model 510. For example, multiple new digital objects can be presented in the user interface 220 as available to be added and positioned in the digital model 510. Digital objects may be created as visual object representations of real objects such as, e.g., furniture, appliances, electronics, and items of home décor. These digital objects may be created by the machine learning system 200 using point cloud 41 data, meshed models, or combinations thereof. Each digital object may be dimensionally accurate to scale relative to the actual object it visually represents. The digital objects can be stored on a computing device, e.g., on the remote computing device 20, the user computing device 12, or the database 22, in the form of object files for later retrieval for viewing and editing. The digital objects 514 may be moved around by a user within the digital model 510 for placement in different locations or points within the digital model 510. The collision meshing used to create the digital model 510 prevents the digital objects 514 from passing through or overlapping space within the digital model 510 that is occupied by static structural elements, architectural elements, items of décor and other objects that may be present in the actual room and that are digitally represented by other digital objects in the digital model 510. In some implementations, the digital model 510 and each of the digital objects 514 includes a respective bounding box which can be used in combination or in place of the collisional meshing to define the limits of the digital room 510 and the digital objects 514. In some implementations, the collisional meshing and/or bounding boxes can facilitate latching mechanisms of the digital objects 514 to the structural elements 512 of the digital model 510. For example, if a digital object 514 is being moved (e.g., based on a user input) in a horizontal direction along a floor plane of the digital model 510, the collisional meshing can detect a collision with a wall plane and halt movement of the digital object 514 in that direction. The collisional meshing can then latch the digital object 514 to the wall and may continue movement in a vertical direction up the wall if such movement is considered allowable by the collisional meshing (e.g., based on the type of digital object). The user interface 220 may also allow removal, deletion, or replacement of digital objects corresponding to furniture and other objects in the room that are present when the 3D scan is initially captured and added to the initial digital model 510 created by the machine learning system 200). For motivation, see claim 3.
Noris, Zhao, and Hill do not explicitly teach in response to a detection, a second prompt is outputted with information used for adjusting capture.
However, Chen teaches in response to a detection, a second prompt is outputted with information used for adjusting capture ([[0054] On a high-level, an exemplary user-guidance system includes the following parts, as shown in FIG. 1. In the 3D scanning process, the user-guidance system takes the output data of any or all of the sensors 11 (e.g., a camera, depth camera, gyro sensor, GPS sensor, motion sensor, etc.), analyzes the data and determines if the user(s) (either a scannee-person 14 or the scanner-person 12, or both) is performing the intended tasks (e.g., scanee-person having the right pose, standing at the right location in relation to the smartphone, the background being correct, etc.) via computation algorithms 13 stored on a computer-readable medium in communication with a computer processor that processes the algorithms (for example, on a smartphone). The processor then issues instructions to generate appropriate real-time feedback 15 communicated to the user via on-screen guidance, voice guidance, haptic guidance, etc., from the smart device. The user(s) then follows these prompts to adjust his/her pose; move himself/herself to the right location in relation to the smart device; move, tilt or rotate the smart device to the right location with the right camera angle; clean up the background environment; or move to a better background. The system functions as a feedback system that can generate and communicate, in real-time, clear instructions to the user 14 and/or user 12, so the user(s) can easily follow these instructions to comply with the requirements of the 3D scanning process).
Noris in view of Zhao, Hill, and Chen teaches claim 5.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Noris and Zhao and Hill’s multi-room capture system with capture lines for capturing closed spaces of a target room including bounding boxes of objects that exceed a closed space and allowing for correction to explicitly include prompts to adjust the location of capture box of target furniture, as taught by Chen, as the references are in the analogous art of 3d rendering of captured data. An advantage of the modification is that it achieves the result of explicitly using a prompt for a user to correct alignments of capture data, so that a user can easily following instructions to comply with scanning requirements for 3d data capture.
Claim(s) 8-9, 12-13, 15, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Noris et al. (US 2023/0127307) in view of Wixson et al. (US 20220269885).
Re claim 8, Noris teaches claim 1. Furthermore, Noris teaches in response to receiving a first instruction, positioning the environmental map according to the environmental data of the environmental map ([0004] Embodiments described herein relate to generating, querying, and managing a scene model. The scene model is an objective (e.g., single source of truth), system-managed, comprehensive, and an up-to-date representation of a user's physical or real world that may be easily indexable and queryable. The scene model may describe static geometry and semantics of the real world. In particular embodiments, the scene model may be composed of a plurality of anchors, where each anchor represents a plane, surface, or an object in a user's physical environment (e.g., user's living room). In some embodiments, the scene model discussed herein may be in the form of a scene graph or hierarchical tree structure comprising of the set of anchors, where each anchor corresponds to an entity in the user's physical environment. These anchors may include, for example, (1) a bounded2D and semanticlabels component to represent a plane (e.g., floor, wall, ceiling, etc.) (2) a bounded3D and semanticlabels component to represent an object (e.g., desk, chair, couch), and (3) a roomlayout and container component to represent an overall scene (e.g., room)).
in response to determining that the positioning of the environmental map succeeds, determining, according to the identifier of the environmental map, identifiers of a plurality of captured rooms that have been associated with the environmental map and acquiring, according to the identifiers of the plurality of captured rooms that have been associated with the environmental map, anchor information of all or part of the plurality of captured rooms that have been associated with the environmental map ([0032] In particular embodiments, a scene realignment solution is provided to mitigate the localization failure discussed herein. The key idea for this mitigation is to rely on users to indicate that they are in a space they have already manually tagged, and ask them to provide enough information for knowing how to align a cache of the room they tagged earlier into a current map. For instance, if a desired room that an application is looking for is not found or associated with the current map, then the user may be asked to identify one or more entities (e.g., walls) of the room they are in. Particularly, the user may be asked to identify an entity that is not subject to change or relocate, such as a wall. Also, if there are multiple caches of the room (e.g., multiple previously saved rooms or room caches), then the user may be asked to identify a particular room cache to load. Based on the user identified entities and/or the room cache, the system (e.g., artificial-reality system 200) may align a previously saved or cached room and load it into the current map. Therefore, the scene realignment solution is able to mitigate the localization failure based on few user inputs without having the user to go through the entire scene capture process again), ([0148] Assisted Localization Option 2: Named Rooms and Annotated Fixtures—when a user runs a scene capture process or workflow (e.g., scene capture workflow 400) for the first time, the system (e.g., artificial-reality system 200) may ask the user to provide a name for their room and annotate surfaces and objects (volumes) that are permanent fixtures. In the future when a localization failure occurs (e.g., unassisted localization fails), the user may be asked to specify which room they are in (e.g., from a list of available rooms) and point at the permanent fixtures that they had previously annotated. By specifying the room name, a localization algorithm running on the system may pick the correct map. With knowledge of permanent fixtures that were previously annotated, the localization algorithm may have a strong pose prior and a focal point for search. The named rooms and annotated fixtures option may particularly help when relocalization fails due to large scene changes. It may not help when relocalization fails due to lighting changes. However, lighting changes are presumable easier to detect, and may be provided as explicit guidance to user as part of the scene capture workflow. For example, the user may be asked to turn on the light to help with relocalization), ([0160] In some embodiments, a cache definition associated with a cached room may be used during the alignment process, such as the alignment process discussed above in reference to FIG. 12. A room cache or a cached room (e.g., previously-created room) may include a unique identifier (e.g., UUID) of the room entity and a list of pairs, each pair including an anchor ID and a pose of the anchor (e.g., <anchor uuid, pose>). The pose is relative to the origin of the room, where the origin may be the corner of the room (or may be at any place, where a spatial anchor may be defined corresponding to the room entity). When information for a room is needed, the system (e.g., artificial-reality system 200) may use the stored UUIDs (e.g., of the room and the anchors) to query relevant information. The information that cannot be queried (e.g., because the map is not loaded) may be provided by the cache definition. In some embodiments, the cache definition may include, for example, cache, what can be loaded, and corner alignment, each of which can be combined to provide entire room information needed to relocalize).
Noris does not explicitly teach displaying acquired content of the captured rooms according to the acquired anchor information of the captured rooms.
However, Wixson teaches displaying acquired content of the captured rooms according to the acquired anchor information of the captured rooms ([0061] FIGS. 2M through 2-O continue the examples of FIG. 2A-2L, and illustrate mapping information that may be generated from the types of analyses discussed in FIGS. 2A-2L and FIGS. 2P-2V, such as by the MIGM system. In particular, FIG. 2M illustrates an example 2D floor plan 230m that may be constructed based on the positioning of determined final estimated room shapes, which in this example includes walls and indications of doorways and windows. In some embodiments, such a floor plan may have further information shown, such as about other features that are automatically detected by the analysis operations and/or that are subsequently added by one or more users. For example, FIG. 2N illustrates a modified floor plan 230n that includes additional information of various types, such as may be automatically identified from analysis operations of visual data from images and/or from depth data, and added to the floor plan 230m, including one or more of the following types of information: room labels (e.g., “living room” for the living room), room dimensions, visual indications of fixtures or appliances or other built-in features, visual indications of positions of additional types of associated and linked information (e.g., of panorama images and/or perspective images acquired at specified acquisition positions, which an end user may select for further display; of audio annotations and/or sound recordings that an end user may select for further presentation; etc.), visual indications of doorways and windows, etc.—in other embodiments and situations, some or all such types of information may instead be provided by one or more MIGM system operator users and/or ICA system operator users. In addition, if assessment and/or other information generated by the BUAM system is available, it may similarly be added to or otherwise associated with the floor plans 230m and/or 230n, whether in addition to or instead of some or all of the other additional types of information shown for floor plan 230n relative to floor plan 230m, although such BUAM system-generated information is not illustrated in this example. Furthermore, when the floor plans 230m and/or 230n are displayed to an end user, one or more user-selectable controls may be added to provide interactive functionality as part of a GUI (graphical user interface) screen 255n, such as to indicate a current floor that is displayed, to allow the end user to select a different floor to be displayed, etc., with a corresponding example user-selectable control 228 added to the GUI in this example—in addition, in some embodiments, a change in floors or other levels may also be made directly from the displayed floor plan, such as via selection of a corresponding connecting passage (e.g., a stairway to a different floor), and other visual changes may be made directly from the displayed floor plan by selecting corresponding displayed user-selectable controls (e.g., to select a control corresponding to a particular image at a particular location, and to receive a display of that image, whether instead of or in addition to the previous display of the floor plan from which the image is selected). In other embodiments, information for some or all different floors may be displayed simultaneously, such as by displaying separate sub-floor plans for separate floors, or instead by integrating the room connection information for all rooms and floors into a single floor plan that is shown together at once. It will be appreciated that a variety of other types of information may be added in some embodiments, that some of the illustrated types of information may not be provided in some embodiments, and that visual indications of and user selections of linked and associated information may be displayed and selected in other manners in other embodiments) and ([0066] FIG. 2U continues the examples of FIGS. 2P-2T, and provide examples of additional data that may be obtained about the living room based at least in part on analysis of one or more initial room-level images of the living room, such as panorama image 250q and/or multiple perspective images that include images 250a-250c and include visual data of all or substantially all of the living room. In particular, FIG. 2U illustrates information 255u that shows alternative examples 237a and 237b of a room shape of the living room (e.g., as may be determined by the MIGM system, as discussed in greater detail elsewhere herein), along with additional data 236u and 238 for room shape 237a that may be determined based at least in part on automated operations of the BUAM system and optionally additional actions of an associated user. In this example, the illustrated information 236u provides an example of expected traffic flow information for the living room, such as based at least in part on a determined layout (not shown) for the living room (e.g., using information about furniture in the living room and inter-wall openings). In addition, the illustrated information 238 indicates that a target attribute of the west window may have been evaluated as showing a mountain view in this example (e.g., based at least in part on an automated determination using visual data that is visible through the window; at least in part using information from an associated user; at least in part using information from other sources, such as publicly available data; etc.). It will be appreciated that these types of additional information are illustrated in FIG. 2U are non-exclusive examples provided for the purpose of illustration, and that similar and/or other types of information may be determined in other manners in other embodiments).
Noris in view of Wixson teaches claim 8. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Noris’s multi-room capture system to explicitly including displaying acquired content of captured rooms according to the acquired anchor information of the captured rooms, as taught by Wixson, as the references are in the analogous art of multi-room capture systems. An advantage of the modification is that it achieves the result of displaying for a user multi-room capture data for easy of information processing and user management of a property.
Re claim 9, Noris and Wixson teaches claim 8. Furthermore, Noris teaches herein acquiring, according to the identifiers of the plurality of captured rooms that have been associated with the environmental map, the anchor information of all or part of the plurality of captured rooms that have been associated with the environmental map comprises:
determining, according to a loading rule and from the identifiers of the plurality of captured rooms that have been associated with the environmental map, an identifier of a second captured room to be loaded; and loading the anchor information of the second captured room into a memory according to the identifier of the second captured room ([0164] In the scene alignment process, the system may instruct the user to select certain entities in their environment (e.g., living room). The certain entities may be those that are fixed at their locations and not movable. For instance, the certain entities may be walls. In particular, the user may be asked to touch or select two walls. Once the user identifies the two walls, the system may determine a room corner (e.g., room corner 1210) based on a point of intersection of the two identified walls. Based on the room corner, the system may align the cached room with the current map. Stated differently, the system may re-position the previously-created map (e.g., Map1 of living room) until it aligns with the room corner of the current map. Once the alignment is complete, at step 1314, the system may move the room or room definition from its original map to currently loaded one. In other words, the system may load the cached room (e.g., Map1) into the current map. It should be noted that unique identifies (e.g., uuid) of anchors may not change because of this move. This is because applications may associate content with anchors, and changing uuid would invalidate them. In particular embodiments, the system may remove the room definition (e.g., scene description or scene model) from the previous map or cached room since it is now associated with or loaded into the current map. This avoids having duplicates which may diverge, and simplifies the system design), ([0161] FIG. 13 illustrates an example relocalization flow or method 1300 for localization failure handling. In particular embodiments, steps 1302-1314 illustrated in the method 1300 may be performed by a localization algorithm running on an artificial-reality system, such as artificial-reality system 200. The method 1300 begins, at step 1302, where a user (e.g., user 202) launches an application on the system (e.g., VR headset). The launched application may require a room or a scene definition, which be associated with a scene model as discussed herein. At step 1304, the system may make a determination whether the room definition or scene description is associated with a current map that is currently loaded on the system. For example, when the user wearing the artificial-reality system (e.g., VR headset) walks into their living room, the system loads a map. The user launches an application that requires a room definition (e.g., scene model) in order to add one or more AR elements to the user's current environment. The system makes the determination of whether the room definition is associated with the current map. As an example, the system may make this determination based on a query “xrQuerySpatialEntity(hasComponent=RoomLayout)”. The query will return a space if the room exists. This query may return only the room in the current map, not in other maps. In some instances, a room entity may be implemented as a spatial anchor internally, so that this spatial anchor is associated to a single map and can be queried), and ([0160] In some embodiments, a cache definition associated with a cached room may be used during the alignment process, such as the alignment process discussed above in reference to FIG. 12. A room cache or a cached room (e.g., previously-created room) may include a unique identifier (e.g., UUID) of the room entity and a list of pairs, each pair including an anchor ID and a pose of the anchor (e.g., <anchor uuid, pose>). The pose is relative to the origin of the room, where the origin may be the corner of the room (or may be at any place, where a spatial anchor may be defined corresponding to the room entity). When information for a room is needed, the system (e.g., artificial-reality system 200) may use the stored UUIDs (e.g., of the room and the anchors) to query relevant information. The information that cannot be queried (e.g., because the map is not loaded) may be provided by the cache definition. In some embodiments, the cache definition may include, for example, cache, what can be loaded, and corner alignment, each of which can be combined to provide entire room information needed to relocalize).
Re claim 12, Noris teaches claim 1. Furthermore, Noris teaches a first management interface in an extended reality space, wherein the first interface comprises a space list of captured spaces of the users, a room list of captured rooms of the captured spaces, and the first captures space belongs to a space of the captured spaces of the user ([0134] In some embodiments, a client application (e.g., first-party application or third-party application running on artificial-reality system) may require only a specific surface (e.g., workrooms require a desk only), not a full room. In this scenario, the client application may query for specific planes or objects and may invoke a partial scene capture process. The partial scene capture process may include capturing only the required planes or objects, required by the client application, in a user's physical environment (e.g., room). In other words, instead of performing an entire scene capture process or workflow (e.g., all steps of the scene capture workflow 400), a subset or a portion of the scene capture process may be performed. FIG. 11 illustrates an example method 1100 for invoking a partial scene capture process. The method 1100 may begin, at step 1102, where an application (e.g., first-party application or third-party application) may query for certain anchors (e.g., spaces with Bounded2D or Bounded3D component). At step 1104, the application may check semantic types in the returned list of space. At step 1106, the application may make a determination whether all required semantic types exist. If all the required semantic types do not exist, at step 1108, the application may request scene capture with a set of required semantic types. At step 1110, the application may invoke a partial scene capture process or workflow (e.g., a portion of scene capture workflow 400 or 600), where existing anchors may be loaded or updated, and new anchors may be added and saved. If on the other hand, the result of the determination in step 1106 is positive (e.g., all the required semantic types do exist), the method 1100 may end and the application may use the scene model), ([0067] In some embodiments, a scene model discussed herein is like a scene graph. The scene graph may be a structed spatial logical hierarchy with scene-related information organized at various levels in the hierarchy. FIG. 3 illustrates an example scene graph 300. In particular embodiments, users (e.g., developers) may be able to query or interact with a scene graph (e.g., scene graph 300) via a set of semantic queries. As shown in FIG. 3, the scene graph 300 represents scene-related information of a real world 302, which may include a plurality of objects 304a-304g (individually or collectively herein referred to as 304) and a plurality of groups 306a-306c (individually or collectively herein referred to as 306). Two or more objects 304 may be grouped together to form a group 306 to represent a larger space or part of the world 302. By way of an example and without limitation, the world 302 may be a house of a user, where different groups 306 may represent different rooms of the house and each group 306 (e.g., room) may include one or more of sub-group(s) or object(s) (e.g., bed, tv, lamp, wall art, couch, etc.) to represent things that are part of that room), and ([0068] In some embodiments, a list of objects 304 in the scene graph 300 may be categorized with semantic meanings and organized by spatial relationships. In some embodiments, rest of the world may be kept as an uncategorized mesh to keep the world watertight for physics or occlusion. Each object in a scene graph may be composed of one or more of the following components: [0069] The plane that a user may place virtual things on. [0070] Mesh, which may represent the most detailed geometry. [0071] Collider mesh, which is typically simpler and suitable to be used by physics engines. [0072] Visual mesh, which is usually with reasonable detail suitable to be used by visualization, occlusion, etc.) and (see [0148], list of rooms.
Noris does not explicitly teach displaying a first management interface in an extended reality space in response to a second instruction, and a 2d view of a space layout of a first captured space. However, Wixson teaches displaying a first management interface in an extended reality space in response to a second instruction, and a 2d view of a space layout of a first captured space (see Fig. 2N, as an example of a 2d view of first captured space) and (see Fig. 2p and 2q displays including listed items and layouts in 3d), and ([0024] Initial images of a room and additional data about the room may be captured in various manners in various embodiments. In some embodiments, some or all of the initial images of a room may be provided to the BUAM system by another system that already acquired those images for other uses, such as by an Image Capture and Analysis (ICA) system and/or a Mapping Information Generation Manager (MIGM) system that uses images of rooms of a building to generate floor plans and/or other mapping information related to the building, as discussed in greater detail below. In other embodiments, some or all of the initial images of a room may be captured by the BUAM system or in response to instructions provided by the BUAM system, such as to an automated image acquisition device in the room and/or to a user (e.g., a BUAM system operator user) in the room with information indicating the types of initial images to capture. In a similar manner, in at least some embodiments, some or all of the additional images for a room may be captured by the BUAM system or in response to instructions provided by the BUAM system, such as to an automated image acquisition device in the room and/or to a user (e.g., a BUAM system operator user) in the room with information indicating the types of additional images to capture. For example, the BUAM system may provide instructions that identify the one or more objects of interest in a room for which to capture additional data, and that identify the one or more target attributes for each of the objects of interest for which to capture additional data that satisfies a defined detail threshold or otherwise satisfies one or more defined detail criteria (or otherwise provides a description of the additional data to capture for the object that causes sufficient data about the one or more target attributes to be captured). Such instructions may be provided in various manners in various embodiments, including to be displayed to a user in a GUI of the BUAM system on a mobile computing device of the user (e.g., a mobile computing device that acts as an image acquisition device and is used to capture some or all of the additional images, such as using one or more imaging sensors of that device and optionally additional hardware components of that device, such as a light, one or more IMU (internal measurement unit) sensors such as one or more gyroscopes and/or accelerometers and/or magnetometers or other compasses, etc.) or otherwise provided to the user (e.g., overlaid on an image of the room that is shown on such a mobile computing device and/or other separate camera device, such as to provide dynamic augmented reality instructions to the user as the image changes in response to movement of the device, and/or to provide static instructions to the user on a previously captured image, and optionally with visual markings on the image(s) of visible objects and/or target attribute), or instead provided to an automated device that acquires the additional images in response to the instructions). Noris and Wixson teaches claim 12. For motivation, see claim 8.
Re claim 13, Noris and Wixson teaches claim 12. Furthermore, Wixson teaches displaying a 3d view of the space layout of the first captured space in the extended reality space in response to a third instruction ([0062] FIG. 2-O continues the examples of FIGS. 2A-2N, and illustrates additional information 265o that may be generated from the automated analysis techniques disclosed herein and displayed (e.g., in a GUI similar to that of FIG. 2N), which in this example is a 2.5D or 3D model floor plan of the house. Such a model 265o may be additional mapping-related information that is generated based on the floor plan 230m and/or 230n, with additional information about height shown in order to illustrate visual locations in walls of features such as windows and doors, or instead by combined final estimated room shapes that are 3D shapes. While not illustrated in FIG. 2-0, additional information may be added to the displayed walls in some embodiments, such as from images taken during the video capture (e.g., to render and illustrate actual paint, wallpaper or other surfaces from the house on the rendered model 265), and/or may otherwise be used to add specified colors, textures or other visual information to walls and/or other surfaces. In addition, some or all of the additional types of information illustrated in FIG. 2N may similarly be added to and shown in the floor plan model 265o). For motivation, see claim 8.
Re claim 15, Noris and Wixson teaches claim 13. Furthermore, Wixson teaches displaying, in the extended reality space, a 3d model of the first room in the first captured space in response to a 3d model evocation instruction ([0062] FIG. 2-O continues the examples of FIGS. 2A-2N, and illustrates additional information 265o that may be generated from the automated analysis techniques disclosed herein and displayed (e.g., in a GUI similar to that of FIG. 2N), which in this example is a 2.5D or 3D model floor plan of the house. Such a model 265o may be additional mapping-related information that is generated based on the floor plan 230m and/or 230n, with additional information about height shown in order to illustrate visual locations in walls of features such as windows and doors, or instead by combined final estimated room shapes that are 3D shapes. While not illustrated in FIG. 2-0, additional information may be added to the displayed walls in some embodiments, such as from images taken during the video capture (e.g., to render and illustrate actual paint, wallpaper or other surfaces from the house on the rendered model 265), and/or may otherwise be used to add specified colors, textures or other visual information to walls and/or other surfaces. In addition, some or all of the additional types of information illustrated in FIG. 2N may similarly be added to and shown in the floor plan model 265o). For motivation, see claim 8.
Re claim 18, Noris teaches claim 1. Furthermore, Noris teaches a second management interface, wherein the second management interface comprises a space list of captured spaces of a user, a room list of captured rooms of the captured spaces, and management controls of the captured rooms of the captured spaces ([0004] Embodiments described herein relate to generating, querying, and managing a scene model. The scene model is an objective (e.g., single source of truth), system-managed, comprehensive, and an up-to-date representation of a user's physical or real world that may be easily indexable and queryable. The scene model may describe static geometry and semantics of the real world. In particular embodiments, the scene model may be composed of a plurality of anchors, where each anchor represents a plane, surface, or an object in a user's physical environment (e.g., user's living room). In some embodiments, the scene model discussed herein may be in the form of a scene graph or hierarchical tree structure comprising of the set of anchors, where each anchor corresponds to an entity in the user's physical environment. These anchors may include, for example, (1) a bounded2D and semanticlabels component to represent a plane (e.g., floor, wall, ceiling, etc.) (2) a bounded3D and semanticlabels component to represent an object (e.g., desk, chair, couch), and (3) a roomlayout and container component to represent an overall scene (e.g., room)).
in response to determining that the positioning of the environmental map succeeds, determining, according to the identifier of the environmental map, identifiers of a plurality of captured rooms that have been associated with the environmental map and acquiring, according to the identifiers of the plurality of captured rooms that have been associated with the environmental map, anchor information of all or part of the plurality of captured rooms that have been associated with the environmental map ([0032] In particular embodiments, a scene realignment solution is provided to mitigate the localization failure discussed herein. The key idea for this mitigation is to rely on users to indicate that they are in a space they have already manually tagged, and ask them to provide enough information for knowing how to align a cache of the room they tagged earlier into a current map. For instance, if a desired room that an application is looking for is not found or associated with the current map, then the user may be asked to identify one or more entities (e.g., walls) of the room they are in. Particularly, the user may be asked to identify an entity that is not subject to change or relocate, such as a wall. Also, if there are multiple caches of the room (e.g., multiple previously saved rooms or room caches), then the user may be asked to identify a particular room cache to load. Based on the user identified entities and/or the room cache, the system (e.g., artificial-reality system 200) may align a previously saved or cached room and load it into the current map. Therefore, the scene realignment solution is able to mitigate the localization failure based on few user inputs without having the user to go through the entire scene capture process again), ([0148] Assisted Localization Option 2: Named Rooms and Annotated Fixtures—when a user runs a scene capture process or workflow (e.g., scene capture workflow 400) for the first time, the system (e.g., artificial-reality system 200) may ask the user to provide a name for their room and annotate surfaces and objects (volumes) that are permanent fixtures. In the future when a localization failure occurs (e.g., unassisted localization fails), the user may be asked to specify which room they are in (e.g., from a list of available rooms) and point at the permanent fixtures that they had previously annotated. By specifying the room name, a localization algorithm running on the system may pick the correct map. With knowledge of permanent fixtures that were previously annotated, the localization algorithm may have a strong pose prior and a focal point for search. The named rooms and annotated fixtures option may particularly help when relocalization fails due to large scene changes. It may not help when relocalization fails due to lighting changes. However, lighting changes are presumable easier to detect, and may be provided as explicit guidance to user as part of the scene capture workflow. For example, the user may be asked to turn on the light to help with relocalization), ([0160] In some embodiments, a cache definition associated with a cached room may be used during the alignment process, such as the alignment process discussed above in reference to FIG. 12. A room cache or a cached room (e.g., previously-created room) may include a unique identifier (e.g., UUID) of the room entity and a list of pairs, each pair including an anchor ID and a pose of the anchor (e.g., <anchor uuid, pose>). The pose is relative to the origin of the room, where the origin may be the corner of the room (or may be at any place, where a spatial anchor may be defined corresponding to the room entity). When information for a room is needed, the system (e.g., artificial-reality system 200) may use the stored UUIDs (e.g., of the room and the anchors) to query relevant information. The information that cannot be queried (e.g., because the map is not loaded) may be provided by the cache definition. In some embodiments, the cache definition may include, for example, cache, what can be loaded, and corner alignment, each of which can be combined to provide entire room information needed to relocalize).
and in response to a second operation by the user on the management control of a fifth captured room of the captured space, performing the following operations on the fifth captured room: room modification and room deletion, wherein the room modification comprises one or more of the following modifications: modifying a name of a room, resetting a space capture result of a room, adding furniture within a room, deleting furniture within a room, or modifying an anchor within a room ([0144] In some embodiments, an artificial-reality system (e.g., artificial-reality system 200, which may be a VR device) may maintain three maps (e.g., scene maps, room maps) on the device or the system. Stored maps may be deleted over time in a least recently used (LRU) order. When this changes, there may be a database of map chunks that have anchors that may be associated or not. In some embodiments, the system may reduce localization failures by, for example and without limitation, (1) prioritizing localization into internal anchors that have spatial anchors attached to them, and (2) prioritizing non-deletion of internal anchors and their backing map data that have spatial anchors attached. The internal anchors may be deleted in an LRU manner when hitting max capacity), see ([0179], manage, retrieve, modify, add, or delete information stored in data store 1564), and (see [0136] asking user to name and localize into specific spaces).
Noris does not explicitly teach displaying in the extended reality space the second management interface. However, Wixson teaches displaying in the extended reality space the second management interface ([0061] FIGS. 2M through 2-O continue the examples of FIG. 2A-2L, and illustrate mapping information that may be generated from the types of analyses discussed in FIGS. 2A-2L and FIGS. 2P-2V, such as by the MIGM system. In particular, FIG. 2M illustrates an example 2D floor plan 230m that may be constructed based on the positioning of determined final estimated room shapes, which in this example includes walls and indications of doorways and windows. In some embodiments, such a floor plan may have further information shown, such as about other features that are automatically detected by the analysis operations and/or that are subsequently added by one or more users. For example, FIG. 2N illustrates a modified floor plan 230n that includes additional information of various types, such as may be automatically identified from analysis operations of visual data from images and/or from depth data, and added to the floor plan 230m, including one or more of the following types of information: room labels (e.g., “living room” for the living room), room dimensions, visual indications of fixtures or appliances or other built-in features, visual indications of positions of additional types of associated and linked information (e.g., of panorama images and/or perspective images acquired at specified acquisition positions, which an end user may select for further display; of audio annotations and/or sound recordings that an end user may select for further presentation; etc.), visual indications of doorways and windows, etc.—in other embodiments and situations, some or all such types of information may instead be provided by one or more MIGM system operator users and/or ICA system operator users. In addition, if assessment and/or other information generated by the BUAM system is available, it may similarly be added to or otherwise associated with the floor plans 230m and/or 230n, whether in addition to or instead of some or all of the other additional types of information shown for floor plan 230n relative to floor plan 230m, although such BUAM system-generated information is not illustrated in this example. Furthermore, when the floor plans 230m and/or 230n are displayed to an end user, one or more user-selectable controls may be added to provide interactive functionality as part of a GUI (graphical user interface) screen 255n, such as to indicate a current floor that is displayed, to allow the end user to select a different floor to be displayed, etc., with a corresponding example user-selectable control 228 added to the GUI in this example—in addition, in some embodiments, a change in floors or other levels may also be made directly from the displayed floor plan, such as via selection of a corresponding connecting passage (e.g., a stairway to a different floor), and other visual changes may be made directly from the displayed floor plan by selecting corresponding displayed user-selectable controls (e.g., to select a control corresponding to a particular image at a particular location, and to receive a display of that image, whether instead of or in addition to the previous display of the floor plan from which the image is selected). In other embodiments, information for some or all different floors may be displayed simultaneously, such as by displaying separate sub-floor plans for separate floors, or instead by integrating the room connection information for all rooms and floors into a single floor plan that is shown together at once. It will be appreciated that a variety of other types of information may be added in some embodiments, that some of the illustrated types of information may not be provided in some embodiments, and that visual indications of and user selections of linked and associated information may be displayed and selected in other manners in other embodiments) and ([0066] FIG. 2U continues the examples of FIGS. 2P-2T, and provide examples of additional data that may be obtained about the living room based at least in part on analysis of one or more initial room-level images of the living room, such as panorama image 250q and/or multiple perspective images that include images 250a-250c and include visual data of all or substantially all of the living room. In particular, FIG. 2U illustrates information 255u that shows alternative examples 237a and 237b of a room shape of the living room (e.g., as may be determined by the MIGM system, as discussed in greater detail elsewhere herein), along with additional data 236u and 238 for room shape 237a that may be determined based at least in part on automated operations of the BUAM system and optionally additional actions of an associated user. In this example, the illustrated information 236u provides an example of expected traffic flow information for the living room, such as based at least in part on a determined layout (not shown) for the living room (e.g., using information about furniture in the living room and inter-wall openings). In addition, the illustrated information 238 indicates that a target attribute of the west window may have been evaluated as showing a mountain view in this example (e.g., based at least in part on an automated determination using visual data that is visible through the window; at least in part using information from an associated user; at least in part using information from other sources, such as publicly available data; etc.). It will be appreciated that these types of additional information are illustrated in FIG. 2U are non-exclusive examples provided for the purpose of illustration, and that similar and/or other types of information may be determined in other manners in other embodiments).
Noris and Wixson teaches claim 18. For motivation, see claim 8.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Noris et al. (US 2023/0127307) in view of Wixson et al. (US 20220269885) and Hill (US 20230230326).
Re claim 16, Noris and Wixson teaches claim 15. Furthermore, Noris teaches upon detecting a first operation, controlling the 3d model of the first room to enter an editing state (see [0179], interfaces that enable AR/VR systems to manage, retrieve, modify, add, or delete information stored) and (see [0110], adding captured entities such as adding ceiling to the user’s room layout).
Noris and Wixson does not explicitly teach displaying with a preset effect after the 3d model enters a state and upon detecting a second operation, controlling the 3d model of the first room to rotate.
However, Hill teaches displaying with a preset effect after the 3d model of the first room enters the editing state and upon detecting a second operation, controlling the 3d model of the first room to rotate ([0071] The machine learning system 200 analyzes and processes the 3D scan 40 to render in a format that is displayable in a web browser, the user interface 220, or other software application using 2D and 3D graphics rendering software (e.g., Unity or WebGL). The digital model 510 is viewable, e.g., as a visual model representation, on a display that is connected to a computing device. The display can be, for example, a computer monitor that has a wired or wireless connection to the user computing device 12, to the remote computing device 20, or to another computing device. In some implementations, the display can be an integrated display of the user device 12, e.g., a touch screen of a smartphone. The digital model 510 generally includes a visual scale, e.g., as represented in a visual model representation, that corresponds to the spatial dimensions of the room 50. The digital model 510 may be manipulated by a user through the user interface 220 using a pointing device (e.g., a computer mouse), gestures on a touch screen, or other interactive method. For example, the digital model 510 may be rotated 360 degrees or tilted at various angles and in various directions to provide views of the digital model 510 on the display from different vantage points relative to the user. In this way, the user can visualize the actual room 50 from different vantage points by visually referring to the digital model 510. The machine learning system 200 or another process of the system 10 may also include features that allow other modifications to the digital model 510 such as, for example, changes to flooring, wall coverings, lighting fixtures, appliances, window treatments, doors, entryways, window types, baseboards, crown molding, chair rail, paneling, wainscoting, ceiling types, ceiling coffering, and colors or textures of any of the foregoing. The machine learning system 200 or another process of the system 10 may also include features that allow modifications to “structural” elements 512 (e.g., digital planes) of the digital model 510 such as, for example, the addition, removal, or movement of walls, doors, entryways, and windows. For example, the user interface 220 can be configured to receive user inputs to adjust a location (e.g., coordinates) of a plane defining a boundary of the space 50 represented by digital model 510. The machine learning system 200 or another process of the system 10 may also include features that allow modifications to lighting direction, color, and intensity in the digital model to simulate actual lighting conditions in the room.
Noris and Wixson and Hill teaches claim 16. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Noris and Wixson’s 3d multi-room capture system with capture policies for objects including editing states in extended realities to explicit include an interface for controlling the 3d model of a first room to rotate, as taught by Hill. An advantage of the modification is that it achieves the result of editing of 3d and augmented reality visualizations such as rotating objects like room/map views.
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Noris et al. (US 2023/0127307) in view of Wixson et al. (US 20220269885) and Ravasz et al. (US 20240320930, hereinafter “Rav”).
Re claim 17, Noris and Wixson teaches claim 15. Noris teaches wherein the first captured space is a space where the user is located currently, the first room is a room wherein the user is located currently ([0082] In particular embodiments, the manual scene capture workflow 400 may be initiated in response to a user wearing an artificial-reality system (e.g., artificial-reality system 200) walking/entering into the room and an application (e.g., first-party application or third-party application) on the artificial-reality system determining that a scene description or room definition for the room is not present. The scene capture workflow 400 may begin, at step 402, with the artificial-reality system 200 presenting a welcome screen to the user 202 wearing the artificial-reality system 200 to initiate a screen capture process. FIG. 5A illustrates an example graphical user interface 500a that may be displayed to the user 202 to initiate the screen capture process. The graphical user interface 500a may include an image 501 and a scene-capture-assist window 502. The image 501 may be displayed as a passthrough image to the user 202. The scene-capture-assist window 502 may be displayed as an AR element on top of the image 501 that the user 202 may be currently seeing. As depicted, the screen-capture-assist window 502 may indicate to the user 202 to setup their room in VR and present two options, including a continue option 506 and a cancel option 508. The continue option 506 may initiate the scene capture process, while the cancel option 508 may cancel the process and exit the scene capture workflow 400. The user 202 may select a desired option via a controller (e.g., controller 206) by hovering over and clicking on the desired option. Once the user confirms the continue option 506 (e.g., as shown by reference numeral 510), the artificial-reality system 200 may initiate the scene capture process).
Noris and Wixon does not explicitly teach the method further comprises: in response to detecting that a rotation angle of a head-mounted device is greater than a preset angle or the 3D model of the first room exceeds a field of view of the user, controlling, according to a current location of the user, the 3D model of the first room to move to a preset location within the field of view of the user.
However, Rav teaches the method further comprises: in response to detecting that a rotation angle of a head-mounted device is greater than a preset angle or the 3D model of the first room exceeds a field of view of the user, controlling, according to a current location of the user, the 3D model of the first room to move to a preset location within the field of view of the user ([0111] Viewpoint-locked virtual object: A virtual object is viewpoint-locked when a computer system displays the virtual object at the same location and/or position in the viewpoint of the user, even as the viewpoint of the user shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the viewpoint of the user is locked to the forward facing direction of the user's head (e.g., the viewpoint of the user is at least a portion of the field-of-view of the user when the user is looking straight ahead); thus, the viewpoint of the user remains fixed even as the user's gaze is shifted, without moving the user's head. In embodiments where the computer system has a display generation component (e.g., a display screen) that can be repositioned with respect to the user's head, the viewpoint of the user is the augmented reality view that is being presented to the user on a display generation component of the computer system. For example, a viewpoint-locked virtual object that is displayed in the upper left corner of the viewpoint of the user, when the viewpoint of the user is in a first orientation (e.g., with the user's head facing north) continues to be displayed in the upper left corner of the viewpoint of the user, even as the viewpoint of the user changes to a second orientation (e.g., with the user's head facing west). In other words, the location and/or position at which the viewpoint-locked virtual object is displayed in the viewpoint of the user is independent of the user's position and/or orientation in the physical environment. In embodiments in which the computer system is a head-mounted device, the viewpoint of the user is locked to the orientation of the user's head, such that the virtual object is also referred to as a “head-locked virtual object).
Noris and Wixson and Rav teaches claim 17. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Noris and Wixson’s 3d multi-room capture system for display 3d views to include moving a 3d model to a preset location within the field of view of the user, as taught by Rav. An advantage of the modification is that it achieves the result of head locking virtual objects to a particular location in a user’s ar/vr view, thus displaying an object in a position regardless of user’ head orientation, as taught by Rav.
Allowable Subject Matter
Claims 6, 10-11, 14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, as well as needed corrections to overcome 112 rejections.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Peter Hoang whose telephone number is (571)270-1346. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hajnik F. Daniel can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PETER HOANG/ Primary Examiner, Art Unit 2616
Prosecution Insights