DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities: Fig. 11 indicates the label of 1110 but does not disclose the label 1110 in the specifications. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 8 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kutliroff et al. (U.S. Pub. No. 20170243352) in view of Bell et al. (U.S. Pub. No. 20160055268) and Kim et al. (U.S. Pub. No. 20220343613).
Regarding claim 1, Kutliroff discloses an augmented reality device comprising (para 26, “In some embodiments, the scene analysis system 130, along with an AR manipulation circuit 140, may be integrated into a mobile platform 102 along with a depth camera 110 and a display element 112. The mobile platform 102 may be a tablet or smartphone or other such similar device.”; also, para 80, “AR manipulation circuit 140 is configured to implement augmented reality operations including removal and insertion of objects and the generation of blueprints based on the scene analysis.”): a camera (para 26, “The depth camera may be configured as a rear mounted camera on the platform to facilitate scanning of the scene 120 as the user moves with respect to the scene to capture 3D image frames from multiple perspectives or camera poses.”); at least one processor including processing circuitry (para 72, “In some embodiments, platform 102 may comprise any combination of a processor 1920, a memory 1930, a scene analysis system 130, an AR manipulation circuit 140, a depth camera 110, a network interface 1940, an input/output (I/O) system 1950, a display element 112, and a storage system 1970.”; also, para 89, “The terms "circuit" or "circuitry," as used in any embodiment herein, may comprise, for example, singly or in any combination, hardwired or purpose-built circuitry, programmable circuitry such as computer processors comprising one or more individual instruction processing cores, state machine circuitry, and/or firmware that stores instructions executed by programmable circuitry.”); memory storing one or more instructions (para 85, “The instructions can be provided in the form of one or more computer software applications and/or applets that are tangibly embodied on a memory device, and that can be executed by a computer having any suitable architecture.”); and wherein the one or more instructions are configured to, when executed by the at least one processor individually or collectively, cause the augmented reality device to (para 87, “Some embodiments may be implemented, for example, using a machine readable medium or article which may store an instruction or a set of instructions that, if executed by a machine, may cause the machine to perform a method and/or operations in accordance with the embodiments.”; also, para 89, “The circuitry may include a processor and/or controller configured to execute one or more instructions to perform one or more operations described herein.”): obtain, through the camera, a spatial image based on a photograph of a real world space (para 24, “In accordance with an embodiment, the 3D scene analysis techniques generate data that describes a real-world scene as imaged by a depth camera from various perspectives.”; also, para 26, “In some embodiments, the depth camera may be configured to capture a stream of images at a relatively high frame rate, for example 30 frames per second or more. Each 3D image frame may comprise a color image frame that provides color (e.g., red, green and blue or RGB) pixels, and a depth map frame that provides depth pixels.”; also, para 30, “As new RGB and depth frames of the scene 120 are captured by depth camera 110 they are provided to the camera pose calculation circuit 302.”; also, a spatial image is a 3d photograph that stores depth data with visual pixels where the depth data can be captured by a depth camera), identify a first plane comprising a wall and a second plane comprising a floor from the spatial image (para 38, “The plane fitting circuit 702 may be configured to scan the 3D reconstruction of the scene for planar surfaces. One method to accomplish this is to calculate normal vectors to the surfaces by scanning the depth maps and calculating the cross product of differences of neighboring depth pixels. The normal vectors are then clustered into groups based on spatial proximity and the values of the vectors. Next, a plane is fitted to each cluster.”; also, para 39, “Floor/wall/ceiling determination circuit 704 may be configured to identify floors, walls and ceilings from the list of planes detected in the scene. In some embodiments, a gyroscope inertial sensor 304 is employed to detect the direction of gravity during scanning, and this data is stored during the scanning process. The direction of gravity may then be used to find the plane corresponding to the floor. Next, planar surfaces that are perpendicular to the floor plane (within a margin of error) and cover a sufficiently large region of the scene are identified as walls.”), segment the object on the spatial image from the real world space based on a 3D model or 3D position information of the object using 3D segmentation (para 27, “The 3D segmentation circuit 136 may be configured to generate an estimate of 3D boundaries of objects of interest in the scene based on 3D images obtained from a number of poses of a depth camera. The estimated boundaries may be expressed as a set of 3D pixels associated with the boundary.”; also, para 37, “The 3D segmentation circuit may be configured to segment out each of the objects detected, finding the points of the 3D reconstruction that correspond to the contours of the objects.”; also, para 63, “For each pixel, the associated 3D position of the 2D pixel is computed, by sampling the associated depth map to obtain the associated depth pixel and projecting that depth pixel to a point in 3D space, at operation 1412. A ray is then generated which extends from the camera to the location of the projected point in 3D space at operation 1414. The point at this 3D position is included in the object boundary set at operation 1416.”). Kutliroff does not disclose generate a three-dimensional (3D) model of the real world space by extending the wall and extending the floor along their respective planes and performing 3D inpainting on an area of the extended wall and the extended floor that is hidden by an obstructive object, segment an object selected by a user input from the spatial image based on two- dimensional (2D) segmentation.
However, in a similar field of endeavor, Bell discloses generate a three-dimensional (3D) model of the real world space by extending the wall and extending the floor along their respective planes and performing 3D inpainting on an area of the extended wall and the extended floor that is hidden by an obstructive object (para 29, “In an aspect, the captured 3D data can be generated (e.g., captured) via at least one 3D reconstruction system. For example, the at least one 3D reconstruction system can employ two-dimensional (2D) image data and/or depth data captured from one or more 3D sensors (e.g., laser scanners, structured light systems, time-of-flight systems, etc.) to automatically and/or semi-automatically generate a 3D model of an interior environment (e.g., architectural spaces, architectural structures, physical objects,”; also, para 32, “Flat surfaces can include walls, floors, and/or ceilings associated with the captured 3D data.”; also, para 40, “Missing data (e.g., holes) can be associated with more than one surface (e.g., holes may extend across multiple planes). As such, in response to a determination that missing data (e.g., a hole) is associated with more than one surface, the data generation component 106 can extend all surfaces adjacent to the missing data (e.g., based on parameterizations associated with the surfaces adjacent to the missing data) until each of the surfaces adjacent to the missing data intersect.”; also, para 66, “Walls and/or floors can be extended behind a moulding to intersect with a portal or with one another.”; also, para 35, “Captured 3D data for an occluded area can be missing when an area on a surface (e.g., a wall, a floor, a ceiling, or another surface) is occluded by an object (e.g., furniture, items on a wall, etc.). Therefore, the data generation component 106 can predict and/or generate data (e.g., geometry data and/or texture data) for a particular surface when an object on the particular surface is moved or removed.”; also, para 36, “For example, the data generation component 106 can determine whether an object or another surface blocked a portion of a particular surface during a 3D capture process. Furthermore, the data generation component 106 can determine that an edge of the particular surface that was occluded during the 3D capture process is an occlusion boundary”; also, para 60, “Examples of an object attached to a particular flat surface include, but are not limited to, a table attached to a floor, a painting attached to a wall, a bookcase attached to a floor and a wall, etc.”; also, para 41, “The data generation component 106 can sample a region surrounding missing data (e.g., a hole) via one or more texture synthesis algorithms, including, but not limited to, one or more tiling algorithms, one or more stochastic texture synthesis algorithms, one or more structured texture synthesis algorithms (e.g., one or more single purpose structured texture synthesis algorithms), one or more pixel-based texture synthesis algorithms, one or more patch-based texture synthesis algorithms, one or more fast texture synthesis algorithms (e.g., using tree-structured vector quantization) and/or one or more other texture synthesis algorithms. An appearance of the region surrounding the missing data (e.g., the hole) can be employed to constrain texture synthesis. Textures can be synthesized separately for each surface in response to a determination that missing data (e.g., a hole) is associated with more than one surface. Furthermore, texture for a particular surface can be synthesized onto a portion of the missing data (e.g., the hole) corresponding to the particular surface.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kutliroff's invention of an augmented reality device having a camera, a processor including processing circuitry and a memory storing instructions, which obtains a spatial image of a real world space through the camera, identifies a first plane comprising a wall and a second plane comprising a floor from that image, and segments an object from the real world space using three-dimensional segmentation, with the features of Bell's invention of generating a three-dimensional model of an interior environment by extending each surface adjacent to a hole until those surfaces intersect and synthesizing texture separately for each such surface onto the portion of the hole corresponding to it. The combination would have been obvious because Kutliroff and Bell address consecutive halves of a single operation on the same indoor scene, and Bell says so in the same words the combination requires. Kutliroff removes an object of interest from the scene and leaves the wall and floor it stood against unreconstructed, having identified those planes only in order to discard their points before clustering. Bell begins from exactly that condition, stating that data for a surface is missing where the surface "is occluded by an object (e.g., furniture, items on a wall, etc.)" and that its data generation component supplies geometry and texture "when an object on the particular surface is moved or removed." A person of ordinary skill working on Kutliroff's object-removal operation would therefore have looked to Bell for the step Kutliroff leaves undone, and the result is predictable, because Kutliroff has already computed the very plane parameters Bell's extension consumes, so the added stage requires no change to how Kutliroff detects planes or clusters objects and yields a scene in which the region the removed object occupied is filled with surface geometry and matching texture rather than left empty.
Kim discloses segment an object selected by a user input from the spatial image based on two- dimensional (2D) segmentation (para 40, “The moving object selection unit 120 is input with the moving object from the user. Here, the moving object is a part corresponding the real object to be moved in the 3D augmented reality and the real object to be moved is selected by a user. That is, the user selects the real object to be moved in the 3D augmented reality”; also, para 40, “The object region division unit 130 divides the region of the moving object selected by the user in a 2D color image (texture information). In addition, the object region division unit 130 performs separation of a foreground and the background on 2D and 3D by using a 2D and 3D relationship to divide the region of the moving object even on the 3D.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, in which an augmented reality device identifies a wall plane and a floor plane, segments an object from the real world space in three dimensions, and reconstructs and textures the surfaces the object occluded, with the features of Kim's invention of dividing the region of a user-selected object in a two-dimensional color image before dividing that same object's region in three dimensions. The combination would have been obvious because the base combination leaves open how the object to be removed is designated and initially isolated, and Kim supplies both in one operation on the same kind of indoor augmented reality scene. Kutliroff already allows a user to select an object to be deleted but reaches that object's extent through a bounding box, which encloses background pixels along with the object and so carries those pixels into the surfaces Bell must afterward reconstruct. Kim's two-dimensional region division of the user-selected object produces the object's actual extent rather than a rectangle enclosing it, and Kim pairs it with the three-dimensional division the base combination already performs, so the result is predictable and improves the accuracy of the boundary handed to the later stages.
Regarding claim 3, Kutliroff as modified by Bell and Kim discloses augmented reality device of claim 1, wherein Kutliroff further discloses the one or more instructions are configured to, when executed by the at least one processor individually or collectively, further cause the augmented reality device to: obtain a plane equation based on the first plane and the second plane (para 38, “The plane fitting circuit 702 may be configured to scan the 3D reconstruction of the scene for planar surfaces. One method to accomplish this is to calculate normal vectors to the surfaces by scanning the depth maps and calculating the cross product of differences of neighboring depth pixels. The normal vectors are then clustered into groups based on spatial proximity and the values of the vectors. Next, a plane is fitted to each cluster. Specifically, the equation for the plane, in an x, y, z coordinate system, may be expressed as: ax+by+cz+d=0 where the constants a, b, c, d which define the plane may be calculated by a least-squares fit or other known techniques in light of the present disclosure.”; also para 39, “Floor/wall/ceiling determination circuit 704 may be configured to identify floors, walls and ceilings from the list of planes detected in the scene. In some embodiments, a gyroscope inertial sensor 304 is employed to detect the direction of gravity during scanning, and this data is stored during the scanning process. The direction of gravity may then be used to find the plane corresponding to the floor. Next, planar surfaces that are perpendicular to the floor plane (within a margin of error) and cover a sufficiently large region of the scene are identified as walls.”; also, prior art uses a method to calculate planar surfaces and such calculations can also be made on floor/wall/ceiling ), plane equation (para 38, “The plane fitting circuit 702 may be configured to scan the 3D reconstruction of the scene for planar surfaces. One method to accomplish this is to calculate normal vectors to the surfaces by scanning the depth maps and calculating the cross product of differences of neighboring depth pixels. The normal vectors are then clustered into groups based on spatial proximity and the values of the vectors. Next, a plane is fitted to each cluster. Specifically, the equation for the plane, in an x, y, z coordinate system, may be expressed as: ax+by+cz+d=0 where the constants a, b, c, d which define the plane may be calculated by a least-squares fit or other known techniques in light of the present disclosure.”), 3D model shape of a virtual wall and a virtual floor, by extending the wall and the floor, identify the object located on the wall and the floor in the real world space from the spatial image, inpaint the area of the wall and the floor that is hidden by the object based on the spatial image, and generate the 3D model of the real world space by applying a texture of an inpainted image to the 3D model shape of the virtual wall and the virtual floor.
However, in a similar field of endeavor, Bell discloses obtain a 3D model shape of a virtual wall and a virtual floor by extending the wall and the floor (para 32, “Flat surfaces can include walls, floors, and/or ceilings associated with the captured 3D data. In one example, the identification component 104 can identify walls, floors and/or ceilings in captured 3D data by identifying planes within a mesh.”; also, para 40, “Missing data (e.g., holes) can be associated with more than one surface (e.g., holes may extend across multiple planes). As such, in response to a determination that missing data (e.g., a hole) is associated with more than one surface, the data generation component 106 can extend all surfaces adjacent to the missing data (e.g., based on parameterizations associated with the surfaces adjacent to the missing data) until each of the surfaces adjacent to the missing data intersect.”; also, para 66, “Walls and/or floors can be extended behind a moulding to intersect with a portal or with one another.”), identify the object located on the wall and the floor in the real world space from the spatial image (para 60, “For example, after the first identification component 202 identifies one or more surfaces, the second identification component 204 can identify one or more objects (e.g., connected components) associated with captured 3D data that are not identified as the one or more surfaces.”; also, para 60, “Examples of an object attached to a particular flat surface include, but are not limited to, a table attached to a floor, a painting attached to a wall, a bookcase attached to a floor and a wall, etc.”; also, para 64, “In an aspect, the second identification component 204 can generate object relation data (e.g., information such as that a bookshelf "is on" a wall and a floor, that a table "is on" a floor) to constrain and/or associate available positions of surfaces and/or objects.”), inpaint the area of the wall and the floor that is hidden by the object based on the spatial image (para 35, “Captured 3D data for an occluded area can be missing when an area on a surface (e.g., a wall, a floor, a ceiling, or another surface) is occluded by an object (e.g., furniture, items on a wall, etc.). Therefore, the data generation component 106 can predict and/or generate data (e.g., geometry data and/or texture data) for a particular surface when an object on the particular surface is moved or removed. The one or more hole-filling techniques can be implemented to generate geometry data and/or texture data for an occluded area.”; also, para 41, “Textures can be synthesized separately for each surface in response to a determination that missing data (e.g., a hole) is associated with more than one surface.”; also, para 41, “An appearance of the region surrounding the missing data (e.g., the hole) can be employed to constrain texture synthesis.”), and generate the 3D model of the real world space by applying a texture of an inpainted image to the 3D model shape of the virtual wall and the virtual floor (para 41, “Furthermore, the data generation component 106 can generate texture data for additional 3D data generated for missing data (e.g., a hole) associated with a flat surface and/or a non-flat surface. The data generation component 106 can sample a region surrounding missing data (e.g., a hole) via one or more texture synthesis algorithms”; also, para 41, “Furthermore, texture for a particular surface can be synthesized onto a portion of the missing data (e.g., the hole) corresponding to the particular surface.”; also, para 89, “In FIG. 8, the missing data 702 can be replaced with mesh data 802. For example, mesh data for the missing data 702 (e.g., the hole 702) can be "filled in". In FIG. 9, texture data can be added to the mesh data 802.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kutliroff's invention of an augmented reality device that fits a plane to each cluster of surface normals, expresses the equation for that plane, and identifies which of the resulting planes are floors and which are walls, with the features of Bell's invention of extending those surfaces until they intersect, identifying an object attached to a floor and a wall, and synthesizing texture separately for each surface onto the portion of the hole corresponding to it. The combination would have been obvious because Kutliroff stops where Bell begins. Kutliroff derives the plane equations and classifies the planes but then removes their points from the scene, so the wall and floor behind a removed object are never reconstructed, while Bell states that surface data is missing where a surface "is occluded by an object (e.g., furniture, items on a wall, etc.)" and supplies geometry and texture for that surface "when an object on the particular surface is moved or removed." A person of ordinary skill would have used the plane equations Kutliroff already computes as the input to Bell's extension, and the result is predictable because Bell's per-surface texture synthesis is designed for exactly the case Bell recites, missing data "associated with more than one surface."
Regarding claim 5, Kutliroff as modified by Bell and Kim discloses the augmented reality device of claim 3, wherein Kutliroff further discloses the one or more instructions are configured to, when executed by the at least one processor individually or collectively, further cause the augmented reality device to: identify the object based on at least one of depth information of the real world space and color information of the real world space (para 35, “The object detection/recognition circuit 504 may be configured to process the RGB image, and in some embodiments the associated depth map as well, along with the 3D reconstruction, to generate a list of any objects of interest recognized in the image.”; also, para 35, “The 3D Reconstruction or the depth maps may also be used as additional channels in a deep convolutional network for the purpose of the object recognition and detection.”), and distinguish the wall and the floor from the object (para 39, “Floor/wall/ceiling determination circuit 704 may be configured to identify floors, walls and ceilings from the list of planes detected in the scene. In some embodiments, a gyroscope inertial sensor 304 is employed to detect the direction of gravity during scanning, and this data is stored during the scanning process. The direction of gravity may then be used to find the plane corresponding to the floor. Next, planar surfaces that are perpendicular to the floor plane (within a margin of error) and cover a sufficiently large region of the scene are identified as walls.”; also, para 39, “Floor/wall/ceiling removal circuit 706 may be configured to remove points that are located within a relatively small distance from one of these planar surfaces from the scene. Connected components clustering circuit 708 may be configured to generate clusters of pixels that are not connected to other regions and classify them as individual, segmented objects of the scene.”).
Regarding claim 8, Kutliroff discloses a method, performed by an augmented reality device, for providing an augmented reality service to control an object in a real world space, the method comprising (para 26, “In some embodiments, the scene analysis system 130, along with an AR manipulation circuit 140, may be integrated into a mobile platform 102 along with a depth camera 110 and a display element 112. The mobile platform 102 may be a tablet or smartphone or other such similar device.”; also, para 80, “AR manipulation circuit 140 is configured to implement augmented reality operations including removal and insertion of objects and the generation of blueprints based on the scene analysis.”; also, para 46, “In some embodiments, the AR manipulation circuit 140 may be configured to allow a user to select one or more objects of interest in the scene to be deleted or to be replaced by a virtual object.”): recognizing a first plane comprising a wall and a second plane comprising a floor from a spatial image based on a photograph of the real world space using a camera (para 24, “In accordance with an embodiment, the 3D scene analysis techniques generate data that describes a real-world scene as imaged by a depth camera from various perspectives”; also, para 26, “Each 3D image frame may comprise a color image frame that provides color (e.g., red, green and blue or RGB) pixels, and a depth map frame that provides depth pixels.”; also, para 39, “Floor/wall/ceiling determination circuit 704 may be configured to identify floors, walls and ceilings from the list of planes detected in the scene. In some embodiments, a gyroscope inertial sensor 304 is employed to detect the direction of gravity during scanning, and this data is stored during the scanning process. The direction of gravity may then be used to find the plane corresponding to the floor. Next, planar surfaces that are perpendicular to the floor plane (within a margin of error) and cover a sufficiently large region of the scene are identified as walls.”); segmenting the object on the spatial image from the real world space based on a 3D model or 3D position information of the object using 3D segmentation (para 37, “The 3D segmentation circuit may be configured to segment out each of the objects detected, finding the points of the 3D reconstruction that correspond to the contours of the objects.”; also, para 40, “More specifically, the segmentation process generates, for each object of interest, a collection of 3D points, referred to as an object boundary set, representing the 3D boundary or contour of the object of interest.”; also, para 63, “For each pixel, the associated 3D position of the 2D pixel is computed, by sampling the associated depth map to obtain the associated depth pixel and projecting that depth pixel to a point in 3D space, at operation 1412.”). Kutliroff does not disclose generating a three-dimensional (3D) model of the real world space by extending the wall and extending the floor along the respective planes and performing 3D in-painting on an area of the extended wall and the extended floor that is hidden by an obstructive object; segmenting an object selected by a user input from the spatial image based on two- dimensional (2D) segmentation.
However, in a similar field of endeavor, Bell discloses generating a three-dimensional (3D) model of the real world space by extending the wall and extending the floor along the respective planes and performing 3D in-painting on an area of the extended wall and the extended floor that is hidden by an obstructive object (para 29, “For example, the at least one 3D reconstruction system can employ two-dimensional (2D) image data and/or depth data captured from one or more 3D sensors (e.g., laser scanners, structured light systems, time-of-flight systems, etc.) to automatically and/or semi-automatically generate a 3D model of an interior environment (e.g., architectural spaces, architectural structures, physical objects,”; also, para 32, “Flat surfaces can include walls, floors, and/or ceilings associated with the captured 3D data”; also, para 40, “As such, in response to a determination that missing data (e.g., a hole) is associated with more than one surface, the data generation component 106 can extend all surfaces adjacent to the missing data (e.g., based on parameterizations associated with the surfaces adjacent to the missing data) until each of the surfaces adjacent to the missing data intersect.”; also, para 66, “Walls and/or floors can be extended behind a moulding to intersect with a portal or with one another.”; also, para 35, “Captured 3D data for an occluded area can be missing when an area on a surface (e.g., a wall, a floor, a ceiling, or another surface) is occluded by an object (e.g., furniture, items on a wall, etc.). Therefore, the data generation component 106 can predict and/or generate data (e.g., geometry data and/or texture data) for a particular surface when an object on the particular surface is moved or removed.”; also, para 36, “For example, the data generation component 106 can determine whether an object or another surface blocked a portion of a particular surface during a 3D capture process. Furthermore, the data generation component 106 can determine that an edge of the particular surface that was occluded during the 3D capture process is an occlusion boundary.”; also, para 60, “Examples of an object attached to a particular flat surface include, but are not limited to, a table attached to a floor, a painting attached to a wall, a bookcase attached to a floor and a wall, etc.”; also, para 41, “Textures can be synthesized separately for each surface in response to a determination that missing data (e.g., a hole) is associated with more than one surface. Furthermore, texture for a particular surface can be synthesized onto a portion of the missing data (e.g., the hole) corresponding to the particular surface.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kutliroff's invention of a method performed by an augmented reality device that recognizes a wall plane and a floor plane from a camera image of a real world space and segments an object from that space using three-dimensional segmentation, with the features of Bell's invention of generating a three-dimensional model of an interior environment by extending each surface adjacent to a hole until those surfaces intersect and synthesizing texture separately for each such surface onto the portion of the hole corresponding to it. The combination would have been obvious because Bell addresses the condition Kutliroff's method creates and leaves unresolved. Kutliroff deletes a selected object and identifies the floor and wall planes only to remove their points from the clustering input, so the surfaces behind the deleted object are never restored. Bell states that surface data is missing where a surface "is occluded by an object (e.g., furniture, items on a wall, etc.)" and supplies geometry and texture for that surface "when an object on the particular surface is moved or removed," which is a direct answer to the state Kutliroff's method leaves the scene in. The result is predictable because the plane parameters Bell's extension consumes are already produced by Kutliroff's plane fitting step.
Kim discloses segmenting an object selected by a user input from the spatial image based on two- dimensional (2D) segmentation (para 40, “Here, the moving object is a part corresponding the real object to be moved in the 3D augmented reality and the real object to be moved is selected by a user. That is, the user selects the real object to be moved in the 3D augmented reality.”; also, para 40, “The object region division unit 130 divides the region of the moving object selected by the user in a 2D color image (texture information).”; also, para 46, “First, the object region division unit 130 divides the region of the moving object corresponding to the moving object selected by the user in the 3D augmented reality (S210). Here, the divided moving object may include the 3D information and the texture information.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, in which a method performed by an augmented reality device recognizes a wall plane and a floor plane, segments an object in three dimensions, and reconstructs and textures the surfaces that object occluded, with the features of Kim's invention of dividing the region of a user-selected object in a two-dimensional color image. The combination would have been obvious because the base combination designates the object to be removed but isolates it with a rectangle enclosing it, and a rectangle carries background pixels into the region the later stages must reconstruct. Kim performs the two-dimensional division on the same kind of user-selected real object in the same kind of augmented reality scene and yields the object's actual extent, so substituting it for the bounding box is a predictable improvement in the accuracy of the boundary passed to the three-dimensional stage.
Regarding claim 10, Kutliroff as modified by Bell and Kim discloses the method of claim 8, wherein Kutliroff further discloses the generating of the 3D model of the real world space comprises: obtaining a plane equation based on the first plane and the second plane (para 38, “Next, a plane is fitted to each cluster. Specifically, the equation for the plane, in an x, y, z coordinate system, may be expressed as: ax+by+cz+d=0 where the constants a, b, c, d which define the plane may be calculated by a least-squares fit or other known techniques in light of the present disclosure.”; also, para 39, “Floor/wall/ceiling determination circuit 704 may be configured to identify floors, walls and ceilings from the list of planes detected in the scene. In some embodiments, a gyroscope inertial sensor 304 is employed to detect the direction of gravity during scanning, and this data is stored during the scanning process. The direction of gravity may then be used to find the plane corresponding to the floor. Next, planar surfaces that are perpendicular to the floor plane (within a margin of error) and cover a sufficiently large region of the scene are identified as walls.”; also, equation of first and second plane would be ax+by+cz+d=0 since when each plane can be determined); plane equation (para 38, “Next, a plane is fitted to each cluster. Specifically, the equation for the plane, in an x, y, z coordinate system, may be expressed as: ax+by+cz+d=0 where the constants a, b, c, d which define the plane may be calculated by a least-squares fit or other known techniques in light of the present disclosure.”); 3D model shape of a virtual wall and a virtual floor, by extending the wall and the floor, identifying the object located on the wall and the floor in the real world space from the spatial image; inpainting the area of the wall and the floor that is hidden by the object based on the spatial image; and generating the 3D model of the real world space by applying a texture of an inpainted image to the 3D model shape of the virtual wall and the virtual floor.
However, in a similar field of endeavor, Bell discloses obtaining a 3D model shape of a virtual wall and a virtual floor, by extending the wall and the floor (para 32, “Flat surfaces can include walls, floors, and/or ceilings associated with the captured 3D data. In one example, the identification component 104 can identify walls, floors and/or ceilings in captured 3D data by identifying planes within a mesh.”; also, para 40, “As such, in response to a determination that missing data (e.g., a hole) is associated with more than one surface, the data generation component 106 can extend all surfaces adjacent to the missing data (e.g., based on parameterizations associated with the surfaces adjacent to the missing data) until each of the surfaces adjacent to the missing data intersect.”; also, para 66, “Walls and/or floors can be extended behind a moulding to intersect with a portal or with one another.”), identifying the object located on the wall and the floor in the real world space from the spatial image (para 60, “For example, after the first identification component 202 identifies one or more surfaces, the second identification component 204 can identify one or more objects (e.g., connected components) associated with captured 3D data that are not identified as the one or more surfaces.”; also, para 60, “Examples of an object attached to a particular flat surface include, but are not limited to, a table attached to a floor, a painting attached to a wall, a bookcase attached to a floor and a wall, etc.”; also, para 64, “In an aspect, the second identification component 204 can generate object relation data (e.g., information such as that a bookshelf "is on" a wall and a floor, that a table "is on" a floor) to constrain and/or associate available positions of surfaces and/or objects.”); inpainting the area of the wall and the floor that is hidden by the object based on the spatial image (para 35, “Captured 3D data for an occluded area can be missing when an area on a surface (e.g., a wall, a floor, a ceiling, or another surface) is occluded by an object (e.g., furniture, items on a wall, etc.). Therefore, the data generation component 106 can predict and/or generate data (e.g., geometry data and/or texture data) for a particular surface when an object on the particular surface is moved or removed.”; also, para 41, “Textures can be synthesized separately for each surface in response to a determination that missing data (e.g., a hole) is associated with more than one surface.”; also, para 41, “An appearance of the region surrounding the missing data (e.g., the hole) can be employed to constrain texture synthesis.”); and generating the 3D model of the real world space by applying a texture of an inpainted image to the 3D model shape of the virtual wall and the virtual floor (para 41, “The data generation component 106 can sample a region surrounding missing data (e.g., a hole) via one or more texture synthesis algorithms”; also, para 41, “Furthermore, texture for a particular surface can be synthesized onto a portion of the missing data (e.g., the hole) corresponding to the particular surface.”; also, para 89, “In FIG. 8, the missing data 702 can be replaced with mesh data 802. For example, mesh data for the missing data 702 (e.g., the hole 702) can be "filled in". In FIG. 9, texture data can be added to the mesh data 802.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kutliroff's invention of a method that fits a plane to each cluster of surface normals, expresses the equation for that plane, and identifies which planes are floors and which are walls, with the features of Bell's invention of extending those surfaces until they intersect, identifying an object attached to a floor and a wall, and synthesizing texture separately for each surface onto the portion of the hole corresponding to it. The combination would have been obvious because Kutliroff's method derives the plane equations and then discards the plane points, leaving the wall and floor behind a removed object unreconstructed, and Bell addresses precisely that condition, supplying geometry and texture for a surface "when an object on the particular surface is moved or removed." The plane equations Kutliroff already computes are the input Bell's extension consumes, so the result is predictable.
Regarding claim 12, Kutliroff as modified by Bell and Kim discloses the method of claim 10, wherein Kutliroff further discloses the identifying of the object located on the wall and the floor in the real world space comprises: identifying the object based on at least one of depth information of the real world space and color information of the real world space (para 35, “The object detection/recognition circuit 504 may be configured to process the RGB image, and in some embodiments the associated depth map as well, along with the 3D reconstruction, to generate a list of any objects of interest recognized in the image.”; also, para 35, “The 3D Reconstruction or the depth maps may also be used as additional channels in a deep convolutional network for the purpose of the object recognition and detection.”); and distinguishing the wall and the floor from the object (para 39, “Floor/wall/ceiling determination circuit 704 may be configured to identify floors, walls and ceilings from the list of planes detected in the scene. In some embodiments, a gyroscope inertial sensor 304 is employed to detect the direction of gravity during scanning, and this data is stored during the scanning process. The direction of gravity may then be used to find the plane corresponding to the floor. Next, planar surfaces that are perpendicular to the floor plane (within a margin of error) and cover a sufficiently large region of the scene are identified as walls.”; also, para 39, “Floor/wall/ceiling removal circuit 706 may be configured to remove points that are located within a relatively small distance from one of these planar surfaces from the scene. Connected components clustering circuit 708 may be configured to generate clusters of pixels that are not connected to other regions and classify them as individual, segmented objects of the scene.”).
Regarding claim 15, Kutliroff discloses a non-transitory computer-readable storage medium storing one or more instructions (para 85, “For example in one embodiment at least one non-transitory computer readable storage medium has instructions encoded thereon that, when executed by one or more processors, cause one or more of the methodologies for generating 3D object image variations, disclosed herein, to be implemented.”; also, para 86, “The aforementioned non-transitory computer readable medium may be any suitable medium for storing digital information, such as a hard drive, a server, a flash memory, and/or random access memory (RAM), or a combination of memories.”), wherein one or more instructions, when executed by one or more processors, cause the one or more processors to (para 87, “Some embodiments may be implemented, for example, using a machine readable medium or article which may store an instruction or a set of instructions that, if executed by a machine, may cause the machine to perform a method and/or operations in accordance with the embodiments.”; also, para 89, “The circuitry may include a processor and/or controller configured to execute one or more instructions to perform one or more operations described herein.”): identify a first plane comprising a wall and a second plane comprising a floor from the spatial image (para 38, “The plane fitting circuit 702 may be configured to scan the 3D reconstruction of the scene for planar surfaces. One method to accomplish this is to calculate normal vectors to the surfaces by scanning the depth maps and calculating the cross product of differences of neighboring depth pixels.”; also, para 39, “Floor/wall/ceiling determination circuit 704 may be configured to identify floors, walls and ceilings from the list of planes detected in the scene. In some embodiments, a gyroscope inertial sensor 304 is employed to detect the direction of gravity during scanning, and this data is stored during the scanning process. The direction of gravity may then be used to find the plane corresponding to the floor. Next, planar surfaces that are perpendicular to the floor plane (within a margin of error) and cover a sufficiently large region of the scene are identified as walls.”); ; and segment the object on the spatial image from the real world space based on a 3D model or 3D position information of the object using 3D segmentation (para 27, “The 3D segmentation circuit 136 may be configured to generate an estimate of 3D boundaries of objects of interest in the scene based on 3D images obtained from a number of poses of a depth camera. The estimated boundaries may be expressed as a set of 3D pixels associated with the boundary.”; also, para 37, “The 3D segmentation circuit may be configured to segment out each of the objects detected, finding the points of the 3D reconstruction that correspond to the contours of the objects.”). Kutliroff does not disclose generate a three-dimensional (3D) model of the real world space by extending the wall and extending the floor along their respective planes and performing 3D inpainting on an area of the extended wall and the extended floor that is hidden by an obstructive object; segment an object selected by a user input from the spatial image based on two- dimensional (2D) segmentation.
However, in a similar field of endeavor, Bell discloses generate a three-dimensional (3D) model of the real world space by extending the wall and extending the floor along their respective planes and performing 3D inpainting on an area of the extended wall and the extended floor that is hidden by an obstructive object (para 29, “For example, the at least one 3D reconstruction system can employ two-dimensional (2D) image data and/or depth data captured from one or more 3D sensors (e.g., laser scanners, structured light systems, time-of-flight systems, etc.) to automatically and/or semi-automatically generate a 3D model of an interior environment (e.g., architectural spaces, architectural structures, physical objects,”; also, para 32, “Flat surfaces can include walls, floors, and/or ceilings associated with the captured 3D data.”; also, para 40, “As such, in response to a determination that missing data (e.g., a hole) is associated with more than one surface, the data generation component 106 can extend all surfaces adjacent to the missing data (e.g., based on parameterizations associated with the surfaces adjacent to the missing data) until each of the surfaces adjacent to the missing data intersect.”; also, para 66, “Walls and/or floors can be extended behind a moulding to intersect with a portal or with one another.”; also, para 35, “Captured 3D data for an occluded area can be missing when an area on a surface (e.g., a wall, a floor, a ceiling, or another surface) is occluded by an object (e.g., furniture, items on a wall, etc.). Therefore, the data generation component 106 can predict and/or generate data (e.g., geometry data and/or texture data) for a particular surface when an object on the particular surface is moved or removed.”; also, para 36, “For example, the data generation component 106 can determine whether an object or another surface blocked a portion of a particular surface during a 3D capture process. Furthermore, the data generation component 106 can determine that an edge of the particular surface that was occluded during the 3D capture process is an occlusion boundary.”; also, para 60, “Examples of an object attached to a particular flat surface include, but are not limited to, a table attached to a floor, a painting attached to a wall, a bookcase attached to a floor and a wall, etc.”; also, para 41, “An appearance of the region surrounding the missing data (e.g., the hole) can be employed to constrain texture synthesis. Textures can be synthesized separately for each surface in response to a determination that missing data (e.g., a hole) is associated with more than one surface. Furthermore, texture for a particular surface can be synthesized onto a portion of the missing data (e.g., the hole) corresponding to the particular surface.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Kutliroff's invention of a non-transitory computer-readable storage medium whose instructions cause one or more processors to identify a wall plane and a floor plane and to segment an object from a real world space using three-dimensional segmentation, with the features of Bell's invention of generating a three-dimensional model of an interior environment by extending each surface adjacent to a hole until those surfaces intersect and synthesizing texture separately for each such surface onto the portion of the hole corresponding to it. The combination would have been obvious because Bell's stated occasion for supplying surface data is the condition Kutliroff's instructions produce, namely a surface that "is occluded by an object (e.g., furniture, items on a wall, etc.)" where the object "is moved or removed." Kutliroff identifies the floor and wall planes and then discards their points, so nothing in Kutliroff restores what stood behind the removed object, and a person of ordinary skill would have looked to Bell to complete the operation. The result is predictable because Bell's extension consumes the same fitted plane data Kutliroff already computes.
Kim discloses segment an object selected by a user input from the spatial image based on two- dimensional (2D) segmentation (para 40, “That is, the user selects the real object to be moved in the 3D augmented reality.”; also, para 40, “The object region division unit 130 divides the region of the moving object selected by the user in a 2D color image (texture information). In addition, the object region division unit 130 performs separation of a foreground and the background on 2D and 3D by using a 2D and 3D relationship to divide the region of the moving object even on the 3D.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, in which the instructions on a non-transitory medium cause one or more processors to identify a wall plane and a floor plane, segment an object in three dimensions, and reconstruct and texture the surfaces the object occluded, with the features of Kim's invention of dividing the region of a user-selected object in a two-dimensional color image. The combination would have been obvious because the base combination isolates the designated object with a rectangle enclosing it, which sweeps background pixels into the region the reconstruction stage must then repair, and Kim performs the two-dimensional region division on a user-selected real object in the same augmented reality setting to yield that object's actual extent. Substituting Kim's region division for the enclosing rectangle produces a predictably cleaner boundary for every stage that follows.
Claim(s) 2 and 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kutliroff et al. (U.S. Pub. No. 20170243352) as modified by Bell et al. (U.S. Pub. No. 20160055268) and Kim et al. (U.S. Pub. No. 20220343613), further in view of Jovanovic et al. (U.S. Pub. No. 20190051054).
Regarding claim 2, Kutliroff as modified by Bell and Kim discloses the augmented reality device of claim 1, wherein further discloses the one or more instructions are configured to, when executed by the at least one processor individually or collectively, further cause the augmented reality device to: orient the 3D model based on positions of the wall and the floor in the real world space, and render an area in the 3D model where the object is segmented based on the 3D segmentation.
However, in a similar field of endeavor, Jovanovic discloses orient the 3D model based on positions of the wall and the floor in the real world space (para 60, “In a non-limiting example, the 3D model is loaded, or reloaded, into the AR application to create, or recreate, a virtual model of the space in alignment with real-world floors, walls, doors, openings, and/or ceilings”; also, para 60, “In some embodiments, the 3D model is stored either on the device or downloaded from a server and regenerated and re-aligned as needed once the user is back in the captured space and/or enters a new space where they want to visualize the original model.”; also, Since when 3D models are loaded or reloaded in an AR application, the virtual model is in alignment with the real world).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, in which an augmented reality device identifies a wall plane and a floor plane, generates a three-dimensional model of the space by extending those surfaces and inpainting the area the obstructive object hid, and segments a user-selected object in two and then three dimensions, with the features of Jovanovic's invention of loading a three-dimensional model of a space into an augmented reality application in alignment with the real-world floors and walls. The combination would have been obvious because the base combination produces a model of the room that is only useful once it sits where the room actually is. Kutliroff already computes the floor plane and the wall planes and already tracks camera pose, and Jovanovic states the purpose of the alignment step in terms, that the model is re-aligned "once the user is back in the captured space" so that it can be visualized against that space. A person of ordinary skill building the base combination's augmented reality service would have looked to Jovanovic for the step that puts the reconstructed model in register with the real floors and walls, and the result is predictable because the floor and wall positions the alignment consumes are the ones Kutliroff has already determined.
Kim discloses render an area in the 3D model where the object is segmented based on the 3D segmentation (para 42, “Meanwhile, the object region background synthesis unit 170 deletes a region corresponding to the moving object divided by the object region division unit 130 from the background, and performs inpainting for the deleted region by using surrounding background information.”; also, para 43, “The synthesis background rendering unit 180 performs rendering for the inpainted part with the surrounding background information by the object region background synthesis unit 170.”; also, para 40, “In addition, the object region division unit 130 performs separation of a foreground and the background on 2D and 3D by using a 2D and 3D relationship to divide the region of the moving object even on the 3D.”; also, para 39, “The environment reconstruction thread unit 110 performs 3D reconstruction of the real environment.”, also, If an object gets removed, the impanting of the deleted object will render an area of the 3D Model).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, further in view of Jovanovic, in which the reconstructed model of the space is placed in alignment with the real floor and walls, with the features of Kim's invention of rendering the region left where the divided object stood. The combination would have been obvious because aligning a reconstructed model to the real room accomplishes nothing visible until the region the removed object occupied is actually drawn from that model, and Kim supplies exactly that step with a dedicated rendering unit operating on the region its own division stage produced. A person of ordinary skill would have applied Kim's rendering of the vacated region to the aligned model of the base combination, and the result is predictable because Kim performs the division that defines that region "on 2D and 3D" in a three-dimensional reconstruction of the real environment, which is the same kind of scene the base combination reconstructs.
Regarding claim 9, Kutliroff as modified by Bell and Kim discloses the method of claim 8, further comprising: orienting the 3D model based on positions of the wall and the floor in the real world space; and rendering an area in the 3D model where the object is segmented based on the 3D segmentation.
However, in a similar field of endeavor, Jovanovic discloses orienting the 3D model based on positions of the wall and the floor in the real world space (para 60, “In a non-limiting example, the 3D model is loaded, or reloaded, into the AR application to create, or recreate, a virtual model of the space in alignment with real-world floors, walls, doors, openings, and/or ceilings”; also, para 60, “In some embodiments, the 3D model is stored either on the device or downloaded from a server and regenerated and re-aligned as needed once the user is back in the captured space and/or enters a new space where they want to visualize the original model.”; also, Since when 3D models are loaded or reloaded in an AR application, the virtual model is in alignment with the real world).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, in which a method performed by an augmented reality device recognizes a wall plane and a floor plane, generates a three-dimensional model of the space by extending those surfaces and inpainting the hidden area, and segments a user-selected object in two and then three dimensions, with the features of Jovanovic's invention of loading a three-dimensional model of a space into an augmented reality application in alignment with the real-world floors and walls. The combination would have been obvious because a reconstructed model of a room serves the method's stated purpose only when it is registered to that room, Jovanovic states that the stored model is regenerated and re-aligned once the user is back in the captured space, and the floor and wall positions that alignment consumes are the ones the base combination has already recognized.
Kim discloses rendering an area in the 3D model where the object is segmented based on the 3D segmentation (para 42, “Meanwhile, the object region background synthesis unit 170 deletes a region corresponding to the moving object divided by the object region division unit 130 from the background, and performs inpainting for the deleted region by using surrounding background information.”; also, para 43, “The synthesis background rendering unit 180 performs rendering for the inpainted part with the surrounding background information by the object region background synthesis unit 170.”; also, para 40, “In addition, the object region division unit 130 performs separation of a foreground and the background on 2D and 3D by using a 2D and 3D relationship to divide the region of the moving object even on the 3D.”; also, para 39, “The environment reconstruction thread unit 110 performs 3D reconstruction of the real environment.”, also, If an object gets removed, the impanting of the deleted object will render an area of the 3D Model).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, further in view of Jovanovic, in which the reconstructed model of the space is aligned to the real floor and walls, with the features of Kim's invention of rendering the region left where the divided object stood. The combination would have been obvious because the aligned model produces no visible result until the vacated region is drawn from it, and Kim supplies a dedicated rendering unit that operates on the region produced by its own division of the object on 2D and 3D. The result is predictable because both the base combination and Kim operate on a three-dimensional reconstruction of a real indoor environment.
Claim(s) 4 and 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kutliroff et al. (U.S. Pub. No. 20170243352) as modified by Bell et al. (U.S. Pub. No. 20160055268) and Kim et al. (U.S. Pub. No. 20220343613), further in view of Yamamoto et al. (U.S. Pub. No. 20160092608).
Regarding claim 4, Kutliroff as modified by Bell and Kim discloses the augmented reality device of claim 3, wherein the one or more instructions are configured to, when executed by the at least one processor individually or collectively, further cause the augmented reality device to: extract an intersection line where the extended wall and the extended floor meet each other, determine the first plane and the second plane based on the intersection line, obtain vertex coordinates of the first plane and the second plane, and generate the 3D model shape of the virtual wall and the virtual floor based on the vertex coordinates.
However, in a similar field of endeavor, Bell discloses extract an intersection line where the extended wall and the extended floor meet each other (para 75, “Planes included in a floorplan can be extended a certain distance (e.g., to intersect past a moulding).”; also, para 76, “A floorplan generated by the floorplan component 302 can comprise a series of lines in 3D space which represent intersections of walls and/or floors, outlines of doorways and/or windows, edges of steps, outlines of other objects of interest (e.g., mirrors, paintings, fireplaces, etc.).”; also, para 66, “Walls and/or floors can be extended behind a moulding to intersect with a portal or with one another.”), determine the first plane and the second plane based on the intersection line (para 55, “In an aspect, the first identification component 202 can employ an intersection line of planes from two identified surfaces of different orientations as a boundary for each identified plane.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, in which an augmented reality device obtains a plane equation for the wall plane and the floor plane and extends those surfaces to build a model shape whose occluded region is inpainted, with the features of Bell's invention of a floorplan whose extended planes yield lines in three-dimensional space representing intersections of walls and floors, and of using an intersection line of planes from two identified surfaces of different orientations as a boundary for each identified plane. The combination would have been obvious because the base combination extends two surfaces of different orientations until they meet without saying what is done at the meeting, and Bell states in consecutive paragraphs of the same component that the extended planes produce intersection lines and that such a line delimits each of the two planes it separates. A person of ordinary skill would have taken that boundary from the extension the base combination already performs, and the result is predictable because the wall and the floor are the perpendicular pair Bell names.
Yamamoto discloses obtain vertex coordinates of the first plane and the second plane (para 53, “From the aforementioned respective knowledge, it is recognized that a model of the three-dimensional structure 30 is represented by vertexes as corners of planes forming the three-dimensional structure 30 and line segments each of which connects two vertexes.”; also, para 88, “The vertex extractor 13 deals respective three equations describing the selected three faces 3 as simultaneous equations and calculates solutions of the equations. The calculated solutions are three-dimensional coordinates of a vertex shared among the three faces 3. By repeating the aforementioned process until every vertex of the three-dimensional structure 30 is obtained, three-dimensional coordinates of every vertex of the three-dimensional structure 30 can be calculated.”; also, para 101, “Even in the case of the room having the aforementioned shape, the vertex extractor 13 can extract a vertex shared among three adjoining faces of the floor 31, the ceiling 32 and the walls 33.”) and generate the 3D model shape of the virtual wall and the virtual floor based on the vertex coordinates (para 95, “The model producer 14 is configured to produce a model, representing the three-dimensional structure 30, formed of a set of boundary lines each of which is a straight line connecting vertexes obtained by the vertex extractor 13 and a straight line along an intersection line between two adjoining faces.”; also, para 55, “That is, the model producer 14 is configured to store a wire frame model in which coordinates of each vertex of the model are associated with a boundary line including the vertex as an end point”; also, para 99, “The model produced by the model producer 14 does not therefore include objects except for the three-dimensional structure 30, openings of the three-dimensional structure 30 and the like.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, in which the extended wall and extended floor meet at an intersection line that bounds each of them, with the features of Yamamoto's invention of calculating the three-dimensional coordinates of the vertexes shared among the floor and the wall faces and producing from those coordinates a model of the room structure that excludes the objects in it. The combination would have been obvious because Yamamoto is directed to the same difficulty the base combination confronts and says so: "when part of the faces 3 is hidden behind furniture, facilities or the like as seen from the measurement device 20, the hidden part cannot be measured," and Yamamoto answers it by solving the plane equations of adjoining faces for their shared corner rather than by measuring the occluded region. A person of ordinary skill holding the base combination's plane equations and intersection line would have looked to Yamamoto to turn them into the corner coordinates that define the surfaces, and the result is predictable because Yamamoto derives those coordinates from the same plane equations the base combination has already obtained.
Regarding claim 11, Kutliroff as modified by Bell and Kim discloses the method of claim 10, wherein the generating of the 3D model shape of the virtual wall and the virtual floor comprises: extracting an intersection line where the extended wall and the extended floor meet each other; determine the first plane and the second plane based on the intersection line; obtaining vertex coordinates based on the first plane and the second plane; and generating the 3D model shape of the virtual wall and the virtual floor based on the vertex coordinates.
However, in a similar field of endeavor, Bell discloses extracting an intersection line where the extended wall and the extended floor meet each other (para 75, “Planes included in a floorplan can be extended a certain distance (e.g., to intersect past a moulding).”; also, para 76, “A floorplan generated by the floorplan component 302 can comprise a series of lines in 3D space which represent intersections of walls and/or floors, outlines of doorways and/or windows, edges of steps, outlines of other objects of interest (e.g., mirrors, paintings, fireplaces, etc.).”; also, para 66, “Walls and/or floors can be extended behind a moulding to intersect with a portal or with one another.”); determine the first plane and the second plane based on the intersection line (para 55, “In an aspect, the first identification component 202 can employ an intersection line of planes from two identified surfaces of different orientations as a boundary for each identified plane.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, in which a method obtains a plane equation for the wall and the floor and extends those surfaces to build a model shape whose occluded region is inpainted, with the features of Bell's invention of a floorplan whose extended planes yield lines in three-dimensional space representing intersections of walls and floors, and of using an intersection line of planes from two identified surfaces of different orientations as a boundary for each identified plane. The combination would have been obvious because the base method extends two perpendicular surfaces until they meet and does nothing with the meeting, while Bell states in consecutive paragraphs of one component that extended planes produce intersection lines and that such a line delimits each of the two planes. The result is predictable because the boundary is computed from the plane geometry already in hand.
Yamamoto discloses obtaining vertex coordinates based on the first plane and the second plane (para 88, “The vertex extractor 13 deals respective three equations describing the selected three faces 3 as simultaneous equations and calculates solutions of the equations. The calculated solutions are three-dimensional coordinates of a vertex shared among the three faces 3.”; also, para 54, “That is, the model producer 14 defines two vertexes on each intersection line shared between two adjoining faces 3 as end points, and defines a line segment between the two end points as a boundary line of the two adjoining faces 3.”; also, para 43, “In the embodiment, since the room has the three-dimensional structure 30, the three-dimensional structure 30 is formed of a floor (face) 31, a ceiling (face) 32 and walls (wall faces) 33 as shown in FIG. 3.”); and generating the 3D model shape of the virtual wall and the virtual floor based on the vertex coordinates (para 95, “The model producer 14 is configured to produce a model, representing the three-dimensional structure 30, formed of a set of boundary lines each of which is a straight line connecting vertexes obtained by the vertex extractor 13 and a straight line along an intersection line between two adjoining faces.”, also, para 95 “The model producer 14 is also configured to store, as model information, information on three-dimensional coordinates of each vertex and each vertex pair connected by a corresponding boundary line.”; also, para 99, “The model produced by the model producer 14 does not therefore include objects except for the three-dimensional structure 30, openings of the three-dimensional structure 30 and the like.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, in which the extended wall and extended floor meet at an intersection line that bounds each of them, with the features of Yamamoto's invention of calculating the three-dimensional coordinates of the vertexes shared among the floor and the wall faces from the plane equations of those faces and producing from those coordinates a model of the room structure that excludes the objects in it. The combination would have been obvious because Yamamoto confronts the same occlusion problem the base method confronts, stating that "when part of the faces 3 is hidden behind furniture, facilities or the like as seen from the measurement device 20, the hidden part cannot be measured," and resolves it by computing the corner from the plane equations instead of measuring it. The result is predictable because those equations are already produced by the base method.
Claim(s) 6 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kutliroff et al. (U.S. Pub. No. 20170243352) as modified by Bell et al. (U.S. Pub. No. 20160055268) and Kim et al. (U.S. Pub. No. 20220343613), further in view of Singh (U.S. Pub. No. 20160275723) and Khazov et al. (U.S. Pub. No. 20200380779).
Regarding claim 6, Kutliroff as modified by Bell and Kim discloses the augmented reality device of claim 1, wherein Kutliroff further discloses the one or more instructions are configured to, when executed by the at least one processor individually or collectively, further cause the augmented reality device to: segment the object from the spatial image based on the 3D position coordinate value based on 3D segmentation (para 58, “FIG. 14 is a detailed block diagram of another embodiment of 3D segmentation circuit 136. The segmentation circuit 136 is shown to include the camera pose calculation circuit 302 and object detection circuit 134 as previously described, as well as an object boundary set matching circuit 1410 and an object boundary set creation circuit 1420.”; also, para 63, “For each unmatched detected object of interest, the 2D bounding box containing the object is scanned to analyze each pixel within the bounding box. For each pixel, the associated 3D position of the 2D pixel is computed, by sampling the associated depth map to obtain the associated depth pixel and projecting that depth pixel to a point in 3D space, at operation 1412. A ray is then generated which extends from the camera to the location of the projected point in 3D space at operation 1414. The point at this 3D position is included in the object boundary set at operation 1416.”). Kutliroff does not disclose determine whether a 3D model of the object is pre-stored in the memory, and based on determining that the 3D model of the object is stored, orient the 3D model to overlap a 2D segmentation outlier on the spatial image based on a direction of the 3D model of the object, obtain a 3D position coordinate value of the object from the 3D model.
However, in a similar field of endeavor, Singh discloses determine whether a 3D model of the object is pre-stored in the memory (para 58, “Further, the general module 124 may determine if the 3D model and contextual objects are available in the memory 208 corresponding to the scanned image of the user or not.”; also, para 39, “For example, if a user selects/scans an initial input object then the general module 124 of the system 100 may determine whether a 3D model or an AR scene or one or more contextual objects related to the initial input object exists in the database 114.”; also, para 70, “Herein, the 3D model may correspond to the scanned object (i.e., chair).”), based on determining that the 3D model of the object is stored (para 69, “The method 400 may proceed to the step 406 (as depicted through 'Yes' pointer from the step 404 to the step 406) if at least one of the 3D model and AR scene (related to the scanned object) is available in the database/memory.”; also, para 70, “Herein, the 3D model may correspond to the scanned object (i.e., chair).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, in which an augmented reality device segments a user-selected object in two and then three dimensions from a scene whose wall and floor have been reconstructed, with the features of Singh's invention of determining whether a three-dimensional model corresponding to the selected object is already available in memory and, on determining that such a model is available, proceeding on the basis of that determination to the step that uses the stored model. The combination would have been obvious because the base combination must reconstruct the geometry of every selected object from the current scan even when that object is a common item the device has modeled before, and Singh answers that inefficiency directly by checking memory for a model corresponding to the scanned object and, when the answer is yes, advancing to the step that operates on that model rather than rebuilding it. A person of ordinary skill would have added the check and its affirmative branch ahead of the base combination's segmentation, and the result is predictable because the branch merely selects between an existing model and the reconstruction path the base combination already performs.
Khazov discloses orient the 3D model to overlap a 2D segmentation outlier on the spatial image based on a direction of the 3D model of the object (para 52, “With reference to FIG. 3, 3D model points 116, as determined based on dense point field 305 are translated into the 3D scene (using position and orientation information for 3D model 115) and projected onto the image plane of each camera of camera array 101 corresponding to one of object mask images 113 (and input images 111) to provide projected 3D model points 306 projected image 118. Such projected 3D model points 306 are then compared with projected dilated object images (e.g., object mask images 113) to adjust the position and orientation information and determine a final position and orientation for 3D model 115.”; also, para 62, “As shown, projected 3D model points 901 (illustrated as white dots) fully or almost fully overlap with and are within 2D representation 511 (illustrated in grey).”; also, para 58, “R represents the rotation parameters for 3D model 115 to orient 3D model 115 in scene 110”), obtain a 3D position coordinate value of the object from the 3D model (para 31, “To locate the 3D model within the scene, the location (e.g., x, y, z coordinates) and orientation (e.g., yaw, pitch roll values) of the 3D model are determined.”; also, para 51, “In some embodiments, the projections (e.g., onto multiple image planes) of each of 3D model points 116 includes determination of a 3D location of each of 3D model points 116 in the 3D scene using an initial position and orientation 117 of 3D model 115 and projection from the 3D location onto image planes using projection matrices corresponding thereto.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, further in view of Singh, in which the device checks memory for a model corresponding to the user-selected object and already produces a two-dimensional segmentation of that object, with the features of Khazov's invention of adjusting the position and orientation information of a three-dimensional model until its projected points overlap the two-dimensional representation of the object, and determining the model's x, y and z coordinates. The combination would have been obvious because Singh's check produces a stored model that must then be placed against the object as it actually appears, and Khazov performs that placement using the one artifact the base combination already has, the two-dimensional segmentation of the selected object. A person of ordinary skill would have used Khazov's projection-and-overlap fit rather than devise another, and the result is predictable because the coordinates the fit yields substitute one known source of the object's three-dimensional position for the depth-derived source Kutliroff already uses to build the object boundary set.
Regarding claim 13, Kutliroff as modified by Bell and Kim discloses the method of claim 8, wherein Kutliroff further discloses the performing of the 3D segmentation comprises: object from the spatial image based on the 3D position coordinate value based on 3D segmentation (para 58, “FIG. 14 is a detailed block diagram of another embodiment of 3D segmentation circuit 136. The segmentation circuit 136 is shown to include the camera pose calculation circuit 302 and object detection circuit 134 as previously described, as well as an object boundary set matching circuit 1410 and an object boundary set creation circuit 1420.”; also, para 63, “For each pixel, the associated 3D position of the 2D pixel is computed, by sampling the associated depth map to obtain the associated depth pixel and projecting that depth pixel to a point in 3D space, at operation 1412. A ray is then generated which extends from the camera to the location of the projected point in 3D space at operation 1414. The point at this 3D position is included in the object boundary set at operation 1416.”). Kutliroff does not disclose determining whether a 3D model of the object is pre-stored; based on determining that the 3D model of the object is stored, orienting the 3D model to overlap a 2D segmentation outlier on the spatial image based on a direction of the 3D model; obtaining a 3D position coordinate value of the object from the 3D model.
However, in a similar field of endeavor, Singh discloses determining whether a 3D model of the object is pre-stored (para 58, “Further, the general module 124 may determine if the 3D model and contextual objects are available in the memory 208 corresponding to the scanned image of the user or not.”; also, para 69, “The method 400 may proceed to the step 406 (as depicted through 'Yes' pointer from the step 404 to the step 406) if at least one of the 3D model and AR scene (related to the scanned object) is available in the database/memory.”). based on determining that the 3D model of the object is stored (para 69, “The method 400 may proceed to the step 406 (as depicted through 'Yes' pointer from the step 404 to the step 406) if at least one of the 3D model and AR scene (related to the scanned object) is available in the database/memory.”; also, para 70, “Herein, the 3D model may correspond to the scanned object (i.e., chair).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, in which a method performed by an augmented reality device segments a user-selected object in two and then three dimensions from a reconstructed scene, with the features of Singh's invention of determining whether a three-dimensional model corresponding to the selected object is already available and, when it is available, proceeding on the basis of that determination to the step that uses the stored model. The combination would have been obvious because the base method reconstructs the geometry of each selected object from the current scan even for objects already modeled, and Singh's check answers that directly, branching on whether a model related to the scanned object is available and routing the available case onward to the step that operates on that model. The result is predictable because the branch selects between a stored model and the reconstruction the base method already performs.
Khazov discloses orienting the 3D model to overlap a 2D segmentation outlier on the spatial image based on a direction of the 3D model (para 52, “Such projected 3D model points 306 are then compared with projected dilated object images (e.g., object mask images 113) to adjust the position and orientation information and determine a final position and orientation for 3D model 115.”; also, para 62, “As shown, projected 3D model points 901 (illustrated as white dots) fully or almost fully overlap with and are within 2D representation 511 (illustrated in grey).”; also, para 58, “R represents the rotation parameters for 3D model 115 to orient 3D model 115 in scene 110”); obtaining a 3D position coordinate value of the object from the 3D model (para 31, “To locate the 3D model within the scene, the location (e.g., x, y, z coordinates) and orientation (e.g., yaw, pitch roll values) of the 3D model are determined.”; also, para 51, “In some embodiments, the projections (e.g., onto multiple image planes) of each of 3D model points 116 includes determination of a 3D location of each of 3D model points 116 in the 3D scene using an initial position and orientation 117 of 3D model 115 and projection from the 3D location onto image planes using projection matrices corresponding thereto.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, further in view of Singh, in which the method checks for a stored model of the user-selected object and already produces a two-dimensional segmentation of that object, with the features of Khazov's invention of adjusting a three-dimensional model's position and orientation until its projected points overlap the two-dimensional representation of the object, and determining the model's x, y and z coordinates. The combination would have been obvious because a retrieved model is of no use until it is placed against the object as imaged, and Khazov performs that placement against the very artifact the base method already produces. The result is predictable because the coordinates the fit yields substitute one known source of the object's three-dimensional position for the depth-derived source the base method already uses.
Claim(s) 7 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kutliroff et al. (U.S. Pub. No. 20170243352) as modified by Bell et al. (U.S. Pub. No. 20160055268) and Kim et al. (U.S. Pub. No. 20220343613), further in view of Singh (U.S. Pub. No. 20160275723) and Zhou et al. (U.S. Pub. No. 20220301205).
Regarding claim 7, Kutliroff as modified by Bell and Kim discloses the augmented reality device of claim 1, wherein Kutliroff further discloses the one or more instructions are configured to, when executed by the at least one processor individually or collectively, further cause the augmented reality device to: feature point, and 3D position coordinate values of pixels of the object based on the spatial image (para 42, “Feature detection circuit 902 may be configured to detect features in both the detected/segmented objects and the source objects (models). These features may include, for example, 3D corners or any other suitable distinctive features of the object. In some embodiments, the RGB image frames are stored and mapped to the 3D reconstruction, enabling the use of 2D feature detection techniques such as Scale Invariant Feature Transform (SIFT) detection and Speeded-Up Robust Feature (SURF) detection.”; also, para 63, “For each unmatched detected object of interest, the 2D bounding box containing the object is scanned to analyze each pixel within the bounding box. For each pixel, the associated 3D position of the 2D pixel is computed, by sampling the associated depth map to obtain the associated depth pixel and projecting that depth pixel to a point in 3D space, at operation 1412.”), and determine whether a 3D model of the object is stored in the memory, based on determining that the 3D model of the object is not stored, obtain an edge, segment the object from the spatial image based on the 3D segmentation and 3D vertex modeling using at least one of the obtained edge, the feature point, and the 3D position coordinate values of the pixels of the object.
However, in a similar field of endeavor, Singh discloses determine whether a 3D model of the object is stored in the memory (para 58, “Further, the general module 124 may determine if the 3D model and contextual objects are available in the memory 208 corresponding to the scanned image of the user or not.”), based on determining that the 3D model of the object is not stored (para 58, “If the 3D model and the contextual objects are not available then the user may capture the input object through a camera, such as the camera 212.”; also, para 73, “Referring to step 404, if 3D model or AR scene related to the scanned object does not exist in the memory then at step 408, the user may be facilitated to generate a video stream or 2D images of the object.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, in which an augmented reality device segments a user-selected object in two and then three dimensions, with the features of Singh's invention of determining whether a three-dimensional model corresponding to the selected object is available in memory and, when it is not, capturing images of that object instead. The combination would have been obvious because the base combination reconstructs every selected object from the current scan without regard to whether it has been modeled before, and Singh supplies the branch that distinguishes the two situations and routes the not-available case to image capture of the object. The result is predictable because the not-available path leads to the same image-based reconstruction the base combination already performs.
Zhou discloses obtain an edge (para 55, “First, depth map generation system 102 (e.g., via depth completion component 108 and/or semantic mesh deformation component 110) can extract semantic segmentation masks from an image (e.g., an RGB image) to isolate the individual object instances.”; also, para 55, “Next, depth map generation system 102 (e.g., via depth completion component 108 and/or semantic mesh deformation component 110) can apply Canny edge detection on the mask images to extract the edges of the object instances.”), segment the object from the spatial image based on the 3D segmentation and 3D vertex modeling using at least one of the obtained edge, the feature point, and the 3D position coordinate values of the pixels of the object (para 68, “In accordance with one or more embodiments described herein, semantic mesh deformation component 110 can apply a semantic mesh reconstruction process to reconstruct a mesh for each object instance in an image (e.g., an RGB image) to calibrate such meshes independently rather than optimizing a single mesh reconstructed from the full depth map globally.”; also, para 70, “Given the camera intrinsic parameters of an augmented reality environment, semantic mesh deformation component 110 can calculate, for instance, the 3D coordinates of each depth sample of depth d following the projection rules.”; also, para 71, “To facilitate such mesh reconstruction, semantic mesh deformation component 110 can, for instance, create a mesh by assigning edge connections between nearby vertices”; also, para 71, “Diagram 500 illustrated in FIG. 5 depicts two semantic meshes that can be created by semantic mesh deformation component 110 by connecting the vertices within each point cloud group.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, further in view of Singh, in which the device branches to image-based reconstruction when no model of the selected object is stored and already computes a three-dimensional position for each pixel of that object, with the features of Zhou's invention of extracting the edges of the object instances by Canny edge detection on the segmentation masks and building a per-object mesh by connecting the vertices within each point cloud group. The combination would have been obvious because the not-stored branch leaves the device holding a set of three-dimensional points for the object and no model built from them, and Zhou consumes exactly that input, segmenting the depth data into per-object portions, computing the three-dimensional coordinates of each depth sample, and connecting those points into a mesh for the object instance. A person of ordinary skill working the not-stored branch would have looked to Zhou for the step that turns the object's points into an object model, and the result is predictable because both operate on per-pixel depth data captured in an augmented reality environment.
Regarding claim 14, Kutliroff as modified by Bell and Kim discloses the method of claim 8, wherein Kutliroff further discloses the segmenting the object on the spatial image from the real world space based on a 3D model or 3D position information of the object using 3D segmentation comprises: , a feature point, and 3D position coordinate values of pixels of the object based on the spatial image (para 42, “Feature detection circuit 902 may be configured to detect features in both the detected/segmented objects and the source objects (models). These features may include, for example, 3D corners or any other suitable distinctive features of the object. In some embodiments, the RGB image frames are stored and mapped to the 3D reconstruction, enabling the use of 2D feature detection techniques such as Scale Invariant Feature Transform (SIFT) detection and Speeded-Up Robust Feature (SURF) detection.”; also, para 63, “For each unmatched detected object of interest, the 2D bounding box containing the object is scanned to analyze each pixel within the bounding box. For each pixel, the associated 3D position of the 2D pixel is computed, by sampling the associated depth map to obtain the associated depth pixel and projecting that depth pixel to a point in 3D space, at operation 1412.”); and . Kutliroff does not disclose determining whether a 3D model of the object is pre-stored; based on determining that the 3D model of the object is not stored, obtaining an edge, segmenting the object from the spatial image based on the 3D segmentation and 3D vertex modeling using at least one of the obtained edge, the feature point, and the 3D position coordinate values of the pixels of the object.
However, in a similar field of endeavor, Singh discloses determining whether a 3D model of the object is pre-stored (para 58, “Further, the general module 124 may determine if the 3D model and contextual objects are available in the memory 208 corresponding to the scanned image of the user or not.”); based on determining that the 3D model of the object is not stored (para 73, “Referring to step 404, if 3D model or AR scene related to the scanned object does not exist in the memory then at step 408, the user may be facilitated to generate a video stream or 2D images of the object.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, in which a method performed by an augmented reality device segments a user-selected object in two and then three dimensions, with the features of Singh's invention of determining whether a three-dimensional model corresponding to the selected object is available and, when it is not, generating images of that object instead. The combination would have been obvious because the base method reconstructs every selected object from the current scan regardless of whether it has been modeled before, and Singh supplies the branch and routes the not-available case to image capture of the object, which is the input the base method already consumes.
Zhou discloses obtaining an edge (para 55, “Next, depth map generation system 102 (e.g., via depth completion component 108 and/or semantic mesh deformation component 110) can apply Canny edge detection on the mask images to extract the edges of the object instances.”; also, para 55, “First, depth map generation system 102 (e.g., via depth completion component 108 and/or semantic mesh deformation component 110) can extract semantic segmentation masks from an image (e.g., an RGB image) to isolate the individual object instances.”), segmenting the object from the spatial image based on the 3D segmentation and 3D vertex modeling using at least one of the obtained edge, the feature point, and the 3D position coordinate values of the pixels of the object (para 68, “In accordance with one or more embodiments described herein, semantic mesh deformation component 110 can apply a semantic mesh reconstruction process to reconstruct a mesh for each object instance in an image (e.g., an RGB image) to calibrate such meshes independently rather than optimizing a single mesh reconstructed from the full depth map globally.”; also, para 70, “Given the camera intrinsic parameters of an augmented reality environment, semantic mesh deformation component 110 can calculate, for instance, the 3D coordinates of each depth sample of depth d following the projection rules.”; also, para 71, “Diagram 500 illustrated in FIG. 5 depicts two semantic meshes that can be created by semantic mesh deformation component 110 by connecting the vertices within each point cloud group.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Kutliroff in view of Bell, further in view of Kim, further in view of Singh, in which the method branches to image-based reconstruction when no model of the selected object is stored and already computes a three-dimensional position for each pixel of that object, with the features of Zhou's invention of extracting the edges of the object instances by Canny edge detection on the segmentation masks and building a per-object mesh by connecting the vertices within each point cloud group. The combination would have been obvious because the not-stored branch leaves the method holding the object's three-dimensional points with no model built from them, and Zhou consumes that input directly, segmenting depth data per object, computing the three-dimensional coordinates of each depth sample, and connecting those points into a mesh for the object instance. The result is predictable because both operate on per-pixel depth data captured in an augmented reality environment.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jai Li whose telephone number is (571)272-1170. The examiner can normally be reached Mon-Thu between 06:00-16:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAI W LI/Junior Patent Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights