DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on/after Mar. 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Response to Arguments
Applicant’s arguments, see pp. 8-9, filed 29 June 2026, with respect to the rejection(s) of claims 1, 10, and 19 under 35 U.S.C. § 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Miller (U.S. PG-PUB 2013/0117377). The Examiner notes that the previously cited BUKOWSKI and LAWRENCE references are no longer relied upon in this Office action. Please see the Office action below for further explanation regarding the rationale for the rejection(s) of the newly amended claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 USC 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-10, 12-20, and 77-82 are rejected under 35 U.S.C. 103 as being unpatentable over Phalak (U.S. PG-PUB 2021/0279950, ‘PHALAK’) in view of Miller (U.S. PG-PUB 2013/0117377, 'MILLER').
Regarding claim 1, PHALAK discloses a method comprising: at a processor:
PNG
media_image1.png
319
520
media_image1.png
Greyscale
generating a … (3D) representation of a physical environment, the 3D representation comprising points of a point cloud and each having a 3D location and representing an appearance of a portion of the physical environment (PHALAK; FIG. 15A; ¶ 0263; “… FIG. 15A illustrates a high-level flow diagram for generating an estimated floorplan with a two-step process. During the 1st step, a room classification and a wall classification may be determined at 1502A from an input image of a scene [‘generating a … (3D) representation of a physical environment’] … An input image may include a static image … captured by a camera …, a sequence of images …, [or] a video … The input image may be processed to determine a set of points or features that may also be referred to as an input point cloud. Classification may be performed to assign a label (e.g., a 1st wall label, a 2nd wall label, a 1st room label, a 2nd room label, [‘appearance of … the physical environment’] etc.) to each point or feature.”);
determining object types for the elements of the 3D representation (PHALAK; ¶ 0310; “For training, some embodiments uniformly sample sub-volumes at … intervals out of each of the [training] scenes. These embodiments keep all sub-volumes containing any non-structural object voxels (e.g., tables, chairs [furniture object type]), and randomly discard sub-volumes that contain only structural voxels (e.g., wall/ceiling/floor [room perimeter object type]) with 90% probability.”), the object types comprising … object type(s) for room perimeter objects;
identifying a room perimeter based on determining the object types, wherein the room perimeter comprises multiple perimeter regions formed by the room perimeter objects (PHALAK; ¶ 0379; “Some embodiments utilize multiple observations of the same real-world scene from … poses to generate a per-frame dense depth map, through the … Multiview Depth Estimation network. These embodiments then optimize a segmentation algorithm for classifying ceiling, floor, and walls [‘comprising/determining object type(s) for room perimeter objects’, ‘multiple perimeter regions formed by the room perimeter objects’] through a standard pyramid scene parsing (PSP) network … After obtaining a depth map and a wall segmentation mask [‘room perimeter object’] for each input frame, some embodiments generate a unified point cloud using only the depth pixels belonging to the wall class … To remove internal wall points, … use the concept of a-shape to create a subset of the point cloud that is representative of its concave hull [‘room perimeter’].”); [and]
replacing a first set of the points of the 3D representation that correspond to the room perimeter with … non-point cloud-based visual feature(s) representing the room perimeter (PHALAK; FIG. 7C; ¶ 0177; “It is also possible to cut out randomly sized rectangular blocks to represent missing points in scenarios where a door or window might be part of the wall.” ¶ 0392; “Various embodiments build a fully synthetic dataset along with normal labels, starting from a room perimeter skeleton randomly sampled from various shapes (rectangle …”).
PHALAK does not explicitly disclose that a second set of the points of the 3D representation that do not correspond to the room perimeter remain in the 3D representation, which MILLER discloses (MILLER; ¶ 0091; “Recognizer software objects ("recognizers") may be deployed … to … assist with recognition of … objects … as a user is navigating data in a world. … if a system has data for a world model comprising 3-D point clouds and pose-tagged images, and there is a desk with … points on it as well as an image of the desk, there may not be a determination that what is being observed is … a desk as humans would know it. In other words, some 3-D points in space and an image from someplace off in space that shows most of the desk may not be enough to instantly recognize that a desk is being observed. To assist with this identification, a specific object recognizer [is] created that will go into the raw 3-D point cloud, segment out a set of points, and, … extract the plane of the top surface of the desk [points of the 3D representation that do not correspond to the room perimeter]. Similarly, a recognizer [is] created to segment out a wall from 3-D points, so that a user could … remove part of the wall in [VR/AR] and have a portal to another room that is not actually there in the real world.” [The Examiner asserts that the removal of points corresponding to a wall {room perimeter} is equivalent to having interior points ‘that do not correspond to the room perimeter [but] remain in the 3D representation’, since the removal of some points necessitates that others must remain. In other words, the perimeter points may [be] part of the wall and removable, leaving the interior non-perimeter/wall points to remain.]).
PHALAK also does not explicitly disclose providing the 3D representation to a remote electronic device, which MILLER discloses (MILLER; FIG. 2; ¶ 0036; “… user device 120 for interacting with the system 100 … A user 210 may interface … digital world(s) through a smart phone 220. The gateway is implemented by a software application 230 stored on and running on the smart phone 220. … the data network 130 includes a wireless mobile network connecting the user device (i.e., smart phone 220) to the computer network 105.”). MILLER further discloses that the 3D representation is a combination of both:
the … non-point cloud-based visual features representing the room perimeter (MILLER; ¶ 0091; “… an [AR/VR] experience [is] based upon recognizing certain types of objects. … it [is] important to understand that a particular object has a depth [to] recognize and understand such object. Recognizer software objects ("recognizers") may be deployed on cloud or local resources to specifically assist with recognition of … objects on … platforms as a user is navigating data in a world. … if a system has data for a world model comprising 3-D point clouds and pose-tagged images, and there is a desk with a bunch of points on it as well as an image of the desk, there may not be a determination that what is being observed is, indeed, a desk as humans would know it. … some 3-D points in space and an image from someplace off in space that shows most of the desk may not be enough to instantly recognize that a desk is being observed. To assist with this identification, a specific object recognizer may be created that will go into the raw 3-D point cloud, segment out a set of points, and … extract the plane of the top surface of the desk. Similarly, a recognizer may be created to segment out a wall from 3-D points [non-point cloud-based visual features representing the room perimeter], so that a user could change wallpaper or remove part of the wall in [AR/VR] and have a portal to another room that is not actually there in the real world.”); and
PNG
media_image2.png
401
454
media_image2.png
Greyscale
the second set of points of the 3D representation that do not correspond to the room perimeter (MILLER; FIG. 16; ¶ 0099; “… a wearable system may … capture image information and extract fiducials and recognized points (52). The wearable local system may calculate pose using one of the pose calculation techniques mentioned below. The cloud (54) may … use images and fiducials to segment 3-D objects [points of the 3D representation that do not correspond to the room perimeter] from more static 3-D background; images provide textures maps for objects and the world (textures may be realtime videos). The cloud resources (56) may … store and make available static fiducials and textures for world registration. The cloud resources may be configured to groom the point cloud for optimal point density for registration. The cloud resources (60) may store and make available object fiducials and textures for object registration and manipulation; the cloud may groom point clouds for optimal density for registration. The cloud resource may be configured (62) to use all valid points and textures to generate fractal solid models of objects; the cloud may groom point cloud information for optimal fiducial density. The cloud resource (64) may … query users for training on identity of segmented objects and the world; an ontology database may use the answers to imbue objects and the world with actionable properties.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of PHALAK to include the disclosure that a second set of the points of the 3D representation that do not correspond to the room perimeter remain in the 3D representation, providing the 3D representation to a remote electronic device, wherein the 3D representation is a combination of both the … non-point cloud-based visual features representing the room perimeter, and the second set of points of the 3D representation that do not correspond to the room perimeter of MILLER. The motivation for this modification is to implement recognizers to operate within the data of a world model and which may be thought of as software "robots" that crawl a world model and imbue that world model with semantic information, or an ontology about what is believed to exist amongst the points in space. Such recognizers or software robots may be configured such that their entire existence is about going around the pertinent world of data and finding things that it believes are walls, or chairs, or other items. They may tag a set of points with the functional equivalent of, "this set of points belongs to a wall” and may comprise a combination of point-based algorithm and pose-tagged image analysis for mutually informing the system regarding what is in the points (MILLER; ¶ [0091]).
Independent claims 10 and 19 recite similar limitations and exhibit similar scope when compared to independent claim 1; therefore, the same motivation(s) to combine references will be maintained.
Regarding claim 10, PHALAK-MILLER discloses a system comprising:
a non-transitory computer-readable storage medium; and
… processor(s) coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the … processor(s), cause the system to perform operations (PHALAK; FIG. 11A; ¶ 0214; “The left PCBA 2502 may include a control subsystem, which may include … controller(s) (e.g., microcontroller, microprocessor, digital signal processor, graphical processing unit, central processing unit, application specific integrated circuit (ASIC), field programmable gate array (FPGA) 2540, and/or programmable logic unit (PLU)).The control system may include … non-transitory computer- or processor readable medium that stores executable logic or instructions and/or data or information. The non-transitory computer- or processor readable medium may take a variety of forms, [e.g.] volatile and nonvolatile forms, for instance read only memory (ROM), random access memory (RAM …)”) comprising: … ([The Examiner notes that the remaining limitations are repeated verbatim from those recited in independent claim 1.]).
Regarding claim 19, PHALAK-MILLER disclose a non-transitory computer-readable storage medium storing program instructions executable via … processor(s) to perform operations (PHALAK; FIG. 11A; ¶ 214; [See treatment of the preamble of parallel independent claim 10.]) comprising: … ([The Examiner notes that the remaining limitations are repeated verbatim from those recited in independent claim 1.]).
Regarding claim 2 and claim 20, PHALAK-MILLER disclose the method of claim 1 and the non-transitory computer-readable storage medium of claim 19, wherein the remote electronic device provides a view of the 3D representation, the view comprising the second set of points and the … non-point cloud-based visual feature(s) (PHALAK; FIG. 1; ¶ 0144; “… a user may be wearing an [AR] system …, which may also be termed a “spatial computing” system in relation to such system's interaction with the [3-D] world around the user when operated [‘remote electronic device provides a view of the 3D representation’]. [This] system … comprises … a head-wearable display component (2), and … features … cameras … which may … map the environment around the user, or to create a “mesh” of such environment, comprising various points [‘second set of points’] representative of the geometry of various objects within the environment around the user, such as walls, floors, chairs, and the like. The spatial computing system may … map or mesh the environment around the user, and to run or operate software … which may … utilize the map or mesh of the room to assist the user in placing, manipulating, visualizing, creating, and modifying various objects and elements in the [3-D] space around the user.”).
Regarding claim 4 and claim 12, PHALAK-MILLER disclose the method of claim 1 and the system of claim 10, wherein determining object types for the points of the 3D representation comprises using a machine learning model to provide a scene understanding of the physical environment (PHALAK; ¶ 0278; “… where the input images or image sequences (e.g., videos) comprise 3-D … data (e.g., 3D video, 4D spatial-temporal image sequence, etc.), … embodiments utilize algorithms such as a Minkowski Net-based algorithm, a ScanComplete-based algorithm, etc. for semantic filtering or segmentation to perform scene analysis and understanding. In … other embodiments where the input images or image sequences (e.g., videos) comprise 2D data, … embodiments utilize algorithms such as a Mask RCNN-based algorithm, a PSPNet-based algorithm, etc. for semantic filtering or segmentation to perform scene analysis and understanding.”).
Regarding claim 5 and claim 13, PHALAK-MILLER disclose the method of claim 1 and the system of claim 10, wherein the object types include furniture object types (PHALAK; FIG. 1; ¶ 0144; ¶ 0310; “For training, some embodiments uniformly sample sub-volumes at … intervals out of each of the train scenes. These embodiments keep all sub-volumes containing any non-structural object voxels (e.g., tables, chairs) and randomly discard sub-volumes that contain only structural voxels (e.g., wall/ceiling/floor) with 90% probability.”).
Regarding claim 6 and claim 14, PHALAK-MILLER disclose the method of claim 1 and the system of claim 10, wherein the … non-point cloud-based visual feature(s) comprise … planar element(s) (PHALAK; FIG. 7C; ¶ 0177; “It is also possible to cut out randomly sized rectangular blocks to represent missing points in scenarios where a door or window might be part of the wall.” ¶ 0392; “Various embodiments build a fully synthetic dataset along with normal labels, starting from a room perimeter skeleton randomly sampled from various shapes (rectangle …”).
Regarding claim 7 and claim 15, PHALAK-MILLER disclose the method of claim 1 and the system of claim 10, wherein the … non-point cloud-based visual feature(s) comprise a geometric element corresponding to … a portion of the room perimeter (PHALAK; ¶ 0367-368; “… FIG. 15E illustrates more details about generating a Deep-Perimeter type of shape at 1502C of FIG. 15C. These embodiments perform a deep estimation at 1502E on … RGB frame(s) in an input image sequence of an environment. A depth map and a wall segmentation mask may be generated at 1504E by using … a multi-view depth estimation network and a PSPNet-based and/or a Resnet-based segmentation module. … a per-frame dense depth map may be generated at 1502E with … a Multiview depth estimation network. A wall point cloud may be extracted at 1506E by fusing … mask depth image(s) with … pose trajectories by using a marching cubes module. These embodiments further isolate depth predictions corresponding to wall points … by training a deep segmentation network at 1508E. The depth predictions may be projected at 1510E to a … (3D) point cloud. The 3D point cloud may be clustered … at 1512E … by detecting, with a deep network, points that belong to the same plane instance. Some embodiments directly cluster wall points so that these embodiments are not handicapped when points (e.g., points corresponding to corners, edges, etc.) are occluded. Some embodiments adopt an end-to-end model for clustering point clouds into long-range planar regions using synthetically generated ground truth. The clusters determined at 1512E may be translated at 1514E into a set of planes that forms a perimeter layout.”).
Regarding claim 8 and claim 16, PHALAK-MILLER discloses the method of claim 1 and the system of claim 10, further comprising:
obtaining an image of the physical environment (PHALAK; ¶ 0148; “Some embodiments extract a floorplan of an indoor environment with single or multiple rooms from captured data, such as a 3D scan of the environment's structural elements, which may include walls, doors, and windows.” ¶ 0154; ¶ 0265; “… FIG. 15B illustrates more details about a determination of a room classification and a wall classification at 1502A in FIG. 15A. … the input image may be identified at 1502B. An image may be obtained from a scan of a scene (e.g., an interior environment having … room(s) with … wall(s)). … an input image may be obtained from a 3D scan of a scene. … an input image may include a static image such as a photograph captured by a camera …, a sequence of images …, [or] a video … An input image may be processed to determine a set of points or features that may also be referred to as an input point cloud. An image may be a … (2D) planar image (or sequence of images) or a higher-dimensional image (or sequence of images such as a 3D image in the Euclidean space, a 4D image with temporal and spatial dimensions, … etc.)” ¶ 0341);
identifying a portion of the image corresponding to the first set of points (PHALAK; ¶ 0171; “FIG. 6 illustrates the input 602 of a set of clustered wall points to a perimeter estimation module for a room, the ordering of the wall segment endpoints 602 determined by the shortest path algorithm, and a room perimeter 606 determined as a polygon by extruding or extending the line segments to generate the polygon vertices.”); and
generating an appearance characteristic of the … non-point cloud-based visual feature(s) based on the portion of the image (PHALAK; ¶ 0376; “FIG. 14K illustrates [a] pipeline for perimeter estimation. Some embodiments begin with a posed monocular sequence of images along with their relative poses. These embodiments extract semantic segmentation maps for walls and a dense depth map through a multi-view stereo algorithm. These two outputs are combined through standard un-projection to form a 3D point cloud consisting of wall pixels only. These wall pixels are then colored into wall instance candidates [‘generating an appearance characteristic of the … non-point cloud-based visual feature(s) based on the portion of the image’] using a deep clustering network and post processed with linear least squares and the shortest path algorithm to form the final perimeter prediction.”).
Regarding claim 9 and claim 17, PHALAK-MILLER disclose the method of claim 1 and the system of claim 10, wherein the … non-point cloud-based visual feature(s) are defined using less data than the first set of points (MILLER; ¶ 0098; “Fiducial points [are] used to optically calculate pose. As the world is mapped with greater detail, more objects may be segmented out and given their own texture maps, but the world still preferably is representable at low spatial resolution in simple polygons with low resolution texture maps.”).
Regarding claim 18, PHALAK-MILLER disclose the system of claim 10, wherein the 3D representation is provided to an electronic device during a communication session (PHALAK; ¶ 0142; “… the terms virtual reality (VR), augmented reality (AR), mixed reality (MR), and extended reality (XR) may be used interchangeably … to denote a method/system for displaying … virtual contents to a user [‘the 3D representation is provided’] via at least a wearable XR devices [‘provided to an electronic device’] as well as … remote computing device(s) [‘electronic device during a communication session’] supporting the wearable XR devices.”).
Regarding claim 77, PHALAK-MILLER disclose the method of claim 1, wherein the one or more non-point cloud-based visual features comprise planar elements (PHALAK; FIG. 7C; ¶ 0177; “It is also possible to cut out randomly sized rectangular blocks to represent missing points in scenarios where a door or window might be part of the wall.” ¶ 0392; “… embodiments build a fully synthetic dataset along with normal labels, starting from a room perimeter skeleton randomly sampled from various shapes (rectangle [etc.]”).
Regarding claim 78, PHALAK-MILLER disclose the method of claim 1, wherein the one or more non-point cloud-based visual features comprise a geometric shell (PHALAK; ¶ 0171; “… FIG. 6 illustrates the input 602 of a set of clustered wall points to a perimeter estimation module for a room, the ordering of the wall segment endpoints 602 determined by the shortest path algorithm, and a room perimeter 606 determined as a polygon [‘geometric shell’] by extruding or extending the line segments to generate the polygon vertices.”).
Regarding claim 79, PHALAK-MILLER disclose the method of claim 1, wherein the one or more non-point cloud-based visual features comprise a 3D rectangle (PHALAK; FIG. 7C; ¶ 0177; “It is … possible to cut out randomly sized rectangular blocks to represent missing points in scenarios where a door/window might be part of the wall.” ¶ 0392; “… embodiments build a fully synthetic dataset along with normal labels, starting from a room perimeter skeleton randomly sampled from various shapes (rectangle [etc.]” [The Examiner interprets the claim limitation ‘3D rectangle’ to mean a 2-D rectangle that exists within a [3-D] environmental context.]).
Regarding claim 80, PHALAK-MILLER discloses the method of claim 1, wherein replacing the first set of points of the 3D representation reduces data size of the 3D representation (MILLER; ¶ 0098; “ As the world is mapped with greater detail, more objects may be segmented out and given their own texture maps, but the world still preferably is representable at low spatial resolution in simple polygons with low resolution texture maps.”).
Regarding claim 81, PHALAK-MILLER discloses the method of claim 1, wherein replacing the first set of points of the 3D representation reduces the number of the points included in the 3D representation (MILLER; ¶ 0098; “ As the world is mapped with greater detail, more objects [are] segmented out and given their own texture maps, but the world still preferably is representable at low spatial resolution in simple polygons with low resolution texture maps.”).
Regarding claim 82, PHALAK-MILLER discloses the method of claim 1, wherein replacing the first set of points of the 3D representation provides a more compact 3D representation (MILLER; ¶ 0098; “ As the world is mapped with greater detail, more objects may be segmented out and given their own texture maps, but the world still preferably is representable at low spatial resolution in simple polygons with low resolution texture maps.”) for more efficient and faster communication to the remote device and more efficient and faster rendering on the remote device
(MILLER; FIG. 16; ¶ 0099; “… a wearable system [remote device] may … capture image information and extract fiducials and recognized points (52). The wearable local system may calculate pose using one of the pose calculation techniques … The cloud (54) may … use images and fiducials to segment 3-D objects from more static 3-D background; images provide textures maps for objects [rendering] and the world (textures may be realtime videos). The cloud resources (56) may … store and make available static fiducials and textures for world registration. The cloud resources may … groom the point cloud for optimal point density for registration. The cloud resources (60) may store and make available object fiducials and textures for object registration and manipulation; the cloud may groom point clouds for optimal density for registration. The cloud resource may be configured (62) to use all valid points and textures to generate fractal solid models of objects; the cloud may groom point cloud information for optimal fiducial density. The cloud resource (64) may … query users for training on identity of segmented objects and the world; an ontology database may use the answers to imbue objects and the world with actionable properties.”).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN M COFINO whose telephone number is (303) 297-4268. The examiner can normally be reached Monday-Friday 10A-4P MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at 571-272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONATHAN M COFINO/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights