DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on/after Mar. 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Response to Arguments
Applicant’s arguments, see pp. 9-10, filed 09 March 2026, with respect to the rejection(s) of claims 1, 10, and 19 under 35 U.S.C. § 103 have been fully considered and are persuasive. Therefore, the rejection(s) have been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Phalak (U.S. PG-PUB 2021/0279950), and further in view of Bukowski et al. (US PG-PUB 2005/0203930) and Lawrence et al. (U.S. PG-PUB 2012/0019522). The Examiner notes that the previously-cited PARCHAMI and KAMUDA references are no longer relied upon in this Office action. Please see the Office action below for further explanation regarding the rationale for the rejection(s) of the newly-amended claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 USC 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-10, 12-20, and 77-79 are rejected under 35 U.S.C. 103 as being unpatentable over Phalak (U.S. PG-PUB 2021/0279950, ‘PHALAK’) in view of Bukowski et al. (U.S. PG-PUB 2005/0203930, ‘BUKOWSKI’) and Lawrence et al. (U.S. PG-PUB 2012/0019522, 'LAWRENCE').
Regarding claim 1, PHALAK discloses a method comprising: at a processor:
PNG
media_image1.png
319
520
media_image1.png
Greyscale
generating a … (3D) representation of a physical environment, the 3D representation comprising points of a point cloud and each having a 3D location and representing an appearance of a portion of the physical environment (PHALAK; FIG. 15A; ¶ 0263; “… FIG. 15A illustrates a high-level flow diagram for generating an estimated floorplan with a two-step process. During the 1st step, a room classification and a wall classification may be determined at 1502A from an input image of a scene [‘generating a … (3D) representation of a physical environment’] … An input image may include a static image … captured by a camera …, a sequence of images …, [or] a video … The input image may be processed to determine a set of points or features that may also be referred to as an input point cloud. Classification may be performed to assign a label (e.g., a 1st wall label, a 2nd wall label, a 1st room label, a 2nd room label, [‘appearance of … the physical environment’] etc.) to each point or feature.”);
determining object types for the elements of the 3D representation (PHALAK; ¶ 0310; “For training, some embodiments uniformly sample sub-volumes at … intervals out of each of the [training] scenes. These embodiments keep all sub-volumes containing any non-structural object voxels (e.g., tables, chairs [furniture object type]), and randomly discard sub-volumes that contain only structural voxels (e.g., wall/ceiling/floor [room perimeter object type]) with 90% probability.”), the object types comprising … object type(s) for room perimeter objects;
identifying a room perimeter based on determining the object types, wherein the room perimeter comprises multiple perimeter regions formed by the room perimeter objects (PHALAK; ¶ 0379; “Some embodiments utilize multiple observations of the same real-world scene from … poses to generate a per-frame dense depth map, through the … Multiview Depth Estimation network. These embodiments then optimize a segmentation algorithm for classifying ceiling, floor, and walls [‘comprising/determining object type(s) for room perimeter objects’, ‘multiple perimeter regions formed by the room perimeter objects’] through a standard pyramid scene parsing (PSP) network … After obtaining a depth map and a wall segmentation mask [‘room perimeter object’] for each input frame, some embodiments generate a unified point cloud using only the depth pixels belonging to the wall class … To remove internal wall points, … use the concept of a-shape to create a subset of the point cloud that is representative of its concave hull [‘room perimeter’].”); [and]
replacing a first set of the points of the 3D representation that correspond to the room perimeter with … non-point cloud-based visual feature(s) representing the room perimeter (PHALAK; FIG. 7C; ¶ 0177; “It is also possible to cut out randomly sized rectangular blocks to represent missing points in scenarios where a door or window might be part of the wall.” ¶ 0392; “Various embodiments build a fully synthetic dataset along with normal labels, starting from a room perimeter skeleton randomly sampled from various shapes (rectangle …”).
PHALAK does not explicitly disclose that a second set of the points of the 3D representation that do not correspond to the room perimeter remain in the 3D representation, which BUKOWSKI discloses (BUKOWSKI; FIG. 14; ¶ 0124; “… the present invention can allow the user to have a dynamic "active set," which can be managed with a set of simple commands. While a number of commands can be used, one command allows a user to define a "fence" to describe a 3D geometric region. … Another command allows a user to remove the points outside a fence such that any points outside the fence are removed from the active set.” [The Examiner asserts that the removal of points outside a given boundary, such as a defined fence, is equivalent to having interior points that do not ‘correspond to the room perimeter remain in the 3D representation’, since the removal of some points necessitates that others must remain. In other words, the perimeter points may be considered to be outside the defined boundary and removable, leaving the interior non-perimeter points to remain.]).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of PHALAK to include the disclosure that a second set of the points of the 3D representation that do not correspond to the room perimeter remain in the 3D representation of BUKOWSKI. The motivation for this modification is to enable storage and processing of large [3-D] data sets in real time by combining and registering the large [3-D] data sets into a single data set. The data can be stored in a data tree structure formed of layers of spatially organized blocks of data. Such storage allows portions of the data to be viewed efficiently, displaying actual point data at an acceptable resolution for the viewing mechanism. Density limited queries can be executed that allow sub-sampling to be done directly and evenly without geometric constraint, to provide a subset of points that is limited in size and includes a spatially-even decomposition of that set of points. This allows the system as a whole to support arbitrarily large point sets while allowing full partitioning functionality, which is efficient to use in both time and space (BUKOWSKI; Abstract). The Examiner notes that BUKOWSKI allows for the partitioning of exterior planar surfaces (walls, ceilings, floors, etc.) from interior contents, wherein the exterior planes may be represented by simple polygons/geometry and the potentially more complex non-planar, interior objects may remain represented by generally non-coplanar point clouds.
PHALAK-BUKOWSKI do not explicitly disclose providing the 3D representation to a remote electronic device, the 3D representation comprising the … non-point cloud-based visual features representing the room perimeter; and the second set of points of the 3D representation that do not correspond to the room perimeter (LAWRENCE; FIG. 4; ¶ 0052-53; “The 3D model generation engine may render the 3D model as a 3D "point cloud" in which each point in the cloud is specified by its position (XYZ) and a color intensity (e.g. R,G,B). … The 3D model generation may alternately render the 3D model as a 3D polygonal model in which individual points are amalgamated into larger polygonal surfaces [‘the … non-point cloud-based visual features representing the room perimeter’] having pixelized textures. The polygonal model may be rendered directly from the images or from a point cloud … The 3D model viewer displays a visual representation 206 of the 3D model from a given viewpoint on the display 208 of the host computer 210 [‘providing the 3D representation to a remote electronic device’]. The viewer represents the 3D model onto the display … The manipulation of the viewpoint provides for the 3D characteristics of the modeled scene. … The viewer allows the host operator to change the viewpoint … or via a touchscreen display.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of PHALAK-BUKOWSKI to include the providing the 3D representation to a remote electronic device, the 3D representation comprising the … non-point cloud-based visual features representing the room perimeter, and the second set of points of the 3D representation that do not correspond to the room perimeter of LAWRENCE. The motivation for this modification is to generate a polygonal model which is more complex but is more efficient to display, manipulate, and transmit [than a dense point-cloud] (LAWRENCE; ¶ [0052]).
Independent claims 10 and 19 recite similar limitations and exhibit similar scope when compared to independent claim 1; therefore, the same motivation(s) to combine references will be maintained.
Regarding claim 10, PHALAK-BUKOWSKI-LAWRENCE disclose a system comprising:
a non-transitory computer-readable storage medium; and
… processor(s) coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the … processor(s), cause the system to perform operations (PHALAK; FIG. 11A; ¶ 0214; “The left PCBA 2502 may include a control subsystem, which may include … controller(s) (e.g., microcontroller, microprocessor, digital signal processor, graphical processing unit, central processing unit, application specific integrated circuit (ASIC), field programmable gate array (FPGA) 2540, and/or programmable logic unit (PLU)).The control system may include … non-transitory computer- or processor readable medium that stores executable logic or instructions and/or data or information. The non-transitory computer- or processor readable medium may take a variety of forms, [e.g.] volatile and nonvolatile forms, for instance read only memory (ROM), random access memory (RAM …)”) comprising: … ([The Examiner notes that the remaining limitations are repeated verbatim from those recited in independent claim 1.]).
Regarding claim 19, PHALAK-BUKOWSKI-LAWRENCE disclose a non-transitory computer-readable storage medium storing program instructions executable via … processor(s) to perform operations (PHALAK; FIG. 11A; ¶ 214; [See treatment of the preamble of parallel independent claim 10.]) comprising: … ([The Examiner notes that the remaining limitations are repeated verbatim from those recited in independent claim 1.]).
Regarding claim 2 and claim 20, PHALAK-BUKOWSKI-LAWRENCE disclose the method of claim 1 and the non-transitory computer-readable storage medium of claim 19, wherein the remote electronic device provides a view of the 3D representation, the view comprising the second set of points and the … non-point cloud-based visual feature(s) (PHALAK; FIG. 1; ¶ 0144; “… a user may be wearing an [AR] system …, which may also be termed a “spatial computing” system in relation to such system's interaction with the [3-D] world around the user when operated [‘remote electronic device provides a view of the 3D representation’]. [This] system … comprises … a head-wearable display component (2), and … features … cameras … which may … map the environment around the user, or to create a “mesh” of such environment, comprising various points [‘second set of points’] representative of the geometry of various objects within the environment around the user, such as walls, floors, chairs, and the like. The spatial computing system may … map or mesh the environment around the user, and to run or operate software … which may … utilize the map or mesh of the room to assist the user in placing, manipulating, visualizing, creating, and modifying various objects and elements in the [3-D] space around the user.”).
Regarding claim 4 and claim 12, PHALAK-BUKOWSKI-LAWRENCE disclose the method of claim 1 and the system of claim 10, wherein determining object types for the points of the 3D representation comprises using a machine learning model to provide a scene understanding of the physical environment (PHALAK; ¶ 0278; “… where the input images or image sequences (e.g., videos) comprise 3-D … data (e.g., 3D video, 4D spatial-temporal image sequence, etc.), some embodiments utilize algorithms such as a Minkowski Net-based algorithm, a ScanComplete-based algorithm, etc. for semantic filtering or segmentation to perform scene analysis and understanding. In some other embodiments where the input images or image sequences (e.g., videos) comprise 2D data, some embodiments utilize algorithms such as a Mask RCNN-based algorithm, a PSPNet-based algorithm, etc. for semantic filtering or segmentation to perform scene analysis and understanding.”).
Regarding claim 5 and claim 13, PHALAK-BUKOWSKI-LAWRENCE disclose the method of claim 1 and the system of claim 10, wherein the object types include furniture object types (PHALAK; FIG. 1; ¶ 0144; ¶ 0310; “For training, some embodiments uniformly sample sub-volumes at … intervals out of each of the train scenes. These embodiments keep all sub-volumes containing any non-structural object voxels (e.g., tables, chairs), and randomly discard sub-volumes that contain only structural voxels (e.g., wall/ceiling/floor) with 90% probability.”).
Regarding claim 6 and claim 14, PHALAK-BUKOWSKI-LAWRENCE disclose the method of claim 1 and the system of claim 10, wherein the … non-point cloud-based visual feature(s) comprise … planar element(s) (PHALAK; FIG. 7C; ¶ 0177; “It is also possible to cut out randomly sized rectangular blocks to represent missing points in scenarios where a door or window might be part of the wall.” ¶ 0392; “Various embodiments build a fully synthetic dataset along with normal labels, starting from a room perimeter skeleton randomly sampled from various shapes (rectangle …”).
Regarding claim 7 and claim 15, PHALAK-BUKOWSKI-LAWRENCE disclose the method of claim 1 and the system of claim 10, wherein the … non-point cloud-based visual feature(s) comprise a geometric element corresponding to … a portion of the room perimeter (PHALAK; ¶ 0367-368; “… FIG. 15E illustrates more details about generating a Deep-Perimeter type of shape at 1502C of FIG. 15C. These embodiments perform a deep estimation at 1502E on … RGB frame(s) in an input image sequence of an environment. A depth map and a wall segmentation mask may be generated at 1504E by using … a multi-view depth estimation network and a PSPNet-based and/or a Resnet-based segmentation module. … a per-frame dense depth map may be generated at 1502E with … a Multiview depth estimation network. A wall point cloud may be extracted at 1506E by fusing … mask depth image(s) with … pose trajectories by using a marching cubes module. These embodiments further isolate depth predictions corresponding to wall points … by training a deep segmentation network at 1508E. The depth predictions may be projected at 1510E to a … (3D) point cloud. The 3D point cloud may be clustered into … cluster(s) at 1512E … by detecting, with a deep network, points that belong to the same plane instance. Some embodiments directly cluster wall points so that these embodiments are not handicapped when points (e.g., points corresponding to corners, edges, etc.) are occluded. Some embodiments adopt an end-to-end model for clustering point clouds into long-range planar regions using synthetically generated ground truth. The clusters determined at 1512E may be translated at 1514E into a set of planes that forms a perimeter layout.”).
Regarding claim 8 and claim 16, PHALAK-BUKOWSKI-LAWRENCE disclose the method of claim 1 and the system of claim 10, further comprising:
obtaining an image of the physical environment (PHALAK; ¶ 0148; “Some embodiments extract a floorplan of an indoor environment with single or multiple rooms from captured data, such as a 3D scan of the environment's structural elements, which may include walls, doors, and windows.” ¶ 0154; ¶ 0265; “… FIG. 15B illustrates more details about a determination of a room classification and a wall classification at 1502A in FIG. 15A. … the input image may be identified at 1502B. An image may be obtained from a scan of a scene (e.g., an interior environment having … room(s) with … wall(s)). … an input image may be obtained from a 3D scan of a scene. … an input image may include a static image such as a photograph captured by a camera …, a sequence of images …, [or] a video … An input image may be processed to determine a set of points or features that may also be referred to as an input point cloud. An image may be a … (2D) planar image (or sequence of images) or a higher-dimensional image (or sequence of images such as a 3D image in the Euclidean space, a 4D image with temporal and spatial dimensions, … etc.)” ¶ 0341);
identifying a portion of the image corresponding to the first set of points (PHALAK; ¶ 0171; “FIG. 6 illustrates the input 602 of a set of clustered wall points to a perimeter estimation module for a room, the ordering of the wall segment endpoints 602 determined by the shortest path algorithm, and a room perimeter 606 determined as a polygon by extruding or extending the line segments to generate the polygon vertices.”); and
generating an appearance characteristic of the … non-point cloud-based visual feature(s) based on the portion of the image (PHALAK; ¶ 0376; “FIG. 14K illustrates [a] pipeline for perimeter estimation. Some embodiments begin with a posed monocular sequence of images along with their relative poses. These embodiments extract semantic segmentation maps for walls and a dense depth map through a multi-view stereo algorithm. These two outputs are combined through standard un-projection to form a 3D point cloud consisting of wall pixels only. These wall pixels are then colored into wall instance candidates [‘generating an appearance characteristic of the … non-point cloud-based visual feature(s) based on the portion of the image’] using a deep clustering network and post processed with linear least squares and a shortest path algorithm to form the final perimeter prediction.”).
Regarding claim 9 and claim 17, PHALAK-BUKOWSKI-LAWRENCE disclose the method of claim 1 and the system of claim 10, wherein the … non-point cloud-based visual feature(s) are defined using less data than the first set of points (LAWRENCE; ¶ 0052; “The 3D model generation may alternately render the 3D model as a 3D polygonal model in which individual points are amalgamated [‘defined using less data than the first set of points’] into larger polygonal surfaces having pixelized textures. The polygonal model may be rendered directly from … a point cloud … The polygonal model is more complex to generate but is more efficient to display, manipulate, and transmit.”).
Regarding claim 18, PHALAK-BUKOWSKI-LAWRENCE disclose the system of claim 10, wherein the 3D representation is provided to an electronic device during a communication session (PHALAK; ¶ 0142; “… the terms virtual reality (VR), augmented reality (AR), mixed reality (MR), and extended reality (XR) may be used interchangeably … to denote a method or system for displaying at least virtual contents to a user [‘the 3D representation is provided’] via at least a wearable XR devices [‘provided to an electronic device’] as well as … remote computing device(s) [‘electronic device during a communication session’] supporting the wearable XR devices.”).
Regarding claim 77, PHALAK-BUKOWSKI-LAWRENCE disclose the method of claim 1, wherein the one or more non-point cloud-based visual features comprise planar elements (PHALAK; FIG. 7C; ¶ 0177; “It is also possible to cut out randomly sized rectangular blocks to represent missing points in scenarios where a door or window might be part of the wall.” ¶ 0392; “… embodiments build a fully synthetic dataset along with normal labels, starting from a room perimeter skeleton randomly sampled from various shapes (rectangle [etc.]”).
Regarding claim 78, PHALAK-BUKOWSKI-LAWRENCE disclose the method of claim 1, wherein the one or more non-point cloud-based visual features comprise a geometric shell (PHALAK; ¶ 0171; “… FIG. 6 illustrates the input 602 of a set of clustered wall points to a perimeter estimation module for a room, the ordering of the wall segment endpoints 602 determined by the shortest path algorithm, and a room perimeter 606 determined as a polygon [‘geometric shell’] by extruding or extending the line segments to generate the polygon vertices.”).
Regarding claim 79, PHALAK-BUKOWSKI-LAWRENCE disclose the method of claim 1, wherein the one or more non-point cloud-based visual features comprise a 3D rectangle (PHALAK; FIG. 7C; ¶ 0177; “It is also possible to cut out randomly sized rectangular blocks to represent missing points in scenarios where a door or window might be part of the wall.” ¶ 0392; “… embodiments build a fully synthetic dataset along with normal labels, starting from a room perimeter skeleton randomly sampled from various shapes (rectangle [etc.]” [The Examiner is interpreting the claim limitation ‘3D rectangle’ to mean a 2-D rectangle that exists within a three-dimensional environmental context.]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN M COFINO whose telephone number is (303) 297-4268. The examiner can normally be reached Monday-Friday 10A-4P MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at 571-272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONATHAN M COFINO/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights