DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 21-29 and 34-37 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-2, 4-9 and 12 of U.S. Patent No. 12,204,821. Although the claims at issue are not identical, they are not patentably distinct from each other because claims in the current application are broader than the reference claims.
Specifically, it is well established that “Omission of element and its function in combination is obvious expedient if remaining elements perform same functions as before” In re KARLSON (CCPA) 136 USPQ 184 (1963). Claims 21-29 and 34-37 in the current application are broader than the reference claim.
Below is a table indicating the corresponding relationship between claims 21-29 and 34-37 of the current application and claims 1-2, 4-9 and 12 of U.S. Patent No. 12,204,821.
Current Application
U.S. Patent No. 12,204,821
21
1
22
1
23
4
24
5
25
6
26
2
27
1
28
7
29
7
34
8
35
9
36
12
37
12
To perform analysis required, claim 21 of the current application is compared to claim 1 of U.S. Patent No. 12,204,821.
Claim 21: Current Application
Claim 1: U.S. Patent No. 12,204,821
A method, comprising:
receiving image data and depth information of at least a portion of a building;
generating, based on the image data, an edge existence probability mask,
wherein the edge existence probability mask represents confidence levels associated with identifying each of the one or more edges of a surface in the at least the portion of the building;
optimizing the edge existence probability mask using the depth information to obtain an optimized edge existence probability mask; and
identifying the one or more edges of the surface based on the optimized edge existence probability mask.
A method comprising:
receiving, by a computer, an image and depth information of at least a portion of a building;
generating, by the computer, based on the image, an edge existence probability mask by executing a segmentation procedure upon a section of the image that includes a visible portion of an edge of a wall and an occluded portion of the edge of the wall using a three-dimensional recursive method that handles occlusions,
wherein the edge existence probability mask represents a confidence level associated with identifying the edge of the wall in the at least the portion of the building;
executing, by the computer, a recursive procedure that uses the depth information to optimize the edge existence probability mask; executing, by the computer, a scene understanding procedure to derive semantic information of the at least the portion of the building;
identifying, by the computer, the edge of the wall based on the optimized edge existence probability mask and the semantic information of the at least the portion of the building; and generating, by the computer, a pictorial rendering of the at least the portion of the building, the pictorial rendering including the edge of the wall.
Claim 21 of the current application is broader than claim 1 of Patent 12,204,821 as shown above. Therefore, this claim is properly subject to ODP rejection.
Similarly, ODP rejection can be shown for claims 22-29 and 34-37 of the current application, as additional limitations in claims 22-29 and 34-37 are similarly recited in the conflicting claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 21, 26-28, 30-36 and 38-40 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
The claim(s) 21 recite(s) steps or acts of receiving image data and depth information of at least a portion of a building, generating an edge existence probability mask, optimizing the edge existence probability mask and identifying the one or more edges of the surface. Thus, the claim is directed to a process, which is one of the statutory categories of invention.
The claim recites the step of generating an edge existence probability mask representing confidence levels associated with identifying each of the one or more edges of a surface in the at least the portion of the building. Under the broadest reasonable interpretation, this step is a process that generates an edge existence probability mask representing confidence data to identify edges of a building and is merely a mathematical concept. This step merely employs mathematical relationships to manipulate existing information to generate additional information in the form of an “edge existence probability mask”. This step falls within the mathematical concept grouping of abstract ideas. The claim further recites the step of optimizing the edge existence probability mask using the depth information. Under the broadest reasonable interpretation, this step is a process of optimizing the edge existence probability mask and is merely a mathematical concept. This step merely employs mathematical relationships to manipulate existing information to generate additional information in the form of an “optimized edge existence probability mask”. This step falls within the mathematical concept grouping of abstract ideas. The claim also recites the step of identifying the one or more edges of the surface based on the optimized edge existence probability mask. Under the broadest reasonable interpretation, this step is a process that can be performed as a mental activity in the human mind by observation, evaluation, judgment and opinion. This step falls within mental process grouping of abstract ideas. See MPEP 2106.04(a)(2), subsection III. Accordingly, the claim recites an abstract idea.
The claim further recites the additional element of receiving image data and depth information of at least a portion of a building. This additional step is mere data gathering and is recited at a high level of generality, and thus is insignificant extra-solution activity. See MPEP 2105(g)(“whether the limitation is significant”). This judicial exception is not integrated into a practical application because the claim does not recite any additional elements or a combination of elements to integrate the abstract idea into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Therefore, the claim is not patent eligible.
Claims 26-27 similarly recites abstract ideas without any additional elements that are sufficient to amount to significantly more than the judicial exception, and therefore are not patent eligible.
The claim(s) 28 recite(s) steps or acts of receiving image data of at least a portion of a structure, receiving depth information associated with the at least the portion of the structure, generating semantic information of the at least the portion of the structure, generating an edge existence probability mask, optimizing the edge existence probability mask and identifying the one or more edges of the surface. Thus, the claim is directed to a process, which is one of the statutory categories of invention.
The claim recites the step of generating semantic information of the at least the portion of the structure based on the image data or a combination of image data and depth information. Under the broadest reasonable interpretation, this step is a process that can be performed using a pen and paper. This step falls within mental process grouping of abstract ideas. The claim recites the step of generating an edge existence probability mask representing confidence levels associated with identifying each of the one or more edges of a surface in the at least the portion of the building. Under the broadest reasonable interpretation, this step is a process that generates an edge existence probability mask representing confidence data to identify edges of a building and is merely a mathematical concept. This step merely employs mathematical relationships to manipulate existing information to generate additional information in the form of an “edge existence probability mask”. The claim further recites the step of optimizing the edge existence probability mask using the depth information. Under the broadest reasonable interpretation, this step is a process of optimizing the edge existence probability mask and is merely a mathematical concept. This step merely employs mathematical relationships to manipulate existing information to generate additional information in the form of an “optimized edge existence probability mask”. This step falls within the mathematical concept grouping of abstract ideas. The claim also recites the step of identifying the one or more edges of the surface based on the optimized edge existence probability mask is a process that under the broadest reasonable interpretation, can be performed as a mental process in the human mind by observation, evaluation, judgment and opinion. This step falls within mental process grouping of abstract ideas. See MPEP 2106.04(a)(2), subsection III. Accordingly, the claim recites an abstract idea.
The claim further recites the additional steps of receiving image data and depth information of at least a portion of a building. These additional steps are mere data gathering and are recited at a high level of generality, and thus are insignificant extra-solution activity. See MPEP 2105(g)(“whether the limitation is significant”). In addition, all uses of the recited judicial exceptions require such data gathering, and as such, these limitations do not impose any meaningful limits on the claim. This judicial exception is not integrated into a practical application because the claim does not recite any additional elements or a combination of elements to integrate the abstract idea into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Therefore, the claim is not patent eligible.
Claims 30-35 similarly recites abstract ideas without any additional elements that are sufficient to amount to significantly more than the judicial exception, and therefore are not patent eligible.
The claim(s) 36 recite(s) a system comprising one or more memories and one or more processors. Thus, the claim is directed to an apparatus, which is one of the statutory categories of invention.
The claim recites the step of generating semantic information of the at least the portion of the structure based on the image data or a combination of image data and depth information is a process that under the broadest reasonable interpretation, is a mental process performed using pen and paper. This step falls within mental process grouping of abstract ideas. The claim recites the step of generating an edge existence probability mask representing confidence levels associated with identifying each of the one or more edges of a surface in the at least the portion of the building. Under the broadest reasonable interpretation, this step is a process that generates an edge existence probability mask representing confidence data to identify edges of a building and is merely a mathematical concept. This step merely employs mathematical relationships to manipulate existing information to generate additional information in the form of an “edge existence probability mask”. This step falls within the mathematical concept grouping of abstract ideas. The claim further recites the step of executing an optimization procedure using the depth information the edge existence probability mask to optimize the probability mask. Under the broadest reasonable interpretation, this step is a process of optimizing the probability mask and is merely a mathematical concept. This step merely employs mathematical relationships to manipulate existing information to generate additional information in the form of an “optimized probability mask”. This step falls within the mathematical concept grouping of abstract ideas. The claim also recites the step of identifying the one or more edges of the surface based on the optimized edge existence probability mask is a process that under the broadest reasonable interpretation, can be performed as a mental process in the human mind by observation, evaluation, judgment and opinion. This step falls within mental process grouping of abstract ideas. See MPEP 2106.04(a)(2), subsection III. Accordingly, the claim recites an abstract idea.
The claim further recites the additional steps of receiving image data of at least a portion of a building using an imaging device and receiving depth information of at least a portion of a building using a depth sensor. These additional elements amounts to mere data gathering, i.e. capturing and sending image data and capturing and sending depth information, that will be used to for further processing by the system. Using an imaging device and a depth sensor to acquire image data and depth information does not impose any meaningful limit on the claim. It is necessary to acquire data in order to use the recited judicial exceptions to perform the mental processes recited by the claim. These elements are recited at a high level of generality, and thus are insignificant extra-solution activity. See MPEP 2105(g)(“whether the limitation is significant”). Therefore, the judicial exception is not integrated into a practical application. The claim does not recite any additional elements or a combination of elements to integrate the abstract idea into a practical application. Therefore, the claim is directed to an abstract idea.
The claim further recites the additional elements of one or more memories and one or more processors executing instructions stored on the one or more memories to perform the steps recited in the claim. These elements are recited at a high level of generality and do no impose any meaningful limit on the claim. The judicial exception is not integrated into a practical application because these additional elements only serve to allow the mental process and mathematical concepts to be performed, which is nothing more than instructions to implement the abstract idea on a generic computer. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements as recited only serve to implement the judicial exception on a computing device with no improvement of the computing device itself, and therefore is not an inventive concept. Therefore, the claim is not patent eligible.
Claims 38-40 similarly recites abstract ideas without any additional elements that are sufficient to amount to significantly more than the judicial exception, and therefore are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 21-22, 26, 28-30, 34-37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tang et al. (US 2021/0225090, hereinafter Tang), and further in view of Hung et al. (US 2014/0002441, hereinafter Hung).
Regarding claim 21, Tang teaches a method (abstract: devices, systems, and methods that generate floorplans and measurements using a three-dimensional (3D) representation of a physical environment generated based on sensor data) comprising:
receiving an image (image data) and depth information (data from depth camera 402, fig. 4) of at least a portion (room) of a building ([0027]: obtaining 3D semantic data of a physical environment generated based on depth data and light intensity image data obtained during a scanning process. For example, a 3D point cloud may be generated based on depth camera information received concurrently with the images during a room scan; [0097]: a system can generate a semantic 3D representation using 3D data and semantic segmentation data based on depth and light intensity image information detected in the physical environment; [0125]: The system flow of the example environment 800A acquires image data (e.g., live camera feed from light intensity camera 406) of a physical environment (e.g., the physical environment 105 of FIG. 1), a semantic 3D representation (e.g., semantic 3D representation 445) from the semantic 3D unit 440, and other sources of physical environment information (e.g., camera positioning information) at the floorplan finalization unit 850 (e.g., floorplan unit 246 of FIG. 2, and/or live floorplan 346 of FIG. 3));
generating, based on the image, an edge existence probability mask (initial edge map 823, fig. 8B) identifying each of the one or more edges of a surface in the at least the portion of the building ([0104]: the preliminary floorplan creation process includes a 2D top-down view of a room based on separately identifying wall structures (e.g., wall edges, door, and windows); [0130]: The edge map neural network 822 generates an initial edge map 823 of the identified walls, and classifies corners 824a-824g (herein referred to as corners 824));
optimizing the edge existence probability mask (initial edge map 823, fig. 8B) using the depth information (data from depth camera 402, fig. 4) to obtain an optimized edge existence probability mask (refined edge map 829, fig. 8B) ([0021]: walls may be identified by generating 2D semantic data (e.g., in layers), using the 2D semantic data to generate an edge map using a neural network, and determining vector parameters to standardize the edge map in a 3D normalized plan. Wall attributes or wall attributes (e.g., doors/windows) may be identified based on RGB images and depth data to generate polygon boundaries; [0077]: the measurement unit 248 obtains a finalized edge map and associated depth data for the walls, 2D outlines and associated depth data for identified wall attributes, and bounding boxes (e.g., refined bounding boxes) for identified objects from the floorplan unit 244. The measurement unit 248 is configured with instructions executable by a processor to generate measurement data based on the 3D representation for the walls identified on the edge map, measurement data for the identified boundaries of the wall attributes, and measurement data for the bounding boxes of the identified objects using one or more processes further disclosed herein with reference to FIGS. 8 and 12; [0119]: At block 704, the method 700 detects positions of wall structures in the physical environment based on the 3D representation. For example, walls may be identified by generating 2D semantic data (e.g., in layers), using the 2D semantic data to generate an edge map using a neural network, and determining vector parameters to standardize the edge map in a 3D normalized plan. Wall attributes or wall attributes (e.g., doors/windows) may be identified based on RGB images and depth data to generate polygon boundaries. This technique for doors and windows provides advantages, especially due to transparency of windows which creates noise/errors in depth data; [0122]: the method 700 further includes detecting positions of wall structures in the physical environment based on the 3D representation includes identifying walls and wall attributes (e.g., doors and windows) of the physical environment from the wall structures based on the 3D representation, and generating an edge map of the identified walls and the wall attributes based on the 3D representation, wherein the 2D floorplan is based on the generated edge map that includes the identified walls and identified wall attributes.; [0130]: The edge map 823 is then refined by the line fitting unit 826 using a line fitting algorithm to generate a line fitted edge map 827. The line fitted edge map 827 is then further refined by the small walls neural network 828 which further classifies and distinguishes each corner to generate a refined edge map 829. For example, corner 824a and 824e was initially identified as a standard corner by the acquired data, but the small walls neural network 828 is trained to identify corners that may actually be a pillar or an indented corner such that a finalized floorplan should reflect for accuracy and completeness. Additionally, corner 824d may actually be an open passthrough to an adjacent room, and not a wall as initially indicated by the edge map. The refined edge map 829 is then sent to the floorplan finalization unit 850);
identifying the one or more edges of the surface (wall edges) based on the optimized edge existence probability mask ([0107]: a preliminary floorplan creation process for the live preview and/or post processing provides a 2D top-down view of a room based on identifying wall structures (wall edges) based on a 2D representation that encodes 3D semantic data in multiple layers; [0110]: the method 500 further includes generating the live preview of the preliminary 2D floorplan by generating an edge map by identifying walls in the physical environment based on the 3D representation, updating the edge map by identifying wall attributes (e.g., doors and windows) in the physical environment based on the 3D representation, updating the edge map by identifying objects in the physical environment based on the 3D representation, and generating the live preview of the preliminary 2D floorplan based on the updated edge map that includes the identified walls, identified wall attributes, and identified objects; [0114]: The edge mapping unit 612 and line fitting unit 613 are utilized to generate and refine an edge map based on the semantic 3D representation for the identified walls using one or more of the techniques disclosed herein. For example, edge mapping unit 612 obtains 3D data (e.g., semantic 3D representation 445) for the identified semantically labeled walls from the semantic 3D unit 440, and generates an initial 2D edge map of the identified walls, and the line fitting unit 613 generates refined 2D edge map using a line fitting algorithm; [0130]: The walls unit 820 includes an edge map neural network 822, line fitting unit 826, and a small walls neural network 828. The system flow of the example environment 800B begins where the edge map neural network 822 acquires a semantic 3D representation (e.g., semantic 3D representation 445), which includes 3D data of identified walls. The edge map neural network 822 generates an initial edge map 823 of the identified walls, and classifies corners 824a-824g (herein referred to as corners 824). The edge map 823 is then refined by the line fitting unit 826 using a line fitting algorithm to generate a line fitted edge map 827. The line fitted edge map 827 is then further refined by the small walls neural network 828 which further classifies and distinguishes each corner to generate a refined edge map 829. For example, corner 824a and 824e was initially identified as a standard corner by the acquired data, but the small walls neural network 828 is trained to identified corners that may actually be a pillar or an indented corner such that a finalized floorplan should reflect for accuracy and completeness. Additionally, corner 824d may actually be an open passthrough to an adjacent room, and not a wall as initially indicated by the edge map. The refined edge map 829 is then sent to the floorplan finalization unit 850).
Tang does not explicitly teach the edge map represents a confidence level associated with identifying each of the one or more edges of a surface.
Hung teaches the edge map (temporally-consistent edge map 140) represents a confidence level (probability) associated with each of the one or more edges of a surface ([0035]: A temporally-consistent edge map 140 of an image from the binocular sequence 110 is generated by determining the probability of a pixel in an image being an object boundary using the segmentation map 130 and the long-range pixel trajectory 120 so that the edges in an image are identified and depth boundary can be preserved when generating a depth map using such a temporally-consistent edge map 140). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Hung’s knowledge of using edge maps that represents probability of a pixel in an image being an object boundary so that edges in an image are identified and modify the process of Tang because such a system is capable of suppressing random foreground artifacts to a large extent, and greatly suppresses the flickering artifacts that improves temporal consistency of depth maps ([0021]).
Regarding claim 22, the combination of Tang and Hung teaches the method of claim 21, further comprising: generating a pictorial rendering of the at least the portion of the building, the pictorial rendering including the one or more edges of the surface (Tang - fig. 6 shows a display preview of the floorplan in 2D displaying wall edges, windows, door, etc.; Tang - [0104]: The method 500 is a process that creates a live preview of a preliminary floorplan being displayed during room scanning (e.g., while walking around with a device, such as a smartphone or tablet). To enable a live preview of the preliminary floorplan, the preview may be generated (at least initially) differently than a final post-scan floorplan (e.g., additional post processing techniques for fine-tuning, increased accuracy for measurement data, etc.). For example, a live preview may use a less computationally intensive neural network or provide an initial floorplan without fine-tuning (e.g., corner correction techniques). The use of 2D semantic data (e.g., for different layers of the room) may also facilitate making the preview determination efficient for live display. According to some implementations, the preliminary floorplan creation process includes a 2D top-down view of a room based on separately identifying wall structures (e.g., wall edges, door, and windows) and detecting bounding boxes for objects (e.g., furniture, appliances, etc.). Additionally, or alternatively, a preliminary floorplan creation process for the live preview and/or post processing provides a 2D top-down view of a room based on identifying wall structures (wall edges) based on a 2D representation that encodes 3D semantic data in multiple layers; Tang - [0115]: as a user scans a room with a device's camera(s), the acquired image data is continuously updating, thus the edge map, wall attribute boundaries, and bounding boxes for objects can be continuously updating with each iteration of updated image data. The floorplan preview unit 610 sends the preliminary 2D floorplan preview feed (e.g., preview 2D floorplan 630) and the live camera feed to the device display 312. The device display 312 can display the live view (e.g., light intensity image data 408) and a picture-in-picture (PIP) display 620 that includes the preview 2D floorplan 630. The preview 2D floorplan 630 includes edge map walls 632a, 632b, 632c (e.g., representing walls 134, 130, 132, respectively), boundary 634a (e.g., representing door 150), boundary 634b (e.g., representing window 152), bounding box 636a (e.g., representing table 142), and bounding box 636b (e.g., representing chair 140)).
Regarding claim 26, the combination of Tang and Hung teaches the method of claim 21, wherein the edge existence probability mask encompasses an area that straddles each of one or more edges of the surface and extends over at least a portion of a length of each of the one or more edges of the surface (Tang - area 824a, 824d or 824e, fig. 8; Tang - [0130]: The edge map neural network 822 generates an initial edge map 823 of the identified walls, and classifies corners 824a-824g (herein referred to as corners 824). The edge map 823 is then refined by the line fitting unit 826 using a line fitting algorithm to generate a line fitted edge map 827. The line fitted edge map 827 is then further refined by the small walls neural network 828 which further classifies and distinguishes each corner to generate a refined edge map 829. For example, corner 824a and 824e was initially identified as a standard corner by the acquired data, but the small walls neural network 828 is trained to identify corners that may actually be a pillar or an indented corner such that a finalized floorplan should reflect for accuracy and completeness. Additionally, corner 824d may actually be an open passthrough to an adjacent room, and not a wall as initially indicated by the edge map).
Regarding claim 28, Tang teaches a method (abstract: devices, systems, and methods that generate floorplans and measurements using a three-dimensional (3D) representation of a physical environment generated based on sensor data) comprising:
receiving, from an imaging device (depth camera 402 and light intensity camera 406, fig. 4), image data (image data) of at least a portion of a structure ([0027]: obtaining 3D semantic data of a physical environment generated based on depth data and light intensity image data obtained during a scanning process. For example, a 3D point cloud may be generated based on depth camera information received concurrently with the images during a room scan; [0097]: a system can generate a semantic 3D representation using 3D data and semantic segmentation data based on depth and light intensity image information detected in the physical environment; [0125]: The system flow of the example environment 800A acquires image data (e.g., live camera feed from light intensity camera 406) of a physical environment (e.g., the physical environment 105 of FIG. 1), a semantic 3D representation (e.g., semantic 3D representation 445) from the semantic 3D unit 440, and other sources of physical environment information (e.g., camera positioning information) at the floorplan finalization unit 850 (e.g., floorplan unit 246 of FIG. 2, and/or live floorplan 346 of FIG. 3));
receiving depth information associated with the at least the portion of the structure ([0113]: The SLAM system may include a multidimensional (e.g., 3D) laser scanning and range measuring system that is GPS-independent and that provides real-time simultaneous location and mapping. The SLAM system may generate and manage data for a very accurate point cloud that results from reflections of laser scanning from objects in an environment. Movements of any of the points in the point cloud are accurately tracked over time, so that the SLAM system can maintain precise understanding of its location and orientation as it travels through an environment, using the points in the point cloud as reference points for the location);
generating, based on at least one of the image data or a combination of the image data and the depth information (depth data and light intensity image data), semantic information (S-3D, fig. 4; semantic 3D representation using 3D data and semantic segmentation data, [0097]) of the at least the portion of the structure ([0020]: obtaining a 3D representation of a physical environment generated based on depth data and light intensity image data obtained during a scanning process. For example, a 3D point cloud may be generated based on depth camera information received concurrently with the images during a room scan. For example, algorithms may be used for semantic segmentation and labeling of 3D point clouds of indoor scenes, where objects in point clouds can have significant variations and complex configurations; [0027]: obtaining 3D semantic data of a physical environment generated based on depth data and light intensity image data obtained during a scanning process. For example, a 3D point cloud may be generated based on depth camera information received concurrently with the images during a room scan; [0098]: The system flow of the example environment 400 acquires image data of a physical environment (e.g., the physical environment 105 of FIG. 1) and the 3D representation unit 410 (e.g., 3D representation unit 242 of FIG. 2, and/or 3D representation unit 342 of FIG. 3) generates a semantic 3D representation 445 representing the surfaces in a 3D environment using a 3D point cloud with associated semantic labels. In some implementations, the semantic 3D representation 445 is a 3D reconstruction mesh using a meshing algorithm based on depth information detected in the physical environment that is integrated (e.g., fused) to recreate the physical environment);
generating, based on the image data, a probability mask (initial edge map 823, fig. 8B) identifying each of one or more structural features (wall structures (e.g., wall edges, door, and windows); [0104]: the preliminary floorplan creation process includes a 2D top-down view of a room based on separately identifying wall structures (e.g., wall edges, door, and windows); [0130]: The edge map neural network 822 generates an initial edge map 823 of the identified walls, and classifies corners 824a-824g (herein referred to as corners 824));
optimizing the probability mask using the depth information to obtain an optimized the probability mask (refined edge map 829, fig. 8B; [0021]: walls may be identified by generating 2D semantic data (e.g., in layers), using the 2D semantic data to generate an edge map using a neural network, and determining vector parameters to standardize the edge map in a 3D normalized plan. Wall attributes or wall attributes (e.g., doors/windows) may be identified based on RGB images and depth data to generate polygon boundaries; [0077]: the measurement unit 248 obtains a finalized edge map and associated depth data for the walls, 2D outlines and associated depth data for identified wall attributes, and bounding boxes (e.g., refined bounding boxes) for identified objects from the floorplan unit 244. The measurement unit 248 is configured with instructions executable by a processor to generate measurement data based on the 3D representation for the walls identified on the edge map, measurement data for the identified boundaries of the wall attributes, and measurement data for the bounding boxes of the identified objects using one or more processes further disclosed herein with reference to FIGS. 8 and 12; [0119]: At block 704, the method 700 detects positions of wall structures in the physical environment based on the 3D representation. For example, walls may be identified by generating 2D semantic data (e.g., in layers), using the 2D semantic data to generate an edge map using a neural network, and determining vector parameters to standardize the edge map in a 3D normalized plan. Wall attributes or wall attributes (e.g., doors/windows) may be identified based on RGB images and depth data to generate polygon boundaries. This technique for doors and windows provides advantages, especially due to transparency of windows which creates noise/errors in depth data; [0122]: the method 700 further includes detecting positions of wall structures in the physical environment based on the 3D representation includes identifying walls and wall attributes (e.g., doors and windows) of the physical environment from the wall structures based on the 3D representation, and generating an edge map of the identified walls and the wall attributes based on the 3D representation, wherein the 2D floorplan is based on the generated edge map that includes the identified walls and identified wall attributes.; [0130]: The edge map 823 is then refined by the line fitting unit 826 using a line fitting algorithm to generate a line fitted edge map 827. The line fitted edge map 827 is then further refined by the small walls neural network 828 which further classifies and distinguishes each corner to generate a refined edge map 829. For example, corner 824a and 824e was initially identified as a standard corner by the acquired data, but the small walls neural network 828 is trained to identify corners that may actually be a pillar or an indented corner such that a finalized floorplan should reflect for accuracy and completeness. Additionally, corner 824d may actually be an open passthrough to an adjacent room, and not a wall as initially indicated by the edge map. The refined edge map 829 is then sent to the floorplan finalization unit 850);
identifying the one or more structural features (wall edges) based on combining the optimized probability mask with the semantic information ([0107]: a preliminary floorplan creation process for the live preview and/or post processing provides a 2D top-down view of a room based on identifying wall structures (wall edges) based on a 2D representation that encodes 3D semantic data in multiple layers; [0110]: the method 500 further includes generating the live preview of the preliminary 2D floorplan by generating an edge map by identifying walls in the physical environment based on the 3D representation, updating the edge map by identifying wall attributes (e.g., doors and windows) in the physical environment based on the 3D representation, updating the edge map by identifying objects in the physical environment based on the 3D representation, and generating the live preview of the preliminary 2D floorplan based on the updated edge map that includes the identified walls, identified wall attributes, and identified objects; [0114]: The edge mapping unit 612 and line fitting unit 613 are utilized to generate and refine an edge map based on the semantic 3D representation for the identified walls using one or more of the techniques disclosed herein. For example, edge mapping unit 612 obtains 3D data (e.g., semantic 3D representation 445) for the identified semantically labeled walls from the semantic 3D unit 440, and generates an initial 2D edge map of the identified walls, and the line fitting unit 613 generates refined 2D edge map using a line fitting algorithm; [0130]: The walls unit 820 includes an edge map neural network 822, line fitting unit 826, and a small walls neural network 828. The system flow of the example environment 800B begins where the edge map neural network 822 acquires a semantic 3D representation (e.g., semantic 3D representation 445), which includes 3D data of identified walls. The edge map neural network 822 generates an initial edge map 823 of the identified walls, and classifies corners 824a-824g (herein referred to as corners 824). The edge map 823 is then refined by the line fitting unit 826 using a line fitting algorithm to generate a line fitted edge map 827. The line fitted edge map 827 is then further refined by the small walls neural network 828 which further classifies and distinguishes each corner to generate a refined edge map 829. For example, corner 824a and 824e was initially identified as a standard corner by the acquired data, but the small walls neural network 828 is trained to identified corners that may actually be a pillar or an indented corner such that a finalized floorplan should reflect for accuracy and completeness. Additionally, corner 824d may actually be an open passthrough to an adjacent room, and not a wall as initially indicated by the edge map. The refined edge map 829 is then sent to the floorplan finalization unit 850).
Tang does not explicitly teach the edge map represents a confidence level associated with identifying each of one or more structural features.
Hung teaches the edge map (temporally-consistent edge map 140) represents a confidence level (probability) associated with identifying each of one or more structural features (edges; [0035]: A temporally-consistent edge map 140 of an image from the binocular sequence 110 is generated by determining the probability of a pixel in an image being an object boundary using the segmentation map 130 and the long-range pixel trajectory 120 so that the edges in an image are identified and depth boundary can be preserved when generating a depth map using such a temporally-consistent edge map 140). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Hung’s knowledge of using edge maps that represents probability of a pixel in an image being an object boundary so that edges in an image are identified and modify the process of Tang because such a system is capable of suppressing random foreground artifacts to a large extent, and greatly suppresses the flickering artifacts that improves temporal consistency of depth maps ([0021]).
Regarding claim 36, Tang teaches a system (abstract: devices, systems, and methods that generate floorplans and measurements using a three-dimensional (3D) representation of a physical environment generated based on sensor data) comprising:
one or more memories (memory 320, fig. 3); and
one or more processors (processing units 302, fig. 3), the one ore more processors configured to execute the instructions stored in the one or more memories to:
receive, from an imaging device (depth camera 402 and light intensity camera 406, fig. 4), image data (image data) of at least a portion of a building ([0027]: obtaining 3D semantic data of a physical environment generated based on depth data and light intensity image data obtained during a scanning process. For example, a 3D point cloud may be generated based on depth camera information received concurrently with the images during a room scan; [0097]: a system can generate a semantic 3D representation using 3D data and semantic segmentation data based on depth and light intensity image information detected in the physical environment; [0125]: The system flow of the example environment 800A acquires image data (e.g., live camera feed from light intensity camera 406) of a physical environment (e.g., the physical environment 105 of FIG. 1), a semantic 3D representation (e.g., semantic 3D representation 445) from the semantic 3D unit 440, and other sources of physical environment information (e.g., camera positioning information) at the floorplan finalization unit 850 (e.g., floorplan unit 246 of FIG. 2, and/or live floorplan 346 of FIG. 3));
receive, from a depth sensor ([0113]: The SLAM system may include a multidimensional (e.g., 3D) laser scanning and range measuring system that is GPS-independent and that provides real-time simultaneous location and mapping), depth information associated with the at least the portion of the building ([0113]: The SLAM system may include a multidimensional (e.g., 3D) laser scanning and range measuring system that is GPS-independent and that provides real-time simultaneous location and mapping. The SLAM system may generate and manage data for a very accurate point cloud that results from reflections of laser scanning from objects in an environment. Movements of any of the points in the point cloud are accurately tracked over time, so that the SLAM system can maintain precise understanding of its location and orientation as it travels through an environment, using the points in the point cloud as reference points for the location);
generate, based on the image data and the depth information (depth data and light intensity image data), semantic information (S-3D, fig. 4; semantic 3D representation using 3D data and semantic segmentation data, [0097]) of the at least the portion of the building ([0020]: obtaining a 3D representation of a physical environment generated based on depth data and light intensity image data obtained during a scanning process. For example, a 3D point cloud may be generated based on depth camera information received concurrently with the images during a room scan. For example, algorithms may be used for semantic segmentation and labeling of 3D point clouds of indoor scenes, where objects in point clouds can have significant variations and complex configurations; [0027]: obtaining 3D semantic data of a physical environment generated based on depth data and light intensity image data obtained during a scanning process. For example, a 3D point cloud may be generated based on depth camera information received concurrently with the images during a room scan; [0098]: The system flow of the example environment 400 acquires image data of a physical environment (e.g., the physical environment 105 of FIG. 1) and the 3D representation unit 410 (e.g., 3D representation unit 242 of FIG. 2, and/or 3D representation unit 342 of FIG. 3) generates a semantic 3D representation 445 representing the surfaces in a 3D environment using a 3D point cloud with associated semantic labels. In some implementations, the semantic 3D representation 445 is a 3D reconstruction mesh using a meshing algorithm based on depth information detected in the physical environment that is integrated (e.g., fused) to recreate the physical environment);
generate, based on the image data, a probability mask (initial edge map 823, fig. 8B) identifying each of one or more structural features (wall structures (e.g., wall edges, door, and windows); [0104]: the preliminary floorplan creation process includes a 2D top-down view of a room based on separately identifying wall structures (e.g., wall edges, door, and windows); [0130]: The edge map neural network 822 generates an initial edge map 823 of the identified walls, and classifies corners 824a-824g (herein referred to as corners 824));
execute, to obtain an optimized probability mask, an optimization procedure using the depth information to optimize the probability mask (refined edge map 829, fig. 8B; [0021]: walls may be identified by generating 2D semantic data (e.g., in layers), using the 2D semantic data to generate an edge map using a neural network, and determining vector parameters to standardize the edge map in a 3D normalized plan. Wall attributes or wall attributes (e.g., doors/windows) may be identified based on RGB images and depth data to generate polygon boundaries; [0077]: the measurement unit 248 obtains a finalized edge map and associated depth data for the walls, 2D outlines and associated depth data for identified wall attributes, and bounding boxes (e.g., refined bounding boxes) for identified objects from the floorplan unit 244. The measurement unit 248 is configured with instructions executable by a processor to generate measurement data based on the 3D representation for the walls identified on the edge map, measurement data for the identified boundaries of the wall attributes, and measurement data for the bounding boxes of the identified objects using one or more processes further disclosed herein with reference to FIGS. 8 and 12; [0119]: At block 704, the method 700 detects positions of wall structures in the physical environment based on the 3D representation. For example, walls may be identified by generating 2D semantic data (e.g., in layers), using the 2D semantic data to generate an edge map using a neural network, and determining vector parameters to standardize the edge map in a 3D normalized plan. Wall attributes or wall attributes (e.g., doors/windows) may be identified based on RGB images and depth data to generate polygon boundaries. This technique for doors and windows provides advantages, especially due to transparency of windows which creates noise/errors in depth data; [0122]: the method 700 further includes detecting positions of wall structures in the physical environment based on the 3D representation includes identifying walls and wall attributes (e.g., doors and windows) of the physical environment from the wall structures based on the 3D representation, and generating an edge map of the identified walls and the wall attributes based on the 3D representation, wherein the 2D floorplan is based on the generated edge map that includes the identified walls and identified wall attributes.; [0130]: The edge map 823 is then refined by the line fitting unit 826 using a line fitting algorithm to generate a line fitted edge map 827. The line fitted edge map 827 is then further refined by the small walls neural network 828 which further classifies and distinguishes each corner to generate a refined edge map 829. For example, corner 824a and 824e was initially identified as a standard corner by the acquired data, but the small walls neural network 828 is trained to identify corners that may actually be a pillar or an indented corner such that a finalized floorplan should reflect for accuracy and completeness. Additionally, corner 824d may actually be an open passthrough to an adjacent room, and not a wall as initially indicated by the edge map. The refined edge map 829 is then sent to the floorplan finalization unit 850);
identify the one or more structural features (wall edges) based on combining the optimized probability mask with the semantic information ([0107]: a preliminary floorplan creation process for the live preview and/or post processing provides a 2D top-down view of a room based on identifying wall structures (wall edges) based on a 2D representation that encodes 3D semantic data in multiple layers; [0110]: the method 500 further includes generating the live preview of the preliminary 2D floorplan by generating an edge map by identifying walls in the physical environment based on the 3D representation, updating the edge map by identifying wall attributes (e.g., doors and windows) in the physical environment based on the 3D representation, updating the edge map by identifying objects in the physical environment based on the 3D representation, and generating the live preview of the preliminary 2D floorplan based on the updated edge map that includes the identified walls, identified wall attributes, and identified objects; [0114]: The edge mapping unit 612 and line fitting unit 613 are utilized to generate and refine an edge map based on the semantic 3D representation for the identified walls using one or more of the techniques disclosed herein. For example, edge mapping unit 612 obtains 3D data (e.g., semantic 3D representation 445) for the identified semantically labeled walls from the semantic 3D unit 440, and generates an initial 2D edge map of the identified walls, and the line fitting unit 613 generates refined 2D edge map using a line fitting algorithm; [0130]: The walls unit 820 includes an edge map neural network 822, line fitting unit 826, and a small walls neural network 828. The system flow of the example environment 800B begins where the edge map neural network 822 acquires a semantic 3D representation (e.g., semantic 3D representation 445), which includes 3D data of identified walls. The edge map neural network 822 generates an initial edge map 823 of the identified walls, and classifies corners 824a-824g (herein referred to as corners 824). The edge map 823 is then refined by the line fitting unit 826 using a line fitting algorithm to generate a line fitted edge map 827. The line fitted edge map 827 is then further refined by the small walls neural network 828 which further classifies and distinguishes each corner to generate a refined edge map 829. For example, corner 824a and 824e was initially identified as a standard corner by the acquired data, but the small walls neural network 828 is trained to identified corners that may actually be a pillar or an indented corner such that a finalized floorplan should reflect for accuracy and completeness. Additionally, corner 824d may actually be an open passthrough to an adjacent room, and not a wall as initially indicated by the edge map. The refined edge map 829 is then sent to the floorplan finalization unit 850).
Tang does not explicitly teach the edge map represents a confidence level associated with identifying each of one or more structural features.
Hung teaches the edge map (temporally-consistent edge map 140) represents a confidence level (probability) associated with identifying each of one or more structural features (edges; [0035]: A temporally-consistent edge map 140 of an image from the binocular sequence 110 is generated by determining the probability of a pixel in an image being an object boundary using the segmentation map 130 and the long-range pixel trajectory 120 so that the edges in an image are identified and depth boundary can be preserved when generating a depth map using such a temporally-consistent edge map 140). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Hung’s knowledge of using edge maps that represents probability of a pixel in an image being an object boundary so that edges in an image are identified and modify the process of Tang because such a system is capable of suppressing random foreground artifacts to a large extent, and greatly suppresses the flickering artifacts that improves temporal consistency of depth maps ([0021]).
Claims 29 and 37 are similar in scope to claim 22, and therefore the examiner provides similar rationale to reject these claims.
Regarding claim 30, the combination of Tang and Hung teaches the method of claim 28, wherein generating the semantic information comprises: executing a scene understanding procedure to identify at least one of: one or more surfaces in the structure (identifying wall edges, door and windows of a building; Tang – [0006]: The generation of floorplans and measurements is facilitated in some implementations using semantically-labelled 3D representations of a physical environment. Some implementations perform semantic segmentation and labeling of 3D point clouds of a physical environment. Techniques disclosed herein may achieve various advantages by using semantic 3D representations, such as a semantically labeled 3D point cloud, encoded onto a two-dimensional (2D) lateral domain. Using semantic 3D representations in 2D lateral domains may facilitate the efficient identification of structures used to generate a floorplan or measurement; Tang – [0009]: a floorplan may be generated based on separately identifying wall structures (e.g., wall edges, door, and windows) and detecting bounding boxes for objects (e.g., furniture, appliances, etc.). The wall structures and objects may be detected separately and thus using differing techniques and the results combined to generate a floorplan that represents both the wall structures and the objects; Tang – [0010]: a floorplan creation process identifies wall structures (e.g., wall edges) based on a 2D representation that encodes 3D semantic data in multiple layers. For example, 3D semantic data may be segmented into a plurality of horizontal layers that are used to identify where the wall edges of the room are located; Tang – [0110]: The 3D representation unit 410 further includes a semantic unit 430 that is configured with instructions executable by a processor to obtain the light intensity image data (e.g., light intensity data 408) and semantically segment wall structures (wall, doors, windows, etc.) and object type (e.g., table, teapot, chair, vase, etc.) using one or more known techniques. For example, the semantic unit 430 receives intensity image data 408 from the image sources (e.g., light intensity camera 406), and generates semantic segmentation data 432 (e.g., RGB-S data). For example, the semantic segmentation 434 illustrates a semantically labelled image of the physical environment 105 in FIG. 1. In some implementations, semantic unit 430 uses a machine learning model, where a semantic segmentation model may be configured to identify semantic labels for pixels or voxels of image data. In some implementations, the machine learning model is a neural network (e.g., an artificial neural network), decision tree, support vector machine, Bayesian network, or the like), orientation of the one or more surfaces, or dimensional measurements of the one or more surfaces (Tang – [0007]: A floorplan may include indications of measurements of boundaries, wall edges, doors, windows, and objects in a room, e.g., including numbers designating a length of a wall, a diameter of a table, a width of a window, etc.; Tang – [0011]: According to some implementations, measurements of a room's wall attributes (e.g., walls, doors, and windows) and objects (e.g., furniture, appliances, etc.) may be acquired using different techniques. For example, for wall attributes, such as doors and windows, light intensity images (e.g., RGB images) may be utilized to generate boundaries (2D polygonal shapes) in addition to or instead of depth data. This may provide various advantages, for example, in circumstances in which depth data may be skewed due to the transparency of windows and doors that may include windows. After the 2D polygonal shapes are determined from the light intensity images, depth data or 3D representations based on the depth data (e.g., a 3D semantic point cloud) can then be used to determine specific measurements of the door or window. In some implementations, objects are measured by first generating 3D bounding boxes for the object based on the depth data, refining the bounding boxes using various neural networks and refining algorithms described herein, and acquiring measurements based on the refined bounding boxes and the associated 3D data points for the respective bounding boxes; Tang – [0041]: a measurement of a boundary associated with a measurement of a particular wall attribute includes a length, a width, and a height of the particular wall attribute. For example, the length, width, and height of a door).
Regarding claim 34, the combination of Tang and Hung teaches the method of claim 28, wherein the probability mask encompasses an area that straddles each of one or more structural features and extends over at least a portion of a length of each of the one or more structural features (Tang - area 824a, 824d or 824e, fig. 8), and wherein the optimizing the probability mask comprises executing a neural network procedure (Tang - [0130]: The edge map neural network 822 generates an initial edge map 823 of the identified walls, and classifies corners 824a-824g (herein referred to as corners 824). The edge map 823 is then refined by the line fitting unit 826 using a line fitting algorithm to generate a line fitted edge map 827. The line fitted edge map 827 is then further refined by the small walls neural network 828 which further classifies and distinguishes each corner to generate a refined edge map 829. For example, corner 824a and 824e was initially identified as a standard corner by the acquired data, but the small walls neural network 828 is trained to identify corners that may actually be a pillar or an indented corner such that a finalized floorplan should reflect for accuracy and completeness. Additionally, corner 824d may actually be an open passthrough to an adjacent room, and not a wall as initially indicated by the edge map).
Regarding claim 35, the combination of Tang and Hung teaches the method of claim 28, wherein the at least the portion of the structure includes a first room (fig. 1, fig. 4, fig. 6 shows a room; [0007]: a floorplan includes a 2D top-down view of a room), and wherein at least one of the one or more structural features is occluded by an object that is present in the first room (as shown in fig. 6, a chair occludes a partial wall of the room).
Claim(s) 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tang, in view of Hung, and further in view of Peng et al. (CN 109408954 A, hereinafter Peng).
Regarding claim 23, the combination of Tang and Hung teaches the method of claim 22, wherein the at least the portion of the building includes a room (Tang - fig. 1, fig. 4, fig. 6 shows a room; Tang - [0007]: a floorplan includes a 2D top-down view of a room), further comprising: displaying the pictorial rendering of the room upon a display screen (Tang - fig. 6 shows a display preview of the floorplan in 2D displaying wall edges, windows, door, etc.; Tang - [0104]: The method 500 is a process that creates a live preview of a preliminary floorplan being displayed during room scanning (e.g., while walking around with a device, such as a smartphone or tablet). To enable a live preview of the preliminary floorplan, the preview may be generated (at least initially) differently than a final post-scan floorplan (e.g., additional post processing techniques for fine-tuning, increased accuracy for measurement data, etc.). For example, a live preview may use a less computationally intensive neural network or provide an initial floorplan without fine-tuning (e.g., corner correction techniques). The use of 2D semantic data (e.g., for different layers of the room) may also facilitate making the preview determination efficient for live display. According to some implementations, the preliminary floorplan creation process includes a 2D top-down view of a room based on separately identifying wall structures (e.g., wall edges, door, and windows) and detecting bounding boxes for objects (e.g., furniture, appliances, etc.). Additionally, or alternatively, a preliminary floorplan creation process for the live preview and/or post processing provides a 2D top-down view of a room based on identifying wall structures (wall edges) based on a 2D representation that encodes 3D semantic data in multiple layers; Tang - [0115]: as a user scans a room with a device's camera(s), the acquired image data is continuously updating, thus the edge map, wall attribute boundaries, and bounding boxes for objects can be continuously updating with each iteration of updated image data. The floorplan preview unit 610 sends the preliminary 2D floorplan preview feed (e.g., preview 2D floorplan 630) and the live camera feed to the device display 312. The device display 312 can display the live view (e.g., light intensity image data 408) and a picture-in-picture (PIP) display 620 that includes the preview 2D floorplan 630. The preview 2D floorplan 630 includes edge map walls 632a, 632b, 632c (e.g., representing walls 134, 130, 132, respectively), boundary 634a (e.g., representing door 150), boundary 634b (e.g., representing window 152), bounding box 636a (e.g., representing table 142), and bounding box 636b (e.g., representing chair 140)).
However, the combination of Tang and Hung does not explicitly teach receiving a query associated with an interior design of the room; and generating a response to the query, the response comprising one of a recommendation to place an object at a location in the room.
Peng teaches receiving a query associated with an interior design of the room (page 6 paragraph 7: the user can be on the web page of the client by selecting options or input mode of character, to select client web page display of decoration goods, decorating goods selected by the user is the target decoration goods, the user can select one or more decoration goods, so that the number of the target commodity can be one or more client end by the user selecting the target commodity information (such as the names of the commodities belonging to the style or the like) to the design device (processor or server). designing device (processor or server) according to the house type information input by the user, determining the target goods selected by the user are placed in which room, placing room of the target commodity is the target room); and generating a response to the query, the response comprising one of a first recommendation to place an object at a location in the room (page 6 paragraph 7: designing device (processor or server) according to the house type information input by the user, determining the target goods selected by the user are placed in which room, placing room of the target commodity is the target room; page 6 last paragraph: said determining user selection of decoration goods is placed after the corresponding target room, according to preference information of the user, comprising an indoor design style liked by user. configuring the corresponding decorative element on the personal interest of the user (and other information can indirectly reflect interior design properties liked by user), the processor or server screening indoor designing style selected by user corresponding to the decoration material database, and according to preset design rule selected by the user in the target room decoration commodity; page 7 paragraph 2: A10, contained in according to the preference information of the user to the function determined by the target room, room, the pattern size of the target and the target commodity of the target room, inquiring pre-stored indoor design rule. obtaining the target decoration corresponding to the article decorating element, the interior design rule from the existing indoor design, using machine learning method obtained room function, size of room, the layout of the room and decorating goods and mapping relationship between the decoration element; page 7 last paragraph – page 8 first paragraph: the client end transmits the said information to the processor or server, processor or server according to the user selection of the target goods and house according to layout information to determine room (determining the placing room of the target commodity is the target room) for placing the target goods selected by the user, indoor designing style information processor or server according to the indoor designing style information preference information of the user, inquiring the indoor designing style found in the database selected by the user and the query corresponding to the indoor designing style of decorating material database, The decoration goods selected by the user. searching the material database, selecting the selected by the user matched with the decoration product decoration element (decoration element according to the indoor designing style of the user and the user-selected target commodity, selecting target goods selected by the user in the interior design scheme matched with the decoration material). after the user selects the commodity matching and good decorating element, processor or server according to the size of the target room, layout of the room, room orientation and interior design rule, adjusting the position relation of the target goods and decoration elements, so as to obtain better design, a design model of the target building). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Peng’s knowledge of recommending products to be placed in the displayed room floorplan and modify the process of Tang and Hung because such a system improves customer’s personalized design experience (page 2 paragraph 4).
Regarding claim 24, the combination of Tang, Hung and Peng teaches the method of claim 23, wherein the object comprises one of a piece of furniture (bed), a decorative object, or a utility object, and wherein the recommendation is based at least in part on matching a first property (placement or position) of the object with a second property (placement or position) of surface (Peng – page 8 paragraph 4: bed, processor or server according to target goods (bed) and layout information (two room one-hall thereafter) selected by the user. determining a user selected goods (bed) is put on is confirmed to be in the room of the bedroom, a processor or server extracts the decoration material of European style in the decorating material database according to the user indoor designing style information (European style), and according to the user selected commodity (bed) and interior design rule, decorating elements (such as cabinet, wardrobe, etc.) the extracted European style decoration goods (bed) in the material database for user selected corresponding, processor or server according to the size and pattern of the bedroom, orientation and interior design rule, inquiring to obtain position mapping relation between bed and bedside cabinet, wardrobe of indoor in the design rule; page 9 first paragraph: user selected corresponding, processor or server according to the size and pattern of the bedroom, orientation and interior design rule, configuring user selection of decoration goods (bed) and a decoration element (night, the position relation of the wardrobe and so on), for example, the bedstand can be placed close to the bed, placed near the wall wardrobe, bedside cupboard placed close to the wall and so on, processor or server according to the size of the target room, layout of the room, room orientation, and interior design rule, adjusting the target position relationship between goods (bed) and a decoration element (cabinet, wardrobe) so as to obtain better design, a design model of the target building; page 9 paragraph 6: the bedside corresponding style of bed, when the night is placed close to the bed, one side of the bed placed against the wall and the bedside cupboard is placed close to the wall. The indoor design style the user likes and interior design-rule to the matching score weighting to obtain 98; when placed close to the bed, the bed is not close to the wall but the bedside cupboard near wall. The indoor design style of the user and interior design rule to the matching score weighting to obtain 88 minutes, when placed close to the bed, the bed is not close to the wall and the bedside cupboard near wall, The indoor design style of the user and interior design rule to the matching score weighting to obtain 78; the bedside cupboard is not close to the bed, the bed is not close to the wall and bed head cabinet is not close to the wall. The indoor design style of the user and interior design rule to the weighted score to obtain 68. Collocating way of said four kinds of ordering from high to low, selecting the grade of design models, so as to screen out the cabinet bed, placed against the wall on the one side of the bed and bedside cupboard placed close to the wall of the placing method).
Claim(s) 25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tang, in view of Hung, in view of Peng, and further in view Beijing Urban Network Neighbor Information Technology Co., Ltd. (CN 111986305 A, hereinafter Beijing).
Regarding claim 25, the combination of Tang, Hung and Peng does not explicitly teach the method of claim 24 wherein the first property comprises at least one of a size of the object, a shape of the object, or a color of the object and wherein the second property comprises at least one of a size of the surface, a shape of the surface, or a color of the surface.
Beijing teaches the first property comprises at least one of a size of the object (page 3 paragraph 10: according to the wall object and the size information, generating furniture layout data of the at least one spatial object), a shape of the object, or a color of the object and wherein the second property comprises at least one of a size of the surface, a shape of the surface, or a color of the surface (page 3 paragraph 9: the spatial attribute at least comprises the at least one space object in the three-dimensional space of the wall object and size information, the step of obtaining the furniture layout data of the at least one spatial object according to the space attribute; page 3 paragraphs 11-14: Optionally, under the condition that the space object is the restaurant object, the size information comprises the first side length and the second side length of the restaurant object, the step of generating furniture layout data of the at least one spatial object according to the wall object and the size information, comprising: obtaining the furniture model object matched with the first side length and the second side length of the restaurant object; and furniture size of the furniture model object; according to the wall body type of the wall object, obtaining the target wall object of the furniture model object; using the furniture size and the target wall object, generating position information of the furniture model object). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Beijing’s knowledge of using the size of the wall and the size of the object for obtaining furniture layout data and modify the process of Tang, Hung and Peng because such a system provides the user with different decoration solution, and sensing and displaying different decoration style, and thereby improving the user experience (page 2: abstract).
Claim(s) 27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tang, in view of Hung, and further in view of Liu et al. (US 2009/0252382, hereinafter Liu).
Regarding claim 27, the combination of Tang and Hung teaches the method of claim 21, wherein generating the edge existence probability mask comprises: executing a segmentation procedure upon a section of the image data that includes a visible portion of at least one edge of the one or more edges (Tang – [0006]: Some implementations perform semantic segmentation and labeling of 3D point clouds of a physical environment; 3D semantic data may be segmented into a plurality of horizontal layers that are used to identify where the wall edges of the room are located; Tang – [0101]: 3D semantic data may be segmented into a plurality of horizontal layers that are used to identify where the wall edges of the room are located).
However, the combination of Tang and Hung do not explicitly teach executing a segmentation procedure upon a section of the image that includes an occluded portion of the at least one edge.
Liu teaches executing a segmentation procedure upon a section of the image that includes an occluded portion of the at least one edge ([0010]: a method for segmentation of an obtained iris image having at least one occluded region therein is provided. Canny transform is performed on the obtained iris image to identify intensity gradients representing edge points within the iris image; performing a circular Hough transform on a plurality of the edge points to identify the pupillary and limbic boundaries within the iris image; performing at least one Radon transform to define two straight line segments each representing a boundary of the occluded region, wherein the occluded region is further bounded by one or more borders of the image; and removing the region bounded by the two straight line segments and the one or more borders from the iris image). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Liu’s knowledge of segmenting an image having at least one occluded region and modify the process of Tang and Hung because such a system more accurately represents the boundary ([0009]).
Claim(s) 31 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tang, in view of Hung, and further in view of Wang et al. (CN 106446844, hereinafter Wang).
Regarding claim 31, the combination of Tang and Hung does not explicitly the method of claim 28, wherein generating the probability mask comprises: extracting key points from the image data; generating a feature map based on the extracted key points; and executing an optimization process to determine a confidence level for each of the one or more structural features based on the feature map.
Wang teaches generating the probability mask (abstract: establishing body part score map) comprises: extracting key points from the image data (page 14 paragraph 4: extracting features from an input image corresponding to the body part of the object; page 15 last paragraph and page 16 1st paragraph: The term "characteristic pattern (feature map)" is intended to represent feature extracted from the image. each point in the feature map corresponding to one image block, the central point of the image block corresponding to the points in the feature map for the image; page 18 paragraph 12: In FIG. 2 the process (2.1), the feature f is divided into several groups, by to represent each group corresponding to the human body of a key point); generating a feature map based on the extracted key points (page 14 paragraph 4: generate a feature map; page 19 paragraph 4: extracting unit 401 extracting features from an input image corresponding to the body part of the object, to generate a feature map); and executing an optimization process to determine a confidence level for each of the one or more structural features based on the feature map (page 14 paragraph 4: predicting the condition of edge probability at least one point of the characteristic pattern in the updated; establishing body part score graph according to the condition edge probability of the prediction by the body part score map; page 16 paragraph 10: using the updated characteristic pattern along the predetermined graph model of the object in the first direction and the second direction and the product (sum-landmark) algorithm, in order to predict the condition of edge probability at least one point in the feature map of the update; page 19 paragraph 5: a graph model updating unit 402 based on the updated object characteristic graph, and establishing a body part score from the feature map of the update).
The combination of Tang and Hung contains a “base” process of identifying edges of a building using an optimized probability map which the claimed invention can be seen as an “improvement” in that the confidence level of the edges in the probability map is determined using feature map.
Wang contains a known technique of using a feature map to determine a probability map representing the confidence level for edges (page 14 paragraph 4, page 15 last paragraph and page 16 1st paragraph, page 16 paragraph 10, page 18 paragraph 12, page 19 paragraph 4 and page 19 paragraph 5) that is applicable to the base process of the combination of Tang and Hung.
Wang’s known technique of using a feature map to determine a probability map representing the confidence level for edges would have been recognized by one skilled in the art as applicable to the “base” process of the combination of Tang and Hung and the results would have been predictable and resulted determining the posture or orientation of the structures which results in an improved process. Therefore, the claimed subject matter would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention.
Claim(s) 32, 38-39 is/are rejected under 35 U.S.C. 103 as being obvious over Tang, in view of Hung, and further in view of Beyhaghi et al. (US 2022/0207202, hereinafter Beyhaghi).
The applied reference has a common inventor with the instant application. Based upon the earlier effectively filed date of the reference, it constitutes prior art under 35 U.S.C. 102(a)(2).
This rejection under 35 U.S.C. 103 might be overcome by: (1) a showing under 37 CFR 1.130(a) that the subject matter disclosed in the reference was obtained directly or indirectly from the inventor or a joint inventor of this application and is thus not prior art in accordance with 35 U.S.C.102(b)(2)(A); (2) a showing under 37 CFR 1.130(b) of a prior public disclosure under 35 U.S.C. 102(b)(2)(B); or (3) a statement pursuant to 35 U.S.C. 102(b)(2)(C) establishing that, not later than the effective filing date of the claimed invention, the subject matter disclosed and the claimed invention were either owned by the same person or subject to an obligation of assignment to the same person or subject to a joint research agreement. See generally MPEP § 717.02.
Regarding claim 32, the combination of Tang and Hung does not explicitly teach the method of claim 28, wherein optimizing the probability mask comprises: generating a pose graph based on successive frames of the image data; forming three-dimensional point cloud fragments by projecting two-dimensional pixels from the image data into three-dimensional space; generating a global pose graph by matching corresponding features in the three- dimensional point cloud fragments; and executing a floorplan estimation procedure using the global pose graph.
Beyhaghi teaches generating a pose graph based on successive frames of the image data ([0108]: The odometry information is obtained from successive frames of an RGB image and used to generate a pose graph); forming three-dimensional point cloud fragments by projecting two-dimensional pixels from the image data into three-dimensional space ([0108]: 3D point cloud fragments may be formed by projecting 2D pixels into a 3D space); generating a global pose graph by matching corresponding features in the three- dimensional point cloud fragments ([0108]: A global pose graph may then be generated by matching corresponding features in various 3D fragments and by use of a feedback generation procedure); and executing a floorplan estimation procedure using the global pose graph ([0108]: The global pose graph may be used for executing a floorplan estimation procedure). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Beyhaghi’s knowledge of executing a floorplan estimation procedure as taught and modify the process of Tang and Hung because such a process can include an optimization pipeline to fit alpha shapes (linear simple curves that can be used for shape reconstruction) with a deep learning pipeline to predict the best-fit corners of each room point cloud. As a result, polygons that best describe each of the rooms present in the global point cloud are estimated. Finally, these polygons are stitched together by referring to the global point cloud and can be used as a usable 2D floor map of a room or a set of rooms ([0108]).
Regarding claim 38, the combination of Tang and Hung does not explicitly teach the system of claim 36, wherein to generate the semantic information, the one or more processors are further configured to execute the instructions to: generate a top view mean normal rendering of the image data; generate a top view projection rendering of the image data; and execute a room segmentation procedure on at least one of the top view mean normal rendering or the top view projection rendering to identify individual rooms in the building.
Beyhaghi teaches the one or more processors are further configured to execute the instructions to ([0063]: FIG. 7 shows a block diagram 700 of a method to generate a floorplan in accordance with an embodiment of the disclosure. The functional blocks shown in the block diagram 700 can be implemented by executing a software program in a computer, such as, for example, the computer 130): generate a top view mean normal rendering of the image data ([0065]: The three-dimensional polygonal mesh representation can be converted to a top view mean normal rendering (block 710) and a top view projection rendering (block 725)); generate a top view projection rendering of the image data ([0065]: The three-dimensional polygonal mesh representation can be converted to a top view mean normal rendering (block 710) and a top view projection rendering (block 725)); and execute a room segmentation procedure on at least one of the top view mean normal rendering or the top view projection rendering to identify individual rooms in the building ([0066]: Stage 1 of room segmentation (block 715) may involve segmenting the top view mean normal rendering and/or the top view projection rendering into individual rooms. The segmenting procedure may involve the use of procedures such as, for example, machine learning, density-based spatial clustering of applications (DBSCAN), and random sample consensus (RANSAC)). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Beyhaghi’s knowledge of using the room segmentation procedure as taught and modify the system of Tang and Hung because such a system efficiently generates the floorplan using machine learning for room segmentation (fig. 7 and [0066]).
Regarding claim 39, the combination of Tang and Hung does not explicitly teach the system of claim 36, wherein to execute the optimization procedure, the one or more processors are further configured to execute the instructions to: execute a sequential model to perform room segmentation; execute a graph-based model to identify relationships between multiple rooms; and generate a global point cloud by matching corresponding features between rooms.
Beyhaghi teaches the one or more processors are further configured to execute the instructions to ([0097][: FIG. 18 illustrates an individual 10 executing a floorplan generation procedure upon a computer 15 in accordance with an embodiment of the disclosure. In this example scenario, the computer 15 is configured to operate as a floorplan generating device): execute a sequential model to perform room segmentation ([0102]: The fully autonomous operation is generally executed in accordance with the disclosure and can, in one example implementation, involve the use of machine learning models such as, for example, a sequential model that performs room segmentation procedures and a graph-based model that identifies relationships between various rooms; [0104]: Furthermore, in some scenarios, the third example floorplan generation procedure (and/or the second floorplan generation procedure) may generate some room properties through room sequence prediction using a sequence model. The sequence model may be applied to one or more rooms. It may be desirable to generate two sets of room data in order to obtain information on individual rooms as well to identify how two or more rooms are interconnected); execute a graph-based model to identify relationships between multiple rooms ([0102]: The fully autonomous operation is generally executed in accordance with the disclosure and can, in one example implementation, involve the use of machine learning models such as, for example, a sequential model that performs room segmentation procedures and a graph-based model that identifies relationships between various rooms; [0105]: converting template floorplans into graphs and using a model that represents graph learning, is one example process to obtain information on how the rooms are interconnected with each other. In an example approach, each room is assumed to be a node and shared walls are assumed as edges. Since graphs do not show a special relationship across rooms, each room may be assigned coordinates in a coordinate plane. For doing so, a graph-to-image algorithm converts a graph of a floorplan to a list of coordinate points, one for each room); and generate a global point cloud by matching corresponding features between rooms ([0108]: A global pose graph may then be generated by matching corresponding features in various 3D fragments and by use of a feedback generation procedure. The global pose graph may be used for executing a floorplan estimation procedure. The floorplan estimation procedure can include an optimization pipeline to fit alpha shapes (linear simple curves that can be used for shape reconstruction) with a deep learning pipeline to predict the best-fit corners of each room point cloud. As a result, polygons that best describe each of the rooms present in the global point cloud are estimated. Finally, these polygons are stitched together by referring to the global point cloud and can be used as a usable 2D floor map of a room or a set of rooms). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Beyhaghi’s knowledge of using a sequential model and a graph-based model as taught and modify the system of Tang and Hung because such a system efficiently generates the floorplan based on the captured information ([0108]).
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Lee et al. (US 2018/0268220) describes identifying the one or more structural features comprises: determining a room type from among a plurality of room types based on maximizing an objective function for each room type ([0071]: When applying a recurrent structure with an encoder-decoder architecture, each layer in the network receives gradients not only across depth but also through time steps between the input and the final objective function during training; claim 1: determine, using the encoder-decoder sub-network, the classifier sub-network and the room image, a predicted room type from the plurality of room types); identifying key points associated with the determined room type (claim 1: determine, using the encoder-decoder sub-network and the room image, a plurality of predicted two-dimensional (2D) keypoint maps corresponding to a plurality of room types … determine, using the plurality of predicted 2D keypoint maps and the predicted room type, a plurality of ordered keypoints associated with the predicted room type).
Allowable Subject Matter
Based on the cited prior art references, the following subject matter is considered as allowable:
“constraining movement of the key points within areas defined by the probability mask”.
Based on the cited prior art references, the following subject matter is also considered as allowable:
“determine a layout relationship between multiple rooms based on identifying interconnecting doors; identify external walls based on evaluating a layout of objects on a periphery of the building; and determine size parameters including at least one of gross living area or net floor area based on the identified external walls”.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JWALANT B AMIN whose telephone number is (571)272-2455. The examiner can normally be reached Monday-Friday 10am - 630pm CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said Broome can be reached at 571-272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JWALANT AMIN/Primary Examiner, Art Unit 2612
Prosecution Insights