DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 05/08/2026 has been entered.
Response to Amendment
This Office Action is in response to Applicant’s amendment/response filed on 05/08/2026, which has been entered and made of record.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-11, and 13-18 are rejected under 35 U.S.C. 103 as being unpatentable over Foley et al. (US 12,197,829 B2, hereinafter “Foley”) in view of Lv et al. (Residential floor plan recognition and reconstruction, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), DOI: 10.1109/CVPR46437.2021.01644, pp. 16712-16721, 20-25 June 2021, hereinafter “Lv”), Rieffel et al. (US 2010/0214284 A1, hereinafter “Rieffel”), and Dai et al. (ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans, arXiv.org, arXiv:1712.10215v2 [cs.CV] 28 Mar 2018, hereinafter “Dai”).
Regarding claim 1, Foley discloses A method, comprising:
receiving, at a server over a network, a raster image of a floor plan of a structure; (col. 35, lines 5-13, “At step 1001, the method includes receiving into a controller a design plan (or a first two-dimensional representation) of at least a portion of a building. As described above, the design plan may include an architectural drawing, floor plan, design drawing and the like. At step 1002, the portion of a design plan (or a first two-dimensional representation) may be represented as a raster image”; col. 11, lines 50-53, “a degree of the processing as described herein may be performed on a controller, which may include a cloud server, a standalone computing device or a smart device”). Note that: a cloud server via a network can perform receiving the raster image of a design plan (floor plan) of at least a portion of a building.
identifying, by the server, semantic information of the floor plan from the raster image; … wherein the semantic information comprises information relevant to rendering of a 3D model, including interior or exterior finishes; (col. 13, lines 30-41, “At step 104, an AI engine may ascertain features included in the design plan, the AI engine may additionally ascertain that a feature is located within a particular set of boundaries or external to the set of boundaries. Features may include … architectural aspects, fixtures, duct work, wiring, piping, or other item included in a two-dimensional reference submitted to be analyzed. The features and boundaries may be determined, for example, via algorithmically processing an input design plan image with a trained AI model the AI engine may process a raster file that is converted for output as an image file of a floorplan”; col. 10, lines 43-51, “AI generated values for parameters may also be useful in a variety of estimation elements, such as (without limitation): flooring (wood, ceramic, carpet, tile, etc.), structural (poured concrete, steel), walls (gypsum, masonry blocks, glass walls, wall base, exterior cladding), doors and frames (hollow metal, wood, glass), windows glazing, insulation, paint (ceilings and walls), acoustical ceilings, code compliance, stucco (ceilings and walls), mechanical, plumbing, and electrical aspects.”). Note that: (1) an AI engine may ascertain or identify the features or semantic information (architectural aspects, fixtures, duct work, wiring, piping, etc.) of the floor plan; (2) the AI engine may process the raster file as the input design plan (floor plan) image with the features; and (3) paint (ceilings and walls) and stucco (ceilings and walls) can be regarded as interior or exterior finishes.
Updating … in response to additional information. (col. 14, lines 12-18, “At step 106, a controller is operative to generate an interactive user interface with dynamic components that may be manipulated by one or both of user interaction and automated processes. Any or all of the components in a user interface may be converted to a version that allows a user to modify an attribute of the components, such as the length, size, beginning point, end point, thickness, or other attribute”; col. 15, lines 19-28, “At step 106A, in some embodiments, components presented in the interactive user interface may be analyzed by a user and refinements may be made to one or more components (e.g., size, shape and/or position of the component). In some embodiments, user modifications may also be input back to the AI engine to train the AI engine. User modifications provided back to the AI Engine may be referenced to make subsequent AI processes more accurate, efficient, fast, trained and/or enable additional types of AI processes”; col. 36, lines 17-20, “Any or all of steps 1101-1107 may be repeated for different portions of the two-dimensional reference descriptive of the building. For example, for a second design plan representing a different portion of the entire building design”). Note that: (1) when a user changes or modifies the attributes of the dynamic components through the user interface, it means additional information regarding the dynamic component is input to a floor plan management system, and it becomes a condition for possible action of the system; (2) the additional information can provide back to or update the input to the AI Engine to enable additional types of AI processes; (3) a second design plan (e.g., a second floor plan raster image) can be regarded as a updated floor plan raster image; and (4) the updated floor plan raster image can also be the additional information.
at the server … by the server … (col. 11, “a degree of the processing as 50 described herein may be performed on a controller, which may include a cloud server, a standalone computing device or a smart device”). Note that: a cloud server can perform the processing.
However, Foley fails to disclose, but in the same art of computer graphics, Lv discloses
generating, by the server, a vector image from the raster image and identified semantic information by processing the raster image through a machine learning model, … (Lv, page 16717, Figure 1: “Pixel (top) and vector (bottom) floor plan image. The pixel floor plan image (top) contains various information, such as room structural elements (magenta), text (blue), symbols (cyan), scales (yellow)”; page 16719, Figure 2: the raster floor plan image on the left is processed with a neural network in the middle to detect, extract, or calculate ROIs, structural elements, text, symbols, and scale, and a vector floor plan image on the right and upper side is generated with “Vectorization” and the identified semantic information; page 16719, col. left, paras. 2-4, “YOLOv4 [11] is used as our basic detection model of region of interest (ROI) detection module … The DeepLabv3+ [12] is adopted as our basic network”; page 16720, col. right, paras. 1-2, “YOLOv4 [11] is our basic network … The network is designed to a modified FCN network [24, 32], and backbone is Resnet50 [16]”; page 16721, col. left, para. 2, “we expect to convert raster information (structural elements, text and symbols) into vector graphics. The segmented floor plan elements can be divided into two categories [33], one is room-boundary pixels, and the other is room-type pixels. The room-boundary pixels represent walls, doors, windows and doorways, while room-type pixels are living room, bathroom, bedroom, etc. The core idea is to obtain vectorized contour information of room type pixels for each room, then calculate the center line of the wall based on the contour information and room boundary pixels. The room type can be refined by the text and symbol detection results. The position information of the doors, windows and doorways can be obtained according to the segmentation and regression results. Detailed steps are shown in Algorithm 1”). Note that: (1) the generation of a vector image can be regarded as a combination of processing a raster image using a machine learning model (YOLOv4, modified YOLOv4, DeepLabv3+, FCN, and Resnet50 based neural network) to detect or extract, or calculate ROIs, structural elements, text and symbols, and scale, and performing vectorization to generate a vector image of a floor plan; (2) the ROIs, structural elements, text and symbols, and scale, can be regarded as the similar identified semantic information as identified by Foley above; and (3) the vector image with vectorization is generated or outputted.
generating, by the server, the 3D model from the vector image and the semantic information, and any predicted semantic information (Lv, page 16719, Figure 2: with “Post-processing” a 3D model on the lower-right side is generated from the vector image with “Vectorization” and “the multi-modal information of the floor plan, such as room structure, text, symbols and scale”; page 16723, Figure 3:” From left to right, an input floor plan image, semantic segmentation result with post-processing, reconstructed vector-graphics representation with room type annotation, the corresponding 3D reconstruction model”). Note that: (1) the multi-modal information of the floor plan, such as room structure, text, symbols and scale can be regarded as the semantic information; and (2) the semantic information and any predicted semantic information for the missing semantic information are combined to form the semantic information for generating the 3D model.
receiving, at the server, a selection of a format for the 3D model: (Lv, Abstract, “An automatic framework is provided that accurately recognizes the structure, type, and size of the room, and outputs vectorized 3D reconstruction results”; page 16719, Figure 2: with “Post-processing” a 3D model on the lower-right side is generated). Note that: the vectorized 3D reconstruction results are regarded as the 3D model with a format (vectorized 3D reconstruction) that is output from the framework. This format of 3D model can be selected as input for a 3D model render later.
updating, by the server, the vector image and 3D model in response to additional information. (Lv, Figure 1: “Pixel (top) and vector (bottom) floor plan image. The pixel floor plan image (top) contains various information, such as room structural elements (magenta), text (blue), symbols (cyan), scales (yellow)”; page 16719, Figure 2: the raster floor plan image on the left side is processed with a neural network in the middle to detect, extract, or calculate ROIs, structural elements, text, symbols, and scale, and a vector floor plan image on the right and upper side is generated with “Vectorization” and the identified semantic information; page 16723, Figure 3:” From left to right, an input floor plan image, semantic segmentation result with post-processing, reconstructed vector-graphics representation with room type annotation, the corresponding 3D reconstruction model”). Note that: (1) Foley discloses above that the users can change or update the parameters of dynamic components of the floor plan and update the input to the AI Engine to enable additional types of AI processes; (2) in combination with Foley’s teaching here, it is obvious to one having ordinary skills in the art that one can repeat the processes of generation of the vector floor plan image and the 3D model above to update them accordingly at the cloud server over a network while the server can perform the processing and store the corresponding updated vector floor plan image(s) and 3D model(s).
Foley and Lv are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply generating a vector floor plan image and generating a 3D model, as taught by Lv into Foley. The motivation would have been “The system significantly increases accuracy and generalization ability, compared with existing methods.” (Lv, page 16717, Abstract). The suggestion for doing so would allow to improve accuracy and generality of floor plan vectorization and 3D model. Therefore, it would have been obvious to combine Foley with Lv.
However, Foley in view of Lv fails to disclose, but in the same art of computer graphics, Rieffel discloses
wherein the semantic information comprises information relevant to rendering of a 3D model (Rieffel, Figure 4A: the process from Semantic Information of “Semantic Information Database 240”, to “Model Builder 226”, to “Model Rendering Instruction Database 242”, to “Model Rendering Instructions 404”, to “Three Dimensional Model Renderer”, to rendering of “Model 120”). Note that: the sematic information of “Semantic Information Database 240” is the information relevant to rendering of a 3D model.
rendering, by the server, the 3D model based on the selected format: (Rieffel, Figure 4A: the process from Semantic Information of “Semantic Information Database 240”, to “Model Builder 226”, to “Model Rendering Instruction Database 242”, to “Model Rendering Instructions 404”, to “Three Dimensional Model Renderer”, to rendering of “Model 120”; page 7, para. [0062], “This defined model may then be rendered (386) by a 3D model renderer, as described in greater detail below with reference to FIGS. 4A-4C.”). Note that: the 3D model defined by the selected format can be rendered by a 3D render.
the semantic information is at least partially obtained from one or more documents that are separate from the raster image, and (Rieffel, page 13, para. [0109], “Alternative methods of associating semantic information with predefined markers 112-1 are also envisioned. For example, markers could be stored in an electronic device capable of producing predefined markers 112-1. A user could then provide an indication of the required semantic information to the electronic device which could, in accordance with some embodiments, produce one or more predefined markers and store the user indicated semantic information with information indicative of those markers in a semantic information database 240 for use in defining a model that is based at least in part on images of a physical space 110 as described herein”). Note that: (1) some sematic information can be associated with the predefined markers by the electronic device while the semantic information with information indicative of those markers is indicated by the user; and (2) this sematic information is separated from the raster image, and is obtained or indicated by the users from the documents in the semantic information database 240.
Foley in view of Lv, and Rieffel are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply rendering a 3D model based on select format with the semantic information relevant to rendering of a 3D model, as taught by Rieffel into Foley in view of Lv. The motivation would have been “This defined model may then be rendered (386) by a 3D model renderer” (Rieffel, page 7, para. [0062]). The suggestion for doing so would allow to render a 3D model with the semantic information relevant to rendering of a 3D model to have a realistic quality rendering. Therefore, it would have been obvious to combine Foley, Lv, and Rieffel.
However, the combination of Foley, Lv, and Rieffel fails to disclose, but in the same art of computer graphics, Dai disclose wherein the machine learning model is trained to further predict any additional missing semantic information needed to render the 3D model; (Dai, page 3, Figure 1: “we propose a hierarchical coarse-to-fine approach, where each level takes a partial 3D scan as input, and predicts a completed scan as well as per-voxel semantic labels at the respective level’s voxel resolution using our autoregressive 3D CNN architecture, “
PNG
media_image1.png
370
918
media_image1.png
Greyscale
”; page 1, Abstract, “we devise a fully-convolutional generative 3D CNN model whose filter kernels are invariant to the overall scene size. The model can be trained on scene subvolumes but deployed on arbitrarily large scenes at test time. In addition, we propose a coarse-to-fine inference strategy in order to produce high-resolution output while also leveraging large input context sizes”). Note that: through a CNN neural network that can be trained on scene subvolumes, the input scan data as the input can be inferred into predicated semantics (semantic classes; sematic information) including predicting any missing semantic information even for previously missing geometry.
The combination of Foley, Lv, and Rieffel, and Dai, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply predicting the missing semantics for the missing geometry, as taught by Dai into Foley in view of Lv and Rieffel. The motivation would have been “we outperform other methods not only in the size of the environments handled and processing efficiency, but also with regard to completion quality and semantic segmentation performance by a significant margin.” (Dai, page 1, Abstract). The suggestion for doing so would allow to predict any additional missing semantic information needed to render the 3D model. Therefore, it would have been obvious to combine Foley, Lv, Rieffel, and Dai.
Regarding claim 2, the combination of Foley, Lv, Rieffel, and Dai discloses The method of claim 1, further comprising training, prior to generating the vector image, the machine learning model. (Lv, page 16719, Figure 2: the raster floor plan image on the left is processed with a neural network in the middle to detect, extract, or calculate ROIs, structural elements, text, symbols, and scale, and a vector floor plan image on the right and upper side is generated with “Vectorization” and the identified semantic information; page 16719, col. left, paras. 2-4, “YOLOv4 [11] is used as our basic detection model of region of interest (ROI) detection module … The DeepLabv3+ [12] is adopted as our basic network”; page 16720, col. right, paras. 1-2, “YOLOv4 [11] is our basic network … The network is designed to a modified FCN network [24, 32], and backbone is Resnet50 [16]”; page 16721, col. left, para. 2, “we expect to convert raster information (structural elements, text and symbols) into vector graphics. The segmented floor plan elements can be divided into two categories [33], one is room-boundary pixels, and the other is room-type pixels. The room-boundary pixels represent walls, doors, windows and doorways, while room-type pixels are living room, bathroom, bedroom, etc. The core idea is to obtain vectorized contour information of room type pixels for each room, then calculate the center line of the wall based on the contour information and room boundary pixels. The room type can be refined by the text and symbol detection results. The position information of the doors, windows and doorways can be obtained according to the segmentation and regression results. Detailed steps are shown in Algorithm 1”; page 16718, col. right, para. 3, “We have crawled 7,000 Residential Floor Plan (RFP) data from Internet search engines, which are mainly home floor plans of Chinese urban buildings”). Note that: (1) the generation of a vector image can be regarded as a combination of processing a raster image using a machine learning model (YOLOv4, modified YOLOv4, DeepLabv3+, FCN, and Resnet50 based neural network) to detect or extract, or calculate ROIs, structural elements, text and symbols, and scale, and performing vectorization to generate a vector image of a floor plan; and (2) before generating the vector image, it is obvious that the machine learning model (YOLOv4, modified YOLOv4, DeepLabv3+, FCN, and Resnet50 based neural network) has been trained with training dataset (RFT dataset).
Regarding claim 3, the combination of Foley, Lv, Rieffel, and Dai discloses The method of claim 1, wherein the additional input comprises an updated raster image of the floor plan. (Foley, col. 36, lines 17-20, “Any or all of steps 1101-1107 may be repeated for different portions of the two-dimensional reference descriptive of the building. For example, for a second design plan representing a different portion of the entire building design”). Note that: (1) a second design plan (e.g., a second floor plan raster image) can be regarded as an updated floor plan raster image; and (2) the updated floor plan raster image can be regarded as the additional information.
Regarding claim 4, the combination of Foley, Lv, and Rieffel discloses The method of claim 1, wherein the additional input comprises a scan of a physical space corresponding to the floor plan. (Foley, col. 33, lines 31-38, “A LiDAR sensing system 951 may also be incorporated into the mobile device 902. The LiDAR system may include a scannable laser light (or other collimated) light source which may operate at nonvisible wavelengths such as in the infrared. An associated sensor device, sensitive to the light of emission may be included in the system to record time and strength of returned signal that is reflected off of surfaces in the environment of the mobile device 902”). Note that: when a user holds a mobile device with a LiDAR in the environment of the corresponding floor plan, he can use scan the physical space; and (2) the scanned data can be regarded as the additional input to be input back to the AI engine as described by Foley above.
Regarding claim 6, the combination of Foley, Lv, Rieffel, and Dai discloses The method of claim 1, wherein the semantic information comprises room segmentation information and non-visible structures. (Foley, col. 13, lines 30-41, “At step 104, an AI engine may ascertain features included in the design plan, the AI engine may additionally ascertain that a feature is located within a particular set of boundaries or external to the set of boundaries. Features may include … architectural aspects, fixtures, duct work, wiring, piping, or other item included in a two-dimensional reference submitted to be analyzed. The features and boundaries may be determined, for example, via algorithmically processing an input design plan image with a trained AI model the AI engine may process a raster file that is converted for output as an image file of a floorplan”; col. 30, lines 49-51, “In a segmentation phase used to determine boundaries of regions or other space features”). Note that: the semantic information includes the features including non-visible structures (e.g., some duct work, wiring, piping) and boundaries of regions that indicate the room segmentation information,
Regarding claim 7, the combination of Foley, Lv, Rieffel, and Dai discloses The method of claim 6, wherein the non-visible structures include one or more of wiring, plumbing, ventilation ducts, insulation, and structural members. (Foley, col. 13, lines 30-41, “At step 104, an AI engine may ascertain features included in the design plan, the AI engine may additionally ascertain that a feature is located within a particular set of boundaries or external to the set of boundaries. Features may include … architectural aspects, fixtures, duct work, wiring, piping, or other item included in a two-dimensional reference submitted to be analyzed”). Note that: the non-visible structures include duct work (e.g., ventilation ducts), wiring, and piping (plumbing).
Claim 8 reciting “A non-transitory computer-readable medium (CRM) comprising instructions that, when executed by the processor of an apparatus, cause the apparatus to:” is corresponding to the method of claim 1. Therefore, claim 8 is rejected for the same rationale for claim 1.
In addition, the combination of Foley, Lv, Rieffel, and Dai discloses A non-transitory computer-readable medium (CRM) comprising instructions that, when executed by the processor of an apparatus, cause the apparatus to: (Foley, col. 32, lines 10-14, “The storage device 803 can store a software program 804 with executable logic for controlling the processor 802. The processor 802 performs instructions of the software program 804, and thereby operates in accordance with the present disclosure”). Note that: The storage device 803 is a non-transitory computer-readable medium (CRM).
Claims 9-11 are corresponding to the method of claims 2-4, respectively. Therefore, claims 9-11 are rejected for the same rationale for claims 2-4, respectively.
Regarding claim 13, the combination of Foley, Lv, Rieffel, and Dai discloses The CRM of claim 8, wherein the apparatus is a server, and the raster image is received from a remote user device over a network. (Foley, col. 31, lines 60-63, “The controller may be included in one or more of the apparatuses described above, such as a Server, and a Network Access Device”; col. 11, lines 37-49, “a drawing or other design plan may be stored in paper format or digital version or may not exist or may never have existed. The input may also be in any raster graphics image … The input process may occur with a user creating, scanning into, or accessing such a file containing a raster graphics image or a vector graphics image. The user may access the file on a desktop or standalone computing device or, in some embodiments, via an application running on a smart device … a user may operate a scanner or a smart device with a charged couple device to create the file containing the image on the smart device”; col. 33, line 3-5, “The mobile device 902 also includes a network interface 916 to communicate data to a network and/or an associated computing device”). Note that: (1) the apparatuses are a Server, and a Network Access Device; (2) a user away physically from the server can be regarded as a remote user, and can create a raster graphics image file of the design plan (floor plan) using a smart device (mobile device) with a CCD device (e.g., camera); and (3) a mobile device can use a network interface over network to communicate data (raster image file and other data) with an associated device (the server) that receives the raster image file and other data.
Regarding claim 14, the combination of Foley, Lv, Rieffel, and Dai discloses The CRM of claim 8, wherein the additional input comprises one or more diagrams of wiring, plumbing, ventilation ducts, insulation, and structural members. (Foley, col. 13, lines 30-41, “At step 104, an AI engine may ascertain features included in the design plan, the AI engine may additionally ascertain that a feature is located within a particular set of boundaries or external to the set of boundaries. Features may include … architectural aspects, fixtures, duct work, wiring, piping, or other item included in a two-dimensional reference submitted to be analyzed. The features and boundaries may be determined, for example, via algorithmically processing an input design plan image with a trained AI model the AI engine may process a raster file that is converted for output as an image file of a floorplan”). Note that: (1) an AI engine may ascertain or identify the features of the floor plan located within a particular set of boundaries or external to the set of boundaries; and (2) the presentations of the features of architectural aspects, fixtures, duct work, wiring, piping, etc. can be regarded as the corresponding diagrams.
Claim 15 reciting “A system, comprising: a server; and a network interface in communication with the server, wherein the server includes instructions to cause the server, when the instructions are executed by a processor of the server, to:” is corresponding to the method of claim 1. Therefore, claim 15 is rejected for the same rationale for claim 1.
In addition, the combination of Foley, Lv, Rieffel, and Dai discloses A system, comprising: a server; and a network interface in communication with the server, wherein the server includes instructions to cause the server, when the instructions are executed by a processor of the server, to: (Foley, lines 60-67, “The controller may be included in one or more of the apparatuses described above, such as a Server, and a Network Access Device. The controller 800 includes a processor unit 802, such as one or more semiconductor-based processors, coupled to a communication device 801 configured to communicate via a communication network (not shown in FIG. 8)”; col. 32, lines 10-14, “The storage device 803 can store a software program 804 with executable logic for controlling the processor 802. The processor 802 performs instructions of the software program 804, and thereby operates in accordance with the present disclosure”). Note that: The controller can be regarded as a system.
Regrading claim 16, the combination of Foley, Lv, Rieffel, and Dai discloses The system of claim 15, wherein the scan is received from a first remote user device over the network interface, and the additional input is received from a second remote user device over the network interface. (Foley, col. 31, lines 60-63, “The controller may be included in one or more of the apparatuses described above, such as a Server, and a Network Access Device”; col. 11, lines 37-49, “a drawing or other design plan may be stored in paper format or digital version or may not exist or may never have existed. The input may also be in any raster graphics image … The input process may occur with a user creating, scanning into, or accessing such a file containing a raster graphics image or a vector graphics image. The user may access the file on a desktop or standalone computing device or, in some embodiments, via an application running on a smart device … a user may operate a scanner or a smart device with a charged couple device to create the file containing the image on the smart device”; col. 33, line 3-5, “The mobile device 902 also includes a network interface 916 to communicate data to a network and/or an associated computing device”). Note that: (1) the apparatuses are a Server, and a Network Access Device; (2) a user away physically from the server can be regarded as a remote user, and can create a raster graphics using a smart device (mobile device) with a CCD device (e.g., camera); (3) a mobile device can use a network interface over network to communicate data (raster image file and other data) with an associated device (the server) that receives the raster image file and other data; (4) another user away physically from the server can be regarded as a second remote user, and can create another raster graphics file (a second raster image file) using the second user’s smart device (mobile device) with a CCD device (e.g., camera), and the mobile device can use a network interface over network to communicate data (the second raster image file and other data) with an associated device (the server) that receives the second raster image file and other data; and (5) the second raster image file can be regarded as the additional input.
Regarding claim 17, the combination of Foley, Lv, Rieffel, and Dai discloses The system of claim 15, wherein the instructions are to further cause the server to transmit, to a remote user device over the network interface, at least a portion of the 3D model. (Foley, col. 32, lines 11-12, “The processor 802 performs instructions of the software program 804”; col. 31 /line 55 – col. 32 / line 2, “Referring now to FIG. 8 an automated controller is illustrated that may be used to implement various aspects of the present disclosure, in various embodiments, and for various aspects of the present disclosure, controller 800 may be included in one or more of: a wireless tablet or handheld device, a server, a rack mounted processor unit. The controller may be included in one or more of the apparatuses described above, such as a Server, and a Network Access Device. The controller 800 includes a processor unit 802, such as one or more semiconductor-based processors, coupled to a communication device 801 configured to communicate via a communication network (not shown in FIG. 8). The communication evice 801 may be used to communicate, for example, with one or more online devices, such as a personal computer, laptop, or a handheld device”). Note that: (1) controller 800 as a server can communicate with via a communication network with one or more online devices (a hand-held device or a remote user device); and (2) at least a portion of the 3D model as a communication content can be transferred from the server to the remote user device.
Regarding claim 18, the combination of Foley, Lv, Rieffel, and Dai discloses The system of claim 15, wherein the additional input comprises a scan performed by a remote user device. (Foley, col. 31, lines 60-63, “The controller may be included in one or more of the apparatuses described above, such as a Server, and a Network Access Device”; col. 11, lines 37-49, “a drawing or other design plan may be stored in paper format or digital version or may not exist or may never have existed. The input may also be in any raster graphics image … The input process may occur with a user creating, scanning into, or accessing such a file containing a raster graphics image or a vector graphics image. The user may access the file on a desktop or standalone computing device or, in some embodiments, via an application running on a smart device … a user may operate a scanner or a smart device with a charged couple device to create the file containing the image on the smart device”; col. 33, line 3-5, “The mobile device 902 also includes a network interface 916 to communicate data to a network and/or an associated computing device”). Note that: (1) the apparatuses are a Server, and a Network Access Device; (2) a user away physically from the server can be regarded as a remote user, and can create a raster graphics using a smart device (mobile device) with a CCD device (e.g., camera); (3) a mobile device can use a network interface over network to communicate data (raster image file and other data) with an associated device (the server) that receives the raster image file and other data; (4) another user away physically from the server can be regarded as a second remote user, and can create another raster graphics file (a second raster image file) using the second user’s smart device (mobile device) with a CCD device (e.g., camera), and the mobile device can use a network interface over network to communicate data (the second raster image file and other data) with an associated device (the server) that receives the second raster image file and other data; and (5) the second raster image file can be regarded as the additional input.
Claims 5, 12, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Foley, Lv, Rieffel, Dai, and Yin et al. (Generating 3D Building Models from Architectural Drawings: A Survey, IEEE Computer Graphics and Applications, vol. 29, no. 1, pp. 20-30, Jan.-Feb. 2009, hereinafter “Yin”).
Regarding claim 5, the combination of Foley, Lv, Rieffel, and Dai discloses The method of claim 1, wherein the raster image of a floor plan is a raster image of a floor plan of a first floor, the 3D model of the vector image is a 3D model of the first floor, and wherein the method further comprises:
receiving, at the server over the network, a raster image of a floor plan of a second floor of the structure; (Foley, col. 35, lines 5-13, “At step 1001, the method includes receiving into a controller a design plan (or a first two-dimensional representation) of at least a portion of a building. As described above, the design plan may include an architectural drawing, floor plan, design drawing and the like. At step 1002, the portion of a design plan (or a first two-dimensional representation) may be represented as a raster image”; col. 11, lines 50-53, “a degree of the processing as described herein may be performed on a controller, which may include a cloud server, a standalone computing device or a smart device”). Note that: a cloud server via a network can perform receiving a raster image of a floor plan of a floor (a second floor) of the building (the structure).
identifying, by the server, semantic information of the floor plan of the second floor from the raster image of the floor plan of the second floor; (col. lines 30-41, “At step 104, an AI engine may ascertain features included in the design plan, the AI engine may additionally ascertain that a feature is located within a particular set of boundaries or external to the set of boundaries. Features may include … architectural aspects, fixtures, duct work, wiring, piping, or other item included in a two-dimensional reference submitted to be analyzed. The features and boundaries may be determined, for example, via algorithmically processing an input design plan image with a trained AI model the AI engine may process a raster file that is converted for output as an image file of a floorplan”). Note that: (1) an AI engine may ascertain or identify the features or semantic information of the plan of the second floor; and (2) the AI engine may process the raster image of the floor plan of the second floor as the input design plan image with the features.
generating, by the server, a vector image from the raster image of the floor plan of the second floor and identified semantic information of the floor plan of the second floor; (Lv, Figure 1: “Pixel (top) and vector (bottom) floor plan image. The pixel floor plan image (top) contains various information, such as room structural elements (magenta), text (blue), symbols (cyan), scales (yellow)”; page 16719, Figure 2: the raster floor plan image on the left is processed with a neural network in the middle to detect, extract, or calculate ROIs, structural elements, text, symbols, and scale, and a vector floor plan image on the right and upper side is generated with “Vectorization” and the identified semantic information; page 16719, col. left, paras. 2-4, “YOLOv4 [11] is used as our basic detection model of region of interest (ROI) detection module … The DeepLabv3+ [12] is adopted as our basic network”; page 16720, col. right, paras. 1-2, “YOLOv4 [11] is our basic network … The network is designed to a modified FCN network [24, 32], and backbone is Resnet50 [16]”; page 16721, col. left, para. 2, “we expect to convert raster information (structural elements, text and symbols) into vector graphics. The segmented floor plan elements can be divided into two categories [33], one is room-boundary pixels, and the other is room-type pixels. The room-boundary pixels represent walls, doors, windows and doorways, while room-type pixels are living room, bathroom, bedroom, etc. The core idea is to obtain vectorized contour information of room type pixels for each room, then calculate the center line of the wall based on the contour information and room boundary pixels. The room type can be refined by the text and symbol detection results. The position information of the doors, windows and doorways can be obtained according to the segmentation and regression results. Detailed steps are shown in Algorithm 1”). Note that: (1) the generation of a vector image of the floor plan of the second floor can be regarded as a combination of processing a raster image of the floor plan of the second floor using a machine learning model (YOLOv4, modified YOLOv4, DeepLabv3+, FCN, and Resnet50 based neural network) to detect or extract, or calculate ROIs, structural elements, text and symbols, and scale, and performing vectorization to generate a vector image of the second floor plan; (2) the ROIs, structural elements, text and symbols, and scale of the floor plan of the second floor, can be regarded as the similar identified semantic information as identified by Foley above; and (3) the vector image of the floor plan of the second floor with vectorization is generated or outputted.
generating, by the server, a 3D model of the vector image of the second floor; and (Lv, page 16719, Figure 2: with “Post-processing” a 3D model on the lower-right side from the vector image with “Vectorization” is generated; page 16723, Figure 3:” From left to right, an input floor plan image, semantic segmentation result with post-processing, reconstructed vector-graphics representation with room type annotation, the corresponding 3D reconstruction model”). Note that: the 3D model is a 3D model of the vector image of the second floor.
However, the combination of Foley, Lv, Rieffel, and Dai fails to disclose, but in the same art of computer graphics, Yin discloses
combining, by the server, the 3D model of the first floor with the 3D model of the second floor to create a multi-level 3D model of a structure. (Yin, page 23, col. left, para. 2, “During 3D modeling, the system separately extrudes a 3D model of each floor and assembles them to form the entire building”). Note that: the 3D models of the first floor and the second floor are generated or extruded by the combination of Foley in view of Lv; (2) the cloud sever can perform a process to combine or assembly them to formulate a multi-floor or multi-level 3D model of the building (structure).
The combination of Foley, Lv, Rieffel, and Dai, and Yin, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply assembling the 3D models of multiply floors and generating a 3D model of a building, as taught by Yin into the combination of Foley, Lv, Rieffel, and Dai. The motivation would have been “During 3D modeling, the system separately extrudes a 3D model of each floor and assembles them to form the entire building” (Yin, page 23, col. left, para. 2). The suggestion for doing so would allow to effectively combine the 3D models of separate floors to formulate a 3D model of a building. Therefore, it would have been obvious to combine Foley, Lv, Rieffel, Dai, and Yin.
Claims 12 and 19 are corresponding to the method of claim 5, respectively. Therefore, claims 12 and 19 are rejected for the same rationale for claims 5, respectively.
Regarding claim 20, the combination of Foley, Lv, Rieffel, Dai, and Yin discloses The system of claim 19, wherein either or both of the scan of the first floor and the scan of the second floor comprise raster images. (Foley, col. 35, lines 5-13, “At step 1001, the method includes receiving into a controller a design plan (or a first two-dimensional representation) of at least a portion of a building. As described above, the design plan may include an architectural drawing, floor plan, design drawing and the like. At step 1002, the portion of a design plan (or a first two-dimensional representation) may be represented as a raster image”; col. 11, lines 37-49, “a drawing or other design plan may be stored in paper format or digital version or may not exist or may never have existed. The input may also be in any raster graphics image … The input process may occur with a user creating, scanning into, or accessing such a file containing a raster graphics image or a vector graphics image. The user may access the file on a desktop or standalone computing device or, in some embodiments, via an application running on a smart device … a user may operate a scanner or a smart device with a charged couple device to create the file containing the image on the smart device”). Note that: (1) The first floor plan’s raster image file can be created by the user using the smart device’s CCD device; and (2) The second floor plan raster image file can be created in the same way for the first floor plan’s raster image file.
Response to Arguments
Applicant's arguments with respect to claim rejection 35 U.S.C. 103, have been fully considered but they are not persuasive.
Applicant alleges, “Rieffel comes closest with its semantic information database; however, Rieffel does not disclose or otherwise teach that the semantic
information in the semantic information database is sourced from one or more separate
documents.” (page 8, lines 16-19). However, Examiner respectfully disagrees about the respective allegations as whole because: Rieffel does disclose the semantic information obtained from one or more documents that are separate from the raster image. Please see the detailed citations and explanations for claim 1 above. The arguments are not persuasive.
Applicant alleges, “None of the cited references disclose a machine learning model that can predict missing semantic information. Rather, Lv (and arguably Foley) discloses use of a machine learning system to extract semantic information from images of building plans that are being analyzed. This is different from predicting missing semantic information, as the semantic information is” (page 8 / line 21 – page 9 / line 2). Examiner agrees that the combination of Foley, Lv, and Rieffel does not disclose the amended limitations. However, Examiner respectfully disagrees about the respective allegations as whole because: Dai discloses predict missing semantic information. Please see the detailed citations and explanations for claim 1 above. The arguments are not persuasive.
Applicant alleges, “For all of the foregoing reasons, independent claim 1 is allowable over the combination of Foley, in view of Lv, in further view of Rieffel. Likewise, independent claims 8 and 15 are allowable by virtue of reciting substantially similar limitations. The remaining claims depend from and add further features to one of the independent claims. It is respectfully submitted that these dependent claims are allowable by reason of depending from an allowable claim, as well as for adding new features, and that it is not necessary to separately address these dependent claims.” (page 9, lines 3-0). However, Examiner respectfully disagrees about the respective allegations as whole because: (1) the combination of Foley, Lv, Rieffel, and Dai discloses all limitations of claim 1. Please see the detailed citations, explanations, and corresponding rationales for claim 1 above; (2) independent claims 8 and 15 are corresponding to claims. Therefore, they are rejected for the same rational for claim 1, respectively; and (3) the dependent claims from the independent claims are rejected for the respective rationale above. The arguments are not persuasive.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BIAO CHEN whose telephone number is (703)756-1199. The examiner can normally be reached M-F 8am-5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee M Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Biao Chen/
Patent Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights