DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office Action is in response to Applicant’s amendment/response filed on 12/23/2025, which has been entered and made of record. Applicant’s amendments to the Specification previously set forth in the Non-Final Office Action mailed 09/30/2025.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-11, and 13-18 are rejected under 35 U.S.C. 103 as being unpatentable over Foley et al. (US 12,197,829 B2, hereinafter “Foley”) in view of Lv et al. (Residential floor plan recognition and reconstruction, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), DOI: 10.1109/CVPR46437.2021.01644, pp. 16712-16721, 20-25 June 2021, hereinafter “Lv”) and Rieffel et al. (US 2010/0214284 A1, hereinafter “Rieffel”).
Regarding claim 1, Foley discloses A method, comprising:
receiving, at a server over a network, a raster image of a floor plan of a structure; (col. 35, lines 5-13, “At step 1001, the method includes receiving into a controller a design plan (or a first two-dimensional representation) of at least a portion of a building. As described above, the design plan may include an architectural drawing, floor plan, design drawing and the like. At step 1002, the portion of a design plan (or a first two-dimensional representation) may be represented as a raster image”; col. 11, lines 50-53, “a degree of the processing as described herein may be performed on a controller, which may include a cloud server, a standalone computing device or a smart device”). Note that: a cloud server via a network can perform receiving the raster image of a design plan (floor plan) of at least a portion of a building.
identifying, by the server, semantic information of the floor plan from the raster image; … wherein the semantic information ; (col. 13, lines 30-41, “At step 104, an AI engine may ascertain features included in the design plan, the AI engine may additionally ascertain that a feature is located within a particular set of boundaries or external to the set of boundaries. Features may include … architectural aspects, fixtures, duct work, wiring, piping, or other item included in a two-dimensional reference submitted to be analyzed. The features and boundaries may be determined, for example, via algorithmically processing an input design plan image with a trained AI model the AI engine may process a raster file that is converted for output as an image file of a floorplan”; col. 10, lines 43-51, “AI generated values for parameters may also be useful in a variety of estimation elements, such as (without limitation): flooring (wood, ceramic, carpet, tile, etc.), structural (poured concrete, steel), walls (gypsum, masonry blocks, glass walls, wall base, exterior cladding), doors and frames (hollow metal, wood, glass), windows glazing, insulation, paint (ceilings and walls), acoustical ceilings, code compliance, stucco (ceilings and walls), mechanical, plumbing, and electrical aspects.”). Note that: (1) an AI engine may ascertain or identify the features or semantic information (architectural aspects, fixtures, duct work, wiring, piping, etc.) of the floor plan; (2) the AI engine may process the raster file as the input design plan (floor plan) image with the features; and (3) paint (ceilings and walls) and stucco (ceilings and walls) can be regarded as interior or exterior finishes.
Updating … in response to additional information. (col. 14, lines 12-18, “At step 106, a controller is operative to generate an interactive user interface with dynamic components that may be manipulated by one or both of user interaction and automated processes. Any or all of the components in a user interface may be converted to a version that allows a user to modify an attribute of the components, such as the length, size, beginning point, end point, thickness, or other attribute”; col. 15, lines 19-28, “At step 106A, in some embodiments, components presented in the interactive user interface may be analyzed by a user and refinements may be made to one or more components (e.g., size, shape and/or position of the component). In some embodiments, user modifications may also be input back to the AI engine to train the AI engine. User modifications provided back to the AI Engine may be referenced to make subsequent AI processes more accurate, efficient, fast, trained and/or enable additional types of AI processes”; col. 36, lines 17-20, “Any or all of steps 1101-1107 may be repeated for different portions of the two-dimensional reference descriptive of the building. For example, for a second design plan representing a different portion of the entire building design”). Note that: (1) when a user changes or modifies the attributes of the dynamic components through the user interface, it means additional information regarding the dynamic component is input to a floor plan management system, and it becomes a condition for possible action of the system; (2) the additional information can provide back to or update the input to the AI Engine to enable additional types of AI processes; (3) a second design plan (e.g., a second floor plan raster image) can be regarded as a updated floor plan raster image; and (4) the updated floor plan raster image can also be the additional information.
at the server … by the server … (col. 11, “a degree of the processing as 50 described herein may be performed on a controller, which may include a cloud server, a standalone computing device or a smart device”). Note that: a cloud server can perform the processing.
However, Foley fails to disclose, but in the same art of computer graphics, Lv discloses
generating, by the server, a vector image from the raster image and identified semantic information by processing the raster image through a machine learning model, … (Lv, page 16717, Figure 1: “Pixel (top) and vector (bottom) floor plan image. The pixel floor plan image (top) contains various information, such as room structural elements (magenta), text (blue), symbols (cyan), scales (yellow)”; page 16719, Figure 2: the raster floor plan image on the left is processed with a neural network in the middle to detect, extract, or calculate ROIs, structural elements, text, symbols, and scale, and a vector floor plan image on the right and upper side is generated with “Vectorization” and the identified semantic information; page 16719, col. left, paras. 2-4, “YOLOv4 [11] is used as our basic detection model of region of interest (ROI) detection module … The DeepLabv3+ [12] is adopted as our basic network”; page 16720, col. right, paras. 1-2, “YOLOv4 [11] is our basic network … The network is designed to a modified FCN network [24, 32], and backbone is Resnet50 [16]”; page 16721, col. left, para. 2, “we expect to convert raster information (structural elements, text and symbols) into vector graphics. The segmented floor plan elements can be divided into two categories [33], one is room-boundary pixels, and the other is room-type pixels. The room-boundary pixels represent walls, doors, windows and doorways, while room-type pixels are living room, bathroom, bedroom, etc. The core idea is to obtain vectorized contour information of room type pixels for each room, then calculate the center line of the wall based on the contour information and room boundary pixels. The room type can be refined by the text and symbol detection results. The position information of the doors, windows and doorways can be obtained according to the segmentation and regression results. Detailed steps are shown in Algorithm 1”). Note that: (1) the generation of a vector image can be regarded as a combination of processing a raster image using a machine learning model (YOLOv4, modified YOLOv4, DeepLabv3+, FCN, and Resnet50 based neural network) to detect or extract, or calculate ROIs, structural elements, text and symbols, and scale, and performing vectorization to generate a vector image of a floor plan; (2) the ROIs, structural elements, text and symbols, and scale, can be regarded as the similar identified semantic information as identified by Foley above; and (3) the vector image with vectorization is generated or outputted.
generating, by the server, the 3D model from the vector image and the semantic information (Lv, page 16719, Figure 2: with “Post-processing” a 3D model on the lower-right side is generated from the vector image with “Vectorization” and “the multi-modal information of the floor plan, such as room structure, text, symbols and scale”; page 16723, Figure 3:” From left to right, an input floor plan image, semantic segmentation result with post-processing, reconstructed vector-graphics representation with room type annotation, the corresponding 3D reconstruction model”). Note that: the multi-modal information of the floor plan, such as room structure, text, symbols and scale can be regarded as the semantic information.
receiving, at the server, a selection of a format for the 3D model: (Lv, Abstract, “An automatic framework is provided that accurately recognizes the structure, type, and size of the room, and outputs vectorized 3D reconstruction results”; page 16719, Figure 2: with “Post-processing” a 3D model on the lower-right side is generated). Note that: the vectorized 3D reconstruction results are regarded as the 3D model with a format (vectorized 3D reconstruction) that is output from the framework. This format of 3D model can be selected as input for a 3D model render later.
updating, by the server, the vector image and 3D model in response to additional information. (Lv, Figure 1: “Pixel (top) and vector (bottom) floor plan image. The pixel floor plan image (top) contains various information, such as room structural elements (magenta), text (blue), symbols (cyan), scales (yellow)”; page 16719, Figure 2: the raster floor plan image on the left side is processed with a neural network in the middle to detect, extract, or calculate ROIs, structural elements, text, symbols, and scale, and a vector floor plan image on the right and upper side is generated with “Vectorization” and the identified semantic information; page 16723, Figure 3:” From left to right, an input floor plan image, semantic segmentation result with post-processing, reconstructed vector-graphics representation with room type annotation, the corresponding 3D reconstruction model”). Note that: (1) Foley discloses above that the users can change or update the parameters of dynamic components of the floor plan and update the input to the AI Engine to enable additional types of AI processes; (2) in combination with Foley’s teaching here, it is obvious to one having ordinary skills in the art that one can repeat the processes of generation of the vector floor plan image and the 3D model above to update them accordingly at the cloud server over a network while the server can perform the processing and store the corresponding updated vector floor plan image(s) and 3D model(s).
Foley and Lv are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply generating a vector floor plan image and generating a 3D model, as taught by Lv into Foley. The motivation would have been “The system significantly increases accuracy and generalization ability, compared with existing methods.” (Lv, page 16717, Abstract). The suggestion for doing so would allow to improve accuracy and generality of floor plan vectorization and 3D model. Therefore, it would have been obvious to combine Foley with Lv.
However, Foley in view of Lv fails to disclose, but in the same art of computer graphics, Rieffel discloses
wherein the semantic information comprises information relevant to rendering of a 3D model (Rieffel, Figure 4A: the process from Semantic Information of “Semantic Information Database 240”, to “Model Builder 226”, to “Model Rendering Instruction Database 242”, to “Model Rendering Instructions 404”, to “Three Dimensional Model Renderer”, to rendering of “Model 120”). Note that: the sematic information of “Semantic Information Database 240” is the information relevant to rendering of a 3D model.
rendering, by the server, the 3D model based on the selected format: (Rieffel, Figure 4A: the process from Semantic Information of “Semantic Information Database 240”, to “Model Builder 226”, to “Model Rendering Instruction Database 242”, to “Model Rendering Instructions 404”, to “Three Dimensional Model Renderer”, to rendering of “Model 120”; page 7, para. [0062], “This defined model may then be rendered (386) by a 3D model renderer, as described in greater detail below with reference to FIGS. 4A-4C.”). Note that: the 3D model defined by the selected format can be rendered by a 3D render.
Foley in view of Lv, and Rieffel are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply rendering a 3D model based on select format with the semantic information relevant to rendering of a 3D model, as taught by Rieffel into Foley in view of Lv. The motivation would have been “This defined model may then be rendered (386) by a 3D model renderer” (Rieffel, page 7, para. [0062]). The suggestion for doing so would allow to render a 3D model with the semantic information relevant to rendering of a 3D model to have a realistic quality rendering. Therefore, it would have been obvious to combine Foley, Lv, and Rieffel.
Regarding claim 2, the combination of Foley, Lv, and Rieffel discloses The method of claim 1, further comprising training, prior to generating the vector image, the machine learning model. (Lv, page 16719, Figure 2: the raster floor plan image on the left is processed with a neural network in the middle to detect, extract, or calculate ROIs, structural elements, text, symbols, and scale, and a vector floor plan image on the right and upper side is generated with “Vectorization” and the identified semantic information; page 16719, col. left, paras. 2-4, “YOLOv4 [11] is used as our basic detection model of region of interest (ROI) detection module … The DeepLabv3+ [12] is adopted as our basic network”; page 16720, col. right, paras. 1-2, “YOLOv4 [11] is our basic network … The network is designed to a modified FCN network [24, 32], and backbone is Resnet50 [16]”; page 16721, col. left, para. 2, “we expect to convert raster information (structural elements, text and symbols) into vector graphics. The segmented floor plan elements can be divided into two categories [33], one is room-boundary pixels, and the other is room-type pixels. The room-boundary pixels represent walls, doors, windows and doorways, while room-type pixels are living room, bathroom, bedroom, etc. The core idea is to obtain vectorized contour information of room type pixels for each room, then calculate the center line of the wall based on the contour information and room boundary pixels. The room type can be refined by the text and symbol detection results. The position information of the doors, windows and doorways can be obtained according to the segmentation and regression results. Detailed steps are shown in Algorithm 1”; page 16718, col. right, para. 3, “We have crawled 7,000 Residential Floor Plan (RFP) data from Internet search engines, which are mainly home floor plans of Chinese urban buildings”). Note that: (1) the generation of a vector image can be regarded as a combination of processing a raster image using a machine learning model (YOLOv4, modified YOLOv4, DeepLabv3+, FCN, and Resnet50 based neural network) to detect or extract, or calculate ROIs, structural elements, text and symbols, and scale, and performing vectorization to generate a vector image of a floor plan; and (2) before generating the vector image, it is obvious that the machine learning model (YOLOv4, modified YOLOv4, DeepLabv3+, FCN, and Resnet50 based neural network) has been trained with training dataset (RFT dataset).
Regarding claim 3, the combination of Foley, Lv, and Rieffel discloses The method of claim 1, wherein the additional input comprises an updated raster image of the floor plan. (Foley, col. 36, lines 17-20, “Any or all of steps 1101-1107 may be repeated for different portions of the two-dimensional reference descriptive of the building. For example, for a second design plan representing a different portion of the entire building design”). Note that: (1) a second design plan (e.g., a second floor plan raster image) can be regarded as an updated floor plan raster image; and (2) the updated floor plan raster image can be regarded as the additional information.
Regarding claim 4, the combination of Foley, Lv, and Rieffel discloses The method of claim 1, wherein the additional input comprises a scan of a physical space corresponding to the floor plan. (Foley, col. 33, lines 31-38, “A LiDAR sensing system 951 may also be incorporated into the mobile device 902. The LiDAR system may include a scannable laser light (or other collimated) light source which may operate at nonvisible wavelengths such as in the infrared. An associated sensor device, sensitive to the light of emission may be included in the system to record time and strength of returned signal that is reflected off of surfaces in the environment of the mobile device 902”). Note that: when a user holds a mobile device with a LiDAR in the environment of the corresponding floor plan, he can use scan the physical space; and (2) the scanned data can be regarded as the additional input to be input back to the AI engine as described by Foley above.
Regarding claim 6, the combination of Foley, Lv, and Rieffel discloses The method of claim 1, wherein the semantic information comprises room segmentation information and non-visible structures. (Foley, col. 13, lines 30-41, “At step 104, an AI engine may ascertain features included in the design plan, the AI engine may additionally ascertain that a feature is located within a particular set of boundaries or external to the set of boundaries. Features may include … architectural aspects, fixtures, duct work, wiring, piping, or other item included in a two-dimensional reference submitted to be analyzed. The features and boundaries may be determined, for example, via algorithmically processing an input design plan image with a trained AI model the AI engine may process a raster file that is converted for output as an image file of a floorplan”; col. 30, lines 49-51, “In a segmentation phase used to determine boundaries of regions or other space features”). Note that: the semantic information includes the features including non-visible structures (e.g., some duct work, wiring, piping) and boundaries of regions that indicate the room segmentation information,
Regarding claim 7, the combination of Foley, Lv, and Rieffel discloses The method of claim 6, wherein the non-visible structures include one or more of wiring, plumbing, ventilation ducts, insulation, and structural members. (Foley, col. 13, lines 30-41, “At step 104, an AI engine may ascertain features included in the design plan, the AI engine may additionally ascertain that a feature is located within a particular set of boundaries or external to the set of boundaries. Features may include … architectural aspects, fixtures, duct work, wiring, piping, or other item included in a two-dimensional reference submitted to be analyzed”). Note that: the non-visible structures include duct work (e.g., ventilation ducts), wiring, and piping (plumbing).
Claim 8 reciting “A non-transitory computer-readable medium (CRM) comprising instructions that, when executed by the processor of an apparatus, cause the apparatus to:” is corresponding to the method of claim 1. Therefore, claim 8 is rejected for the same rationale for claim 1.
In addition, the combination of Foley, Lv, and Rieffel discloses A non-transitory computer-readable medium (CRM) comprising instructions that, when executed by the processor of an apparatus, cause the apparatus to: (Foley, col. 32, lines 10-14, “The storage device 803 can store a software program 804 with executable logic for controlling the processor 802. The processor 802 performs instructions of the software program 804, and thereby operates in accordance with the present disclosure”). Note that: The storage device 803 is a non-transitory computer-readable medium (CRM).
Claims 9-11 are corresponding to the method of claims 2-4, respectively. Therefore, claims 9-11 are rejected for the same rationale for claims 2-4, respectively.
Regarding claim 13, the combination of Foley, Lv, and Rieffel discloses The CRM of claim 8, wherein the apparatus is a server, and the raster image is received from a remote user device over a network. (Foley, col. 31, lines 60-63, “The controller may be included in one or more of the apparatuses described above, such as a Server, and a Network Access Device”; col. 11, lines 37-49, “a drawing or other design plan may be stored in paper format or digital version or may not exist or may never have existed. The input may also be in any raster graphics image … The input process may occur with a user creating, scanning into, or accessing such a file containing a raster graphics image or a vector graphics image. The user may access the file on a desktop or standalone computing device or, in some embodiments, via an application running on a smart device … a user may operate a scanner or a smart device with a charged couple device to create the file containing the image on the smart device”; col. 33, line 3-5, “The mobile device 902 also includes a network interface 916 to communicate data to a network and/or an associated computing device”). Note that: (1) the apparatuses are a Server, and a Network Access Device; (2) a user away physically from the server can be regarded as a remote user, and can create a raster graphics image file of the design plan (floor plan) using a smart device (mobile device) with a CCD device (e.g., camera); and (3) a mobile device can use a network interface over network to communicate data (raster image file and other data) with an associated device (the server) that receives the raster image file and other data.
Regarding claim 14, the combination of Foley, Lv, and Rieffel discloses The CRM of claim 8, wherein the additional input comprises one or more diagrams of wiring, plumbing, ventilation ducts, insulation, and structural members. (Foley, col. 13, lines 30-41, “At step 104, an AI engine may ascertain features included in the design plan, the AI engine may additionally ascertain that a feature is located within a particular set of boundaries or external to the set of boundaries. Features may include … architectural aspects, fixtures, duct work, wiring, piping, or other item included in a two-dimensional reference submitted to be analyzed. The features and boundaries may be determined, for example, via algorithmically processing an input design plan image with a trained AI model the AI engine may process a raster file that is converted for output as an image file of a floorplan”). Note that: (1) an AI engine may ascertain or identify the features of the floor plan located within a particular set of boundaries or external to the set of boundaries; and (2) the presentations of the features of architectural aspects, fixtures, duct work, wiring, piping, etc. can be regarded as the corresponding diagrams.
Claim 15 reciting “A system, comprising: a server; and a network interface in communication with the server, wherein the server includes instructions to cause the server, when the instructions are executed by a processor of the server, to:” is corresponding to the method of claim 1. Therefore, claim 15 is rejected for the same rationale for claim 1.
In addition, the combination of Foley, Lv, and Rieffel A system, comprising: a server; and a network interface in communication with the server, wherein the server includes instructions to cause the server, when the instructions are executed by a processor of the server, to: (Foley, lines 60-67, “The controller may be included in one or more of the apparatuses described above, such as a Server, and a Network Access Device. The controller 800 includes a processor unit 802, such as one or more semiconductor-based processors, coupled to a communication device 801 configured to communicate via a communication network (not shown in FIG. 8)”; col. 32, lines 10-14, “The storage device 803 can store a software program 804 with executable logic for controlling the processor 802. The processor 802 performs instructions of the software program 804, and thereby operates in accordance with the present disclosure”). Note that: The controller can be regarded as a system.
Regrading claim 16, the combination of Foley, Lv, and Rieffel discloses The system of claim 15, wherein the scan is received from a first remote user device over the network interface, and the additional input is received from a second remote user device over the network interface. (Foley, col. 31, lines 60-63, “The controller may be included in one or more of the apparatuses described above, such as a Server, and a Network Access Device”; col. 11, lines 37-49, “a drawing or other design plan may be stored in paper format or digital version or may not exist or may never have existed. The input may also be in any raster graphics image … The input process may occur with a user creating, scanning into, or accessing such a file containing a raster graphics image or a vector graphics image. The user may access the file on a desktop or standalone computing device or, in some embodiments, via an application running on a smart device … a user may operate a scanner or a smart device with a charged couple device to create the file containing the image on the smart device”; col. 33, line 3-5, “The mobile device 902 also includes a network interface 916 to communicate data to a network and/or an associated computing device”). Note that: (1) the apparatuses are a Server, and a Network Access Device; (2) a user away physically from the server can be regarded as a remote user, and can create a raster graphics using a smart device (mobile device) with a CCD device (e.g., camera); (3) a mobile device can use a network interface over network to communicate data (raster image file and other data) with an associated device (the server) that receives the raster image file and other data; (4) another user away physically from the server can be regarded as a second remote user, and can create another raster graphics file (a second raster image file) using the second user’s smart device (mobile device) with a CCD device (e.g., camera), and the mobile device can use a network interface over network to communicate data (the second raster image file and other data) with an associated device (the server) that receives the second raster image file and other data; and (5) the second raster image file can be regarded as the additional input.
Regarding claim 17, the combination of Foley, Lv, and Rieffel discloses The system of claim 15, wherein the instructions are to further cause the server to transmit, to a remote user device over the network interface, at least a portion of the 3D model. (Foley, col. 32, lines 11-12, “The processor 802 performs instructions of the software program 804”; col. 31 /line 55 – col. 32 / line 2, “Referring now to FIG. 8 an automated controller is illustrated that may be used to implement various aspects of the present disclosure, in various embodiments, and for various aspects of the present disclosure, controller 800 may be included in one or more of: a wireless tablet or handheld device, a server, a rack mounted processor unit. The controller may be included in one or more of the apparatuses described above, such as a Server, and a Network Access Device. The controller 800 includes a processor unit 802, such as one or more semiconductor-based processors, coupled to a communication device 801 configured to communicate via a communication network (not shown in FIG. 8). The communication evice 801 may be used to communicate, for example, with one or more online devices, such as a personal computer, laptop, or a handheld device”). Note that: (1) controller 800 as a server can communicate with via a communication network with one or more online devices (a hand-held device or a remote user device); and (2) at least a portion of the 3D model as a communication content can be transferred from the server to the remote user device.
Regarding claim 18, the combination of Foley, Lv, and Rieffel discloses The system of claim 15, wherein the additional input comprises a scan performed by a remote user device. (Foley, col. 31, lines 60-63, “The controller may be included in one or more of the apparatuses described above, such as a Server, and a Network Access Device”; col. 11, lines 37-49, “a drawing or other design plan may be stored in paper format or digital version or may not exist or may never have existed. The input may also be in any raster graphics image … The input process may occur with a user creating, scanning into, or accessing such a file containing a raster graphics image or a vector graphics image. The user may access the file on a desktop or standalone computing device or, in some embodiments, via an application running on a smart device … a user may operate a scanner or a smart device with a charged couple device to create the file containing the image on the smart device”; col. 33, line 3-5, “The mobile device 902 also includes a network interface 916 to communicate data to a network and/or an associated computing device”). Note that: (1) the apparatuses are a Server, and a Network Access Device; (2) a user away physically from the server can be regarded as a remote user, and can create a raster graphics using a smart device (mobile device) with a CCD device (e.g., camera); (3) a mobile device can use a network interface over network to communicate data (raster image file and other data) with an associated device (the server) that receives the raster image file and other data; (4) another user away physically from the server can be regarded as a second remote user, and can create another raster graphics file (a second raster image file) using the second user’s smart device (mobile device) with a CCD device (e.g., camera), and the mobile device can use a network interface over network to communicate data (the second raster image file and other data) with an associated device (the server) that receives the second raster image file and other data; and (5) the second raster image file can be regarded as the additional input.
Claims 5, 12, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Foley in view of Lv and Rieffel, and further in view of Yin et al. (Generating 3D Building Models from Architectural Drawings: A Survey, IEEE Computer Graphics and Applications, vol. 29, no. 1, pp. 20-30, Jan.-Feb. 2009, hereinafter “Yin”).
Regarding claim 5, the combination of Foley, Lv, and Rieffel discloses The method of claim 1, wherein the raster image of a floor plan is a raster image of a floor plan of a first floor, the 3D model of the vector image is a 3D model of the first floor, and wherein the method further comprises:
receiving, at the server over the network, a raster image of a floor plan of a second floor of the structure; (Foley, col. 35, lines 5-13, “At step 1001, the method includes receiving into a controller a design plan (or a first two-dimensional representation) of at least a portion of a building. As described above, the design plan may include an architectural drawing, floor plan, design drawing and the like. At step 1002, the portion of a design plan (or a first two-dimensional representation) may be represented as a raster image”; col. 11, lines 50-53, “a degree of the processing as described herein may be performed on a controller, which may include a cloud server, a standalone computing device or a smart device”). Note that: a cloud server via a network can perform receiving a raster image of a floor plan of a floor (a second floor) of the building (the structure).
identifying, by the server, semantic information of the floor plan of the second floor from the raster image of the floor plan of the second floor; (col. lines 30-41, “At step 104, an AI engine may ascertain features included in the design plan, the AI engine may additionally ascertain that a feature is located within a particular set of boundaries or external to the set of boundaries. Features may include … architectural aspects, fixtures, duct work, wiring, piping, or other item included in a two-dimensional reference submitted to be analyzed. The features and boundaries may be determined, for example, via algorithmically processing an input design plan image with a trained AI model the AI engine may process a raster file that is converted for output as an image file of a floorplan”). Note that: (1) an AI engine may ascertain or identify the features or semantic information of the plan of the second floor; and (2) the AI engine may process the raster image of the floor plan of the second floor as the input design plan image with the features.
generating, by the server, a vector image from the raster image of the floor plan of the second floor and identified semantic information of the floor plan of the second floor; (Lv, Figure 1: “Pixel (top) and vector (bottom) floor plan image. The pixel floor plan image (top) contains various information, such as room structural elements (magenta), text (blue), symbols (cyan), scales (yellow)”; page 16719, Figure 2: the raster floor plan image on the left is processed with a neural network in the middle to detect, extract, or calculate ROIs, structural elements, text, symbols, and scale, and a vector floor plan image on the right and upper side is generated with “Vectorization” and the identified semantic information; page 16719, col. left, paras. 2-4, “YOLOv4 [11] is used as our basic detection model of region of interest (ROI) detection module … The DeepLabv3+ [12] is adopted as our basic network”; page 16720, col. right, paras. 1-2, “YOLOv4 [11] is our basic network … The network is designed to a modified FCN network [24, 32], and backbone is Resnet50 [16]”; page 16721, col. left, para. 2, “we expect to convert raster information (structural elements, text and symbols) into vector graphics. The segmented floor plan elements can be divided into two categories [33], one is room-boundary pixels, and the other is room-type pixels. The room-boundary pixels represent walls, doors, windows and doorways, while room-type pixels are living room, bathroom, bedroom, etc. The core idea is to obtain vectorized contour information of room type pixels for each room, then calculate the center line of the wall based on the contour information and room boundary pixels. The room type can be refined by the text and symbol detection results. The position information of the doors, windows and doorways can be obtained according to the segmentation and regression results. Detailed steps are shown in Algorithm 1”). Note that: (1) the generation of a vector image of the floor plan of the second floor can be regarded as a combination of processing a raster image of the floor plan of the second floor using a machine learning model (YOLOv4, modified YOLOv4, DeepLabv3+, FCN, and Resnet50 based neural network) to detect or extract, or calculate ROIs, structural elements, text and symbols, and scale, and performing vectorization to generate a vector image of the second floor plan; (2) the ROIs, structural elements, text and symbols, and scale of the floor plan of the second floor, can be regarded as the similar identified semantic information as identified by Foley above; and (3) the vector image of the floor plan of the second floor with vectorization is generated or outputted.
generating, by the server, a 3D model of the vector image of the second floor; and (Lv, page 16719, Figure 2: with “Post-processing” a 3D model on the lower-right side from the vector image with “Vectorization” is generated; page 16723, Figure 3:” From left to right, an input floor plan image, semantic segmentation result with post-processing, reconstructed vector-graphics representation with room type annotation, the corresponding 3D reconstruction model”). Note that: the 3D model is a 3D model of the vector image of the second floor.
However, the combination of Foley, Lv, and Rieffel fails to disclose, but in the same art of computer graphics, Yin discloses
combining, by the server, the 3D model of the first floor with the 3D model of the second floor to create a multi-level 3D model of a structure. (Yin, page 23, col. left, para. 2, “During 3D modeling, the system separately extrudes a 3D model of each floor and assembles them to form the entire building”). Note that: the 3D models of the first floor and the second floor are generated or extruded by the combination of Foley in view of Lv; (2) the cloud sever can perform a process to combine or assembly them to formulate a multi-floor or multi-level 3D model of the building (structure).
The combination of Foley, Lv, and Rieffel, and Yin, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply assembling the 3D models of multiply floors and generating a 3D model of a building, as taught by Yin into the combination of Foley, Lv, and Rieffel. The motivation would have been “During 3D modeling, the system separately extrudes a 3D model of each floor and assembles them to form the entire building” (Yin, page 23, col. left, para. 2). The suggestion for doing so would allow to effectively combine the 3D models of separate floors to formulate a 3D model of a building. Therefore, it would have been obvious to combine Foley, Lv, Rieffel, and Yin.
Claims 12 and 19 are corresponding to the method of claim 5, respectively. Therefore, claims 12 and 19 are rejected for the same rationale for claims 5, respectively.
Regarding claim 20, the combination of Foley, Lv, Rieffel, and Yin discloses The system of claim 19, wherein either or both of the scan of the first floor and the scan of the second floor comprise raster images. (Foley, col. 35, lines 5-13, “At step 1001, the method includes receiving into a controller a design plan (or a first two-dimensional representation) of at least a portion of a building. As described above, the design plan may include an architectural drawing, floor plan, design drawing and the like. At step 1002, the portion of a design plan (or a first two-dimensional representation) may be represented as a raster image”; col. 11, lines 37-49, “a drawing or other design plan may be stored in paper format or digital version or may not exist or may never have existed. The input may also be in any raster graphics image … The input process may occur with a user creating, scanning into, or accessing such a file containing a raster graphics image or a vector graphics image. The user may access the file on a desktop or standalone computing device or, in some embodiments, via an application running on a smart device … a user may operate a scanner or a smart device with a charged couple device to create the file containing the image on the smart device”). Note that: (1) The first floor plan’s raster image file can be created by the user using the smart device’s CCD device; and (2) The second floor plan raster image file can be created in the same way for the first floor plan’s raster image file.
Response to Arguments
Applicant's arguments with respect to claim rejection 35 U.S.C. 103, have been fully considered but they are not persuasive.
Applicant alleges, “neither Foley nor Lv disclose semantic information that includes interior or exterior finishes, that are relevant to rendering of a 3D model. In fact, neither Foley nor Lv disclose use of any information specific to finishes” (page 10, lines 14-16). However, the arguments are respectfully mooted because the corresponding newly amended limitations, “… wherein the semantic information ;”, have been addressed in the detailed claim rejection 35 U.S.C. above. The arguments are not persuasive.
Applicant alleges, “Further, neither Foley nor Lv disclose selection of a format for a 3D model that is generated from a vector image and the semantic information, or further rendering the 3D model based on the selected format. Rather, Foley and Lv are simply silent on rendering of a 3D model or any specific format that such a rendering would take, let alone selection of a format for the 3D model” (page 10, lines 16-20). However, the arguments are respectfully mooted because the corresponding newly amended limitations, “receiving, at the server, a selection of a format for the 3D model:”, “wherein the semantic information comprises information relevant to rendering of a 3D model”, and rendering, by the server, the 3D model based on the selected format:”, have been addressed in the detailed claim rejection 35 U.S.C. above. The arguments are not persuasive.
Applicant alleges, “The remaining claims depend from and add further features to one of the independent claims. It is respectfully submitted that these dependent claims are allowable by reason of depending from an allowable claim, as well as for adding new features, and that it is not necessary to separately address these dependent claims.” (page 10, lines 21-24). However, Examiner respectfully disagrees about the allegations as whole because the dependent claims of independent claims 1, 8, and 15 are rejected for the respective rationale above. The arguments are not persuasive.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BIAO CHEN whose telephone number is (703)756-1199. The examiner can normally be reached M-F 8am-5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee M Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Biao Chen/
Patent Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights