DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 14 is objected to because of the following informalities:
Claim 14 Lines 9-10 recites the limitation “the trained machine learning neural network”. This has a lack of antecedent basis and should instead be “the trained machine learning model” to be consistent with an earlier limitation. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-5, 8-11, 13-20 are rejected under 35 U.S.C. 103 as being unpatentable over Hartfiel et al. (US 20200388059 A1)(Hereinafter referred to as Hartfiel) in view of Dong et al. (“Vectorization of Floor Plans Based on EdgeGAN”)(Hereinafter referred to as Dong) and in further view of Dahm et al. (US 20200402192 A1)(Hereinafter referred to as Dahm).
Regarding Claim 1, Hartfiel discloses A method for machine-assisted digital map editing, comprising: (See Abstract, “A system and method for tracing polygons in a drawing source file.” Also see [0002], “Design files (e.g. CAD, SVG, DXF) are used to create, draft, edit, modify and optimize the design of buildings, machines, electronics, vehicles and other 3D objects in a number of industries.”)
generating a vectorized polygon representation of architectural data in a base map; (See Abstract, “The method includes extracting vector imagery from the source file, creating a planar representation of the vector imagery as a plurality of lines, filtering the plurality of lines to create simplified line art, morphologically dilating the simplified line art to generate a polygonal approximation,” See [0005], “Accordingly, there is a need for a self-improving polygon tracing/correction system to generate polygons from lines in design files while minimizing human error and observer bias.”)
receiving tracing of an architectural feature in the base map; and (See [0006], “Provided is a method for tracing polygons in a drawing source file.”)
proposing or automatically placing a polygon object representing the traced feature in an editable map. (See [0001], “The embodiments disclosed herein relate to automated methods of editing design files, and, in particular to a system and method for tracing polygons in design files containing only line segments.” Note that “editing” implies an editable map.
Also see [0006], “Provided is a method for tracing polygons in a drawing source file. The method include extracting vector imagery from the source file . . . and filtering the visual polygons according to one or more geometric parameters to identify salient polygons.”
Lastly, see [0007], “The method may further include generating a composite raster representation of the salient polygons overlaid on the planar representation of the vector imagery, . . .” In this case, “overlaid” would correspond to “automatically placing a polygon object”.)
However, Hartfiel fails to explicitly disclose classifying architectural data in a digital base map by a trained machine learning neural network;
generating a vectorized polygon representation of the classified architectural data;
presenting the base map on a display interface;
receiving user input tracing an architectural feature in the base map; and
Dong teaches classifying architectural data in a digital base map by a trained machine learning neural network; (See Abstract, “A 2D floor plan (FP) often contains structural, decorative, and functional elements and annotations. Vectorization of floor plans (VFP) is an object detection task that involves the localization and recognition of different structural primitives in 2D FPs.” Also see Page 2 Fig. 2 showing a pipeline that includes “Recognition of pillars”, “Recognition of doors”, “Recognition of ordinary windows”. Lastly, see Page 2 Paragraph 2, “Artificial neural networks have been applied in VFP with the development of DL. Dodge et al. [14] used a fully convolutional neural network (CNN) to detect structural elements and achieve a mean intersection-over-union score of 89.9\% on R-FP and 94.4\% on the public CVC-FP dataset.” In this case, “recognition” would be considered as “classifying architectural data in a digital base map”.)
generating a vectorized polygon representation of the classified architectural data; (See Abstract, “Second, a new edge-extracting GAN (EdgeGAN) is designed for the new task by formulating the VFP task as an image translation task innovatively that involves the projection of the original 2D FPs into a primitive space. The output of EdgeGAN is a primitives feature map . . .”
See Page 2 Fig. 1(a) and 1(b) showing the vectorization of the floor plan. Specifically, the structural primitives in the floor plan are detected and then assembled into a primitives feature map.
In this case, after the structural elements are recognized (classified architectural data), a GAN then outputs a primate feature map, which is a vectorization of the floor plan. Note, in combination with Hartfiel which teaches to generate polygons from lines, see Hartfiel Abstract and [0005], and Dong teaching the primitives feature map of recognized structural elements, the above limitation of generating “vectorized polygon representation of the classified architectural data” is taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Hartfiel with Dong to include classifying architectural data in a digital base map by a machine learning neural network and generating a vectorized polygon representation of the classified architectural data.
The motivation to combine Hartfiel with Dong would have been obvious as both arts are in the same of field of processing architectural floor plans, see Dong Abstract. The benefit of classifying architectural data is that it allows for improved accuracy and efficiency when outputting primitives feature maps.
However, Hartfiel in view of Dong still fail to explicitly disclose presenting the base map on a display interface;
receiving user input tracing an architectural feature in the base map; and
Dahm teaches presenting the base map on a display interface; (See Abstract, “The method may commence with receiving floor plan data and generating a floor plan of a facility. The method may continue with creating an interactive web-based map of the facility by superimposing the floor plan onto a map of an area associated with the facility.” Also see Fig. 6 showing the base map on a display interface.)
receiving user input tracing an architectural feature in the base map; and (See [0071], “A floor plan 605 (e.g., in Joint Photographic Experts Group (JPEG) or portable network graphics (PNG) format) may be superimposed on a map 610 (e.g., a Google Earth image) to generate a web-based interactive map 615. Upon generation of the web-based interactive map 615, multiple tools to trace and edit the floor plan 605 can be added to the web-based interactive map 615.” Here, Dahm directly teaches a tool which allows a user to “trace” the floor plan (base map).)
Lastly, note that Dahm additionally teaches an editable map. (See [0071] teaching a superimposed floor plan and map which has tools that can edit the interactive map.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Hartfiel in view of Dong with Dahm to include presenting the based on a display interface and receiving user input tracing an architectural feature in the base map.
The motivation to combine Hartfiel in view of Dong with Dahm would have been obvious as Dahm is simply teaching the basic and common idea of displaying a base map on an interface and receiving user input tracing of an architectural feature in the base map base.
Regarding Claim 2, Hartfiel in view of Dong and Dahm disclose The method of claim 1, further comprising: training the neural network using a paired set of training images, comprising: a set of base map images as inputs; and (See Dong Page 4 “Framework Based on EdgeGAN”, “Thus, the new dataset ZSCVFP is established for this reason. ZSCVFP contains 8800 FPs in the training set and 2000 FPs in the test set.” Note that FPs is Floor Plans (a set of base map images).)
a set of vectorized polygon representations as outputs, wherein each base map image is associated to a vectorized polygon representation. (See Dong Page 5 “3.1 EdgeGAN” Paragraph 1, “EdgeGAN learns a map from the input FPs X to the output Z, and Y is the ground truth.” Note that output Z refers to the primitives feature map (vectorized floor plan).
Also See Hartfiel [0082], “The machine learning model is configured to detect deficiencies in polygons according to a training dataset of accepted polygons as validated by a user.” Note that for the combination of the two arts, the EdgeGAN taught by Dong would be trained using a dataset of accepted polygons (a set of vectorized polygon representations) and thus be configured to directly output a set of vectorized polygon representations. In this machine learning scenario, each floor plan (base map image) would be associated with a polygon (vectorized polygon representation). The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 3, Hartfiel in view of Dong and Dahm disclose The method of claim 1, further comprising: receiving user input modifying the polygon object; and updating the editable map to show the modified polygon object. (See Hartfiel [0020], “The system further includes a display for showing candidate suggested modifications to a user and an input device for entering user validation.”
Also see Hartfiel [0019], “The processor may be further configured to generate a composite raster representation of the salient polygons overlaid on the planar representation of the vector imagery, implement a first machine learning model to identify candidate regions for salient polygon improvement, implement a second machine learning model to generate suggested modifications to salient polygon vertices, receive user validation of the suggested modifications, update the first and second machine learning models based on the validated modifications, and apply the validated modifications to create improved polygons . . .”)
Regarding Claim 4, Hartfiel in view of Dong and Dahm disclose The method of claim 3, further comprising: feeding back the user input modifying the polygon object to the neural network to further train the neural network to automatically place the polygon object in the editable map. (See Hartfiel [0019], “receive user validation of the suggested modifications, update the first and second machine learning models based on the validated modifications, and apply the validated modifications to create improved polygons . . .”)
Regarding Claim 5, Hartfiel in view of Dong and Dahm disclose The method of claim 1, further comprising: providing the base map as a CAD file, an image file or a scanned file of a hardcopy architectural drawing. (See Hartfiel [0002], “Design files (e.g. CAD, SVG, DXF) . . .”)
Regarding Claim 8, Hartfiel in view of Dong and Dahm disclose The method of claim 1, further comprising: presenting the editable map superimposed on the base map on the display interface. (See Dahm [0071], “A floor plan 605 (e.g., in Joint Photographic Experts Group (JPEG) or portable network graphics (PNG) format) may be superimposed on a map 610 (e.g., a Google Earth image) to generate a web-based interactive map 615. Upon generation of the web-based interactive map 615, multiple tools to trace and edit the floor plan 605 can be added to the web-based interactive map 615.” The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 9, Hartfiel in view of Dong and Dahm disclose The method of claim 1, further comprising: identifying an enclosed area in the editable map; (See Dong Page 2 Paragraph 1, “Gimenez et al. [8] also discussed methods that can be used to recognize walls, openings, and spaces. Special segmentation and recognition methods for text annotations, which could obtain high-level semantic information about scale [9], measurement [10], type of subspace [11], were proposed.”)
classifying polygon objects, symbols and/or text within the enclosed area by the neural network; and assigning a room type for the enclosed space based on classification of the polygon objects, symbols and/or text within the enclosed area. (See Dong Page 2 Paragraph 1, “Gimenez et al. [8] also discussed methods that can be used to recognize walls, openings, and spaces. Special segmentation and recognition methods for text annotations, which could obtain high-level semantic information about scale [9], measurement [10], type of subspace [11], were proposed.”
See Dong Page 12 “4.3. Classifying of Subspaces Based on GNN” Paragraph 1, “A new dataset that contains feature matrices annotated with subspace types is established to validate the advantage of GNN. The distributions of instances in the dataset are listed in Table 1. The features used here include window ratio, area ratio, number of doors, number of windows, and number of edges.” Also see Table 1 showing different subspace types (room type for the enclosed space). The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 10, Hartfiel in view of Dong and Dahm disclose The method of claim 1, wherein the vectorized polygon representation of the classified architectural data comprises: line strings representing walls; and (See Dong Page 2 Fig. 1(a) and 1(b) showing the vectorization of the floor plan. Specifically, in 1(b), line strings are representing walls.)
polygon bounding boxes representing doors and windows. (See Dong Page 5 Fig. 3 showing bounding boxes representing doors and windows. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 11, Hartfiel in view of Dong and Dahm disclose The method of claim 1, wherein tracing the architectural feature in the base map comprises: tracing only a portion of the architectural feature. (See Hartfiel [0004], “In other cases, a design file may be “noisy,” with incomplete polygon tracing.”)
Regarding Claim 13, Hartfiel in view of Dong and Dahm disclose The method of claim 1, wherein tracing the architectural feature in the base map comprises: commencing tracing of the architectural feature at a vertex. (See Dahm [0071] teaching to be able to trace the architectural feature in the base map. Since Dahm doesn’t specify how to trace, and merely provides a trace tool, then it would be obvious to be able to commence tracing of the architectural feature at a vertex. The motivation to combine would have been similar that of Claim 1 rejection motivation.)
Regarding Claim 14, Hartfiel in view of Dong and Dahm disclose A system for machine-assisted digital map editing, comprising: (See Hartfiel Abstract, “A system and method for tracing polygons in a drawing source file.”)
a display interface; an input device; (Hartfiel [0020], “The system further includes a display for showing candidate suggested modifications to a user and an input device for entering user validation.”)
a processor; and a memory for storing processor-executable instructions including a trained machine learning model, wherein upon execution of the processor-executable instructions by the processor, the system is configured to: (Hartfiel [0018], “Provided is a system for tracing polygons in a source file. . . a memory coupled to the processor . . .”)
classify architectural data in a digital base map by the trained machine learning neural network; generate a vectorized polygon representation of the classified architectural data; present the base map on the display interface; receive user input via the input device tracing an architectural feature in the base map on the display interface; and propose or automatically place a polygon object representing the traced architectural feature in an editable map. (The above limitations are similar to those of Claim 1 and is therefore rejected under a similar rationale as that of Claim 1.)
Regarding Claim 15, Claim 15 contains similar limitations as to Claim 3 and is therefore rejected under a similar rationale as Claim 3.
Regarding Claim 16, Claim 16 contains similar limitations as to Claim 8 and is therefore rejected under a similar rationale as Claim 8.
Regarding Claim 17, Claim 17 contains similar limitations as to Claim 9 and is therefore rejected under a similar rationale as Claim 9.
Regarding Claim 18, Hartfiel in view of Dong and Dahm disclose The system of claim 14, wherein the trained machine learning model comprises: a generative adversarial network (GAN) trained using a paired set of training images, comprising: (See Dong Abstract, “A 2D floor plan (FP) often contains structural, decorative, and functional elements and annotations. Vectorization of floor plans (VFP) is an object detection task that involves the localization and recognition of different structural primitives in 2D FPs. . . Second, a new edge-extracting GAN (EdgeGAN) is designed for the new task by formulating the VFP task as an image translation task innovatively that involves the projection of the original 2D FPs into a primitive space.”)
a set of base map images as inputs; and a set of vectorized polygon representations as outputs, wherein each base map image is associated to a vectorized polygon representation. (The above limitations are similar to those of Claim 2 and is therefore rejected under a similar rationale as that of Claim 2.)
Regarding Claim 19, Hartfiel in view of Dong and Dahm disclose The system of claim 18, wherein the GAN comprises: a generator neural network configured to: generate a sample polygon representation upon input of a random noise vector; and (See Hartfiel Abstract and [0005] teaching to generate polygons from lines.
See Dong Page 6 Paragraph 2, “generator G to generate better PFM Z and the discriminator to recognize the difference between the distribution of Z and that of the ground truth Y.” Note that generator G corresponds with “a generator neural network” and in combination with Hartfiel, the output would be “a sample polygon representation” instead of a just a primitives feature map. Lastly, note that although Dong doesn’t explicitly disclose “input of a random noise vector”, it is well-known in the art that a random noise vector is the starting point for a GAN's generator.)
a discriminator neural network configured to: determine whether the sample is real or generated in comparison to real-world data; and (See Dong Page 6 Paragraph 2, “generator G to generate better PFM Z and the discriminator to recognize the difference between the distribution of Z and that of the ground truth Y.” Here, Dong teaches a discriminator (a discriminator neural network) that determines if the sample generated by generator is real compared to ground truth Y (real-world data).)
feed back a result to the generator neural network. (See Dong Page 6 showing the loss functions used to guide the generator to create better results. Once again, this feature is standard for all GAN. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 20, Hartfiel in view of Dong and Dahm The system of claim 19, wherein the generator neural network is trained to: receive the base map as an input; and generate the vectorized polygon representation as an output. (See Dong Page 2 Fig. 1(a) and 1(b) showing the vectorization of the floor plan. In this case, the floor plan (base map) is used as input to generate a primitives feature map. Also see Hartfiel Abstract and [0005] teaching to generate polygons from lines. The combination of the these two arts would output “the vectorized polygon representation” of the input base map. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Claims 6-7 is rejected under 35 U.S.C. 103 as being unpatentable over Hartfiel in view of Dong and Dahm and in further view of Colburn et al. (US 20200116493 A1)(Hereinafter referred to as Colburn).
Regarding Claim 6, Hartfiel in view of Dong and Dahm fail to explicitly disclose The method of claim 1, further comprising: classifying architectural features as connections in the base map by the trained neural network; automatically placing the connections in the editable map; and assigning a floor span for each connection.
Colburn teaches classifying architectural features as connections in the base map by the trained neural network; (See Claim 8, “. . . the building has multiple stories, wherein the using of the obtained information to automatically generate the floor map further includes automatically generating, by the computing device, a sub-map of the floor map for each of the multiple stories, and identifying connections between sub-maps based at least in part on connecting passages between the sub-maps,”
In combination with Hartfiel and Dong teaching the trained neural network (see Hartfiel and Dong in Claim 1), classifying the connections would be another structural element that needs to be recognized.)
automatically placing the connections in the editable map; and (See Claim 8 teaching identifying connections in the base map. See Hartfiel [0007] teaching automatically placing polygons. In combination, with would be obvious to also automatically place the connections in the editable map as well.)
assigning a floor span for each connection. (See Claim 8, “. . . the building has multiple stories, wherein the using of the obtained information to automatically generate the floor map further includes . . .” In this case, the obtained information would inherently have the “floor span” for each connection as that would be standard in Floor Plans to contain such information regarding connection. Even if not inherently, assigning a floor span for each connection would be obvious to implement as once again, floor span information is well known in the space of Floor Plan information.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Hartfiel in view of Dong and Dahm with Colburn to include being able to classify and place connections from the base map.
The motivation to combine Hartfiel in view of Dong and Dahm with Colburn would have been obvious as Hartfiel, Dong, and Colburn are all in the same field of floor plan processing, see Colburn Abstract. Connections between sub-maps are a very common architectural feature, so being able to classify and identify them would be a beneficial task when trying to process the elements of a floor plan as they can provide additional information regarding the room/layout of the floor plan.
Regarding Claim 7, Hartfiel in view of Dong, Dahm, Colburn disclose The method of claim 6, further comprising: proposing placement of the connections in the editable map; and receiving user input confirming the placement of the connections. (See Colburn Claim 8 teaching identifying connections. See Hartfiel [0007] for placement of polygons.
Also see Hartfiel [0020], “The system further includes a display for showing candidate suggested modifications to a user and an input device for entering user validation.”
Lastly see Hartfiel [0019], “receive user validation of the suggested modifications, update the first and second machine learning models based on the validated modifications, and apply the validated modifications to create improved polygons . . .”
Here, Hartfiel [0020] and [0019] shows the feature of shows proposals for modifying polygon which requires user validation (user input confirming). In this case, proposing placement of the connections in the editable map which needs user input confirming the placement would be a well-known and inherent feature in floor plans. Even if it is not inherent, it would be obvious to implement based on the current systems of Hartfiel, a way of proposing the placement of connections and needing user input confirming that placement. The motivation to combine would have been similar to that of Claim 6 rejection motivation.)
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Hartfiel in view of Dong and Dahm and in further view of Gootee et al. (US 20140229426 A1)(Hereinafter referred to as Gootee).
Regarding Claim 12, Hartfiel in view of Dong and Dahm disclose The method of claim 1, wherein tracing the architectural feature in the base map comprises: tracing an area around the architectural feature. (See Dahm [0071] teaching to be able to trace the architectural feature in the base map. Since Dahm doesn’t specify how to trace, and merely provides a trace tool, then it would be obvious to be able to trace an area around the architectural feature.)
Gootee additionally shows tracing an area around the architectural feature. (See Fig. 4 showing ability to annotate with drawing a box or circle around architectural features.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Hartfiel in view of Dong and Dahm with Gootee to include tracing an area around the architectural feature.
The motivation to combine Hartfiel in view of Dong and Dahm with Gootee would obvious as Hartfiel, Dong, and Gootee are all in the same field of floor plan processing. Note that being able to trace an area around the architectural feature a very common and well-known idea.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THANG G HUYNH whose telephone number is (571)272-5432. The examiner can normally be reached Mon-Thu 7:30am-4:30pm EST | Fri 7:30am-11:30am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.G.H./Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611