DETAILED ACTION
Claims 1-27 are presented for examination.
This Office Action is in response to submission of documents on February 26, 2024.
Rejection of claims 1-2, 5-19, 21, 23, and 25-26 under 35 U.S.C. 102(a)(1) as being anticipated by Segev.
Rejection of claims 3-4, 20, and 24 under 35 U.S.C. 103 as being obvious over Segev in view of Yin.
Rejection of claim 22 under 35 U.S.C. 103 as being obvious over Segev in view of Rawat.
Rejection of claim 27 under 35 U.S.C. 103 as being obvious over Segev in view of Woh.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on February 27, 2022; January 9, 2024; and February 26, 2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 5-19, 21, 23, and 25-26 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Segev, et al. (U.S. Pat. Pub. No. 2021/0073435, hereinafter “Segev”).
Claim 1
Segev discloses:
A computer-implemented method comprising:
Disclosed embodiments, systems, methods, and computer readable media related to extracting data from a 2D floor plan and retaining it in a building information model are disclosed. Segev at [0012].
obtaining, by one or more computing devices, data about a house with multiple rooms, including a plurality of images acquired at the house,
For example, a floor plan may be accessed by receiving a digital, textual, hand draw, hard copy, photographic, or any other representation of a floor plan as stored in a data structure. Segev at [0217].
In some embodiments, analyzing image data (for example by the methods, steps and modules described herein) may comprise analyzing the image data to obtain a preprocessed image data, and subsequently analyzing the image data and/or the preprocessed image data to obtain the desired outcome. Some non-limiting examples of such image data may include one or more images, videos, frames, footages, 2D image data, 3D image data, and so forth.
a floor plan for the house that includes a room layout with at least two-dimensional room shapes and relative positions of the multiple rooms, and
FIG. 3A illustrates an example generative analysis for generating a coverage map, consistent with the disclosed embodiments. As described above, the disclosed system may include accessing a floor plan 310, which may depict several rooms. Segev at [0187].
a textual description of the house;
As discussed above, semantic enrichment may include employing a computational or classification algorithm along with an analysis method to generate semantic designations for a floor plan. Segev at [0333].
The analysis method may produce output data of several types indicating the spaces within the floor plan and information about the spaces including geometric data (including the shapes, relative sizes, relative locations, and other information related to floor plan or space geometry), textual data (including any existing semantic designations, labels, or other text on a floor plan related to spaces of the floor plan)… Segev at [0333].
generating, by the one or more computing devices, additional information about the house based on the obtained data, including:
Accordingly, disclosed embodiments provide additional information at scale by enabling the identification of accurate, consistent semantic designations in floor plans. Segev at [0777].
analyzing, by the one or more computing devices and using one or more trained first neural networks models, the plurality of images to identify multiple objects inside the house, to determine attributes of the multiple objects, and to determine positions of the multiple objects within the multiple rooms;
In some embodiments generatively analyzing the room may include use of machine learning. Embodiments consistent with the present disclosure may include using artificial intelligence (i.e., machine learning models) for identifying wall contours, rooms, windows, walls, doors, door sills, architectural features, semantic features, many other possible features as well as predicting energy consumption and equipment types and locations. Segev at [0483].
The analysis method may produce output data of several types indicating the spaces within the floor plan and information about the spaces including geometric data (including the shapes, relative sizes, relative locations, and other information related to floor plan or space geometry). Segev at [0333].
analyzing, by the one or more computing devices and using one or more trained second neural network models, the floor plan to determine further attributes of the house that each corresponds to a characteristic of the room layout; and
Generative analysis may include implementing a trained model to provide a solution. As an example, machine learning model 380 may accept floor plan 370 as input and provide solution 390 identifying sensor placement location to maximize sensor coverage. Segev at [0203].
More generally, the generative analysis process illustrated in FIG. 3B may be applied to train models to provide solutions that at least partly conform to any functional requirement for any floor plan, and the solutions may include one or more equipment placement locations. Segev at [0203].
generating, by the one or more computing devices and using one or more trained language
The system may mark an area on the floor plan with a visual indication or representation, including lines, dots, shading, color, highlighting, text, symbols, or other means of identifying an area. Segev at [0228].
A semantic analysis may include Natural Language Processing algorithms such as word2vector used to infer semantic meaning even when it is not clearly represented in the data. For example, a Natural Language Processing or similar algorithm may be used to infer that “hallway” is another word for a “corridor” or that a “recliner” is a type of “sofa.” Segev at [0329].
A Natural Language Processor is a language model.
updating, by the one or more computing devices, the textual description of the house to add data from the generated additional information to contents of the textual description;
In some embodiments, semantic enrichment may include updating or augmenting a prior semantic designation. As described above, a floor plan may include one or more existing designations. When semantic enrichment is performed on the floor plan, one or more of these prior existing semantic designations may be updated by replacement with a new designation. Segev at [0350].
receiving, by the one or more computing devices, one or more search criteria;
A floor plan may be accessed based on a search of floor plans using search criteria. For example, the at least one processor may identify a floor plan using a Boolean search method of textual data associated with a floor plan, such as identifiers of the floor plan. Segev at [0136].
determining, by the one or more computing devices and based on at least some of the added data from the generated additional information in the updated textual description, that the house matches the one or more search criteria; and
Alternatively or additionally, accessing a floor plan may include identifying a floor plan which satisfies a minimum or maximum size, has a specific number of rooms, has a date of creation that meets particular criteria, has a room of a particular type, is associated with a functional requirement, or which satisfies any other search condition. Segev at [0136].
presenting, by the one or more computing devices and in response to the determining, search results that indicate the house and
The system may then output the technical specifications for cameras 322, 324, and 326, as well as the equipment placement locations. For example, the output may be displayed via a user interface 330. The output may include a coverage map 350, which may include the equipment placement locations for cameras 322, 324, and 326, as well as the corresponding coverage areas. Segev at [0187].
include at least some of the updated textual description with the added data from the generated additional information.
The output may further include a materials list 340 (which may correspond to the output technical specification, discussed above). Materials list 340 may list the cameras included in the coverage map along with a model number or other technical information associated with cameras 322, 324, and 326. Segev at [0187].
Claim 2
Segev discloses:
wherein the updating of the textual description of the house further includes validating, by the one or more computing devices, some of the contents of the existing textual description based on the generated additional information, and correcting, by the one or more computing devices, other of the contents of the existing textual description based on the generated additional information, and
In some embodiments, the NLP algorithm may be configured to classify incomplete or incorrect text. As an example, the NLP algorithm may classify a furniture label “mtress” as a “bed.” In some embodiments, the NLP algorithm may be configured to classify text from a plurality of languages. In some embodiments, the classification may remain within the source location of the existing labels. Segev at [0337].
wherein the presenting of the search results further includes presenting the updated textual description with one or more indications of at least one of the validating or the correcting.
In some embodiments, a semantic enrichment may include indicating a confidence rating for a semantic designation. A confidence rating for semantic designations may relates to the degree of the match identified between a space and a determined semantic designation. In some cases, characteristics of spaces may not be identified with a 100% certainty in the semantic enrichment process. The reasons for this may be an ambiguous or meaningless room tag (“IR 300”) or a geometry that lacks any clearly identifying features. To account for this, the semantic enrichment algorithm may be configured to provide a confidence rating. For example, a room already tagged as “office” may be assigned the semantic designator “office” with a 100% confidence, while a room tagged as “Guy's room” may be also designated as an office, but with a comparatively low (e.g., 30%) confidence rating. Segev at [0354].
Indicating a confidence rating is analogous to “presenting an indication of at least one of the validating or the correcting.”
Claim 5
Segev discloses:
A computer-implemented method comprising:
Disclosed embodiments, systems, methods, and computer readable media related to extracting data from a 2D floor plan and retaining it in a building information model are disclosed. Segev at [0012].
obtaining, by one or more computing devices, data about an indicated building with multiple rooms, including a plurality of images acquired at the indicated building and a floor plan for the indicated building having information about the multiple rooms that includes at least two-dimensional room shapes and relative positions of the multiple rooms;
For example, a floor plan may be accessed by receiving a digital, textual, hand draw, hard copy, photographic, or any other representation of a floor plan as stored in a data structure. Segev at [0217].
In some embodiments, analyzing image data (for example by the methods, steps and modules described herein) may comprise analyzing the image data to obtain a preprocessed image data, and subsequently analyzing the image data and/or the preprocessed image data to obtain the desired outcome. Some non-limiting examples of such image data may include one or more images, videos, frames, footages, 2D image data, 3D image data, and so forth. Segev at [0212].
generating, by the one or more computing devices, additional information about the indicated building based on the obtained data, including:
Accordingly, disclosed embodiments provide additional information at scale by enabling the identification of accurate, consistent semantic designations in floor plans. Segev at [0777].
Generative analysis may include implementing a trained model to provide a solution. As an example, machine learning model 380 may accept floor plan 370 as input and provide solution 390 identifying sensor placement location to maximize sensor coverage. Segev at [0203].
determining, by the one or more computing devices and using one or more trained machine learning models, a plurality of attributes about the indicated building that are part of the additional information, including analyzing the plurality of images to determine some of the plurality of attributes based at least in part on objects identified in the plurality of images, and further including analyzing the floor plan to determine one or more additional attributes of the plurality of attributes; and
The system may mark an area on the floor plan with a visual indication or representation, including lines, dots, shading, color, highlighting, text, symbols, or other means of identifying an area. Segev at [0228].
generating, by the one or more computing devices and using one or more trained language models, a textual description of the indicated building that is part of the additional information and is based at least in part on the determined plurality of attributes;
In some embodiments, semantic enrichment may include updating or augmenting a prior semantic designation. As described above, a floor plan may include one or more existing designations. When semantic enrichment is performed on the floor plan, one or more of these prior existing semantic designations may be updated by replacement with a new designation. Segev at [0350].
determining, by the one or more computing devices, that the indicated building matches one or more indicated criteria based on at least some of the generated additional information; and
A floor plan may be accessed based on a search of floor plans using search criteria. For example, the at least one processor may identify a floor plan using a Boolean search method of textual data associated with a floor plan, such as identifiers of the floor plan. Segev at [0136].
Alternatively or additionally, accessing a floor plan may include identifying a floor plan which satisfies a minimum or maximum size, has a specific number of rooms, has a date of creation that meets particular criteria, has a room of a particular type, is associated with a functional requirement, or which satisfies any other search condition. Segev at [0136].
presenting, by the one or more computing devices and in response to the determining that the indicated building matches the one or more indicated criteria, the at least some of the generated additional information about the indicated building.
The system may then output the technical specifications for cameras 322, 324, and 326, as well as the equipment placement locations. For example, the output may be displayed via a user interface 330. The output may include a coverage map 350, which may include the equipment placement locations for cameras 322, 324, and 326, as well as the corresponding coverage areas. Segev at [0187].
The output may further include a materials list 340 (which may correspond to the output technical specification, discussed above). Materials list 340 may list the cameras included in the coverage map along with a model number or other technical information associated with cameras 322, 324, and 326. Segev at [0187].
Claim 6
Segev discloses:
receiving one or more search criteria that include the one or more indicated criteria, wherein the determining that the indicated building matches the one or more indicated criteria is performed as part of determining search results that satisfy the one or more search criteria and include the indicated building, and
Aspects of this disclosure may relate to systems, methods and computer readable media for automatically indexing rooms in a floor plan and group similar rooms so that actions implemented on one room in the group may be applied to others in the group. Such indexes may facilitate rapid searching and may enable efficient application of bulk actions (e.g., locating equipment in positions within a group of rooms based on shared semantic designations among the group of rooms). Segev at [0763].
wherein the presenting of the at least some of the generated additional information about the indicated building includes transmitting, by the one or more computing devices and over one or more computer networks, the determined search results to one or more client devices for display on the one or more client devices.
For ease of discussion, a method is described below with the understanding that aspects of the method apply equally to systems, devices, and computer readable media. For example, some aspects of such a method may occur electronically over a network that is either wired, wireless, or both. Segev at [0119].
Additionally or alternatively, an output may be displayed via a graphical user interface on a display or virtually. A solution may include a floor plan containing one or more areas of interest or disinterest. Segev at [0243].
Claim 7
Segev discloses:
obtaining, by the one or more computing devices, an existing textual description of the indicated building that is separate from the generated textual description
In some embodiments, identifying room contours, a generative analyzing process, and/or other processes such as the performing of semantic enrichment may be computed on a cloud-based servers or interface or retrieved from a cloud-based interface. Segev at [0137].
performing, by the one or more computing devices, at least one of
validating contents of the existing textual description based on the generated additional information, or
In some embodiments, the NLP algorithm may be configured to classify incomplete or incorrect text. As an example, the NLP algorithm may classify a furniture label “mtress” as a “bed.” In some embodiments, the NLP algorithm may be configured to classify text from a plurality of languages. In some embodiments, the classification may remain within the source location of the existing labels. Segev at [0337].
updating the contents of the existing textual description to add data from the generated additional information, and
In some embodiments, semantic enrichment may include updating or augmenting a prior semantic designation. As described above, a floor plan may include one or more existing designations. When semantic enrichment is performed on the floor plan, one or more of these prior existing semantic designations may be updated by replacement with a new designation. Segev at [0350].
wherein the presenting of the at least some of the generated additional information about the indicated building includes presenting the existing textual description with the at least one of the validated contents or the updated contents.
In some embodiments, a semantic enrichment may include indicating a confidence rating for a semantic designation. A confidence rating for semantic designations may relates to the degree of the match identified between a space and a determined semantic designation. In some cases, characteristics of spaces may not be identified with a 100% certainty in the semantic enrichment process. The reasons for this may be an ambiguous or meaningless room tag (“IR 300”) or a geometry that lacks any clearly identifying features. To account for this, the semantic enrichment algorithm may be configured to provide a confidence rating. For example, a room already tagged as “office” may be assigned the semantic designator “office” with a 100% confidence, while a room tagged as “Guy's room” may be also designated as an office, but with a comparatively low (e.g., 30%) confidence rating. Segev at [0354].
Claim 8
Segev discloses:
wherein the one or more trained machine learning models include one or more first neural networks used for the analyzing of the plurality of images and
In some embodiments, artificial neural networks may be configured to analyze inputs and generate corresponding outputs. Segev at [0126].
one or more second neural networks used for the analyzing of the floor plan, and wherein the method further comprises
For example, artificial intelligence methods including, but not limited to, deep learning networks may be used to segment walls and room contours from a floor plan… Segev at [0127].
training, by the one or more computing devices and before the generating of the additional information, the one or more first neural networks to identify objects and determine positions of the identified objects, and the one or more second neural networks to determine building characteristics that are each based on room layout of a plurality of rooms, and wherein the generating of the additional information includes:
FIG. 24F is an illustration depicting an exemplary machine learning furniture analysis. Machine learning furniture analysis may be used to identify furniture within a floor plan, as described herein. Machine learning furniture analysis may include training a machine learning detection or classification model 2481 to detect furniture of a floor plan using a training data set 2475. Segev at [0710].
identifying, by the one or more computing devices using the trained one or more first neural networks, multiple objects inside the indicated building;
FIG. 24F is an illustration depicting an exemplary machine learning furniture analysis. Machine learning furniture analysis may be used to identify furniture within a floor plan, as described herein. Machine learning furniture analysis may include training a machine learning detection or classification model 2481 to detect furniture of a floor plan using a training data set 2475. Segev at [0710].
determining, by the one or more computing devices using the trained one or more first neural networks, positions of the multiple objects within the multiple rooms; and
In some embodiments, semantic enrichment may include performing a geometric analysis on the floor plan. As used herein, a geometric analysis may include any form analysis for extracting information from a floor plan based on geometries represented in the floor plan. A geometric analysis may include an interrogation of a floor plan represented in a BIM, CAD, PDF or any file format containing geometric entities such as lines, polylines, arcs circles or vectors. The geometric analysis may use coordinates (e.g., XYZ coordinates) of end points of entities to determine properties of the entities and their relationship each other. For example, information derived from a geometric analysis may include line length, line direction, the location of geometric objects in the floor plan… Segev at [0131].
determining, by the one or more computing devices using the trained one or more second neural networks, multiple building characteristics that each corresponds to a characteristic of the room layout.
As used herein, a geometric analysis may include any form analysis for extracting information from a floor plan based on geometries represented in the floor plan. Segev at [0131].
Claim 9
Segev discloses:
A system comprising: one or more hardware processors of one or more computing devices; and
Process 400 may be performed by a processing device, such as any of the processors described throughout the present disclosure. It is to be understood that throughout the present disclosure, the term “processor” is used as a shorthand for “at least one processor.” Segev at [0204].
one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations including at least:
The instructions executed by at least one processor may, for example, be pre-loaded into a memory integrated with or embedded into the controller or may be stored in a separate memory. The memory may include a Random Access Memory (RAM), a Read-Only Memory (ROM), a hard disk, an optical disk, a magnetic medium, a flash memory, other permanent, fixed, or volatile memory, or any other mechanism capable of storing instructions. Segev at [0204].
obtaining data about an indicated building with multiple rooms, including a plurality of images acquired at the indicated building, a floor plan for the indicated building having information about the multiple rooms that includes at least two-dimensional room shapes and relative positions of the multiple rooms, and
For example, a floor plan may be accessed by receiving a digital, textual, hand draw, hard copy, photographic, or any other representation of a floor plan as stored in a data structure. Segev at [0217].
In some embodiments, analyzing image data (for example by the methods, steps and modules described herein) may comprise analyzing the image data to obtain a preprocessed image data, and subsequently analyzing the image data and/or the preprocessed image data to obtain the desired outcome. Some non-limiting examples of such image data may include one or more images, videos, frames, footages, 2D image data, 3D image data, and so forth. Segev at [0212].
a textual description of the building;
The analysis method may produce output data of several types indicating the spaces within the floor plan and information about the spaces including geometric data (including the shapes, relative sizes, relative locations, and other information related to floor plan or space geometry), textual data (including any existing semantic designations, labels, or other text on a floor plan related to spaces of the floor plan)…
generating additional information about the indicated building based on the obtained data, including:
Accordingly, disclosed embodiments provide additional information at scale by enabling the identification of accurate, consistent semantic designations in floor plans. Segev at [0777].
determining a plurality of attributes about the indicated building that are part of the additional information, including analyzing the plurality of images to determine some of the plurality of attributes based at least in part on objects identified in the plurality of images, and further including
Generative analysis may include implementing a trained model to provide a solution. As an example, machine learning model 380 may accept floor plan 370 as input and provide solution 390 identifying sensor placement location to maximize sensor coverage. Segev at [0203].
More generally, the generative analysis process illustrated in FIG. 3B may be applied to train models to provide solutions that at least partly conform to any functional requirement for any floor plan, and the solutions may include one or more equipment placement locations. Segev at [0203].
analyzing the floor plan to determine one or more additional attributes of the plurality of attributes; and
Generative analysis may include implementing a trained model to provide a solution. As an example, machine learning model 380 may accept floor plan 370 as input and provide solution 390 identifying sensor placement location to maximize sensor coverage. Segev at [0203].
More generally, the generative analysis process illustrated in FIG. 3B may be applied to train models to provide solutions that at least partly conform to any functional requirement for any floor plan, and the solutions may include one or more equipment placement locations. Segev at [0203].
generating, using one or more trained language models, a further textual description of the indicated building that is part of the additional information and is based at least in part on the determined plurality of attributes;
The system may mark an area on the floor plan with a visual indication or representation, including lines, dots, shading, color, highlighting, text, symbols, or other means of identifying an area. Segev at [0228].
A semantic analysis may include Natural Language Processing algorithms such as word2vector used to infer semantic meaning even when it is not clearly represented in the data. For example, a Natural Language Processing or similar algorithm may be used to infer that “hallway” is another word for a “corridor” or that a “recliner” is a type of “sofa.” Segev at [0329].
A Natural Language Processor is a language model.
updating the textual description of the building by adding at least some of the generated additional information; and
In some embodiments, semantic enrichment may include updating or augmenting a prior semantic designation. As described above, a floor plan may include one or more existing designations. When semantic enrichment is performed on the floor plan, one or more of these prior existing semantic designations may be updated by replacement with a new designation. Segev at [0350].
providing information about the indicated building that includes the at least some generated additional information.
The system may then output the technical specifications for cameras 322, 324, and 326, as well as the equipment placement locations. For example, the output may be displayed via a user interface 330. The output may include a coverage map 350, which may include the equipment placement locations for cameras 322, 324, and 326, as well as the corresponding coverage areas. Segev at [0187].
The output may further include a materials list 340 (which may correspond to the output technical specification, discussed above). Materials list 340 may list the cameras included in the coverage map along with a model number or other technical information associated with cameras 322, 324, and 326. Segev at [0187].
Claim 10
Segev discloses:
wherein the at least one computing device includes a server computing device and wherein the one or more computing devices further include a client computing device of a user, and wherein the stored instructions include software instructions that, when executed by the one or more computing devices, cause the one or more computing devices to perform further automated operations including:
In some embodiments, the system may be a cloud-based system. For example, the methods described herein may be executed in a virtual instance by a remote computer, rather than a local processor. Cloud-based computation can provide advantages in scalability, cost, and security over approaches that rely on local computational or conventional servers. Segev at [0250].
receiving, by the server computing device, one or more search criteria from the client computing device;
A floor plan may be accessed based on a search of floor plans using search criteria. For example, the at least one processor may identify a floor plan using a Boolean search method of textual data associated with a floor plan, such as identifiers of the floor plan. Segev at [0136].
determining, by the server computing device, search results for the search criteria that include the indicated building based at least in part on the at least some generated additional information;
Alternatively or additionally, accessing a floor plan may include identifying a floor plan which satisfies a minimum or maximum size, has a specific number of rooms, has a date of creation that meets particular criteria, has a room of a particular type, is associated with a functional requirement, or which satisfies any other search condition. Segev at [0136].
performing, by the server computing device, the providing of the information about the indicated building by transmitting the information about the indicated building over one or more computer networks to the client computing device, the transmitted information including the determined search results; and
The system may then output the technical specifications for cameras 322, 324, and 326, as well as the equipment placement locations. For example, the output may be displayed via a user interface 330. The output may include a coverage map 350, which may include the equipment placement locations for cameras 322, 324, and 326, as well as the corresponding coverage areas. Segev at [0187].
receiving, by the client computing device, the transmitted information including the determined search results, and displaying the determined search results on the client computing device.
This may include a manual selection, applying one or more filters, applying one or more searches, or a combination thereof. In some embodiments, enabling the user to select a room from the index may include enabling the user to click in the floor plan on a room that was previously indexed. Accordingly, the disclosed methods may include displaying the floor plan for selection of rooms by a user. Segev at [0795].
Claim 11
Segev discloses:
wherein the determining of the plurality of attributes includes using one or more trained machine learning models, and wherein the providing of the information about the indicated building includes
In some embodiments generatively analyzing the room may include use of machine learning. Embodiments consistent with the present disclosure may include using artificial intelligence (i.e., machine learning models) for identifying wall contours, rooms, windows, walls, doors, door sills, architectural features, semantic features, many other possible features as well as predicting energy consumption and equipment types and locations. Segev at [0483].
The analysis method may produce output data of several types indicating the spaces within the floor plan and information about the spaces including geometric data (including the shapes, relative sizes, relative locations, and other information related to floor plan or space geometry). Segev at [0333].
presenting the updated textual description for the indicated building.
In some embodiments, semantic enrichment may include updating or augmenting a prior semantic designation. As described above, a floor plan may include one or more existing designations. When semantic enrichment is performed on the floor plan, one or more of these prior existing semantic designations may be updated by replacement with a new designation. Segev at [0350].
Claim 12
Segev discloses:
wherein the one or more trained machine learning models include one or more first neural networks used for the analyzing of the plurality of images and
In some embodiments generatively analyzing the room may include use of machine learning. Embodiments consistent with the present disclosure may include using artificial intelligence (i.e., machine learning models) for identifying wall contours, rooms, windows, walls, doors, door sills, architectural features, semantic features, many other possible features as well as predicting energy consumption and equipment types and locations. Segev at [0483].
The analysis method may produce output data of several types indicating the spaces within the floor plan and information about the spaces including geometric data (including the shapes, relative sizes, relative locations, and other information related to floor plan or space geometry). Segev at [0333].
one or more second neural networks used for the analyzing of the floor plan, and wherein the automated operations further include
Generative analysis may include implementing a trained model to provide a solution. As an example, machine learning model 380 may accept floor plan 370 as input and provide solution 390 identifying sensor placement location to maximize sensor coverage. Segev at [0203].
More generally, the generative analysis process illustrated in FIG. 3B may be applied to train models to provide solutions that at least partly conform to any functional requirement for any floor plan, and the solutions may include one or more equipment placement locations. Segev at [0203].
training, before the generating of the additional information, the one or more first neural networks to identify objects and determine positions of the identified objects, and the one or more second neural networks to determine building characteristics that are each based on room layout of a plurality of rooms, and wherein the
For example, the generative analysis may include implementing a machine learning algorithm trained to identify technical specifications and placement locations based on a training set of floor plans that partially comply with a particular functional requirement. Segev at [0158].
In some embodiments, machine learning models, such as one or more ResNet models trained to classify architectural features, or classifications models such as XGBoost trained to predict room functions, may be stored, for example, in cloud based file hosting services (such as Amazon S3, Azure Blob storage, Google cloud storage, DigitalOcean spaces, etc.). Segev at [0138].
generating of the additional information includes: identifying, using the trained one or more first neural networks, multiple objects inside the indicated building;
FIG. 24F is an illustration depicting an exemplary machine learning furniture analysis. Machine learning furniture analysis may be used to identify furniture within a floor plan, as described herein. Machine learning furniture analysis may include training a machine learning detection or classification model 2481 to detect furniture of a floor plan using a training data set 2475. Segev at [0710].
determining, using the trained one or more first neural networks, positions of the multiple objects within the multiple rooms; and
In some embodiments, semantic enrichment may include performing a geometric analysis on the floor plan. As used herein, a geometric analysis may include any form analysis for extracting information from a floor plan based on geometries represented in the floor plan. A geometric analysis may include an interrogation of a floor plan represented in a BIM, CAD, PDF or any file format containing geometric entities such as lines, polylines, arcs circles or vectors. The geometric analysis may use coordinates (e.g., XYZ coordinates) of end points of entities to determine properties of the entities and their relationship each other. For example, information derived from a geometric analysis may include line length, line direction, the location of geometric objects in the floor plan… Segev at [0131].
determining, using the trained one or more second neural networks, multiple building characteristics that each corresponds to a characteristic of the room layout.
As used herein, a geometric analysis may include any form analysis for extracting information from a floor plan based on geometries represented in the floor plan. Segev at [0131].
Claim 13
Claim 13 recites A non-transitory computer-readable medium that stores instructions to perform essentially the same steps as the method recited in claim 1. Accordingly, for at least the same reasons as asserted for claim 1, claim 13 is anticipated by Segev.
Claim 14
Segev discloses:
wherein the stored contents include software instructions that, when executed by the one or more computing devices, cause the one or more computing devices to perform further automated operations including receiving one or more search criteria from the client computing device, wherein the determining that the indicated building matches the one or more indicated criteria is performed as part of determining search results that satisfy the one or more search criteria and include the indicated building, and
A floor plan may be accessed based on a search of floor plans using search criteria. For example, the at least one processor may identify a floor plan using a Boolean search method of textual data associated with a floor plan, such as identifiers of the floor plan. Segev at [0136].
wherein the providing of the at least some of the generated additional information about the indicated building includes transmitting, by the one or more computing devices and over one or more computer networks, the determined search results to one or more client devices for display on the one or more client devices.
The system may then output the technical specifications for cameras 322, 324, and 326, as well as the equipment placement locations. For example, the output may be displayed via a user interface 330. The output may include a coverage map 350, which may include the equipment placement locations for cameras 322, 324, and 326, as well as the corresponding coverage areas. Segev at [0187].
Claim 15
Segev discloses:
wherein the automated operations further include:
obtaining, by the one or more computing devices, an existing textual description of the indicated building that is separate from the generated textual description; and
As discussed above, semantic enrichment may include employing a computational or classification algorithm along with an analysis method to generate semantic designations for a floor plan. Segev at [0333].
The analysis method may produce output data of several types indicating the spaces within the floor plan and information about the spaces including geometric data (including the shapes, relative sizes, relative locations, and other information related to floor plan or space geometry), textual data (including any existing semantic designations, labels, or other text on a floor plan related to spaces of the floor plan)… Segev at [0333].
updating, by the one or more computing devices, contents of the existing textual description to add data from the generated additional information, and wherein the providing of the at least some of the generated additional information about the indicated building includes providing the existing textual description with the updated contents.
In some embodiments, semantic enrichment may include updating or augmenting a prior semantic designation. As described above, a floor plan may include one or more existing designations. When semantic enrichment is performed on the floor plan, one or more of these prior existing semantic designations may be updated by replacement with a new designation. Segev at [0350].
Claim 16
Segev discloses:
wherein the automated operations further include:
obtaining, by the one or more computing devices, an existing textual description of the indicated building that is separate from the generated textual description; and
As discussed above, semantic enrichment may include employing a computational or classification algorithm along with an analysis method to generate semantic designations for a floor plan. Segev at [0333].
The analysis method may produce output data of several types indicating the spaces within the floor plan and information about the spaces including geometric data (including the shapes, relative sizes, relative locations, and other information related to floor plan or space geometry), textual data (including any existing semantic designations, labels, or other text on a floor plan related to spaces of the floor plan)… Segev at [0333].
validating, by the one or more computing devices, contents of the existing textual description based on the generated additional information, and wherein the providing of the at least some of the generated additional information about the indicated building includes
In some embodiments, the NLP algorithm may be configured to classify incomplete or incorrect text. As an example, the NLP algorithm may classify a furniture label “mtress” as a “bed.” In some embodiments, the NLP algorithm may be configured to classify text from a plurality of languages. In some embodiments, the classification may remain within the source location of the existing labels. Segev at [0337].
providing the existing textual description with an indication of the validated contents or the updated contents.
In some embodiments, a semantic enrichment may include indicating a confidence rating for a semantic designation. A confidence rating for semantic designations may relates to the degree of the match identified between a space and a determined semantic designation. In some cases, characteristics of spaces may not be identified with a 100% certainty in the semantic enrichment process. The reasons for this may be an ambiguous or meaningless room tag (“IR 300”) or a geometry that lacks any clearly identifying features. To account for this, the semantic enrichment algorithm may be configured to provide a confidence rating. For example, a room already tagged as “office” may be assigned the semantic designator “office” with a 100% confidence, while a room tagged as “Guy's room” may be also designated as an office, but with a comparatively low (e.g., 30%) confidence rating. Segev at [0354].
Claim 17
Segev discloses:
wherein the generating of the additional information includes performing the analyzing of the images to determine some of the plurality of attributes and performing the analyzing of the floor plan to determine one or more additional attributes of the plurality of attributes, and further includes:
identifying, by the one or more computing devices, multiple objects inside the indicated building;
FIG. 24F is an illustration depicting an exemplary machine learning furniture analysis. Machine learning furniture analysis may be used to identify furniture within a floor plan, as described herein. Machine learning furniture analysis may include training a machine learning detection or classification model 2481 to detect furniture of a floor plan using a training data set 2475. Segev at [0710].
determining, by the one or more computing devices, positions of the multiple objects within the multiple rooms; and
In some embodiments, semantic enrichment may include performing a geometric analysis on the floor plan. As used herein, a geometric analysis may include any form analysis for extracting information from a floor plan based on geometries represented in the floor plan. A geometric analysis may include an interrogation of a floor plan represented in a BIM, CAD, PDF or any file format containing geometric entities such as lines, polylines, arcs circles or vectors. The geometric analysis may use coordinates (e.g., XYZ coordinates) of end points of entities to determine properties of the entities and their relationship each other. For example, information derived from a geometric analysis may include line length, line direction, the location of geometric objects in the floor plan… Segev at [0131].
determining, by the one or more computing devices, multiple building characteristics that each corresponds to a characteristic of the room layout.
As used herein, a geometric analysis may include any form analysis for extracting information from a floor plan based on geometries represented in the floor plan. Segev at [0131].
Claim 18
Segev discloses:
wherein the one or more trained machine learning models include one or more first neural networks used for the analyzing of the plurality of images and one or more second neural networks used for the analyzing of the floor plan, wherein the automated operations further include
In some embodiments generatively analyzing the room may include use of machine learning. Embodiments consistent with the present disclosure may include using artificial intelligence (i.e., machine learning models) for identifying wall contours, rooms, windows, walls, doors, door sills, architectural features, semantic features, many other possible features as well as predicting energy consumption and equipment types and locations. Segev at [0483].
The analysis method may produce output data of several types indicating the spaces within the floor plan and information about the spaces including geometric data (including the shapes, relative sizes, relative locations, and other information related to floor plan or space geometry). Segev at [0333].
Generative analysis may include implementing a trained model to provide a solution. As an example, machine learning model 380 may accept floor plan 370 as input and provide solution 390 identifying sensor placement location to maximize sensor coverage. Segev at [0203].
More generally, the generative analysis process illustrated in FIG. 3B may be applied to train models to provide solutions that at least partly conform to any functional requirement for any floor plan, and the solutions may include one or more equipment placement locations. Segev at [0203].
training, by the one or more computing devices and before the generating of the additional information, the one or more first neural networks to identify objects and determine positions of the identified objects, and
For example, the generative analysis may include implementing a machine learning algorithm trained to identify technical specifications and placement locations based on a training set of floor plans that partially comply with a particular functional requirement. Segev at [0158].
In some embodiments, machine learning models, such as one or more ResNet models trained to classify architectural features, or classifications models such as XGBoost trained to predict room functions, may be stored, for example, in cloud based file hosting services (such as Amazon S3, Azure Blob storage, Google cloud storage, DigitalOcean spaces, etc.). Segev at [0138].
the one or more second neural networks to determine building characteristics that are each based on room layout of a plurality of rooms, wherein the identifying of the multiple objects and the determining of the positions of the multiple objects are performed using the trained one or more first neural networks, and wherein the determining of the multiple building characteristics is performed using the trained one or more second neural networks.
As used herein, a geometric analysis may include any form analysis for extracting information from a floor plan based on geometries represented in the floor plan. Segev at [0131].
Claim 19
Segev discloses:
wherein the at least some of the generated additional information used for the determining that the indicated building matches the one or more indicated criteria includes one or more attributes of the determined plurality of attributes, and
Conventionally, identifying similar rooms may be challenging as their semantic designations (e.g. labels indicating a private office, corridor, or other space type) may not be structured. The semantic designation of architectural features, furniture, and equipment located inside rooms may similarly not be structured. The disclosed systems may use machine learning and/or other methods to generate semantic designations of a plurality of rooms and the architectural features, furniture, and equipment located inside the rooms to obviate conventional problems of working directly in floor plans or using labor intensive and inconsistent methods to generate semantic designations. Segev at [0763].
See also FIGS. 28A-E.
wherein the providing of the at least some generated additional information about the indicated building includes identifying the indicated building and including information about the one or more attributes of the indicated building.
The output may further include a materials list 340 (which may correspond to the output technical specification, discussed above). Materials list 340 may list the cameras included in the coverage map along with a model number or other technical information associated with cameras 322, 324, and 326. Segev at [0187].
Claim 21
Segev discloses:
wherein the at least some of the generated additional information used for the determining that the indicated building matches the one or more indicated criteria includes one or more attributes of the determined plurality of attributes, and
Conventionally, identifying similar rooms may be challenging as their semantic designations (e.g. labels indicating a private office, corridor, or other space type) may not be structured. The semantic designation of architectural features, furniture, and equipment located inside rooms may similarly not be structured. The disclosed systems may use machine learning and/or other methods to generate semantic designations of a plurality of rooms and the architectural features, furniture, and equipment located inside the rooms to obviate conventional problems of working directly in floor plans or using labor intensive and inconsistent methods to generate semantic designations. Segev at [0763].
See also FIGS. 28A-E.
wherein the one or more attributes used as part of the determining that the indicated building matches the one or more indicated criteria include one or more local attributes that are generated from the analyzing of the plurality of images and are each associated with one of the multiple rooms, and
In some embodiments the area of disinterest may define an area outside a desired sensor coverage area For example, as described earlier, rather than positively defining a space or area where equipment may be located or sensor coverage may be desired, an area of disinterest may define a negative limitation where equipment placement is not desired or where sensor coverage should be avoided. Sensor coverage may refer to an area where something is able to be detected by a sensor (e.g., a. video surveillance camera, temperature sensor, motion sensor, occupancy sensor, light sensor, water leak sensor, humidity sensors, door contact, glass break sensor, or other detector configured to detect local conditions). Segev at [0232].
at least one global attribute that is generated from the analyzing of the floor plan and is associated with all of the indicated building.
In stage 140, the generative analysis may determine an optimal solution which, in this example, may represent the maximum coverage percentage of 85%. This optimal solution may be output as a result of the generative analysis, as described in greater detail below. For example, the output may include a technical specification for camera 114 and a suggested location for camera 114 that maximizes coverage region 116 and at least partially complies with the functional requirements. While FIG. 1 shows two stages of simulations (120 and 130), it is understood that any number of iterations or simulations may be performed as part of an optimized analysis, consistent with the disclosed embodiments. For example, as shown in FIG. 1, the system may be configured to re-test the best alternatives to hone in on the optimal solution. Segev at [0169].
Claim 23
Segev discloses:
wherein the at least some of the generated additional information used for the determining that the indicated building matches the one or more indicated criteria includes one or more attributes of the determined plurality of attributes, and
Conventionally, identifying similar rooms may be challenging as their semantic designations (e.g. labels indicating a private office, corridor, or other space type) may not be structured. The semantic designation of architectural features, furniture, and equipment located inside rooms may similarly not be structured. The disclosed systems may use machine learning and/or other methods to generate semantic designations of a plurality of rooms and the architectural features, furniture, and equipment located inside the rooms to obviate conventional problems of working directly in floor plans or using labor intensive and inconsistent methods to generate semantic designations. Segev at [0763].
See also FIGS. 28A-E.
wherein the plurality of images include multiple images acquired inside the indicated building, and
In some embodiments, the processor may access a floor plan using an associated scanner, camera, or other device capable of capturing images. Floor plans may also be accessed in other suitable ways, as described herein. Segev at [0576].
wherein the one or more attributes used as part of the determining that the indicated building matches the one or more indicated criteria include one or more architectural features of an interior of the indicated building, the one or more architectural features including at least one of a type of floor of one of the multiple rooms, or a type of ceiling of one of the multiple rooms, or a type of built-in structural element of at least one of the multiple rooms.
Architectural features that might impact such analyses may include room area, ceiling height, lines of sight, window locations, door locations, railing locations, column or furniture locations, locations of other pieces of equipment specified, or any other physical detail. For example, a pitched ceiling or ceiling with dropped beams may necessitate different smoke detector coverage ranges. Segev at [0238].
Claim 25
Segev discloses:
wherein the obtaining of the data about the indicated building further includes acquiring, using one or more cameras, the plurality of images at a plurality of acquisition locations associated with the indicated building, and
In some embodiments, the processor may access a floor plan using an associated scanner, camera, or other device capable of capturing images. Floor plans may also be accessed in other suitable ways, as described herein. Segev at [0576].
generating, by the one or more computing devices, the floor plan based at least in part on analysis of visual data of the plurality of images.
Embodiments may include using artificial intelligence to segment walls and rooms from images of floor plans. For example, artificial intelligence may be used to detect and classify architectural features on floor plan images, detecting features such as windows, doors, door sills, stairs, furniture, equipment or any other features. As another example, artificial intelligence may determine room type using feature based ML models such as XGBoost. Segev at [0667].
Claim 26
Segev discloses:
wherein the stored contents include one or more data structures, the one or more data structures including at least one of the one or more trained machine learning models, or of the one or more trained language models.
In some embodiments, machine learning models such as one or more ResNet models trained to classify architectural features or classifications model such as XGBoost trained to predict room functions, may be stored, for example, in cloud based file hosting services (such as Amazon S3, Azure Blob storage, Google cloud storage, DigitalOcean spaces, etc). The model, for example, may be fetched and loaded into memory (RAM) by microservices during their initialization. Segev at [0510].
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 3, 4, 20, and 24 are rejected under 35 U.S.C. 103 as being obvious over Segev in view of Yin, et al., (U.S. Pat. Pub. No. 2022/0092227, hereinafter “Yin”).
Claim 3
Segev recites:
wherein the identified multiple objects in the house include at least appliances and fixtures and structural elements,
As used herein, equipment may refer to any piece of electronic, mechanic or any other type of hardware, device, machinery, cabinetry, or furniture that has a function and a technical specification. Segev at [0151].
wherein the determined attributes of the multiple objects include colors and types of surface materials,
A technical characteristic may include properties such as mounting device type, resolution, DB rating, luminosity, IR rating, lens type, network speed, wattage requirement, color, dimensions, material, flammability rating, or water-resistance. Segev at [0234].
wherein the determined further attributes of the house include both objective attributes about the house that are able to be independently verified and
Generative analysis may include implementing a trained model to provide a solution. As an example, machine learning model 380 may accept floor plan 370 as input and provide solution 390 identifying sensor placement location to maximize sensor coverage. Segev at [0203].
subjective attributes for the house that are predicted by the one or more trained second neural network models,
In some embodiments, other text data related to a space as whole, but not necessarily to a specific element of the space, may be input into a machine learning algorithm, such a natural language processing (NLP) algorithm. For example, a text data input may include a floor plan room label of “Bob's sleeping quarters.” The NLP algorithm may classify this label as a “bedroom” label because of a learned association between “sleeping quarters” and “bedroom.” Segev at [0341].
one or more types of objects, and
As used herein, a retention data structure is any data structure or other type of data storage used to store data associated with identified wall boundaries as well as other architectural features such as doors, windows, furniture, equipment and room geometric data. As described herein, a data structure may refer to any type of data stored in an organized, searchable manner and include a variety of data storage formats. Segev at [0679].
wherein the determining that the house matches the one or more search criteria is based on one or more of the identified multiple objects and on one or more of the determined attributes of the multiple objects and on one or more subjective attributes of the determined further attributes.
Conventionally, identifying similar rooms may be challenging as their semantic designations (e.g. labels indicating a private office, corridor, or other space type) may not be structured. The semantic designation of architectural features, furniture, and equipment located inside rooms may similarly not be structured. The disclosed systems may use machine learning and/or other methods to generate semantic designations of a plurality of rooms and the architectural features, furniture, and equipment located inside the rooms to obviate conventional problems of working directly in floor plans or using labor intensive and inconsistent methods to generate semantic designations. Segev at [0763].
See also FIGS. 28A-E.
Segev does not appear to disclose:
wherein the search criteria include indications of one or more colors and one or more types of surface materials
Yin, which is analogous art, discloses:
wherein the search criteria include indications of one or more colors and one or more types of surface materials and
In addition, the MIGM system may in at least some embodiments perform further automated operations to determine and associate additional information with a building floor plan and/or specific rooms or locations within the floor plan, such as to analyze images and/or other environmental information (e.g., audio) captured within the building interior to determine particular attributes (e.g., a color and/or material type and/or other characteristics of particular elements… Yin at [0030].
Yin is analogous art to the disclosed invention because both are directed to determining attributes of floor plans in an automated manner. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to identify the attributes disclosed in Yin using the techniques disclosed in Segev to result in a system that generates color and material types of objects in an environment as attributes. Motivation to combine includes improving the usability of the system of Segev by allowing for additional types of attributes to be identified from images.
Claim 4
Segev recites:
wherein the search criteria further include indications of one or more positions of the one or more types of objects,
As disclosed, at least one data structure may refer to any type of data stored in a processor in an organized, searchable manner. Technical specifications may include but are not limited to one or more of an equipment specification, a model identifier, or any technical characteristic describing one or more properties of a piece of equipment. Locations may include information regarding the position of a piece of equipment within a floor plan. Segev at [0534].
wherein the determining that the house matches the one or more search criteria is further based on the determined positions of the one or more identified objects within the multiple, and
In some embodiments, combinations of architectural features may be considered. For example, an architectural feature designated as “chair” may indicate a room is an office when included in the same room as an architectural feature designated as “desk,” but may indicate a room is a living room when included in the same room as an architectural feature designated as “sofa.” Further, the number of features designated as “chair” may indicate a room is a conference room rather than an office. In some embodiments, the spatial architectural features and furniture may be considered. For example, the position of chairs relative to a table may indicate whether the table is a meeting table or a dining table. Segev at [0780].
wherein the method further comprises, before the generating of the additional information about the house: training, by the one or more computing devices, the one or more second neural network models to identify the objective and subjective attributes;
In some embodiments, machine learning models, such as one or more ResNet models trained to classify architectural features, or classifications models such as XGBoost trained to predict room functions, may be stored, for example, in cloud based file hosting services (such as Amazon S3, Azure Blob storage, Google cloud storage, DigitalOcean spaces, etc.). Segev at [0138].
training, by the one or more computing devices, the one or more first neural network models to identify objects and to determine attributes of objects and to determine positions of objects; and
For example, the generative analysis may include implementing a machine learning algorithm trained to identify technical specifications and placement locations based on a training set of floor plans that partially comply with a particular functional requirement. Segev at [0158].
training, by the one or more computing devices, the one or more language models to generate textual descriptions based on attributes of houses.
An NLP algorithm may be configured to classify a room element based on the text associated with it. For example, a text data input may include a furniture label of “mattress.” The NLP algorithm may classify this label as a “bed” label because of a learned association between “mattress” and “bed.” As another example, text data input may include a furniture label of “loveseat.” The NLP algorithm may classify this label as a “sofa” label because of a learned association between “loveseat” and “sofa.” Segev at [0337].
Segev does not appear to disclose:
wherein the one or more subjective attributes include at least one of an atypical floor plan that differs from typical floor plans, or an open floor plan, or an accessible floor plan, or a non- standard floor plan
Yin discloses:
wherein the one or more subjective attributes include at least one of an atypical floor plan that differs from typical floor plans, or an open floor plan, or an accessible floor plan, or a non- standard floor plan
…training one or more classification neural networks or other machine learning models to classify building floor plans according to one or more subjective factors (e.g., accessibility friendly, an open floor plan, an atypical floor plan, a non-standard floor plan, etc.)… Yin at [0100].
It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to identify the attributes disclosed in Yin using the techniques disclosed in Segev to result in a system that subjectively determines a type of floor plan as an attribute. Motivation to combine includes improving the usability of the system of Segev by allowing for additional types of attributes to be identified from images.
Claim 20
Segev discloses:
wherein the at least some of the generated additional information used for the determining that the indicated building matches the one or more indicated criteria includes one or more attributes of the determined plurality of attributes, and
Conventionally, identifying similar rooms may be challenging as their semantic designations (e.g. labels indicating a private office, corridor, or other space type) may not be structured. The semantic designation of architectural features, furniture, and equipment located inside rooms may similarly not be structured. The disclosed systems may use machine learning and/or other methods to generate semantic designations of a plurality of rooms and the architectural features, furniture, and equipment located inside the rooms to obviate conventional problems of working directly in floor plans or using labor intensive and inconsistent methods to generate semantic designations. Segev at [0763].
See also FIGS. 28A-E.
Segev does not appear to disclose:
wherein the one or more attributes used as part of the determining that the indicated building matches the one or more indicated criteria include one or more subjective attributes generated from the analyzing of the floor plan, the one or more subjective attributes including at least one of an open floor plan, or an accessible floor plan, or a non-standard floor plan.
Yin discloses:
wherein the one or more attributes used as part of the determining that the indicated building matches the one or more indicated criteria include one or more subjective attributes generated from the analyzing of the floor plan, the one or more subjective attributes including at least one of an open floor plan, or an accessible floor plan, or a non-standard floor plan.
…training one or more classification neural networks or other machine learning models to classify building floor plans according to one or more subjective factors (e.g., accessibility friendly, an open floor plan, an atypical floor plan, a non-standard floor plan, etc.)… Yin at [0100].
Claim 24
Segev discloses:
wherein the determining of the plurality of attributes includes performing the analyzing the plurality of images to determine attributes based at least in part on objects identified in the plurality of images, and
FIG. 24F is an illustration depicting an exemplary machine learning furniture analysis. Machine learning furniture analysis may be used to identify furniture within a floor plan, as described herein. Machine learning furniture analysis may include training a machine learning detection or classification model 2481 to detect furniture of a floor plan using a training data set 2475. Segev at [0710].
FIG. 24F is an illustration depicting an exemplary machine learning furniture analysis. Machine learning furniture analysis may be used to identify furniture within a floor plan, as described herein. Machine learning furniture analysis may include training a machine learning detection or classification model 2481 to detect furniture of a floor plan using a training data set 2475. Segev at [0710].
In some embodiments, semantic enrichment may include performing a geometric analysis on the floor plan. As used herein, a geometric analysis may include any form analysis for extracting information from a floor plan based on geometries represented in the floor plan. A geometric analysis may include an interrogation of a floor plan represented in a BIM, CAD, PDF or any file format containing geometric entities such as lines, polylines, arcs circles or vectors. The geometric analysis may use coordinates (e.g., XYZ coordinates) of end points of entities to determine properties of the entities and their relationship each other. For example, information derived from a geometric analysis may include line length, line direction, the location of geometric objects in the floor plan… Segev at [0131].
wherein the one or more attributes used as part of the determining that the indicated building matches the one or more indicated criteria include at least one of one or more of the objects,
In addition, the MIGM system may in at least some embodiments perform further automated operations to determine and associate additional information with a building floor plan and/or specific rooms or locations within the floor plan, such as to analyze images and/or other environmental information (e.g., audio) captured within the building interior to determine particular attributes (e.g., a color and/or material type and/or other characteristics of particular elements… Yin at [0030].
Segev does not appear to disclose:
a color and a type of surface material for each of one or more of the objects
Yin, which is analogous art, discloses:
a color and a type of surface material for each of one or more of the objects
In addition, the MIGM system may in at least some embodiments perform further automated operations to determine and associate additional information with a building floor plan and/or specific rooms or locations within the floor plan, such as to analyze images and/or other environmental information (e.g., audio) captured within the building interior to determine particular attributes (e.g., a color and/or material type and/or other characteristics of particular elements… Yin at [0030].
Yin is analogous art to the disclosed invention because both are directed to determining attributes of floor plans in an automated manner. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to identify the attributes disclosed in Yin using the techniques disclosed in Segev to result in a system that generates color and material types of objects in an environment as attributes. Motivation to combine includes improving the usability of the system of Segev by allowing for additional types of attributes to be identified from images.
Claim 22 is rejected under 35 U.S.C. 103 as being obvious over Segev in view of Rawat, et al., (U.S. Pat. No. 10,430,902, hereinafter “Rawat”).
Claim 22
Segev discloses:
wherein the at least some of the generated additional information used for the determining that the indicated building matches the one or more indicated criteria includes one or more attributes of the determined plurality of attributes, and
Conventionally, identifying similar rooms may be challenging as their semantic designations (e.g. labels indicating a private office, corridor, or other space type) may not be structured. The semantic designation of architectural features, furniture, and equipment located inside rooms may similarly not be structured. The disclosed systems may use machine learning and/or other methods to generate semantic designations of a plurality of rooms and the architectural features, furniture, and equipment located inside the rooms to obviate conventional problems of working directly in floor plans or using labor intensive and inconsistent methods to generate semantic designations. Segev at [0763].
See also FIGS. 28A-E.
Segev does not appear to teach or disclose:
wherein the plurality of images include one or more images acquired external to the indicated building, and wherein the one or more attributes used as part of the determining that the indicated building matches the one or more indicated criteria include an architectural style of the indicated building that is based at least in part on the indicated building.
Rawat, which is analogous art, discloses:
wherein the plurality of images include one or more images acquired external to the indicated building, and wherein the one or more attributes used as part of the determining that the indicated building matches the one or more indicated criteria include an architectural style of the indicated building that is based at least in part on the indicated building.
In various embodiments, use cases for the image tags that have been used to update attributes of a real estate database entry include: Visual Search or Similarity search (e.g., a search where the query is an image)—for example, a search is initiated using an image (e.g., an image taken by a user device); the system determines tags for the image or a visual representation(s) for similarity, the tags and/or the visual representation(s) are used to initiate a search, results returned based on matches to tags or attributes associated with properties in the database, images and/or their associated properties are provided to the initiator of the search using the image. In some embodiments, the user of the system can query the database to search for a property or an image using a photo (e.g., a photo captured by a device). For example, scenarios using a photo query include: scenario 1) consumer is visiting an open house, likes the kitchen, consumer uses an app on their mobile device to capture an image of the kitchen, which is then sent to a server for processing, the photo is analyzed and a result set of homes is returned with similar kitchens to the consumer; and scenario 2) same as scenario #1, however focused on external features of the house—for example a photo of the front of the home and we return result set of homes with similar architectural style.
Rawat is analogous to the claimed invention because both are directed to determining attributes of buildings based on images of the buildings. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to combine the system described in Segev with the attributes determined by the system of Rawat. Motivation to combine includes a more robust system that, based on images, can determine an architectural style. Thus, the resulting system has additional capabilities and can be utilized to perform the additional tasks without requiring another system to analyze exterior images of a building.
Claim 27 is rejected under 35 U.S.C. 103 as being obvious over Segev in view of Woh, et al., (“Does the Vision-Language Model Really Judge Like a Human Being?,” hereinafter “Woh”).
Claim 27
Segev does not appear to disclose:
wherein the generating of the textual description of the indicated building includes using one or more language models that are trained to use, as input, at least information about the determined plurality of attributes, wherein the one or more language models include at least one of a Vision and Language Model (VLM) that is trained using image/caption tuples, or a Knowledge Enhanced Natural Language Generation (VENLG) model that is trained using one or more defined knowledge sources, or a language model that uses a knowledge graph in which nodes represent entities and edges represent predicate relationships.
Woh, which is analogous art, discloses:
wherein the generating of the textual description of the indicated building includes using one or more language models that are trained to use, as input, at least information about the determined plurality of attributes, wherein the one or more language models include at least one of a Vision and Language Model (VLM) that is trained using image/caption tuples, or a Knowledge Enhanced Natural Language Generation (VENLG) model that is trained using one or more defined knowledge sources, or a language model that uses a knowledge graph in which nodes represent entities and edges represent predicate relationships.
Vision-Language task interprets the context of the given image and attempts to represent it in a human language. Image captioning…for example, analyzes context (relations between objects, situational background, etc.) to con struct a sentence that adequately explains the given image. Woh at pg. 1.
Woh is analogous art to the claimed invention because both are related to generating textual data from an image. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to utilize a VLM, as disclosed in Woh, to generate the attributes of a floor plan as a textual representation based on an image of the location. Motivation to combine includes increased accuracy of VLMs over other techniques, as well as decreased computation time.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Sfar, et al., U.S. Pat. Pub. No. 2019/0243928
Gallo, et al., U.S. Pat. No. 11,768,974
Wixon, et al., U.S. Pat. Pub. No. 2022/0269885
Communication
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSEPH MORRIS whose telephone number is (703)756-5735. The examiner can normally be reached M-F 8:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ryan Pitaro can be reached at (571) 272-4071. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JOSEPH MORRIS
Examiner
Art Unit 2188
/JOSEPH P MORRIS/Examiner, Art Unit 2188
/RYAN F PITARO/Supervisory Patent Examiner, Art Unit 2188
Prosecution Insights