DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are pending.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 and are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which applicant regards as the invention.
Claim 1 is directed to a “system comprising one or more storage devices”, it is unclear if applicant is claiming a system or storage devices because the claim is directed to both, placing claim 1 into two separate and distinct categories of patentable subject matter.
Claim 1 recites the limitation "the one or more data storage elements" in line(s) 7. There is insufficient antecedent basis for this limitation in the claim. There are no "data storage elements" anteceding this limitation in the claim.
As to claim(s) 8, 15, the same deficiency applies.
Claim 1 recites the limitation "the operational data" in line(s) 9. There is insufficient antecedent basis for this limitation in the claim. There is no "operational data" anteceding this limitation in the claim.
As to claim(s) 8, 15, the same deficiency applies.
Claim 8 recites the limitation "the knowledge graph" in line(s) 6. There is insufficient antecedent basis for this limitation in the claim. There is no "knowledge graph" anteceding this limitation in the claim.
As to claim(s) 15, the same deficiency applies.
Claim 12 recites the limitation "the one or more solutions" in line(s) 2. There is insufficient antecedent basis for this limitation in the claim. There are no "one or more solutions" anteceding this limitation in the claim.
Claim 16 recites the limitation "the one or more processors" in line(s) 4. There is insufficient antecedent basis for this limitation in the claim. There are no "one or more processors" anteceding this limitation in the claim.
Appropriate correction or clarification is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Examiner would like to point out that any reference to specific figures, pages, columns and lines should not be considered limiting in any way, the entire reference is considered to provide disclosure relating to the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ettinger et al., (Ettinger hereinafter), U.S. Patent 11216663, taken in view of Harvey et al., (Harvey hereinafter), U.S. Pre–Grant publication 20210383235. (see IDS).
As to claim 1, Ettinger discloses a building system (see “BIM is a digital representation of physical and functional characteristics of a facility, building, space, etc., which will necessarily incorporate information about objects present therein. BIM is associated with a shared knowledge resource for information about a facility forming a reliable basis for decisions during its lifecycle, defined as existing from earliest conception to demolition. BIM involves representing a design as combinations of “objects” - vague and undefined, generic, or product-specific, solid shapes or void-space oriented (like the shape of a cone or more), that carry their geometry, relations and attributes. BIM design tools allow extraction of different views from a building model for drawing production and other uses” in col. 27, line 57 to col. 28, line 3) comprising one or more storage devices storing instructions thereon that, when executed by one or more processors, cause the one or more processors (see “Stored in the memory 1609 are… components that are executable by the processor“ in col. 62, lines 48-51) to: receive an indication to (see “information about scene type, structure type, structure size, and location of the objects etc.… provided by the user“ in col. 25, line 66 to col. 26, line 5) execute a service; execute the service (see “system 1600 can be… computing devices” in col. 62, lines 31-22; “Stored in the memory 1609 are… components that are executable by the processor“ in col. 62, lines 48-51) based on data of a building (see “sensor data types can comprise… building information model (“BIM”) data” in col. 7, lines 35-41) to generate an inference or a prediction of a condition of a building (see “by leveraging available information about the user's behavior as indicated by the real time navigation and positioning of his scene camera relative to one or more objects, features, scene, or locations displayed to him as one or more viewports, likely user intent in relation to a user information goal or activity can be inferred, at least in part, to provide him with information that is more likely relevant to his user activity in context and any information goal associated therewith. For a better inference, this can be augmented by information about scene type, structure type, structure size, and location of the objects etc.… provided by the user“ in col. 25, line 58 to col. 26, line 5); store, by the service, the inference or the prediction, or a link to the inference or the prediction, in the one or more data storage elements of a knowledge (see “BIM is associated with a shared knowledge resource for information about a facility forming a reliable basis for decisions during its lifecycle, defined as existing from earliest conception to demolition. BIM involves representing a design as combinations of “objects” - vague and undefined, generic, or product-specific, solid shapes or void-space oriented (like the shape of a cone or more), that carry their geometry, relations and attributes” in col. 27, line 57 to col. 28, line 3) graph, the knowledge graph including representations of entities of the building, relationships between the entities of the building (see “Machine learning-based object identification, segmentation, and/or labeling algorithms can be used to identify the 2D/3D boundaries, geometry, type and health of objects, components, and scene or locations of interest so that it can be replaced by an object representing a physical asset or elements thereof with corresponding semantic data from an existing 3D model library of a subject object, feature, scene, or location of interest, such as, for example, by providing recognition of components and identification of aspects on or associated with a physical asset. In this regard, Deep Convolutional Neural Networks (DCNNs) can be used to assigning a label to one or more portions of an image (e.g., bounding box, region enclosed by a contour, or a set of pixels creating a regular or irregular shape) that include a given object, feature, scene, or location of interest, a collection of the physical assets of interest, or components or features on or relevant to the asset(s). An object mask can also indicate which portions of the image include the asset(s). A directed graph can be used to build the neural networks. Each unit can be represented by a node labeled according to its output and the units are interconnected by directed edges. Once the multiple bounded views or enclosed free-shape regions and directed graphs are built, a model can be used to assign labels based on resemblance and statistics” in col. 52, lines 39-60), and one or more storage elements storing or linking the operational data (see “stored in the memory 1609 are both data and several components that are executable by the processor” in col. 62, lines 48-49); query the knowledge graph to retrieve the inference or the prediction from the knowledge graph (see “continuously process the user's scene camera navigation and positioning substantially in real time to identify one or more 3D point(s), one or more 3D surfaces for an object (or components/features thereof), one or more combinations of connected 3D surfaces for an object (or components/features thereof), the whole of a 3D object, and/or one or more combinations of connected/nearby 3D objects. This is then used to infer the intent of the user in the context of his scene camera's positioning in relation to the object, feature, scene, or location of interest. To this end, any point or location at which the user places his scene camera relative to a displayed viewport will be mirrored on one or more dependent viewports” in col. 25, lines 24-36); update, responsive to the query, a graphical model of the building including graphical representations of the entities (see “BIM is a digital representation of physical and functional characteristics of a facility, building, space, etc., which will necessarily incorporate information about objects present therein. BIM is associated with a shared knowledge resource for information about a facility forming a reliable basis for decisions during its lifecycle, defined as existing from earliest conception to demolition. BIM involves representing a design as combinations of “objects” - vague and undefined, generic, or product-specific, solid shapes or void-space oriented (like the shape of a cone or more), that carry their geometry, relations and attributes. BIM design tools allow extraction of different views from a building model for drawing production and other uses” in col. 27, line 57 to col. 28, line 3) to include the inference or the prediction (see “for the purpose of detecting a damaged lead jack on a commercial roof structure… a model that is trained on DEM (Digital Elevation Model) imagery and could predict any obstructions on a roof surface with a particular shape. The prediction can then be augmented via a model that predicts/calculates height of roof obstructions from oblique imagery. The combination of all these steps, each with a different goal but complementary to each other, would maximizes the probability of a confident detection, identification, segmentation, and modeling while also minimizing the probability of missing the intended object in the dataset” in col. 53, lines 47-64); and cause a display device of a user device to display the updated graphical model (see “synchronized data display concept… overlaying or superimposing the representation of… data types concurrently with the representation of… other data types while preserving the real-time seamless interaction functionalities which allow interacting with a specific data type even in the context of another data type space” in col. 20, lines 5-11).
While Ettinger discloses execute the service as “system 1600 can be… computing devices” (see col. 62, lines 31-22) or “Stored in the memory 1609 are… components that are executable by the processor“ (see col. 62, lines 48-51), Ettinger fails to expressly disclose execute the service.
Harvey discloses execute the service. (See “[0059]… a neural network that more closely mimics its digital twin building… [0061]… memory 120 stores software 185 implementing the described methods of heterogenous neural network creation and implementation”).
Ettinger and Harvey are analogous art because they are related to digital twins.
Therefore, it would have been obvious to one of ordinary skill in this art before the effective filing date of the claimed invention to use Harvey with Ettinger, because Harvey discloses that his "[0059]… neural networks… have potentially different activation functions that may be equations that model portions of physical systems. The neural network may have more than one type of input. When a neural network is optimized, it may be optimized for less than all the inputs… When the neural network is modeling a physical structure inputs of type one may be of temporary values in the structure, such as temperature and humidity. Inputs of type two may be of permanent values of the structure, such as layer mass and heat transfer rates. Running the neural net optimizing the inputs of type two may optimize the amount of energy that the building might use, while optimizing the inputs of type two may optimize the characteristics of the structure itself, giving a neural network that more closely mimics its digital twin building".
As to claim 2, Ettinger discloses wherein the instructions cause the one or more processors to: receive an input through the graphical model (see “infer user intent from the user's navigation and positioning of the scene camera in and around each of the viewport including information about the object, feature, scene, or location of interest… incorporate an activity or task-oriented processing step applied to the sensor data from which the one or more visualizations is generated, provided, or rendered, so as to enhance visual context for the user as he navigates in and around the display” in col. 16, lines 5-14); and ingest the input into the knowledge graph, wherein the input is to execute the service (see “for the purpose of detecting a damaged lead jack on a commercial roof structure… a model that is trained on DEM (Digital Elevation Model) imagery and could predict any obstructions on a roof surface with a particular shape. The prediction can then be augmented via a model that predicts/calculates height of roof obstructions from oblique imagery. The combination of all these steps, each with a different goal but complementary to each other, would maximizes the probability of a confident detection, identification, segmentation, and modeling while also minimizing the probability of missing the intended object in the dataset” in col. 53, lines 47-64).
As to claim 3, Ettinger discloses wherein the instructions cause the one or more processors to link the inference or the prediction, or the link to the inference or the prediction to at least a portion of the graphical model of the building (see “portion of the graphical model of the building“ as “3D rendering of a roof structure“, “user's scene camera navigation and positioning may indicate that the user is looking at a 3D rendering of a roof structure from a far-range viewpoint… user activity can be inferred as obtaining of an overall understanding of the building geometry, a rough inventory of the objects on the roof, and/or the relationship among those objects” in col. 31, lines 4-30).
As to claim 4, Ettinger discloses wherein the prediction or the inference comprises one or more solutions to an issue that affects the building; and wherein the instructions cause the one or more processors to cause the display device of the user device to display the one or more solutions and a result of implementing the one or more solutions (see “issue that affects the building“ as “observational aspect likely to be associated with the inferred user intent… information goal as determinable from the user's positioning of the scene camera“, “user's scene camera navigation and positioning may indicate that the user is looking at a 3D rendering of a roof structure from a far-range viewpoint… user activity can be inferred as obtaining of an overall understanding of the building geometry, a rough inventory of the objects on the roof, and/or the relationship among those objects… user intent is derivable at least from the user's real time navigation and positioning of his scene camera… recommend for concurrent display with the 3D rendering base viewport and one or more 2D images as dependent viewports that optimally represent that observational aspect likely to be associated with the inferred user intent, as well as the associated user activity and an appropriate information goal as determinable from the user's positioning of the scene camera relative to the object(s), feature(s), scenes(s), or location(s) in the overall scene vis-à-vis the 3D rendering on the user's display” in col. 31, lines 4-30).
As to claim 5, Ettinger discloses wherein the instructions cause the one or more processors to implement the one or more solutions in response to a receipt of an entitlement request that verifies the one or more solutions (see ‘“intelligent and interactive recommendation engine” which could recommend a sorted list of options from an at least one additional sensor dataset based on the user interaction history with a base sensor dataset such that the best outcome could be achieved for a user task event. The “best outcome” could refer to the solution with the maximum confidence, minimum error (in terms of distance, linear measurement, surface measurement, volume measurement, geo-localization, angle, orientation, direction, location, coordinates of 2D/3D point, line, polygon, or a volumetric shape), highest accuracy and recall in predictions, minimum number of user input data, minimum occlusion, or optimum geometry, topology, and semantics’ in col. 39, line 56 to col. 40, line 2).
As to claim 6, Ettinger discloses wherein the service is a first service, and the instructions cause the one or more processors to: feed the prediction or the inference into a second service; and execute the second service to implement the one or more solutions (see “Once identified via user observation on his display and, optionally, image analytics and machine learning, derived information for the object, feature, scene, or location of interest could then allow the user to obtain, via an up-close inspection process, an identification of the presence or absence of damage, such as corrosion, rust, erosion, a short hazard, drainage issues, discoloration, deformation, and the like. As the user navigates and positions his scene camera in and around one or more displayed viewports, for example a 3D point cloud of the subject tower as the base viewport, the system will select and, if necessary first perform processing on, one or more 2D images that are determined to best match the orientation, perspective, or viewpoint that is the inferred user intent, where the inferred user intent is inferred from the user's navigation and positioning of the scene camera in and around the 3D point cloud” in col. 26, lines 41-56).
As to claim 7, Ettinger discloses wherein the data of the building (see “sensor data types can comprise… BIM”) data” in col. 7, lines 35-41) includes timeseries data (see “historical performance information indicating that a type of mechanical equipment often present on a commercial roof is often subject to failure can be incorporated in a machine learning library that is used to enhance the user navigation around and evaluation of objects, components, and features of interest, such as by populating a directed user workflow and/or highlighting, annotating or otherwise marking objects, features scene, or locations” in col. 57, lines 11-19); and wherein the instructions cause the one or more processors to update the timeseries data based at least in part on the inference or the prediction (see “existing database information relevant to the known presence of objects, components, or features on the roof can enrich and inform user activity during navigation and viewport of displayed 2D and/or 3D information during a user activity and, as such, can further enhance the content and quality of information provided associated therewith” in col. 57, lines 25-30).
As to claims 8-20, these claims recite a method performed by and a building system comprising processing circuits like the building system comprising storage devices of claims 1-7. Ettinger discloses “system 1600 can be… computing devices” (see col. 62, lines 31-22) performed by the apparatus system that teaches claims 1-7. Therefore, claims 8-20 are rejected for the same reasons given above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JUAN CARLOS OCHOA whose telephone number is (571)272-2625. The examiner can normally be reached Mondays, Tuesdays, Thursdays, and Fridays 9:30AM - 8:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Renee Chavez can be reached at 571-270-1104. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JUAN C OCHOA/Primary Examiner, Art Unit 2186
Prosecution Insights