DETAILED ACTION
This action is in response to the amendment filed on Feb. 17th, 2026. A summary of this action:
Claims 1-4, 6, 8-11, 13-16, 18-23 have been presented for examination.
Claims 1, 8, 13, 23 are objected to because of informalities
Claims 1-4, 6, 8-11, 13-16, 18-23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of both a mathematical concept and mental process without significantly more.
No § 102/103 rejection. See the Final Act. Feb. 2025, at page 8 for the rationale on why the prior rejection was withdrawn, and that rationale is incorporated herein by reference.
This action is Final
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments/Amendments
Regarding the § 112(a) Rejection
Withdrawn in view of amendment. Newly amended subject matter is objected to below, with a suggested amendment to more precisely reflect what is expressly disclosed.
Regarding the § 101 Rejection
Maintained, updated as necessitated by amendment.
With respect to the prong 1 remarks, see instant fig. 2. See MPEP § 2106.04(a)(2)(III), for the exact phrasing is: “Synopsys, Inc. v. Mentor Graphics Corp., 839 F.3d 1138, 1139, 120 USPQ2d 1473, 1474 (Fed. Cir. 2016) (holding that claims to a mental process of "translating a functional description of a logic circuit into a hardware component description of the logic circuit" are directed to an abstract idea, because the claims "read on an individual performing the claimed steps mentally or with pencil and paper").
To clarify, see the opinion of Synopsys for the discussion of the four lines of codes. Also, the quote to Synopsys is taken out of context – full quote: “Synopsys' reliance on TQP Development, LLC v. Intuit Inc., No. 2:12-cv-180-WCB, [2014 BL 43742], 2014 U.S. Dist. LEXIS 20077 , [2014 BL 43742], 2014 WL 651935 (E.D. Tex. Feb. 19, 2014), is therefore misplaced.
See Appellant's Opening Br. 39 n.8. In that case, the district court denied the defendant's motion for summary judgment that claims for a specific data encryption method for computer communication were invalid under § 101. TQP, [2014 BL 43742], 2014 U.S. Dist. LEXIS 20077 , [2014 BL 43742], 2014 WL 651935 , at *1. It distinguished the claims at issue from the mental processes found unpatentable in cases like Gottschalk. It explained that unlike those "simple," "basic" processes, the plaintiff's "invention involves a several-step manipulation of data that, except in its most simplistic form, could not conceivably be performed in the human mind or with pencil and paper." [2014 BL 43742], 2014 U.S. Dist. LEXIS 20077 , [WL] at *4 (emphasis added). This case is different”
See the Fed. Register Notice, July 2024, for PersonalWeb as well, as was cited to in the action: “Claims to “the use of an algorithm-generated content-based identifier to perform the claimed data-management functions,” which include limitations to “controlling access to data items,” “retrieving and delivering copies of data items,” and “marking copies of data items for deletion,” where the claims cover “a medley of mental processes that, taken together, amount only to a multistep mental process,” such that the steps can be practically performed in the human mind, PersonalWeb Techs. LLC v. Google LLC, 8 F.4th 1310, 1316-18 (Fed. Cir. 2021).”
To clarify, MPEP § 2106.04(a)(2)(III)(A) states: “a claim to a specific data encryption method for computer communication involving a several-step manipulation of data, Synopsys., 839 F.3d at 1148, 120 USPQ2d at 1481 (distinguishing the claims in TQP Development, LLC v. Intuit Inc., 2014 WL 651935 (E.D. Tex. 2014)); and” – what is claimed is not a particular data encryption method.
With respect to fig. 2, this shows directly the simplicity of the graphs to be generated in the exemplary form. These remarks allege complexity of data, but the specification provides a simple set of graphs.
To clarify, see Synopsys, Inc. v. Mentor Graphics Corp., 839 F.3d 1138, 1139, 120 USPQ2d 1473, 1474 (Fed. Cir. 2016): “Synopsys disputes the district court's characterization of the claims as mental processes. It suggests that the "complexity" of the claimed methods would make it implausible—if not impossible—for a skilled logic circuit designer to perform the methods mentally or with pencil and paper. Appellant's Opening Br. 21. It distinguishes these supposedly "complex" claims from the "simple" concepts found unpatentable in cases like Alice and Bilski 11. Appellant's Opening Br. 39. But, Synopsys' argument is belied by the actual claims at issue.…As demonstrated above, supra at 8-11, and in the patent specification itself, '841 patent , 21:45-22:23, the method can be performed mentally or with pencil and paper. The skilled artisan must simply analyze a four-line snippet of HDL code:…” – the fact pattern in these remarks is akin to that Synopsys alleged, and were “belied by the actual claims at issue” as demonstrated “in the patent specification itself”.
With respect to prong 2 remarks, these claims don’t improve graph database technology, but rather only are alleged to improve the data in the database. MPEP § 2106.05(a)(I): “Examples that the courts have indicated may not be sufficient to show an improvement in computer-functionality:… vii. Providing historical usage information to users while they are inputting data, in order to improve the quality and organization of information added to a database, because "an improvement to the information stored by a database is not equivalent to an improvement in the database’s functionality," BSG Tech LLC v. Buyseasons, Inc., 899 F.3d 1281, 1287-88, 127 USPQ2d 1688, 1693-94 (Fed. Cir. 2018); and”
To further clarify, these remarks don’t even point to any evidence in the decision to allege the improvement, but per MPEP § 2106.05(a): “That is, the disclosure must provide sufficient details such that one of ordinary skill in the art would recognize the claimed invention as providing an improvement. The specification need not explicitly set forth the improvement, but it must describe the invention such that the improvement would be apparent to one of ordinary skill in the art.” – see ¶¶ 76, 31, and 33, i.e. these are generically describe graph databases simply used to store information. Also, ¶ 33: “though other types of databases, such as SQL databases, may be used in various implementations”
Claim Objections
Claims 1, 8, 13, 23 are objected to because of the following informalities:
Independent claims recite, in part: claim 1 and 13: “and simulate real life characteristics of the objects in the physical environment;”, claim 8: “simulate real life characteristics of the objects in the physical environment and consequences of the interactions;” – the Examiner notes that these are interpreted in view of the specification, as cited below, and suggests amending this to more clearly reflect what is disclosed for precision of language, i.e. “and simulate real-world characteristics of the objects…”.
¶ 6: “generating a digital twin of the physical environment using the semantic model and a model library including models corresponding to the components, where the models corresponding to the components include information allowing the digital twin to reflect real-life characteristics of the components”
environment” and ¶ 72: “Sources of representative data 604 may include information about, or accurately reflect, the relative size and position [example of characteristics] of the included components (e.g. a street map), or the sources may be more abstract, only showing the relationships between components without regard to where the components are in the real world or where they are shown on the diagram (e.g. a circuit diagram).” And ¶ 18: “In this manner, digital twins can be created that are realistic and reflect the real-world conditions of the physical site and equipment”
Claim 23 is objected to for “processing element” because the term “element” could be considered a nonce term for § 112(f), and thus to avoid ambiguity in the claims the Examiner suggests that the claims are amended to expressly recite “processor” instead (e.g. see claim 1), and interprets it in this manner.
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-4, 6, 8-11, 13-16, 18-23 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea of both a mathematical concept and mental process without significantly more.
Step 1
Claim 1 is directed towards the statutory category of a process.
Claim 8 is directed towards the statutory category of an apparatus.
Claim 13 is directed towards the statutory category of an article of manufacture.
Claim 23 is directed towards a process.
Claims 13, and the dependents thereof, are rejected under a similar rationale as representative claim 1, and the dependents thereof.
Step 2A – Prong 1
The claims recite an abstract idea of both a mental process and mathematical concept.
See MPEP § 2106.04: “...In other claims, multiple abstract ideas, which may fall in the same or different groupings, or multiple laws of nature may be recited. In these cases, examiners should not parse the claim. For example, in a claim that includes a series of steps that recite mental steps as well as a mathematical calculation, an examiner should identify the claim as reciting both a mental process and a mathematical concept for Step 2A Prong One to make the analysis clear on the record.”
To clarify, see the USPTO 101 training examples, available at https://www.uspto.gov/patents/laws/examination-policy/subject-matter-eligibility.
Math concept is (claims 1, 8, 13, and 23; claim 1 as representative):
and modifying, via the processor, the vertex in the third graph database by performing an optimizing operation to modify a configuration of the vertex in the third graph database based on a probabilistic constraint of the vertex in the third graph database
¶ 32: “In some implementations, the constraints may be represented as probability distributions. For example, for a pipe angle radius attribute, the relational model 112 may include that P(45°) = 0.05, P(90°) = 0.94, and P(Other Angle)= 0.01, conveying that where a pipe changes direction, there is a 94% probability the angle radius is 90°, a 5% probability the angle radius is 45°, and a 1 % probability the angle radius is an angle besides 45° or 90°. Additionally, constraints may be represented as continuous probability distributions or encoded as mathematical functions. For example, a function may take position, rotation, and other relevant information about parts as input and may output a relative likelihood or probability. Other attributes in the relational model may include, for example, a probability that the diameter of core piping changes without a reducer or a set of likely orientations of the primary body axis and flow axis relative to the ground plane or infrastructure plane”
¶ 59: “To optimize the poses of the valves, information from the relational model 112 is used to constrain the pipes connecting the valves. For example, the relational model 112 shows a high probability that pipes connecting the valves will be either horizontal or vertical and will run in a straight line The poses of the valves and pipes may then be adjusted to maximize the probability distribution given the probabilities and constraints in the process model 110 and the physical model 108.”
Math calculations/relationships in textual form.
The mathematical concept recited in claim 8 is:
a relational model comprising probability distributions regarding attributes of at least some of the components in the physical environment and relationships between the components within the physical environment; - math relationships in textual form
The math concept recited in claim 23:
wherein the CAD library comprises mathematical representations of physical components within the physical environment and the mathematical representations are modified to match specific features of the physical components and construct three dimensional interactive components in the three-dimensional virtual model; - see ¶ 77: “The library 616 may be, like the CAD library 122, a parametric CAD library in which components (e.g., industrial components) and their variations are constructed from mathematical representations of their geometry based on simple paths and shapes augmented through a series of parametric procedural modifiers.” – and ¶ 35: “The CAD library 122 may be a custom, parametric CAD library in which specific components (e.g., industrial components) and their variations are constructed from mathematical representations of their geometry based on simple paths and shapes augmented through a series of parametric mathematically-defined modifiers.” And ¶ 49: “In some implementations, models within the CAD library 122 may be defined mathematically using shapes such as vectors and curves, and procedural modifiers such that, after the correct parameters are applied, the defined 3D shape can be replaced with a polygonal mesh at the appropriate scale which can then be added to the digital twin to allow interaction with the other components.” – i.e. math relationships in the mathematical field of geometry in textual form, i.e. this is the mathematical relationships in geometry describe the shape of objects, such as using the mathematical relationships (and/or equations) for a cylinder to represent a “pipe” (fig. 2) geometry, etc.; and the modification is merely math calculations in textual form to adjust parameters in such relationships/geometry (e.g. the diameter of the cylinder) to match the physical component, akin to setting the value of “r” and re-calculating it in the equation of a circle (MPEP § 2106.05(h): “iii. Limiting the use of the formula C = 2 (pi) r to determining the circumference of a wheel as opposed to other circular objects, because this limitation represents a mere token acquiescence to limiting the reach of the claim, Flook, 437 U.S. at 595, 198 USPQ at 199.
See MPEP § 2106.04(a)(2)(I)(A) which gives an example of such math relationships in geometry: “iii. a mathematical relationship between enhanced directional radio activity and antenna conductor arrangement (i.e., the length of the conductors with respect to the operating wave length and the angle between the conductors), Mackay Radio & Tel. Co. v. Radio Corp. of America, 306 U.S. 86, 91, 40 USPQ 199, 201 (1939) (while the litigated claims 15 and 16 of U.S. Patent No. 1,974,387 expressed this mathematical relationship using a formula that described the angle between the conductors, other claims in the patent (e.g., claim 1) expressed the mathematical relationship in words);”
Under the broadest reasonable interpretation, the claim recites a mathematical concept – the above limitations are steps in a mathematical concept such as mathematical relationships, mathematical formulas or equations, and mathematical calculations. If a claim, under its broadest reasonable interpretation, is directed towards a mathematical concept, then it falls within the Mathematical Concepts grouping of abstract ideas. In addition, as per MPEP § 2106.04(a)(2): “It is important to note that a mathematical concept need not be expressed in mathematical symbols, because "[w]ords used in a claim operating on data to solve a problem can serve the same purpose as a formula." In re Grams, 888 F.2d 835, 837 and n.1, 12 USPQ2d 1824, 1826 and n.1 (Fed. Cir. 1989). See, e.g., SAP America, Inc. v. InvestPic, LLC, 898 F.3d 1161, 1163, 127 USPQ2d 1597, 1599 (Fed. Cir. 2018)”
See MPEP § 2106.04(a)(2).
To clarify, see the USPTO 101 training examples, available at https://www.uspto.gov/patents/laws/examination-policy/subject-matter-eligibility.
As an initial matter, the Examiner notes figure 2 and its accompanying description, in particular # 110, 108, 112, and 118, which depict simple graphs readily mentally visualized (such as part of a mental evaluation/judgement/observation) or drawn on pen and paper. The present claims are directed to this abstract idea, which, at a high level of abstract, is the mental process of creating a series of simple graphs to represent a physical environment. This is akin, but much simpler, than the mental process a person such as a cartographer or surveyor may do to create a map of a region/area, but for the mere instructions to do it in a computer environment.
The mental process recited in claim 1 is:
generating, via a processor, a first graph database that represents the physical environment using image color data and depth data collected from the physical environment, the first graph database including spatial data about objects in the physical environment,, … traversing, via the processor, the image data and the depth data and iteratively identifying the objects in the physical environment on an object-by-object basis; for each object, generating, via the processor, a vertex in the first graph database that represents the physical model, each vertex representing an object in the physical model and a connection between objects in the physical model;and generating, via the processor, edges in the first graph database physical model, the edges representing relationships between objects; - a mental process, but done in a computer environment of mentally determined data which merely nests the mental process into a token post-solution activity to do it on a computer (see prong 2 below for clarification, i.e. generate these graphs, but do it with generic databases and a processor for the computer enviroment). See ¶ 42: “Additionally, in some implementations, a human-in-the-loop may be used to identify objects or verify the identity of objects generated by the system. For example, the system may present image data to a human, representing either part of or all of one or more photographs, depth images, rendered images of the collected spatial data, or other graphics representing real or abstract data. For example, this may be presented via a display associated with a user device accessible or viewable by the human user when the system is unable to identify a component based on RGB data. In this manner, the human user may provide input to” – i.e. a person is readily able, and contemplated, to be part of this, e.g. a person mentally observing images on a generic display of a computer and mentally observing objects in the images (e.g. observing a pipe in the image of a camera). Such a person, e.g. an industrial plant engineer, is readily able to observe all of the objects in a physical environment from an image, and then create a simple graph using pen and paper to record the results of their observation, including their mental evaluations of how the parts are connected to each other (e.g. observe and evaluate there is a valve, connected to a pipe, connected to another pipe, etc.)
generating, via the processor, a second graph database that represents the physical environment using a digitized diagram of the physical environment, the generating comprising: extracting, via the processor, from the digitized diagram, components in the physical environment and interconnections between the components reflecting connections between the components in the physical environment; generating, via the processor, a vertex in the second graph database that represents the process model, the vertex representing each component; and generating, via the processor, edges in the second graph database, the edges representing relationships between components; A mental process, but for the mere instructions to do it with a computer/in a computer environment. See the instant disclosure, ¶ 31 and ¶ 46, and figure 2 # 110. A person is readily able to mentally observe a diagram of the physical environment (e.g. ¶ 31) and for a simple diagram mentally evaluate the diagram so as to generate a process model of the environment, e.g. by using pen and paper to draw out a process model in the form of a simple graph (fig. 2, # 110).
The graph matching operation is a mental process, i.e. a person observing the graphs, e.g. on paper, and determining whether or not they math by visual inspection (observations + evaluations/judgements), but done in a generic computer enviroment. ¶ 46: “During the graph matching, the process model 110 may be used as ground truth where the vertices and edges of the process model 110 are used as a checklist for or to otherwise verify initial graph matching steps…” – e.g. simple mental observations in view of the instant graphs in the disclosure, or using physical aids, e.g. pen and paper, to make a checklist as part of this. ¶ 46: “In some implementations, these vertices may be transmitted to a human-in-the-loop for verification. For example, the image of the object may be sent to a computing device where a user may match the object to a component of the process model 110, provide additional information about the object, or request that the vertex be removed from the semantic model 118.”
generating, via the processor, a third graph database that represents the physical environment by; iterating, via the processor, through all vertices and edges in the first graph database and all vertices and edges in the second graph database; determining, via the processor, that a vertex in the second graph database matches a vertex in the first graph database by performing a graph matching operation based on the first graph database and second graph database generating, via the processor, a vertex in the third graph database that represents the vertex in the second graph database; and modifying, via the processor, the vertex in the third graph database by performing an optimizing operation to modify a configuration of the vertex in the third graph database based on a probabilistic constraint of the vertex in the third graph database - a continuation of the mental process, but for the mere instructions (incl. the “storing…”) to do it on a computer.
See the instant disclosure, fig. 2 # 110, # 108, and # 118 – a person would readily be able to mental observe such simple graphs (# 108 and # 110), and mentally evaluate them so as to generate a new graph to model the physical environment using pen and paper (# 118)
To clarify, when read in view of the disclosure (¶¶ 30-34; fig. 2; ¶ 59) this is a mental process that an engineer, e.g. a process control engineer at an industrial facility, is readily able to perform when mentally evaluating the facility (suppose a simple facility, or a portion of it, e.g. fig. 2 # 120 for the visual depiction), wherein they make a series of mental observations, evaluations, and judgements, so as to generated simple 2D graphs representing the components in the facility and how they are used in the process of the facility.
To further clarify, see fig. 4, and its accompanying description including in ¶ 52: “In some circumstances, particularly when modeling natural systems, there may be no process model available, and creation of the semantic model may rely on the information from the physical model, the relational model, and any potential humans-in-the-loop.”
To clarify on the modifying, ¶ 46: “Instead, the semantic model may be produced by traversing the process model concurrently with RGB-D data, normalized point clouds, other input data, and the relational model, and fully and probabilistically determining the most likely identity and pose of a component before moving on to the next component. This process may similarly make use of procedural algorithms, machine trained algorithms, or a human-in-the-loop.”
“¶ 32: “In some implementations, the constraints may be represented as probability distributions. For example, for a pipe angle radius attribute, the relational model 112 may include that P(45°) = 0.05, P(90°) = 0.94, and P(Other Angle)= 0.01, conveying that where a pipe changes direction, there is a 94% probability the angle radius is 90°, a 5% probability the angle radius is 45°, and a 1 % probability the angle radius is an angle besides 45° or 90”
A person is readily able to mentally evaluate a graph such as the ones depicted and do a mental trial-and-error process to optimize the graph such as by modifying vertexes in the graph (but, as claimed, its merely done in a computer environment), wherein a person is readily able to make mental decisions/evaluations based on probabilistic constraints, e.g. “94% probability the angle radius is 90°” – e.g. see fig. 2, “45 pipe” – a person, if they observe the odds, is readily able to optimize this graph by mentally judging it is highly unlikely that it’s a 45 degree pipe (5% chance), and much more likely to be a 90 degree pipe.
To clarify, people routinely make decisions constrained by probability, e.g. see blackjack, wherein a skilled player knows the odds (the probability) of various outcomes, and makes betting decisions on it (or they can simply ask the dealer for help, as most skilled dealers will provide them advice on what to do). Hence, casino games such as blackjack long pre-date the computer, and the odds have long been in favor of the house because people have mentally calculated the probabilities of various outcomes (a commonly taught mental process in most collegiate level introduction to probability course, usually using blackjack for the main example problem), and the rules of the game are usually set to slightly favor the house (the casino).
generating, via the processor, a three-dimensional virtual model of the physical environment using the third graph database and component models stored in memory to construct three dimensional operational virtual objects in the virtual model that represent the objects in the physical environment … - a mental process, but for the mere instructions to do it on a computer. See ¶ 21: “The semantic model may then be used to generate a digital twin by selecting and placing models (e.g., CAD models [note ¶ 49, i.e. “models…defined mathematically using shapes…”, e.g. simple equations representing shapes for squares/rectangles, circles, or in 3D cuboids and cylinders/spheres, and similar other such geometric primitive shapes; which are easily modelled mentally or with pen and paper used as an aid]) representing the various components in the environment in the digital twin of the environment.” See the July 2024 Fed. Register notice for “Claims to “the use of an algorithm-generated content-based identifier to perform the claimed data-management functions,” which include limitations to “controlling access to data items,” “retrieving and delivering copies of data items,” and “marking copies of data items for deletion,” where the claims cover “a medley of mental processes that, taken together, amount only to a multistep mental process,” such that the steps can be practically performed in the human mind, PersonalWeb Techs. LLC v. Google LLC, 8 F.4th 1310, 1316-18 (Fed. Cir. 2021).” – and see MPEP § 2106.05(f) for its discussion of Intellectual Ventures I v. Capital One Fin. Corp., 850 F.3d 1332, 121 USPQ2d 1940 (Fed. Cir. 2017).
To clarify, the specification conveys that this is merely the mental process of “retrieving and delivering copies of data items” based on the mental graphs (fig. 2), but done in a generic computer environment (e.g. in the “Unity” game engine; ¶ 51).
The mental process recited in claim 8 is:
one or more memories configured to store: a first graph database that represents a first model of the physical environment comprising spatial data about objects in the physical environment, the first graph database storing vertices representing the objects in the physical environment and the spatial data comprising image data and depth data registered to each other; - a similar mental process as the physical model of claim 1, but for nesting it into a limitation with mere instructions to do it on a computer (i.e. “store…” this data, rather than generate it). To clarify, generating this information is considered as the mental process, the act of storing it is merely instructions to do it on a computer and a token post-solution activity (see prong 2 below).
a second graph database that represents a second model of the physical environment storing vertices that represent components extracted from a digitized diagram of the physical environment and at least one edge that represent a relationship between two components; - a similar mental process as the process model of claim 1, but for nesting it into a limitation with mere instructions to do it on a computer (i.e. “store…” this data, rather than generate it). To clarify, generating this information is considered as the mental process, the act of storing it is merely instructions to do it on a computer and a token post-solution activity (see prong 2 below).
a relational model comprising probability distributions regarding attributes of at least some of the components in the physical environment and relationships between the components within the physical environment; - a mental process, but for nesting it into a limitation with mere instructions to do it on a computer (i.e. “store…” this data, rather than generate it). To clarify, generating this information (fig. 2, # 112) is considered as the mental process, the act of storing it is merely instructions to do it on a computer and a token post-solution activity (see prong 2 below).
To further clarify, see ¶ 59: “For example, a semantic model 118 may include three valves represented by vertices 142, 144, and 152, where each pair is connected by a pipe. The physical model 108 includes a pose estimation for each of the three valves, shown roughly by the angle of edges between the vertices. During graph matching to generate the semantic model, the types of valves, as well as the connections between the valves are constrained by the process model 110... The poses of the valves and pipes may then be adjusted to maximize the probability distribution given the probabilities and constraints in the process model 110 and the physical model 108. For example, the poses of the vertices 144, 142, and 146 are adjusted in the semantic model 118 such that pipe segments between the valves run either vertical or horizontal and match up to, for example, connections in the T-pipe represented by the vertex 146.” – a mental process of graph generation, performed such as a person observing two simple graphs such as the ones shown in figure 2 # 110 and # 108, and then mentally judging/evaluating how to combine these graphs into a new graph (# 188) with aid of pen and paper to draw the graph, wherein the probability information is used to perform corrections to angles of lines in the drawing (e.g. a person observing a table of 3 pipe angle probabilities such as discussed in ¶ 32, and using that table to adjust angles of lines in the graph).
To further clarify see ¶ 32: “The relational model 112 may include information about standard configurations and attributes of a typical process operation environment, which may be domain knowledge 106. Domain knowledge 106 may include information not included in the P&ID 104 but generally known to human operators, derived from statistics, or known principles in plant design. For example, the knowledge that most pipes run in a straight line either parallel or perpendicular to the ground plane may be included as domain knowledge 106. The relational model 112 generally encodes the domain knowledge 106 to specify constraints on component attributes and relationships. In some implementations, the constraints may be represented as probability distributions. For example, for a pipe angle radius attribute, the relational model 112 may include that P(45°) = 0.05, P(90°) = 0.94, and P(Other Angle)= 0.01, conveying that where a pipe changes direction, there is a 94% probability the angle radius is 90°, a 5% probability the angle radius is 45°, and a 1 % probability the angle radius is an angle besides 45° or 90°.” – the relational model, including the probability distributions, are merely part of the collection of information that is used in the mental process as discussed above.
For example, an engineer familiar with plant design would have recognized that most pipes run parallel or perpendicular to each other such as by mental observations of the environment and other similar environments. To clarify, the engineer would have been able to mentally observe piping systems in various physical environments, and use pen and paper, or another physical aid, to tabulate their mental observation, e.g. tabulating that out of 100 pipes that turn, 94 of the pipes turn perpendicular/at 90 degrees, 5 of the pipes turn at 45 degrees, and 1 pipe turns at 15 degrees. From this tabulation, they would have readily been able to perform simple calculations, such as with pen and paper, to mentally evaluate the probability of the angle of the pipes at a turn, e.g. 94/100 = 94 % probability of the pipe turning at 90 degrees/perpendicularly, 5/100 = 5% probability of pipes that turn at 45 degrees, etc. They may have also substituted in their own experience and provided a mental professional opinion, e.g. providing an opinion that the vast majority of pipes that turn is perpendicular/90 degree turns, such as providing an opinion that this is roughly 90 % of all pipes, etc. Alternatively, the person, when performing the mental process, would have been able to look this information up (mental observation) such as in a design book or a textbook, or the like.
generating a third graph database that represents a third model of the physical environment by repeatedly: determining a correspondence point between a vertex in the first graph database and a vertex in the second graph database; traversing, along a direction, from the correspondence point to a next vertex in the first graph database and a next vertex in the second graph database; and determining that the next vertex in the second graph database matches the next vertex in the first graph database by performing a graph matching operation based on the first graph database and second graph database, generating a vertex in the third graph database that represents the vertex in the second graph database; and modifying the vertex in the third graph database by performing an optimizing operation to modify a configuration of the vertex in the third graph database based on a probabilistic constraint of the vertex in the third graph database - a mental process, but for the mere instructions to do it on a computer, incl. the “storing…” of the mentally determined data. See instant fig. 7, as discussed in ¶¶ 79-86 – and see ¶ 71 for: “The spatial database 606 may be an implementation of the physical model 108, organizing and representing the measured data 602 of the physical environment.” – i.e. this is merely a mental process of graph matching (instant fig. 2; as discussed above), wherein a person is readily able to mentally observe matching vertices in the graph (fig. 2, # 110 and # 108, e.g. “GV-122” is the “Gate vale” # 132 by simple mental observation), observe portions of the physical model not in the process model by traversing (in the observation) both models, e.g. “pipe” # 136, judge to add them into the semantic model, e.g. “Pipe class A” #148, etc. In other words, this is a mental process of a series of mental observations, evaluations, and judgements, in graph matching two simple 2D graphs (fig. 2) readily able to be performed using pen and paper as an aid.
See above rationale to clarify on the determining and modifying, for similar limitations were discussed above for claim 1.
generating the three-dimensional virtual model of the physical environment using the third graph database that represents a fire, the probability distributions regarding the attributes of the at least some of the components in the relational model, and the component models, the generating comprising selecting and placing some or all of the component models in the three-dimensional virtual model … - rejected under a similar rationale as the similar limitation in claim 1 above.
The mental process in claim 23 is:
Analyzing, by a processing element, the image color data by a machine learned algorithm to detect and localize components within a two dimensional representation of the physical environment; - this is a mental process, akin to what was discussed above, but for the mere instructions to do it on a computer (incl. the “by a machine learned algorithm”) – see ¶ 42: “In other implementations, the combined RGB-D data, processed or raw, with or without additional sensor data types included, may be used to train machine learning algorithms to perform similar functions as those described above in association with the information collection operation 202. Additionally, in some implementations, a human-in-the-loop may be used to identify objects or verify the identity of objects generated by the system. For example, the system may present image data to a human, representing either part of or all of one or more photographs, depth images, rendered images of the collected spatial data, or other graphics representing real or abstract data. For example, this may be presented via a display associated with a user device accessible or viewable by the human user when the system is unable to identify a component based on RGB data”
analyzing, by the processing element, with a machine learned model the two- dimensional representation of the physical environment and the depth data to identify the components and estimate a pose of the components in the physical environment; - a continuation of the above discussed mental process (¶ 42), but for the mere instructions to do it on a computer – see ¶ 41 to further clarify, including a generic listing of “various computer vision algorithms” that may be used as part of using a computer as a tool to perform this step. To clarify on poses, ¶ 47: “poses (e.g., position, orientation, and possibly additional parameters)” – a person is readily able to observe in 2D images components/objects in the images, and poses of the objects, e.g. observing the orientation and positioning of a pipe relative to other components in images.
generating, by the processing element, a digital process and instrumentation (P&ID) file by extracting symbols and annotations from a P&ID diagram; - a mental process, but for the mere instructions to do it on a computer. An engineer’s mind is readily equipped to mentally observe and evaluate a P&ID diagram (these are akin to circuit schematics, but for piping and instrumentation in industrial facilities) and observe/evaluate symbols and annotations in them, e.g. by using pen and paper to create a tabular or graphical representation of these.
modeling, by the processing element, the digital P&ID file as a process model, wherein the process model comprises a graph database comprising a plurality of vertices and a plurality of edges spanning between the plurality of vertices, wherein each vertex of the plurality of the vertices represents a component within the physical environment and each edge of the plurality of edges represent a relationship between two components, wherein the process model further comprises unique identifiers for the plurality of vertices and the plurality of edges; - a mental process, but for the mere instructions to do it on a computer, akin to the ones founds in the claims above. This is a mental process of a person, e.g. an industrial plant or control engineer, mentally observing and evaluating P&ID diagrams such as by tabulating on paper the information found in them, or by using graph paper and pen/pencil to create a simple graph representing the P&ID diagram (e.g. fig. 2; as discussed above). See the above discussions of the process model for more clarification on this feature, in view of the disclosure, is readily a mental process, but for the mere instructions to do it on a computer.
optimizing, by the processing element, by graph matching between the graph database of the processing model and the spatial database of the physical model and using information in a relational model;; - a mental graph matching process, but for the mere instructions to do it on a computer, as was discussed above with respect to figures 2, 4, 7, and their accompanying descriptions.
While the claims, in view of the disclosure, describe a number of mental steps to be performed (and with mere instructions to do it on a computer), this abstract idea at its core is merely ““a medley of mental processes that, taken together, amount only to a multistep mental process,” such that the steps can be practically performed in the human mind, PersonalWeb Techs. LLC v. Google LLC, 8 F.4th 1310, 1316-18 (Fed. Cir. 2021).” As discussed in the July 2024 Fed. Register notice.
To further clarify on the graph matching, see the above rationale on the graph matching step when given its BRI consistent with the instant disclosure.
generating, by the processing element, a semantic model based on the spatial database and the graph database, wherein the semantic model comprises digital components representative of each component within the physical environment; - a mental process, but for the mere instructions to do it on a computer, similar to the ones discussed above (fig. 2 # 118 is the resulting semantic model) based on a graph matching # 110 and # 108, wherein a “Relational Model” (# 112; ¶¶ 32 and 59 as discussed above) is used which has “Domain knowledge 106 may include information not included in the P&ID 104 but generally known to human operators, derived from statistics, or known principles in plant design…”(¶ 32) – see the above discussion for more detail.
As a point of clarity regarding humans doing object detection using images and using a computer as a tool to perform the mental process, and to clarify on the BRI of RGBD (images with both color and depth data), see an example of this in Silberman, Nathan. Reasoning about Object Instances, Relations and Extents in RGBD Scenes. Diss. New York University, 2015. See the abstract, then pages 3-4 the paragraph split between the pages: “These alternative domains of visual reasoning have attracted increased attention concurrent with another trend in computer vision: combining RGB and depth (RGBD) inputs for visual processing… The depth image provided by a depth camera provides direct access to part of the scene's geometry and allows for the exploration of algorithms that are able to reason about the full room geometry, object's true physical extents, physical support relations and improved semantic and instance segmentation.” – e.g. see fig. 3.1 which visually depicts: “A typical indoor scene captured by the Microsoft Kinect. (a): Webcam image. (b) Raw depth map (red=close, blue=far). (c) Labels obtained via Amazon Mechanical Turk. (d) After a homography, followed by pre-processing to ll in missing regions, the depth map (hue channel) can be seen to closely aligned with the image (intensity channel) [example of registering these datasets to each other].” - with respect to the discussion of Amazon’s Mechanical Turk, this is a “human-in-the-loop” as discussed in the instant disclosure - see § 3.2.5: “The selected frames from each dataset were uploaded to Amazon Mechanical Turk and manually annotated using the LabelMe interface [?]. The annotators were instructed to provide polygonal labels for every object instance in the scene such that no object was left unlabeled (see Fig. 3.1(c)). Furthermore, each polygon label was named with both a semantic class and an instance (e.g. cup1, cup2, cup3).” And § 3.2.6: “Along with the semantic and instance annotations, we also provide support annotations for the NYU Depth V2 dataset. These annotations define a physical support relation type between two objects in the scene”
modifying, by the processing element, the semantic model by performing an optimizing operation to modify a configuration a vertex of the semantic model based on a probabilistic constraint of the vertex of the semantic model; -rejected under a similar rationale as discussed above.
and generating, by the processing element, a three-dimensional virtual model of the physical environment using the semantic model and a computer aided (CAD) library, wherein the CAD library comprises mathematical representations of physical components within the physical environment and the mathematical representations are modified to match specific features of the physical components and construct three dimensional interactive components in the three-dimensional virtual model; - rejected under a similar rationale as the similar limitation in claim 1 above. With respect to the modifications, given the lack of detail on how, this is readily a mental process, but to do it on a computer, e.g. represent all pipe objects with a math representation of a cylinder and its associated equations; and then modify variables for the cylinder so as to modify the cylinder to match each individual pipe (e.g. by setting height, radius/diameter, thickness of the walls of the cylinder, etc.).
Under the broadest reasonable interpretation, these limitations are process steps that cover mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of physical aids but for the recitation of a generic computer component. If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the "Mental Process" grouping of abstract ideas. A person would readily be able to perform this process either mentally or with the assistance of physical aids. See MPEP § 2106.04(a)(2).
To clarify, see the USPTO 101 training examples, available at https://www.uspto.gov/patents/laws/examination-policy/subject-matter-eligibility. In particular, with respect to the physical aids, see example # 45, analysis of claim 1 under step 2A prong 1, including: “Note that even if most humans would use a physical aid (e.g., pen and paper, a slide rule, or a calculator) to help them complete the recited calculation, the use of such physical aid does not negate the mental nature of this limitation.”; also see example # 49, analysis of claim 1, under step 2A prong 1: “Moreover, the recited mathematical calculation is simple enough that it can be practically performed in the human mind. Even if most humans would use a physical aid, like a pen and paper or a calculator, to make such calculations, the use of a physical aid would not negate the mental nature of this limitation.”
As such, the claims recite an abstract idea of both a mental process and mathematical concept.
Step 2A, prong 2
The claimed invention does not recite any additional elements that integrate the judicial exception into a practical application. Refer to MPEP §2106.04(d).
The following limitations are merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, as discussed in MPEP § 2106.05(f), including the “Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more”:
Claim 1 - A computer-implemented method of generating a model of a physical environment in a digital environment comprising: - and similar recitations in the other independent claims are rejected under a similar rationale
Claim 8 – the “storing…” is the “Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data)”, i.e. mentally determine such data, then store it, and the similar recitations in the other independent claims is rejected under a similar rationale
Claim 1 – “via the processor” and claim 23 “by the processing element”
Claim 23 – “by a machine learned algorithm…with a machine learned model…” – mere instructions to do it on a computer, given the generic nature of what this is and the non-limited listing of algorithm described with a high degree of generality in ¶¶ 41-42 as discussed above. Also see ¶ 26: “For example, a machine learning or machine trained algorithm may attempt to match sensor data about a physical environment to data obtained through schematics of the physical environment and, where the algorithm is unable to match or reconcile the data, a procedural algorithm or human-in-the-loop may provide additional context to generate the digital twin 109.”
The above claims have been amended to recite various databases for graphs. See ¶¶ 30-32, including: “However, it should be noted that various types of storage structures and encoded data may be used, such as, but not limited to, graph and other database structures, SQL databases and the like.” – given what is claimed and the disclosure, these databases amount to nothing more than part of the mere instructions to do it on the computer, given the generic nature of their description. See MPEP § 2106.05(f); also see MPEP § 2106.05(a) for: “vii. Providing historical usage information to users while they are inputting data, in order to improve the quality and organization of information added to a database, because "an improvement to the information stored by a database is not equivalent to an improvement in the database’s functionality," BSG Tech LLC v. Buyseasons, Inc., 899 F.3d 1281, 1287-88, 127 USPQ2d 1688, 1693-94 (Fed. Cir. 2018); and”.
Claims 1, 8, 13, and 23 (using claim 1 as representative): generating, via the processor, a three-dimensional virtual model of the physical environment using the third graph database and component models stored in memory to construct three dimensional operational virtual objects in the virtual model that represent the objects in the physical environment and simulate real life characteristics of the objects in the physical environment; - these are considered as both mere instructions to use a computer as a tool to implement an abstract idea and mere instructions to “apply it” as they “cover any solution to an identified problem with no restriction on how the result is accomplished and no description of the mechanism for accomplishing the result” as discussed in MPEP § 2106.05(f). To clarify, the claim places no restrictions on how the semantic model and model library are used to generate the digital twin, outside of simply retrieving and delivering copies of data items, and merely specifying what data is to be retrieved (see MPEP § 2106.05(h) for “Intellectual Ventures I LLC v. Erie Indem. Co., 850 F.3d 1315, 1328-29, 121 USPQ2d 1928, 1939 (Fed. Cir. 2017) (limiting use of abstract idea to use with XML tags”; and MPEP § 2106.05(f) as well: “Intellectual Ventures I v. Capital One Fin. Corp., 850 F.3d 1332, 121 USPQ2d 1940 (Fed. Cir. 2017), the steps in the claims described "the creation of a dynamic document based upon ‘management record types’ and ‘primary record types.’" 850 F.3d at 1339-40; 121 USPQ2d at 1945-46. The claims were found to be directed to the abstract idea of "collecting, displaying, and manipulating data." 850 F.3d at 1340; 121 USPQ2d at 1946. In addition to the abstract idea, the claims also recited the additional element of modifying the underlying XML document in response to modifications made in the dynamic document. 850 F.3d at 1342; 121 USPQ2d at 1947-48. Although the claims purported to modify the underlying XML document in response to modifications made in the dynamic document, nothing in the claims indicated what specific steps were undertaken other than merely using the abstract idea in the context of XML documents”).
To further clarify, ¶ 18 of the instant disclosure: “A digital twin provides an accurate computerized model of a physical environment, such as a chemical operation environment, oil refinery, other industrial environment, and/or natural environment (e.g., forest, national park, or the like). Digital twins may, for example, employ computer assisted design (CAD) models of components within these environments and may be presented in two dimensions or in three dimensions… “ and ¶ 21: “For example, the various information collected may be combined into a semantic model of the environment, where the semantic model includes information about the components of the environment and spatial relationships between various components in the environment. The semantic model may then be used to generate a digital twin by selecting and placing models (e.g., CAD models) representing the various components in the environment in the digital twin of the environment.” – i.e. this is mere instructions to apply a computer as a tool to implement a computer model of the environment in place of a mental model, such as the one discussed above with respect to claim 1 (e.g. a person making a series of mental observations and judgements so as to mentally visualize a model of the physical environment, such as in their own mind or with the aid of pen and paper drawings as discussed in detail above).
In other words, this limitation would be akin to a cartographer deciding to use a computer as a tool make digital maps of a region instead of paper maps, wherein the digital maps use a library/database of standard symbols to represent certain features on a map (e.g. a tree symbol to represent each park in the map). The recitation in the claims of “simulate real life conditions…” is "so result focused, so functional, as to effectively cover any solution to an identified problem" (MPEP § 2106.05(f)), e.g. the cartographer deciding to have the map mimic/reflect real-life conditions/functionality of roads on the map by marking on the digital map what roads are closed and which ones are open.
As to the simulating, this is merely using the commercially available commonplace software of the “Unity” game engine in its ordinary capacity. ¶¶ 36, 48, 51.
In addition, the machine learning recitations are also considered as generally linking to a particular technological environment wherein such algorithms are used in place of other algorithms, or a human, given what is disclosed (e.g. ¶¶ 26 and 42 for generic descriptions).
The following limitations are adding insignificant extra-solution activity to the judicial exception, as discussed in MPEP § 2106.05(g):
Claim 1 - generating, via a processor, a first graph database that represents the physical environment using image color data and depth data collected from the physical environment, the first graph database including spatial data about objects in the physical environment, wherein the image color data and the depth data are registered to each other and the generating of the first graph database comprises: traversing, via the processor, the image data and the depth data and iteratively identifying the objects in the physical environment on an object-by-object basis; for each object, generating, via the processor, a vertex in the first graph database that represents the physical model, each vertex representing an object in the physical model and a connection between objects in the physical model; and generating, via the processor, edges in the first graph database physical model, the edges representing relationships between objects;- the registering of the image color data and depth data to each other are mere data gathering and insignificant activities involved in the mere data gathering. The remaining portions of this, should it be found not to be part of the abstract idea for the reasons discussed above (¶ 42), would also be considered as mere data gathering with insignificant activities involved in the data gathering.
The similar recitations in the other independent claims are rejected under a similar rationale
Claim 8 – the “storing…” is a token post-solution activity, i.e. “adding a final step of storing data to a process that only recites computing the area of a space (a mathematical relationship) does not add a meaningful limitation to the process of computing the area.” As discussed in MPEP § 2106.05(g) , and the similar recitations in the other independent claims is rejected under a similar rationale. A similar rationale is also used for the graph databases, in view of ¶¶ 30-33, which merely convey the generic databases are to store the models.
Claim 1- generating, via the processor, a three-dimensional virtual model of the physical environment using the third graph database and component models stored in memory to construct three dimensional operational virtual objects in the virtual model that represent the objects in the physical environment and simulate real life characteristics of the objects in the physical environment; - the simulation is an example of token post-solution activities that are nominally/tangentially linked to the primary process of the claimed invention as the “simulate…” is a token post solution activity (¶ 51) recited and described in a high degree of generality without any particular details on how it is to perform this simulation of real life conditions (¶ 6) outside of simply invoking the commonplace commercially available software of the Unity game engine (see citations to disclosure above). Should the generation of the virtual model be found not to be part of the abstract idea then this would be another token post-solution activity akin to mere data displaying (fig. 2, # 120-122; ¶ 36). The similar recitations in the other independent claims is rejected under a similar rationale
To further clarify, ¶ 21 of the instant disclosure: “The semantic model may then be used to generate a digital twin by selecting and placing models (e.g., CAD models) representing the various components in the environment in the digital twin of the environment” - this is an insignificant computer implementation of generating a computerized model of the physical environment in place of a mental model/pen and paper model, wherein the described technique to generate the digital twin would readily be performable by a person using a computer as a tool to implement the computerized model (e.g. the person mentally observing the semantic model # 118 in fig. 2, and then dragging-and-dropping CAD models to generate a 3D CAD model to reflect the semantic model, wherein the 3D CAD model reflects a mental observation, e.g. a mental visualization, of the physical environment).
Claim 1- and transmitting via the processor the virtual model to a computing device to enable digital interaction with the three-dimensional virtual model.. – mere data transmission, with a functional recitation (MPEP § 2173.05(g)) of “to enable…” without even requiring the interaction to be performed (MPEP § 2111.04(I)), wherein this is merely describing the capability of a process step positively recited (the “transmitting…”) which is mere data transmission. Should the interaction be given patentable weight in that it is required, then it would be mere data gathering recited in a high degree of generality. The similar recitations in the other independent claims is rejected under a similar rationale
Claim 23 - registering, by a processing element, image color data captured by a camera with depth data captured by a depth scan of a physical environment; analyzing the image color data by a machine learned algorithm to detect and localize components within a two dimensional representation of the physical environment; analyzing, by a processing element, with a machine learned model the two-dimensional representation of the physical environment and the depth data to identify the components and estimate a pose of the components in the physical environment; - rejected under a similar rationale as the similar limitations found in the other independent claims (i.e. mere data gathering with insignificant activities involved in the data gathering).
generating, by the processing element, a spatial database forming a first graph database of the physical environment, the spatial database including the components of the physical environment; - mere data gathering/storing, see ¶ 69: “For example, measured data 602 may be utilized to create a spatial database 606 or other physical model and representative data 604 may be utilized to create a process model 608” – also see ¶ 71, i.e. this is merely creating a collection of the measured data into a database (i.e. gather, then store information in a database).
and generating, by the processing element, a three-dimensional virtual model of the physical environment using the semantic model and a computer aided (CAD) library, wherein the CAD library comprises mathematical representations of physical components within the physical environment and the mathematical representations are modified to match specific features of the physical components and construct three dimensional interactive components in the three-dimensional virtual model; - rejected under a similar rationale as the generation of the virtual model step above.
and outputting , by a processing element, the three-dimensional virtual model to a display to allow a user to navigate through and interact with the three-dimensional interactive components in the three-dimensional virtual model.. – mere data outputting, rejected under a similar rationale as the transmitting step above
Of particular note for the above analysis, given the particularity recited in the data gathering, the Examiner notes example 45, dependent claim 3; and Thales Visionix, Inc. v. United States, 850 F.3d 1343, 1348-49, 121 USPQ2d 1898, 1902 (Fed. Cir. 2017) as discussed in MPEP § 2106.05(a) – i.e. in the Examiner’s view, the question of whether or not the data gathering steps render this claimed invention as being eligible subject matter is a step 2B consideration at the WURC consideration (see below). To clarify, in example 45, claim 3 see its discussion of the thermocouple, wherein the improvement was only discussed at step 2B for mere data gathering.
With respect to claim 23, “… the component models comprising at least one of parametric models of the objects, standardized models of the objects, combinations of parametric models and standardized models, or polygonal meshes; and” – this is considered as part of the mere instructions to do it on a computer, as well as generally linking to a particular technological environment.
To clarify, see ¶ 35, and ¶ 49. Neither the claims nor the disclosure provides any particular disclosure on what these models are, or how they are to be particularly manipulated (e.g. particular methods of manipulating their particular data structures; see Research Corp. in MPEP 2106.04(a)(2)(III)(A)). Rather, this is akin to “Although the claims purported to modify the underlying XML document in response to modifications made in the dynamic document, nothing in the claims indicated what specific steps were undertaken other than merely using the abstract idea in the context of XML document” in Intellectual Ventures I v. Capital One Fin. Corp., 850 F.3d 1332, 121 USPQ2d 1940 (Fed. Cir. 2017) in MPEP § 2106.05(f).
A claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception. See MPEP § 2106.04(d).
E.g. MPEP § 2106(I): “Mayo, 566 U.S. at 80, 84, 101 USPQ2dat 1969, 1971 (noting that the Court in Diamond v. Diehr found “the overall process patent eligible because of the way the additional steps of the process integrated the equation into the process as a whole,”” – and see MPEP § 2106.05(e). To clarify, unlike Diamond v. Diehr there is no particular control action (example 45, dependent claims 2 and 4) in the present claims, in addition to the additional steps, that integration the abstract idea into a practical application.
The claimed invention does not recite any additional elements that integrate the judicial exception into a practical application. Refer to MPEP §2106.04(d).
Step 2B
The claimed invention does not recite any additional elements/limitations that amount to significantly more.
The following limitations are merely reciting the words "apply it" (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, as discussed in MPEP § 2106.05(f), including the “Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more”:
Claim 1 - A computer-implemented method of generating a model of a physical environment in a digital environment comprising: - and similar recitations in the other independent claims are rejected under a similar rationale
Claim 8 – the “storing…” is the “Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data)”, i.e. mentally determine such data, then store it, and the similar recitations in the other independent claims is rejected under a similar rationale
Claim 1 – “via the processor” and claim 23 “by the processing element”
Claim 23 – “by a machine learned algorithm…with a machine learned model…” – mere instructions to do it on a computer, given the generic nature of what this is and the non-limited listing of algorithm described with a high degree of generality in ¶¶ 41-42 as discussed above. Also see ¶ 26: “For example, a machine learning or machine trained algorithm may attempt to match sensor data about a physical environment to data obtained through schematics of the physical environment and, where the algorithm is unable to match or reconcile the data, a procedural algorithm or human-in-the-loop may provide additional context to generate the digital twin 109.”
The above claims have been amended to recite various databases for graphs. See ¶¶ 30-32, including: “However, it should be noted that various types of storage structures and encoded data may be used, such as, but not limited to, graph and other database structures, SQL databases and the like.” – given what is claimed and the disclosure, these databases amount to nothing more than part of the mere instructions to do it on the computer, given the generic nature of their description. See MPEP § 2106.05(f); also see MPEP § 2106.05(a) for: “vii. Providing historical usage information to users while they are inputting data, in order to improve the quality and organization of information added to a database, because "an improvement to the information stored by a database is not equivalent to an improvement in the database’s functionality," BSG Tech LLC v. Buyseasons, Inc., 899 F.3d 1281, 1287-88, 127 USPQ2d 1688, 1693-94 (Fed. Cir. 2018); and”.
Claims 1, 8, 13, and 23 (using claim 1 as representative): generating, via the processor, a three-dimensional virtual model of the physical environment using the third graph database and component models stored in memory to construct three dimensional operational virtual objects in the virtual model that represent the objects in the physical environment and simulate real life characteristics of the objects in the physical environment; - these are considered as both mere instructions to use a computer as a tool to implement an abstract idea and mere instructions to “apply it” as they “cover any solution to an identified problem with no restriction on how the result is accomplished and no description of the mechanism for accomplishing the result” as discussed in MPEP § 2106.05(f). To clarify, the claim places no restrictions on how the semantic model and model library are used to generate the digital twin, outside of simply retrieving and delivering copies of data items, and merely specifying what data is to be retrieved (see MPEP § 2106.05(h) for “Intellectual Ventures I LLC v. Erie Indem. Co., 850 F.3d 1315, 1328-29, 121 USPQ2d 1928, 1939 (Fed. Cir. 2017) (limiting use of abstract idea to use with XML tags”; and MPEP § 2106.05(f) as well: “Intellectual Ventures I v. Capital One Fin. Corp., 850 F.3d 1332, 121 USPQ2d 1940 (Fed. Cir. 2017), the steps in the claims described "the creation of a dynamic document based upon ‘management record types’ and ‘primary record types.’" 850 F.3d at 1339-40; 121 USPQ2d at 1945-46. The claims were found to be directed to the abstract idea of "collecting, displaying, and manipulating data." 850 F.3d at 1340; 121 USPQ2d at 1946. In addition to the abstract idea, the claims also recited the additional element of modifying the underlying XML document in response to modifications made in the dynamic document. 850 F.3d at 1342; 121 USPQ2d at 1947-48. Although the claims purported to modify the underlying XML document in response to modifications made in the dynamic document, nothing in the claims indicated what specific steps were undertaken other than merely using the abstract idea in the context of XML documents”).
To further clarify, ¶ 18 of the instant disclosure: “A digital twin provides an accurate computerized model of a physical environment, such as a chemical operation environment, oil refinery, other industrial environment, and/or natural environment (e.g., forest, national park, or the like). Digital twins may, for example, employ computer assisted design (CAD) models of components within these environments and may be presented in two dimensions or in three dimensions… “ and ¶ 21: “For example, the various information collected may be combined into a semantic model of the environment, where the semantic model includes information about the components of the environment and spatial relationships between various components in the environment. The semantic model may then be used to generate a digital twin by selecting and placing models (e.g., CAD models) representing the various components in the environment in the digital twin of the environment.” – i.e. this is mere instructions to apply a computer as a tool to implement a computer model of the environment in place of a mental model, such as the one discussed above with respect to claim 1 (e.g. a person making a series of mental observations and judgements so as to mentally visualize a model of the physical environment, such as in their own mind or with the aid of pen and paper drawings as discussed in detail above).
In other words, this limitation would be akin to a cartographer deciding to use a computer as a tool make digital maps of a region instead of paper maps, wherein the digital maps use a library/database of standard symbols to represent certain features on a map (e.g. a tree symbol to represent each park in the map). The recitation in the claims of “simulate real life conditions…” is "so result focused, so functional, as to effectively cover any solution to an identified problem" (MPEP § 2106.05(f)), e.g. the cartographer deciding to have the map mimic/reflect real-life conditions/functionality of roads on the map by marking on the digital map what roads are closed and which ones are open.
As to the simulating, this is merely using the commercially available commonplace software of the “Unity” game engine in its ordinary capacity. ¶¶ 36, 48, 51.
In addition, the machine learning recitations are also considered as generally linking to a particular technological environment wherein such algorithms are used in place of other algorithms, or a human, given what is disclosed (e.g. ¶¶ 26 and 42 for generic descriptions).
The following limitations are adding insignificant extra-solution activity to the judicial exception, as discussed in MPEP § 2106.05(g):
Claim 1 - generating, via a processor, a first graph database that represents the physical environment using image color data and depth data collected from the physical environment, the first graph database including spatial data about objects in the physical environment, wherein the image color data and the depth data are registered to each other and the generating of the first graph database comprises: traversing, via the processor, the image data and the depth data and iteratively identifying the objects in the physical environment on an object-by-object basis; for each object, generating, via the processor, a vertex in the first graph database that represents the physical model, each vertex representing an object in the physical model and a connection between objects in the physical model; and generating, via the processor, edges in the first graph database physical model, the edges representing relationships between objects;- the registering of the image color data and depth data to each other are mere data gathering and insignificant activities involved in the mere data gathering. The remaining portions of this, should it be found not to be part of the abstract idea for the reasons discussed above (¶ 42), would also be considered as mere data gathering with insignificant activities involved in the data gathering.
The similar recitations in the other independent claims are rejected under a similar rationale
Claim 8 – the “storing…” is a token post-solution activity, i.e. “adding a final step of storing data to a process that only recites computing the area of a space (a mathematical relationship) does not add a meaningful limitation to the process of computing the area.” As discussed in MPEP § 2106.05(g) , and the similar recitations in the other independent claims is rejected under a similar rationale. A similar rationale is also used for the graph databases, in view of ¶¶ 30-33, which merely convey the generic databases are to store the models.
Claim 1- generating, via the processor, a three-dimensional virtual model of the physical environment using the third graph database and component models stored in memory to construct three dimensional operational virtual objects in the virtual model that represent the objects in the physical environment and simulate real life characteristics of the objects in the physical environment; - the simulation is an example of token post-solution activities that are nominally/tangentially linked to the primary process of the claimed invention as the “simulate…” is a token post solution activity (¶ 51) recited and described in a high degree of generality without any particular details on how it is to perform this simulation of real life conditions (¶ 6) outside of simply invoking the commonplace commercially available software of the Unity game engine (see citations to disclosure above). Should the generation of the virtual model be found not to be part of the abstract idea then this would be another token post-solution activity akin to mere data displaying (fig. 2, # 120-122; ¶ 36). The similar recitations in the other independent claims is rejected under a similar rationale
To further clarify, ¶ 21 of the instant disclosure: “The semantic model may then be used to generate a digital twin by selecting and placing models (e.g., CAD models) representing the various components in the environment in the digital twin of the environment” - this is an insignificant computer implementation of generating a computerized model of the physical environment in place of a mental model/pen and paper model, wherein the described technique to generate the digital twin would readily be performable by a person using a computer as a tool to implement the computerized model (e.g. the person mentally observing the semantic model # 118 in fig. 2, and then dragging-and-dropping CAD models to generate a 3D CAD model to reflect the semantic model, wherein the 3D CAD model reflects a mental observation, e.g. a mental visualization, of the physical environment).
Claim 1- and transmitting via the processor the virtual model to a computing device to enable digital interaction with the three-dimensional virtual model.. – mere data transmission, with a functional recitation (MPEP § 2173.05(g)) of “to enable…” without even requiring the interaction to be performed (MPEP § 2111.04(I)), wherein this is merely describing the capability of a process step positively recited (the “transmitting…”) which is mere data transmission. Should the interaction be given patentable weight in that it is required, then it would be mere data gathering recited in a high degree of generality. The similar recitations in the other independent claims is rejected under a similar rationale
Claim 23 - registering, by a processing element, image color data captured by a camera with depth data captured by a depth scan of a physical environment; analyzing the image color data by a machine learned algorithm to detect and localize components within a two dimensional representation of the physical environment; analyzing, by a processing element, with a machine learned model the two-dimensional representation of the physical environment and the depth data to identify the components and estimate a pose of the components in the physical environment; - rejected under a similar rationale as the similar limitations found in the other independent claims (i.e. mere data gathering with insignificant activities involved in the data gathering).
generating, by the processing element, a spatial database forming a first graph database of the physical environment, the spatial database including the components of the physical environment; - mere data gathering/storing, see ¶ 69: “For example, measured data 602 may be utilized to create a spatial database 606 or other physical model and representative data 604 may be utilized to create a process model 608” – also see ¶ 71, i.e. this is merely creating a collection of the measured data into a database (i.e. gather, then store information in a database).
and generating, by the processing element, a three-dimensional virtual model of the physical environment using the semantic model and a computer aided (CAD) library, wherein the CAD library comprises mathematical representations of physical components within the physical environment and the mathematical representations are modified to match specific features of the physical components and construct three dimensional interactive components in the three-dimensional virtual model; - rejected under a similar rationale as the generation of the virtual model step above.
and outputting , by a processing element, the three-dimensional virtual model to a display to allow a user to navigate through and interact with the three-dimensional interactive components in the three-dimensional virtual model.. – mere data outputting, rejected under a similar rationale as the transmitting step above
Of particular note for the above analysis, given the particularity recited in the data gathering, the Examiner notes example 45, dependent claim 3; and Thales Visionix, Inc. v. United States, 850 F.3d 1343, 1348-49, 121 USPQ2d 1898, 1902 (Fed. Cir. 2017) as discussed in MPEP § 2106.05(a) – i.e. in the Examiner’s view, the question of whether or not the data gathering steps render this claimed invention as being eligible subject matter is a step 2B consideration at the WURC consideration (see below). To clarify, in example 45, claim 3 see its discussion of the thermocouple, wherein the improvement was only discussed at step 2B for mere data gathering.
With respect to claim 23, “… the component models comprising at least one of parametric models of the objects, standardized models of the objects, combinations of parametric models and standardized models, or polygonal meshes; and” – this is considered as part of the mere instructions to do it on a computer, as well as generally linking to a particular technological environment.
To clarify, see ¶ 35, and ¶ 49. Neither the claims nor the disclosure provides any particular disclosure on what these models are, or how they are to be particularly manipulated (e.g. particular methods of manipulating their particular data structures; see Research Corp. in MPEP 2106.04(a)(2)(III)(A)). Rather, this is akin to “Although the claims purported to modify the underlying XML document in response to modifications made in the dynamic document, nothing in the claims indicated what specific steps were undertaken other than merely using the abstract idea in the context of XML document” in Intellectual Ventures I v. Capital One Fin. Corp., 850 F.3d 1332, 121 USPQ2d 1940 (Fed. Cir. 2017) in MPEP § 2106.05(f).
In addition, the above insignificant extra-solution activities are also considered as well-understood, routine, and conventional activities, as discussed in MPEP § 2106.05(d):
The “storing…” limitations, the graph database limitations, and the generating a spatial database in claim 23 are akin to MPEP § 2106.05(d)(II) of: “iii. Electronic recordkeeping, Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. 208, 225, 110 USPQ2d 1984 (2014) (creating and maintaining "shadow accounts"); Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755 (updating an activity log); iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;” – also, see the below discussed WURC evidence.
Also, graph databases are indicated as WURC in the instant disclosure, e.g. see ¶¶ 30-34, wherein these merely list: “However, it should be noted that various types of storage structures and encoded data may be used, such as, but not limited to, graph and other database structures, SQL databases and the like.”, i.e. a generic description of databases that POSITA would have considered WURC in the art, given the omission of any particular details on these that indicates, by the test of enablement, that the disclosure merely omitted what was well-known. For additional evidence, see Amazon, “What is a Graph Database”, URL: aws(dot)amazon(dot)com/nosql/graph/, accessed via WayBack machine with an archive date of Feb. 2017, for: “A graph database stores vertices and directed links called edges. Graphs can be built on relational (SQL) and non-relational (NoSQL) databases. Vertices and edges can each have properties associated with them. The diagram below depicts a simple graph of relationships between friends and their interests… Amazon Web Services (AWS) provides a variety of graph database options. You can operate your graph database in the cloud on Amazon EC2 and Amazon EBS and work with Amazon solution providers….”
Sierla et al., “Towards Semi-Automatic Generation of a Steady State Digital Twin of a Brownfield Process Plant”, 2020, page 11 last paragraph
Abri et al., “Probabilistic relational models learning from graph Databases”, 2018, chapter 4. Include seeing § 4.1 ¶ 4.
The “transmitting…” limitations are akin to MPEP § 2106.05(d)(II) of: “i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network);”
The outputting to a display and having using interaction with the display (claim 23, the interaction functional limitation in the other independent claims) is akin to “…vi. A Web browser’s back and forward button functionality, Internet Patent Corp. v. Active Network, Inc., 790 F.3d 1343, 1348, 115 USPQ2d 1414, 1418 (Fed. Cir. 2015)….iv. Presenting offers and gathering statistics, OIP Techs., 788 F.3d at 1362-63, 115 USPQ2d at 1092-93;…” as discussed in MPEP § 2106.05(d)(II); as well as example 46, claim 1, step 2B analysis: “Similarly, limitation (c) is just a nominal or tangential addition to the claim, and displaying data is also well-known.” – also, see the below discussed WURC evidence
The act, in claims 1, 8, and 13 of generating the physical model, including its sub steps (including its generation of a graph, e.g. fig. 2 # 108), and the similar recitations in claim 23, is considered WURC in view of the below:
Izadinia, Hamid, Qi Shan, and Steven M. Seitz. "Im2cad." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. Abstract, then see § 1 including: “In his 1963 Ph.D. thesis, Lawrence Roberts [34] demonstrated a system that infers a 3D scene from a single photo (Figure 2). Leveraging a database of known 3D objects, his system analyzed edges in the image to infer the locations and orientations of these objects in the scene. Unlike the vast majority of modern 3D reconstruction techniques, which capture only visible surfaces, Robert’s method was capable of inferring back-facing and occluded surfaces, object segments, and recognized which objects are present…. One major limitation is the need for an accurate, a priori 3D model of each object in the scene. While a chair model, e.g., is not hard to come by, obtaining exact 3D models of every chair in the world is not presently feasible. A further challenge is the need to reliably match between features in photographs and CAD models, particularly when the model does not exactly match the object photographed… We therefore introduce a variant of Robert’s original problem, that we call IM2CAD, in which the goal is to reconstruct a scene that is as similar as possible to the scene depicted in a photograph, where the reconstruction is composed of objects drawn from a database of available 3D object models… Our work builds on a number of recent advances in the computer vision and graphics research community. First, we leverage ShapeNet [6], which contains millions of 3D models of objects, including thousands of different chairs, tables, and other household items… Second, we use state-of-the-art object recognition algorithms [32] to identify common objects like chairs, tables, windows, etc.; these methods work impressively well in practice. Third, we leverage deep features trained by convolutional neural nets (CNNs) [21] to reliably match between photographs and CAD renderings [3, 20, 36, 18].” – then see § 2, then see §§ 3.2-3.4
Wald, Johanna, et al. "Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions." 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020. See the abstract: “Scene understanding has been of high interest in computer vision. It encompasses not only identifying objects in a scene, but also their relationships within the given context. With this goal, a recent line of works tackles 3D semantic segmentation and scene layout prediction. In our work we focus on scene graphs, a data structure that organizes the entities of a scene in a graph, where objects are nodes and their relationships modeled as edges. We leverage inference on scene graphs as a way to carry out 3D scene understanding, mapping objects and their relationships….” – then see § 1 including: “3D scene understanding relates to the perception and interpretation of a scene from 3D data, with a focus on its semantic and geometric nature, which includes not only recognizing and localizing the objects present in the 3D space therein, but also their context and relationships. This thorough understanding is of high interest for various applications such as robotic navigation, augmented and virtual reality. Current 3D scene understanding works include perception tasks such as instance segmentation [12, 21, 44, 50], semantic segmentation [34, 36, 5, 38] as well as 3D object detection and classification [40, 34, 35, 54]… Scene understanding from images has recently explored the use of scene graphs to aid understanding object relationships in addition to characterizing objects individually. Before that, scene graphs have been used in computer graphics to arrange spatial representations of a graphical scene, where nodes commonly represent scene entities (object instances), while the edges represent relative transformations between two nodes. This is a flexible representation of a scene which encompasses also complex spatial relations and operation grouping. Some of these concepts where successively adapted or extended in computer vision datasets, such as support structures [32], semantic relationships and attributes [19] and hierarchical mapping of scene entities [3]. Scene graphs have been shown to be relevant, for instance, for partial [46] and full matching [17] in image search, as well as image generation [16]… In 3D, scene graphs have only recently gained more popularity [3]. In this work, we want to focus on the semantic aspects of 3D scene graphs as well as their potential…” – e.g. fig. 1; then see § 2 including: “Johnson et al. [17] introduced scene graphs – motivated by image retrieval – as a representation that semantically describes an image, where each node is an object while edges represent interactions between them. Additionally, the object nodes contain attributes that describe object properties [see the remaining citations of related art]… An active research area within 3D scene understanding focuses on 3D semantic segmentation [34, 4, 36, 7, 38, 12] and object detection and classification [41, 34, 35, 52]. These works mostly focus on object semantics and context is only used to improve object class accuracy… Another line of works incorporate graphs structures for object-level understanding, rather than entire scenes… Only recently the community started to explore semantic relationships in 3D and on real world data…” as well as “…Many image-based 3D retrieval works focus on retrieving 3D CAD models from RGB images: IM2CAD generates a 3D scene from a single image by detecting the objects, estimating the room layout and retrieving a corresponding CAD model for each bounding box [each object detected by a bounding box] [14]. Pix3D on the other hand propose a dataset for single image 3D shape modeling based on highly accurate 3D model alignments in the 2D images [42]. Liu et al. show improved 2D-3D model retrieval by simulating local context to generate false occlusion [27]. The SHREC benchmark [1, 2], enables 2D-3D retrieval of diverse scenes (beach, bedroom or castle), while [30] and [9] operate on indoor environments but also only focus on synthetic data rather than real 3D reconstructions.” – then, see § 3.1 including fig. 3; and § 3.3, and fig. 4-6
Zollhöfer, Michael, et al. "State of the art on 3D reconstruction with RGB‐D cameras." Computer graphics forum. Vol. 37. No. 2. 2018. Abstract: “The advent of affordable consumer grade RGB-D cameras has brought about a profound advancement of visual scene reconstruction methods. Both computer graphics and computer vision researchers spend significant effort to develop entirely new algorithms to capture comprehensive shape models of static and dynamic scenes with RGB-D cameras. This led to significant advances of the state of the art along several dimensions…” – e.g., see § 1 ¶ 1 which describes commercially available RGB-D cameras, then ¶ 2: “First, highly innovative new algorithms for RGB-D-based dense 3D geometry reconstruction of static environments were developed… Second, entirely new methods for capturing dense 3D geometry models of dynamic scenes and scene elements were proposed, such as models of moving humans and rigid objects, or of general deformable surfaces…” – see § 1.1 for more clarification on the sensing devices used ; then see § 2.1 including: “Although there are many different algorithms for RGB-D reconstruction of static scenes, most if not all of these approaches have a very similar processing pipeline, which we describe here for reference (c.f. Fig. 3)…. In the first stage, the Depth Map Preprocessing, noise reduction and outlier removal is applied to the incoming RGB-D data… In the subsequent stage, the Camera Pose Estimation, the best aligning transformation T for the current frame (c.f. Sec. 2.3) is computed..” – see fig. 3 for a visual overview of this; see §§ 2.2 and 2.3 for more details on these steps, e.g. in § 2.3.1: “Early works on off-line 3D shape registration [object identification] heavily inspire current approaches for real-time camera tracking based on depth streams. The first proposed techniques employed simple frameto- frame variants of the Iterative Closest Point Algorithm (ICP) [BM92, YM92] and were based on a point-to-point [BM92] or point-to-plane [YM92] error metric…” – e.g. ¶ 41 of the instant disclosure – then see § 2.3.2
Song, Shuran, Samuel P. Lichtenberg, and Jianxiong Xiao. "Sun rgb-d: A rgb-d scene understanding benchmark suite." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015. Abstract: “Although RGB-D sensors have enabled major breakthroughs for several vision tasks, such as 3D reconstruction, we have not attained the same level of success in high level scene understanding. Perhaps one of the main reasons is the lack of a large-scale benchmark with 3D annotations and 3D evaluation metrics” then § 1 including: “RGB-D sensors have also enabled rapid progress for scene understanding (e.g. [20, 19, 53, 38, 30, 17, 32, 49]). However, while we can crawl color images from the Internet easily, it is not possible to obtain large-scale RGB-D data online… To this end, we introduce SUN RGB-D, a dataset containing 10,335 RGB-D images with dense annotations in both 2D and 3D, for both objects and rooms. Based on this dataset, we focus on six important recognition tasks towards total scene understanding, which recognizes objects, room layouts and scene categories. For each task, we propose metrics in 3D and evaluate baseline algorithms derived from the state-of-the-arts. Since there are several popular RGB-D sensors available, each with different size and power consumption, we construct our dataset using four different kinds of sensors to study how well the algorithms generalize across sensors.” - then, see fig. 2 which visually depicts the “color” and the “raw depth” data obtained from these sensors, as well as “raw points”, and corresponding “refined depth” and “refined points” – then see fig. 3 which shows “Example images with annotation from our dataset.”; § 2.1 gives a brief discussion of several examples of popular RGB-D sensors; then see § 2.2: “For RGB-D sensors, we must calibrate the camera intrinsic parameters and the transformation between the depth and color cameras… We use the stereo calibration function to calibrate the transformation between the depth (IR) and the color cameras [the transformation being an example of registering the data sets to each other]” – and see § 3 including for “…Object detection is another important step for scene understanding… Object Orientation Besides predicting the object location and category, another important vision task is to estimate its pose. For example, knowing the orientation of a chair is critical to sit on it properly. Because we assume that an object bounding box is aligned with gravity, there is only one degree of freedom in estimating the yaw angle for orientation… Room Layout Estimation The spatial layout of the entire space of the scene allows more precise reasoning about free space (e.g., where can I walk?) and improved object reasoning. It is a popular but challenging task for color-based scene understanding (e.g. [22, 23, 24]). With the extra depth information in the RGB-D image, this task is considered to be much more feasible [74]. We evaluate the room layout estimation in 3D by calculating the Intersection over Union (IoU) between the free space from the ground truth and the free space predicted by the algorithm output… As shown in Figure 7, the free space is defined as the space that satisfies four conditions: 1) within camera field of view, 2) within effective range, 3) within the room, and 4) outside any object bounding box (for room layout estimation, we assume empty rooms without objects)… The final task for our scene understanding benchmark is to estimate the whole scene including objects and room layout in 3D [38]. This task is also referred to “Basic Level Scene Understanding” in [71]. We propose this benchmark task as the final goal to integrate both object detection and room layout estimation to obtain a total scene understanding, recognizing and localizing all the objects and the room structure…” – see § 4 for more clarification, including various uses of machine learning algorithms for these tasks such as using various types of a “Convolutional Neural Net” (instant disclosure, ¶ 41) as well as other machine learning algorithms, e.g. “SVM”, wherein it clarifies on page 574: “We use RGB-D RCNN and Sliding Shapes for object detection and combine them with Manhattan Box for room layout estimation.” – see fig. 11 for a “Visualization of total scene understanding results”
As discussed above, see Silberman, Nathan. Reasoning about Object Instances, Relations and Extents in RGBD Scenes. Diss. New York University, 2015.
Ren, Zhile, and Erik B. Sudderth. "Clouds of oriented gradients for 3D detection of objects, surfaces, and indoor scene layouts." IEEE Transactions on Pattern Analysis and Machine Intelligence 42.10 (2019): 2670-2683. § 1: “SEMANTIC understanding of three-dimensional (3D) scenes plays an increasingly important role in modern robotic systems and autonomous vehicles… Advances in depth sensor technology can reduce ambiguities in standard RGB images, enabling breakthroughs in scene layout prediction [3], [4], [5], support surface prediction [6], [7], [8], semantic parsing [9], and object detection [10], [11], [12], [13]…. Holistic indoor scene understanding [16] requires integrated detection of objects and the room layouts (walls, floors, and ceilings) that surround them… We instead propose to detect the 3D size, position, and orientation of object instances via bounding cuboids (convex polyhedra). 3D cuboid detection is a standard task in indoor and outdoor scene understanding benchmarks [16], [17]… Descriptors constructed from point cloud representations of RGB-D images are frequently used for 3D object detection.” – then see § 2 incl: “Two-dimensional object detection is a widely studied problem. Dalal and Triggs [25] introduced the histogram of oriented gradient (HOG) descriptor to model 2D object appearance using image gradients… Increasingly, real-world computer vision systems often incorporate depth data as an additional input to increase accuracy and robustness. With depth maps we can reconstruct point cloud representations of scenes, leading to significant recent advances in 3D object classification [34], [35], point cloud segmentation [36], [37], cuboid-based geometric modeling [38], [39], [40], room layout prediction [41], [42], 3D contextual modeling [43], [44], and 3D shape reconstruction [45], [46]. Here, we focus on the related problem of 3D object detection… In outdoor scenes, localizing objects with 3D cuboids has become a standard in the popular KITTI autonomous driving benchmark [17]…. For robotics applications, a 3D convolutional neural network was designed to detect simple objects in real time [59]… Detecting support surfaces is an essential first step in understanding the geometry of 3D scenes for such tasks as surface normal estimation [7], [62] and shape retrieval [63]. Silberman et al. [6] use semantic segmentation to model object support relationships; this work was later extended by Guo et al. [8] for support surface prediction… Some related work has predicted 2D projections of the 3D layout [5], [23], [41], [42], [64], [65], or used CNNs to directly predict the 3D layout [66]…. More broadly, holistic scene understanding systems integrate forms of semantic object reasoning, spatial context modeling, and scene type identification [16], [67]. Often, models for each sub-task are learned independently, and then integrated via conditional random fields (CRFs) like that proposed by Lin et al. [53]. However, rich scene models lead to complex graph structures and challenging inference problems.” – e.g. see fig. 1; also see fig. 9 (as discussed in § 6)
Lahoud, Jean, and Bernard Ghanem. "2d-driven 3d object detection in rgb-d images." Proceedings of the IEEE international conference on computer vision. 2017. Abstract and §§ 1-2, including § 2 ¶ 1 which discusses the numerous times “CNN[s]” are used in the process of object detection; see ¶ 2 of § 2 for further clarification, then see § 2 ¶ 3: “The method of [30] uses renderings of 3D CAD models from multiple viewpoints to classify all 3D bounding boxes obtained from sliding a window over the whole space. Using CAD models restricts the classes and variety of objects that can be detected, as it is much more difficult to find 3D models of different types and object classes than photograph them. Also, the sliding window strategy is computationally demanding, rendering this technique quite slow. Similar detectors use object segmentation along with pose estimation to represent objects that have corresponding 3D models in a compiled library [7]…” and later in § 2: “Recent works have also applied ConvNets for 3D object detection…” – then see fig. 2-3, in particular see the “Factor graph” in fig. 3 (page 4626, col. 1, for its accompanying description)
Kermani, Z. Sadeghipour, et al. "Learning 3D Scene Synthesis from Annotated RGB‐D Images." Computer Graphics Forum. Vol. 35. No. 5. 2016. See the abstract; see fig. 1 including its caption: “Progressive synthesis of 3D indoor scenes. Starting from an empty room, results from steps 1, 4, and 7 of the synthesis procedure are shown. Two close-ups are given on the side for the last result to highlight the placement of small objects into the scene. Object selection and arrangement are implemented fully automatically based on models learned from a large set of annotated RGB-D Images.” – then see § 1 including: “The best known and state-of-the-art method for synthesizing 3D indoor scenes is the work of Fisher et al. [FRS12]…. Similar to Fisher et al. [FRS12], our probabilistic model also consists of two main components: a cooccurrence model to guide which objects are to be inserted into the scene and an arrangement model to determine where each object should be placed. However, there are two key distinctions related to the probabilistic model and the associated learning process: In addition to considering pairwise object relations, we extract and learn salient higher-order relations involving more than two objects, e.g., two nightstands symmetrically surrounding a bed.” – e.g. instant fig. 2 # 108 which shows “pairwise object relations” for the arrangement model/physical model – see Kermain, fig. 2 for a visual clarification of these “pairwise spatial relations”; also see § 5 and figures 5-6, wherein fig. 5 shows a presumably manually annotated arrangement model drawn over an image. Page 200, subsection “Proximity Relations” for its accompanying description.
Previously cited Czerniawski, Thomas. Updating digital models of existing commercial buildings using deep learning. Diss. 2020. Page 32, ¶¶ 1-3 including: “Computer vision hardware is a dominant source of digital representations. Available computer vision hardware includes: visible light cameras (e.g. 2D image, 360 image [133], and video), structured-light 3D scanners (e.g. range cameras and RGB-D), thermographic or infrared cameras [100,134], hyperspectral imagers [280], LiDAR scanners, and ground-penetrating radar. Selection of hardware is based on: acquisition cost and pre-existing hardware ownership, operational expertise, information interoperability (e.g. vision hardware and computing hardware, vision hardware and processing software, and sensor fusion [148]), computational power for data processing, and the hardware’s ability to digitize relevant reality (e.g. specular or dark surface limitations [108]) at the required scope, resolution, and accuracy.”, then see § 2.3.3.4 including: “Table 2-5 classifies recognition methods from these articles based on the type of input data used for representing the search space: RGB, RGB-D, and point cloud.” And see table 2-5 which shows that “15” articles used “RGB”, “17”, used “RGB-D”, and “32” used a “Point Cloud”; also see page 62 ¶¶ 2-3 then see table 3-1, and then see § 3.3.2.1: “There are many RGB-D sensors available on the market…”, also see the abstract: “To address this limitation, I show how a deep convolutional neural network can be trained to semantically segment RGB-D (i.e. color and depth) images into thirteen building component classes using a new annotated dataset called 3DFacilities.”
Previously cited Frebet, V., et al. "Interactive semantics-driven reconstruction methodology from Point Clouds." 29th CIRP DESIGN 2019 OPEN DESIGN and DESIGN AS EXPONENTIAL TECHNOLOGY. 2019. See § 1 including ¶ 1
The act of generating a virtual model based on component models stored in a library in claims 1, 8, and 13; and the similar recitation in claim 23 wherein these are CAD component models, are considered WURC in view of:
The above discussed evidence (as cited above), including Izadinia, Hamid, Qi Shan, and Steven M. Seitz. "Im2cad." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017 as well as Wald, Johanna, et al. "Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions." 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020.
For additional evidence, see the previously cited:
Son, Hyojoo, Changmin Kim, and Changwan Kim. "3D reconstruction of as-built industrial instrumentation models from laser-scan data and a 3D CAD database based on prior knowledge." Automation in Construction 49 (2015): 193-200. Abstract, then see § 1 including: “Instrumentation plays a vital role in operations carried out in industrial plants. In the operation and maintenance phase of such facilities, it is important to ensure that the as-built condition of each installed piece of instrumentation, together with any subsequent changes, is properly recorded—and adjusted for reference by engineers and managers in the interpretation process. The three-dimensional (3D) reconstruction of as built industrial plant models is an important task in many applications, as it allows for the generation of a digital representation of the current status of an existing industrial plant… It has recently become possible to efficiently carry out 3D measurements of as-built conditions by using terrestrial laser scanners [4–6,11, 10,7]…. Under such circumstances, users can utilize and retrieve the existing 3D models of individual pieces of instrumentation. Although this process may seem simple, however, identifying the instrumentation type (e.g., davits, heaters, heat exchangers, pumps, tanks, vessels, butterfly valves, check valves, gate valves, and globe valves) and size, and retrieving 3D models for a number of different types and sizes, is a complex procedure requiring skilled personnel, a high level of knowledge, and tremendous effort. Users need to place 3D models for each piece of instrumentation at an installed location in a 3D environment, and this task becomes time consuming… There have been efforts over the past decade to automate some of these tasks…” and in § 2 see: “In addition to this, this study utilized 3D databases from existing industrial plants which contain a library of reusable models of instrumentation.”
Martinez, Gerardo Santillan, et al. "Automatic generation of a simulation-based digital twin of an industrial process plant." IECON 2018-44th Annual Conference of the IEEE Industrial Electronics Society. IEEE, 2018. Abstract, then see § 1: “Recent advances in modelling and simulation technology and in industrial interoperability standards have resulted in the development of Digital Twins of production plants [1], [2]. A Digital Twin (DT) is a digital replica of the physical assets of an industrial plant which contains the structure and the dynamics of how the devices and the process operate [3]. They are a powerful application for decision support of operational process plants in sectors such as chemical, power generation, mineral processing, pulp & paper and oil & gas… In contrast, simulation-based DTs (SBDTs) are based on online first-principles simulation models [4]–[6]. First principles models (FPMs) rely on engineering, physics or chemical descriptions to represent the behavior of the plant [7]. As shown in Fig. 1, in SBDTs, a simulation model runs together with the plant, while estimation techniques keep the simulation state in the same state as the targeted device or process [8]. These simulation configurations are also known as online model-based applications [5]. A SBDT can be used to obtain high-fidelity predictions, including production forecasts of operating regions from which no measurement data is available [9]. Furthermore, SBDTs can be used for developing operator training simulation systems, for production optimization, or for troubleshooting and failure diagnoses. SBDTs are a holistic and powerful application for plant operation support of modern industrial plants…”, then see § II.A
Sierla, Seppo, et al. "Towards semi-automatic generation of a steady state digital twin of a brownfield process plant." Applied Sciences 10.19 (2020): 6959. Abstract, then see § 1 including: “Despite much recent research on digital twins,…”
Kim, Hyungki, et al. "Deep-learning-based retrieval of piping component catalogs for plant 3D CAD model reconstruction." Computers in Industry 123 (2020): 103320. Abstract: “In process plants, 3D computer-aided design (CAD) plant models are frequently generated via reverse engineering using point cloud data. Generating a 3D model from scan data consists of point cloud collection, preprocessing, and modeling. The process of 3D modeling to create a plant 3D CAD model from a registered point cloud consists of grouping similar point clouds into several segmented point clouds, identifying the components represented by each segment, selecting catalogs for the components, and placing them into 3D design space. The core of a 3D modeling process is to identify components represented by the segmented point clouds. This study proposes a deep learning-based method to retrieve catalogs for piping components to support the reconstruction of a plant 3D CAD model from point clouds”, then see § 1 including ¶¶ 2-5, and page 2, col. 2, ¶¶ 4-7 including: “Then, the components represented by each separate segmented point cloud are identified. For example, in the case of a building, a segmented point cloud could represent a door, window, or a wall, and, in the case of a plant, it could represent a pipe, valve, or flange. 3D CAD systems in the building or plant field provide a catalog of 3D shapes for various components. Therefore, when a component represented by the segmented point cloud is identified, the corresponding catalog is selected, and the 3D shapes in the catalog are placed in the same position and orientation as those of the segmented point cloud… The core of a 3D modeling process is to identify components represented by the segmented point clouds…”; also see §§ 2.1-2.2, then see § 3 including ¶¶ 1-2 including: “The conventional approach to reconstructing a plant 3D CAD model for piping design proceeds as follows…If the segmented point cloud is not recognized as a pipe type, the nominal diameters of pipes adjacent to the segmented point cloud are determined, retrieval of a piping component cata-log begins, and the catalog corresponding to the segmented point cloud is returned. Finally, the piping component catalog is selected and placed in the 3D design space”
Sierla, Seppo, Mohammad Azangoo, and Valeriy Vyatkin. "Generating an industrial process graph from 3d pipe routing information." 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA). Vol. 1. IEEE, 2020. Abstract: “The automatic generation of digital twins of industrial processes requires the integration of several sources of information. If the twin is expected to accurately capture thermo-hydraulic phenomena, dimensions of tanks and other process components as well as detailed pipe routing information is relevant… Further research is expected on applying graph matching methods for integrating these separate graphs to a common graph-based data structure that captures all of the desired information. This common model could support further work to develop digital twins. A major obstacle to this is that the graphs that have currently been generated from P&IDs and 3D CAD models are at very different abstraction levels, so graph matching methods are not feasible….” Then see §§ I-II including: “A digital twin is a comprehensive physical and functional description of a component, product or system. The level of detail that is captured by the twin determines its potential for applications along the life cycle of its physical counterpart [5]. Due to high demand for digital twins, their automatic generation is interesting, since manual generation involves significant time and cost. The most two popular sources of information for automatic generation of digital twins are design phase documents, such as P&IDs, and 3D models. 3D point clouds obtained by scanning technologies have been used to provide visualizations for existing manufacturing simulations [6] and for object detection in factories [7] and other built environments [8]…”
Sierla, Seppo, et al. "Integrating 2D and 3D digital plant information towards automatic generation of digital twins." 2020 IEEE 29th international symposium on industrial electronics (ISIE). IEEE, 2020. Abstract, §§ I-II
Kalasapudi, Vamsi Sai, Yelda Turkan, and Pingbo Tang. "Toward automated spatial change analysis of MEP components using 3D point clouds and as-designed BIM models." 2014 2nd International Conference on 3D Vision. Vol. 2. IEEE, 2014. Abstract: “The architectural, engineering, construction and facilities management (AEC-FM) industry is going through a transformative phase by adapting new technologies and tools into its change management practices. AEC-FM Industry has adopted Building Information Modeling (BIM) and three dimensional (3D) laser scanning technologies in tracking changes in the whole lifecycle of building and infrastructure projects, from planning to design and construction, and finally to facilities management…”, § II.A including: “Three-dimensional (3D) laser scanning is an imaging technology increasingly adopted in the Architectural, Engineering, Construction, and Facilities Management (AEC-FM) industry.”
Czerniawski, Thomas. Updating digital models of existing commercial buildings using deep learning. Diss. 2020. Page 2, ¶ 2: “Representing the physical world in digital form provides many efficiencies. First and foremost, a ‘digital twin’ reduces an investigator’s dependence on many of the slow and occasionally destructive methods for information gathering in the physical world, e.g. traveling long-distances, acquiring and using lift equipment, and dismantling obfuscating barriers” and page 6, ¶¶ 1-2, then see page 32, ¶¶ 1-3 including: “Computer vision hardware is a dominant source of digital representations. Available computer vision hardware includes: visible light cameras (e.g. 2D image, 360[Calibri font/0xB0] image [133], and video), structured-light 3D scanners (e.g. range cameras and RGB-D), thermographic or infrared cameras [100,134], hyperspectral imagers [280], LiDAR scanners, and ground-penetrating radar. Selection of hardware is based on: acquisition cost and pre-existing hardware ownership, operational expertise, information interoperability (e.g. vision hardware and computing hardware, vision hardware and processing software, and sensor fusion [148]), computational power for data processing, and the hardware’s ability to digitize relevant reality (e.g. specular or dark surface limitations [108]) at the required scope, resolution, and accuracy.”, then see § 2.3.3.4 including: “Table 2-5 classifies recognition methods from these articles based on the type of input data used for representing the search space: RGB, RGB-D, and point cloud.” And see table 2-5 which shows that “15” articles used “RGB”, “17”, used “RGB-D”, and “32” used a “Point Cloud”; also see page 62 ¶¶ 2-3 then see table 3-1, and then see § 3.3.2.1: “There are many RGB-D sensors available on the market…”
Rausch, Chris, et al. "Computational algorithms for digital twin support in construction." Construction Research Congress 2020. Reston, VA: American Society of Civil Engineers, 2020. Abstract, then see page 2, ¶¶ 2-3 including: “…This is a key step in working towards digitization processes that are required in the creation of digital twins in construction. The use of computational algorithms to build, support and maintain digital twins is a new area of research that is expected to gain wide attention and industry use in the near future… The concept of a virtual or digital equivalent to a physical product has roots dating back to the early 2000s, where it was initially presented as a tool for product life cycle management in the manufacturing industry (Schleich et al. 2017). Since then, the concept has matured, and the term "digital twin" is being used in a wide range of activities. Across definitions and applications, every digital twin can be distilled into three main components: (1) physical assets, systems, and processes, (2) the virtual representation of physical assets, systems and processes and (3) the interconnectedness of an object’s actual state to measurable information allows for mapping the evolution of the state of an object over time. The key for every digital twin lies in the symbiotic relationship between the real world and digital objects, wherein real-world data on a physical object is used to update a virtual model, and analysis of a virtual model is used to make predictions on the real world model…”
Cho, Chi Yon, Xuesong Liu, and Burcu Akinci. "Automated building information models reconstruction using 2D mechanical drawings." Advances in Informatics and Computing in Civil and Construction Engineering: Proceedings of the 35th CIB W78 2018 Conference: IT in Design, Construction, and Management. Cham: Springer International Publishing, 2018. § 60.1: “All aspects described above signify a need to create BIMs for existing buildings in a cost- and time- efficient way. To address this need, a wide range of research studies have been conducted using 3D point cloud data (i.e., terrestrial laser-scanned data), photos, or building drawings to reconstruct B” and see table 60.1 for more clarification.
Son, Hyojoo, and Changwan Kim. "Automatic segmentation and 3D modeling of pipelines into constituent parts from laser-scan data of the built environment." Automation in Construction 68 (2016): 203-211. Abstract, then see § 1 ¶¶ 1-3 including: “…Several commercially available software programs have been developed to assist the current manual process of 3D modeling (e.g., Leica CloudWorx by Leica Geosystems, AutoCAD Plant 3D by Autodesk, and EdgeWise Plant by ClearEdge3D, all of which were introduced in 2014). These programs are user-friendly tools for the 3D modeling of the pipelines, provide several features for manipulation of laser-scan data in the form of 3D point clouds acquired from the built environment, and have the capability to create and modify pipeline models to help and guide users in what would otherwise be a repetitive, tedious, and time-consuming modeling process [14]…”, also see § 2.
Son, Hyojoo, Frédéric Bosché, and Changwan Kim. "As-built data acquisition and its use in production monitoring and automated layout of civil infrastructure: A survey." Advanced Engineering Informatics 29.2 (2015): 172-183. Abstract, then see § 1 including: “Advancements in on-site spatial survey technologies (e.g., photo/video-grammetry and terrestrial laser scanning) enable more efficient acquisition of 3D data on as-built civil infrastructure (hereinafter referred to as ‘‘as-built data’’)… Three-dimensional as-built data acquired from civil infrastructure have been used to establish geometric properties of entire facilities and their constituent components…”- then see § 2 including § 2.2.1.2, then see § 3 including: “Currently, modeling which is done to represent the existing state of an as-built pipeline or the 3D layout of an as-built building is mostly performed manually – in an interactive manner – by the user. Especially, 3D layout of as-built pipelines from 3D point clouds has been extensively investigated, and several commercially available software programs have been developed to assist the current manual process of 3D layout. Most providers of laser-scanning systems (e.g., Leica Geosystems and Trimble) have developed software that enables the 3D layout of as-built pipelines from 3D point clouds…. The leading 3D CAD vendors (Autodesk, Bentley, Aveva, and Intergraph) have also developed software that enables the 3D layout of as-built pipelines from 3D point clouds. One example of this is AutoCAD Plant 3D, which can be used with Kubit’s PointSense Plant add-in for AutoCAD (see Fig. 14a). PointSense Plant by Kubit provides several functions for pattern recognition that can identify pipelines from 3D point clouds…”
Flynn, Patrick J., and Anil K. Jain. "CAD-based computer vision: from CAD models to relational graphs." Conference Proceedings., IEEE International Conference on Systems, Man and Cybernetics. IEEE, 1989. See § 1 for the subsection “Adaptation of Preexisting Models” including: “In many industrial applications, the 3D geometry of a solid object may have been predefined for some other purpose. (e.g., robotic assembly of printed-circuit boards requires a model of the board as well an models of components to be mounted). If the model descriptions in the existing database can be automatically adapted, augmented, or restructured for vision tasks, little additional human intervention would be required to construct models for object recognition. In other words, the ‘sharing’ of models between manufacturing disciplines allows database consistency to be maintained, minimizes redundant design effort, and allows a closer integration of the system involved…”
Tang et al., US 2017/0337299, abstract, ¶¶ 5-8 including: “…Several researchers explored the potential of using three-dimensional (3D) imaging technologies, namely 3D laser scanning and photogrammetry coupled with computer vision, for change analysis between as-designed and as-built conditions….Recent studies have explored the application of relational graphs to match and compare objects from 3D as-designed models with the objects in the corresponding 3D as-built model accurately, which has significant advantages over data-model comparison tools that are available in commercial 3D data processing and reverse engineering environments, such as InnovMetric Polyworks. However, comparing relational graphs generated from as designed models and 3D laser scan data of large-scale building systems (e.g., hundreds of interconnected ductworks) involves computational complexity that grows exponentially with the number of building elements…” and ¶ 10: “Manual comparison of 2D and 3D imagery data against as-designed models is also tedious and error-prone.”
Perez, Yeritza. Semantically-rich as-built 3D modeling of the built environment from point cloud data. Diss. University of Illinois at Urbana-Champaign, 2020. See § 1.1 including: “Despite their benefits, as-is 3D models are often not readily available for existing facilities. To address such needs, 3D geometric and semantic modeling of the built environment from 3D point cloud data is gaining significant popularity…. – Point cloud data can be collected using Laser Scanners, Airborne Laser Swath Mapping (ALSM) techniques, Light Detection, and Ranging (LiDAR) devices, handheld 3D laser scanner, cameras, among others. Once 3D point clouds are captured from several locations and orientations, a complete representation of the built environment can be formed by registering them in a common 3D coordinate system…” – also see # 2-3, and fig. 1.2 including: “Fig.1.2 shows an example of a point cloud and the Building Information Modeling (BIM) generated using it. In today’s best practice (see Fig. 1.3), the Scan2BIM steps are performed by surveyors, designers, and engineers”
Mohamed, Ahmed Gouda, Mohamed Reda Abdallah, and Mohamed Marzouk. "BIM and semantic web-based maintenance information for existing buildings." Automation in Construction 116 (2020): 103209. Abstract, then see § 1 ¶¶ 1-2, then see §§ 2-2.2, then see § 3.1 including: “One of the primary roles of maintaining existing building facilities is appraising the as-is information of these facilities and their components. BIM, as a data-rich and object-oriented model, has been employed in the area of data acquisition for existing building facilities to construct an as-is information BIM model. Within this context, Scan-to- BIM provides physical 3D laser scanning means for indoor spaces or environments to build a precise digital portrayal of these spaces” and § 3.1.1. including: “As-built 3D model creation demands as-is information acquisition of the building facilities components. Data or information capture techniques depend profoundly on the level of the required details needed to scan and to reconstruct existing building facilities for as-is BIM model establishment. The most commonly employed approach for data scanning and retrieving is laser scanning using a terrestrial laser scanner”, then see § 3.1.3 including: “The following as-is information BIM establishment procedure is modeling scanned and segmented indoor point clouds, which designates the constructed BIM features that resemble building components, regarding geometric and non-geometric features relationships. The modeling process depends heavily on commercial applications and software; there are numerous commercial software implementations accessible on the market, which assist in decreasing the workload and time for modeling the segmented building facilities point clouds.”; also see § 3.3.1
Frebet, V., et al. "Interactive semantics-driven reconstruction methodology from Point Clouds." 29th CIRP DESIGN 2019 OPEN DESIGN and DESIGN AS EXPONENTIAL TECHNOLOGY. 2019. Abstract, §§ 1-1.1.2.1, then see §§ 2.1-2.3, 2.3.2, 2.4, and 3.
Kwon, Soon-Wook-Bosche, and Youngki Frederic-Huh. "“MODEL SPELL CHECKER" FOR PRIMITIVE-BASED AS-BUILT MODELING IN CONSTRUCTION." Korean Journal of Construction Engineering and Management. Volume 5 Issue 5 Serial No. 21 / Pages.163-171 / 2004. Abstract, then see § 1 including ¶¶ 2- 4 including: “A recently developed method for workspace modeling, using the human ability to recognize targets, has been developed at the University of Texas at Austin (UT) (Cho et al, 2001, Kwon, 2003). This method takes advantage of the human ability to recognize geometric forms and distinguish target from non-target objects which can significantly reduce acquisition and modeling processing times (Kwon, 2003, McLaughlin, 2001 ). Rapid and accurate methods for fitting and matching primitives to "sparse range point clouds" is a major feature of the method (Kwon, 2003)…”
Holi, Pavitra, et al. "Intelligent reconstruction and assembling of pipeline from point cloud data in smart plant 3D." Advances in Multimedia Information Processing--PCM 2015: 16th Pacific-Rim Conference on Multimedia, Gwangju, South Korea, September 16-18, 2015. Abstract, § 1 including: “In recent years the advancement in laser technology, incremented the demand for reconstruction of pipeline of an industrial plant from the laser-scanned point cloud data [1]. Conventional method of engendering 3D models of pipeline is being done utilizing commercial software’s such as Leica Cyclone [2],Smart Plant 3D [3] and AutoCAD Plant 3D. The data obtained from laser scan is sizably voluminous, manually detecting explicit geometric information available from point cloud data to reconstruct pipe-line is time consuming and intricate”
The claimed invention is directed towards an abstract idea of both a mathematical concept and a mental process without significantly more.
Regarding the dependent claims
Claims 2-3 are considered as further limiting the mere data gathering in a manner that is WURC in view of the evidence cited above for claim 1, including Czerniawski, Thomas. Updating digital models of existing commercial buildings using deep learning. Diss. 2020 and the other references cited above for the physical model generation step
Claim 4 is considered as further limiting the mental process, such as by the person performing the mental process mentally observing a “piping and instrumentation diagram”, a “map…”, or, on a printout from a computer or on a display of a computer, a 2D CAD model. Should it be found that these are not mental, then the Examiner submits that these would be generally linking to a particular technological environment and/or an insignificant extra-solution activity of selecting a particular data source or type of data to be manipulated, wherein these are WURC in view of the above discussed evidence with respect to the independent claims, including:
Son, Hyojoo, Changmin Kim, and Changwan Kim. "3D reconstruction of as-built industrial instrumentation models from laser-scan data and a 3D CAD database based on prior knowledge." Automation in Construction 49 (2015): 193-200. See § 2 ¶ 1: “Specifically, the pieces of equipment and valves in industrial plants have a relationship with neighboring pipelines. In the case of an industrial plant, such information can be derived from the piping and instrumentation diagrams (P&IDs). These diagrams, which are the overall documents used to define the industrial process, contain an instrumentation list, a pipeline list, and information about their functional interrelationships [26–28]” and see fig. 5(a) for a visual example
Martinez, Gerardo Santillan, et al. "Automatic generation of a simulation-based digital twin of an industrial process plant." IECON 2018-44th Annual Conference of the IEEE Industrial Electronics Society. IEEE, 2018. § I, ¶¶ 1-3 inclduing: “In particular, laborious FPM development is tackled by exploring the utilization of automatic model generation (AMG) methods. Existing AMG methods utilize data from engineering sources that are accessible already during the process design phase. These sources include piping and instrumentation diagrams (P&ID), equipment technical data sheets and control application programs [11]–[13].”, and see § III including: “AMG approaches use information mapping algorithms to generate a simulation model based on the targeted system information. These algorithms automatically map the accessed data into the model logic specified by the simulation language utilized [28]. There are a number of AMG methods available. These methods utilize different data sources for automatic model generation, including P&ID [12], [29], [30], 3D plant models [14] and control application programs [31].” – also see # 2-3, and see fig. 5 for an example “P&ID” diagram
Sierla, Seppo, et al. "Towards semi-automatic generation of a steady state digital twin of a brownfield process plant." Applied Sciences 10.19 (2020): 6959. § 2.3, ¶¶ 1-4 icnlduing: “…There are different available sources of information at process plants for the automatic generation of a digital display [11], such as datasheets, Process Flow Diagram (PFD) and Piping & Instrumentation Diagram (P&ID) diagrams, IO lists, 3D plant models and logic diagrams. The required information for simulation model creation can be extracted from these documents. The source information for the digital twin creation is not limited to design and engineering documents; for example, in [53] it was shown that a low fidelity digital twin has been generated automatically from high level requirements of the initial design phase of the project. Ref. [11] presents an automatic generation of simulation based digital twins for industrial process plants from 3D models. Sierla et al. [6] present an automatic solution to create the abstract graph model of the process system from a digital P&ID and a 3D CAD model of the system. This work was continued towards integrating the P&ID and CAD information by first converting the extracted information to the same level of abstraction [54]….” And see § 3.1 # 1: “The main process design document that is generally available at a brownfield process plant is a P&ID. Some leading P&ID CAD vendor’s tools are able to export P&IDs in a machine readable format according to the standardized Proteus XML schema, but this capability is present only in the most recent tool versions and is thus not applicable to brownfield plants”
Perez, Yeritza. Semantically-rich as-built 3D modeling of the built environment from point cloud data. Diss. University of Illinois at Urbana-Champaign, 2020. See § 1.1 as discussed above, then see pages 2-3 including: “ Also, it is common the facility contains different source of information (PDF, CAD, drawing, etc.) and attributes, making the extraction of information difficult [13].
Son, Hyojoo, Frédéric Bosché, and Changwan Kim. "As-built data acquisition and its use in production monitoring and automated layout of civil infrastructure: A survey." Advanced Engineering Informatics 29.2 (2015): 172-183. See page 173, col. 1, including: “Another important aspect of the construction, operation, and maintenance phases of civil infrastructure is automated layout. The Oxford English Dictionary defines layout as ‘‘the way in which the parts of something are arranged or laid out.’’ The Collins English Dictionary defines layout, in its technical sense, as ‘‘a drawing showing the relative disposition of parts in a machine, etc.’’ In this review, the term automated layout is used to mean the process of automatically determining geometric properties (dimensions, shape, and 3D position (location and orientation)) and other semantic (real-world) attributes of individual components of a structure, as well as the relationships between them, from 3D as-built data… Recording of information on the as-built status of individual components of a facility is needed, because the as-designed state, such as CAD drawings or early component selections made by the design team, may not correspond to the infrastructure actually produced.”, also see § 2.1.2.1 including ¶¶ 1-5
Claim 6 is further limiting the mental process as discussed above.
Claim 9 is rejected under a similar rationale as claim 4 above.
Claims 10-11 are rejected under a similar rationale as claims 2-3 as discussed above
Claims 14-15 are rejected under a similar rationale as claims 2-3 as discussed above
Claim 16 is rejected under a similar rationale as claim 5 as discussed above
Claims 18 is further limiting the mental process as discussed above.
Claim 19 is adding in the use of math relationships in textual form; wherein these are also considered part of the mental process. See the above discussion of a similar feature in claim 8 above
Claim 20 is considered as both a mental process step and further limiting the math concept for similar reasons as discussed with respect to a similar feature in claim 8 above
Claim 21 is considered as generally linking to particular fields of use/technological environments as this is merely specifying the type of data used, also considered as part of the mere instructions to do this on a computer – see the discussion of Intellectual Ventures I v. Capital One Fin. Corp in MPEP § 2106.05(f), including its discussion of in the “context of XML”; see the citation to “Intellectual Ventures I LLC v. Erie Indem. Co., 850 F.3d 1315, 1328-29, 121 USPQ2d 1928, 1939 (Fed. Cir. 2017) (limiting use of abstract idea to use with XML tags)” in MPEP § 2106.05(h)
Claim 22 is adding additional mental judgements/evaluations to the mental process, with an insignificant extra-solution activity to be performed of storing the vertex based on such a mental process, wherein storing is WURC in view of the above discussed evidence.
The claimed invention is directed towards an abstract idea of both a mathematical concept and a mental process without significantly more.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID A. HOPKINS whose telephone number is (571)272-0537. The examiner can normally be reached Monday to Friday, 10AM to 7 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ryan Pitaro can be reached at (571) 272-4071. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/David A Hopkins/Primary Examiner, Art Unit 2188