Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The Action is responsive to the Amendments and Remarks filed with the Request for Continued Examination on 4/1/2026. Claims 1, 3-4, 6-11, and 14-16 are pending claims. Claims 1, 15, and 16 are written in independent form. Claims 2, 5, and 12-13 have been cancelled.
Priority
Acknowledgment is made of a claim for priority as a continuation of PCT/JP2021/031985, filed 08/31/2021, which claims foreign priority to JP2020-167558, filed 10/2/2020, under 35 U.S.C. § 119(a)-(d) or (f), and is also acknowledged. Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Objections
Claims 1 and 14 are objected to because of the following informalities:
Claim 1 appears to recite a typographical error by first reciting “acquire…inspection record data created in a predetermined format in which text data of predetermined information is input…” and then reciting “analyze the text data of the inspection report to specify a member…, a position…, and a type…corresponding to the text data” and it is unclear which text data is being referred to by “…corresponding to the text data”. The language is being understood as intended to recite “…corresponding to the text data of the inspection report”.
Claim 14 appears to recite a typographical error by reciting “acquire…inspection record data created in a predetermined format in which text data of predetermined information is input for each of a plurality of inspection points of the structure” in Independent Claim 1 and “Wherein the inspection record data is created in the predetermined format in which the text data of information of the member of the structure, the damage to the structure, the position of the damage, and findings of the damage is input for each of the plurality of inspection points.” in Dependent Claim 14. The language of Dependent Claim 14 is being understood as intended to recite “Wherein the inspection record data is created in the predetermined format in which the text data of predetermined information of the member of the structure, the damage to the structure, the position of the damage, and findings of the damage is input for each of the plurality of inspection points.”
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-4, 6-11, and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Omansky et al. (U.S. Pre-Grant Publication No. 2014/0337286, hereinafter referred to as Omansky) and further in view of Liu (U.S. Pre-Grant Publication No. 2018/0025659).
Regarding Claim 1:
Omansky teaches an inspection support device for a structure comprising a processor (Para. [0029]) configured to:
Acquire three-dimensional model data of the structure, inspection data of the structure associated with the three-dimensional model data, and inspection record data created in a predetermined format in which text data of predetermined information is input for each of a plurality of inspection points of the structure,
Omansky teaches “Data structures and data sets of Building Information Model objects in the BIM Database Software Application 90 are transferred 92 to the Field Database Software Application 94. The unique data structures and data sets, related to one or may field operations, authored out in field, on the job site and at the point of construction, by construction field personnel, are then transferred 93 from the Field Database Software Application 94 back to the respective Building Information Model objects in the BIM Database Software Application 90.” (Para. [0049]) where “In the graphical user interface of the BIM Database Software Application 90, the data structures and data sets of Building Information Model objects are generally represented by visual or pictorial, three-dimensional (3D) shapes including building geometry and spatial relationships 91.” (Para. [0050]) and “The tabular format with text values in columns and rows 95 facilitates use by construction field personnel and generally lowers barriers to adoption. The tabular format is faster and easier to navigate, search and read through a large amount of information in a list format, filtered, grouped and sorted as required, than in a three-dimensional (3D) shapes format.” (Para. [0052]).
Omansky further teaches inspection data by teaching “the data structures and data sets relate to building assemblies, materials, systems, sub-systems, equipment, components, and inter-relations thereof in a building project. In other embodiments, the data structures and data sets relate to at least one field process selected from the group consisting of systems commissioning, issue tracking, field reports, materials tracking, safety, quality assurance, quality control, work lists, punch lists, and handover.” (Para. [0010])Therefore Omansky teaches acquiring data structures represented by visual or pictoral, 3d shapes, inspection data of the structure associated with the visual or pictoral 3d shapes, and inspection record data created in a predetermined format in which text data of predetermine dinformaiotn ins input for the inspection points, such as field reports, quality assurance, etc., and text data of the data sets related to the inspection work of the building structure.
Display the inspection record data on a display device;
Omansky teaches displaying the list of text data on a display device (Figure 5) where the text data is inspection record data by teaching “the data structures and data sets relate to building assemblies, materials, systems, sub-systems, equipment, components, and inter-relations thereof in a building project. In other embodiments, the data structures and data sets relate to at least one field process selected from the group consisting of systems commissioning, issue tracking, field reports, materials tracking, safety, quality assurance, quality control, work lists, punch lists, and handover.” (Para. [0010])
Analyze the text data of the inspection record to specify a member of the structure, a position of damage to the structure, and a type of the damage to the structure corresponding to the text data;
Omansky teaches 3d model data where “FIG. 9, which shows a representation of a user interface 120 for an example BIM database application with visualization of model objects from business rules and logic by the field database software application of the present invention” (Para. [0055]) and “the Status value 121 of each unique virtual model object in the Field database” (Para. [0055]) thereby teaching specifying the unique member of the structure, position of the structure, and corresponding text data in the Field database.
Omansky further teaches the structure data as being associated with damage to the structure by teaching “Exemplary documents include…emergency operating procedures, warranty, warranty guarantor, and spare parts lists” (Para. [0009]) which are all understood as information related to damage.
Display the portion of the three-dimensional model data and the portion of the inspection data on the display device; and
Omansky teaches “transmitting the data structures and data sets includes transmitting data at a level of detail required for the field operations or processes” (Claim 5) where “FIG. 5 is a drawing showing a representation of graphical user interfaces and data displays of an example BIM database application and of the field database software application of the present invention” (Para. [0017]) and Figs. 5, 6, and 9 depict displaying corresponding portions of the 3d model data and/or a screen of the extracted/relevant inspection data “at a level of detail required for the field operations or processes”.
Omansky explicitly teaches all of the elements of the claimed invention as recited above except:
Receive selection of an inspection record from the inspection record data;
Extract a portion of the three-dimensional model data corresponding to the member, the position of the damage, and the type of the damage;
Extract a portion of the inspection data associated with the portion of the three-dimensional model data;
Formulate a maintenance plan for the structure based on the extracted corresponding portion of the three-dimensional model data.
However, in the related field of endeavor of linking relevant text information with virtual models, Liu teaches the processor:
Receive selection of an inspection record from the inspection record data;
Liu teaches “a user input selecting a linked text displayed in the textual instruction section, identifying a component of the object in the 3D virtual model that is linked to the linked text, and highlighting, in response to identifying the component of the object in the 3D virtual model, the component of the object in the 3D virtual model displayed on display interface of the display device.” (Abstract).
extract a portion of the three-dimensional model data corresponding to the member, the position of the damage, and the type of the damage;
Liu teaches “identifying a component of the object in the 3D virtual model that is linked to the linked text, and highlighting, in response to identifying the component of the object in the 3D virtual model, the component of the object in the 3D virtual model displayed on display interface of the display device.” (Abstract). Therefore, Liu teaches extracting both the portion of linked text and the corresponding portion of the object in the 3D virtual model.It is noted that the extracting is not performed using the corresponding information, just that the extracted portion corresponds to the member, the position of the damage, and the type of the damage.
Extract a portion of the inspection data associated with the portion of the three-dimensional model data;
Liu teaches “identifying a component of the object in the 3D virtual model that is linked to the linked text, and highlighting, in response to identifying the component of the object in the 3D virtual model, the component of the object in the 3D virtual model displayed on display interface of the display device.” (Abstract). Therefore, Liu teaches extracting both the portion of linked text and the corresponding portion of the object in the 3D virtual model.
Formulate a maintenance plan for the structure based on the extracted corresponding portion of the three-dimensional model data.
Liu teaches “The display of the 3D model 506 of an object in contrast to a 2D model 504 of the object may facilitate tasks such as use of the object, maintenance or repair of the object, etc. by providing improved understanding of the object, for example, within the context of instructions provided along with the 2D model 504.” (Para. [0050]) thereby teaching using an extracted portion of a 3d model being displayed (“The display of the 3D model 506 of an object in contrast to a 2D model 504 of the object may facilitate tasks”) to plan and perform a task of “maintenance or repair of the object”.
Thus, it would have been obvious to one of ordinary skill in the art, having the teachings of Liu and Omansky at the time that the claimed invention was effectively filed, to have combined the highlighting feature of a particular component when selecting a particular piece of text, as taught by Liu, with the systems and methods for construction field management and operations with building information modeling, as taught by Omansky.
One would have been motivated to make such combination because Omansky teaches mapping data structures to data sets and corresponding text in the data sets (Paras. [0012], [0034], & Fig. Fig. 5) and Liu teaches “The method may also include rotating the object in the virtual model section 104 to provide an improved view of the component in response to a selection of a linked text in the textual instruction section 102 of the electronic manual 100. The method may also include providing a zoomed in view of the component in response to a selection of a linked text in the textual instruction section 102 of the electronic manual 100. The method may also include tilting the object to provide an improved view of the component in response to a selection of a linked text in the textual instruction section 102 of the electronic manual 100. The method may also include highlighting a linked text in the textual instruction section 102 in response to a selection of a component of the object 106 in the virtual model section 104.” (Para. [0054]) and it would have been obvious to a person having ordinary skill in the art that providing improved views of particular objects in the displayed visual would improve the user’s ability to review and inspect the details about the object of interest.
Regarding Claim 3:
Liu and Omansky further teach:
A memory configured to store the three-dimensional model data, the inspection data associated with the three-dimensional model data, and the inspection record data,
Omansky teaches “The unique data structures and data sets, related to one or may field operations, authored out in field, on the job site and at the point of construction, by construction field personnel, are then transferred from the Field Database 43 to the BIM Database 40 by the Integration Adapter 42 over the Internet 41, via either a wireless or wired connection. The unique data structures and data sets, related to one or may field operations, are associated with the unique Building Information Model objects representing building assemblies, systems, equipment and components.” (Para.[0044]).Omansky further explicitly teaches memory as part of teaching computer storage (Para. [0031]).
Wherein the processor acquires the three-dimensional model data, the inspection data, and the list of text data from the memory.
Omansky teaches “Data structures and data sets of Building Information Model objects in the BIM Database Software Application 90 are transferred 92 to the Field Database Software Application 94. The unique data structures and data sets, related to one or may field operations, authored out in field, on the job site and at the point of construction, by construction field personnel, are then transferred 93 from the Field Database Software Application 94 back to the respective Building Information Model objects in the BIM Database Software Application 90.” (Para. [0049]) where “In the graphical user interface of the BIM Database Software Application 90, the data structures and data sets of Building Information Model objects are generally represented by visual or pictorial, three-dimensional (3D) shapes including building geometry and spatial relationships 91.” (Para. [0050]).
Regarding Claim 4:
Liu and Omansky further teach wherein the processor is configured to:
Map the portion of the inspection data to the portion of the three-dimensional model data and
Omansky teaches “attributes from the Building Information Modeling software are mapped to the objects in the field database and are linked to the database” (Para. [0034]).
Omansky further teaches “Data structures and data sets of Building Information Model objects in the BIM Database Software Application 90 are transferred 92 to the Field Database Software Application 94. The unique data structures and data sets, related to one or may field operations, authored out in field, on the job site and at the point of construction, by construction field personnel, are then transferred 93 from the Field Database Software Application 94 back to the respective Building Information Model objects in the BIM Database Software Application 90.” (Para. [0049]) where “In the graphical user interface of the BIM Database Software Application 90, the data structures and data sets of Building Information Model objects are generally represented by visual or pictorial, three-dimensional (3D) shapes including building geometry and spatial relationships 91.” (Para. [0050]).
display the portion of the inspection data mapped to the portion of the three-dimensional model data on the display device.
Omansky teaches displaying the list of text data on a display device (Figure 5). Omansky further teaches Figure 10A as a screenshot, and thus is displaying, imported or received data structures and data sets (inspection data) which are “mapped to the objects in the field database and are linked to the database” (Paras. [0022] & [0034]). Omansky also teaches “In the graphical user interface of the BIM Database Software Application 90, the data structures and data sets of Building Information Model objects are generally represented by visual or pictorial, three-dimensional (3D) shapes including building geometry and spatial relationships 91.” (Para.[0050]) and Figure 10A also shows a three-dimensional model to which data is mapped being displayed.
Regarding Claim 6:
Liu and Omansky further teach:
Wherein the three-dimensional model data includes at least a member region of the structure and data of a member of the structure.
Omansky teaches 3d model data where “FIG. 9, which shows a representation of a user interface 120 for an example BIM database application with visualization of model objects from business rules and logic by the field database software application of the present invention” (Para. [0055]). Omansky further teaches “the Status value 121 of each unique virtual model object in the Field database” (Para. [0055]) thereby teaching a member region and data of the member.
Regarding Claim 7:
Liu and Omansky further teach:
Wherein the inspection data includes a plurality of types of data.
Omansky teaches a plurality of types of data by teaching “the invention selects and filters the data sets from the BIM Database 10 to the Field Database 13, via the Integration Adapter 12 and set of rules in XML or JSON, to include only data sets relevant to and material in value to the construction field operation, and to exclude data sets not relevant to and immaterial in value to the construction field operation. For example, in a Mechanical Commissioning field operation, the invention selects and filters only data sets related to the Mechanical Commissioning of Heating, Ventilating, and Air-Conditioning (HVAC) assemblies, systems, equipment and components, such as System Number, System Name, Equipment Number, Equipment Name, Manufacturer, Serial Number, Model Number, and Type, and not data sets related to building geometry, spatial relationships and geographic information. Different construction field operations require different data sets for proper execution and administration.” (Para. [0037]).
Regarding Claim 8:
Liu and Omansky further teach:
Wherein the plurality of types of data include
a captured image,
Omansky teaches “Exemplary documents include operations and maintenance manuals, shop drawings, test reports, construction drawings, "as-builts" or as-installed drawings, specifications, riser diagrams, P&ID piping and instrumentation diagrams, submittals, preventive maintenance plans, safety tasks, troubleshooting procedures, start-up procedures, shut-down procedures, emergency operating procedures, warranty, warranty guarantor, and spare parts lists.” (Para. [0009])
a panoramic composite image,
Omansky teaches “systems of the invention are able to deliver data structures and data sets generated in Building Information Modeling software (BIM) to construction field operations out in the field, on the job site, and at the point of construction. In FIG. 1 there is shown a BIM Level of Detail "pyramid" diagram. In the building construction phase, the invention only requires the Level of Detail with Approximate Geometry (Level of Detail 200) 31 as a minimum level of detail with approximate geometry, not precise geometry, in the Field Database for construction field operations, whereas the BIM Database requires varying levels of detail from Conceptual Level of Detail (Level of Detail 100) 30 to As-built Level of Detail (Level of Detail 500) 34 depending on the phase of the project.” (Para. [0025] & Fig. 1) thereby teaching a panoramic composite image as the “conceptual level of detail 100” when the term “panoramic composite image” is given its broadest reasonable interpretation.
damage information, and
Omansky teaches “In a Commissioning construction field operation example, the Status value 121 of each unique virtual model object in the Field database determines its color 122, 123, 124 in the BIM Database, thereby creating a virtual "heat map" or virtual "weather map" of the current status systems and equipment in the physical construction. In a Commissioning construction field operation example, if the current Status value=Functional Test, then the associated model object renders in a green color. If the current Status value=Pre-Functional Test, then the associated model object renders in a blue color. If the current Status value=Powered Up, then the associated model object renders in a red color. If the current Status value=null or no value, then the associated model object renders in a grey color.” (Para. [0055]).Omansky further teaches “exemplary documents include…emergency operating procedures, warranty, warranty guarantor, and spare parts lists” (Para. [0009]) which are all understood as information related to damage.
a two-dimensional drawing.
Omansky teaches “Exemplary documents include operations and maintenance manuals, shop drawings, test reports, construction drawings, "as-builts" or as-installed drawings, specifications, riser diagrams, P&ID piping and instrumentation diagrams, submittals, preventive maintenance plans, safety tasks, troubleshooting procedures, start-up procedures, shut-down procedures, emergency operating procedures, warranty, warranty guarantor, and spare parts lists.” (Para. [0009]).
Regarding Claim 9:
Liu and Omansky further teach wherein the processor:
Displays at least one type of data on the display device from the plurality of types of data included in the inspection.
Omansky teaches a plurality of types of data by teaching “the invention selects and filters the data sets from the BIM Database 10 to the Field Database 13, via the Integration Adapter 12 and set of rules in XML or JSON, to include only data sets relevant to and material in value to the construction field operation, and to exclude data sets not relevant to and immaterial in value to the construction field operation. For example, in a Mechanical Commissioning field operation, the invention selects and filters only data sets related to the Mechanical Commissioning of Heating, Ventilating, and Air-Conditioning (HVAC) assemblies, systems, equipment and components, such as System Number, System Name, Equipment Number, Equipment Name, Manufacturer, Serial Number, Model Number, and Type, and not data sets related to building geometry, spatial relationships and geographic information. Different construction field operations require different data sets for proper execution and administration.” (Para. [0037]).
Regarding Claim 10:
Liu and Omansky further teach:
Wherein the inspection data includes a plurality of captured images; and
Omansky teaches “Exemplary documents include operations and maintenance manuals, shop drawings, test reports, construction drawings, "as-builts" or as-installed drawings,” (Para. [0009]) thereby teaching capturing a plurality of documents as images in the form of drawings.
The processor is configured to display the captured image satisfying a condition on the display device from the plurality of captured images to be displayed.
Omansky teaches “Exemplary documents include operations and maintenance manuals, shop drawings, test reports, construction drawings, "as-builts" or as-installed drawings,” (Para. [0009]) thereby teaching capturing a plurality of documents as images in the form of drawings, where the drawings are “‘as-builts’ or as-installed drawings” thus satisfying a condition of installation/built.Omansky further teaches “data to be added/updated to the model” including properties such as installation information such as “install date” and “install complete” (Paras. [0068] – [0069]) thereby further supporting the teaching of capturing and displaying information satisfying a condition.
Regarding Claim 11:
Liu and Omansky further teaches wherein the processor is configured to:
Analyze the text data of the inspection record to extract past inspection data corresponding to the member, the position of the damage, and the type of the damage; and
Liu teaches “identifying a component of the object in the 3D virtual model that is linked to the linked text, and highlighting, in response to identifying the component of the object in the 3D virtual model, the component of the object in the 3D virtual model displayed on display interface of the display device.” (Abstract).
Omansky teaches “transmitting the data structures and data sets includes transmitting data at a level of detail required for the field operations or processes” (Claim 5) and “The invention selects and filters the data structures of building assemblies, systems, equipment and components from the BIM Database 10 to the Field Database 13, via the Integration Adapter 12 and set of rules in XML or JSON, to include only data structures relevant to and material in value to the construction field operation, and to exclude data structures not relevant to and immaterial in value to the construction field operation” (Para. [0036]).Omansky further teaches past inspection data by teaching “associate documents or electronic links to documents with the selected data structures and data sets. Exemplary documents include…test reports” (Para. [0009]) and “the data structures and data sets relate to building assemblies, materials, systems, sub-systems, equipment, components, and inter-relations thereof in a building project. In other embodiments, the data structures and data sets relate to…issue tracking, field reports, materials tracking, safety, quality assurance, quality control, work lists, punch lists, and handover.” (Para. [0010]).
Display the past inspection data on the display device.
Omansky teaches “transmitting the data structures and data sets includes transmitting data at a level of detail required for the field operations or processes” (Claim 5) where “FIG. 5 is a drawing showing a representation of graphical user interfaces and data displays of an example BIM database application and of the field database software application of the present invention” (Para. [0017]) and Figs. 5, 6, and 9 depict displaying corresponding portions of the 3d model data and/or a screen of the extracted/relevant inspection data “at a level of detail required for the field operations or processes”.
Regarding Claim 14:
Liu and Omansky further teach:
Wherein the inspection record data is created in the predetermined format in which the text data of information of the member of the structure, the damage to the structure, the position of the damage, and findings of the damage is input for each of the plurality of inspection points.
Omansky teaches the data being created in a predetermined format by teaching “In the graphical user interface (QUI) of the Field Database Software Application 94, the unique data structures and data sets, related to one or may field operations, and associated with respective Building Information Model objects are generally represented in a tabular format with text values in columns and rows 95 or a list view. Building Information Model data structures and data sets are translated from shapes 91 to text 95, retaining the hierarchical relationships of the Building Information Model. Then, text and text-based values 95 are transferred back 93 to the associated objects in the Building Information Model.” (Para. [0050]) where “The tabular format with text values in columns and rows 95 facilitates use by construction field personnel and generally lowers barriers to adoption. The tabular format is faster and easier to navigate, search and read through a large amount of information in a list format, filtered, grouped and sorted as required, than in a three-dimensional (3D) shapes format.” (Para. [0052]).
Regarding Claim 15:
Liu and Omansky further teach the processor is configured to:
Acquire a list of text data of the plurality of inspection points related to an inspection work of the structure,
Omansky teaches “Data structures and data sets of Building Information Model objects in the BIM Database Software Application 90 are transferred 92 to the Field Database Software Application 94. The unique data structures and data sets, related to one or may field operations, authored out in field, on the job site and at the point of construction, by construction field personnel, are then transferred 93 from the Field Database Software Application 94 back to the respective Building Information Model objects in the BIM Database Software Application 90.” (Para. [0049]) where “In the graphical user interface of the BIM Database Software Application 90, the data structures and data sets of Building Information Model objects are generally represented by visual or pictorial, three-dimensional (3D) shapes including building geometry and spatial relationships 91.” (Para. [0050]).Omansky further teaches inspection data by teaching “the data structures and data sets relate to building assemblies, materials, systems, sub-systems, equipment, components, and inter-relations thereof in a building project. In other embodiments, the data structures and data sets relate to at least one field process selected from the group consisting of systems commissioning, issue tracking, field reports, materials tracking, safety, quality assurance, quality control, work lists, punch lists, and handover.” (Para. [0010])Therefore Omansky teaches acquiring data structures represented by visual or pictoral, 3d shapes, inspection data sets such as field reports, quality assurance, etc., and text data of the data sets related to the inspection work of the building structure.
Wherein the inspection data include:
a captured image obtained by imaging the structure, or
Omansky teaches “Exemplary documents include operations and maintenance manuals, shop drawings, test reports, construction drawings, "as-builts" or as-installed drawings,” (Para. [0009]) thereby teaching capturing a plurality of documents as captured images in the form of drawings.
the captured image and a damage detection result image in which a damage detected from the captured image is expressed;
Omansky teaches “Exemplary documents include…emergency operating procedures, warranty, warranty guarantor, and spare parts lists” (Para. [0009]) which are all understood as information related to damage.
Display the list of text data on the display device;
Omansky teaches displaying the list of text data on a display device (Figure 5).
Receive selection of a text data of at least one of the plurality of inspection points from the list of text data;
Liu teaches “a user input selecting a linked text displayed in the textual instruction section, identifying a component of the object in the 3D virtual model that is linked to the linked text, and highlighting, in response to identifying the component of the object in the 3D virtual model, the component of the object in the 3D virtual model displayed on display interface of the display device.” (Abstract)
Analyze the text data to extract the inspection data corresponding to the text data of the at least one of the plurality of inspection points;
Liu teaches “identifying a component of the object in the 3D virtual model that is linked to the linked text, and highlighting, in response to identifying the component of the object in the 3D virtual model, the component of the object in the 3D virtual model displayed on display interface of the display device.” (Abstract).
Display, on the display device, as the inspection data, the three-dimensional model data to which:
the captured image is mapped or
Omansky teaches Figure 10A as a screenshot, and thus is displaying, imported or received data structures and data sets (inspection data) which are “mapped to the objects in the field database and are linked to the database” (Paras. [0022] & [0034]). Omansky further teaches “In the graphical user interface of the BIM Database Software Application 90, the data structures and data sets of Building Information Model objects are generally represented by visual or pictorial, three-dimensional (3D) shapes including building geometry and spatial relationships 91.” (Para.[0050]) and Figure 10A also shows a three-dimensional model to which data is mapped being displayed.It is noted that Omansky teaches “Exemplary documents include operations and maintenance manuals, shop drawings, test reports, construction drawings, "as-builts" or as-installed drawings,” (Para. [0009]) thereby teaching capturing a plurality of documents as captured images that can be mapped and displayed as mapped inspection data.
the captured image and the damage detection result image are mapped.
Omansky teaches Figure 10A as a screenshot, and thus is displaying, imported or received data structures and data sets (inspection data) which are “mapped to the objects in the field database and are linked to the database” (Paras. [0022] & [0034]). Omansky further teaches “In the graphical user interface of the BIM Database Software Application 90, the data structures and data sets of Building Information Model objects are generally represented by visual or pictorial, three-dimensional (3D) shapes including building geometry and spatial relationships 91.” (Para.[0050]) and Figure 10A also shows a three-dimensional model to which data is mapped being displayed.It is noted that Omansky teaches ““Exemplary documents include operations and maintenance manuals, shop drawings, test reports, construction drawings, "as-builts" or as-installed drawings,…emergency operating procedures, warranty, warranty guarantor, and spare parts lists” (Para. [0009]) which are all understood as information related to damage, and therefore teaches capturing a plurality of documents as captured images and damage detection result images that can be mapped and displayed as mapped inspection data.
Claim(s) 16 is rejected under 35 U.S.C. 103 as being unpatentable over Omansky and Liu, and further in view of (U.S. Pre-Grant Publication No. 2018/0025659).
Regarding Claim 16:
Some of the limitations herein are similar to some or all of the limitations of Claim 15.
Liu and Omansky explicitly teach all of the elements of the claimed invention as recited above except:
Analyze the text data to determine whether the text data include text data related to progressiveness of damage; and
In response to determining that the text data include the text data related to the progressiveness of the damage, extract past inspection data corresponding to the at least one of the plurality of inspection points, and display the past inspection data on the display device.
However, in the related field of endeavor of inspection result retrieval, Yamagishi teaches:
Analyze the text data to determine whether the text data include text data related to progressiveness of damage; and
Yamagishi teaches “Accordingly, changes over time in an inspection portion of the construction 9 and changes over time in the damage D that has occurred in the inspection portion can be checked.” (Para. [0109]).
In response to determining that the text data include the text data related to the progressiveness of the damage, extract past inspection data corresponding to the at least one of the plurality of inspection points, and display the past inspection data on the display device.
Yamagishi teaches “the inspection result retrieval device 10A according to the second embodiment performs image processing for matching the shape and position of the construction 9 in the inspection image data 27 included in the first inspection result 13 and those of the construction 9 in the inspection image data 27 included in each of the second inspection results 17, the first inspection result 13 and the second inspection results 17 being the results of inspection of the same construction 9, and thereafter, causes the display unit 20 to perform display so as to enable a comparison.” (Para. [0109]).Yamagishi further teaches “in a case where a user selects the second inspection result 17 (including the specific inspection result 17A in the fifth embodiment) that is a desired one on the retrieval result display screen 38 (see FIG. 7) displayed on the display unit 20, the inspection history of the construction 9 (hereinafter referred to as the specific construction 9) that is the inspection target in the selected second inspection result 17 is displayed on the display unit 20.” (Para. [0129]).
Thus, it would have been obvious to one of ordinary skill in the art, having the teachings of Yamagishi, Liu, and Omansky at the time that the claimed invention was effectively filed, to have modified the highlighting feature of a particular component when selecting a particular piece of text, as taught by Liu, and the systems and methods for construction field management and operations with building information modeling, as taught by Omansky, with the tracking of changes in the damage of the construction over time in an inspection portion of the construction, as taught by Yamagishi.
One would have been motivated to make such modification because Yamagishi teaches tracking damage history and predicting the development of damage such that “ the severity of the damage D that has occurred in the construction 9 can be determined, and furthermore, the time when the damage D of the construction 9 needs to be repaired can be appropriately determined.” (Para. [0093]) and it would be obvious to a person having ordinary skill in the art that knowing the time when damaged components need to be repaired would improve the ability to manage the structure and proactively schedule repairs as well as order any necessary replacement parts in advance.
Response to Amendment
Applicant’s Amendments, filed on 4/1/2026, are acknowledged and accepted.
Response to Arguments
On page 6 of the Remarks filed on 4/1/2026, Applicant states that “On page 2 of the Office Action, the rejection of claim 1 is based on the assertion that claim 1 recites "acquire ... inspection record data created in a predetermined format in which text data of predetermined information is input ... " and then reciting "analyze the text data of the inspection report to specify a member ... , a position ... , and a type ... corresponding to the text data" (emphasis in original). Applicant respectfully submits that claim 1 recites "text data of predetermined information ... selection of the text data ... the selected text data ... the selected text data." Accordingly, claim 1 does not recite the limitations identified in the Office Action, and therefore Applicant respectfully submits that claim 1 is clear.”Applicant’s argument is not convincing to overcome the claim objection because they don’t seem to be referring to language in the claims filed on 4/1/2026. The claims do not recite at least “the selected text data” as is being argued, but instead only recite any form of selecting in the limitation: “receiving selection of an inspection record from the inspection record data”.It is further noted that the claim 1 was objected to, not rejected, as Applicant states in the remarks.
On page 6 of the Remarks filed on 4/1/2026, Applicant states that “Claim 14 recites "wherein the inspection record data is created in the predetermined format in which the text data of information of a member, damage, a position of the damage and findings of the damage is input for each of the inspection points." Accordingly, Applicant respectfully submits that the text data of claim 14 is also clear.”Applicant’s argument is not convincing to overcome the claim objection because it is unclear whether the “information” of “the text data of information” recited in dependent claim 14 is the same as the “predetermined information” of “text data of predetermined information” recited in independent claim 1, upon which Claim 14 depends.
In light of the Amendments and Remarks filed on 4/1/2026 and further review of the Application’s specification, the 101 rejection of claims 1, 3-4, 6-11, and 14-16 for being directed to an abstract idea without significantly more has been withdrawn. In particular, Applicant convincingly argues on Pages 6-10 of the Remarks that the claims “amended claim 1 is patent-eligible because amended claim 1 recites limitations that ‘reflect[ ] the improvement disclosed in the specification ... the claim includes the components or steps of the invention that provide the improvement described in the specification’ (USPTO Memorandum, Charles Kim, Deputy Commissioner for Patents, ‘Advance notice of change to the MPEP in light of Ex Parte Desjardins’ at p. 2, December 5, 2025).” (Remarks Page 7) where “Claim 1 recites limitations that ‘reflect [ ] the improvement disclosed in the specification’ (USPTO Memorandum, Charles Kim, Deputy Commissioner for Patents, ‘Advance notice of change to the MPEP in light of Ex Parte Desjardins’ at p. 2, December 5, 2025), including ‘analyze the selected text data of the at least one of the inspection points to specify a member, a position and damage of the at least one of the inspection points; extract, according to information of the specified member, position and damage of the at least one of the inspection points, a corresponding portion of the three-dimensional model data corresponding to the selected text data of the at least one of the inspection points.’” and “amended claim 1 recites to "formulate a maintenance plan for the structure based on the extracted corresponding portion of the three-dimensional model data." Applicant respectfully submits that such a limitation constitutes a practical application of the improvement disclosed in the Specification and recited in claim 1, as discussed above. See e.g., 84 Fed. Reg. 4 at 54-55” (Remarks Page 10).In the Remarks, Applicant specifically cites Paragraphs [0003], [0006], [0049], [0052]-[0053], and [0055]-[0057] and Figures 2, 7, and 9 of the Specification for support related to the 35 U.S.C. 101 analysis argument, and that “the amendments are supported by at least paragraph 3 of the Specification”.
On page 11 of the Remarks filed on 4/1/2026, Applicant argues that “the combination of Omansky and Liu fails to disclose, suggest, or otherwise render obvious the combination of limitations recited in amended claim 1, including to ‘formulate a maintenance plan for the structure based on the extracted corresponding portion of the three-dimensional model data.’” because “Omansky may appear to focus on construction field management, specifically transferring data structures and sets (like field reports or quality control) between a BIM database and a field database. While Omansky may appear to mention "maintenance manuals" and "operations," any such description is limited to the context of stored documents, not the active formulation of a plan based on extracted 3D model portions. Liu appears to be directed toward an electronic manual that cross-links text instructions with 3D virtual models to highlight components. Applicant respectfully submits that Lui does not, however, address the formulation of maintenance plans or any logic related to structure maintenance scheduling or strategy.”Upon further time spent for search and consideration, the amendments and remarks were not found to overcome the previously cited prior art because Liu was found to teach the newly added limitation as is addressed in the rejection above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Bussey et al. (U.S. Patent No. 8,319,792) teaches composing a computer-aided design (CAD) drawing that includes modeled components and non-modeled components. The method includes receiving a selection of properties for a non-modeled component to include in the CAD drawing, where the non-modeled component defines an element of an object modeled in the CAD drawing that is not included in a display representation of the object including an instance of the non-modeled component in the CAD drawing. Although not shown in display representations, the non-modeled elements may be displayed in a browse view showing the elements included in a CAD drawing, and in a bill of materials generated from the CAD drawing.The reference further teaches “Illustratively, 3D model 120 includes 3D model parts 122, virtual components 124, and part associations 126. 3D model parts 122 are elements of a mechanical design that are displayed in a 3D model view 112. Virtual components 124 are elements of the mechanical design represented by 3D model 120 that are not displayed in 3D model view 112. Virtual components 124 may; however, be included in displays generated by a model browser tool 114 and in BOMs generated using a BOM tool 116. Part associations 126 enable 3D model parts 122 to be associated with an associative virtual component, as appropriate in a given case. In other words, part associations 126 specify what virtual components 124 are associated with a given 3D model part. Further, virtual components 122 may have properties that describe aspects of the real-world object modeled by the virtual component. For example, properties such as mass and volume may be included in the properties of a virtual component. Such proprieties may be useful for engineering calculations of the expected weight, center of gravity, mass, volume and other physical properties of the real-world object modeled by 3D model 120.” (Col. 4 Lines 12-32).
Basile et al. (U.S. Patent No. 6,725,097) teaches assisting an operator in surveying defects on a building site which consists in providing him with a portable terminal displaying images of the works and inputting data concerning his observations, under the control of a software operating with a database of the works, comprising files defining the images displayed on the terminal graphic interface, data structures relating respectively to types of defects liable to occur in the works, and if any, descriptive data of defects previously observed on the works, organized in accordance with the associated data structures. In response to the selection of a type of fault by the operator, said software activates a dialogue, by means of the associated data structure, to input parameters describing said defects, which are used to constitute a representation of the defect on the image displayed on the graphic interface and to update the data describing the defects.
Mildrew et al. (U.S. Patent No. 10,139,985) teaches defining, displaying and interacting with tags in a 3D model. In an embodiment, a method includes generating, by a system including a processor, a three-dimensional model of an environment based on sets of aligned three-dimensional data captured from the environment, and associating tags with defined locations of the three-dimensional model, wherein the tags are respectively represented by tag icons that are spatially aligned with the defined locations of the three-dimensional model as included in different representations of the three-dimensional model rendered via an interface of a device, wherein the different representations correspond to different perspectives of the three-dimensional model, and wherein selection of the tag icons causes the tags respectively associated therewith to be rendered at the device.The reference further teaches “a tag entry for a point of interest in a 3D model may provide any amount of textual information associated with the point of interest, such as title, byline, description, details, summary or report. In another example, a tag associated with a point of interest can identify any number of images associated with the point of interest, including photorealistic and panorama images. In another example, a tag associated with a point of interest can include audio associated with the point of interest, a video associated the point of interest, 360° panorama associated with the point of interest, or a combination of image, video and audio associated with the point of interest. In another example, a tag associated with a point of interest can include a hyperlink to external websites, as indicated by a uniform resource locator (URL) or uniform resource identifier (URI)” (Col. 22 Lines 24-38).
Stanic (U.S. Pre-Grant Publication No. 2015/0186472) teaches an automated process collects and organizes field data from an inspection of a building or other structure such as pipe supports, bridges, buildings, over head supports, and smoke stacks.
Connary et al. (U.S. Pre-Grant Publication No. 2022/0130145) teaches improved methodologies for visualization on a user's display of sensor data (e.g., 2D and 3D information obtained from or derived from sensors) for objects, components, or features of interest in a scene. The previously acquired sensor data is processable for concurrent display of objects/features/scene or location visualizations to a user during their real-time navigation of a scene camera during a variety of user visualization activities. Sensor data can be acquired via the operation of vehicles configured with one or more sensors, such as unmanned aerial vehicles, or from other methodologies, or from any other suitable sensor data acquisition activities. Objects etc. for which acquired sensor data can be visualized by a user on a display includes buildings, parts of buildings, and infrastructure elements, among other things. The improved display of information to a user for visualization and information generation therefrom provides significant benefits over prior art display methodologies and exhibits notable utility for user activities such as, inspection, condition assessment, performance assessment, insurance applications, construction, inventorying, building information modeling, asset management and the like. Information derivable from the methodologies herein can be used for machine learning libraries and digital twin processes.The reference further teaches “In an example relating to the commercial roof illustrated previously, an appearance of a pool of water on the roof can be modeled using a digital twin of that roof to determine the effect of a coming storm—or more broadly, the predicted rainfall for the location for the upcoming year—to generate a plan for conducting maintenance or repair activities associated with the roof. When a plan is generated, the processes can operate automatically in the background to monitor the plan in relation to the amount of rainfall that was predicted and that which was used to generate the maintenance plan versus the actual amount of rainfall. Moreover, from time to time, additional imagery of the roof can be generated and processed according to the methodology herein to provide updated roof condition information related to the water pooling, for example. The maintenance or repair plan can be updated or modified for that roof. Other maintenance and repair plans for other roofs at other locations can also be automatically updated.” (Para. [0225]).
Foreign Publication CN 111340948A teaches a BIM-based operation and maintenance system for a water purification plant. The system includes: a physical pipe network layer, a transmission perception layer and a three-dimensional display layer; the physical pipe network layer includes water treatment such as pipes, pumps, valves, pools, etc. equipment/facilities; the transmission perception layer is used to detect the production data of the water treatment equipment in the physical pipe network layer through the detection equipment; the three-dimensional display layer is used to detect the production data of the water purification plant according to the production data The three-dimensional BIM model of the water purification plant, the equipment data of the water purification plant, and the operation information of the water purification plant are generated to generate a virtual digital operation and maintenance model of the water purification plant, so as to operate and maintain the water purification plant according to the virtual digital operation and maintenance model. The BIM-based water purification plant operation and maintenance system of the present invention can assist in reducing the operation cost of the water purification plant and improve the production efficiency and economic benefit of the water purification plant.The reference further teaches “The above-described file management system generates predictive maintenance plans and predictive maintenance plans to achieve intelligent management of water purification plant equipment.” (Page 3 Paragraph 18) and “The equipment maintenance management module is used for planned maintenance, declaration maintenance, fault maintenance, maintenance work order and approval, maintenance organization (department or personnel), maintenance plan, safety measures, maintenance acceptance, maintenance cost, maintenance resources, maintenance statistics, maintenance effect/ Parts replacement record filing, etc. It can be divided into different levels such as daily maintenance, preventive maintenance, compulsory maintenance (based on the operating time), fault maintenance, and overhaul. The system should be able to automatically sequence maintenance plans according to different levels. The equipment maintenance management module establishes corresponding maintenance plans for different equipment, and can automatically generate maintenance work orders.” (Page 6 Section “4. Equipment maintenance management module”).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT F MAY whose telephone number is (571)272-3195. The examiner can normally be reached Monday-Friday 9:30am to 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Boris Gorney can be reached at 571-270-5626. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT F MAY/Examiner, Art Unit 2154 4/18/2026
/BORIS GORNEY/Supervisory Patent Examiner, Art Unit 2154
Prosecution Insights