DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Status of the Claims
The pending claims in the present application are original claims 1-10 and 12-21 of the "REPLY TO NON-FINAL OFFICE ACTION" of 24 August 2026 (hereinafter referred to as the "Reply").
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-10 and 12-21 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The paragraphs below provide rationales for the rejection. The rationales are based on the multi-step subject matter eligibility test outlined in MPEP 2106.
Step 1 of the eligibility analysis involves determining whether a claim falls within one of the four enumerated categories of patentable subject matter recited in 35 USC 101. (See MPEP 2106.03(I).) That is, Step 1 asks whether a claim is to a process, machine, manufacture, or composition of matter. (See MPEP 2106.03(II).) The “method” of claims 1-10, 12, and 21 constitutes a process under 35 USC 101, the “system” of claims 13-18 constitutes a machine under the statute, and the “non-transitory computer readable media” of claims 19 and 20 constitutes a manufacture under the statute. Accordingly, claims 1-10 and 12-21 meet the criteria of Step 1 of the eligibility analysis. The claims, however, fail to meet the criteria of subsequent steps of the eligibility analysis, as explained in the paragraphs below.
The next step of the eligibility analysis, Step 2A, involves determining whether a claim is directed to a judicial exception. (See MPEP 2106.04(II).) This step asks whether a claim is directed to a law of nature, a natural phenomenon (product of nature) or an abstract idea. (See id.) Step 2A is a two-prong inquiry. (See MPEP 2106.04(II)(A).) Prong One and Prong Two are addressed below.
In the context of Step 2A of the eligibility analysis, Prong One asks whether a claim recites an abstract idea, law of nature, or natural phenomenon. (See MPEP 2106.04(II)(A)(1).) Using independent claim 1 as an example, the claim recites the following abstract idea limitations:
“A ... method for ... tracking construction progress, the ... method comprising: ...” - See below regarding MPEP 2106.04(a), mental processes
“... receiving ... a set of processed imaging data comprising one or more images of a construction site, wherein the one or more images have been captured ...” - See below regarding MPEP 2106.04(a), mental processes
“... processing ... the one or more images to identify a status at the construction site of (i) at least one physical structure or (ii) at least one site condition, wherein the processing comprises: ...” - See below regarding MPEP 2106.04(a), mental processes
“... performing feature extraction to generate one or more structured data representations of the one or more images; ...” - See below regarding MPEP 2106.04(a), mental processes
“... identifying an object within the one or more structured data representations indicative of a status at the construction site; ...” - See below regarding MPEP 2106.04(a), mental processes
“... classifying the object ... to determine a semantic category of the status; and ...” - See below regarding MPEP 2106.04(a), mental processes
"... identifying, by applying heuristics data to the classified object ..., completion of one or more prerequisite construction elements associated with the object, wherein the one or more prerequisite construction elements are obscured by the object within the one or more images; and ..." - See below regarding MPEP 2106.04(a), mental processes
“... generating a textual description that i) characterizes the identified status at the construction site based on at least the semantic category of the status and ii) indicates completion of the one or more prerequisite construction elements associated with the object.” - See below regarding MPEP 2106.04(a), mental processes
The above-listed limitations of claim 1, when applying their broadest reasonable interpretations in light of their context in the claim as a whole, fall under enumerated groupings of abstract ideas outlined in MPEP 2106.04(a). For example, limitations of the claim can be characterized as concepts performed in the human mind, including observation (e.g., the recited “receiving” step), and evaluation, judgment, and/or opinion (e.g., the recited “processing,” “performing,” “identifying,” “classifying,” "identifying," and “generating steps) which fall under the mental processes grouping of abstract ideas (see MPEP 2106.04(a)). Accordingly, for at least these reasons, claim 1 fails to meet the criteria of Step 2A, Prong One of the eligibility analysis.
In the context of Step 2A of the eligibility analysis, Prong Two asks if the claim recites additional elements that integrate the judicial exception into a practical application. (See MPEP 2106.04(II)(A)(2).) Continuing to use independent claim 1 as an example, the claim recites the following additional element limitations:
The claimed “method” is “computer-implemented” and occurs “automatically” - See below regarding MPEP 2106.05(a)-(c) and (f)-(h)
The claimed “receiving” is “at one or more processors communicatively coupled to one or more image sensors” - See below regarding MPEP 2106.05(a)-(c) and (f)-(h)
The claimed “captured” is “by the one or more image sensors each equipped with an imaging system” - See below regarding MPEP 2106.05(a)-(c) and (f)-(h)
The claimed “processing” is “using a vision large language model (LLM) executed by the one or more processors” - See below regarding MPEP 2106.05(a)-(c) and (f)-(h)
The claimed “classifying” is “using a transformer-based neural network” - See below regarding MPEP 2106.05(a)-(c) and (f)-(h)
The above-listed additional element limitations of claim 1, when applying their broadest reasonable interpretations in light of their context in the claim as a whole, are analogous to: accelerating a process of analyzing audit log data when the increased speed comes solely from the capabilities of a general-purpose computer, and mere automation of manual processes, which courts have indicated may not be sufficient to show an improvement in computer-functionality (see MPEP 2106.05(a)(I)); a commonplace business method being applied on a general purpose computer, and selecting a particular generic function for computer hardware to perform from within a range of fundamental or commonplace functions performed by the hardware, which courts have indicated may not be sufficient to show an improvement to technology (see MPEP 2106.05(a)(II)); a general purpose computer that applies a judicial exception, such as an abstract idea, by use of conventional computer functions, and merely adding a generic computer, generic computer components, or a programmed computer to perform generic computer functions, which do not qualify as a particular machine or use thereof (see MPEP 2106.05(b)(I)); a machine that is merely an object on which the method operates, which does not integrate the exception into a practical application (see MPEP 2106.05(b)(II)); use of a machine that contributes only nominally or insignificantly to the execution of the claimed method, which does not integrate a judicial exception (see MPEP 2106.05(b)(III)); transformation of an intangible concept such as a contractual obligation or mental judgment, which is not likely to provide significantly more (see MPEP 2106.05(c)); use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea, a commonplace business method or mathematical algorithm being applied on a general purpose computer, and requiring the use of software to tailor information and provide it to the user on a generic computer, which courts have found to be mere instructions to apply an exception, because they do no more than merely invoke computers or machinery as a tool to perform an existing process (see MPEP 2106.05(f)); mere data gathering in the form of obtaining information about transactions using the Internet to verify transactions and consulting and updating an activity log, which courts have found to be insignificant extra-solution activity (see MPEP 2106.05(g)); and specifying that the abstract idea of monitoring audit log data relates to transactions or activities that are executed in a computer environment, because this requirement merely limits the claims to the computer field, i.e., to execution on a generic computer, which courts have described as merely indicating a field of use or technological environment in which to apply a judicial exception (see MPEP 2106.05(h)). For at least these reasons, claim 1 fails to meet the criteria of Step 2A, Prong Two of the eligibility analysis.
The next step of the eligibility analysis, Step 2B, asks whether a claim recites additional elements that amount to significantly more than the judicial exception. (See MPEP 2106.05(II).) The step involves identifying whether there are any additional elements in the claim beyond the judicial exceptions, and evaluating those additional elements individually and in combination to determine whether they contribute an inventive concept. (See id.) The ineligibility rationales applied at Step 2A, Prong Two, also apply to Step 2B. (See id.) For all of the reasons covered in the analysis performed at Step 2A, Prong Two, independent claim 1 fails to meet the criteria of Step 2B. As a result, claim 1 is rejected under 35 USC 101 as ineligible for patenting.
Regarding claims 2-10, 12, and 21 the claims depend from independent claim 1, and expand upon limitations introduced by claim 1. The dependent claims are rejected at least for the same reasons as claim 1. For example, the dependent claims recite abstract idea elements similar to the abstract idea elements of claim 1, that fall under the same abstract idea groupings as the abstract idea elements of claim 1 (e.g., the “wherein the one or more images comprise one or more first images that have been captured ... at a first time and one or more second images that have been captured ... at a second time, and wherein the identified object within the one or more structured data representations is a modification between the one or more first images and the one or more second images, and the status at the construction site is a change at the construction site indicated by the modification” of claim 2, the “wherein the one or more images each include associated metadata, and the one or more structured data representations are generated further based on the metadata associated with the one or more images” of claim 3, the “wherein the set of processed imaging data is based on one or more source files that have had at least one of feature extraction, semantic segmentation, or edge detection applied” of claim 4, the “wherein the set of processed imaging data includes metadata comprising at least one of timestamp data, geolocation data, sensor data, lens data, photographic setting data, sensor calibration data, or sensor angle data, and wherein the identifying of the status at the construction site is further based on the metadata” of claim 5, the “wherein the textual description includes at least one standardized descriptive term” of claim 6, the “further comprising: transmitting ... a scheduling request ...; receiving ... a scheduling response including scheduling data ...; and ... generate a progress report, wherein the progress report is based on at least one of the at least one description or the scheduling data” of claim 7, the “generate an analysis of the construction site, wherein the analysis is based on the progress report” of claim 8, the “further comprising: receiving ... a data inquiry ..., wherein the data inquiry includes at least one reference to a portion of the construction site” of claim 9, the “further comprising: ... generate a data response, wherein the data response is based on at least one of the progress report, the scheduling data, or the data inquiry” of claim 10, the “wherein the progress report is a first progress report, wherein the first progress report is further based on a second progress report generated” of claim 12, and the "wherein the one or more images comprise a 2:1 equirectangular image and wherein the processing further comprises: preventing a warping error in a portion of the image by converting the 2:1 equirectangular to a cubic project prior to analysis" of claim 21). The dependent claims recite further additional elements that are similar to the additional elements of claim 1, that fail to warrant eligibility for the same reasons as the additional elements of claim 1 (e.g., the “computer-implemented ..., wherein the one or more image sensors comprise a first image sensor and a second image sensor, the first image sensor and the second image sensor being a same image sensor or different image sensors, ... by the first image sensor ... by the second image sensor” of claim 2, the “computer-implemented” of claim 3, the “computer-implemented ... by a preprocessing server” of claim 4, the “computer-implemented” of claim 5, the “computer-implemented” of claim 6, the “computer-implemented ... by the one or more processors, ... to a scheduling server; ... by the one or more processors, ... from the scheduling server; and utilizing, by the one or more processors, the vision LLM to” of claim 7, the “computer-implemented ... utilizing, by the one or more processors, the vision LLM to” of claim 8, the “computer-implemented ..., by the one or more processors, ... from a user device” of claim 9, the “computer-implemented ... utilizing, by the one or more processors, the vision LLM to” of claim 10, the “computer-implemented ... by the vision LLM” of claim 12, and the "computer-implemented ... by the vision LLM" of claim 21). Accordingly, claims 2-10, 12, and 21 also are rejected as ineligible under 35 USC 101.
Regarding claims 13-18, while the claims are of different scope relative to claims 1-6, the claims recite limitations similar to the limitations of claims 1-6. As such, the rejection rationales applied to reject claims 1-6 also apply for purposes of rejecting claims 13-18. Limitations recited by claims 13-18 that do not have a counterpart in claims 1-6, such as the recited “system for automatically ..., the system comprising: one or more processors; and one or more non-transitory computer readable media storing instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising” limitations of independent claim 13, fail to warrant a finding of eligibility, because such limitations amount to additional elements that fail to meet the criteria of Step 2A, Prong Two and Step 2B, for the same reasons as the additional elements of claims 1-6. Claims 13-18 are, therefore, also rejected as ineligible under 35 USC 101.
Regarding claims 19 and 20, while the claims are of different scope relative to claims 1, 2, 13, and 14, the claims recite limitations similar to the limitations of claims 1, 2, 13, and 14. As such, the rejection rationales applied to reject claims 1, 2, 13, and 14 also apply for purposes of rejecting claims 19 and 20. Claims 19 and 20 are, therefore, also rejected as ineligible under 35 USC 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 4, 6-10, 12-14, 16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ersoz, Ahmet Bahaddin. "Demystifying the Potential of ChatGPT-4 Vision for Construction Progress Monitoring." arXiv preprint arXiv:2412.16108 (2024) (hereinafter referred to as “Ersoz”), in view of U.S. Pat. App. Pub. No. 2023/0290122 A1 to Burkett et al. (hereinafter referred to as Burkett), and further in view of U.S. Pat. App. Pub. No. 2025/0371019 A1 to Chen et al. (hereinafter referred to as “Chen”).
Regarding independent claim 1, Ersoz discloses the following limitations:
“A computer-implemented method for automatically tracking construction progress, the computer-implemented method comprising: ...” - Ersoz discloses, “The integration of Large Vision-Language Models (LVLMs) such as OpenAI's GPT-4 Vision into various sectors has marked a significant evolution in the field of artificial intelligence, particularly in the analysis and interpretation of visual data. This paper explores the practical application of GPT-4 Vision in the construction industry, focusing on its capabilities in monitoring and tracking the progress of construction projects” (p. 1). The application of GPT-4 Vision for monitoring and tracking the progress of construction projects, in Ersoz, reads on the recited limitation.
“... receiving ... a set of processed imaging data comprising one or more images of a construction site, wherein the one or more images have been captured ...” - See the aspects of Ersoz that have been cited above. Ersoz also discloses, “Utilizing high-resolution aerial imagery of construction sites, the study examines how GPT-4 Vision performs detailed scene analysis and tracks developmental changes over time” (p. 1). Receiving the aerial imagery of the construction sites, in Ersoz, reads on the recited limitation.
"... identifying, by applying heuristics data to the classified object using the vision LLM, completion of one or more prerequisite construction elements associated with the object, wherein the one or more prerequisite construction elements are obscured by the object within the one or more images; and ..." - See the aspects of Ersoz that have been cited above. Ersoz also discloses, "Figure 3 - Input images for construction progress tracking" and "Then, the AI tool was provided with a list of expected tasks at the construction site and asked to, "From this list which tasks are completed? Categorize, completed, ongoing and not started tasks only. Prepare a table" (p. 6); and "GPT-4 Vision identified the progress and tabulated the tasks into categories as below: Based on the images provided and the construction list, the categorization of tasks is as follows: 1 Site - Earth - Completed" "2 Foundations - Completed," "3 Structural Columns B1 Basement - Completed," "4 Structural framing - B1 Basement - Completed," "5 Floor Slabs - B1 Basement - Completed," "6 Structural Framing - Ground Floor Completed," "7 Floor Slabs - Ground Floor - Completed," "8 Structural Columns - Ground Floor - Completed," "9 Columns - Ground Floor - Completed," and "10 Structural Framing - 1st Floor - Ongoing," and "The above categorization is inferred from the images with the understanding that the first floor is above the ground floor. The tasks that are 'Ongoing' are assumed because there is visible work in progress related to those tasks in the images, wherein 'Not Started' tasks are for those for which there is no visible work or structure present" (p. 7). Identifying, by applying inferences to the construction site using the AI tool and GPT-4 Vision, completion of, for example, structural framing of B1 Basement from the list of expected tasks associated with the construction site, wherein the construction list elements are obscured by the ground floor and 1st floor within the input images, in Ersoz, reads on the recited limitation.
“... generating a textual description that i) characterizes the identified status at the construction site ... and ii) indicates completion of the one or more prerequisite construction elements associated with the object.” - See the aspects of Ersoz that have been cited above. Generating the textual descriptions characterizing the construction progress at the construction site, and that indicates completion of elements of the basement from the list of expected tasks, upon which the ground floor and 1st floor are supported, in Ersoz, reads on the recited limitations.
The combination of Ersoz and Burkett (hereinafter referred to as “Ersoz/Burkett”) teaches limitations below of independent claim 1:
The claimed “receiving" is "at one or more processors communicatively coupled to one or more image sensors" - See the aspects of Ersoz that have been cited above. Ersoz does not appear to provide details about the hardware involved in its processes. Burkett discloses “inspection and monitoring construction progress” (Abstract), similar to the claimed invention and to Ersoz. Burkett also discloses, “multiple UAVs 102, 104 can contemporaneously inspect (monitor) an outdoor construction site 100” (para. [0021]), “The UAV 102 flying over the construction site 100 can aerially monitor actual individual pieces of equipment, or components thereof, externally placed in the outdoor construction site 100. The construction activities associated with externally placing actual individual pieces of equipment, or components thereof, in the outdoor construction site 100 can include being delivered, installed, constructed, or a combination of the three. An examples of captured aerial images of actual individual pieces of equipment externally placed in the outdoor construction site 100 is shown in FIG. 1. The actual aerial images of the pieces of equipment captured by the UAV 102 can be wirelessly transmitted via a network 762 to an information processing system 700” (para. [0034]), “The first set of observed aerial images of the outdoor construction site 100, comprising this particular portion, can be observed and aerially captured by the UAV 102 using an onboard camera system optically coupled to a receiving optical train that coupled optical data information to the camera device” (para. [0041]), “Regarding FIG. 7, an example of an information processing system 700 includes various components. A processor 702 executes instructions 707 that cause the information processing system 700 to perform operations” (para. [0078]), “As illustrated in FIG. 7, a site inspection process controller 740 instructs the information processing system 700 how to inspect and monitor the construction site using the UAV 102 flying over construction 100” (para. [0089]), and “System 700, at step 1108, collects a set of aerial images of the construction site that were observed and aerially captured by at least one autonomous UAV from the construction site during a construction project” (para. [0104]). Receiving, at the information processing system in wireless communication via the network, in Burkett, reads on the recited limitation. Burkett discloses “inspection and monitoring construction progress” (Abstract), similar to the claimed invention and to Ersoz. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have implemented the GPT-4 Vision aspects, of Ersoz, using the hardware and componentry, of Burkett, so inspection and monitoring can be performed autonomously and continuously, per Burkett (para. [0019]). Additionally or alternatively, the combination would have been obvious as it constitutes: combining prior art elements according to known methods to yield predictable results; simple substitution of one known element for another to obtain predictable results; use of known technique to improve similar devices (methods, or products) in the same way; and/or applying a known technique to a known device (method, or product) ready for improvement to yield predictable results. (See MPEP 2143(I).)
The claimed "captured" is "by the one or more image sensors each equipped with an imaging system” - See the aspects of Ersoz and Burkett that have been cited above. Operation of the cameras of the on board camera systems of the UAVs, in Burkett, reads on the recited limitation. The rationales for combining the cited references, as stated above, also apply here.
“... processing, using a vision large language model (LLM) executed by the one or more processors, the one or more images to identify a status at the construction site of (i) at least one physical structure or (ii) at least one site condition, wherein the processing comprises: ...” - See the aspects of Ersoz and Burkett that have been cited above. Ersoz also discloses, “Formwork” and “Scaffolding” (p. 6). Processing, using GPT-4 and other LVLMs, the imagery of the construction sites, to identify formwork and scaffolding, and the progress of construction, in Ersoz, reads on the recited “processing, using a vision large language model (LLM) ..., the one or more images to identify a status at the construction site of (i) at least one physical structure or (ii) at least one site condition, wherein the processing comprises” limitation. Operation of the processor, in Burkett, reads on the recited “executed by the one or more processors” limitation. The rationales for combining the cited references, as set forth above, also apply here.
“... performing feature extraction to generate one or more structured data representations of the one or more images; ...” - See the aspects of Ersoz and Burkett that have been cited above. Ersoz also discloses, "In the construction site depicted in the image, the following properties and features can be identified" (p. 5). While the examiner views identifying features in images and providing text descriptions thereof, likely reads on the recited limitation, the wording in Ersoz is not explicit. Burkett also discloses, “Such an ML/AI system can include, according to various embodiments, a set of classifiers 720 that are used to extract features (e.g., using the feature set values extractor 744 in the instruction 707) of certain pieces of equipment that are installed in a construction site 100. Classifiers 720 can also include functions and operations stored as a collection of classifiers 748 in instructions 702. The ML/AI system trains the classifiers 720 with a training data set 722 that includes images of certain pieces of equipment. In this example ML/AI system, each training data set image is labeled by an expert in construction project technology and equipment used in the construction project in the construction site 100. The ML/AI system receives aerially captured images from UAV 102 and stores the information in a repository as session data 726. The ML/AI system uses the trained classifiers 720, the decision rules 724, and the decision rules processor 750 to identify individual pieces of equipment in the aerially captured images. The ML/AI system uses machine learning based artificial intelligence image recognition and identification of at least one externally placed piece of equipment in the outdoor construction site 100” (para. [0083]). Performing extracting of features to generate information identifying equipment associated with the captured images, in Burkett, reads on the recited limitation. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have implemented the GPT-4 Vision aspects, of Ersoz, using the feature extraction, of Burkett, so inspection and monitoring can be performed autonomously and continuously, per Burkett (para. [0019]). Additionally or alternatively, the combination would have been obvious as it constitutes: combining prior art elements according to known methods to yield predictable results; simple substitution of one known element for another to obtain predictable results; use of known technique to improve similar devices (methods, or products) in the same way; and/or applying a known technique to a known device (method, or product) ready for improvement to yield predictable results. (See MPEP 2143(I).)
“... identifying an object within the one or more structured data representations indicative of a status at the construction site; ...” - See the aspects of Ersoz that have been cited above. Ersoz also discloses, “Scaffolding is in place for work on the above-ground structure” (p. 6). Identifying the scaffolding within the visual information gleaned from the input images, wherein the scaffolding is indicative of work being done on the above-ground structure, in Ersoz, reads on the recited limitation.
The combination of Ersoz, Burkett, and Chen (hereinafter referred to as “Ersoz/Burkett/Chen”) teaches limitations below of independent claim 1:
“... classifying the object using a transformer-based neural network to determine a semantic category of the status; and ...” - See the aspects of Ersoz that have been cited above. Ersoz also discloses, “ChatGPT effectively recognizes the type of building and construction materials, correctly identifying type of steel and reinforced concrete structures,” “ChatGPT demonstrates an understanding of different equipment types,” and “object classification” (p. 8). Characterizing or classifying objects to determine descriptions of construction progress based thereon, in Ersoz, reads on the recited “classifying the object ... to determine a semantic category of the status” limitation. While it is possible, and even likely, that ChatGPT uses neural networks with transformer architecture, Ersoz does not clearly disclose such features. Chen discloses applying machine learning models to images (Abstract), similar to the claimed invention and to Ersoz/Burkett. Chen also discloses, “Encoding model 220 may generate a vector/embedding 230 indicative of a feature set of image 210A by using a neural network model such as a convolutional neural network (CNN) or a vision transformer (ViT)” (para. [0021]), and “LVLM 320 is a particular type of machine learning model that combines natural language processing (NLP) and computer vision to determine a set of features associated with an image 120 based on a received prompt 310. LVLM 320 may correspond to any suitable LVLM such as Large Language-and-Vision Assistant (LLaVa), GPT-4 Vision, or Fuyu” (para. [0027]). Generating vectors/embeddings for images of objects using the vision transformer neural network model, to determine sets of features associated with the image, in Chen, when applied in the context of determining construction progress, in Ersoz, reads on the recited limitation. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the LVLM and ChatGPT, in Ersoz/Burkett, to include the transformer, neural network, and operations performed in association therewith, of the LVLM and ChatGPT of Chen, because Chen provides a more detailed example of what already is disclosed in Ersoz. Also, use of such features facilitates automation, per Chen (Abstract). Additionally or alternatively, the combination would have been obvious as it constitutes: combining prior art elements according to known methods to yield predictable results; simple substitution of one known element for another to obtain predictable results; use of known technique to improve similar devices (methods, or products) in the same way; and/or applying a known technique to a known device (method, or product) ready for improvement to yield predictable results. (See MPEP 2143(I).)
The claimed “generating" is "based on at least the semantic category of the status” - See the aspects of Ersoz and Chen that have been cited above. Generating the textual descriptions of the construction progress based on characteristics of the images indicative of progress, in Ersoz, involving the vectors/embeddings and features, in Chen, reads on the recited limitation. The rationales for combining the cited references, as stated above, also apply here.
Regarding claim 2, Ersoz/Burkett/Chen teaches the following limitations:
“The computer-implemented method of claim 1, wherein the one or more image sensors comprise a first image sensor and a second image sensor, the first image sensor and the second image sensor being a same image sensor or different image sensors, ...” - See the aspects of Burkett that have been cited above. The cameras on the UAVs, in Burkett, read on the recited limitation.
“... wherein the one or more images comprise one or more first images that have been captured by the first image sensor at a first time and one or more second images that have been captured by the second image sensor at a second time, and ...” - See the aspects of Burkett that have been cited above. Burkett also discloses, “The first set of observed aerial images of the construction site 100, such as the one shown in FIG. 1, are aerially captured by at least one UAV 102, 104, following a flight path over the construction site 100. The first set of observed aerial images of the construction site 100, in this example, can be captured by the UAV 102 at one or more time points within one or more time spans” (para. [0040]). The images includes images taken by the camera of one UAV at one or more time points, and images taken by the camera of another UAV at one or more time points, in Burkett, reads on the recited limitation.
“... wherein the identified object within the one or more structured data representations is a modification between the one or more first images and the one or more second images, and the status at the construction site is a change at the construction site indicated by the modification.” - See the aspects of Ersoz that have been cited above. The identified objects of the data from the images, that shows differences between the images, indicative of changes in progress at the construction site going from the first image to the second image, in Ersoz (p. 6), reads on the recited limitation. The rationales for combining the teachings of the cited references, from the rejection of claim 1, also apply to this rejection of claim 2.
Regarding claim 4, Ersoz/Burkett/Chen teaches the following limitations:
“The computer-implemented method of claim 1, wherein the set of processed imaging data is based on one or more source files that have had at least one of feature extraction, semantic segmentation, or edge detection applied by a preprocessing server.” - See the aspects of Burkett that have been cited above. The performing of feature extraction on the received images, in Burkett, reads on the recited limitation. The rationales for combining the teachings of the cited references, from the rejection of claim 1, also apply to this rejection of claim 4.
Regarding claim 6, Ersoz/Burkett/Chen teaches the following information:
“The computer-implemented method of claim 1, wherein the textual description includes at least one standardized descriptive term.” - See the aspects of Ersoz that have been cited above. The standard English words generated by ChatGPT, in Ersoz (p. 6), read on the recited limitation.
Regarding claim 7, Ersoz/Burkett/Chen teaches the following information:
“The computer-implemented method of claim 1, further comprising: transmitting, by the one or more processors, a scheduling request to a scheduling server; ...” - See the aspects of Ersoz and Burkett that have been cited above. Ersoz also discloses, “Then, the AI tool was provided with a list of expected tasks at the construction site and asked to, “From this list which tasks are completed? Categorize, completed, ongoing and not started tasks only. Prepare a table” (p. 6). Transmitting the request for the table to the AI tool, in Ersoz, reads on the recited “further comprising: transmitting ... a scheduling request” limitation. Use of the computers (and/or servers), in Burkett, reads on the recited “by the one or more processors ... to a scheduling server” limitation.
“... receiving, by the one or more processors, a scheduling response including scheduling data from the scheduling server; and ...” - See the aspects of Ersoz and Burkett that have been cited above. Receiving the table, in Ersoz, reads on the recited “receiving ... a scheduling response including scheduling data” limitation. Use of the computers and servers, in Burkett, reads on the recited “by the one or more processors ... from the scheduling server” limitation.
“... utilizing, by the one or more processors, the vision LLM to generate a progress report, wherein the progress report is based on at least one of the at least one description or the scheduling data.” - See the aspects of Ersoz and Burkett that have been cited above. Using the LVLM to generate the entries for the table, wherein the entries of the table include descriptions of construction site objects, locations, and progress, in Ersoz, reads on the recited “utilizing ... the vision LLM to generate a progress report, wherein the progress report is based on at least one of the at least one description or the scheduling data” limitation. Use of the computers, in Burkett, reads on the recited “by the one or more processors” limitation. The rationales for combining the teachings of the cited references, from the rejection of claim 1, also apply to this rejection of claim 7.
Regarding claim 8, Ersoz/Burkett/Chen teaches the following information:
“The computer-implemented method of claim 7, further comprising: utilizing, by the one or more processors, the vision LLM to generate an analysis of the construction site, wherein the analysis is based on the progress report.” - See the aspects of Ersoz and Burkett that have been cited above. Using the LVLM to generate the analysis of the construction progress, based on the elements of the table, in Ersoz, reads on the recited “further comprising: utilizing ... the vision LLM to generate an analysis of the construction site, wherein the analysis is based on the progress report” limitation. Use of the computers, in Burkett, reads on the recited “by the one or more processors” limitation. The rationales for combining the teachings of the cited references, from the rejection of claim 1, also apply to this rejection of claim 8.
Regarding claim 9, Ersoz/Burkett/Chen teaches the following information:
“The computer-implemented method of claim 7, further comprising: receiving, by the one or more processors, a data inquiry from a user device, wherein the data inquiry includes at least one reference to a portion of the construction site.” - See the aspects of Ersoz and Burkett that have been cited above. Ersoz also discloses, “The system was instructed to “Two images will be uploaded, after upload, categorize and create a list of construction progress.”” (p. 6). Receiving the instructions, including reference to the construction site shown in the input images, in Ersoz, reads on the recited “further comprising: receiving ... a data inquiry ..., wherein the data inquiry includes at least one reference to a portion of the construction site” limitation. Additionally or alternatively, the prompts associated with Figure 2 of Ersoz (pp. 4 and 5), read on the limitation. Use of the computers, in Burkett, reads on the recited “by the one or more processors ... from a user device” limitation. The rationales for combining the teachings of the cited references, from the rejection of claim 1, also apply to this rejection of claim 9.
Regarding claim 10, Ersoz/Burkett/Chen teaches the following information:
“The computer-implemented method of claim 9, further comprising: utilizing, by the one or more processors, the vision LLM to generate a data response, wherein the data response is based on at least one of the progress report, the scheduling data, or the data inquiry.” - See the aspects of Ersoz and Burkett that have been cited above. Using the LVLM to generate responses to prompts, wherein the responses include information about progress at the construction site from the table, in Ersoz, reads on the recited “further comprising: utilizing ... the vision LLM to generate a data response, wherein the data response is based on at least one of the progress report, the scheduling data, or the data inquiry” limitation. Use of the computers, in Burkett, reads on the recited “by the one or more processors” limitation. The rationales for combining the teachings of the cited references, from the rejection of claim 1, also apply to this rejection of claim 10.
Regarding claim 12, Ersoz/Burkett/Chen teaches the following information:
“The computer-implemented method of claim 7, wherein the progress report is a first progress report, wherein the first progress report is further based on a second progress report generated by the vision LLM.” - See the aspects of Ersoz that have been cited above. Continuous operation of the LVLM, to generate responses to prompts, including listings of progress information and progress tables, in Ersoz, reads on the recited limitation.
Regarding claims 13, 14, 16, and 18-20, while the claims are of different scope relative to claims 1, 2, 4, and 6, the claims recite limitations similar to those recited by claims 1, 2, 4, and 6. As such, the rationales applied for purposes of rejecting claims 1, 2, 4, and 6 also apply for purposes of rejecting claims 13, 14, 16, and 18-20. Limitations recited by claims 13, 14, 16, and 18-20 that do not appear to be recited by claims 1, 2, 4, and 6, such as the system componentry and hardware limitations of claims 13 and 19, are disclosed by Burkett (FIG. 7). Claims 13, 14, 16, and 18-20 are, therefore, also rejected under 35 USC 103 as obvious in view of Ersoz/Burkett/Chen.
Claims 3, 5, 15, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Ersoz, in view of Burkett, further in view of Chen, and further in view of U.S. Pat. App. Pub. No. 2021/0158207 A1 to Alsahlawi et al. (hereinafter referred to as “Alsahlawi”).
Regarding claim 3, the combination of Ersoz, Burkett, Chen, and Alsahlawi (hereinafter referred to as “Ersoz/Burkett/Chen/Alsahlawi”) teaches the following limitations:
“The computer-implemented method of claim 1, wherein the one or more images each include associated metadata, and the one or more structured data representations are generated further based on the metadata associated with the one or more images.” - See the aspects of Ersoz that have been cited above. While Ersoz discloses, "This section delves into the diverse applications of GPT-4V across different domains, demonstrating its capability to interpret and respond to visual data alongside text" (p. 2), Ersoz does not appear to specifically disclose metadata associated with the visual data. Alsahlawi discloses “inspection and monitoring construction progress” (Abstract), similar to the claimed invention and to Ersoz/Burkett/Chen. Alsahlawi also discloses, “In addition to descriptions of images, other types of metadata that may be used to enhance and refine prediction models may include: location data (which can be verified against visual images of known locations at the worksite), assigned task status updates, information about whether or not an accident occurred, information on how long a task or sub-task took to complete, as well as other data and information” (para. [0162]). The images including descriptive metadata, including location data, in Alsahlawi, when associated with the images of Ersoz/Burkett/Chen, reads on the recited limitation. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the image data, of Ersoz/Burkett/Chen, to include the location data metadata, of Alsahlawi, to enhance and refine prediction models, per Alsahlawi (para. [0162]). Additionally or alternatively, the combination would have been obvious as it constitutes: combining prior art elements according to known methods to yield predictable results; simple substitution of one known element for another to obtain predictable results; use of known technique to improve similar devices (methods, or products) in the same way; and/or applying a known technique to a known device (method, or product) ready for improvement to yield predictable results. (See MPEP 2143(I).)
Regarding claim 5, Ersoz/Burkett/Chen/Alsahlawi teaches limitations below:
“The computer-implemented method of claim 1, wherein the set of processed imaging data includes metadata comprising at least one of timestamp data, geolocation data, sensor data, lens data, photographic setting data, sensor calibration data, or sensor angle data, and wherein the identifying of the status at the construction site is further based on the metadata.” - See the aspects of Ersoz and Alsahlawi that have been cited above. The image data including metadata comprising location data, in Alsahlawi, wherein progress within the construction site is identified based on said image data, in Ersoz, reads on the recited limitation. The rationales for combining the teachings of the cited references, from the rejection of claim 3, also apply to this rejection of claim 5.
Regarding claims 15 and 17, while the claims are of different scope relative to claims 3 and 5, the claims recite limitations similar to those recited by claims 3 and 5. As such, the rationales applied for purposes of rejecting claims 3 and 5 also apply for purposes of rejecting claims 15 and 17. Claims 15 and 17 are, therefore, also rejected under 35 USC 103 as obvious in view of Ersoz/Burkett/Chen/Alsahlawi.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Ersoz, in view of Burkett, further in view of Chen, and further in view of JP Pat. App. Pub. No. 2019029002 A to Li et al. (hereinafter referred to as "Li").
Regarding claim 21, the combination of Ersoz, Burkett, Chen, and Li (hereinafter referred to as Ersoz/Burkett/Chen/Li teaches the following limitations:
"The computer-implemented method of claim 1, wherein the one or more images comprise a 2:1 equirectangular image and wherein the processing further comprises: preventing a warping error in a portion of the image by converting the 2:1 equirectangular to a cubic projection prior to analysis by the vision LLM." - See the aspects of Ersoz that have been cited above. While Ersoz discloses visual data and images, Ersoz does not explicitly disclose technical characteristics of the visual data and images. Li discloses "an image processing technique." (English-language translation, p. 1.) Li also discloses, "Further processing is performed on the image acquired after the above processing to reduce distortion at the top of the image. Specifically, an equirectangular projection image having a 2:1 width-to-height ratio output from the panoramic camera Theta S is converted into a cubic projection." (Id. at p. 4.) The images including the equirectangular projection image having the 2:1 width-to-height ratio, and the act of reducing distortion by converting the image to the cubic projection, in Li, when applied in the context of preparing the visual data and images, of Ersoz, reads on the recited limitation. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the visual data and images, of Ersoz, to have undergone the processing, of Li, to reduce distortion, per Li. (See id.) Additionally or alternatively, the combination constitutes: combining prior art elements according to known methods to yield predictable results; simple substitution of one known element for another to obtain predictable results; use of known technique to improve similar devices (methods, or products) in the same way; and/or applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (see MPEP 2143(I)).
Response to Arguments
On pp. 11-19 of the Reply, the applicant requests reconsideration and withdrawal of the claim rejection under 35 USC 101. More specifically, the applicant emphasizes the "identifying" and "generating" limitations found, for example, at the end of independent claim 1. (See Reply at pp. 11 and 12.) With respect to Step 2A, Prong Two, the applicant argues that prior solutions require extensive preparation of building information models that take a long time to prepare, and that the prior solutions also involve a labor-intensive manual review involving human experts comparing site imagery against planned elements in the models. (See id. at pp. 13 and 14.) According to the applicant, the claimed invention is a solution to the problems and provides technical benefits. (See id. at p. 14.) The examiner finds these arguments unpersuasive. The prior solutions encompass things like preparing updated blueprints or other site plans, and having humans compare site pictures to the blueprints or site plans. The time it takes to perform the prior solutions is not a technical problem. Thus, the applicant's claimed invention does not offer an unconventional technical solution to a technical problem, but instead, offers a technical solution to a non-technical problem. The latter is not grounds for eligibility. Further, the applicant's claimed invention, and purported solution, appears to be nothing more than accelerating a process where the increased speed comes solely from the capabilities of a general-purpose computer, and/or mere automation of manual processes, neither of which warrants eligibility. (See MPEP 2106.05(a)(I).)
The applicant also argues that independent claim 1 recites a specific technical pipeline by which captured imaging data is transformed into a machine-generated status description of a physical construction site, with the applicant emphasizing the recited "performing feature extraction to generate one or more structured data representations of the one or more images," "identifying an object within the one or more structured data representations indicative of a status at the construction site," and "classifying the object using a transformer-based neural network to determine a semantic category of the status" limitations of the claim. (See Reply at p. 14.) The applicant argues that these are not recitations of a generic computer applied to an idea, especially in light of description in the applicant's specification of computer vision techniques, deep learning models, and vectors, being involved. (See id.) The applicant argues that this constitutes specifying the particular form of the intermediate data representation upon which the subsequent identification and transformer-based classification operate, and specifies the particular ML architecture that performs the classification. (See id. at pp. 14 and 15.) The examiner finds these arguments unpersuasive. The examiner's perspective is that there is nothing technical about the pipeline established by the above-recited "performing" and "identifying" limitations, except for inclusion of a generic "transformer-based neural network." by "nothing technical," the examiner means that the steps are performed manually, by humans, and that the pipeline involves nothing more than swapping in a type of NN for a human. Summarizing what one sees in an image, whether the summarizing results in a textual description, a checklist, a table, or some other structure, can be performed manually and mentally. Classifying or categorizing what one sees in an image also can be performed manually and mentally. The NN is adding nothing--it is just performing a step in the exact way a human would using their mind and pen and paper. The NN is an additional element that is not being analyzed in a vacuum, but rather, in the context of the step(s) being performed, and the analysis shows that the NN is, or is acting like, any generic computer running ML software. Also, the situation here is much like that of ineligible claim 2 of the Office's Example 47.
The applicant also argues for eligibility in light of independent claim 1 requiring the vision LLM to identify "completion of one or more prerequisite construction elements associated with the object, wherein the one or more prerequisite construction elements are obscured by the object within the one or more images." (See Reply at p .15.) The applicant argues that the claimed system does not merely report the visual content that the image sensors captured, and instead, produces a status description not recoverable from the captured image data by observation, due to elements being occluded from view. (See id.) The applicant characterizes this a technical improvement. (See id. at pp. 15 and 16.) The examiner finds these arguments unpersuasive. The limitation above is nothing more than performing a manual/mental task, but using a generic vision LLM instead of one's mind. A person looking at an image of a top floor of a construction site can know that there must be floors under the top floor that are at least partially completed, and would come to such a conclusion via intuition and experience. Doing the same thing using a vision LLM is not a technical improvement. Rather, it is merely applying technology described at a high level of generality to perform an abstract idea. This is not an eligibility-warranting improvement (because it is an improvement to the abstract idea--see MPEP 2106.05(a)(II)), and is likely insignificant extra-solution activity as well (see MPEP 2106.05(g)).
The applicant also argues that the examiner isolates the identification and description steps, characterizes them as observation and evaluation capable of performance in the human mind, and treats the recited image sensors, imaging system, processors, vision LLM, and transformer-based NN as generic components invoked merely as tools. (See Reply at p. 16.) The applicant argues that doing so is evaluating the additional elements in a vacuum, completely separate from the recited judicial exception. (See id.) The examiner finds the arguments unpersuasive. While the 35 USC 101 section above separates abstract idea elements from additional elements for purposes of clearly identifying which is which, the rejection should not be viewed as analyzing the additional elements in a vacuum. All additional elements have been considered in the context of the abstract idea steps they perform, and it is the examiner's view that all additional elements amount to nothing more than applying various forms of computer-based technology, claimed at a high-level of generality, to perform steps that can be performed mentally by a human. This is a proper as-a-whole inquiry.
The applicant also argues that reasoning from Desjardin call for a finding of eligibility here. (See Reply at p. 16.) The applicant emphasizes reasoning involving cautions against reducing technological claims to high-level abstractions or dismissing operational limitations as generic components. (See id.) The applicant contends that the examiner has violated this when characterizing the claims as tracking construction progress. (See id. at pp. 16 and 17.) The examiner finds these arguments unpersuasive. The 35 USC 101 rejection here does not involve any reduction or generalization. It is based on the explicit wording of the claims themselves. Any characterization should be viewed as a short hand reference to the actual wording of the claims. The actual wording of the claims are abstract idea elements, and additional elements, that fail to warrant eligibility. For example, the rejection should not be viewed as stating that tracking construction progress is a mental process, and instead, that "identifying, by applying heuristics data to the classified object ..., completion of one or more prerequisite construction elements associated with the object, wherein the one or more prerequisite construction elements are obscured by the object within the one or more images" is the mental process (as any human that looks at an image of a construction site can assume what is below the top floor, based on, for example, inference, knowledge, and experience).
With respect to Step 2B, the applicant argues that the examiner has not met the burden of demonstrating that the additional elements represent well-understood, routine, and conventional activity. (See Reply at p. 17.) The examiner finds the arguments unpersuasive. The rejection is based on other ineligibility rationales. The ineligibility rationales applied at Step 2A, Prong Two of the rejection also apply at Step 2B.
On pp. 19-21 of the Reply, the applicant argues for reconsideration and withdrawal of the claim rejections under 35 USC 103. More specifically, the applicant argues that the combined references fail to teach or suggest all claimed limitations, as required. (See Reply at p. 19.) The applicant emphasize the language of the recent additions to the independent claims. (See id. at pp. 19 and 20.) According to the applicant, Ersoz's inferential reasoning is tethered to what is visually observable, per p. 7 of Ersoz describing ongoing and not started tasks. (See id. at p. 20.) The applicant also contends that inferences drawing from images with the understanding that the first floor is above the ground floor is misplaced, as that passage reflects spatial disambiguation among floors that Ersoz can plainly see, not an inference concerning work concealed behind a detected object. (See id. at pp. 20 and 21.) The applicant also argues that Burkett and Chen fail to remedy the deficiencies of Ersoz. (See id. at p. 21.) The examiner finds these arguments unpersuasive. The inference in Ersoz go beyond spatial disambiguation. Structural framing of the basement, which cannot be seen from the image, can be inferred as completed when work on the floor slabs of the 1st floor is ongoing. See p. 7 of Ersoz. One of ordinary skill in the art would understand that, in Ersoz, the system infers that the structural framing of the basement is finished in view of seeing the floor slabs of the 1st floor having been laid on top of the basement (thereby obstructing any view of the structural framing of the basement).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THOMAS Y. HO whose telephone number is (571)270-7918. The examiner can normally be reached Monday through Friday, 9:30 AM to 5:30 PM Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jerry O'Connor, can be reached at 571-272-6787. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THOMAS YIH HO/Primary Examiner, Art Unit 3624