DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This is a final rejection in response to remarks/amendments filed on 07/06/2026. Claims 1, 14, 27, and 30 are amended herein. Claims 3, 15, 16, and 18 were previously cancelled. Therefore, claims 1, 2, 4, 6-15, 17, and 19-31 are pending and are examined herein.
Priority
The earliest filing date is the filing date of the present application which is 12/06/2023.
Claim Rejections – 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 2, 4, 6-15, 17, and 19-31 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Is the claim to a Process, Machine, Manufacture, or Composition of Matter?
Claims 1, 2, 4, 6-13, and 28-31: A computer system comprising at least one processor in communication with at least one memory device, wherein the at least one processor programmed to:
Claims 14, 15, 17, 19-26: A computer-implemented method performed by a computer device including at least one processor in communication with at least one memory device, the method comprising: Claim 27: At least one non-transitory computer-readable media having computer-executable instructions embodied thereon, wherein when executed by a computing device including at least one processor in communication with at least one memory device, the computer-executable instructions cause the at least one processor to:
All of the claims fall under at least potentially eligible subject matter category, at least “process, machine, or manufacture,” therefore the claims are to be further analyzed under step 2.
Step 2a Prong 1: Is the claim reciting a Judicial Exception(A Law of Nature, a Natural Phenomenon (Product of Nature), or An Abstract Idea?)
The claims under the broadest reasonable interpretation in light of the specification are analyzed herein. Representative claims 1, 14 and 27 are marked up, isolating the abstract idea from additional elements, wherein the abstract idea is in bold and the additional elements have been italicized as follows:
Claim 1 Preamble: A computer system comprising at least one processor in communication with at least one memory device, wherein the at least one processor programmed to:
Claim 14 Preamble: A computer-implemented method performed by a computer device including at least one processor in communication with at least one memory device, the method comprising: Claim 27 Preamble: At least one non-transitory computer-readable media having computer-executable instructions embodied thereon, wherein when executed by a computing device including at least one processor in communication with at least one memory device, the computer-executable instructions cause the at least one processor to:
Claim 1, 14, 27 Body:
store(ing) one or more trained machine-learning models for analyzing images to identify issues pictured in the images;
receive, from a user computer device associated with a location, a real-time video stream of a location;
control a camera of the user computer device to capture a plurality of initial images of the location from the real-time video stream;
store a plurality of initial images of a location;
receive(ing), from the user computer device, a plurality of current images of the location;
compar(ing) the plurality of current images to a plurality of reference images to determine whether the plurality of current images are suitable for image processing by the one or more trained machine-learning models;
upon determining that the plurality of current images are suitable, execute(ing) a trained image comparison model of the one or more trained machine-learning models to compare the plurality of initial images to the plurality of current images, wherein an output from the trained image comparison model includes an identification of a first fixture captured in both the plurality of initial images and the plurality of current images and an identification of one or more differences in the first fixture between the plurality of initial images and plurality of current images;
based on the output, select a trained classification model from the one or more trained machine-learning models that is trained to identify issues with fixtures of a type of the first fixture, wherein others of the trained machine-learning models are respectively trained to identify issues with a corresponding fixture type of other fixtures;
execute the selected trained classification model to identify an issue representing a reason for the difference; and
schedule at least one appointment at the location to remediate the issue.
When evaluating the bolded limitations of the claims under the broadest reasonable interpretation in light of the specification, it is clear that representative claims 1, 14, and 27 recite an abstract idea under the category “certain methods of organizing human activity.” More specifically, the present invention falls under the sub-groupings “commercial or legal interactions” which include agreements in the form of contracts, legal obligations, advertising, marketing or sales activities or behaviors, and business relations” and “managing personal behavior, or interactions, or relationships between people.” The representative claims recite a commercial or legal interaction known as performing inspections, which is made clear in at least specification paragraph [0004], and [0005] is a practice performed in order to manage rental properties. Therefore, the claims, which perform data recordation, processing, and outputting in order to detect issues within a rental property are merely steps to comply with legal obligations and facilitate business relations. Furthermore, the steps merely transmit the notifications in order to manage the personal behavior, relationships and interactions between individuals which is another subcategory of “certain methods of organizing human activity” outlined in MPEP 2106.04(a)(2)(II)(C) which include social activities, teaching and following rules or instructions.
Even when considering “receiving a real-time video stream of a location,” and “capture a plurality of initial images of the location from the real-time video stream”, are recited with such breadth that the scope of the claims includes instructions to perform the intended functions of receiving video data and capturing images from the data. The broadest reasonable interpretation of the claims includes any manner of receiving real-time video data, and capturing images from the real-time video data. This includes instructions to manage the behavior of a person in order to capture the intended images, which would fall under the scope of “certain methods of organizing human activity.” Furthermore, these steps are also mere data collection steps associated with the abstract idea.
Even when considering the amended limitations that the executing of a “trained image comparison model,” which outputs “differences between the initial images and current images,” followed by “an identification of a first fixture captured in both the plurality of initial images and the plurality of current images and an identification of one or more differences in the first fixture between the plurality of initial images and plurality of current images,” and “based on the output, select a trained classification model from the one or more trained machine-learning models that is trained to identify issues with fixtures of a type of the first fixture, wherein others of the trained machine-learning models are respectively trained to identify issues with a corresponding fixture type of other fixtures.” These are still recitations of an abstract idea because it merely claims the use of mathematical models to perform the “certain method of organizing human activity” at a high-level of generality that it is no more than claiming the models as “a black box” to perform the intended outcome. Therefore, the claim limitations in their broadest reasonable interpretation include mere instructions to an individual to perform the abstract tasks at hand using any trained image comparison model, or any trained classification model. When considering the activity itself for example, identifying fixtures in images, and identifying differences within the fixtures, this is merely equivalent to instructions to manage personal behavior, interactions, or relationships between people. Furthermore, based on the output of the fixture type, selecting a particular out of a list of models trained for that particular fixture type is also equivalent to “managing personal behavior, interactions, or relationships between people,” as it is merely equivalent to an instruction to an individual to manage their behavior.
Furthermore, “scheduling an appointment to remediate the issue” is still an example of “commercial or legal interactions,” or “managing personal behavior, interactions, or relationships between people,” because it is merely a reciting a social task.
To further show that the amended limitations are broad enough to be mere instructions to a user to manage their personal behavior, the following arguments describe how the amended limitations are merely mental processes that can be performed by a human using a pen and paper.” The particular steps, starting from “compar(ing) the plurality of current images to a plurality of reference images to determine whether the plurality of current images are suitable for image processing by the one or more trained machine-learning models;” and ending with “wherein others of the trained machine-learning models are respectively trained to identify issues with a corresponding fixture type of other fixtures;” Nothing in these limitations, other than merely instructing them to be performed on a computer (which will be addressed later), merely limits the claims from being performed in the human mind. See MPEP 2106.04(a)(2)(III), “Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, judgments, and opinions.” For example, a human mind can compare the initial images to the current images, and identify one or more differences between the plurality of initial images and plurality of current images. The human mind, using a pen and paper can determine whether images are suitable for image processing based on observations. Furthermore, a human can select particular models to use based on the type of fixture. Therefore, the claims recite “collecting information, analyzing it, and displaying certain results of the collection analysis,” where the data analysis steps are recited at such a high level of generality such that they could practically be performed in the human mind. Requiring that a “trained image comparison model,” and a “trained classification model,” are used to perform the tasks is still broad enough to recite a mental process cause it can include any use of a model (mathematical calculation), such that the use of a physical aid (pencil and paper or a slide rule) would enable a human to perform the steps manually. Therefore, when considering these particular amendments, they fall under “managing personal behavior, or relationships, or interactions between people,” because they encapsulate mere instructions to an individual to manage their personal behavior, supported by the fact that the functions can be performed mentally by a human with a pen and paper.
Therefore, the claims recite an abstract idea of “detecting issues within images of a rental property and transmitting notifications based on the issues” wherein the steps are recited at a high level of generality that they are no more than “certain methods of organizing human activity.” The examiner notes that even though the interactions may be between an individual and a computer, the activity itself is what is considered. MPEP 2106.04(a)(2)(II) states, “Finally, the sub-groupings encompass both activity of a single person (for example, a person following a set of instructions or a person signing a contract online) and activity that involves multiple people (such as a commercial interaction), and thus, certain activity between a person and a computer (for example a method of anonymous loan shopping that a person conducts using a mobile phone) may fall within the "certain methods of organizing human activity" grouping.”
In addition, the claims and are to be further analyzed under Step 2A Prong 2 and step 2b.
Step 2A Prong 2: Does the claim recite additional elements that integrate the judicial exception into a practical application?
Claims 1, 14, and 27 recite the following additional elements:
-computer system in claim 1
-processor in claims 1, 14, 27
-memory device in claims 1, 14, 27
-computer-implemented method in claims 14
-computer device in claims 14, 27
-non-transitory computer-readable medium in claim 27
- a user computer device in claims 1, 14, 27
- control a camera of the user computer device; in claims 1, 14, 27
- trained machine learning models in claims 1, 14, and 27
The additional elements listed above, when considered individually and in combination with the claim as a whole, are no more than a recitation of the words “apply it” (or an equivalent) or mere instructions to implement an abstract idea or other exception on generic computing components as outlined in MPEP 2106.05(f). In this case, the abstract idea of “detecting issues within images of a rental property and transmitting notifications based on the issues” are performed on generic computing devices such as computer devices, processors, memory devices, user computer device and non-transitory computer-readable medium. The claims do provide steps of creating models and analyzing images, but these are merely abstract idea steps that can be performed by a generic computing device. Particularly, the ”trained image comparison model,” and “trained machine learning model” are invoked to perform the abstract idea but are used in a black box manner that does not specify the details of how a solution to a problem is accomplished. There are no specific improvements to image comparison or machine learning models, and the claims recite that they are already “trained,” without citing any specific training process. Therefore, the claims are merely invoking computers or other machinery to perform an existing process, and recite only the idea of an outcome or solution, therefore, the additional elements are merely equivalent to the words “apply it,” because they are mere instructions to implement an abstract idea or other exception on a computer. Even when considering the amended limitation of capturing “real-time video stream of a location” this is still an example of utilizing the computer to perform economic tasks (such as data capture), in order to perform the abstract idea.
Furthermore, the steps of “control a camera of the user computer device... to capture a plurality of initial images of the location from the real-time video stream;” also falls under “apply it” in MPEP 2106.05(f), because it is using a device (such as a camera) in its ordinary capacity to perform the capturing of images without providing an improvement to camera technology or any technical field (See MPEP 2106.05(a)). The control of a camera of the user computer device, as supported in [0033], includes,
“In additional embodiments, the IA computer device controls the camera of the user's mobile device when taking images of the property. For example, the IA computer device instructs the user where to stand and at what angle to point to take the image.” Therefore, the control a camera of the user computer device step merely includes a set of instructions to perform the abstract idea on a generic computing device or an ordinary device in its ordinary capacity (instructions to capture a photo using a camera).
Even when considering the additional elements individually or as an ordered combination, nothing in claims integrates the abstract idea into a practical application. Therefore, the claims are directed to an abstract idea without integration into a practical application.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
Claims 1, 14, and 27 recite the following additional elements:
-computer system in claim 1
-processor in claims 1, 14, 27
-memory device in claims 1, 14, 27
-computer-implemented method in claims 14
-computer device in claims 14, 27
-non-transitory computer-readable medium in claim 27
- a user computer device in claims 1, 14, 27
- control a camera of the user computer device; in claims 1, 14, 27
- trained machine learning models in claims 1, 14, and 27
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered separately and as an ordered combination, they do not add significantly more (also known as an “inventive concept”) to the exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using computers devices, processors, memory devices, user computer device, camera and non-transitory computer-readable medium to perform the “storing one or more models to identify issues pictured in images, receiving a real-time video stream, capturing initial images, receiving current images, executing a model to compare the images and identify the differences, determine if the differences exceed a threshold and identify the issue using a classification model, and schedule an appointment to remediate the issue” amounts to no more than mere instructions to apply the exception using generic computer components. Even when viewing the claims as a whole, nothing meaningfully limits the claims such that they are significantly more than the abstract idea. There is no improvement to how any models are trained including machine learning models, and there is no improvement to computer functionality, technology, or any technical field. Therefore, the claims are directed to an abstract idea without integration to a practical application or significantly more.
Dependent claims 2, 4, 6-13, 15, 17, and 19-31 are also given the full two part analysis both individually and in combination with the claims they depend on herein:
Claims 2, 15 further define the abstract idea by limiting the data to a particular source, such as historical images of a property, or images received from a computer device associated with the location. This is more of the same abstract idea because they are merely indicating a data source or generally stating a data processing state, whilst still performing the same abstract idea of “detecting issues within images of a rental property and transmitting notifications based on the issues.” The additional elements of processor, and user computing device are still merely linking the abstract idea to generic computing components. Since there are no further additional elements to analyze, the claims are directed to an abstract idea without integration into a practical application or significantly more.
Claims 4, 6, 7, 9, 17, 19, 20, 22 further define the abstract idea by providing instructions to the user in order to capture the data. This is merely a further embellishment of the abstract idea, since it is an “instruction” in order to facilitate human behavior, which is also a “certain method of organizing human activity” in the subcategory of “managing personal behavior, interactions, or relationship between individuals” in MPEP 2106.05(a)(2)(II)(C). The only new additional elements are the “display screen” in claims 7, 20. However, this is merely a generic computing device performing the abstract idea, operating in its ordinary capacity to perform the display of an output, as outlined in MPEP 2106.05(f). Therefore, the claims are directed to an abstract idea without integration into a practical application or significantly more.
Claims 8, 21 further define the abstract idea by “controlling the user computer device to capture the plurality of initial images.” This is merely a data collection step in the processing of performing the abstract idea of “detecting issues within images of a rental property and transmitting notifications based on the issues.” There are no further additional elements to analyze, therefore, the claims are directed to an abstract idea without integration into a practical application or significantly more.
-Claim 10, 23 further define the abstract idea by scheduling an appointment with a service provider to resolve the one or more issues. This is merely a further embellishment of the abstract idea, since transmitting to the scheduling to a service provider is still an instruction to an individual and it is no more than managing an individual’s workflow, which is also a “certain method of organizing human activity” in the subcategory of “managing personal behavior, interactions, or relationship between individuals” in MPEP 2106.05(a)(2)(II)(C). There are no further additional elements to analyze, therefore, the claims are directed to an abstract idea without integration into a practical application or significantly more.
-Claims 11, 12, 13, 24, 25, 26 receives images overtime and determines a trend or calculates a score. These are merely embellishments of the abstract idea, since they recite additional data processing steps that still perform the abstract idea of “detecting issues within images of a rental property and transmitting notifications based on the issues.” There are no further additional elements to analyze, therefore, the claims are directed to an abstract idea without integration into a practical application or significantly more.
-Claims 28 and 29 recite more of the same abstract idea because it merely adds the steps of defining a predefined period of time since images were captured and transmit instructions to the user device to capture the images. The abstract idea is still recited because the instructions are merely to capture the initial images (28) and store the plurality of periodic images in a timeline (29) which are mere data collection and storage steps associated with the abstract idea. The additional element of sending the instructions to the user computer device periodically is still equivalent to apply because performing the abstract idea on a computer at multiple periods is still “mere instructions to carry out the abstract idea.” Even when considering the additional elements individually or in an ordered combination, they still fail to integrate the abstract idea into a practical application because all of the claimed functions are invoking the use of generic computing components to perform an existing abstract idea process. Even when viewed as a whole, nothing meaningfully limits the abstract idea such that it provides significantly more.
Claim 30 further limits the abstract idea by prioritizing issues based on urgency and scheduling an appointment to address the first priority issue. This is more of the same “certain methods of organizing human activity,” because it is merely managing the workflow and performing business relationships. There are no additional elements to consider that have not already been identified as “apply it” level elements, therefore the claims are directed to an abstract idea without integration into a practical application without significantly more.
Claim 31 further limits the abstract idea by adding the step of receiving audio information, and analyzing the audio information to detect the issues at the location. However, this is more of the same abstract idea because it is recited with such generality that it does exclude instructions to an individual to perform the task. Furthermore, it does not specify how the issues are detecting from the audio, therefore, it only recites the idea of the outcome or solution. There are no additional elements to consider that have not already been identified as “apply it” level elements, therefore the claims are directed to an abstract idea without integration into a practical application without significantly more.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 4, 6-9, 11-15, 17, 19-22, 24-28, 29, and 31 are rejected under 35 U.S.C. 103 as being obvious over Nelson et al. (US 20200410278 A1) hereinafter Nelson, in view of Chavez et al. (US 11600063 B1) hereinafter Chavez, further in view of Fields et al. (US 20240053816 A1) hereinafter Fields.
Regarding Claims 1, 14, 27:
Nelson discloses a method of determining damage to a property by comparing first images to second images and using a trained model that determines the damage. Nelson teaches
Claim 1 Preamble: A computer system comprising at least one processor in communication with at least one memory device, wherein the at least one processor programmed to: (Nelson [0004] An illustrative non-transitory computer readable medium has instructions stored thereon that, upon execution by a computing device, cause the computing device to perform operations.)
Claim 14 Preamble: A computer-implemented method performed by a computer device including at least one processor in communication with at least one memory device, the method comprising: (Nelson [0005] An illustrative method for capturing an image of an object associated with a property using a mobile computing device includes displaying, by a processor of the mobile computing device, a first image representing a field of view of a camera of the mobile computing device on a display of the mobile computing device. The method further includes overlaying, by the processor, a representation of an object associated with a property on the display that is also displaying the first image.)
Claim 27 Preamble: At least one non-transitory computer-readable media having computer-executable instructions embodied thereon, wherein when executed by a computing device including at least one processor in communication with at least one memory device, the computer-executable instructions cause the at least one processor to: (Nelson [0004] An illustrative non-transitory computer readable medium has instructions stored thereon that, upon execution by a computing device, cause the computing device to perform operations.)
Claims 1, 14, 27 Body:
-stor(ing) one or more trained machine-learning models for analyzing images to identify issues pictured in the images; (Nelson [0027] In addition, the various electronic devices may be used to train and implement machine learning models for identifying and assessing damage to objects and property as described herein. For example, the servers 112 may be used to train a model that is stored on one or more of the servers 112 for identifying and assessing damage.)
-storing a plurality of initial images of a location; (Nelson [0074] FIG. 28 is a flow chart illustrating an example method 2800 of determining whether damage to an object exceeds expected wear and tear depreciation damages, in embodiments. As described herein, a user may capture images of various objects or aspects of a property during a property inspection. [0075] In an operation 2802 of the method 2800, a first image of an object is received, the first image being captured at a first time. A first image of an object may be received by virtue of a user using an electronic device to capture an image (e.g., as shown in FIG. 19-26 or 30-37), or a first image may be received by a server or other electronic device from a device that was actually used to capture the image. [0054] FIG. 8 shows a user interface 800 for a move-in inspection that has been completed. The user interface 800 may be navigated to, for example, if a completed inspection is selected at the interface 500 of FIG. 5. A back arrow 802 may be used to navigate back to a list of inspections such as that of FIG. 5. Information 804 may indicate details of the selected inspection, such as a status (e.g., inspection completed), the date completed, and who completed the inspection. [0087] In addition to images relating to a specific inspection, aggregated images and metadata from other inspections may be sanitized and selected at an operation 2910 for use in training a damage identification model. At an operation 2912, those images may be categorized for contents (e.g., what type of object is in the photographs) and presence of damages (e.g., is there damage shown in the photographs).) The first image is mapped to the initial images at a location taught by Nelson since these images are taking at move in. These images are a plurality of image as seen in Figs 6-8, there are multiple initial(first) images being taken.
-receiv(ing), from the user computer device, a plurality of current images of the location; (Nelson [0076] In an operation 2804, a second image of the object is received, the second image being captured at a second time different from the first time. Similar to the first image, the second image of an object may be received by virtue of a user using an electronic device to capture an image (e.g., as shown in FIG. 19-26 or 30-37), or the second image may be received by a server or other electronic device from a device that was actually used to capture the image. The first and second images in this embodiment are captured at different times. For example, the first image may be captured at or near a time when a tenant/renter moved into a property (or unit of the property), and the second image may be captured at or near a time when a tenant/renter moved out of the property. In this way, the images may represent what the object was like when the tenant/renter moved in and what the object was like when they moved out.)
- compar(ing) the plurality of current images to a plurality of reference images to determine whether the plurality of current images are suitable for image processing by the one or more trained machine-learning models(Nelson [0078] In an operation 2808, the first and second images are aligned such that features of the object in the first and second images align. Such alignment may include resizing, cropping, or otherwise adjusting the first and/or second images so that the features of the object properly align. [0079] In various embodiments, the alignment of the two images may be used to determine that the object in the two images is the same object. For example, if the edges or corners in an image can be aligned, the likelihood that the images capture the same object is high, while if the corners or edges in images cannot be aligned it is likely that the images do not capture the same object (or at least are of insufficient quality to properly compare the objects in the images).)
-upon determining that the plurality of current images are suitable, execut(ing) a trained image comparison model the one or more trained machine-learning models to compare the plurality of initial images to the plurality of current images; (Nelson [0077] In other embodiments, the system may perform an image analysis to determine that the objects in an image are the same object. For example, an image detection system may look for differences that indicate the images do not capture a same object to prevent errors or fraud. In some embodiments, a property manager or landlord may place a visual sticker or other indicator with a code (e.g., QR code, bar code, or other code) on an object so that a visual detection system may identify the code to ensure that the same object is captured. [0078] In various embodiments, a machine learning model trained as described herein may compare images and the objects therein without any alignment, resizing, cropping, etc. [0082] A machine learning model may be trained to consider multiple images of an object together to identify and assess damage as described herein. [0082] For example, additional images taken at different times from the first and second images may be compared to determine damage and/or wear and tear to an object over time. In other examples, the system may be configured to receive multiple contemporaneously taken images of an object to compare to one or more images of the object taken at a different time. For example, multiple images of an object may be taken from different angles, and those images may be used to compare to one or more images of the objects captured at a different time. A machine learning model may be trained to consider multiple images of an object together to identify and assess damage as described herein. [0084] FIG. 29 is a flow chart illustrating an example method 2900 of identifying and assessing damages and training a model for identifying and assessing damages to an object, in embodiments. As described herein, determining that at least one difference between two or more images is associated with damage to the object (e.g., wear and tear depreciation damage or damage that exceeds wear and tear) may include processing the images with a trained machine learning model. In addition to using training a model to identify and assess damages and using a trained model to identify and assess damages as shown in the method 2900, images captured during inspections may be added to data sets that are used to further refine and train models for identifying and assessing damage in images captured of objects over time.)
-wherein an output from the trained image comparison model includes an identification of a first fixture captured in both the plurality of initial images and the plurality of current images(Nelson [0087] In addition to images relating to a specific inspection, aggregated images and metadata from other inspections may be sanitized and selected at an operation 2910 for use in training a damage identification model. At an operation 2912, those images may be categorized for contents (e.g., what type of object is in the photographs. [0092] This, together with other inputs such as damage value, type of damage, object type, etc., may train the model to more accurately identify and assess damage in images based on an amount of time that has passed between the capture of images. For example, images showing a scuffed hardwood floor or bent window blinds may indicate material damage if such scuffs or bends are identified in an image taken one year after an image showing a newly stained hardwood floor or an image taken one year after new blinds were installed, respectively. However, images showing a scuffed hardwood floor compared to an image from a year before that show scuffs, or images showing a scuffed hardwood floor compared to an image from fifteen years before that does not show scuffs, may not indicate material damage. Similarly, while year old blinds that are bent may indicate material damage, ten-year-old blinds that have bends in them may not indicate damage. Accordingly, identifying objects in images and tracking them over time may be helpful in assessing the extent of and value of damage. Thus, inputting images with data indicating their relationship to one another (whether two images show the same object and the time passed between the capture of the images) may be helpful in adequately identifying and assessing damages. Thus, when inspections are performed as described herein, it may be valuable to determine, either through manual tagging, automatic tagging based on image recognition, or any other method, when images taken at different times are of the same object. [0032] For example, a user may photograph a bedroom and not the windows specifically, but an image recognition process may identify windows in the photograph.) Windows, flooring, and blinds are all examples of fixture types which are identified (object types).
-and an identification of one or more differences in the first fixture between the plurality of initial images and plurality of current images;(Nelson [0087] At an operation 2912, those images may be categorized for contents (e.g., what type of object is in the photographs) and presence of damages (e.g., is there damage shown in the photographs). In other words, images of objects may be manually categorized for their contents and whether damage is shown and used to train the model for identifying damage. [0080] In an operation 2810, at least one difference associated with the object based on a comparison between the first image and the second image is determined. [0084] FIG. 29 is a flow chart illustrating an example method 2900 of identifying and assessing damages and training a model for identifying and assessing damages to an object, in embodiments. As described herein, determining that at least one difference between two or more images is associated with damage to the object (e.g., wear and tear depreciation damage or damage that exceeds wear and tear) may include processing the images with a trained machine learning model.)
- execute the selected trained classification model to identify an issue representing a reason for the first difference; and(Nelson [0088] In other words, the model and manual review may only be looking for damage that exceeds a particular threshold indicating material damage. In other embodiments, where the manual review and model are configured to recognize any damage, a model for assessing value of the damage may be used to categorize whether the damage is wear and tear or material damage based on, for example, an amount of damage. [0089] If a manual review at the operation 2914 identifies damage in the images, the images may be tagged as incorrect (or may be re-tagged such that the metadata associated with the image indicates that the image includes damage), so that they can be resubmitted to refine or re-train the model that missed the damage in the images initially at an operation 2920. [0093] Some models may also be trained using data that indicates the extent of and/or presence of damage shown in images or pairs of images. For example, for the model for identifying damage as applied at the operation 2906 of FIG. 29, images and a tag indicating whether or not damage is present in the image, alone or in combination with other information, may be used to train such a model. In another example, information indicating whether damage is wear and tear or material damage may be indicated and input to train a model to identify when damage rises to a material damage threshold. [0090] If damage is identified in images at the operation 2906, the images may be further analyzed at an operation 2916 using a damage pricing model... For example, information may be identified at an operation 2928 to be used for training a damage pricing model. Such information may include, for example, images categorized/tagged by types of damage, types of objects, cost of repair or replacement, geological data, geographic data, demographic data, or any other type of data that may factor into the value of damage to various objects associated with a property... [0111] damage assessment model may use datasets that do not require pre-processing such as image classification or other types of pre-processing...FIG. 38 may be used to build/train models.) The trained models in Nelson satisfy “trained classification model,” because they perform categorization, tagging, and image classification. In Nelson, when damage exceeds a threshold it is determined as an issue, “material damage vs wear and tear” for which “a tenant should pay in certain embodiments. Various embodiments in Nelson above satisfy “identifying a reason for the difference,” such as “types of damage, types of objects,” and [104] where a hole is exceeds a particular threshold, the reason is that the hole is larger than the bounding box.
-schedule at least one appointment at the location to remediate the issue(Nelson [0113] In another example, if a tenant submits a maintenance request because their furnace is broken, the landlord may input that the furnace was repaired or replaced so that the value and/or depreciation of the furnace may be accurately calculated according to the various systems and methods described herein. [0117] Landlord scores may also be calculated using tenant inputs about landlords, such as how good landlords are at maintenance repair and response, scheduling maintenance and damage repairs, overall condition of property, likeliness to rent from again, security deposit deductions, and/or time to return security deposit.) This limitation is satisfied by Nelson because the broadest reasonable interpretation of the limitation covers any scheduling of an appointment, even if performed manually by a landlord, to remediate the issue at the location. The language of “remediate the issue” is still broad enough such that the scheduling can be performed manually be a landlord, even if the issue identified was identified automatically.
However, Nelson fails to teach:
-receive, from a user computer device associated with a location, a real-time video stream of a location;
- control a camera of the user computer device to capture a plurality of initial images of the location from the real-time video stream;
-based on the output, select a trained classification model from the one or more trained machine-learning models that is trained to identify issues with fixtures of a type of the first fixture;
-wherein others of the trained machine-learning models are respectively trained to identify issues with a corresponding fixture type of other fixtures;
Alternatively, Chavez discloses a guided inspection software using augmented reality to perform an inspection in a physical space, utilizing machine learning to automatically detect and classify damage to the space, and prompt a user with instructions to move closer to the detected damages. Chavez teaches:
-receive, from a user computer device associated with a location, a real-time video stream of a location; (Chavez [Col. 4 Lines 33-35] Remote device 204 may comprise a device that can be brought to the location where an inspection is to occur. [Col. 5 Lines 2-5] Additionally, in some embodiments, remote device 204 may include a GPS receiver for receiving GPS information that can be used to determine the location of the remote device. [Col. 7 Lines 30-42] As the user is guided through the living space, remote device 204 may capture images (photos or video) of a living space (i.e., a physical space) during step 402. In some cases, remote device 204 may prompt a user to aim the camera and/or take images of one or more physical structures at the location... In some other embodiments, remote device 204 may automatically take pictures or video without prompting a user. Optionally, remote device 204 could prompt a user to aim the camera at a certain area or feature in the room but may take images or videos automatically without further user action. [Col. 10 Line 23-35] In FIG. 7, the user has moved closer to the door resulting in remote device 204 obtaining better quality images of the region just above door 620. At this point the newly obtained image information is sent to centralized computer system 202 for processing and assessment... To clarify what structure is possibly damaged, remote device 204 may display a highlighted boundary 720 around the damage in the live video feed of the area.)
- control a camera of the user computer device to capture a plurality of initial images of the location from the real-time video stream; (Chavez [Col. 7 Lines 42-47] In step 404, remote device 204 sends image information to server 203 of centralized computer system 202 over a network (for example, network 206). The term “image information”, as used herein, refers to any information corresponding to photos or videos [Col. 10 Lines 42-58] It may be appreciated that during this process centralized computer system 202 could provide other kinds of instructions. As one example, if an image processed at centralized computer system 202 is out of focus or not centered sufficiently on a given physical structure, centralized computer system 202 may prepare and send instructions to have images retaken, either manually by a user or automatically by remote device 204. (56) A guided inspection system may also include provisions for detecting when a structure to be inspected is obscured. For example, during an inspection the blinds on a window may be down. This allows the system to determine if the blinds are damaged, but obscures the window itself from view. In that situation a guided inspection system could be configured to automatically detect the obscured window and prompt the user to raise the blinds so the window can be inspected.) Since Chavez teaches that the remote device (user computer device) sends image information (including videos) to the server and receives instructions to perform the additional capturing (including instructions to the user or to the device itself to automatically control the camera) then the limitation has been taught.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present disclosure to modify Nelson by adding the teachings of Chavez including transmitting a live video stream and controlling a camera to perform automatic capturing from the live video feed. One of ordinary skill in the art would have been motivated to perform this combination by the benefit of simplifying the inspection process enabling a user to quickly and accurately assess possible damage. (Chavez [Col. 3 Lines 4-12] By automatically capturing and analyzing image information about structures in the physical space to determine if there is damage, the system and method improve the efficiency of the inspection process. By using an augmented reality system to prompt a user, the system and method simplify the inspection process and allow users with little to no experience in inspecting properties to quickly and accurately assess possible damage in a physical space.)
However, neither Nelson nor Chavez teach or suggest:
- based on the output, select a trained classification model from the one or more trained machine-learning models that is trained to identify issues with fixtures of a type of the first fixture;
- wherein others of the trained machine-learning models are respectively trained to identify issues with a corresponding fixture type of other fixtures;
Alternatively, Fields discloses an augmented reality-assisted home inspection software which includes various layers of classifiers, with different machine learning classifiers trained on different segments of data based on the tag associated with the inputs (such as an object type). Fields teaches:
- based on the output, select a trained classification model from the one or more trained machine-learning models that is trained to identify issues with fixtures of a type of the first fixture;(Fields [0053] Depending on implementation, one or more machine learning models may be implemented to train multiple classifiers at the same time. The different trained classifiers may be further operated separately or in conjunction to detect different property inspection indicia. Accordingly, the training data may be associated with any number of tags associated with different property inspection indicia such that a classifier can be trained to detect attributes indicative of the tag. In this sense, the tag may be an indication that the underlying training data includes attributes indicative of the tag and the absence of a tag may be an indication that the underlying training data does not include the attributes indicative of the tag. [0064] Additionally, the machine learning model may include multiple layers. For example, in a first layer, the machine learning model may be configured to segment the input underlay layer data to identify and/or label objects. For example, the first layer may identify that the underlay layer data includes image data representative of an appliance, smart device, wall, pipe, or any other object associated with property inspection indicia. Accordingly, in addition to applying a tag that indicates the detected object type, the classifiers in the first layer may identify a segment of the input underlay layer data that includes the image data representative of the object. In this example, the second layer may then be configured to analyze the segmented image data to identify the particular conditions of the object associated with property inspection indicia (e.g., damage to the object, a dimension of the object, or any other condition associated with a property inspection indicia). Accordingly, the machine learning model may include different classifiers in the second layer that are applied in response to different tags applied by the first layer.) In the steps above, Fields teaches the first machine-learning model identifier that identifies an object type, and then uses a model particularly trained on that specific type of object to determine the damage with the object of that object type.
- wherein others of the trained machine-learning models are respectively trained to identify issues with a corresponding fixture type of other fixtures; (Fields [0064] Accordingly, the machine learning model may include different classifiers in the second layer that are applied in response to different tags applied by the first layer. [0066] It should be appreciated that the particular classifiers and models that are implemented in the hierarchical model are not necessarily trained at the same time and/or based on the same training data sets. For example, some of the particular classifiers may be useful for other tasks performed by the operator of the disclosed systems (e.g., an object classifier that builds automated inventory lists). As such, some of the component classifiers may be integrated into multiple machine learning models. Accordingly, the individual classifiers, segmenters, parsers, etc. that form the overall machine learning hierarchy may be modularly trained. Additionally, the ability to identify new objects and/or detect additional indicia for objects may be added by including an additional classifier to the machine learning hierarchy described herein.) Thus, each classifier is trained specifically for sub-set of training data based on the type of object.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present disclosure to further modify Nelson and Chavez by adding the teachings of Fields, particularly the use of different classifiers, specific to an object type, to identify damage related to an object. By modifying Nelson’s teachings, which is indiscriminate to the type of fixture, to be trained on more specific subsets of data based on the object type, one of ordinary skill would have arrived at the claimed invention. One of ordinary skill would have been motivated by the benefit of expediting calculations and increasing the specificity of predictions for each particular instance space. (Fields [0053] Training multiple classifiers may provide an advantage of expediting calculations and further increasing specificity of prediction for each classifier's particular instance space.)
Regarding Claims 2, 15:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 1, wherein the at least one processor is further programmed to/The computer-implemented method of Claim 14 further comprising:
Furthermore, Nelson teaches:
-receiv(ing) a plurality of historical images of a plurality of properties; (Nelson [0087] In addition to images relating to a specific inspection, aggregated images and metadata from other inspections may be sanitized and selected at an operation 2910 for use in training a damage identification model. At an operation 2912, those images may be categorized for contents (e.g., what type of object is in the photographs) and presence of damages (e.g., is there damage shown in the photographs). In other words, images of objects may be manually categorized for their contents and whether damage is shown and used to train the model for identifying damage. [0084] In addition to using training a model to identify and assess damages and using a trained model to identify and assess damages as shown in the method 2900, images captured during inspections may be added to data sets that are used to further refine and train models for identifying and assessing damage in images captured of objects over time.)
-and generating the one or more models based upon the plurality of historical images. (Nelson [0087] In various embodiments, the images used to train such a model may be pairs of images with elapsed time between them (e.g., two images showing a same object with and without damage, respectively), or may be images unrelated to other images used to train the model. In other words, in an example, a model may be trained by inputting into an untrained machine learning model a plurality of pairs of images associated with a plurality of objects, wherein each pair of the plurality of pairs of images comprises an earlier captured image of an example object of the plurality of objects and a later captured image of the example object. In such examples, a machine learning algorithm may be used that can incorporate the relationship between two photographs of the same object taken at different times into its learning. Models for assessing an amount of damages (e.g., a damage pricing model as discussed below with respect to operations 2916, 2926, and 2928 below) may be trained in similar or different ways.)
Regarding Claims 4, 17:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 1, wherein the at least one processor is further programmed to/The computer-implemented method of Claim 14 further comprising:
Furthermore, Nelson teaches:
-Transmit(ting) instructions to the user computer device for capturing one or more of the plurality of initial images. (Nelson [0034] In an operation 206, the step-by-step instructions for inspecting the property are displayed on an electronic device (e.g., a mobile device like a smartphone or tablet) so that the user can complete the inspection. Examples of user interfaces that may be used to display step-by-step instructions may include the user interfaces shown in FIGS. 16-27. A user may complete the step-by-step instructions by, for example, answering prompts displayed on a user interface, entering notes related to aspects of a property being inspected, taking photographs of objects on the property, identifying damaged portions of objects in the photographs, etc. as described herein. [0049] Inspection indicator 506 shows an example of an inspection that has yet to be started, and may be started by selecting the inspection indicator 506 (as indicated by the “Start Inspection” text that occurs in the inspection indicator 506. The inspection indicator 506 also indicates the type of inspection being performed, such as an annual or periodic inspection. Other inspection types may also be used, such as move-in, move-out, before-remodel, post-remodel, post-repair, etc., so that the property and its various objects may be properly tracked for proper recordkeeping and tracking over time.) These instructions are shown both during move in and move out, therefore the limitation has been taught.
Regarding Claims 6, 19:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 1, wherein the at least one processor is further programmed to/The computer-implemented method of Claim 14 further comprising:
Furthermore, Nelson teaches:
- transmit(ting) instructions to a user computer device associated with a tenant of the location, (Nelson [0034] In an operation 206, the step-by-step instructions for inspecting the property are displayed on an electronic device (e.g., a mobile device like a smartphone or tablet) so that the user can complete the inspection. Examples of user interfaces that may be used to display step-by-step instructions may include the user interfaces shown in FIGS. 16-27. A user may complete the step-by-step instructions by, for example, answering prompts displayed on a user interface, entering notes related to aspects of a property being inspected, taking photographs of objects on the property, identifying damaged portions of objects in the photographs, etc. as described herein. [0049] Inspection indicator 506 shows an example of an inspection that has yet to be started, and may be started by selecting the inspection indicator 506 (as indicated by the “Start Inspection” text that occurs in the inspection indicator 506. The inspection indicator 506 also indicates the type of inspection being performed, such as an annual or periodic inspection. Other inspection types may also be used, such as move-in, move-out, before-remodel, post-remodel, post-repair, etc., so that the property and its various objects may be properly tracked for proper recordkeeping and tracking over time.) These instructions are shown both during move in and move out, therefore the limitation has been taught.
-wherein the instructions identify one or more items to capture in the plurality of current images of at the location. (Nelson [0042] In an operation 306, instructions for capturing a second image may be displayed on the display along with the representation and the first image. Examples of such instructions are shown in FIGS. 19, 21, 23, and 30-37. In FIG. 19, for example, instructions state “Front Door Exterior—While standing outside the front door, take a picture of the entire front door.”)
Regarding Claims 7, 20:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 6, /The computer-implemented method of Claim 19:
Furthermore, Nelson teaches:
-wherein the instructions include an overlay to display on a display screen of the user computer device. (Nelson [0096] FIGS. 30 and 31 are example user interfaces 3000 and 3100, respectively, for overlaying a representation of an object onto a user interface of a mobile computing device, in embodiments. For example, such overlays may be used as described in the operation 304 of FIG. 3. The user interface 3000 shows a first image 3002 that represents a field of view of a camera of an electronic device such as a smartphone. Overlaid on the first image 3002 is a representation 3004 of a refrigerator, and in particular an image of a refrigerator. The overlaid representation 3004 may be used to align the field of view with an actual refrigerator that may appear in the first image 3002 if the actual refrigerator is within the field of view of the camera (though not shown in FIG. 30). In some embodiments, the overlaid representation may be a prior image captured of the actual refrigerator associated with a property. For example, if a property was inspected a year prior to a current inspection, prior photos of objects taken during the prior inspection may be modified to be partially transparent and used as overlays for images captured for the current inspection. As described herein, the overlaid representations may also cause users to capture more consistent and/or better-quality images that may more easily be used to identify and/or assess damage in the captured images.)
Regarding Claims 8, 21:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 6, wherein the at least one processor is further programmed to/The computer-implemented method of Claim 19 further comprising:
Furthermore, Nelson teaches:
-Control(ling) the user computer device to capture the plurality of current images. (Nelson [0067] For example, in FIG. 19, instructions 1904 overlaid onto the first image 1902 explain to the user what to take a picture of. While the user interface 1900 does not show a representation of the front door for aligning a photograph, such a representation may optionally be overlaid as described herein. A user may press a button 1906 to capture a second image that coincides with the first image currently being displayed at the time the button 1906 is pressed. In the alternative, the user may select a skip button 1908 to skip capturing an image at this stage.) The broadest reasonable interpretation (BRI) of this limitation in the plain language that the claim is written includes any control of the user device to capture the initial images, such as pressing a button which then indicates the computing device to capture the images. In view of the specification paragraph [0056], the limitation does not necessarily require an augmented reality overlay, or automatic capture of the images, since the claim language is so broad.
Regarding Claims 9, 22:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 6, wherein the at least one processor is further programmed to/The computer-implemented method of Claim 19 further comprising:
Furthermore, Nelson teaches:
- transmit(ting) the instructions to the user computer device of the tenant on a periodic basis. (Nelson [0049] The inspection indicator 506 also indicates the type of inspection being performed, such as an annual or periodic inspection. [0051] FIG. 6 shows a user interface 600 for a periodic inspection, for example if an inspection indicator in FIG. 5 is selected. The user interface 600 includes a back arrow 602 that may navigate the user back to the user interface 500 of FIG. 5.)
Regarding Claims 11, 24:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 1, wherein the at least one processor is further programmed to/The computer-implemented method of Claim 14 further comprising:
Furthermore, Nelson teaches:
- receiv(ing) a plurality of sets of current images of a first location over a plurality of periods of time; and (Nelson [0023] As described herein, if multiple inspections of a property are performed, the images or other documentation related to those inspections may be analyzed over time to automatically identify changes to the property or objects on the property to determine a material wear and tear depreciation or specific damage value that exceeds wear and tear associated with a property, building, or object on the property. [0074] In addition, other images may be captured of those objects or aspects of a property during subsequent inspections. As described in the method 2800 those subsequent images of a same object may be compared to determine changes to the object, which may be wear and tear due to normal use and passage of time, or may be material damage for which a tenant should be liable for repairs or replacement of the object.)
-compar(ing) the plurality of sets of current images of the first location to determine at least one trend at the first location. (Nelson [0094] As described herein, a machine learning model may also be trained to determine an estimated value associated with the damage to the object. That value may be associated with wear and tear (e.g., a depreciation of the object) or may be associated with material damage for which a tenant should be liable. The material damage value may represent a cost to repair or replace the object. In various embodiments, a landlord or property manager may input the value of various objects when they are installed in a property. In this way, the systems and methods herein may be used to track the value of those objects as they are inspected over time. For example, a washing machine may be purchased for $500, and the systems and methods herein may indicate during an annual inspection that such a washing machine depreciates approximately $50 a year. In other examples, various objects may depreciate at varying rates over time. Thus, a value remaining of an object may be used to determine a replacement cost of the washing machine if a tenant damages it beyond repair, or the new cost of the object may be used as the replacement cost. For example, if the washing machine described above is damaged beyond repair after three years of use, the system may recommend a replacement cost for which a tenant is liable of $350.) Estimating the value overtime falls within the scope of the limitation since it is a determination of a “trend.”
Regarding Claims 12, 25:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 1, wherein the at least one processor is further programmed to/The computer-implemented method of Claim 14 further comprising:
Furthermore, Nelson teaches:
-receiv(ing) a plurality of sets of current images of a first location over a plurality of periods of time; and (Nelson [0074] In addition, other images may be captured of those objects or aspects of a property during subsequent inspections. As described in the method 2800 those subsequent images of a same object may be compared to determine changes to the object, which may be wear and tear due to normal use and passage of time, or may be material damage for which a tenant should be liable for repairs or replacement of the object.)
-calculat(ing) a score for a tenant at the first location based upon the plurality of sets of current images of the first location. (Nelson [0112] Other features related to the inspection systems and methods described herein may also be implemented in various embodiments. For example, the systems and methods herein may be used to determine a score for any of landlords, property managers, and/or renters. For example, parties may rate other parties they interact with based on their experiences. For example, a renter may rate their landlord and vice versa. In addition, the parties’ scores may be adjusted based on data collected and/or entered related to inspections. For example, if after an inspection, an object is determined to be damaged beyond wear and tear, a renter’s score may be adjusted downward based on damage to the object during the time in which the renter occupied the property. In some embodiments, an extent of damage to a property may be used to adjust a renter score up or down, such that a renter score may improve or decrease over time depending on how well the renter took care of properties they rented. Thus, a renter score may indicate how likely it is that the renter will cause more than wear and tear damage to a property.)
Regarding Claims 13, 26:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 1, wherein the at least one processor is further programmed to/The computer-implemented method of Claim 14 further comprising:
Furthermore, Nelson teaches:
-receiv(ing) a plurality of sets of current images from one or more locations associated with a landlord; and (Nelson [0074] In addition, other images may be captured of those objects or aspects of a property during subsequent inspections. As described in the method 2800 those subsequent images of a same object may be compared to determine changes to the object, which may be wear and tear due to normal use and passage of time, or may be material damage for which a tenant should be liable for repairs or replacement of the object.)
-calculat(ing) a score for the landlord based upon the plurality of sets of current images of the one or more locations.( Nelson [0112] Other features related to the inspection systems and methods described herein may also be implemented in various embodiments. For example, the systems and methods herein may be used to determine a score for any of landlords, property managers, and/or renters. [0117] Landlord ratings may also be adjusted based on condition of property and objects on the property as determined by automatic image analysis as described herein.)
Regarding Claim 28:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 1, wherein the at least one processor is further programmed to:
Furthermore, Nelson teaches:
-determine a predefined period of time has elapsed since a plurality of previous images of the location were captured; and(Nelson [0083] In various embodiments, the automated system for identifying and assessing damage in an image may consider an amount of elapsed time between when a first image was captured of an object and when a second image of the object was captured. In addition, renters and landlords may choose to each perform an inspection around the same time (e.g., around move-in, move-out, etc.). Inspections performed within a predetermined threshold of time (e.g., within a day, within a week, within a month) of one another may be aligned inspections.)
-transmit a prompt to the user computer device to initiate the capture of the plurality of current images. (Nelson [0049] Inspection indicator 506 shows an example of an inspection that has yet to be started, and may be started by selecting the inspection indicator 506 (as indicated by the “Start Inspection” text that occurs in the inspection indicator 506. The inspection indicator 506 also indicates the type of inspection being performed, such as an annual or periodic inspection. Other inspection types may also be used, such as move-in, move-out, before-remodel, post-remodel, post-repair, etc., so that the property and its various objects may be properly tracked for proper recordkeeping and tracking over time. As also shown in the user interface 500, other inspection indicators may indicate inspections at various states of completion (e.g., “Continue Inspection”). Other state indicators could be “In-Progress,” “Completed,” “Not Started,” etc. [0018] A user of a mobile application may be presented with a number of prompts based on the type of property being inspected. At each prompt, the user may be asked to provide a response to said prompt, which may be an answer to a question or a picture taken using the camera of the mobile device.)
Regarding Claim 29:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 28, wherein the at least one processor is further programmed to:
Furthermore, Nelson teaches:
- transmit further prompts to the user computer device according to a predefined periodicity, each further prompt requesting the initiation of capture of a respective plurality of periodic current images;(Nelson [0119] For example, inspections may be designed to focus more on items that are determined to be more likely to be damaged by a renter. In another example, more frequent and brief inspections may focus only on problem areas identified from aggregated data, while less frequent and exhaustive inspections may inspect and document all aspects of a property. Thus, the most problematic areas of a property that are likely to be damaged may be inspected more often. [0049] (as indicated by the “Start Inspection” text that occurs in the inspection indicator 506. The inspection indicator 506 also indicates the type of inspection being performed, such as an annual or periodic inspection. Other inspection types may also be used, such as move-in, move-out, before-remodel, post-remodel, post-repair, etc., so that the property and its various objects may be properly tracked for proper recordkeeping and tracking over time.)
-receive, in response to each further prompt, the respective plurality of periodic current images; and(Nelson [0082] Although the method 2800 describes comparing two images, additional images may be considered and/or compared by the system. For example, additional images taken at different times from the first and second images may be compared to determine damage and/or wear and tear to an object over time. In other examples, the system may be configured to receive multiple contemporaneously taken images of an object to compare to one or more images of the object taken at a different time. For example, multiple images of an object may be taken from different angles, and those images may be used to compare to one or more images of the objects captured at a different time.)
-store each plurality of periodic images linked to a timeline of when the respective plurality of periodic images was captured, relative to one another and to the plurality of current images.(Nelson [0092] In such embodiments, times at which each of the images in a pair of images were taken or an elapsed time indicating the time passed between the earlier captured image and the later captured image of the image pairs may also be input. This, together with other inputs such as damage value, type of damage, object type, etc., may train the model to more accurately identify and assess damage in images based on an amount of time that has passed between the capture of images... Accordingly, identifying objects in images and tracking them over time may be helpful in assessing the extent of and value of damage. Thus, inputting images with data indicating their relationship to one another (whether two images show the same object and the time passed between the capture of the images) may be helpful in adequately identifying and assessing damages. Thus, when inspections are performed as described herein, it may be valuable to determine, either through manual tagging, automatic tagging based on image recognition, or any other method, when images taken at different times are of the same object. In this way, a trained model may be able to better track the extent and value of damage sustained to various objects. Thus, the methods and systems herein may accommodate inputs from a user to indicate when objects are replaced so that the system may begin the timeline for tracking damage to an object again. [0036] In an operation 210, the images captured are timestamped and stored. The images may be timestamped at or near the moment the images are captured. The images may be stored in memory of a device which captured the images. In various embodiments, the images along with their timestamps may be stored in other devices such as servers. [0049] Other inspection types may also be used, such as move-in, move-out, before-remodel, post-remodel, post-repair, etc., so that the property and its various objects may be properly tracked for proper recordkeeping and tracking over time)
Regarding Claim 31:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 1, wherein the at least one processor is further programmed to:
However, Nelson fails to teach:
-receive audio information from the real-time video stream; and
-analyze the audio information to detect one or more further issues at the location.
Alternatively, Chavez teaches:
-receive audio information from the real-time video stream; and(Chavez [Col. 4 Lines 44-58] Still further, remote device 204 can include speakers and a microphone for receiving and generating audible sounds. In the exemplary embodiment of FIG. 2, remote device 204 comprises a tablet computing device. In other embodiments, however, a remote device could comprise a smartphone, a laptop, or similar kind of device. (22) Remote device 204 may include hardware components for capturing sensory information, as well as storing and/or transmitting captured information. As used herein the term “sensory information” can include visual information, audible information, tactile information and/or information related to the motion of the remote device (for example, acceleration information). In an exemplary embodiment, remote device 204 includes a camera for capturing images in the form of photos or video.)
-analyze the audio information to detect one or more further issues at the location. (Chavez [Col. 12 Lines 14-27] To detect and classify structures and/or damage, the embodiments may utilize a machine learning system. As used herein, the term “machine learning system” refers to any collection of one or more machine learning algorithms. Some machine learning systems may incorporate various different kinds of algorithms, as different tasks may require different types of machine learning algorithms. Generally, a machine learning system will take input data and output one or more kinds of predicted values. The input data could take any form including image data, text data, audio data or various other kinds of data. The output predicted values could be numbers taking on discrete or continuous values. The predicted values could also be discrete classes (for example, a “damaged” class and an “undamaged” class).)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present disclosure to modify Nelson by adding the teachings of Chavez including using the audio from the video stream as another source of input data to determine one or more issues. One of ordinary skill in the art would have been motivated to perform this combination by the benefit of using more sources of information to make a more accurate detection of damage (Chavez [Col. 3 Lines 4-12] By automatically capturing and analyzing image information about structures in the physical space to determine if there is damage, the system and method improve the efficiency of the inspection process. By using an augmented reality system to prompt a user, the system and method simplify the inspection process and allow users with little to no experience in inspecting properties to quickly and accurately assess possible damage in a physical space.)
Claims 10, 23, and 30 are rejected under 35 U.S.C. 103 as being unpatentable over Nelson (US 20200410278 A1), in view of Chavez (US 11600063 B1), further in view of Fields et al. (US 20240053816 A1) hereinafter Fields, further in view of Jorey E. Ramer (US 20230141593 A1) hereinafter Ramer.
Regarding Claims 10, 23:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 1, wherein the at least one processor is further programmed to/The computer-implemented method of Claim 14 further comprising:
However, neither Nelson, Chavez, nor Fields teach or suggest:
-schedule the at least one appointment with a service provider to resolve the one or more issues. (Both Nelson and Chavez merely teach the inspection of the location but not the process of resolving any detected problems)
Alternatively, Ramer discloses a system of sending service calls to maintenance and repair services based on sensor data. Ramer teaches:
-schedule the at least one appointment with a service provider to resolve the one or more issues (Ramer [0070] In example embodiments, the data associated with the entities at each property of each subscriber may be sensor data sent from one or more entities at the properties. In example embodiments, the prediction engine may analyze the sensor data to determine whether the entities may be in need of repair or servicing...[0239] When the action type is “proactive execution,” the sensor management system 250 may configure the platform 10, in example embodiments, to automatically schedule a service visit. [0334] The claim adjudication system 260 may receive the request for repair service, and may automatically adjudicate the request. For this purpose, in one example, the system 260 may associate a job with the service request and designate an appropriate servicer to perform the job. [0335] Additionally and/or alternatively, the claim adjudication system 260 may also pre-authorize or automatically enable the servicer to perform the repair service at the home, without the servicer requiring manual approval to perform the repair.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the present disclosure to modify Nelson by adding Ramer’s features of automatically determining and predicting what services are needed from sensor data and automatically scheduling a service visit in response to the detection of a problem. Nelson’s system already determines a type of damage and predicted cost of damage, therefore a combination would yield the predictable outcome of contacting the appropriate service provider based on the damage. One of ordinary skill in the art would have been motivated to combine as it would provide the benefit of automatically finding high quality repair services at fair prices and with predictable costs (Ramer [0009] As a result, a need exists for methods and systems that improve the consumer/homeowner experience of scheduling repair services, such as by assuring the provision of available, high quality repair services at fair prices and with predicable costs.)
Regarding Claim 30:
The combination of Nelson, Chavez, and Fields teach The computer system of Claim 31, wherein the at least one processor is further programmed to:
However, neither Nelson, Chavez, nor fields teach or suggest:
-prioritize each issue based on a respective urgency of the issue; and
-schedule the at least one appointment including a first appointment to address a first priority issue.
However, Ramer teaches:
-prioritize each issue based on a respective urgency of the issue; and(Ramer [0237] Upon finding a matching alert message in the action table based upon the sensor data 321, the prediction engine 252 may include the contents of the alert in a message, may send the message to various stakeholders, and/or execute a set of actions in response. The actions to take may incorporate varying levels of urgency and an action type (e.g., recommendation or proactive execution). Example actions may include recommending scheduling of a service job on a next visit, recommending scheduling a visit within a time period (e.g., within the next two weeks), recommending an immediate service call, recommending an action pending a service call (e.g., stopping usage of a potentially damaged item to prevent further harm, replacing an element (e.g., a filter or battery, or the like), and/or recommending complete replacement of an item, among others. [0254This may include adjusting the weights of a set of parameters in a scheduling rule, such as to rank order a set or servicers for a job.)
-schedule the at least one appointment including a first appointment to address a first priority issue.(Ramer [0237] Example actions may include recommending scheduling of a service job on a next visit, recommending scheduling a visit within a time period (e.g., within the next two weeks), recommending an immediate service call,) These examples of recommending an immediate service call, or scheduling a job on the next visit satisfy “scheduling...to address a first priority issue.”
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present disclosure to further modify Nelson and Chavez by adding Ramer’s features of prioritizing issues based on levels of urgency, and scheduling an immediate appointment for urgent items. By adding Ramer’s system at the end of Nelson’s system, one would arrive at the predictable outcome of performing the limitations of claim 30 because by the end of Nelson the problem is already identified and Ramer merely performs the prioritization and scheduling steps. One of ordinary skill in the art would have been motivated to perform the combination by the benefit of preventing further harm, and improving the efficiency of scheduling an appointment with a servicer. (Ramer [0129] Where the host system 100 is permitted to schedule the first appointment of the window, it can determine if an actual appointment time has an effect on customer satisfaction. If the host system 100 schedules most or all of the appointments within a window, it can determine the most efficient routing, and it can use time estimation to reduce the appointment window (or provide an actual appointment time) for a consumer.)
Response to Arguments
Applicant's arguments filed 07/06/2026 have been fully considered but are not persuasive for the following reasons:
Regarding arguments over rejections under 35 U.S.C. 101, the applicant’s arguments have been fully considered but are not persuasive. Firstly, the applicant submits that the pending claims are eligible for reasons similar to the ARP Decision (Ex Parte Desjardins), however, the examiner respectfully disagrees. The applicant asserts that page 8 of ARP decision identifies “improvements in training the machine learning model itself,” and page 9 explains that the improvements are reflected in the claims that allow systems to “use less of their storage of capacity,” and enables “reduced system complexity.” However, while the examiner acknowledges the alleged improvements in [0002] and [0003] in the specification, the examiner does not find these improvements to be the same type of improvements that enabled Ex Parte Desjardins to be eligible over 101, which were specifically directed to improvements to the training of machine learning models itself. While the present specification may include training related steps, the scope of the present claims does not recite any training steps, because the trained machine learning models are claimed without specific training steps. Furthermore, the claims are claimed in an outcome-based manner, not a technical manner that provides any specific improvements over existing machine learning techniques. One cannot show an improvement to machine learning, without claiming the specific steps necessary to make it apparent to one of ordinary skill in the art that the claim realizes the improvement. Therefore, the applicant’s argument that the “above-noted aspects of the Specification are reflected in amended Claim 1 at least by the following recitations…” is not persuasive because none of the steps actually recited in the claim language provide any improvements to machine learning itself or any technology or technical field. While the claims are read in view of the specification, limitations from the specification are not read directly into the claims. Furthermore, the applicant’s argument that the claims reflect improvements to the “overall technology field of image analysis and assessment,” is not persuasive because “image analysis” is not a specific technical field within the computing arts, and thus providing improvements to analyzing images, in the context of home inspection is merely an improvement to the abstract idea itself, not an improvement to a technical field.
Furthermore, determining whether images are “good enough,” for further processing, is not a technical step because it does not recite a specific technical manner in determining whether the images are sufficient, it is merely claiming such filtering in an abstract manner, therefore, this can’t possibly provide an improvement to a technical field. Furthermore, based on the initial output, selected a particular type of model is not an improvement to machine learning models, because it is not an improvement to how they are trained, nor does it use machine learning to improve a computer component, it is merely reciting the selection of a type of machine learning, without meaningfully limiting how the selection occurs. Therefore, even when taken as a whole, the claims do not integrate the abstract idea into a practical application or provide significantly more because the steps of “ensuring suitability quality of images,” “using a trained model to identify a fixture,” and “selecting one specific classification model to classify a type of issue,” is merely a series of abstract steps which claim the intended outcome but do not specify how the invention arrives at the outcome. Therefore, the applicant’s argument that the claims “represent an improvement in the technical field of image processing that leverages the improved application of artificial intelligence,” rather than being a generic use of artificial intelligence to implement the abstract idea is not persuasive because the claims merely claim machine learning by name, without any specific training steps, or any potential improved technique that can be analyzed as a technical improvement. Therefore, the claims stand rejected under 35 U.S.C. 101.
Regarding arguments over rejections under 35 U.S.C. 103, the applicant’s arguments have been fully considered but are not persuasive in view of the updated rejection which is now based upon a combination of Nelson, Chavez, and Fields. Each and every limitation of claims 1, 14 and 27 is taught or suggested by the combination, therefore, the applicant’s arguments that neither Nelson nor Chavez teach or suggest each and every limitation is moot, because the combination of Nelson, Chavez and Fields satisfies (i) compare images to reference images to ensure they are suitable for model-based image processing, (ii) only when the images are indeed suitable, execute a (first) model to identify a fixture depicted in the compared images and differences in the fixture over time, (iii) based on the identified fixture, select a (second) model specifically applicable to that type of fixture, and (iv) execute that (second) model to identify an issue with the fixture, and (v) automatically schedule appointments to remediate that identified issue.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
-Mariotti et al. (US 20230334586 A1) discloses a remote real estate property inspection technique that uses images and other sensor data to identify objects within the property, identifying the data associated with the objects, identifying preventative maintenance, and provide a transparent history of the property.
-Gonzales et al. (US 20220383128 A1) discloses training an inspection model for physical products to identify an anomaly on a product that would prevent the product from meeting quality standards, wherein the various classification models are trained based on objects that are the same type as the inspected object.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICO LAUREN PADUA whose telephone number is (703)756-1978. The examiner can normally be reached Mon to Fri: 8:30 to 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jessica Lemieux can be reached at (571) 270-3445. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NICO L PADUA/Junior Patent Examiner, Art Unit 3626
/RASHIDA R SHORTER/Primary Examiner, Art Unit 3626
Prosecution Insights