DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 27 February, 2025 has been entered.
Response to Amendment
Claims 1-20 are pending. Claims 1-20 are amended directly or by dependency on an amended claim.
Response to Arguments
Applicant’s arguments, see pages 6-8, filed 27 February, 2025 with respect to the 35 USC 103 rejections of claim(s) 1 and 11 have been considered but are moot because the new ground of rejection does not rely on the combination of references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Examiner recommends if there is a particular way the percentage of each type of damage is calculated that is different from the Ruda reference applied and the references cited in the conclusion, this could potentially improve conditions for allowance.
All other arguments are by similarity or dependency and are addressed by the above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 6, 9, 11, 12, 16, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Okazaki (US 20180089763 A1) in view of Hakimi-Boushehri et al. (US 10607295 B1) in view of Ruda et al. (US 20180247416 A1).
Regarding claims 1 and 11, Okazaki discloses a computer vision system and computer vision method for determining a condition of a roof from an image (Two-dimensional aerial images, in some examples, can be used to determine location coordinates of the property, street name, occupancy type, floor area, existence of skylights, existence of chimneys, roof condition, roof shape, roof covering, roof anchors, roof equipment, and/or pounding. Three-dimensional aerial images, in comparison, can be used to determine location coordinates, street name, construction type, occupancy type, year built, building height, soft stories, number of stories, roof condition, roof shape, roof covering, roof anchors, roof equipment, cladding, and pounding. Where there is an overlap in characteristics identifiable using either a two-dimensional image or a three-dimensional image, in some embodiments, machine learning analysis of both images can be combined to provide increase confidence in identification of the individual characteristics, [0041]) comprising: an image database storing at least one image of a roof (multiple remote database systems to obtain both a two-dimensional aerial image and a three-dimensional aerial image, roof condition, roof shape, [0041]); and a processor in communication with the image database, the processor: retrieving the image of the roof from the database (query multiple remote database systems to obtain both a two-dimensional aerial image and a three-dimensional aerial image, roof condition, roof shape, [0041]); processing the image of the roof to determine a footprint of the roof using a first neural network trained to segment roof structures (deep learning methodology could be applied to risk exposure database population to analyze aerial imagery and automatically extract characteristics of individual properties, [0007], “As illustrated in FIG. 1, in some implementations, the system identifies (106) features of each aerial image 102c to classify property characteristics. Using machine learning for analysis, for example, the system can extract features of the aerial image of a particular property location 102b. Groupings of extracted features, such as angles, outlines, substantially homogenous fields, etc. can be used to identify property features such as rooftop, swimming pool, chimney, and sky lights. Upon extracting image-related features identifiable as a property feature (e.g., rooftop), the extracted features may be analyzed to determine one or more property characteristics of that feature (e.g., type of rooftop). In one example, a swimming pool may be identified as the property feature of a pool (e.g., various shapes of outlines of a particular size or greater bordering a substantially homogenous field of blue), then characterized as a particular shape (e.g., rectangular, round, or kidney bean, etc.), type (e.g., above ground or in-ground), and/or size (e.g., approximate area). The machine learning classifier used in the machine learning analysis, in some embodiments, includes a convolutional neural network (CNN) to preprocess the aerial image 102c of the particular property location 102b and to classify the property features as property characteristics 110,” [0045], A test was conducted using NIN deep learning algorithms to classify rooftop shape in two-dimensional aerial images including a mix of gambrel, gable, hipped, square, and flat roof shapes of assorted colors, [0048], Further to roof shape, in some embodiments, feature analysis can be used to discern additional roof features such as, in some examples, roof covering, roof anchors, roof equipment, skylights, widow's walks, turrets, towers, dormers, and/or chimneys. Furthermore, upon identifying the outline of the roof, a footprint of the property location 102b (e.g., size of the roof) can be calculated based upon a scale of the aerial image 102c, [0050]); determining at least one condition of the roof using a second neural network trained to segment roof structure condition features (“Systems, methods, and computing system platforms described herein support matching aerial image features of one or more properties to corresponding property conditions (e.g., maintenance levels of property features) through machine learning analysis,” [0009], “In some implementations, a condition of each property feature may be classified (112) as a corresponding condition characteristic 116. New properties are in good condition, but property feature conditions can deteriorate over time due to normal wear-and-tear on the property. Further, property features can suffer damage due to external forces such as storms and natural disasters. Eventually, conditions of housing features can deteriorate to the point where repair and/or replacement may be necessary. As with property characteristics described above at block 106, machine learning algorithms can be used to classify a present condition of individual detected property features. Using machine learning for analysis, for example, the system can extract pixel intensity distributions of previously identified property features of the aerial image of the particular property location 102b. In some examples, newly constructed property features generally have sharp contrast and well-defined features in machine learning image analysis. Conversely, weathered or damaged property features can have softened edges, blurred contrasts, and asymmetrical patches of wear. The machine learning classifier used in the machine learning condition analysis, in some embodiments, includes a machine learning analysis to process the aerial image 102c of the particular property location 102b and to classify the condition of previously identified property characteristics 110 as condition characteristics 116. The machine learning analysis, in some examples, can include two-dimensional color histogram analysis or three-dimensional color histogram analysis. In other embodiments, the machine learning analysis may be performed using pattern recognition algorithms (e.g., determining missing fence posts or missing/misaligned rooftop shingles). In other embodiments, the machine learning classifier includes deep learning analysis such as CNN or NIN”, [0053]); and generating and transmitting a roof condition report indicating the at least one condition of the roof and a respective contribution of the at least one condition toward a total roof structure (Rather than sharing cost estimate data 350 and/or risk estimate data 352 directly with the clients 306, in other embodiments, the system 302 may include a report generation engine (not illustrated) that prepares reports regarding condition, damage, and risk assessments of one or more properties. Further, in some embodiments, the system 302 may compare a current condition characteristic 344 to a historic condition characteristic 344 to confirm whether a property owner made repairs to a property (e.g., based upon payment of an insurance claim). Other modifications of the system 302 are possible, [0091]).
Okazaki does not disclose a respective percentage of contribution of the at least one condition toward a total roof structure.
Hakimi-Boushehri et al. teach a respective contribution of the at least one condition toward a total roof structure (“The present methods and systems may also use the data gathered by the smart home controller within a property to determine the sequence of events that led to the loss. Determining the events that led to the property damage may be performed for individual properties. The methods and systems may determine that one source of damage (e.g., water) should be allocated a certain percentage of the loss or property damage, and another source of damage (e.g., wind) should be allocated another percentage of the loss or property damage”, col. 6, Iines 1-10; “After determining the primary and secondary causes of loss, the smart home controller may assign a portion of the overall damage to the first and second causes of loss. In some embodiments, the primary cause of loss may be assigned a larger portion of the overall damage than the secondary cause of loss. Accordingly, the smart home controller may assign each cause of loss identified, a percentage of the overall amount of damage. For instance, 80% of total damage to the property may be assigned to the tree branch falling through the roof, with the remaining 20% of damage being assigned to the subsequent water damage. In some embodiments, the primary cause of loss may be assigned 100% of the overall damage… As an example, the homeowner's policy may cover 100% of roof damage and 50% of water damage. The smart home controller may determine that the overall damage to the property is $10,000 and that 80% of the damage is assigned to the falling branch and 20% assigned to the water damage. Accordingly, the generated proposed insurance claim may indicate that a total $8,000 (10,000*1*0.8) may be recovered for the roof damage and $1,000 (10,000*0.5*0.2) may be recovered for the water damage, thus enabling the homeowner to recover an overall amount of $9,000 (8,000+1,000)”, col. 16, lines 30-65
PNG
media_image1.png
741
412
media_image1.png
Greyscale
).
Okazaki and Hakimi-Boushehri et al. are in the same art of assessing property damage (Okazaki, abstract; Hakimi-Boushehri et al., abstract). The combination of Hakimi-Boushehri et al. with Okazaki enables calculating a respective contribution of the at least one condition toward a total roof structure. It would have been obvious at the time of filing to combine the calculation of Hakimi-Boushehri et al. with the invention of Okazaki as this was known at the time of filing, the combination would have predictable results, and as Hakimi-Boushehri et al. state “The systems and methods therefore may offer a benefit to an insurance provider by enabling insurance claims to be more accurately estimated and/or more expediently processed as facilitated by the present embodiments. This increased accuracy and expediency may save the insurance provider money that was wasted on covering damage that was out-of-policy or time spent dispatching claims adjusters to the affected property. As such, the savings may be passed onto customers in the form of offering cheaper insurance products or through the creation of new insurance products better tailored to meet the needs of homeowners. Further, the methods and systems described herein may improve the technological fields of insurance, urban planning, disaster relief, building development, and/or others” (col. 7, lines 25-40) thus indicating a benefit to both insurance companies and customers when a specific cause of the total damage can be ascertained.
Okazaki and Hakimi-Boushehri et al. do not explicitly disclose expressing as a percentage.
Ruda et al. teach a respective percentage of contribution of the at least one condition toward a total roof structure (“Example damage types may include, but are not limited to, weather-related damage such as damage attributable to hail, water, wind, lightning, etc. For example, different types of weather conditions may result in missing sections (e.g., tiles, shingles, etc.), flipped or bent sections, hail markings, or the like”, [0052], “In some embodiments, damage analyzer 304 may also base the measure of consistency on a comparison between the detected extent and type of damage with the weather event. For example, damage analyzer 304 may further assess the distribution of damage on a given slope of the roof, the localization of the damage (e.g., middle vs. edge of shingles, etc.), the direction that the slope faces, the pitch of the slope, or any other damage features, in view of the specific details of the weather event (e.g., wind directions, wind strengths, etc.)”, [0053], “As noted above, the machine learning process of damage analyzer 304 may be trained to classify whether a given image depicts damage and, if so, the extent and/or type of damage present in the input image. For example, a label may be applied to a given portion of the image (e.g., a colored box, etc.) that indicates whether damage is present and the type of damage. In turn, this labeled information can be used to train the classifier of damage analyzer 304 to identify areas of damage on a given roof, as well as the type/cause of the damage,” [0079], “In various embodiments, the classifier may be configured to detect damage done to a particular one of the shingles or tiles depicted in a particular subdivision image under analysis and to assign a damage type to the detected damage. For example, the classifier may be configured to detect damage to a shingle or tile within a subdivision image and label the damage as hail damage, ice damage, wind damage, wear and tear damage, and/or human-made damage”, [0096], “At step 1125, as detailed above, the device may send display data for display that is indicative of an extent of damage to the rooftop associated with the assigned damage type. For example, the extent of damage may correspond to an area measurement (e.g., 400 square feet of damage, etc.), area percentage (e.g., 40% of the rooftop is damaged, etc.), shingle or tile count (e.g., 200 damaged shingles, etc.), or the like. In some cases, the display data may also indicate further information about the rooftop, such as material type, total area, etc. In doing so, the user is able to quickly diagnose the damage and determine which repairs are needed to the rooftop”, [0097]) [Note Ruda reference specific to roof damage, and damage can be identified down to per tile as a specific type of damage].
Okazaki and Hakimi-Boushehri et al. and Ruda et al. are in the same art of assessing property damage (Okazaki, abstract; Hakimi-Boushehri et al., abstract; Ruda et al., abstract). The combination of Ruda et al. with Okazaki and Hakimi-Boushehri et al. enables expressing as a percentage. It would have been obvious at the time of filing to combine the percentage of Ruda et al. with the invention of Okazaki and Hakimi-Boushehri et al.as this was known at the time of filing, the combination would have predictable results, and as Ruda et al. state “After the occurrence of a storm, assessing weather-related damage to buildings, vehicles, etc., is a potentially dangerous and error-prone activity. In particular, assessing roof damage to a building may require an inspector to physically climb onto the roof and take pictures of the damage. In doing so, this exposes the inspector to potential injury, such as when proper safety procedures are not followed. Further, there is no guarantee that this process will correctly assess the extent of the damage and/or the type of the damage” ([0005]) and “According to the techniques described herein, systems and methods are disclosed in which machine learning is used to assess images of weather damage. In some aspects, the techniques herein may assess and identify the features of a roof or other structure (e.g., building façade, etc.), such as the type of roof, shingle type, and/or shingle count. In further aspects, the techniques herein may assess and identify the features of any damage to the structure, such as the type and extent of the damage” ([0019]) thereby improving the claim assessments of Okazaki and Hakimi-Boushehri et al. which will result in safer assessments and potential cost savings for the insurance company.
Regarding claims 2 and 12, Okazaki and Hakimi-Boushehri et al. and Ruda et al. disclose the system and method of claims 1 and 11. Okazaki further discloses the processor receives a geospatial region of interest (ROI) specified by a user and retrieves the image of the roof from the image database using the geospatial region of interest (Two-dimensional aerial images, in some examples, can be used to determine location coordinates of the property, [0041], The overlay of the aerial image 102c with the shape map image 102a, further, can be used in aiding in cropping the aerial image 102c to focus analysis on a particular property location 102b. For example, turning to FIG. 5C, property location 524 is substantially aligning with the corresponding planning map image, and the general outline can be used in aiding in cropping of the property location 524, [0043], the image may be cropped to include the property of interest, or the property of interest plus a portion of its surroundings, [0049]).
Regarding claims 6 and 16, Okazaki and Hakimi-Boushehri et al. and Ruda et al. disclose the system and method of claims 1 and 11. Okazaki further indicates the processor obtains the footprint of the roof from a roof structure footprint database in communication with the processor (one or more processors may execute these system functions, where the processors are distributed across multiple components communicating in a network, [0018], remote databases, [0039]-[0041], database includes roof shape/footprint, [0041], property characteristic classification and condition analysis system 302 in communication with client computing system, [0065], cloud computing environment 730 may also include one or more databases 738, [0119]).
Regarding claims 9 and 19, Okazaki and Hakimi-Boushehri et al. and Ruda et al. disclose the system and method of claims 1 and 11. Hakimi-Boushehri et al. further indicate the respective contribution of the at least one condition toward the total roof structure comprises a percentage of composition of the total roof structure (“After determining the primary and secondary causes of loss, the smart home controller may assign a portion of the overall damage to the first and second causes of loss. In some embodiments, the primary cause of loss may be assigned a larger portion of the overall damage than the secondary cause of loss. Accordingly, the smart home controller may assign each cause of loss identified, a percentage of the overall amount of damage. For instance, 80% of total damage to the property may be assigned to the tree branch falling through the roof, with the remaining 20% of damage being assigned to the subsequent water damage. In some embodiments, the primary cause of loss may be assigned 100% of the overall damage… As an example, the homeowner's policy may cover 100% of roof damage and 50% of water damage. The smart home controller may determine that the overall damage to the property is $10,000 and that 80% of the damage is assigned to the falling branch and 20% assigned to the water damage. Accordingly, the generated proposed insurance claim may indicate that a total $8,000 (10,000*1*0.8) may be recovered for the roof damage and $1,000 (10,000*0.5*0.2) may be recovered for the water damage, thus enabling the homeowner to recover an overall amount of $9,000 (8,000+1,000)”, col. 16, Iines 30-65)
Claim(s) 3, 4, 13 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Okazaki (US 20180089763 A1) and Hakimi-Boushehri et al. (US 10607295 B1) and Ruda et al. (US 20180247416 A1) as applied to claims 1 and 11 above, further in view of Kottenstette et al. (US 20190213413 A1).
Regarding claims 3 and 13, Okazaki and Hakimi-Boushehri et al. and Ruda et al. disclose the system and method of claims 1 and 11. Okazaki and Hakimi-Boushehri et al. and Ruda et al. do not disclose processes the image of the roof using neural network segmentation processing to generate a single channel image that maps each pixel in the image to a binary classification indicative of whether each pixel is or is not representative of a roof structure.
Kottenstette et al. teach processes the image of the roof using neural network segmentation processing to generate a single channel image that maps each pixel in the image to a binary classification indicative of whether each pixel is or is not representative of a roof structure (“An example of bottom-up object detection is image segmentation. Image segmentation is a process where pixels in an image are labeled in such a way that pixels sharing the same characteristics (e.g., color, texture, proximity) have the same label. In this way, an image is parceled into distinct segments, typically in a bottom-up approach, where low-level local image properties are used to detect coherent regions”, [0096], A pixel that is part of a road can be labeled as the road class, while another pixel that is part of a roof can be labeled as the roof class, [0121], In a two-class label, each color can correspond to whether a specific pixel corresponds to the specified object. Any suitable color scheme can be used. For example, in identifying a roof in an image, if a black and white scheme is used, white can be used to identify a roof, and black can be used to identify any other object that is not a roof, or vice versa, [0129], “At step 306, a classifier can be created based on the received training images and the received labels. The classifier can be configured to identify the class of each pixel of an image. In some embodiments, the classifier can be created using a machine learning system, which includes one or more classifiers such as an artificial neural network (ANN) including but not limited to a convolutional neural network (CNN), as would be appreciated by one of ordinary skill in the art”, [0140], binary foreground mask, [0150], “Yet in another example, unless otherwise noted, the term image can cover any kind of image from any detectable band of the electromagnetic spectrum, including IR, UV, panchromatic, multi-spectral, or hyperspectral images. An image can be RGB, grayscale, or black and white. An image can be any resolution, including high resolution and low resolution. An image may be taken by various methods and/or devices, including satellites, drones, robots, stationary or moving camera or sensor devices, and humans using a camera or sensor device. An image may be taken as a digital image or as an analog image. An image may have been originated as an analog image but later converted to a digital format. An image can indicate a 2D array of pixels with one or more mutually co-registered channels or bands per pixel, summarizing the same area. Non-limiting examples include (1) a 1-channel image that can be a panchromatic image, a grayscale image, a label map, a binary mask, or any other suitable image with one channel”, [0304]).
Okazaki and Hakimi-Boushehri et al. and Kottenstette et al. are in the same art of assessing property damage (Okazaki, abstract; Hakimi-Boushehri et al., abstract; Kottenstette et al., [0284]). The combination of Kottenstette et al. with Okazaki and Hakimi-Boushehri et al. and Ruda et al. enables using a single channel image and binary classification. It would have been obvious at the time of filing to combine the single channel image and binary classification of Kottenstette et al. with the invention of Okazaki and Hakimi-Boushehri et al. and Ruda et al. as this was known at the time of filing, the combination would have predictable results, and as Kottenstette et al. state “Thus, method 500B can result in a more accurate binary foreground mask 510B than binary foreground mask 510A produced by prior art method 500A. Moreover, unlike prior art method 500A, method 500B can be scalable to process a large number of pixels” ([0150]), that this provides efficient classification ([0151], [0154]) and it is known different channels can be used to better assess different image features, thus indicating adding the processes of Kottenstette et al. will improve the detection and segmentation accuracy and efficiency of Okazaki and Hakimi-Boushehri et al. and Ruda et al..
Regarding claims 4 and 14, Okazaki and Hakimi-Boushehri et al. and Ruda et al. and Kottenstette et al. disclose the system and method of claims 3 and 13. Okazaki and Kottenstette et al. further indicate the processor executes a contour extraction algorithm on the single channel image to determine the footprint of the roof structure (Okazaki, Further to roof shape, in some embodiments, feature analysis can be used to discern additional roof features such as, in some examples, roof covering, roof anchors, roof equipment, skylights, widow's walks, turrets, towers, dormers, and/or chimneys. Furthermore, upon identifying the outline of the roof, a footprint of the property location 102b (e.g., size of the roof) can be calculated based upon a scale of the aerial image 102c, [0050]; Kottenstette et al., single channel image, [0304]).
Claim(s) 5 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Okazaki (US 20180089763 A1) and Hakimi-Boushehri et al. (US 10607295 B1) and Ruda et al. (US 20180247416 A1) and Kottenstette et al. (US 20190213413 A1) as applied to claims 4 and 14 above, further in view of Chen et al. (US 20150130797 A1).
Regarding claims 5 and 15, Okazaki and Hakimi-Boushehri et al. and Ruda et al. and Kottenstette et al. disclose the system and method of claims 4 and 14. Okazaki further indicates contour extraction ([0050]) and Ruda et al. teach “In another embodiment, a more adaptable approach uses a trained CNN to indicate pixels that look like “edges” of shingles or tiles. In other words, shingle/tile counter 306 may use a deep learning network that takes as input small portions of images 312 and classifies them according to whether they are located on the boundaries of shingles or tiles” ([0058]) however Okazaki and Hakimi-Boushehri et al. and Kottenstette et al. and Ruda et al. do not explicitly disclose the contour extraction algorithm determines pixel boundary locations of the roof structure.
Chen et al. teach the contour extraction algorithm determines pixel boundary locations of the roof structure (“The threshold will operate to divide the pixels into groups defined by pixels having values above and below the threshold. For example, a threshold value of 50 may be established, and all pixels having a value below 50 are designated as black, and all pixels having a value above 50 may be designated as white. The white pixels may be determined to have elevation, and black pixels may be determined to not have elevation. Boundary white pixels representing the boundaries of structures may be determined. In an embodiment, a pixel may have 8 surrounding, or neighbor, pixels. As pixels within the boundary will be surrounded by pixels having similar elevation, or color, boundary pixels may be identified as pixels having at least one neighbor pixel that is black after thresholding. After determining boundary pixels, the boundary pixels may be traced or grouped based on other proximal boundary pixels. For example, a boundary pixel may be grouped with a nearest other boundary pixels. In an embodiment, a line model may be used to project and determine a boundary pixel grouping by fitting a line through determined boundary pixels. In an embodiment, resulting lines of an image constructed using thresholding techniques represent the boundaries of structures in an intensity image”, [0036], “In act 260 a shape representing the geographic footprint of the structure in the geographic area is detected from the segmented intensity image. The shape may be detected using any technique. In an embodiment, a shape detection algorithm is applied to the segmented intensity image to detect a shape”, [0037]).
Okazaki and Chen et al. are in the same art of processing aerial imagery (Okazaki, abstract; Chen et al., [0021]). The combination of Chen et al. with Okazaki and Hakimi-Boushehri et al. and Ruda et al. and Kottenstette et al. enables using pixel boundary locations. It would have been obvious at the time of filing to combine the pixel boundary locations of Chen et al. with the invention of Okazaki and Hakimi-Boushehri et al. and Ruda et al. and Kottenstette et al. as this was known at the time of filing, the combination would have predictable results, as this is likely how the segmentation described by Kottenstette et al. is being performed, and as in general to provide the most accurate assessment of roof damage as would be desired in Okazaki and Hakimi-Boushehri et al. and Ruda et al., the true location of the metes and bounds of the damage would need to be known with pixel or sub-pixel accuracy.
Claim(s) 7, 8, 10, 17, 18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Okazaki (US 20180089763 A1) and Hakimi-Boushehri et al. (US 10607295 B1) and Ruda et al. (US 20180247416 A1) as applied to claims 1 and 11 above, further in view of Howe et al. (US 20170270650 A1).
Regarding claims 7 and 17, Okazaki and Hakimi-Boushehri et al. and Ruda et al.disclose the system and method of claims 1 and 11. Ruda et al. further imply the processor determines the at least one condition of the roof using a segmentation-based neural network that segments roof condition features (“FIGS. 4A-4C illustrate examples of the use and training of a machine learning process, according to various embodiments. As noted previously, any or all of sub-processes 302-310 of image analysis process 248 may be configured to use machine learning models, such as a trained deep neural network (e.g., a CNN, etc.). As shown in the example 400 of FIG. 4A, the trained CNN may receive input data such as image data and, in turn, output one or more labels for the data. For example, in the case of shingle/tile counter 306, the CNN may label the edges of each shingle or tile, to obtain a total count of shingles or tiles present in the image”, [0069], “Certainly wind-damage that consists of a number of missing shingles will be visible, and there is still a need to estimate the damage. The first step is to use a version of the previously described CNN for boundary detection that has been trained for detecting roof boundaries in the same type imagery. After thresholding and applying the (modified) probabilistic Hough transform a number of edge segments are created. The second step is to take these edge segments and turn them into separate slopes of a roof”, [0093]) however another reference is added for further clarity.
Howe et al. teach determining one condition of the roof using a segmentation-based neural network that segments roof condition features (In some embodiments, when processing the images according to the image processing criteria, the system may segment the images into a plurality of segmented images, extract one or more features from the segmented images, and apply a classifier to determine whether any of the images is a candidate damage image. In this case, when processing the images to determine whether the claim processing criteria are satisfied, the system may only process those images that are determined to be a candidate damage image, [0010], “The system will access a data storage facility to retrieve, or otherwise receive, image processing criteria 212 and process the images 213 according to a selected set of image processing criteria to identify one or more features in the images. In some embodiments, the image processing criteria may include criteria that are associated with the category. In some embodiments, the image processing criteria may include criteria that are associated with the imaging device, or that are defined by the insurance provider, or that are otherwise established. Example image processing criteria may include an image recognition algorithm such as an edge detection process or a pattern recognition algorithm, an image segmentation requirement (such as a requirement to divide images of a roof into shingle-specific or sector-specific sub-segments. For example, one roof segmentation approach may include an edge refining and filling algorithm that traces the boundaries of individual shingles based on their largely consistent orientations within a roof image. Another example includes running damage detection algorithms within each segmented shingle. Other image processing criteria may include a requirement to compare an image of a particular property segment with a previously-captured image of the property segment to determine whether the images have a difference in value that exceeds a threshold amount”, [0047], In one embodiment, when analyzing an ostensibly damaged part of the property, such as a shingled roof, the segmentation may perform at the following levels: separating the roof from all other elements of the frame, segmenting each individual (and usually, but not always) planar section of the roof, and/or segmenting each shingle tab or “tooth.”, [0064], “The above described classifier-based system 400 has two stages, a training stage and an inference stage. With reference to FIG. 4A, a classifier-based system 400 includes a feature extraction process 402 and a classifier that performs a classification process 403. The same feature extraction and classifier processes may be applied to both the training and inference stages. In one embodiment, in the training stage, the labels for each training image, that is, the class to which the image or pixels in the image belong to (e.g., damaged vs. undamaged) are, for example, assigned by a trained professional. The images, or more specifically their feature representations in some chosen feature space (extracted by the feature extraction 402), can be used to train a classifier. Various classifiers, such as a support vector machine (SVM), decision tree, clustering algorithm or neural network, can be used. The features can be hand-engineered and can include attributes such as size, shape, density, location, texture, color, and the like, or combinations thereof. For example, the features can correspond to the RGB values of the pixels, that is, the input image itself. The features can also be learned by a deep learning framework (e.g., a convolutional neural network) or a discriminative analysis process. In the training stage, the classifier learns to separate the feature descriptions of the damaged samples from those of the undamaged samples”, [0070]).
Okazaki and Hakimi-Boushehri et al. and Howe et al. are in the same art of assessing property damage (Okazaki, abstract; Hakimi-Boushehri et al., abstract; Howe et al., abstract). The combination of Howe et al. with Okazaki and Hakimi-Boushehri et al. and Ruda et al. enables using a segmentation-based neural network that segments roof condition features. It would have been obvious at the time of filing to combine the segmentation-based neural network that segments roof condition features of Howe et al. with the invention of Okazaki and Hakimi-Boushehri et al. and Ruda et al. as this was known at the time of filing, the combination would have predictable results, and as Howe et al. state “For example, due to illumination, weathering, or other variations across the different areas of the roof, the difference in appearance between similarly damaged portions (or conversely, between different undamaged roof portions) may vary greatly. Making that determination on a per-region basis may result in higher accuracy damage detection results” ([0072]) thereby improving the claim assessments of Okazaki and Hakimi-Boushehri et al. and Ruda et al. which will result in potential cost savings for the insurance company and a lowering of customer aggravation by decreasing mistaken damage assessment, thereby also improving customer satisfaction.
Regarding claims 8 and 18, Okazaki and Hakimi-Boushehri et al. and Ruda et al. and Howe et al. disclose the system and method of claims 7 and 17. Howe et al. further indicate the processor generates a single channel image (green channel, [0073]) based on output of the segmentation-based neural network that maps each pixel in the image to a condition label indicative of the at least one condition of the roof (“DAMAGE LOCATION LABELING RESOLUTION: Damage location can be labeled at different resolutions, from pixel- to frame-level. For example, each picture or frame could be annotated at the pixel level, indicating whether that pixel represents a damaged area or not. This could be in addition to having the same image available without any annotation. Damage location could also be labeled at the tab or “tooth” level for roof shingles, at roof section level, at the whole roof/property level or at the whole image level” [0055], “The above described classifier-based system 400 has two stages, a training stage and an inference stage. With reference to FIG. 4A, a classifier-based system 400 includes a feature extraction process 402 and a classifier that performs a classification process 403. The same feature extraction and classifier processes may be applied to both the training and inference stages. In one embodiment, in the training stage, the labels for each training image, that is, the class to which the image or pixels in the image belong to (e.g., damaged vs. undamaged) are, for example, assigned by a trained professional. The images, or more specifically their feature representations in some chosen feature space (extracted by the feature extraction 402), can be used to train a classifier. Various classifiers, such as a support vector machine (SVM), decision tree, clustering algorithm or neural network, can be used. The features can be hand-engineered and can include attributes such as size, shape, density, location, texture, color, and the like, or combinations thereof. For example, the features can correspond to the RGB values of the pixels, that is, the input image itself. The features can also be learned by a deep learning framework (e.g., a convolutional neural network) or a discriminative analysis process. In the training stage, the classifier learns to separate the feature descriptions of the damaged samples from those of the undamaged samples”, [0070], With reference to FIG. 4B, a heuristics-based damage detection process 410 is shown. This system uses the local disparity in image color and texture to detect damaged pixels in the imagery. For example, in a roof damage context, the system may first change the original RGB image to a color space 411 that provides a channel specific to brightness. In one example, the system can use any combination of the color channels in a color image, e.g., the green channel in a RGB image, or multiple channels in a hyperspectral image, to obtain a standard grayscale conversion. In another example, the system can use the value channel from the hue-saturation-value (HSV) color space to do the conversion. The process may further include thresholding 412, which separates each pixel into two classes based on the assumption that regions that contain hail damage often have different light reflectance values when compared with those pixels in the undamaged regions, and that the number of pixels that correspond to roof damage is far fewer than those that do not. For example, as shown in FIG. 5, the pixels that correspond to hail damaged regions contain higher reflectance values. For simplicity, the pixels that correspond to hail damaged regions are referred to as “foreground pixels” and those that correspond to undamaged roof portions as “background pixels.” Groups or clusters of foreground pixels are also referred to as “blobs.”, [0073]).
Regarding claims 10 and 20, Okazaki and Hakimi-Boushehri et al. and Ruda et al. disclose the system and method of claims 1 and 11. Okazaki and Hakimi-Boushehri et al. and Ruda et al. do not disclose the processor generates a score indicating a severity of the at least one condition and includes the score in the roof condition report.
Howe et al. teach a processor generates a score indicating a severity of the at least one condition and includes the score in the roof condition report (“DAMAGE SEVERITY: Damage severity could be provided in association with each damage location. For example, it may include how severe the damage was, e.g., was it just cosmetic, did it dent the surface, did it fracture the mat (the structural part of the shingle), etc. Alternatively and/or additionally, the industry may have a standard measurement scale, reference images, tools, and/or vocabulary to describe the damage (beyond what is depicted in literature used in the assessment industry such as the Haag Composition Roofs Damage Assessment Field Guide), and the damage severity could be provided using that scale and/or vocabulary,” [0056], “By way of example, and with reference to FIG. 4A, a classifier-based damage detection process 400 is further illustrated in the context of roof damage. It can be assumed that labeled data corresponding to images of damaged and undamaged roofs is available. The resolution of the labels dictates the resolution of the classification output. For example, damage can be labeled at the pixel-level, e.g., by identifying the location of pixels corresponding to damaged roof areas in the image. In another example, the damage labels can correspond to images, if the label identifies an image as containing a particular type of damage (which can include hail, wind, rot, etc.). A system that is trained based on these types of labels, will consequently produce classification labels of input imagery, effectively classifying the image as either containing a type of damage or not. Moreover, if either in addition to or in absence of a classification label, a severity score is known, a system that is based on a regression analysis can be trained to estimate the severity of damage for a given input image,” [0069]).
Okazaki and Hakimi-Boushehri et al. and Howe et al. are in the same art of assessing property damage (Okazaki, abstract; Hakimi-Boushehri et al., abstract; Howe et al., abstract). The combination of Howe et al. with Okazaki and Hakimi-Boushehri et al. and Ruda et al. enables using a score. It would have been obvious at the time of filing to combine the score of Howe et al. with the invention of Okazaki and Hakimi-Boushehri et al. and Ruda et al. as this was known at the time of filing, the combination would have predictable results, and as quantifying with a definitive score thereby improving the claim assessments of Okazaki and Hakimi-Boushehri et al. and Ruda et al. which will result in potential cost savings for the insurance company and a lowering of customer aggravation by decreasing mistaken damage assessment, thereby also improving customer satisfaction.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: US 20150073834 A1 (where djk=j is simply a numerical indicator of the damage state, namely d.sub.1k=1 for damage state D1 in component set k, d.sub.2k=2 for damage state D2 in component set k, and so on. The notation ejk indicates the percentage of all components k (e.g., 20% of all columns and beams) that are in damage state j (e.g., moderate damage, D2), [0062]); “Estimation of repair costs for RC and masonry residential buildings based on damage data collected by post-earthquake visual inspection” (In spite of the numerous studies revised above, a reliable direct correlation between empirical damage and Actual Repair Costs, ARC (i.e. cost computed by practitioners to repair the damage suffered by each structural and non-structural component) is still lacking
PNG
media_image2.png
532
550
media_image2.png
Greyscale
).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE M ENTEZARI HAUSMANN whose telephone number is (571)270-5084. The examiner can normally be reached 10-7 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent M Rudolph can be reached on (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE M ENTEZARI HAUSMANN/Primary Examiner, Art Unit 2671