DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office Action is responsive to Applicant’s remarks received on April 16, 2026. Claims 1-6, 8-16, 18, 19 and 21-23 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 6, 9-11, 16, 19 and 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Malreddy et al. (US 2021/0342997) and Lambert et al. (US 2021/0350470).
Regarding claim 1, Malreddy et al. discloses an image processing apparatus, comprising:
a memory storing computer-executable code (“FIG. 37 is a diagram showing hardware and software components of a computer system 400 on which the system of the present disclosure can be implemented. The computer system 400 can include a storage device 404, computer vision software code 406” at paragraph 0085, line 1); and
at least one processor configured to access the memory and execute the code (“The CPU 412 could include any suitable single-core or multiple-core microprocessor of any suitable architecture that is capable of implementing and running the computer vision software code 406 (e.g., Intel processor)” at paragraph 0086, second to last sentence), wherein the code comprises instructions for the at least one processor to generate an input image by detecting a part in an image (“The vehicle component segmentation can be classified into six classes including a vehicle left front door, a vehicle right front door, a vehicle left front fender, a vehicle right front fender, a vehicle hood and a background” at paragraph 0073, line 7), apply the input image to an image analysis model to obtain a defect type and a defect size of the part (“Damage severity classification can be classified for each vehicle component segmentation class according to one of undamaged, mildly damaged and extremely damaged by cropping each vehicle component along with its corresponding context from the obtained segmentation” at paragraph 0073, last sentence; “For example, the system 10 can determine whether the location of the damage includes at least one of a front of the vehicle (e.g., a hood or windshield) in step 80, a rear of the vehicle (e.g., a bumper and trunk) in step 82 and/or a side of the vehicle (e.g., a passenger door) in step 84. In step 86, the system 10 determines a severity classification of the damage sustained by the detected vehicle in the received image. For example, the system 10 can determine whether the sustained damage is minor in step 88, moderate in step 90 or severe in step 92” at paragraph 0057, line 9), and generate an output image by displaying the defect type and the defect size of the part, wherein the defect size corresponds to a pixel amount defined by a number of pixels representing a defective area in the output image (the output is a labeled image where the defect pixels are identified according to severity, which reflects size and type; implied that the severity is partially determined by how big the defect is, which is reflected in the amount of pixels constituting the defect).
Malreddy et al. does not explicitly disclose determining an availability of the part based on the defect size and the defect type by: determining that the part is similar to a new product and is an available part when the pixel amount is less than a first threshold; determining that the part has a slight damage, but is an available part, when the pixel amount is greater than or equal to the first threshold and is less than a second threshold; determining that the part has substantial damage and is an unavailable part, when the pixel amount is greater than or equal to the second threshold and is less than a third threshold; and determining that the part is damaged and is an unavailable part, when the pixel amount is greater than or equal to the third threshold.
Lambert et al. teaches an image processing apparatus in the same field of endeavor of image based vehicle damage defect detection, comprising:
a memory storing computer-executable code (“The memory element 32 may include, or may constitute, a “computer-readable medium.” The memory element 32 may store instructions, code, code segments, software, firmware, programs, applications, apps, services, daemons, or the like, including the mobile application 40, that are executed by the processing element 34” at paragraph 0052, line 7); and
at least one processor configured to access the memory and execute the code (“The processing element 34 may include one or more processors” at paragraph 0053, line 1), wherein the code comprises instructions for the at least one processor to:
determining an availability of the part based on the defect size and the defect type by:
determining that the part is similar to a new product (“In addition, the OEM parts database 88 may include information as to whether a selected part of a vehicle is available as an OEM part or a direct replacement aftermarket part” at paragraph 0065, last sentence) and is an available part when the pixel amount is less than a first threshold (“In one implementation, the damage estimator computing device 43 may determine whether the damage is light damage below a first threshold damage level” at paragraph 0081, line 1; it is feasible that a person of ordinary skill in the art could designate an even lesser damage category in lieu of “light damage”);
determining that the part has a slight damage, but is an available part, when the pixel amount is greater than or equal to the first threshold and is less than a second threshold (“whether the damage is heavy damage above the first threshold and below a second threshold” at paragraph 0081, line 4; similar to the above reasoning, the “light damage” category would be subsequent to the very light damage category above as defined by the first and second threshold);
determining that the part has substantial damage and is an unavailable part, when the pixel amount is greater than or equal to the second threshold and is less than a third threshold (similar to the above reasoning, a “heavy damage” category would be shifted to encompass the second and third thresholds); and
determining that the part is damaged and is an unavailable part, when the pixel amount is greater than or equal to the third threshold (“whether the damage is a total loss above the second threshold” at paragraph 0081, line 6; therefore the total loss category would be shifted to constitute the third threshold).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize a damage severity as taught by Lambert et al. on the images of Malreddy et al. as a clear way to define the severity of the damage for evaluation of proper repair.
Regarding claim 11, Malreddy et al. discloses an image processing method, comprising:
generating an input image by detecting a part in an image (“The vehicle component segmentation can be classified into six classes including a vehicle left front door, a vehicle right front door, a vehicle left front fender, a vehicle right front fender, a vehicle hood and a background” at paragraph 0073, line 7),
generating an output image by displaying the defect type and the defect size of the part, wherein the defect size corresponds to a pixel amount defined by a number of pixels representing a defective area in the output image (“Damage severity classification can be classified for each vehicle component segmentation class according to one of undamaged, mildly damaged and extremely damaged by cropping each vehicle component along with its corresponding context from the obtained segmentation” at paragraph 0073, last sentence; “For example, the system 10 can determine whether the location of the damage includes at least one of a front of the vehicle (e.g., a hood or windshield) in step 80, a rear of the vehicle (e.g., a bumper and trunk) in step 82 and/or a side of the vehicle (e.g., a passenger door) in step 84. In step 86, the system 10 determines a severity classification of the damage sustained by the detected vehicle in the received image. For example, the system 10 can determine whether the sustained damage is minor in step 88, moderate in step 90 or severe in step 92” at paragraph 0057, line 9), and
applying the defect type and the defect size to the input image to generate an output image (the output is a labeled image where the defect pixels are identified according to severity, which reflects size and type).
Malreddy et al. does not explicitly disclose generating an input image by preprocessing an image around a part.
Lambert et al. teaches an image processing method in the same field of endeavor of image based vehicle damage defect detection, comprising:
determining an availability of the part based on the defect size and the defect type by:
determining that the part is similar to a new product (“In addition, the OEM parts database 88 may include information as to whether a selected part of a vehicle is available as an OEM part or a direct replacement aftermarket part” at paragraph 0065, last sentence) and is an available part when the pixel amount is less than a first threshold (“In one implementation, the damage estimator computing device 43 may determine whether the damage is light damage below a first threshold damage level” at paragraph 0081, line 1; it is feasible that a person of ordinary skill in the art could designate an even lesser damage category in lieu of “light damage”);
determining that the part has a slight damage, but is an available part, when the pixel amount is greater than or equal to the first threshold and is less than a second threshold (“whether the damage is heavy damage above the first threshold and below a second threshold” at paragraph 0081, line 4; similar to the above reasoning, the “light damage” category would be subsequent to the very light damage category above as defined by the first and second threshold);
determining that the part has substantial damage and is an unavailable part, when the pixel amount is greater than or equal to the second threshold and is less than a third threshold (similar to the above reasoning, a “heavy damage” category would be shifted to encompass the second and third thresholds); and
determining that the part is damaged and is an unavailable part, when the pixel amount is greater than or equal to the third threshold (“whether the damage is a total loss above the second threshold” at paragraph 0081, line 6; therefore the total loss category would be shifted to constitute the third threshold).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize a damage severity as taught by Lambert et al. on the images of Malreddy et al. as a clear way to define the severity of the damage for evaluation of proper repair.
Regarding claims 6 and 16, Malreddy et al. discloses an apparatus and method wherein the defect type comprises:
a first defect including a surface scratch of a painted surface area of the part (“A real dataset and a simulated dataset can each illustrate vehicle damage including, but not limited to, superficial damage such as a scratch or paint chip” at paragraph 0053, line 1);
a second defect including shapeshifting of the part (“deformation damage such as a dent” at paragraph 0053, line 4);
a third defect in which at least a portion of a shape of the part is different from an existing shape of the part (“The severe damage class can include damage indicative of a broken axle, a bent or twisted frame” at paragraph 0053, second to last sentence); or
a fourth defect including a gap generated in a joint of the part (“The minor damage class can include damage indicative of a scratch, a scrape, a ding, a small dent, a crack in a headlight, etc. The moderate damage class can include damage indicative of a large dent, a deployed airbag” at paragraph 0053, third to last sentence).
Regarding claims 9, 10 and 19, Malreddy et al. discloses an apparatus and method further comprising:
generating the image analysis model including a plurality of pooling layers and a plurality of unpooling layers, based on the image analysis model being a U-Net neural network; and training the image analysis model, using a loss function defined as a sum of losses between the input image and the output image, wherein the plurality of pooling layers are connected with non-linearity of a rectified linear unit (ReLU) function included in the image analysis model, and wherein the image analysis model includes a skip connection from the plurality of pooling layers to the plurality of unpooling layers (“Alternatively, segmentation processing can be performed with a U-Net-CNN. It is noted that a U-Net-CNN works well with small datasets. Advantageously, the segmentation processing provides for identifying a damaged vehicle component instead of a damaged vehicle region in two steps via vehicle component segmentation and damage severity classification” at paragraph 0073, line 1; “The system 10 can use an error metric, such as the per-pixel cross-entropy loss function, to measure the error of the neural network 16. The cross-entropy loss function evaluates class predictions for each pixel vector individually and then averages over all pixels” at paragraph 0065, second to last sentence; Figure 24A shows the pooling and unpooling layers of the U-net, the ReLU layers and the skip connections, which are referred to as copy and crop).
Regarding claims 21 and 22, the Malreddy et al. and Lambert et al. combination discloses the elements of claims 1 and 11 above.
The Malreddy et al. and Lambert et al. combination does not explicitly disclose that the first threshold is 400 pixels, wherein the second threshold is 1500 pixels, and wherein the third threshold is 4000 pixels.
However, as the Malreddy et and Lambert et al. combination discloses that the defect severity is determined by the overall size of said defect, it therefore follows that particular thresholds constituting a number of pixels would be established. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the claimed pixel thresholds to clearly define the severity classes in evaluation of damage repair.
Regarding claim 23, Malreddy et al. discloses an apparatus wherein the defect type comprises:
a first defect including a surface scratch of a painted surface area of the part (“A real dataset and a simulated dataset can each illustrate vehicle damage including, but not limited to, superficial damage such as a scratch or paint chip” at paragraph 0053, line 1);
a second defect including shapeshifting of the part (“deformation damage such as a dent” at paragraph 0053, line 4);
a third defect in which at least a portion of a shape of the part is different from an existing shape of the part (“The severe damage class can include damage indicative of a broken axle, a bent or twisted frame” at paragraph 0053, second to last sentence); and
a fourth defect including a gap generated in a joint of the part (“The minor damage class can include damage indicative of a scratch, a scrape, a ding, a small dent, a crack in a headlight, etc. The moderate damage class can include damage indicative of a large dent, a deployed airbag” at paragraph 0053, third to last sentence).
Claim(s) 2, 5, 12 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Malreddy et al. and Lambert et al. as applied to claims 1 and 11 above, and further in view of Hyatt et al. (US 2023/0281791).
Regarding claims 2 and 12, the Malreddy et al. and Lambert et al. combination discloses an apparatus and method wherein the code comprises instructions for the at least one processor to generate the input image by use of a cropped image of all portions of the detected part (“Damage severity classification can be classified for each vehicle component segmentation class according to one of undamaged, mildly damaged and extremely damaged by cropping each vehicle component along with its corresponding context from the obtained segmentation” Malreddy et al. at paragraph 0073, last sentence).
The Malreddy et al. and Lambert et al. combination does not explicitly disclose applying normalization to pixel values included in the cropped image depending on a selected pixel interval to change the pixel values.
Hyatt et al. teaches an apparatus and method wherein the code comprises instructions for the at least one processor to generate the input image by use of a cropped image, and apply normalization to pixel values included in the cropped image depending on a selected pixel interval to change the pixel values (“Input module 312 may include an image processing component to process an inspection image prior to it being received at the classification component. The image processing component may modify statistics of input data (e.g., images). For example, changes to input images (namely to inspection images 5) may be performed by input module 312, based on prior processing of reference images (e.g., images 4, 4′ and 4″). Changes to parameters of input images may include, for example, adjusting coefficients or changing pixel values for normalization, colorization, contrast, balance, lighting-fixing, as well as, aligning, cropping, warping an image or parts of it to specific coordinates” at paragraph 0085, line 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the image adjustment as taught by Hyatt et al. to generate the input images of the Malreddy et al. and Lambert et al. combination to provide input images to the classifier in a better form for performance.
Regarding claims 5 and 15, the Malreddy et al. and Lambert et al. combination discloses the elements of claims 1 and 11 as described above.
The Malreddy et al. and Lambert et al. combination does not explicitly disclose that the at least one processor to adjust one of or any combination of a color feature of the input image, an edge feature of the input image, a polygon feature of the input image, a saturation feature of the input image, a color temperature feature of the input image, a definition feature of the input image, a contrast feature of the input image, a blur feature of the input image, and a brightness feature of the input image, and apply the adjusted image to the image analysis model to obtain the defect type and the defect size.
Hyatt et al. teaches an apparatus and method wherein the code comprises instructions for the at least one processor to adjust one of or any combination of a color feature of the input image, an edge feature of the input image, a polygon feature of the input image, a saturation feature of the input image, a color temperature feature of the input image, a definition feature of the input image, a contrast feature of the input image, a blur feature of the input image, and a brightness feature of the input image (“Input module 312 may include an image processing component to process an inspection image prior to it being received at the classification component. The image processing component may modify statistics of input data (e.g., images). For example, changes to input images (namely to inspection images 5) may be performed by input module 312, based on prior processing of reference images (e.g., images 4, 4′ and 4″). Changes to parameters of input images may include, for example, adjusting coefficients or changing pixel values for normalization, colorization, contrast, balance, lighting-fixing, as well as, aligning, cropping, warping an image or parts of it to specific coordinates” at paragraph 0085, line 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the image adjustment as taught by Hyatt et al. to generate the input images of the Malreddy et al. and Lambert et al. combination to provide input images to the classifier in a better form for performance.
Claim(s) 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Malreddy et al. and Lambert et al. as applied to claims 1 and 11 above, and further in view of Chen et al. (US 10,706,321).
Regarding claims 8 and 18, the Malreddy et al. and Lambert et al. combination discloses the elements of claims 1 and 11 as described above.
The Malreddy et al. and Lambert et al. combination does not explicitly disclose providing an interface to receive the image from a parts supplier supplying used parts and provide the parts supplier with the output image, the defect type, and the defect size stored in an image storage server through the interface.
Chen et al. teaches an apparatus and method wherein the code comprises instructions for the at least one processor to provide an interface to receive the image from a parts supplier supplying used parts and provide the parts supplier with the output image, the defect type, and the defect size stored in an image storage server through the interface (“At any rate, at a block 1610, the method 1600 includes generating an indication of the one or parts needed to repair the vehicle, e.g., as determined/identified at the block 1608. At a block 1612, the indication is provided to at least one of a user interface or to another computing device. In some scenarios, providing the indication of the one or more parts needed to repair the vehicle (block 1612) includes ordering the one or parts needed to repair the vehicle. For example, the indication of the one or more parts needed to repair the vehicle may be electronically transmitted to a parts ordering system, a parts procurement system, or a parts store” at col. 33, line 62).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize part availability determination as taught by Chen et al. in the system of the Malreddy et al. and Lambert et al. combination as a way to “estimate repair costs, estimate the amount of change that has occurred in the object, estimate the amount of time or effort needed to correct or fix the change (Chen et al. at col. 2, line 62).
Claim(s) 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Malreddy et al., Lambert et al. and Hyatt et al. as applied to claims 2 and 12 above, and further in view of Noda et al. (US 2019/0205668).
The Malreddy et al., Lambert et al. and Hyatt et al. combination discloses the elements of claims 2 and 12 as described above.
The Malreddy et al., Lambert et al. and Hyatt et al. combination does not explicitly disclose changing a channel of the cropped image, the pixel values of which are changed, to a first channel or a second channel, changing a plurality of pixel values included in the cropped image, the pixel values of which are changed, to a one-dimensional array in an order of channel features included in the first channel changes to the first channel including a plurality of channels, based on the channel of the cropped image, the pixel values of which are changed.
Noda et al. teaches an apparatus and method wherein the code comprises instructions for the at least one processor to:
change a channel of the image, the pixel values of which are changed, to a first channel or a second channel, change a plurality of pixel values included in the image, the pixel values of which are changed, to a one-dimensional array in an order of channel features included in the first channel changes to the first channel including a plurality of channels, based on the channel of the image, the pixel values of which are changed (“The image data input the neural network may also be an R, G, B color image, or an image resultant of a color space conversion, such as a Y, U, V color image. Furthermore, the image input to the neural network may be a one-channel image resultant of converting the color image into a monochromatic image. Furthermore, instead of inputting the image as it is, assuming that an R, G, B color image is to be input, for example, the neural network may also receive an image from which an average pixel value in each channel is subtracted, or a normalized image from which an average value is subtracted and divided by a variance, as an input. Furthermore, a captured image corresponding to some point in time, or a part thereof may be also input to the neural network. It is also possible to input a captured image including a plurality of frames corresponding to several points in time with reference to one point in time, or a part of each captured image including a plurality of frames may be input to the neural network” at paragraph 0042).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the color conversion as taught by Noda et al. on the input images of the Malreddy et al., Lambert et al. and Hyatt et al. combination to provide input images to the classifier in a better form for performance.
Claim(s) 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Malreddy et al., Lambert et al., Hyatt et al. and Noda et al. as applied to claims 3 and 13 above, and further in view of Price et al. (US 10,970,835).
The Malreddy et al., Lambert et al., Hyatt et al. and Noda et al. combination discloses the elements of claims 3 and 13 as described above.
The Malreddy et al., Lambert et al., Hyatt et al. and Noda et al. combination does not explicitly disclose concatenating the image, the input image, and the output image, and transmit the concatenated image to an image storage server.
Price et al. teaches an apparatus and method in the same field of endeavor of vehicle damage identification, wherein the code comprises instructions for the at least one processor to concatenate the image and vehicle information, and transmit the concatenated data to an image storage server (“In a third implementation, alone or in combination with one or more of the first and second implementations, process 500 may include storing, in a data structure, information identifying the damaged part of the vehicle and the information regarding the damage on the vehicle in association with the information identifying the vehicle, receiving another plurality of images of the vehicle, and identifying, in the other plurality of images, a location of the damaged part in one or more of the other plurality of images based on the information identifying the damaged part of the vehicle” at col. 20, line 48).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the data consolidation as taught by Price et al. in the system of the Malreddy et al., Lambert et al., Hyatt et al. and Noda et al. combination to ensure all the relevant damage data for the vehicle is contained in a central location.
Response to Arguments
Summary of Remarks (@ response page labeled 10): Malreddy et al., Hida, Hyatt et al., Noda et al. and Price et al. do not teach or disclose determining an availability of the part based on the defect size and the defect type by: determining that the part is similar to a new product and is an available part when the pixel amount is less than a first threshold; determining that the part has a slight damage, but is an available part, when the pixel amount is greater than or equal to the first threshold and is less than a second threshold; determining that the part has substantial damage and is an unavailable part, when the pixel amount is greater than or equal to the second threshold and is less than a third threshold; and determining that the part is damaged and is an unavailable part, when the pixel amount is greater than or equal to the third threshold.
Examiner’s Response: This argument is moot in view of the newly cited Lambert et al. reference.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/Primary Examiner, Art Unit 2672
Prosecution Insights