DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1-13 and 15-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Malreddy et al (US2021/0342997) in view of Tan et al (US2020/0111203) and further in view of Chen et al (US10319094).
Regarding claims 1, 10 and 16, Malreddy teaches a method of detecting vehicle segments within vehicle images, comprising:
obtaining a plurality of electronic images of a plurality of vehicles, a first subset of the plurality of vehicles including damage and a second subset of the plurality of vehicles being undamaged, and the plurality of electronic images including respective sets of pixels depicting respective vehicle segments of the plurality of vehicles;
(Malreddy, Fig. 3, “the raw input data 12 can include real digital images of vehicles with or without damage”, [0052]; these images are used for training a neural network (NN))
marking, via one or more computer processing devices, each pixel of the respective sets of pixels depicting the respective vehicle segments of the plurality of vehicles to indicate a respective pre-defined vehicle segment which the each pixel depicts thereby creating a plurality of marked vehicle images, the respective pre-defined vehicle segment delineated based on a respective base object segmentation model
developing, using the one or more computer processing devices, a pre-defined vehicle segments in an image on a pixel by pixel basis;
(Malreddy, Fig. 22, “in step 224, the system 10 segments the simulated input image of the vehicle into corresponding components of the vehicle. For example, the system 10 can segment the simulated input image to distinguish the hood, fender and door components of the vehicle”, [0071]; the vehicle component segmentation may be trained using labeled samples as the base models for segmentation of the vehicle components such as hood, fender and door in a supervised training of a machine learning model as suggested by Tan, “The image segmentation model may be obtained by training using sample images labelled with vehicle component areas”, [0080]; “the classification model may alternatively be obtained by supervised training of an existing machine learning model using machine learning methods and training samples”, [0065]; “an image segmentation model may be constructed based on a convolutional neural network (CNN) and a region proposal network (RPN) in combination with a pooling layer, a fully connected layer, and the like”, [0080]; image segmentation is a process to mark particular objects in an image; the segmentation marking is pixel by pixel)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Tan into the system or method of Malreddy in order to use CNN and RPN to perform image segmentation to partition a digital image into multiple regions (segments), making it easier to analyze and process. The combination of Malreddy and Tan also teaches other enhanced capabilities.
The combination of Malreddy and Tan does not expressly disclose but Chen teaches:
… a respective base object segmentation model specific to a year, a make, and a model of a respective vehicle corresponding to the respective pre-defined vehicle segment;
(Chen, Figs. 1 and 2; “when a set of new images has been collected/selected for a target object to be analyzed for changes, such as for damage, a block 204 determines a base object model (i.e., one of the base object models 120 of FIG. 1) to use in the change detection processing. As an example, the block 204, when analyzing images of an automobile for damage, may determine the make/model/year of the automobile within the collected images and may obtain a base object model 120 for that make/model/year from the database 108 of FIG. 1”, c9:40-50; “FIG. 9 illustrates a base object model associated with the vehicle for which damage is to be detected”, c18:25-30)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Chen into the modified system or method of Malreddy and Tan in order to detect damages in a vehicle specific to a particular vehicle type, e.g., make/model/year of the vehicle using a particular basic model for the particular vehicle type (make/model/year). The combination of Malreddy, Tan and Chen also teaches other enhanced capabilities.
The combination of Malreddy, Tan and Chen further teaches:
… a regression model…
(Chen, “a CNN may determine, for each pixel or set of pixels in the image corresponding to a particular component or part of a component, a likelihood of change or damage at that point, an estimate of the likelihood of change or damage at that point, and/or a type or severity of change or damage at that point, and may represent this likelihood of change or other change parameter as a number on a scale, such as a number between 0-99, in which 0 represents no likelihood of change or damage and 99 represents a high likelihood of change or damage”, c11:66-c12:10; the CNN of this type can be considered to have a regression model; the key indicator is that it outputs a continuous value on a scale (a number between 0 and 99) rather than a discrete class label)
obtaining at least one electronic image of a further vehicle;
(Tan, “acquiring a damage area image of a target vehicle”, [0006]; obtaining/acquiring an image of a (target) vehicle for subsequent processing)
applying, using the one or more computer processing devices, the regression model to the at least one electronic image of the further vehicle to determine, for each pixel of a set of pixels of the at least one electronic image of the further vehicle, a measure of a respective probability of the each pixel depicting each pre-defined vehicle segment included in one or more pre-defined vehicle segments of the further vehicle, the one or more pre-defined vehicle segments delineated based on the base object model; and
(Chen, “each CNN 134 can be used to identify damage (e.g., the likelihood of damage and/or a quantification of damage such as amount of damage or a type of damage) at each part (e.g., pixel, flat surface area such as the triangular surface segments of FIG. 5, etc.) of the body panel or segment being analyze”, c26:10-20; “The CNN 1302 applies a series of weights and calculations (such weights being determined during the training of the CNN) to the inputs of the CNN 1302 to produce a damage parameter value or damage determination for the target vehicle image pixel being analyzed; the damage parameter value may represent a likelihood of damage, an amount of damage, and/or an indication of the type of damage”, c26:30-40; applying CNN models pixel-by-pixel to determine probabilities (“likelihood”) of the damage; as discussed above, the CNN model may be a regression model; Tan, “A detection result may include location information of the second suspected damage area, a probability that each pixel in the image is included in the second suspected damage area”, [0049]; detecting a probability of each pixel in the image belonging to a particular damage segment)
producing, using the one or more computer processing devices and based on the applying, a segmented vehicle image for the further vehicle, the segmented vehicle image indicating at least one of the one or more pre-defined vehicle segments depicted within the electronic image of the further vehicle.
(Malreddy, Fig. 22; “in step 224, the system 10 segments the simulated input image of the vehicle into corresponding components of the vehicle. For example, the system 10 can segment the simulated input image to distinguish the hood, fender and door components of the vehicle”, [0071]; Chen, “The block 310 may thus create a background eliminated, segmented, corrected damaged vehicle image that depicts the target vehicle with the contours of the body panels or other segments of the target vehicle defined and with the background image pixels removed”, c25:54-60; producing segmented vehicle images showing vehicle components/segments)
Regarding claim 2, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the method of claim 1, wherein the regression model is a neural network model.
(Malreddy, Tan, see comments on claim 1)
Regarding claim 3, the combination Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the method of claim 1, wherein the regression model is a convolutional neural network model.
(Malreddy, Tan, see comments on claim 1)
Regarding claim 4, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the method of claim 3, wherein developing the regression model includes selecting a size and a shape for a pixel area surrounding a pixel being processed to use as inputs to the regression model.
(Tan, Fig. 11, “The image segmentation model may be obtained by training using sample images labelled with vehicle component areas”, [0080])
Regarding claim 5, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the method of claim 1, wherein the plurality of electronic images includes a plurality of electronic images of a plurality of damaged and undamaged vehicles in which different segments of different vehicles are depicted.
(Malreddy, Fig. 3, “the raw input data 12 can include real digital images of vehicles with or without damage”, [0052]; Tan, Fig. 22, “The image segmentation model may be obtained by training using sample images labelled with vehicle component areas”, [0080])
Regarding claim 6, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the method of claim 1, wherein the plurality of electronic images includes a plurality of electronic images of a plurality of damaged and undamaged vehicles of different years, makes, models or trims.
(Tan, “outputting, according to the determined damage location, damage type, numerical information, and preset corresponding relationships between damage location, damage type, numerical information and vehicle maintenance information. The vehicle maintenance information may include a vehicle maintenance method, such as touch-up painting, replacing components, and may also include maintenance costs, maintenance work hours, and the like. The corresponding relationships between the damage location, the damage type, the numerical information and the vehicle maintenance information may be set according to the actual situations of the areas”, [0069]; in order to identify components for replacement, it is obvious that the basic vehicle information such as years, makes, models or trims of a vehicle must be used in the training of the damage classification model)
Regarding claim 7, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the method of claim 1, wherein the plurality of electronic images includes depictions of vehicles having one or more vehicle segments, each vehicle segment of the one or more vehicle segments being a one of a front panel, a wheel, a windshield, a hood, a rear panel, a side panel, and a door.
(Malreddy, Fig. 22, “in step 224, the system 10 segments the simulated input image of the vehicle into corresponding components of the vehicle. For example, the system 10 can segment the simulated input image to distinguish the hood, fender and door components of the vehicle”, [0071])
Regarding claim 8, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the method of claim 1, wherein the one or more pre-defined vehicle segments includes at least two pre-defined vehicle segments, and applying the regression model to the at least one electronic image of the further vehicle includes
storing the determined, respective probability of the each pixel, of the set of pixels, depicting the each pre-defined vehicle segment included in the at least two pre-defined vehicle segments.
(Malreddy, Fig. 22, “in step 224, the system 10 segments the simulated input image of the vehicle into corresponding components of the vehicle. For example, the system 10 can segment the simulated input image to distinguish the hood, fender and door components of the vehicle”, [0071]; Fig. 24B, IOU results, [0074])
Regarding claim 9, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the method of claim 8, further including using the determined, respective probabilities to determine which of the at least two pre-defined vehicle segments the each pixel of the set of pixels depicts.
(Malreddy, see comments on claim 8)
Regarding claim 11, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the image processing system of claim 10,
wherein the characterization engine is a neural network engine and
wherein the image model includes neural network weights developed using the plurality of marked vehicle images in which the pre-defined vehicle segments were known on the pixel by pixel basis.
(Tan, CNN and RPN, [0080]; Malreddy, “Training the neural network 16 can include an iterative learning process in which input values (e.g., data from the dataset) are sequentially presented to the neural network 16 and weights associated with the input values are sequentially adjusted”, [0054])
Regarding claim 12, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the image processing system of claim 10, wherein the characterization engine applies, for each pixel of a multiplicity of pixels within the one or more images of the vehicle, a filter that specifies a corresponding set of pixels around a pixel being analyzed to use as inputs to the image model.
(Tan, see comments on claim 1; segmentation RPN is a regional filter)
Regarding claims 13 and 19, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the image processing system of claim 10,
wherein the each image of the one or more images of the vehicle includes information about a view of the vehicle depicted in the each image, and
wherein the characterization engine further uses the information about the view of the vehicle depicted in the each image to detect, on the pixel by pixel basis, the respective pre-defined vehicle segment to which the each pixel belongs.
(Tan, Figs. 22 and 23)
Regarding claim 15, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the image processing system of claim 10, wherein the characterization engine determines, using the image model, a respective probability of a particular vehicle segment at each pixel included in a group of pixels in the one or more images of the vehicle.
(Malreddy, see comments on claim 8)
Regarding claim 17, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the method of claim 16, wherein the regression model is a neural network model that includes neural network weights developed using the plurality of marked vehicle images in which a respective pre-defined vehicle segment is known for each pixel of a plurality of pixels.
(Malreddy, Tan, see comments on claim 11)
Regarding claim 18, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination further teaches the method of claim 16, wherein applying the regression model to the at least one of the one or more images of the vehicle comprises applying the regression model at each pixel of a multiplicity of pixels within the one or more images of the vehicle and using a specified set of pixels around a pixel being analyzed to use as inputs to the regression model.
(Tan, Figs. 22 and 23)
Claim(s) 14 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Malreddy et al (US2021/0342997) in view of Tan et al (US2020/0111203) and further in view of Chen et al (US10319094) and Tian et al (US2021/0142464).
Regarding claims 14 and 20, the combination of Malreddy, Tan and Chen teaches its/ their respective base claim(s),
The combination does not expressly disclose but Tian teaches the image processing system of claim 10,
wherein the each image of the one or more images of the vehicle includes information about a zoom level of the each image, and
wherein the characterization engine further uses the information about the zoom level of the each image to detect, on the pixel by pixel basis, the respective pre-defined vehicle segment to which the each pixel belongs.
(Tian, “various embodiments use machine learning and image segmentation trained on vehicle images obtained from different views/angles. The use of such disparate images in training helps in providing a generalized image analysis and inferencing technique that can accurately identify vehicle damage from images of vehicles taken from many different angles, under different light conditions, and/or at different levels of zoom”, [0023]; Fig. 4A-2, “in step 412a, an image is selected from the received images based on the zoom information included in the image or represented by the image pixels. Specifically, a moderately zoomed image, e.g., having a scaling factor in the range 0.3 to 0.7 is selected, and in step 412a it is determined if the image includes both an identifiable external part and an associated damage segment. The scaling factor has a range of 0 to 1, where 0 indicates a zoomed-in image with no parts boundary shown and 1 indicates a zoomed-out image where the entire vehicle is shown”, [0035])
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Tian into the modified system or method of Malreddy and Tan in order to determine whether one or more damages is included in an image for damage identification based on zoom information embedded in the image. The combination of Malreddy, Tan and Tian also teaches other enhanced capabilities.
Response to Arguments
Applicant's arguments filed on 10/21/2025 with respect to one or more of the pending claims have been fully considered but they are not persuasive.
Regarding claim(s) 1, 10 and 16, Applicant, in the remarks, argues that the combination of the cited reference(s) fails to teach the newly amended limitations in the claims.
The Examiner respectfully disagreed. The office action has been updated to address applicant’s argument. See the updated review comments for details.
Regarding claim(s) 1, 10 and 16, Applicant, in the remarks, argues that the combination of the cited references fails to teach “applying, using one or more computer processing devices, a regression model to at least one of the one or more images of the vehicle to determine, for each pixel of a set of pixels of the at least one of the one or more images of the vehicle, a respective pre-defined vehicle segment to which each pixel of the set of pixels belongs and a measure of a respective probability of the each pixel belonging to the respective pre-defined vehicle segment ... ” as recited in claim 16.
The Examiner respectfully disagreed. The office action has been updated to address applicant’s argument. See the updated review comments for details.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, Applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 12/31/2025
Prosecution Insights