DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 07/07/2025 has been entered.
Claim Status
Claims 1-4, 6-12, and 14-20 are pending for examination in the application filed 07/07/2025. Claims 1-4, 6-12, and 14-20 have been amended and claims 5 and 13 have been cancelled.
Response to Arguments
Applicant’s arguments, filed on 07/07/2025, with respect to independent claims 1 and 9 have been considered but are moot because the new grounds of rejection do not rely on the combination of references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier, as explained in MPEP §2181, subsection I (note that the list of generic placeholders below is not exhaustive, and other generic placeholders may invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph):
A. The Claim Limitation Uses the Term “Means” or “Step” or a Generic Placeholder (A Term That Is Simply A Substitute for “Means”)
With respect to the first prong of this analysis, a claim element that does not include the term “means” or “step” triggers a rebuttable presumption that 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, does not apply. When the claim limitation does not use the term “means,” examiners should determine whether the presumption that 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, paragraph 6 does not apply is overcome. The presumption may be overcome if the claim limitation uses a generic placeholder (a term that is simply a substitute for the term “means”). The following is a list of non-structural generic placeholders that may invoke 35 U.S.C. 112(f) or pre- AIA 35 U.S.C. 112, paragraph 6: “mechanism for,” “module for,” “device for,” “unit for,” “component for,” “element for,” “member for,” “apparatus for,” “machine for,” or “system for.” Welker Bearing Co., v. PHD, Inc., 550 F.3d 1090, 1096, 89 USPQ2d 1289, 1293-94 (Fed. Cir. 2008); Massachusetts Inst. of Tech. v. Abacus Software, 462 F.3d 1344, 1354, 80 USPQ2d 1225, 1228 (Fed. Cir. 2006); Personalized Media,161 F.3d at 704, 48 USPQ2d at 1886–87; Mas- Hamilton Group v. LaGard, Inc., 156 F.3d 1206, 1214-1215, 48 USPQ2d 1010, 1017 (Fed. Cir.1998). This list is not exhaustive, and other generic placeholders may invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, paragraph 6.
In claims 9 and 20:
at least one first image capturing unit for capturing…
at least one second image capturing unit for capturing…
[pg. 9] According to an embodiment the at least one first image capturing unit ICU11, ICU12, ICU13, ICU14, ICU15, ICU16 is a camera, such as a line scan camera, an analog area scan camera, a digital area scan camera, a FullHD resolution camera or a camera of a mobile device such as a smartphone.
[pg. 9] According to an embodiment the at least one second image capturing unit ICU21, ICU22, ICU23, ICU24, ICU25, ICU26 is a camera, such as a line scan camera. The at least one second image capturing unit may also be an analog area scan camera, a digital area scan camera, a FullHD resolution camera or a camera of a mobile device such as a smartphone.
In claim 9 and 14
at least one image analysis unit for analyzing…
In claim 9
at least one image combination unit configured to provide…
In claim 11
at least one image database unit configured to store…
In claim 12
at least one image retrieval and comparison unit configured to retrieve…
[pg. 22] The image analysis unit IAU and image analysis applications therein, the image combination unit ICU, the image retrieval and comparison unit IRCU, or the functionalities thereof may be implemented in one or more dedicated data processing means, that can execute a computer program product comprising executable code that when executed, cause the respective operations for detecting damages in the freight container.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Objections
Claim 6 is objected to because of the following informalities: Claim 6 as written depends on claim 5, which is currently cancelled. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 16 and 18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 16 and 18 recite the limitation “at least one analysis unit”. There is insufficient antecedent basis for this limitation in the claim, as independent claim 9 recites “at least one image analysis unit”. For the sake of compact prosecution, the Examiner is interpreting the analysis unit and the image analysis unit to be equivalent.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 6-12, and 14-19 are rejected under 35 U.S.C. 103 as being unpatentable over Heikkila (US20180038805A1) in view of Lavie (US11915479B1).
Regarding claim 1, Heikkila teaches a method of detecting damages in a freight container ([0001] The aspects of the disclosed embodiments relate to a container inspection arrangement and a method for inspecting a container. The aspects of the disclosed embodiments also relate to software for performing the method steps, and an apparatus for executing the software), the method comprising:
capturing at least one first image of at least one part of the freight container at an angle deviating from a perpendicular direction in respect of the at least one part of the freight container, capturing at least one second image of the same at least one part of the freight container at an angle substantially perpendicular in respect of the same at least one part of the freight container ([0027] FIG. 1 shows an arrangement for inspecting a container according to an embodiment of the invention. A container 100 is gripped by a gripper element 104 of a container handling apparatus for transferring the container 100. The gripper element 104 is equipped with at least one camera or more cameras 102. The camera 102 is placed in the gripper element 104 in such a way that when the container 100 is gripped by the gripper element 104, the camera 102 is trained on the container 100, being placed in that part of the gripper element 104 that is facing the container 100 to be gripped. The camera 102 placed in the gripper element 104 makes it possible to take images of the top of the container 100, for imaging the top side of the container 100),
PNG
media_image1.png
462
540
media_image1.png
Greyscale
Fig. 1 of Heikkila shows two cameras (102), one at an angle deviating from a perpendicular direction and one at an angle substantially perpendicular to the freight container.
analyzing the at least one first image captured at the angle deviating from the perpendicular direction in respect of the at least one part of the freight container to detect in the at least one first image at least one damage in the respective at least one part of the freight container, analyzing the at least one second image captured at the angle substantially perpendicular in respect of the same at least one part of the freight container to detect in the at least one second image at least one damage in the respective at least one part of the freight container ([0026] According to the embodiments, images of a container are taken at the container handling stage. The image data is transferred to software for analysis. On the basis of the analysis, potential damage points may be detected),
in response to detecting at least one damage in the at least one part of the freight container providing at least one damage information image regarding to the respective at least one part of the freight container, wherein the damage information image comprises the second image captured at the angle substantially perpendicular in respect of the respective part of the freight container and includes the at least one damage detected in at least one of the respective first image captured at the angle deviating from the perpendicular direction in respect of the respective part of the freight container or in this second image captured at the angle substantially perpendicular in respect of the same respective part of the freight container ([0026] A three-dimensional object may be formed of the image data, also including the detected potential damage points. For example, an existing 3D model of the container may be supplemented with image data, other received information, and/or information derived from the image data. [0046] The 3D object includes image data on the container and the results of the analysis on possible damage and other aspects analyzed),
Heikkila does not teach wherein at least one damage detected in the first image captured at the angle deviating from the perpendicular direction in respect of the respective part of the freight container is projected into the respective second image captured at the angle substantially perpendicular in respect of the same respective part of the freight container when providing the damage information image from the respective part of the freight container.
Lavie, in the same field of endeavor of detecting damage, teaches wherein at least one damage detected in the first image captured at the angle deviating from the perpendicular direction (angle image) in respect of the respective part of the freight container is projected into the respective second image captured at the angle substantially perpendicular (side image) in respect of the same respective part of the freight container when providing the damage information image from the respective part of the freight container ([col. 9 ln. 8-9] Further, for each projection, a pair of angle-side images will be identified using orientation detection results. [col. 8 ln. 48-54] For each of the four angled views, two lists of panels will be created that will be used to determine which side view the damage on the panel should be projected to. For example, in the back-left view (see FIG. 8, view A) both back-left doors and boot are visible, but the back-left door should be projected to left view and boot should be projected to the back view (FIG. 8, view E)).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Heikkila with the teachings of Lavie to project detected damage because "allometry and homography are used to calculate the size of damage on panels at different camera angles allowing for improved accuracy in damage estimates as compared to prior techniques" [Lavie col. 7 ln. 35-38].
Regarding claim 2, Heikkila and Lavie teach the method of claim 1. Heikkila further teaches presenting visually the at least one damage information image in response to detecting at least one damage in the respective at least one part of the freight container ([0046] In this way, a 3D object is formed which corresponds to the imaged object (container). Furthermore, the 3D object is supplemented with the indication of damage detected on the basis of the image analysis. The corresponding data may be presented in a written report and/or list. The location of the damage is displayed in the 3D object, and it may be visible on two sides of the 3D object, in the same location of the wall. The 3D object includes image data on the container and the results of the analysis on possible damage and other aspects analyzed. The 3D object may be displayed to the user who may examine the object three-dimensionally, rotate it and zoom it in and out. Furthermore, the user may be shown a specific part of the object seen from different angles or directions, next to each other).
Regarding claim 3, Heikkila and Lavie teach the method of claim 1. Heikkila further teaches storing at least one of the at least one first image, the at least one second image or the at least one damage information image of the at least one part of the freight container as well as an individualization identifier of the freight container in response to detecting at least one damage in the respective at least one part of the freight container ([0068] On the basis of the detected damage, a repair card may be associated with said detected damage, containing information relating to the repair of the damage. The repair card may be stored in a memory, or it may be available, for example downloadable. The repair card may be retrieved on the basis of detected damage and associated with the detected damage. Image data, data formed of the images, modified data, and/or associated data may be transmitted between devices in an electronic and/or wireless way. [0055] The characteristic features and/or structural data and/or shapes, and/or the identification of the container may be stored in a memory for comparison and/or analysis).
Regarding claim 4, Heikkila and Lavie teach the method of claim 1. Heikkila further teaches retrieving from a group of images stored previously at least one of the at least one first image, the at least one second image or the at least one damage information image of the same at least one part of the same freight container in response to detecting at least one damage in the at least one part of the freight container under analysis, and comparing the previously stored at least one of the at least one first image, the at least one second image or the at least one damage information image of the same at least one part of the same freight container to the respective at least one of the at least one first image, the at least one second image or the at least one damage information image of the at least one part of the freight container under analysis to determine a change in the at least one damage in the freight container ([0043] In an embodiment, a large number of reference images are stored in a storage unit, to be accessible to the running software. There may be hundreds or thousands of reference images, or even more. Reference images may be collected, for example, from received image data on containers, and/or they may be entered separately…The received image data may be compared with reference images…From the received image data, it is also possible to analyze whether it approaches an image defined to be unacceptable. A relatively large number of reference images enables a reliable analysis based on them. As the number of images increases, various defects and/or discontinuities analyzed may be represented in a reliable way).
Regarding claim 6, Heikkila and Lavie teach the method of claim 1. Lavie teaches analyzing the at least one first image and the at least one second image to find at least one distinguishable non-damage related feature (panel) in the images, comparing the at least one distinguishable non-damage related feature found in the at least one first image and in the at least one second image to each other to find at least one same distinguishable non-damage related feature in the at least one first image and in the at least one second image, such that the at least one same distinguishable non-damage related feature forms in the at least one first image and in the at least one second image at least one corresponding keypoint, and to provide the damage information image ([col. 6 ln. 36-49] Vehicle Panel Identification sub-module 38 identifies the panels of the vehicle visible in the image. As illustrated in FIG. 6, more than 50 panels around the vehicle may be identified, including, for example, the bonnet, the front right door, the back left wheel, etc. In some embodiments, panel identification may being with edge identification based on the prior sub-module. With edges identified, a starting edge may be selected according to a rule-set and then by moving across the image identifying edges based on the rules, panels may be identified. In one embodiment, three parameters may be considered in parallel: edges distinguishing panels, panels indicating expected next edges and location analysis. In some embodiments, all three factors are compared and must match. [col. 7 ln. 50-59] Homography facilitates consolidation and comparison of damages detected on panels between different orientation images. Sizes of panels and damages calculated in one orientation are applied to the second related orientations. For example, sizes calculated on the side orientations are projected on the diagonal orientations. Homography enables model output cross comparison and validation between images. Vehicle orientation detection in the image analysis step is used to determine the appropriate homographic projections)
projecting into the at least one second image at least one damage of the freight container detected in the at least one first image based on the position of the at least one corresponding keypoint in the at least one first image and in the at least one second image ([col. 8 ln. 48-59] For each of the four angled views, two lists of panels will be created that will be used to determine which side view the damage on the panel should be projected to. For example, in the back-left view (see FIG. 8, view A) both back-left doors and boot are visible, but the back-left door should be projected to left view and boot should be projected to the back view (FIG. 8, view E). Some of the panels (e.g., taillights, rear bumper) may not appear on either list (no damage estimation will be done) or will be in two lists simultaneously (each damage will have two size estimations). For these panels identifying averaging or best fit techniques may be used to finalize damage size estimation).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Heikkila with the teachings of Lavie to compare features between images and project the detected damage based on the feature(s) because "allometry and homography are used to calculate the size of damage on panels at different camera angles allowing for improved accuracy in damage estimates as compared to prior techniques" [Lavie col. 7 ln. 35-38].
Regarding claim 7, Heikkila and Lavie teach the method of claim 1. Heikkila further teaches wherein at least one visually perceptible sign in the damage information image indicates the at least damage in the part of the freight container by at least one visually perceptible sign ([0046] The location of the damage is displayed in the 3D object, and it may be visible on two sides of the 3D object, in the same location of the wall. The 3D object includes image data on the container and the results of the analysis on possible damage and other aspects analyzed. The 3D object may be displayed to the user who may examine the object three-dimensionally, rotate it and zoom it in and out. Furthermore, the user may be shown a specific part of the object seen from different angles or directions, next to each other).
Regarding claim 8, Heikkila and Lavie teach the method of claim 1. Heikkila further teaches indicating a possible need to interrupt the transportation of the freight container ([0040] In an embodiment, a container carried by a truck is driven through a gate to a container handling area. Cameras are placed at the gate so that when the truck is driven through the gate, images of the top, the long sides and possibly the doors of the end wall of the container are taken. If damage in the container is detected at this stage when the truck arrives at the container handling area via the gate, the container may be directly forwarded to closer inspection and/or stored to wait for rejection, acceptance or measures of a repair report).
Heikkila does not teach determining a degree of damage of the damaged part of the freight container and in response to the degree of damage exceeding a predetermined limit value set for the degree of damage, providing a warning signal.
Lavie teaches determining a degree of damage of the damaged part of the freight container and in response to the degree of damage exceeding a predetermined limit value set for the degree of damage, providing a warning signal ([col. 10 ln. 31-39] In the use case where vehicles are being automatically assessed for classification in vehicle auctions, expert rules are and applied created to identify the auction category of the vehicle based on the vehicle make and model, and determined parameters such as panel damaged, damage type, and damage extent and/or size. For example, if the vehicle has any panel that is damaged by more than 50% regardless of damage type then the vehicle is classed as a Category C vehicle, i.e. a repairable salvage vehicle. [col. 15 ln. 46-53] The system backend identifies the vehicle, automatically estimates extent of damage, and compares current assessment with previous assessments, among other functions as described herein. The system backend provides also may provide services such as notification and reporting management or portal. FIG. 21 presents a block diagram of one example of such an overall system).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Heikkila with the teachings of Lavie to provide a notification of the damage extent because "reporting may span the entire vehicle lifecycle providing single instance reporting as well as comparative temporal reporting expanding use cases to before-and-after scenarios for insurance, vehicle rental, ride sharing, and vehicle leasing" [Lavie col. 10 ln. 43-47].
Regarding claim 9, Heikkila teaches a system configured to detect damages in a freight container ([0001] The aspects of the disclosed embodiments relate to a container inspection arrangement and a method for inspecting a container. The aspects of the disclosed embodiments also relate to software for performing the method steps, and an apparatus for executing the software), the system comprising:
at least one first image capturing unit (cameras 102) for capturing at least one first image of at least one part of the freight container at an angle deviating from a perpendicular direction in respect of the at least one part of the freight container, at least one second image capturing unit (cameras 102) for capturing at least one second image of the same at least one part of the freight container at an angle substantially perpendicular in respect of the same at least one part of the freight container ([0027] FIG. 1 shows an arrangement for inspecting a container according to an embodiment of the invention. A container 100 is gripped by a gripper element 104 of a container handling apparatus for transferring the container 100. The gripper element 104 is equipped with at least one camera or more cameras 102. The camera 102 is placed in the gripper element 104 in such a way that when the container 100 is gripped by the gripper element 104, the camera 102 is trained on the container 100, being placed in that part of the gripper element 104 that is facing the container 100 to be gripped. The camera 102 placed in the gripper element 104 makes it possible to take images of the top of the container 100, for imaging the top side of the container 100),
PNG
media_image1.png
462
540
media_image1.png
Greyscale
Fig. 1 of Heikkila shows two cameras (102), one at an angle deviating from a perpendicular direction and one at an angle substantially perpendicular to the freight container.
at least one image analysis unit (software unit 901) for analyzing the at least one first image captured at the angle deviating from the perpendicular direction in respect of the at least one part of the freight container to detect in the at least one first image at least one damage in the respective at least one part of the freight container, analyzing the at least one second image captured at the angle substantially perpendicular in respect of the same at least one part of the freight container to detect in the at least one second image at least one damage in the respective at least one part of the freight container ([0051] The software unit 901 and/or the cloud 903 may comprise an analysis program for processing and/or analyzing image data. [0026] According to the embodiments, images of a container are taken at the container handling stage. The image data is transferred to software for analysis. On the basis of the analysis, potential damage points may be detected),
at least one image combination unit (image processing unit) configured to provide at least one damage information image regarding to the respective at least one part of the freight container in response to the detection of at least one damage in the at least one part of the freight container, wherein the damage information image comprises the second image captured at the angle substantially perpendicular in respect of the respective part of the freight container and includes the at least one damage detected in at least one of the respective first image captured at the angle deviating from the perpendicular direction in respect of the respective part of the freight container or in this second image captured at the angle substantially perpendicular in respect of the same respective part of the freight container ([0026] A three-dimensional object may be formed of the image data, also including the detected potential damage points. For example, an existing 3D model of the container may be supplemented with image data, other received information, and/or information derived from the image data. [0046] The software unit 901 may comprise an image processing unit for processing the received image data 1020, identifying the location of said image data in the 3D model, and attaching it to the correct location in the 3D model),
Heikkila does not teach wherein the at least one image combination unit is configured to project at least one damage detected in the first image captured at the angle deviating from the perpendicular direction in respect of the respective part of the freight container into the respective second image captured at the angle substantially perpendicular in respect of the same respective part of the freight container to provide the damage information image from the respective part of the freight container.
Lavie, in the same field of endeavor of detecting damage, teaches to project at least one damage detected in the first image captured at the angle deviating from the perpendicular direction (angle image) in respect of the respective part of the freight container into the respective second image captured at the angle substantially perpendicular (side image) in respect of the same respective part of the freight container to provide the damage information image from the respective part of the freight container ([col. 9 ln. 8-9] Further, for each projection, a pair of angle-side images will be identified using orientation detection results. [col. 8 ln. 48-54] For each of the four angled views, two lists of panels will be created that will be used to determine which side view the damage on the panel should be projected to. For example, in the back-left view (see FIG. 8, view A) both back-left doors and boot are visible, but the back-left door should be projected to left view and boot should be projected to the back view (FIG. 8, view E)).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the system of Heikkila with the teachings of Lavie to project detected damage because "allometry and homography are used to calculate the size of damage on panels at different camera angles allowing for improved accuracy in damage estimates as compared to prior techniques" [Lavie col. 7 ln. 35-38].
Regarding claim 10, Heikkila and Lavie teach the system of claim 9. Heikkila further teaches wherein the system comprises at least one display unit (display 904) configured to present visually the at least one damage information image in response to detecting at least one damage in the respective at least one part of the freight container ([0046] In this way, a 3D object is formed which corresponds to the imaged object (container). Furthermore, the 3D object is supplemented with the indication of damage detected on the basis of the image analysis. The corresponding data may be presented in a written report and/or list. The location of the damage is displayed in the 3D object, and it may be visible on two sides of the 3D object, in the same location of the wall. The 3D object includes image data on the container and the results of the analysis on possible damage and other aspects analyzed. The 3D object may be displayed to the user who may examine the object three-dimensionally, rotate it and zoom it in and out. Furthermore, the user may be shown a specific part of the object seen from different angles or directions, next to each other).
Regarding claim 11, Heikkila and Lavie teach the system of claim 9. Heikkila further teaches wherein the system comprises at least one image database unit configured to store at least one of the at least one first image, the at least one second image or the at least one damage information image of the at least one part of the freight container as well as an individualization identifier of the freight container in response to detecting at least one damage in the respective at least one part of the freight container ([0068] On the basis of the detected damage, a repair card may be associated with said detected damage, containing information relating to the repair of the damage. The repair card may be stored in a memory, or it may be available, for example downloadable. The repair card may be retrieved on the basis of detected damage and associated with the detected damage. Image data, data formed of the images, modified data, and/or associated data may be transmitted between devices in an electronic and/or wireless way. [0055] The characteristic features and/or structural data and/or shapes, and/or the identification of the container may be stored in a memory for comparison and/or analysis).
Regarding claim 12, Heikkila and Lavie teach the system of claim 9. Heikkila further teaches wherein the system comprises at least one image retrieval and comparison unit (image processing unit) configured to retrieve from a group of images stored previously at least one of the at least one first image, the at least one second image or the at least one damage information image of the same at least one part of the same freight container in response to detecting at least one damage in the at least one part of the freight container under analysis, and comparing the previously stored at least one of the at least one first image, the at least one second image or the at least one damage information image of the same at least one part of the same freight container to the respective at least one of the at least one first image, the at least one second image or the at least one damage information image of the at least one part of the freight container under analysis to determine a change in the at least one damage in the freight container ([0043] In an embodiment, a large number of reference images are stored in a storage unit, to be accessible to the running software. There may be hundreds or thousands of reference images, or even more. Reference images may be collected, for example, from received image data on containers, and/or they may be entered separately…The received image data may be compared with reference images…From the received image data, it is also possible to analyze whether it approaches an image defined to be unacceptable. A relatively large number of reference images enables a reliable analysis based on them. As the number of images increases, various defects and/or discontinuities analyzed may be represented in a reliable way).
Regarding claim 14, Heikkila and Lavie teach the system of claim 9. Lavie teaches wherein the image analysis unit is configured to analyze the at least one first image and the at least one second image to find at least one distinguishable non-damage related feature (panel) in the images, comparing the at least one distinguishable non-damage related feature found in the at least one first image and in the at least one second image to each other to find at least one same distinguishable non-damage related feature in the at least one first image and in the at least one second image, such that the at least one same distinguishable non-damage related feature forms in the at least one first image and in the at least one second image at least one corresponding keypoint ([col. 6 ln. 36-49] Vehicle Panel Identification sub-module 38 identifies the panels of the vehicle visible in the image. As illustrated in FIG. 6, more than 50 panels around the vehicle may be identified, including, for example, the bonnet, the front right door, the back left wheel, etc. In some embodiments, panel identification may being with edge identification based on the prior sub-module. With edges identified, a starting edge may be selected according to a rule-set and then by moving across the image identifying edges based on the rules, panels may be identified. In one embodiment, three parameters may be considered in parallel: edges distinguishing panels, panels indicating expected next edges and location analysis. In some embodiments, all three factors are compared and must match. [col. 7 ln. 50-59] Homography facilitates consolidation and comparison of damages detected on panels between different orientation images. Sizes of panels and damages calculated in one orientation are applied to the second related orientations. For example, sizes calculated on the side orientations are projected on the diagonal orientations. Homography enables model output cross comparison and validation between images. Vehicle orientation detection in the image analysis step is used to determine the appropriate homographic projections)
and the image combination unit is configured to project into the at least one second image at least one damage of the freight container detected in the at least one first image based on the location of the at least one corresponding keypoint in the at least one first image and in the at least one second image ([col. 8 ln. 48-59] For each of the four angled views, two lists of panels will be created that will be used to determine which side view the damage on the panel should be projected to. For example, in the back-left view (see FIG. 8, view A) both back-left doors and boot are visible, but the back-left door should be projected to left view and boot should be projected to the back view (FIG. 8, view E). Some of the panels (e.g., taillights, rear bumper) may not appear on either list (no damage estimation will be done) or will be in two lists simultaneously (each damage will have two size estimations). For these panels identifying averaging or best fit techniques may be used to finalize damage size estimation).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the system of Heikkila with the teachings of Lavie to compare features between images and project the detected damage based on the feature(s) because "allometry and homography are used to calculate the size of damage on panels at different camera angles allowing for improved accuracy in damage estimates as compared to prior techniques" [Lavie col. 7 ln. 35-38].
Regarding claim 15, Heikkila and Lavie teach the system of claim 9. Heikkila further teaches wherein the system is arranged to indicate the at least one damage in the part of the freight container by at least one visually perceptible sign in the damage information image ([0046] The location of the damage is displayed in the 3D object, and it may be visible on two sides of the 3D object, in the same location of the wall. The 3D object includes image data on the container and the results of the analysis on possible damage and other aspects analyzed. The 3D object may be displayed to the user who may examine the object three-dimensionally, rotate it and zoom it in and out. Furthermore, the user may be shown a specific part of the object seen from different angles or directions, next to each other).
Regarding claim 16, Heikkila and Lavie teach the system of claim 9. Heikkila further teaches wherein the at least one analysis unit is configured to indicating a possible need to interrupt the transportation of the freight container ([0040] In an embodiment, a container carried by a truck is driven through a gate to a container handling area. Cameras are placed at the gate so that when the truck is driven through the gate, images of the top, the long sides and possibly the doors of the end wall of the container are taken. If damage in the container is detected at this stage when the truck arrives at the container handling area via the gate, the container may be directly forwarded to closer inspection and/or stored to wait for rejection, acceptance or measures of a repair report).
Heikkila does not teach to determine a degree of damage of the damaged part of the freight container and in response to the degree of damage exceeding a predetermined limit value set for the degree of damage, the system is configured to provide a warning signal
Lavie teaches to determine a degree of damage of the damaged part of the freight container and in response to the degree of damage exceeding a predetermined limit value set for the degree of damage, the system is configured to provide a warning signal ([col. 10 ln. 31-39] In the use case where vehicles are being automatically assessed for classification in vehicle auctions, expert rules are and applied created to identify the auction category of the vehicle based on the vehicle make and model, and determined parameters such as panel damaged, damage type, and damage extent and/or size. For example, if the vehicle has any panel that is damaged by more than 50% regardless of damage type then the vehicle is classed as a Category C vehicle, i.e. a repairable salvage vehicle. [col. 15 ln. 46-53] The system backend identifies the vehicle, automatically estimates extent of damage, and compares current assessment with previous assessments, among other functions as described herein. The system backend provides also may provide services such as notification and reporting management or portal. FIG. 21 presents a block diagram of one example of such an overall system).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the system of Heikkila with the teachings of Lavie to provide a notification of the damage extent because "reporting may span the entire vehicle lifecycle providing single instance reporting as well as comparative temporal reporting expanding use cases to before-and-after scenarios for insurance, vehicle rental, ride sharing, and vehicle leasing" [Lavie col. 10 ln. 43-47].
Regarding claim 17, Heikkila and Lavie teach the system of claim 9. Heikkila further teaches wherein the part of the freight container is one of a side wall of the freight container, an end wall of the freight container, a roof of the side container, or a bottom of the freight container ([0029] The side walls and the topmost wall of the exterior walls of the container 100 may be imaged by cameras 102 in the masts 180).
Regarding claim 18, Heikkila and Lavie teach the system of claim 9. Lavie teaches wherein the at least one analysis unit is configured to detect in the at least one first image at least one different kind of damage than in the at least one second image [col. 13 ln. 59-65] FIG. 16 shows an example of a typical image input showing of a front end collision. In the photo we can identify a damage hood, passenger side wheels and tires, a driver side headlight, damaged front grill, windshield, etc. We know the car is a Honda Element and it collided with some object on the passenger side approximately at a 45 degree angle relative to the front of the car. [col. 3 ln. 36-39] FIG. 15 illustrates and example of a training step for training the system to identify damage and location of damage using an image of a right front bumper with a scuff and some scratches).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the system of Heikkila with the teachings of Lavie to determine different types of damage in the different images "to calculate the size of damage on panels at different camera angles allowing for improved accuracy in damage estimates as compared to prior techniques" [Lavie col. 7 ln. 36-38].
Regarding claim 19, Heikkila and Lavie teach the system of claim 9. Heikkila teaches detecting damage in a freight container (Fig. 1). Lavie teaches wherein the system is configured to detect in the at least one first image at least one of scratch, protrusion, or depression ([col. 13 ln. 59-65] FIG. 16 shows an example of a typical image input showing of a front end collision. In the photo we can identify a damage hood, passenger side wheels and tires, a driver side headlight, damaged front grill, windshield, etc. We know the car is a Honda Element and it collided with some object on the passenger side approximately at a 45 degree angle relative to the front of the car), and in the at least one second image at least one of deformation, hole, scratch, or rust ([col. 3 ln. 36-39] FIG. 15 illustrates and example of a training step for training the system to identify damage and location of damage using an image of a right front bumper with a scuff and some scratches).
Therefore, it would have been obvious to a person of ordinar