Detailed Action
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . See 35 U.S.C. § 100 (note).
Drawings
The drawing(s) filed on February 20, 2025, are accepted by the Examiner.
Status of Claims
Claims 1–14 are pending in this application.
Preliminary Amendment
The Preliminary Amendment submitted on February 20, 2025, containing amendments to the claims are acknowledged.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on February 20, 2025, is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Issues Under 35 U.S.C. § 112
Claim Rejection
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, 4 and 12 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
With respect to claim 3: Claim 3 recites:
The road surface diagnosis system according to claim 2, wherein the at least one processor is further configured to execute the instructions to: display the information indicating the certainty with an icon indicating a site having the certainty lower than a certainty of another site on a map.
The phrase “certainty lower than a certainty of another site” renders the claim indefinite because “another site” is not identified, and the claim does not provide an objective comparison baseline for determining when the certainty is lower. This is a relative/degree-type limitation, and the metes and bounds of the claim are unclear under MPEP § 2173. The specification describes certainty based on similarity relative to a threshold and describes displaying an icon for a site whose certainty is lower than that of another site. See Publication ¶¶ [0072]–[0074]. However, the claim does not recite the threshold, reference site, average certainty, predetermined criterion, or other objective standard used to determine the comparative certainty. For purposes of examination, Examiner is reading the claim as requiring display of an icon on a map for a site having a determination certainty that is lower than the certainty associated with some other site.
With respect to claim 4: Claim 4 recites:
The road surface diagnosis system according to claim 2, wherein the at least one processor is further configured to execute the instructions to: display the information indicating the road surface deterioration including a road surface image at a site where the certainty is low among road surface images in which it is determined whether the recognized road surface deterioration is identical.
The phrase “certainty is low” renders the claim indefinite because “low” is a term of degree and the claim does not provide an objective boundary for determining when the certainty is low. See MPEP § 2173. The specification discusses certainty and low certainty. See Publication ¶¶ [0072]–[0074]. However, the claim does not tie the term “low” to a threshold, range, calculated value, or other objective criterion. Applicant may clarify the claim by reciting an objective standard, for example, “where the certainty is below a predetermined threshold.” For purposes of examination, Examiner is reading the claim as requiring display of a road surface image corresponding to a site for which the determination certainty is relatively low.
With respect to claim 12: Claim 12 recites:
The road surface diagnosis system according to claim 1, wherein the at least one processor is further configured to execute the instructions to: weigh a degree of deterioration calculated from each of the plurality of road surface images based on similarity between road surface deterioration; and calculate a degree of deterioration at a site where the each road surface image is captured based on a plurality of the weighted degrees of deterioration.
The phrase ‘where the each road surface image is captured’ is grammatically improper (the article ‘the’ should not precede ‘each’) and renders the claim unclear as to which site’s degree of deterioration is being calculated. Applicant may clarify by amending to ‘where each road surface image is captured’ or ‘at a site where each of the road surface images is captured.’ Additionally, the term “weigh” appears to be intended to mean “weight,” which creates ambiguity in the calculation step. The specification supports weighting deterioration degrees based on similarity and calculating a weighted average of deterioration degrees. See Publication ¶ [0083]; see also ¶¶ [0136]–[0137]. Applicant may clarify the claim by amending “weigh” to “weight” and revising the phrase to recite, for example, “calculate a degree of deterioration at a site where each road surface image is captured based on a plurality of the weighted degrees of deterioration.” For purposes of examination, Examiner is reading the claim as requiring weighting of deterioration-degree values calculated from road surface images based on similarity between road surface deterioration, and calculating a site-level degree of deterioration based on the weighted values.
Art Rejections
Obviousness Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 5, 11, 13 and 14 are rejected under 35 U.S.C. § 103 as being unpatentable over the combination of US Patent Application Publication Shimomura et al. (US 20130169794 A1) (hereinafter referred to as “Shimomura”) in view of US Patent Yoshida et al. (US 12,442,773 B2) (hereinafter referred to as “Yoshida”).
Claims 2–4, 10 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over “Shimomura” in view of “Yoshida” as applied to claims 1 above, and further in view of US Patent Application Publication Suzuki et al. (US 20130170701 A1) (hereinafter referred to as “Suzuki”).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over “Shimomura” in view of “Yoshida” as applied to claims 1 above, and further in view of US Patent Application Publication Yonekawa et al. (US 20180195973 A1) (hereinafter referred to as “Yonekawa”).
Claims 7 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over “Shimomura” in view of “Yoshida” as applied to claims 1 above, and further in view of US Patent Application Publication Ihara et al. (US 20080013790 A1) (hereinafter referred to as “Ihara”).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over “Shimomura” in view of “Yoshida” and further in view of “Ihara” as applied to claims 1 and 7 above, and further in view of US Patent Application Publication Suzuki et al. (US 20130170701 A1) (hereinafter referred to as “Suzuki”) and further in view of US Patent Application Publication Spaeth et al. (US 20130125002 A1) (hereinafter referred to as “Spaeth”).
With respect to claim 1, Shimomura discloses a road surface diagnosis system (Fig. 1 – a road surface inspection system) comprising:
at least one memory storing instructions (Shimomura teaches “the storage unit 13 is a storage device such as a semiconductor memory device (for example, a flash memory), a hard disk, or an optical disc. The storage unit 13 is not limited to the above-mentioned storage device, but a random access memory (RAM) or a read only memory (ROM) may be used..” See Shimomura, ¶ [0056] and [0057]; Fig. 3.); and
at least one processor configured to execute the instructions to (Shimomura teaches “the storage unit 13 stores an operating system (OS) that is executed by the control unit 15.” See Shimomura, ¶¶ [0057], Fig. 3. Shimomura further discloses “The control unit 15 includes a program in which various processing procedures are described and an internal memory for storing control data, and executes various processes using the program and the control data.” See Shimomura, ¶ [0066];):
acquire a plurality of road surface images (Shimomura teaches “Among these sensors, the camera 31 is an imaging device that captures an image using an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). As an embodiment, when capturing a road image at a predetermined frame rate, the camera 31 correlates the road image with a captured time by embedding the captured time in the frames of the road image as header information and then stores the road image in the storage unit 34.” See Shimomura, ¶ [0042]. Shimomura further discloses “As an embodiment, when the video data is read from the memory card 20 by the reader/writer 11, the registration unit 15a registers the video data in the storage unit 13 for each vehicle number of the patrol car 3.” See Shimomura, ¶ [0067]. Shimomura further discloses “As an embodiment, the abnormal region detecting unit 15b starts its processing when new video data 13a is registered in the storage unit 13. First, the abnormal region detecting unit 15b sequentially reads the frames of a road image included in the video data 13a stored in the storage unit 13.” See Shimomura, ¶ [0069].);
recognize road surface deterioration from each of the acquired road surface images (Shimomura teaches “After that, the abnormal region detecting unit 15b detects an abnormal region, in which it can be estimated that a discoloration or the like is present in the pavement of the road surface, from the specified image processing execution target region E.” See Shimomura, ¶ [0070]. Shimomura further discloses “By determining the size of the area, the overlap determining unit 15c determines whether the abnormal region has a size such that it can be estimated that the abnormal region is a bump, a groove, or a crack on the road surface, that is, whether the abnormal region is a small stone or the like.” See Shimomura, ¶ [0072]); and
display, based on the determination, information of the recognized road surface deterioration as information indicating one road surface deterioration or information indicating different road surface deterioration (Shimomura teaches “upon receiving a browse request from the subscriber terminal 50, the providing unit 15f generates a map screen in which the coordinate position of the position data included in the sensing data 13b is mapped onto map information of a predetermined range (for example, a jurisdictional area). In this case, the providing unit 15f maps the coordinate positions corresponding to the deterioration data and the deterioration candidate data included in the browsing data 13d among the coordinate positions of the position data included in the sensing data 13b in a display form different from that of the other coordinate positions.” See Shimomura, ¶ [0084]; Fig. 10. Shimomura further discloses “a map screen 400 on which the coordinate positions of the position data included in the sensing data 13b are mapped is displayed on the subscriber terminal 50. In the map screen 400, a coordinate position 400a corresponding to the deterioration data included in the browsing data 13d is displayed as a black mark. Further, in the map screen 400, a coordinate position 400b corresponding to the deterioration candidate data is displayed as a dotted mark.” See Shimomura, ¶ [0085]; Fig. 10).
However, Shimomura fails to explicitly disclose convert at least one of the plurality of road surface images; compare the plurality of road surface images in each of which the road surface at the identical site is imaged, the plurality of road surface images including the converted road surface image, and determine whether road surface deterioration recognized from the road surface images is an identical road surface deterioration.
Yoshida, working in the same field of endeavor, recognizes this problem and teaches convert at least one of the plurality of road surface images (Yoshida teaches “a crack detection device includes: processing circuitry to: acquire image data acquired by imaging a road surface from an oblique direction with respect to the road surface; classify the acquired image data into an acceptable range with a resolution higher than a standard value, and an unacceptable range with a resolution equal to or less than the standard value; and output acceptable data being image data of a part classified into the acceptable range, as data to detect a crack on the road surface.” See Yoshida, claim 1; Abstract. Yoshida’s classification of image data into acceptable and unacceptable ranges based on resolution constitutes converting the image data);
compare the plurality of road surface images in each of which the road surface at the identical site is imaged, the plurality of road surface images including the converted road surface image (Yoshida teaches “the image data is repeatedly acquired by the imaging device while the moving object is moving,” and “the processing circuitry extracts effective data by retaining acceptable data with a high resolution with respect to an overlapping area that overlaps with other acceptable data, from acceptable data acquired from each of a plurality of pieces of image data repeatedly acquired.” See Yoshida, claim 3. Yoshida further discloses “the plurality of pieces of image data are acquired by taking images of the overlapping area depending on imaging intervals. Further, there is a case wherein acceptable ranges with respect to each image data also overlap with each other. An area where the acceptable ranges overlap with one another is the overlapping area 34.” See Yoshida, col. 6, lines 54-61; Figs. 12-13), and
determine whether road surface deterioration recognized from the road surface images is an identical road surface deterioration (Yoshida teaches “In the area classified into the acceptable range, the closer the area is to the imaging device 30, the higher the resolution is. Therefore, with respect to the overlapping area 34, acceptable data in image data acquired earlier has a higher resolution than that of acceptable data in image data acquired later. Therefore, with respect to the overlapping area 34, the image extraction unit 26 retains the acceptable data in the image data acquired earlier, and deletes the acceptable data in the image data acquired later.” See Yoshida, col. 6, line 61 - col. 7, line 7; Figs. 12-13. Yoshida’s extraction of effective data by comparing overlapping areas from multiple same-site images and retaining high-resolution data while deleting lower-resolution data teaches determining which detected deterioration represents identical road surface deterioration. Yoshida further discloses “the processing circuitry displays the output data on a display device,” and “the processing circuitry displays at a time the effective data extracted from each of the plurality of pieces of image data, by arranging the effective data according to an acquisition order of the plurality of pieces of image data.” See Yoshida, claims 4-5.).
At the time of the invention, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have included in the invention of Shimomura to apply Yoshida’s overlapping-image processing and effective-data extraction as taught by Yoshida since doing so would have predictably and advantageously allowed Shimomura’s road surface inspection system to reduce redundant or erroneous detections when the same road surface area is imaged multiple times from a moving vehicle, and to display one deterioration indication for the same road surface deterioration detected in overlapping same-site images while displaying separate indications for different deterioration. Specifically, both Shimomura and Yoshida address detecting road surface deterioration from vehicle-acquired road surface images. Shimomura detects abnormal regions on a road surface from road images captured by a camera mounted on a patrol car and displays deterioration positions on a map. See Shimomura, ¶¶ [0026], [0029]-[0031], [0084]-[0085]. Yoshida addresses the problem that “in road deterioration detection by image analysis or acceleration analysis described above, a detection result with sufficient accuracy may not be obtained due to the influence of weather, an artifact on a road, or the like,” and “when a crack is detected by simply using image data obtained by taking an image from an angle, there is a possibility that the crack cannot be detected appropriately.” See Yoshida, Background, col. 1, lines 35-48.
Therefore, the claimed subject matter would have been obvious to a person having ordinary skill in the art at the time the invention was made.
With respect to claim 5, which claim 1 is incorporated, Shimomura discloses receive, from a user, selection (Shimomura teaches “when the coordinate values of the latitude and longitude corresponding to the deterioration data or the deterioration candidate data are selected on the map screen by the subscriber terminal 50, the providing unit 15f generates a browsing screen including a road image, a change in the gravitational acceleration before a predetermined period elapses from the captured time of the road image, and a map screen and transmits the browsing screen to the subscriber terminal 50.” See Shimomura, ¶ [0086]; Fig. 11. Shimomura teaches receiving user selection on the map screen to trigger display of detailed deterioration information).
However, Shimomura fails to explicitly disclose selection as to whether road surface deterioration is identical, and change determination as to whether the road surface deterioration is identical according to the received selection.
Yoshida, working in the same field of endeavor, recognizes this problem and teaches selection as to whether road surface deterioration is identical, and change determination as to whether the road surface deterioration is identical according to the received selection (Yoshida teaches “the processing circuitry extracts effective data by retaining acceptable data with a high resolution with respect to an overlapping area that overlaps with other acceptable data, from acceptable data acquired from each of a plurality of pieces of image data repeatedly acquired.” See Yoshida, claim 3. Yoshida further discloses “with respect to the overlapping area 34, the image extraction unit 26 retains the acceptable data in the image data acquired earlier, and deletes the acceptable data in the image data acquired later.” See Yoshida, col. 7, lines 4-7. Yoshida teaches processing overlapping same-site images and determining which detected deterioration represents identical deterioration (i.e., retaining one representation and deleting redundant data).
At the time of the invention, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have included in the invention of Shimomura in view of Yoshida’s overlapping-image processing and effective-data extraction as taught by Yoshida, and to further allow user selection to change determination as to whether deterioration is identical, since doing so would have predictably and advantageously allowed Shimomura’s road surface inspection system to reduce redundant or erroneous detections when the same road surface area is imaged multiple times from a moving vehicle, to display one deterioration indication for the same road surface deterioration detected in overlapping same-site images, and to provide users with flexibility to override automatic determinations when manual review indicates different results. Specifically, both Shimomura and Yoshida address detecting road surface deterioration from vehicle-acquired road surface images. Shimomura detects abnormal regions on a road surface from road images captured by a camera mounted on a patrol car, displays deterioration positions on a map, and receives user selection to view detailed deterioration information. See Shimomura, ¶¶ [0026], [0029]-[0031], [0084]-[0087]; Fig. 10-11. Yoshida addresses the problem that “in road deterioration detection by image analysis or acceleration analysis described above, a detection result with sufficient accuracy may not be obtained due to the influence of weather, an artifact on a road, or the like,” and “when a crack is detected by simply using image data obtained by taking an image from an angle, there is a possibility that the crack cannot be detected appropriately.” See Yoshida, Background, col. 1, lines 35-48. Yoshida teaches automatically determining identical deterioration from overlapping same-site images. See Yoshida, claim 3; col. 6-7. Combined with Shimomura’s teaching of receiving user selection to interact with displayed deterioration information (see Shimomura, ¶ [0086]), it would have been obvious to allow user selection to change determination as to whether deterioration is identical because providing user override functionality for automatic determinations is a well-known design principle in inspection and monitoring systems, allowing human expertise to correct automated errors or ambiguous cases where automatic determination may be uncertain.
Therefore, the claimed subject matter would have been obvious to a person having ordinary skill in the art at the time the invention was made.
With respect to claim 11, which claim 1 is incorporated, Shimomura discloses display a position of identical road surface deterioration on a map with one icon (Shimomura teaches generating a map screen and displaying deterioration positions as icons on the map. Specifically, Shimomura teaches that “upon receiving a browse request from the subscriber terminal 50, the providing unit 15f generates a map screen in which the coordinate position of the position data included in the sensing data 13b is mapped onto map information of a predetermined range,” and displays coordinate positions of deterioration positions as icons on the map. See Shimomura, ¶¶ [0084]-[0085]; Fig. 10).
With respect to claim 13, (drawn to a method) the proposed combination of Shimomura in view of Yoshida, explained in the rejection of system claim 1 renders obvious the steps of the method of claim 13, because these steps occur in the operation of the apparatus as discussed above. Thus, the arguments similar to that presented above for claim 1 are equally applicable to claim 13.
With respect to claim 14, (drawn to a computer-readable program) the proposed combination of Shimomura in view of Yoshida, explained in the rejection of system claim 1 renders obvious the steps of the computer-readable program of claim 14, because these steps occur in the operation of the apparatus as discussed above. Thus, the arguments similar to that presented above for claim 1 are equally applicable to claim 14. Further Shimomura teaches that “The storage unit 13 stores an operating system (OS) that is executed by the control unit 15 and various programs such as a road surface inspection program for inspecting the road surface,” and “The control unit 15 includes a program in which various processing procedures are described and an internal memory for storing control data, and executes various processes using the program and the control data.” See Shimomura, ¶¶ [0057], [0066]. Yoshida expressly teaches “a non-transitory computer readable medium storing a crack detection program to make a computer function as a crack detection device.” See Yoshida, claim 11; col. 3, lines 15-17.
Claims 2–4, 10 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over “Shimomura” in view of “Yoshida” as applied to claims 1 above, and further in view of US Patent Application Publication Suzuki et al. (US 20130170701 A1) (hereinafter referred to as “Suzuki”).
With respect to claim 2, which claim 1 is incorporated, neither Shimomura nor Yoshida appears to explicitly disclose display the information indicating the road surface deterioration including information indicating certainty of determination in determining whether the road surface deterioration is identical.
Suzuki, working in the same field of endeavor, recognizes this problem and teaches display the information indicating the road surface deterioration including information indicating certainty of determination in determining whether the road surface deterioration is identical (Under the broadest reasonable interpretation, “certainty of determination in determining whether the road surface deterioration is identical” encompasses reliability or confidence information associated with deterioration detection and comparison. Suzuki teaches determining reliability of deterioration detection based on road surface conditions. Specifically, Suzuki teaches “lowering the reliability of the road deterioration detection result in a case where a first condition is detected as the road surface condition compared to a case where the first condition is not detected as the road surface condition,” and displaying reliability information (high, medium, low) with deterioration detection results. See Suzuki, claim 2; Fig. 6; Fig. 13; ¶¶ [0073]-[0076], [0097]).
At the time of the invention, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have included in the invention of Shimomura in view of Yoshida’s overlapping-image processing and effective-data extraction as taught by Yoshida, and further in view of Suzuki’s reliability determination and display as taught by Suzuki, since doing so would have predictably and advantageously allowed Shimomura’s road surface inspection system to reduce redundant or erroneous detections when the same road surface area is imaged multiple times from a moving vehicle, to display one deterioration indication for the same road surface deterioration detected in overlapping same-site images while displaying separate indications for different deterioration, and to provide users with reliability information to evaluate detection results.
Specifically, Shimomura, Yoshida, and Suzuki all address detecting road surface deterioration from vehicle-acquired road surface images. Shimomura detects abnormal regions on a road surface from road images captured by a camera mounted on a patrol car and displays deterioration positions on a map. See Shimomura, ¶¶ [0026], [0029]-[0031], [0084]-[0085]. Yoshida addresses the problem that “in road deterioration detection by image analysis or acceleration analysis described above, a detection result with sufficient accuracy may not be obtained due to the influence of weather, an artifact on a road, or the like,” and “when a crack is detected by simply using image data obtained by taking an image from an angle, there is a possibility that the crack cannot be detected appropriately.” See Yoshida, Background, col. 1, lines 35-48. Suzuki similarly recognizes that detection reliability varies based on road surface conditions and teaches displaying reliability information with deterioration detection results to allow users to evaluate detection quality. See Suzuki, ¶¶ [0053]-[0062]; Fig. 6; Fig. 7. Combined with Yoshida’s teaching of processing overlapping same-site road surface images and retaining one effective representation (see Yoshida, claim 3; col. 6-7), it would have been obvious to display reliability or certainty information when determining whether deterioration detected in overlapping images represents identical deterioration, because Suzuki teaches that detection reliability varies based on conditions and that displaying reliability allows users to evaluate detection results. See Suzuki, ¶¶ [0053]-[0062].
Therefore, the claimed subject matter would have been obvious to a person having ordinary skill in the art at the time the invention was made.
With respect to claim 3, which claim 2 is incorporated, Shimomura discloses display the information indicating the certainty with an icon indicating a site having the certainty lower than a certainty of another site on a map (Under the broadest reasonable interpretation, “an icon indicating a site having the certainty lower than a certainty of another site on a map” encompasses displaying different icons for sites with different levels of confidence or detection certainty. Shimomura teaches displaying different icons on a map for deterioration data and deterioration candidate data, where deterioration candidate data represents sites with lower certainty requiring further inspection. Specifically, Shimomura teaches “In this case, the providing unit 15f maps the coordinate positions corresponding to the deterioration data and the deterioration candidate data included in the browsing data 13d among the coordinate positions of the position data included in the sensing data 13b in a display form different from that of the other coordinate positions,” and “In the map screen 400, a coordinate position 400a corresponding to the deterioration data included in the browsing data 13d is displayed as a black mark. Further, in the map screen 400, a coordinate position 400b corresponding to the deterioration candidate data is displayed as a dotted mark.” See Shimomura, ¶¶ [0084]-[0085]; Fig. 10. Shimomura further discloses that deterioration candidate data is generated for sites with lower certainty. See Shimomura, ¶¶ [0063], [0079]. Therefore, Shimomura displays sites having different levels of certainty with different display forms (icons) on a map).
With respect to claim 4, which claim 2 is incorporated, Shimomura discloses display the information indicating the road surface deterioration including a road surface image at a site where the certainty is low among road surface images in which it is determined whether the recognized road surface deterioration is identical (Under the broadest reasonable interpretation, “a road surface image at a site where the certainty is low” encompasses displaying road surface images for sites identified as deterioration candidates or sites with lower confidence than confirmed deterioration sites. Shimomura teaches displaying road surface images for deterioration positions including positions with lower certainty (deterioration candidate positions). Specifically, Shimomura teaches that “when the coordinate values of the latitude and longitude corresponding to the deterioration data or the deterioration candidate data are selected on the map screen by the subscriber terminal 50, the providing unit 15f generates a browsing screen including a road image, a change in the gravitational acceleration before a predetermined period elapses from the captured time of the road image, and a map screen and transmits the browsing screen to the subscriber terminal 50,” and “FIG. 11 is a diagram illustrating an example of a screen transmitted to the subscriber terminal 50. As illustrated in FIG. 11, the browsing screen 500 including a road image 510, a change 520 in the gravitational acceleration before a predetermined period elapses from the captured time of the road image 510, and a map screen 540 is displayed on the subscriber terminal 50.” See Shimomura, ¶¶ [0086]-[0087]; Fig. 11. Shimomura’s deterioration candidate data represents sites with lower certainty. See Shimomura, ¶¶ [0063], [0079]. Therefore, Shimomura displays road surface images for sites with varying certainty levels, including sites with lower certainty).
With respect to claim 10, which claim 1 is incorporated, Shimomura in view of Yoshida fails to explicitly disclose determine that the road surface deterioration is identical when similarity of the road surface deterioration between road surface images to be compared exceeds a threshold value determined according to driving information including at least one of a traveling direction, a steering wheel operation, a speed, or an acceleration of a mobile object on which an imaging device that captures the road surface images is mounted.
Suzuki, working in the same field of endeavor, recognizes this problem and teaches determine that the road surface deterioration is identical when similarity of the road surface deterioration between road surface images to be compared exceeds a threshold value determined according to driving information including at least one of a traveling direction, a steering wheel operation, a speed, or an acceleration of a mobile object on which an imaging device that captures the road surface images is mounted (Suzuki teaches adjusting thresholds for detecting road surface events based on vehicle speed and road type. Specifically, Suzuki teaches “when a threshold value to be compared with a deceleration is set, by setting a threshold value for an expressway so as to be lower than that of an ordinary road, it is possible to collect a larger number of abrupt deceleration samples from the high-speed region. Moreover, when a threshold value to be compared with the horizontal G is set, by setting a threshold value for a high-speed corner so as to be lower than that of a low-speed corner, it is possible to collect a larger number of hard turn samples from the high-speed region.” See Suzuki, ¶ [0081]. Suzuki’s teaching of adjusting threshold values based on vehicle speed and road type teaches adjusting threshold values according to driving information. {Under the broadest reasonable interpretation, “a threshold value determined according to driving information” encompasses adjusting detection or comparison thresholds based on vehicle driving conditions such as speed, acceleration, traveling direction, or road type})
At the time of the invention, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have included in the invention of Shimomura in view of Yoshida’s overlapping-image processing and effective-data extraction as taught by Yoshida, and further in view of Suzuki’s threshold adjustment based on driving information as taught by Suzuki, since doing so would have predictably and advantageously allowed Shimomura’s road surface inspection system to reduce redundant or erroneous detections when the same road surface area is imaged multiple times from a moving vehicle, to display one deterioration indication for the same road surface deterioration detected in overlapping same-site images while displaying separate indications for different deterioration, and to improve detection accuracy by adjusting thresholds based on vehicle driving conditions. Specifically, Shimomura, Yoshida, and Suzuki all address detecting road surface deterioration from vehicle-acquired road surface images. Shimomura detects abnormal regions on a road surface from road images captured by a camera mounted on a patrol car and displays deterioration positions on a map. See Shimomura, ¶¶ [0026], [0029]-[0031], [0084]-[0085]. Yoshida addresses the problem that “in road deterioration detection by image analysis or acceleration analysis described above, a detection result with sufficient accuracy may not be obtained due to the influence of weather, an artifact on a road, or the like,” and “when a crack is detected by simply using image data obtained by taking an image from an angle, there is a possibility that the crack cannot be detected appropriately.” See Yoshida, Background, col. 1, lines 35-48. Suzuki recognizes that vehicle driving conditions affect detection and teaches adjusting thresholds based on driving conditions to improve detection accuracy. See Suzuki, ¶ [0081].
Combined with Yoshida’s teaching of comparing overlapping same-site road surface images and retaining data based on resolution/quality (see Yoshida, claim 3; col. 6-7), it would have been obvious to adjust thresholds used in determining whether detected deteriorations are identical based on driving information, because Suzuki recognizes that vehicle driving conditions affect detection and that adjusting thresholds based on driving conditions improves detection accuracy. See Suzuki, ¶ [0081]; Yoshida, claim 3.
Therefore, the claimed subject matter would have been obvious to a person having ordinary skill in the art at the time the invention was made.
With respect to claim 12, which claim 1 is incorporated, Shimomura fails to explicitly disclose weigh a degree of deterioration calculated from each of the plurality of road surface images based on similarity between road surface deterioration; and calculate a degree of deterioration at a site where the each road surface image is captured based on a plurality of the weighted degrees of deterioration.
Yoshida, working in the same field of endeavor, recognizes this problem and teaches weigh a degree of deterioration calculated from each of the plurality of road surface images based on similarity between road surface deterioration (Yoshida teaches “the processing circuitry extracts effective data by retaining acceptable data with a high resolution with respect to an overlapping area that overlaps with other acceptable data, from acceptable data acquired from each of a plurality of pieces of image data repeatedly acquired See Yoshida, claim 3. Yoshida further discloses “with respect to the overlapping area 34, acceptable data in image data acquired earlier has a higher resolution than that of acceptable data in image data acquired later. Therefore, with respect to the overlapping area 34, the image extraction unit 26 retains the acceptable data in the image data acquired earlier, and deletes the acceptable data in the image data acquired later.” See Yoshida, col. 7, lines 4-7. Yoshida’s retention of acceptable high-resolution data for overlapping areas and deletion of lower-resolution data is functionally equivalent to assigning full weight to high-quality data and zero weight to low-quality data, thereby teaching weighing deterioration data based on quality/similarity).
At the time of the invention, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have included in the invention of Shimomura in view of Yoshida’s overlapping-image processing and effective-data extraction as taught by Yoshida, since doing so would have predictably and advantageously allowed Shimomura’s road surface inspection system to reduce redundant or erroneous detections when the same road surface area is imaged multiple times from a moving vehicle, and to display one deterioration indication for the same road surface deterioration detected in overlapping same-site images while displaying separate indications for different deterioration. Specifically, both Shimomura and Yoshida address detecting road surface deterioration from vehicle-acquired road surface images. Shimomura detects abnormal regions on a road surface from road images captured by a camera mounted on a patrol car and displays deterioration positions on a map. See Shimomura, ¶¶ [0026], [0029]-[0031], [0084]-[0085]. Yoshida addresses the problem that “in road deterioration detection by image analysis or acceleration analysis described above, a detection result with sufficient accuracy may not be obtained due to the influence of weather, an artifact on a road, or the like,” and “when a crack is detected by simply using image data obtained by taking an image from an angle, there is a possibility that the crack cannot be detected appropriately.” See Yoshida, Background, col. 1, lines 35-48. Yoshida teaches retaining acceptable high-resolution data for overlapping areas and discarding lower-resolution data, which is functionally equivalent to assigning full weight to high-quality data and zero weight to low-quality data. See Yoshida, col. 7, lines 4-7.
However neither Shimomura nor Yoshida appears to explicitly disclose calculate a degree of deterioration at a site where the each road surface image is captured based on a plurality of the weighted degrees of deterioration.
Suzuki, working in the same field of endeavor, recognizes this problem and teaches calculate a degree of deterioration at a site where the each road surface image is captured based on a plurality of the weighted degrees of deterioration (Suzuki teaches determining reliability of deterioration detection based on road surface conditions and excluding low-reliability data. Specifically, Suzuki teaches “in a case where a second condition is detected as the road surface condition, exclude, from a road deterioration detection target, at least one of the image and the acceleration acquired together with the sound in which the second condition is detected,” where the second condition represents conditions with low reliability such as very wet, snow, freeze, manhole, or joint. See Suzuki, claim 5; Fig. 9; ¶¶ [0112]-[0114]. Suzuki further discloses a reliability table showing different reliability levels (high, medium, low) based on road surface conditions. See Suzuki, Fig. 6; ¶¶ [0053]-[0056]. Suzuki’s teaching of excluding low-reliability data and applying different reliability levels based on conditions teaches calculating an overall deterioration assessment by weighting data based on reliability/quality).
At the time of the invention, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have included in the invention of Shimomura in view of Yoshida’s overlapping-image processing and effective-data extraction as taught by Yoshida, and further in view of Suzuki’s reliability-based data selection as taught by Suzuki, since doing so would have predictably and advantageously allowed Shimomura’s road surface inspection system to reduce redundant or erroneous detections when the same road surface area is imaged multiple times from a moving vehicle, to display one deterioration indication for the same road surface deterioration detected in overlapping same-site images while displaying separate indications for different deterioration, and to calculate more accurate deterioration assessments by weighting detection results based on quality and reliability. Specifically, Shimomura, Yoshida, and Suzuki all address detecting road surface deterioration from vehicle-acquired road surface images. Shimomura detects abnormal regions on a road surface from road images captured by a camera mounted on a patrol car and displays deterioration positions on a map. See Shimomura, ¶¶ [0026], [0029]-[0031], [0084]-[0085]. Yoshida addresses the problem that “in road deterioration detection by image analysis or acceleration analysis described above, a detection result with sufficient accuracy may not be obtained due to the influence of weather, an artifact on a road, or the like,” and “when a crack is detected by simply using image data obtained by taking an image from an angle, there is a possibility that the crack cannot be detected appropriately.” See Yoshida, Background, col. 1, lines 35-48. Suzuki similarly recognizes that detection quality varies based on road surface conditions and teaches excluding or adjusting detection results based on reliability to improve overall accuracy. See Suzuki, ¶¶ [0053]-[0056], [0112]-[0114]. It would have been obvious to calculate a weighted degree of deterioration by assigning greater weight to higher-quality or higher-reliability deterioration detections from multiple same-site images, because both Yoshida and Suzuki teach that detection quality varies based on imaging/detection conditions and that using higher-quality data improves accuracy. See Yoshida, col. 6-8; Suzuki, ¶¶ [0053]-[0056], [0112]-[0114].
Therefore, the claimed subject matter would have been obvious to a person having ordinary skill in the art at the time the invention was made.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over “Shimomura” in view of “Yoshida” as applied to claims 1 above, and further in view of US Patent Application Publication Yonekawa et al. (US 20180195973 A1) (hereinafter referred to as “Yonekawa”).
With respect to claim 6, which claim 1 is incorporated, neither Shimomura nor Yoshida appears to explicitly disclose display the information indicating the road surface deterioration including a road surface image captured at a later date and time among road surface images in which the recognized road surface deterioration is determined to be identical road surface deterioration.
Yonekawa, working in the same field of endeavor, recognizes this problem and teaches display the information indicating the road surface deterioration including a road surface image captured at a later date and time among road surface images in which the recognized road surface deterioration is determined to be identical road surface deterioration (Yonekawa teaches displaying pavement crack images obtained at different imaging dates/times, including past and latest road images. Yonekawa expressly teaches displaying crack-superimposed images from multiple dates—e.g., five years ago, three years ago and recently—and displaying the past images and latest image for comparison. It additionally generates comparison images from the past and latest images. See Yonekawa, Fig. 1; Fig. 5; Fig. 10; ¶¶ [0030], [0051], [0059], [0061]).
At the time of the invention, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have included in the invention of Shimomura in view of Yoshida’s display the information indicating the road surface deterioration including a road surface image captured at a later date and time among road surface images in which the recognized road surface deterioration is determined to be identical road surface deterioration as taught by Yonekawa, since doing so would have predictably and advantageously allows to display the later/latest image among images associated with the same road-surface deterioration, as taught by Yonekawa, to provide the user with the most current representation of the deterioration and facilitate evaluation of the current road condition and changes over time. Yonekawa expressly identifies this benefit: comparison of the past and latest road images allows the user to recognize changes in road-surface properties over time.
Therefore, the claimed subject matter would have been obvious to a person having ordinary skill in the art at the time the invention was made.
Claims 7 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over “Shimomura” in view of “Yoshida” as applied to claims 1 above, and further in view of US Patent Application Publication Ihara et al. (US 20080013790 A1) (hereinafter referred to as “Ihara”).
With respect to claim 7, which claim 1 is incorporated, neither Shimomura nor Yoshida appears to explicitly disclose estimate, based on a traveling direction of a mobile object on which an imaging device is mounted or driving information including a steering wheel operation, an imaging direction of the imaging device that captures the road surface image; and convert the road surface image based on the estimated imaging direction.
Ihara, working in the same field of endeavor, recognizes this problem and teaches estimate, based on a traveling direction of a mobile object on which an imaging device is mounted or driving information including a steering wheel operation, an imaging direction of the imaging device that captures the road surface image (Ihara teaches a camera mounted on a vehicle and oriented to image the road in the vehicle's moving direction. The camera optical axis has a defined relationship with the vehicle direction. The reference expressly equates vehicle moving direction/yaw angle with “orientation of the camera” and teaches determining vehicle direction from road images. See Ihara, Fig. 4; Fig. 5; ¶¶ [0027]–[0030]); and convert the road surface image based on the estimated imaging direction (Ihara teaches coordinate conversion of vehicle-camera road images into overhead road images. The homographic conversion for the second road image depends upon vehicle movement and moving direction/yaw angle, and corresponding transformed road images are generated according to assumed vehicle/camera directions. FIG. 5 expressly concerns the relationship between camera orientation and vehicle moving direction, while its conversion unit converts road images based upon those geometrical conditions, See Ihara, Fig. 4; Fig. 5; ¶¶ [0020], [0026], [0029], [0029]).
At the time of the invention, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have included in the invention of Shimomura in view of Yoshida’s to use the known vehicle-direction/camera-orientation relationship and direction-dependent image conversion of Ihara in the vehicle-mounted road-imaging system of the claim 1 combination to compensate for changes in the vehicle/camera imaging direction and place road images into a common orientation for more reliable comparison of corresponding road-surface areas. Since doing so would have predictably and advantageously allows to calculates the degree of similarity between the overhead images, and compares the degrees of similarity to each other to obtain the probable change in the moving state, so that a conversion accuracy of the coordinate converting unit is improved, thus realizing more accurate detection of the lateral displacement amount of the vehicle and more accurate detection of a curve amount and a meandering amount, and hence stably executing a control using the lateral displacement amount of the vehicle all the time without being affected by the road environment (see Ihara, Abstract ¶¶ [0006]).
Therefore, the claimed subject matter would have been obvious to a person having ordinary skill in the art at the time the invention was made.
With respect to claim 8, which claim 7 is incorporated, neither Shimomura nor Yoshida appears to explicitly disclose rotate the road surface image based on the driving information.
Ihara, working in the same field of endeavor, recognizes this problem and teaches rotate the road surface image based on the driving information (Ihara teaches that rotation of the vehicle-mounted camera occurs integrally with change in vehicle direction and that the camera optical axis accords with the moving direction. Vehicle moving direction is represented by yaw angle θ and treated as camera orientation. The second road image is coordinate-converted using a homography determined in part by the vehicle moving direction/yaw angle. Thus the geometric conversion compensates for the rotational/yaw relationship produced by vehicle movement, See Ihara, at least ¶ [0027]).
At the time of the invention, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have included in the invention of Shimomura in view of Yoshida’s to rotate/geometrically convert the road image according to vehicle-direction information to compensate for the corresponding change in camera orientation and thereby align road-surface images obtained while the vehicle travels in different directions. (see Ihara, Abstract ¶¶ [0006]).
Therefore, the claimed subject matter would have been obvious to a person having ordinary skill in the art at the time the invention was made.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over “Shimomura” in view of “Yoshida” and further in view of “Ihara” as applied to claims 1 and 7 above, and further in view of US Patent Application Publication Suzuki et al. (US 20130170701 A1) (hereinafter referred to as “Suzuki”) and further in view of US Patent Application Publication Spaeth et al. (US 20130125002 A1) (hereinafter referred to as “Spaeth”).
With respect to claim 9, which claim 1 is incorporated, Shimomura in view of Yoshida fails to explicitly disclose determine that the road surface deterioration is identical when similarity of the road surface deterioration between road surface images to be compared exceeds a threshold value determined according to driving information including at least one of a traveling direction, a steering wheel operation, a speed, or an acceleration of a mobile object on which an imaging device that captures the road surface images is mounted.
Suzuki, working in the same field of endeavor, recognizes this problem and teaches determine that the road surface deterioration is identical when similarity of road surface deterioration between road surface images to be compared exceeds a threshold value (Suzuki discloses adjusting thresholds for detecting road surface events based on vehicle speed and road type. Specifically, Suzuki teaches “when a threshold value to be compared with a deceleration is set, by setting a threshold value for an expressway so as to be lower than that of an ordinary road, it is possible to collect a larger number of abrupt deceleration samples from the high-speed region. Moreover, when a threshold value to be compared with the horizontal G is set, by setting a threshold value for a high-speed corner so as to be lower than that of a low-speed corner, it is possible to collect a larger number of hard turn samples from the high-speed region.” See Suzuki, ¶ [0081]. Suzuki’s teaching of adjusting threshold values based on vehicle speed and road type teaches adjusting threshold values according to driving information. {Under the broadest reasonable interpretation, “a threshold value determined according to driving information” encompasses adjusting detection or comparison thresholds based on vehicle driving conditions such as speed, acceleration, traveling direction, or road type})
At the time of the invention, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have included in the invention of Shimomura in view of Yoshida’s overlapping-image processing and effective-data extraction as taught by Yoshida, and further in view of Suzuki’s threshold adjustment based on driving information as taught by Suzuki, since doing so would have predictably and advantageously allowed Shimomura’s road surface inspection system to reduce redundant or erroneous detections when the same road surface area is imaged multiple times from a moving vehicle, to display one deterioration indication for the same road surface deterioration detected in overlapping same-site images while displaying separate indications for different deterioration, and to improve detection accuracy by adjusting thresholds based on vehicle driving conditions. Specifically, Shimomura, Yoshida, and Suzuki all address detecting road surface deterioration from vehicle-acquired road surface images. Shimomura detects abnormal regions on a road surface from road images captured by a camera mounted on a patrol car and displays deterioration positions on a map. See Shimomura, ¶¶ [0026], [0029]-[0031], [0084]-[0085]. Yoshida addresses the problem that “in road deterioration detection by image analysis or acceleration analysis described above, a detection result with sufficient accuracy may not be obtained due to the influence of weather, an artifact on a road, or the like,” and “when a crack is detected by simply using image data obtained by taking an image from an angle, there is a possibility that the crack cannot be detected appropriately.” See Yoshida, Background, col. 1, lines 35-48. Suzuki recognizes that vehicle driving conditions affect detection and teaches adjusting thresholds based on driving conditions to improve detection accuracy. See Suzuki, ¶ [0081].Combined with Yoshida’s teaching of comparing overlapping same-site road surface images and retaining data based on resolution/quality (see Yoshida, claim 3; col. 6-7), it would have been obvious to adjust thresholds used in determining whether detected deteriorations are identical based on driving information, because Suzuki recognizes that vehicle driving conditions affect detection and that adjusting thresholds based on driving conditions improves detection accuracy. See Suzuki, ¶ [0081].
However, nether Shimomura, Yoshida, nor Suzuki appears to explicitly disclose a threshold value determined according to a length of an interval between imaging dates and times of the road surface images.
Spaeth, working in the same field of endeavor, recognizes this problem and teaches a threshold value determined according to a length of an interval between imaging dates and times of the road surface images (Spaeth teaches changing the required visual-similarity threshold according to temporal proximity/separation between images. Specifically, images having time proximity can be grouped when similarity reaches a lower threshold, whereas images lacking time proximity must satisfy a higher similarity threshold. An exemplary implementation uses 70% similarity for temporally proximate images and 80% for images lacking temporal proximity, See Spaeth, at least ¶ [0040]).
At the time of the invention, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have included in the invention of Shimomura in view of Yoshida’s and further in view of Suzuki to apply a threshold value determined according to a length of an interval between imaging dates and times of the road surface images as taught by Spaeth since doing so would have predictably and advantageously allows such that the applicable similarity threshold is selected based upon the temporal separation of the compared images to account for the reduced reliability of visual correspondence as images become more temporally separated and thereby reduce erroneous matching of separate image instances, (See Spaeth, at least ¶ [0002]-[0005])).
Therefore, the claimed subject matter would have been obvious to a person having ordinary skill in the art at the time the invention was made.
Summary
Claims 1–14 are rejected under at least one of 35 U.S.C. §§ 102 and 103 as being unpatentable over the cited prior art. In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 C.F.R. § 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
ADDITIONAL CITATIONS
The following table lists several references that are relevant to the subject matter claimed and disclosed in this Application. The references are not relied on by the Examiner, but are provided to assist the Applicant in responding to this Office action.
Citation
Relevance
Banitt et al. (20160292518)
Describes a system for detecting and classifying defects in a paved surface. Uses include but are not limited to paved roadways, bridge surfaces, car parking and airplane runways. The system is fully automatic and compact in structure, and has high energy efficiency, and does not need correction or compensation for ambient light conditions. The system provides estimate of the rate of change in and projected residual lifetime of the paved surface based on change of the dimensions of the surface damage divided by the time interval between the two surveys. The system performs additional surveys to improve accuracy and time resolution of estimates, and can train an image processing device with machine learning capabilities to identify defects or other objects of interest in the images of the paved surface.
Du et al. (20190339209)
Describes a binocular image analysis-based asphalt road surface damage detection system. The system utilizes the mobile platform to achieve rapid and accurate detection of cracks on an asphalt road surface. The system collects sufficient damage data to provide decision support and realize intelligent management of an entire road network..
Table 1
CONCLUSION
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HENOK A SHIFERAW whose telephone number is (571)272-4637. The examiner can normally be reached Monday-Friday, 8:30AM - 5:00PM, (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676
Prosecution Insights