Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) were filed on 09/23/2024 and 02/20/2025. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 7-13 and 17-19 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Omiya et. al., hereafter Omiya (US Pub. No. 20200361378 A1 ).
As per claim 1, Omiya teaches “A processor-implemented method, comprising:
generating a first view image of a first moving object view and generating a second view image of a second moving object view;” (See paragraphs 20-22 “[0020] As shown in FIG. 1, an image generation system 1 of the present embodiment is a system mounted to a vehicle such as a passenger car, and includes at least a control unit 10. The image generation system 1 may also include a front camera 21F, a rear camera 21B, a right camera 21R, a left camera 21L, various sensors 25, a display device 30, and the like”. See also paragraph 26 (shows processor and memory). Omiya)
determining a first occlusion region in the first moving object view image based on an obstacle in the first moving object view, and determining a second occlusion region in the second moving object view image based on an obstacle in the second moving object view; (See paragraphs 44-54 “[0044] Subsequently, in S220, the control unit 10 performs conversion so that the obstacles in the captured images are raised. Specifically, when subjecting the captured images to coordinate-conversion into a bird's eye view image…In the processing in S220, the enlarged obstacles are shrunk to be reduced in size, and thus the areas occupied by the obstacles in the bird's eye view image become small, so that blank regions are generated.” Examiner interprets “occlusion region” as a blank region. The blank regions are determined for all obstacles present. “[0045] Subsequently, in S230, the control unit 10 complements the blank regions formed by the above-described conversion using the other captured images. Specifically, the control unit 10 shrinks the obstacles so that blank regions are generated due to the absence of the image data. The control unit 10 complements the regions using the captured images other than the images captured by the cameras which are closest in distance to the respective obstacles.” On paragraph 47 “0047] In this case, for the left pedestrian 511, a region 511A is a region to be enlarged in the bird's eye view, and, for the right pedestrian 512, a region 512A is a region to be enlarged in the bird's eye view. However, the respective pedestrians 511 and 512 are imaged also by the other cameras. In the captured images of the other cameras, regions 511B and 512B are each regions to be enlarged in the bird's eye view, and the regions 511A and 512A are well imaged by the other cameras.” See also fig. 3 and fig. 6. See also paragraph 62. Omiya )
“determining a first temporary boundary of the obstacle in the first moving object view with respect to a first overlap region between the first moving object view and the second moving object view, based on the first occlusion region; determining a second temporary boundary of the obstacle in the second moving object view with respect to the first overlap region, based on the second occlusion region; and” (See paragraphs 65-71 “[0065] (2b) In the image generation system 1 described above, the control unit 10 may temporarily set, in S140 and S160, a plurality of boundary line candidates within the overlap region while avoiding the specific obstacles and set, as the boundary line, the boundary line candidate which is closest to a preliminarily set reference position among the plurality of boundary line candidates.” “[0066] In this case, for example, a boundary line setting condition B shown in the middle column in FIG. 4 is employed, and a plurality of boundary lines are temporarily set as shown in FIG. 8. Under the boundary line setting condition B, a plurality of boundary lines are temporarily set while avoiding regions containing the obstacles. At this time, (2) the lower end parts of the obstacles are avoided as the regions of the obstacles. For example, in the example shown in FIG. 8, boundary lines 23A, 23B, 23C, 23D and 23E passing between the respective pedestrians 501, 502, 503 and 504 are set as the boundary line candidates.” Therefore it teaches that it detects temporary boundaries based on the overlap region, obstacles and the respective occlusion region. It is based on the occlusion region as seen on paragraph 71 “[0071] Specifically, in the example shown in FIG. 9, two pedestrians 521 and 522 are positioned within the overlap region, however, in the image captured by the front camera 21F, the right pedestrian 522 is in a region 521A which is the shadow of the left pedestrian 521, and thus is not recognized. In this case, the number of the obstacles recognized is 1 in the image captured by the front camera 21F, on the other hand, it is 2 in the image captured by the right camera 21R. In this case, preferably, as indicated in the boundary line setting condition C, the boundary is determined so that obstacles are contained in the captured image in which there are a larger number of the obstacles detected in the overlap region, and, at this time, (2) the lower end parts of the obstacles are avoided, and (3) the boundary line is made closer to the predetermined position. [0072] Consequently, the boundary line 23G is set so that the pedestrians 521 and 522 are contained in the image captured by the right camera 21R, as shown in FIG. 9.” The region 512A is a blank region as seen on paragraphs 44-48. See also fig. 3, 9 and fig, 7. See also paragraph 43. Omiya)
“generating a top-view image of the moving object based on the first temporary boundary and the second temporary boundary.” (See paragraph 59 “[0059] (1d) In the image generation system 1 described above, the control unit 10 is, in S170, configured to generate a bird's eye view image in which the periphery of the vehicle is as seen from above, as the composite image. ” See also fig. 3, fig. 2, fig.7 and paragraphs 48, Omiya)
Claim 12 is rejected under the same analysis of claim 1. (See also paragraph 26. Omiya)
Claim 18 is rejected under the same analysis of claim 1. (See also paragraph 26. Omiya)
As per claim 2, Omiya teaches “the method of claim 1, wherein the determining of the first occlusion region and the determining of the second occlusion region comprises: detecting obstacle candidates disposed in the first view image by performing a semantic segmentation based on the first view image; and determining, among the detected obstacle candidates, a region corresponding to the obstacle of the first moving object view to be the first occlusion region. (See paragraphs 43-48 “[0043] Firstly, in S210 of the obstacle conversion processing, the control unit 10 extracts regions of the obstacles. In this processing, well-known image processing is performed to identify the contours of the obstacles, and regions enclosed by the contours are extracted as the regions of the obstacles. When the contours of the obstacles are identified, for example, deep learning of semantic segmentation can be used.” “[0044] Subsequently, in S220, the control unit 10 performs conversion so that the obstacles in the captured images are raised… In the processing in S220, the enlarged obstacles are shrunk to be reduced in size, and thus the areas occupied by the obstacles in the bird's eye view image become small, so that blank regions are generated.” “[0048] Therefore, the regions 511A and 512A in the captured images including the pedestrians 511 and 512 are contained in the bird's eye view image, are complemented by replacing with the regions 511A and 512A in the captured images including the pedestrians 511 and 512 are not included in the bird's eye view image.” It teaches “obstacle candidates” since it detects the regions that may be considered as obstacles as seen on paragraph 71 “[0071] Specifically, in the example shown in FIG. 9, two pedestrians 521 and 522 are positioned within the overlap region, however, in the image captured by the front camera 21F, the right pedestrian 522 is in a region 521A which is the shadow of the left pedestrian 521, and thus is not recognized. In this case, the number of the obstacles recognized is 1 in the image captured by the front camera 21F, on the other hand, it is 2 in the image captured by the right camera 21R.” See also paragraph 34 and paragraphs 53-58 which shows that all obstacles are detected and processing is performed on specific obstacles, therefore it also implicitly teaches candidate obstacles. Omiya )
Claim 13 is rejected under the same analysis as claim 2.
Claim 19 is rejected under the same analysis as claim 2.
As per claim 3, Omiya teaches “The method of claim 2, wherein the obstacle candidates comprise a vehicle, a lane, a road, or a combination thereof, and the obstacle of the first view is the vehicle.” (See paragraph 29 “This bird's eye view image contains obstacles on the road. It should be noted that the obstacles mean any objects that can disturb the traveling of the vehicle, such as other vehicles which are vehicles except the own vehicle, pedestrians, bicycles, curbs, and walls.” See also paragraphs 20-23 ” [0020] As shown in FIG. 1, an image generation system 1 of the present embodiment is a system mounted to a vehicle such as a passenger car, and includes at least a control unit 10. The image generation system 1 may also include a front camera 21F, a rear camera 21B, a right camera 21R, a left camera 21L, various sensors 25, a display device 30, and the like. The vehicle in which the image generation system 1 is mounted is referred to also as an own vehicle.” Omiya)
As per claim 7, Omiya teaches “The method of claim 1, wherein the generating of the top-view image comprises: comparing a visibility of the first overlap region of the first view image identified by the first temporary boundary with a visibility of the first overlap region of the second view image identified by the second temporary boundary; and generating the top-view image by selectively using one of the first view image and the second view image with respect to the first overlap region, based on a final boundary determined according to a result of the comparing.” (See paragraphs 37-41 “[0037] Specifically, under the boundary line setting condition A, the boundary line is set so that each of the obstacles is contained in the image captured by the camera 21F, 21B, 21L or 21R which is closest in distance to this obstacle, among the plurality of cameras 21F, 21B, 21L and 21R.” “[0038] For example, in the example shown in FIG. 5, the control unit 10 detects four pedestrians 501, 502, 503 and 504 in the overlap region 22D. At this time, the rightmost pedestrian 504 is closest to the right camera 21R among the cameras 21F, 21B, 21L and 21R, and thus setting is made so that the rightmost pedestrian 504 is contained in the image captured by the right camera 21R.” Since it determines which is “closest” to the camera in the overlap region, it determines a “visibility” of the obstacle. See also paragraphs 57, 61, 65-68 and 71. “[0057] (1c) In the image generation system 1 described above, the control unit 10, in S140 and S160, sets the boundary line so that each of the specific obstacles is contained in the image captured by the imaging unit which is closest in distance to the specific obstacle among the plurality of imaging units.” Omiya)
Claim 17 is rejected under the same analysis as claim 7.
As per claim 8, Omiya teaches “the method of claim 1, further comprising: generating a third view image of a third moving object view; determining a third occlusion region generated in the third view image based on an obstacle in the third moving object view; determining a third temporary boundary with respect to a second overlap region between the first moving object view and the third moving object view, based on the first occlusion region; and determining a fourth temporary boundary with respect to the second overlap region, based on the third occlusion region. (The system presented in Omiya teaches the cases where the processing for other overlapping regions are performed for all obstacles and all regions. It includes 4 cameras as seen in fig. 2, and the claim limitation here is also covered within a BRI with the information used already to reject claim 1. See paragraphs 20-22, 44-54, fig. 3, fig. 6, fig. 9, fig. 7 and paragraphs 65-71 along with paragraph 59 and 48. On paragraph 65 it shows “[0065] (2b) In the image generation system 1 described above, the control unit 10 may temporarily set, in S140 and S160, a plurality of boundary line candidates within the overlap region while avoiding the specific obstacles and set, as the boundary line, the boundary line candidate which is closest to a preliminarily set reference position among the plurality of boundary line candidates.” “[0066] In this case, for example, a boundary line setting condition B shown in the middle column in FIG. 4 is employed, and a plurality of boundary lines are temporarily set as shown in FIG. 8. Under the boundary line setting condition B, a plurality of boundary lines are temporarily set while avoiding regions containing the obstacles. At this time, (2) the lower end parts of the obstacles are avoided as the regions of the obstacles. For example, in the example shown in FIG. 8, boundary lines 23A, 23B, 23C, 23D and 23E passing between the respective pedestrians 501, 502, 503 and 504 are set as the boundary line candidates.” Therefore it teaches that it detects temporary boundaries based on the overlap region, obstacles and the respective occlusion region. It is based on the occlusion region as seen on paragraph 71 “[0071] Specifically, in the example shown in FIG. 9, two pedestrians 521 and 522 are positioned within the overlap region, however, in the image captured by the front camera 21F, the right pedestrian 522 is in a region 521A which is the shadow of the left pedestrian 521, and thus is not recognized. In this case, the number of the obstacles recognized is 1 in the image captured by the front camera 21F, on the other hand, it is 2 in the image captured by the right camera 21R. In this case, preferably, as indicated in the boundary line setting condition C, the boundary is determined so that obstacles are contained in the captured image in which there are a larger number of the obstacles detected in the overlap region, and, at this time, (2) the lower end parts of the obstacles are avoided, and (3) the boundary line is made closer to the predetermined position. [0072] Consequently, the boundary line 23G is set so that the pedestrians 521 and 522 are contained in the image captured by the right camera 21R, as shown in FIG. 9.” The region 512A is a blank region as seen on paragraphs 44-48. See paragraph 59 “[0059] (1d) In the image generation system 1 described above, the control unit 10 is, in S170, configured to generate a bird's eye view image in which the periphery of the vehicle is as seen from above, as the composite image. ” Omiya)
As per claim 9, Omiya teaches “the method of claim 8, wherein the generating of the top-view image comprises: comparing a visibility of the second overlap region of the first view image identified by the third temporary boundary with a visibility of the second overlap region of the third view image identified by the fourth temporary boundary; and generating the top-view image by selectively using one of the first view image and the third view image with respect to the second overlap region, based on a final boundary determined according to a result of the comparing.” (The system presented in Omiya teaches the cases where the processing for other overlapping regions are performed for all obstacles and all regions. It includes 4 cameras as seen in fig. 2, and the claim limitation here is also covered within a BRI with the information used already to reject claim 1 and claim 7. See paragraphs 37-41 “[0037] Specifically, under the boundary line setting condition A, the boundary line is set so that each of the obstacles is contained in the image captured by the camera 21F, 21B, 21L or 21R which is closest in distance to this obstacle, among the plurality of cameras 21F, 21B, 21L and 21R.” “[0038] For example, in the example shown in FIG. 5, the control unit 10 detects four pedestrians 501, 502, 503 and 504 in the overlap region 22D. At this time, the rightmost pedestrian 504 is closest to the right camera 21R among the cameras 21F, 21B, 21L and 21R, and thus setting is made so that the rightmost pedestrian 504 is contained in the image captured by the right camera 21R.” Since it determines which is “closest” to the camera in the overlap region, it determines a “visibility” of the obstacle. See also paragraphs 57, 61, 65-68 and 71. “[0057] (1c) In the image generation system 1 described above, the control unit 10, in S140 and S160, sets the boundary line so that each of the specific obstacles is contained in the image captured by the imaging unit which is closest in distance to the specific obstacle among the plurality of imaging units.” Omiya)
As per claim 10, Omiya teaches “the method of claim 1, wherein: the first moving object view is one of a front view, a rear view, a left view, and a right view of the moving object, and the second moving object view is another one of the front view, the rear view, the left view, and the right view of the moving object, which is adjacent to the first moving object view. (See fig. 2 and paragraphs 20-22 “[0020]… The image generation system 1 may also include a front camera 21F, a rear camera 21B, a right camera 21R, a left camera 21L, various sensors 25, a display device 30, and the like.” “[0021] Each of the cameras 21F, 21B, 21L and 21R is arranged in the periphery of the vehicle, as shown in FIG. 2. ” Omiya)
As per claim 11, Omiya teaches “A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.” (See paragraph 26 “[0026] The control unit 10 includes a microcomputer having a CPU 11 and a semiconductor memory (hereinafter referred to as memory 12) such as a RAM or ROM. The respective functions of the control unit 10 are realized by the CPU 11 executing programs stored in a non-transitory tangible storage medium. In this example, the memory 12 corresponds to the non-transitory tangible storage medium having the programs stored therein.” Omiya)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 4-6, 14-16 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Omiya in view of Kim et. al. (US Pub. No. 20210001776 A1) .
As per claim 4, Omiya teaches “the method of claim 1, further comprising: generating a first partial top-view image and a second partial top-view image… the first moving object view image and the second moving object view image, respectively.” , however Omiya does not teach “a first partial… image”, “a second partial… image” and “by warping”.
Kim teaches “a first partial… image”, “a second partial… image” and “by warping”. (See paragraph 54 “[0054] To generate an elevated top view or a bird's eye view image (hereinafter simply “bird-view image”) of a vehicle, it needs to merge a plurality of images captured by a plurality of cameras of the vehicle. To merge the images, it needs to match coordinate systems of the images. That is, it needs to generate partial bird-view images by transforming the coordinate systems of the images into a common coordinate system that is set in advance for the vehicle, and generate a final bird-view image by merging the generated partial bird-view images. Here, a partial bird-view image may be generated by warping an image, for example. The generated final bird-view image may be used for an around view monitor (AVM). The term “bird-view image” used herein may refer to an image captured from a bird's eye viewpoint.” It therefore shows that it produces several partial top view images to then merge them. Kim)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Omiya with the teachings of Kim to create a partial top view image by warping the image. The modification would have been motivated by the desire to remove blind spots of the vehicle periphery and also improve accuracy, therefore it is an improvement, as suggested by Kim (See paragraph 3 “The top view system may generate a top-view image or a bird's-eye view image (hereinafter simply “bird-view image”) using a plurality of images captured by a plurality of cameras. The bird-view image may provide a driver with an elevated top view of a vehicle that is viewed from above, and thus contribute to completely removing a blind spot on a front side, a rear side, a left side, and/or a right side of the vehicle.” See also paragraph 56 and paragraph 142 “[0142]… When the images captured using a fish-eye lens are transformed into bird-view images, accuracy in matching feature points in ROIs for same features may increase.’ Kim)
Claim 14 is rejected under the same analysis as claim 4.
As per claim 5, Omiya in view of Kim already teaches “the method of claim 4,”, however wherein the determining of the first temporary boundary comprises:
setting boundary candidates with respect to the first overlap region in the first partial top-view image; determining partial regions of the first overlap region by dividing the first overlap region into the boundary candidates; comparing a first corresponding occlusion region of the first occlusion region of the first partial top-view image with the partial regions; and determining one of the boundary candidates to be the first temporary boundary based on a result of the comparing. (See paragraphs 65-71 “[0065] (2b) In the image generation system 1 described above, the control unit 10 may temporarily set, in S140 and S160, a plurality of boundary line candidates within the overlap region while avoiding the specific obstacles and set, as the boundary line, the boundary line candidate which is closest to a preliminarily set reference position among the plurality of boundary line candidates.” “[0066] In this case, for example, a boundary line setting condition B shown in the middle column in FIG. 4 is employed, and a plurality of boundary lines are temporarily set as shown in FIG. 8. Under the boundary line setting condition B, a plurality of boundary lines are temporarily set while avoiding regions containing the obstacles. At this time, (2) the lower end parts of the obstacles are avoided as the regions of the obstacles. For example, in the example shown in FIG. 8, boundary lines 23A, 23B, 23C, 23D and 23E passing between the respective pedestrians 501, 502, 503 and 504 are set as the boundary line candidates.” Therefore it teaches that it detects temporary boundaries based on the overlap region, obstacles and the respective occlusion region . It is based on the occlusion region as seen on paragraph 71 “[0071] Specifically, in the example shown in FIG. 9, two pedestrians 521 and 522 are positioned within the overlap region, however, in the image captured by the front camera 21F, the right pedestrian 522 is in a region 521A which is the shadow of the left pedestrian 521, and thus is not recognized. In this case, the number of the obstacles recognized is 1 in the image captured by the front camera 21F, on the other hand, it is 2 in the image captured by the right camera 21R. In this case, preferably, as indicated in the boundary line setting condition C, the boundary is determined so that obstacles are contained in the captured image in which there are a larger number of the obstacles detected in the overlap region, and, at this time, (2) the lower end parts of the obstacles are avoided, and (3) the boundary line is made closer to the predetermined position. [0072] Consequently, the boundary line 23G is set so that the pedestrians 521 and 522 are contained in the image captured by the right camera 21R, as shown in FIG. 9.” The region 512A is a blank region as seen on paragraphs 44-48. See also fig. 3, 9 and fig, 7. On fig. 7 it shows partial regions. See also paragraph 43. Omiya),
however Kim also teaches the claim limitations along with specific “partial regions”. (See paragraph 26 “… The camera calibration method includes obtaining a plurality of images of surroundings of the vehicle captured by a plurality of cameras of the vehicle, determining a partial region of each of the images based on an ROI that is set in advance with respect to a bird's eye viewpoint, generating a plurality of partial bird-view images by transforming the determined partial regions by the bird's eye viewpoint, in which a first partial bird-view image among the partial bird-view images corresponds to a first ROI and a second partial bird-view image among the partial bird-view images corresponds to a second ROI and the first ROI and the second ROI include a common region, detecting a first feature point of the first partial bird-view image and a second feature point of the second partial bird-view image, matching the first feature point and the second feature point… See also paragraphs 139 and 140. “[0140]… The partial regions of the images determined in operation 1420 may be images captured using a fish-eye lens, and thus shapes in the images may be distorted. By transforming the partial regions by the bird's eye viewpoint, it is possible to restore the shapes in the images. A partial bird-view image will be described hereinafter in detail with reference to FIG. 17.” It would have been obvious to combine the references for the same reasons presented in the rational used for claim 4. Kim)
Claim 15 is rejected under the same analysis as claim 5.
As per claim 6, Omiya in view of Kim teaches “the method of claim 5, wherein the comparing of the first corresponding occlusion region with the partial regions comprises comparing the first corresponding occlusion region with the partial regions based on at least one of an area occupied by the first corresponding occlusion region in each of the partial regions and a distance between the first corresponding occlusion region shown in each of the partial regions and a representative position with respect to the first occlusion region of the moving object. (See paragraphs 65-71 “[0065] (2b) In the image generation system 1 described above, the control unit 10 may temporarily set, in S140 and S160, a plurality of boundary line candidates within the overlap region while avoiding the specific obstacles and set, as the boundary line, the boundary line candidate which is closest to a preliminarily set reference position among the plurality of boundary line candidates.” “[0066] In this case, for example, a boundary line setting condition B shown in the middle column in FIG. 4 is employed, and a plurality of boundary lines are temporarily set as shown in FIG. 8. Under the boundary line setting condition B, a plurality of boundary lines are temporarily set while avoiding regions containing the obstacles. At this time, (2) the lower end parts of the obstacles are avoided as the regions of the obstacles. For example, in the example shown in FIG. 8, boundary lines 23A, 23B, 23C, 23D and 23E passing between the respective pedestrians 501, 502, 503 and 504 are set as the boundary line candidates.” Therefore it teaches that it detects temporary boundaries based on the overlap region, obstacles and the respective occlusion region . It is based on the occlusion region as seen on paragraph 71 “[0071] Specifically, in the example shown in FIG. 9, two pedestrians 521 and 522 are positioned within the overlap region, however, in the image captured by the front camera 21F, the right pedestrian 522 is in a region 521A which is the shadow of the left pedestrian 521, and thus is not recognized. In this case, the number of the obstacles recognized is 1 in the image captured by the front camera 21F, on the other hand, it is 2 in the image captured by the right camera 21R. In this case, preferably, as indicated in the boundary line setting condition C, the boundary is determined so that obstacles are contained in the captured image in which there are a larger number of the obstacles detected in the overlap region, and, at this time, (2) the lower end parts of the obstacles are avoided, and (3) the boundary line is made closer to the predetermined position. [0072] Consequently, the boundary line 23G is set so that the pedestrians 521 and 522 are contained in the image captured by the right camera 21R, as shown in FIG. 9.” The region 512A is a blank region as seen on paragraphs 44-48. See also fig. 3, 9 and fig, 7. On fig. 7 it shows partial regions. See also paragraph 43. It is based on an area occupied since there is an obstacle in the area and the distance is used to select the camera for the specific regions and the “representative position” is interpreted as seen on paragraph 68 and 62 “[0068] Such a configuration makes it possible to set the boundary line at a position close to the reference position while avoiding the obstacles.” It is a comparison as seen in paragraph 24 “[0024] The various sensors 25 are well-known sensors that detect the state of the own vehicle. For example, the various sensors 25 detect the vehicle speed, acceleration, yaw rate, current location, obstacles around the own vehicle, position of a gear shift lever, and the like. The various sensors 25 send the detection results to the control unit 10.” See also paragraph 41. Omiya), however Kim also teaches the claim limitations along with specific “partial regions” and “comparing the first corresponding… region with the partial regions based on at least one of an area occupied by the first corresponding… region in each of the partial regions and a distance between the first corresponding… region shown in each of the partial regions and a representative position with respect to the first…region of the moving object”. (See paragraph 26 “… The camera calibration method includes obtaining a plurality of images of surroundings of the vehicle captured by a plurality of cameras of the vehicle, determining a partial region of each of the images based on an ROI that is set in advance with respect to a bird's eye viewpoint, generating a plurality of partial bird-view images by transforming the determined partial regions by the bird's eye viewpoint, in which a first partial bird-view image among the partial bird-view images corresponds to a first ROI and a second partial bird-view image among the partial bird-view images corresponds to a second ROI and the first ROI and the second ROI include a common region, detecting a first feature point of the first partial bird-view image and a second feature point of the second partial bird-view image, matching the first feature point and the second feature point… See also paragraphs 139 and 140. “[0140]… The partial regions of the images determined in operation 1420 may be images captured using a fish-eye lens, and thus shapes in the images may be distorted. By transforming the partial regions by the bird's eye viewpoint, it is possible to restore the shapes in the images. A partial bird-view image will be described hereinafter in detail with reference to FIG. 17.” It would have been obvious to combine the references for the same reasons presented in the rational used for claim 4. Kim)
Claim 16 is rejected under the same analysis as claim 6.
As per claim 20, Omiya teaches “the moving object of claim 18, wherein the one or more processors is configured to: generate a first partial top-view image and a second partial top-view image by… the first view image and the second view image, respectively; ()
set boundary candidates with respect to the first overlap region in the first partial top-view image; determine partial regions of the first overlap region by dividing the first overlap region into the boundary candidates; compare a first corresponding occlusion region of the first occlusion region of the first partial top-view image with the partial regions; and determine one of the boundary candidates to be the first temporary boundary based on a result of the comparison.” (See paragraphs 65-71 “[0065] (2b) In the image generation system 1 described above, the control unit 10 may temporarily set, in S140 and S160, a plurality of boundary line candidates within the overlap region while avoiding the specific obstacles and set, as the boundary line, the boundary line candidate which is closest to a preliminarily set reference position among the plurality of boundary line candidates.” “[0066] In this case, for example, a boundary line setting condition B shown in the middle column in FIG. 4 is employed, and a plurality of boundary lines are temporarily set as shown in FIG. 8. Under the boundary line setting condition B, a plurality of boundary lines are temporarily set while avoiding regions containing the obstacles. At this time, (2) the lower end parts of the obstacles are avoided as the regions of the obstacles. For example, in the example shown in FIG. 8, boundary lines 23A, 23B, 23C, 23D and 23E passing between the respective pedestrians 501, 502, 503 and 504 are set as the boundary line candidates.” Therefore it teaches that it detects temporary boundaries based on the overlap region, obstacles and the respective occlusion region . It is based on the occlusion region as seen on paragraph 71 “[0071] Specifically, in the example shown in FIG. 9, two pedestrians 521 and 522 are positioned within the overlap region, however, in the image captured by the front camera 21F, the right pedestrian 522 is in a region 521A which is the shadow of the left pedestrian 521, and thus is not recognized. In this case, the number of the obstacles recognized is 1 in the image captured by the front camera 21F, on the other hand, it is 2 in the image captured by the right camera 21R. In this case, preferably, as indicated in the boundary line setting condition C, the boundary is determined so that obstacles are contained in the captured image in which there are a larger number of the obstacles detected in the overlap region, and, at this time, (2) the lower end parts of the obstacles are avoided, and (3) the boundary line is made closer to the predetermined position. [0072] Consequently, the boundary line 23G is set so that the pedestrians 521 and 522 are contained in the image captured by the right camera 21R, as shown in FIG. 9.” The region 512A is a blank region as seen on paragraphs 44-48. See also fig. 3, 9 and fig, 7. On fig. 7 it shows partial regions. See also paragraph 43. Omiya), however Omiya does not teach “a first partial… image”, “a second partial… image” and “by warping”.
Kim teaches “a first partial… image”, “a second partial… image” and “by warping”. (See paragraph 54 “[0054] To generate an elevated top view or a bird's eye view image (hereinafter simply “bird-view image”) of a vehicle, it needs to merge a plurality of images captured by a plurality of cameras of the vehicle. To merge the images, it needs to match coordinate systems of the images. That is, it needs to generate partial bird-view images by transforming the coordinate systems of the images into a common coordinate system that is set in advance for the vehicle, and generate a final bird-view image by merging the generated partial bird-view images. Here, a partial bird-view image may be generated by warping an image, for example. The generated final bird-view image may be used for an around view monitor (AVM). The term “bird-view image” used herein may refer to an image captured from a bird's eye viewpoint.” It therefore shows that it produces several partial top view images to then merge them. Kim), In addition Kim also teaches the claim limitations along with specific “partial regions”. (See paragraph 26 “… The camera calibration method includes obtaining a plurality of images of surroundings of the vehicle captured by a plurality of cameras of the vehicle, determining a partial region of each of the images based on an ROI that is set in advance with respect to a bird's eye viewpoint, generating a plurality of partial bird-view images by transforming the determined partial regions by the bird's eye viewpoint, in which a first partial bird-view image among the partial bird-view images corresponds to a first ROI and a second partial bird-view image among the partial bird-view images corresponds to a second ROI and the first ROI and the second ROI include a common region, detecting a first feature point of the first partial bird-view image and a second feature point of the second partial bird-view image, matching the first feature point and the second feature point… See also paragraphs 139 and 140. “[0140]… The partial regions of the images determined in operation 1420 may be images captured using a fish-eye lens, and thus shapes in the images may be distorted. By transforming the partial regions by the bird's eye viewpoint, it is possible to restore the shapes in the images. A partial bird-view image will be described hereinafter in detail with reference to FIG. 17.”. Kim)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Omiya with the teachings of Kim to create a partial top view image by warping the image and then determine partial regions used to detect the boundary candidate. The modification would have been motivated by the desire to remove blind spots of the vehicle periphery and also improve accuracy, therefore it is an improvement, as suggested by Kim (See paragraph 3 “The top view system may generate a top-view image or a bird's-eye view image (hereinafter simply “bird-view image”) using a plurality of images captured by a plurality of cameras. The bird-view image may provide a driver with an elevated top view of a vehicle that is viewed from above, and thus contribute to completely removing a blind spot on a front side, a rear side, a left side, and/or a right side of the vehicle.” See also paragraph 56 and paragraph 142 “[0142]… When the images captured using a fish-eye lens are transformed into bird-view images, accuracy in matching feature points in ROIs for same features may increase.’ Kim)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DYLAN J MENDEZ MUNIZ whose telephone number is (703)756-5672. The examiner can normally be reached M-F, 8AM - 5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Moyer can be reached at (571) 272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DYLAN JOHN MENDEZ MUNIZ/Examiner, Art Unit 2675
/ANDREW M MOYER/Supervisory Patent Examiner, Art Unit 2675
Prosecution Insights