DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Amendments
Applicant’s Amendment filed on 5/27/2026 has been entered and made of record.
Currently Pending claims: 1-13
Independent claims: 1 and 10
Amended claims: 1-10
New claims: 12 and 13
Response to Arguments
This office action is responsive to Applicant’s Arguments/Remarks Made in an Amendment received on 5/27/2026.
Applicant’s arguments, see Page 7, filed 5/27/2026, with respect to objections to the specification have been fully considered and are persuasive. The objections to the specification have been withdrawn.
Applicant’s arguments, see Page 7, filed 5/27/2026, with respect to claim interpretation under 35 U.S.C. § 112(f) have been fully considered and are persuasive. The interpretation of claims under 35 U.S.C. § 112(f) has been withdrawn.
Applicant's arguments, see Page 7-9, filed 5/27/2026, with respect to rejections of claims 1-11 under 35 U.S.C. § 103 have been fully considered but they are not persuasive.
Applicant, on page 8, argues:
PNG
media_image1.png
422
714
media_image1.png
Greyscale
The Examiner respectfully disagrees. As cited in the previous Office Action, Yamamoto discloses the calculation of adjustment coefficients in a region of interest which is equated to the boundary portion, “In view of this, the γ-curve coefficient calculator 34c calculates a first coefficient for … the first region of interest (e.g., the region of interest 40FL)”, at ¶0063. Yamamoto does not limit the type of images that can be used for determining either the region of interest or the coefficient. The images used in Yamamoto comprise sections of a “bird’s-eye-view image”, as illustrated in Fig. 4, and thus determining a region of interest in a section would also be determining a region of interest in the corresponding location of the bird’s-eye-view image. Kohara was relied upon, in the previous Office Action, to provide the overlapping image generated by the accumulated images at Fig. 3, 4, and “The display image creating unit 29 creates a supplementary image D corresponding to the non-imaging area B by sequentially joining partial areas corresponding to the non-imaging area B from the new history image of the accumulated history image group”, at Col. 8 lines 5-10. Further, Kohara also discloses the full bird’s-eye-view image in Fig.3 label E. Therefore, in combination, Yamamoto and Kohara disclose determining a region of interest between a bird’s-eye-view image and an overlapping image, and calculating an adjustment coefficient in the region of interest, as claimed in amended claim 1.
In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007).
In this case, both Yamamoto and Kohara are solving the same problem of improving bird’s-eye-view images of vehicles.
PNG
media_image2.png
346
675
media_image2.png
Greyscale
Applicant, on page 8, argues:
The improvement cited in Kohara is directed towards the synthesized bird’s-eye-view images “Therefore, it is possible to suppress discontinuity of images of a road surface rendered in a non-imaging area in a synthesized display bird's-eye image”, as suggested at Col. 10, lines 18-21, and “Therefore, in the display bird's-eye image synthesized from the captured bird's-eye image and the history image, it is possible to suppress discontinuity of images of the road surface rendered in the non-imaging area” at Col. 3, lines 12-16. Further, Yamamoto states “the imagers 14 successively shoot the surroundings outside the vehicle 10 including road surfaces on which the vehicle 10 is movable” in ¶0039. Successively shooting images indicate a temporal sequence of images which would fall under history images. Therefore, the clear motivation to combine Yamamoto and Kohara would be the improvement of bird’s-eye-view images.
PNG
media_image3.png
302
650
media_image3.png
Greyscale
Applicant, on page 9, further argues:
However, Yamamoto only utilizes overlapping images to maintain spatial continuity in the combined image, “Image data (images) generated by the imagers 14 includes mutually overlapping regions to prevent missing regions in combined images”, ¶0041. The selection of the region of interest is not required to be in an overlapping region, “The region-of-interest setter 30 may set the position of each region of interest 40 to a predefined position”, ¶0053. Therefore, Yamamoto is technically compatible with Kohara as Kohara generates continuous combined images. Further, even if the method Yamamoto necessitated overlapping images for luminance adjustment, the history images of Kohara are shown to be overlapping with peripheral image of the vehicle, in Fig. 4 labels C0-C4 and A0-A7. Kohara simply selects parts of C0 and C4 that overlap with the non-imaging area for the purpose of maintaining continuity in the forward direction of the vehicle, “However, in the periods at times T1 to T3, no new history images are acquired. As a result, the region of the image synthesized from the last acquired history image C0 for the non-imaging area gradually expands”, Col. 9, lines 49-52. Therefore, a person skilled in the art would be capable creating the combined history images that include an overlapping area with the peripheral images, see Fig. 4, C0/C4 overlapping with A7 in timepoint T7.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 and 8-13 are rejected under 35 U.S.C. 103 as being unpatentable over Yamamoto et al. (US 2021/0160432) (hereafter, “Yamamoto”) in view of Kohara et al. (US 10,873,725) (hereafter, “Kohara”).
Regarding claim 1, Yamamoto discloses an image processing device comprising: a processor; and a memory storing program instructions that cause the processor to: [accumulate images] obtained from a plurality of image capturing devices (¶0039, the vehicle 10 includes, for example, four imagers 14a to 14d as a plurality of imagers 14) [as accumulated images], the plurality of image capturing devices being provided in a moving body (¶0039, The imagers 14 can output moving image data. Examiner considers moving image data to imply a moving body); [generate an overlapping image from the accumulated images, the overlapping image showing an area that overlaps the moving body and is not included in a field of view of the plurality of image capturing devices]; generate a first bird's-eye-view image that shows surroundings of the moving body by merging the images obtained from the plurality of image capturing devices together (¶0041, The image processing device of the present embodiment performs computation or image processing to image data generated by the imagers 14 to generate an image having a wider viewing angle or a virtual image of the vehicle 10 viewed from above, front, or laterally (e.g., a bird's-eye image)), [and generate a second bird's-eye-view image by merging the first bird's-eye-view image and the overlapping image together]; calculate adjustment coefficients in a boundary portion formed between the first bird's-eye-view image and the overlapping image (¶0063, In view of this, the γ-curve coefficient calculator 34c calculates a first coefficient for … the first region of interest (e.g., the region of interest 40FL). Similarly, the γ-curve coefficient calculator 34c calculates a second coefficient … for the second target luminance of the second region of interest. Examiner considers the regions of interest as “boundary regions”), based on property information of the first bird's-eye-view image and property information of the [accumulated] images used to generate the overlapping image (¶0052, As illustrated in FIG. 5, the region-of-interest setter 30 sets regions of interest 40 (40FL, 40RL, 40RR, and 40FR) in each of the overlapping regions 38 of the imaging regions 36 acquired by the acquirer 28. … The luminance of the regions of interest 40 refers to, for example, the average of luminance of pixels included in the corresponding regions of interest 40. Examiner considers the luminance to be a “property” of the image. Examiner relies upon Kohara to provide the accumulated/overlapping images and considers the overlapping image to be interchangeable with one of the images used in Yamamoto as the primary difference is in the position of the image (under the vehicle vs. around) and should have no impact on the processing steps described by Yamamoto); and adjust the property of the overlapping image to be merged with the first bird’s-eye-view image (¶0085, As a result, the luminance setter 34d can calculate corrected luminance (output luminance Y.sub.1) of the region between the pair of regions of interest 40. Through such correction to the rest of the images, it is possible to generate a recognizable peripheral image), based on the calculated adjustment coefficients (¶0085, The luminance setter 34d substitutes the calculated γ-curve coefficient and the luminance of the position of the object to be corrected (luminance value of original image, i.e., input luminance Y.sub.0) into the relation: Y1=Y.sub.0.sup.1/γ between the input luminance Y.sub.0 (luminance of original image) and the output luminance Y.sub.1).
However, Yamamoto fails to explicitly disclose accumulate images; and generate an overlapping image from the accumulated images, the overlapping image showing an area that overlaps the moving body and is not included in a field of view of the plurality of image capturing devices; and generate a second bird's-eye-view image by merging the first bird's-eye-view image and the overlapping image together.
Kohara teaches an image accumulate images (Col. 4, lines 66-67 – Col. 5, lines 1-2, The history image acquiring unit 27 stores the extracted history images in a time series in the history area 43 of the storage unit 15. Examiner considers the history images as the “accumulated images”); and generate an overlapping image from accumulated images (Fig. 3, 4; Col. 8, lines 5-10, The display image creating unit 29 creates a supplementary image D corresponding to the non-imaging area B by sequentially joining partial areas corresponding to the non-imaging area B from the new history image of the accumulated history image group. Fig. 3D illustrates the accumulated images F1-F10 being assembled into overlapping image D while Fig. 4 illustrates the assembly process over time), the overlapping image showing an area that overlaps the moving body and is not included in a field of view of the plurality of image capturing devices (Fig. 3 #B, #D; Col. 8, lines 5-8, The display image creating unit 29 creates a supplementary image D corresponding to the non-imaging area B. Examiner considers the non-imaging area as “not included in a field of view”); and generate a second bird's-eye-view image by merging the first bird's-eye-view image and the overlapping image together (Col. 6, lines 27-30, The display image creating unit 29 then creates the display bird's-eye image by combining the created supplementary image with the non-imaging area of the latest captured bird's-eye image).
Both Yamamoto and Kohara are analogous to the claimed invention because they are in the field of generating bird's-eye-view images of vehicles. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the accumulated images from Kohara into the luminance adjustment of Yamamoto. The suggestion/motivation for doing so would have been to suppress discontinuity in the final image, as suggested by Kohara at Col. 10, lines 18-21, Therefore, in the display bird's-eye image synthesized from the captured bird's-eye image and the history image, it is possible to suppress discontinuity of images of the road surface rendered in the non-imaging area.
This method of improving Yamamoto was within the ordinary ability of one of ordinary skill in the art based on the teachings of Kohara.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yamamoto with the teachings of Kohara to obtain the invention as specified in claim 1.
Regarding claim 8, in which claim 1 is incorporated, Yamamoto discloses wherein the program instructions further cause the processor to calculate an average value of property information of a plurality of pixels in the boundary portion (¶0058, For example, the average luminance of 125 of the region of interest 40FL and the region of interest 40FR is set to the target luminance), and calculate adjustment coefficients based on the average value calculated (¶0063, In view of this, the γ-curve coefficient calculator 34c calculates a first coefficient for a first γ curve, which is calculated as a curve formula for the first target luminance of the luminance of the first region of interest (e.g., the region of interest 40FL)).
Regarding claim 9, in which claim 1 is incorporated, Yamamoto discloses wherein the first bird's-eye-view image is generated by merging together (¶0041, The image processing device of the present embodiment performs computation or image processing to image data generated by the imagers 14 to generate an image having a wider viewing angle or a virtual image of the vehicle 10 viewed from above, front, or laterally (e.g., a bird's-eye image)), [on a real-time basis], the images obtained from the plurality of image capturing devices (¶0039, the vehicle 10 includes, for example, four imagers 14a to 14d as a plurality of imagers 14), [wherein the overlapping image is generated from past accumulated images stocked, and wherein the second bird’s-eye-view image is generated by merging the first bird's-eye-view image and the overlapping image together].
However, Yamamoto fails to explicitly disclose on a real-time basis; and wherein the overlapping image is generated from past accumulated images stocked, and wherein the second bird’s-eye-view image is generated by merging the first bird's-eye-view image and the overlapping image together.
Kohara teaches on a real-time basis (Col. 7, lines 50-51, In S116, the image processing unit 13 creates a display bird's eye image by combining the real-time image); and wherein the overlapping image is generated from past accumulated images stocked (Col. 8, lines 5-10, The display image creating unit 29 creates a supplementary image D corresponding to the non-imaging area B by sequentially joining partial areas corresponding to the non-imaging area B from the new history image of the accumulated history image group C), and wherein the second bird’s-eye-view image is generated by merging the first bird's-eye-view image and the overlapping image together (Col. 6, lines 27-30, The display image creating unit 29 then creates the display bird's-eye image by combining the created supplementary image with the non-imaging area of the latest captured bird's-eye image).
Both Yamamoto and Kohara are analogous to the claimed invention because they are in the field of generating bird's-eye-view images of vehicles. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the accumulated images from Kohara into the luminance adjustment of Yamamoto. The suggestion/motivation for doing so would have been to suppress discontinuity in the final image, as suggested by Kohara at Col. 10, lines 18-21, Therefore, in the display bird's-eye image synthesized from the captured bird's-eye image and the history image, it is possible to suppress discontinuity of images of the road surface rendered in the non-imaging area.
This method of improving Yamamoto was within the ordinary ability of one of ordinary skill in the art based on the teachings of Kohara.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yamamoto with the teachings of Kohara to obtain the invention as specified in claim 9.
Regarding claim 10, Yamamoto discloses an image processing method comprising: [accumulating images obtained] from a plurality of image capturing devices (¶0039, the vehicle 10 includes, for example, four imagers 14a to 14d as a plurality of imagers 14) [as accumulated images], the plurality of image capturing devices being provided in a moving body (¶0039, The imagers 14 can output moving image data. Examiner considers moving image data to imply a moving body); [generating an overlapping image from the accumulated images, the overlapping image showing an area that overlaps the moving body and is not included in a field of view of the plurality of image capturing devices]; generating a first bird's-eye-view image that shows surroundings of the moving body by merging the images obtained from the plurality of image capturing devices together (¶0041, The image processing device of the present embodiment performs computation or image processing to image data generated by the imagers 14 to generate an image having a wider viewing angle or a virtual image of the vehicle 10 viewed from above, front, or laterally (e.g., a bird's-eye image)), [and generating a second bird's-eye-view image by merging the first bird's-eye-view image and the overlapping image together]; calculating adjustment coefficients in a boundary portion formed between the first bird's-eye-view image and the overlapping image (¶0063, In view of this, the γ-curve coefficient calculator 34c calculates a first coefficient for … the first region of interest (e.g., the region of interest 40FL). Similarly, the γ-curve coefficient calculator 34c calculates a second coefficient … for the second target luminance of the second region of interest. Examiner considers the regions of interest as “boundary regions”), based on property information of the first bird's-eye-view image and property information of the accumulated images used to generate the overlapping image (¶0052, As illustrated in FIG. 5, the region-of-interest setter 30 sets regions of interest 40 (40FL, 40RL, 40RR, and 40FR) in each of the overlapping regions 38 of the imaging regions 36 acquired by the acquirer 28. … The luminance of the regions of interest 40 refers to, for example, the average of luminance of pixels included in the corresponding regions of interest 40. Examiner considers the luminance to be a “property” of the image. Examiner relies upon Kohara to provide the accumulated/overlapping images and considers the overlapping image to be interchangeable with one of the images used in Yamamoto as the primary difference is in the position of the image (under the vehicle vs. around) and should have no impact on the processing steps described by Yamamoto); and adjusting a property of the overlapping image to be merged with the first bird's-eye-view image (¶0085, As a result, the luminance setter 34d can calculate corrected luminance (output luminance Y.sub.1) of the region between the pair of regions of interest 40. Through such correction to the rest of the images, it is possible to generate a recognizable peripheral image), based on the adjustment coefficients calculated (¶0085, The luminance setter 34d substitutes the calculated γ-curve coefficient and the luminance of the position of the object to be corrected (luminance value of original image, i.e., input luminance Y.sub.0) into the relation: Y1=Y.sub.0.sup.1/γ between the input luminance Y.sub.0 (luminance of original image) and the output luminance Y.sub.1).
However, Yamamoto fails to explicitly disclose accumulating images obtained; generating an overlapping image from accumulated images, the overlapping image showing an area that overlaps the moving body and is not included in a field of view of the plurality of image capturing devices; and generating a second bird's-eye-view image by merging the first bird's-eye-view image and the overlapping image together.
Kohara teaches accumulating images obtained (Col. 4, lines 66-67 – Col. 5, lines 1-2, The history image acquiring unit 27 stores the extracted history images in a time series in the history area 43 of the storage unit 15. Examiner considers the history images as the “accumulated images”); generating an overlapping image from accumulated images, the overlapping image showing an area that overlaps the moving body (Fig. 3, 4; Col. 8, lines 5-10, The display image creating unit 29 creates a supplementary image D corresponding to the non-imaging area B by sequentially joining partial areas corresponding to the non-imaging area B from the new history image of the accumulated history image group. Fig. 3D illustrates the accumulated images F1-F10 being assembled into overlapping image D while Fig. 4 illustrates the assembly process over time) and is not included in a field of view of the plurality of image capturing devices (Col. 8, lines 5-8, The display image creating unit 29 creates a supplementary image D corresponding to the non-imaging area B. Examiner considers the non-imaging area as “not included in a field of view”); and generating a second bird's-eye-view image by merging the first bird's-eye-view image and the overlapping image together (Col. 6, lines 27-30, The display image creating unit 29 then creates the display bird's-eye image by combining the created supplementary image with the non-imaging area of the latest captured bird's-eye image).
Both Yamamoto and Kohara are analogous to the claimed invention because they are in the field of generating bird's-eye-view images of vehicles. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the accumulated images from Kohara into the luminance adjustment of Yamamoto. The suggestion/motivation for doing so would have been to suppress discontinuity in the final image, as suggested by Kohara at Col. 10, lines 18-21, Therefore, in the display bird's-eye image synthesized from the captured bird's-eye image and the history image, it is possible to suppress discontinuity of images of the road surface rendered in the non-imaging area.
This method of improving Yamamoto was within the ordinary ability of one of ordinary skill in the art based on the teachings of Kohara.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yamamoto with the teachings of Kohara to obtain the invention as specified in claim 10.
Regarding claim 11, Yamamoto in view of Kohara discloses the image processing method of claim 10.
However, Yamamoto fails to explicitly disclose a computer-readable non-transitory recording medium .
Kohara teaches a computer-readable non-transitory recording medium (Claim 5, a non-transitory computer-readable storage medium).
Both Yamamoto and Kohara are analogous to the claimed invention because they are in the field of generating bird's-eye-view images of vehicles. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the accumulated images from Kohara into the luminance adjustment of Yamamoto. The suggestion/motivation for doing so would have been to suppress discontinuity in the final image, as suggested by Kohara at Col. 10, lines 18-21, Therefore, in the display bird's-eye image synthesized from the captured bird's-eye image and the history image, it is possible to suppress discontinuity of images of the road surface rendered in the non-imaging area.
This method of improving Yamamoto was within the ordinary ability of one of ordinary skill in the art based on the teachings of Kohara.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yamamoto with the teachings of Kohara to obtain the invention as specified in claim 11.
Regarding claim 12, Yin which claim 1 is incorporated, Yamamoto discloses wherein the boundary portion includes a point outside an area (Fig. 5; ¶0053, the part of the imaging region 36F corresponding to the region of interest 40FL and the part of the imaging region 36SL corresponding to the region of interest 40FL. The regions of interest may include areas from both images at the boundary. Therefore, parts not corresponding to 36F, for example, would be considered “outside an area” of 36F) of the [overlapping image].
However, Yamamoto fails to explicitly disclose overlapping image.
Kohara teaches overlapping image (Fig. 3, 4; Col. 8, lines 5-10, The display image creating unit 29 creates a supplementary image D corresponding to the non-imaging area B by sequentially joining partial areas corresponding to the non-imaging area B from the new history image of the accumulated history image group. Fig. 3D illustrates the accumulated images F1-F10 being assembled into overlapping image D while Fig. 4 illustrates the assembly process over time).
Both Yamamoto and Kohara are analogous to the claimed invention because they are in the field of generating bird's-eye-view images of vehicles. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the accumulated images from Kohara into the luminance adjustment of Yamamoto. The suggestion/motivation for doing so would have been to suppress discontinuity in the final image, as suggested by Kohara at Col. 10, lines 18-21, Therefore, in the display bird's-eye image synthesized from the captured bird's-eye image and the history image, it is possible to suppress discontinuity of images of the road surface rendered in the non-imaging area.
This method of improving Yamamoto was within the ordinary ability of one of ordinary skill in the art based on the teachings of Kohara.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yamamoto with the teachings of Kohara to obtain the invention as specified in claim 12.
Regarding claim 13, Yin which claim 1 is incorporated, Yamamoto discloses wherein the boundary portion has a rectangular shape surrounding an area (Fig. 5; ¶0052, he regions of interest 40 are, for example, rectangular regions. The regions of interest may include areas from both images at the boundary. Therefore, parts corresponding to 36F, for example, would be considered “an area” of 36F) of the [overlapping image].
However, Yamamoto fails to explicitly disclose overlapping image.
Kohara teaches overlapping image (Fig. 3, 4; Col. 8, lines 5-10, The display image creating unit 29 creates a supplementary image D corresponding to the non-imaging area B by sequentially joining partial areas corresponding to the non-imaging area B from the new history image of the accumulated history image group. Fig. 3D illustrates the accumulated images F1-F10 being assembled into overlapping image D while Fig. 4 illustrates the assembly process over time).
Both Yamamoto and Kohara are analogous to the claimed invention because they are in the field of generating bird's-eye-view images of vehicles. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the accumulated images from Kohara into the luminance adjustment of Yamamoto. The suggestion/motivation for doing so would have been to suppress discontinuity in the final image, as suggested by Kohara at Col. 10, lines 18-21, Therefore, in the display bird's-eye image synthesized from the captured bird's-eye image and the history image, it is possible to suppress discontinuity of images of the road surface rendered in the non-imaging area.
This method of improving Yamamoto was within the ordinary ability of one of ordinary skill in the art based on the teachings of Kohara.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yamamoto with the teachings of Kohara to obtain the invention as specified in claim 13.
Claims 2 and 4-7 are rejected under 35 U.S.C. 103 as being unpatentable over Yamamoto et al. (US 2021/0160432) (hereafter, “Yamamoto”) in view of Kohara et al. (US 10,873,725) (hereafter, “Kohara”) as applied to claims 1 and 8-11 above, and further in view of Ouchi (US 2015/0170559).
Regarding claim 2, in which claim 1 is incorporated, Yamamoto discloses [wherein the program instructions further cause the processor to calculate reference adjustment coefficients at positions of vertices of the boundary portion], which is rectangular in shape (¶0052, The regions of interest 40 are, for example, rectangular regions), and calculate an adjustment coefficient for every pixel in the overlapping image by using the reference adjustment coefficients calculated (Fig. 7; ¶0072, The linear interpolator 34a generates a linear interpolation formula 42 (42F) using the correction value (N=−50) of the region of interest 40FL and the correction value (N=+50) of the region of interest 40FR set by the first setter 32 (S110) … Then, the luminance setter 34d corrects (sets) the luminance of the region between the region of interest 40FL and the region of interest 40FR with the correction value (individual correction value) calculated by the generated linear interpolation formula 42F. Examiner considers the individual correction value to indicate a pixel-wise representation as Fig. 7 shows the interpolation to span the entire image).
However, Yamamoto fails to explicitly disclose wherein the program instructions further cause the processor to calculate reference adjustment coefficients at positions of vertices of the boundary portion.
Ouchi teaches wherein the program instructions further cause the processor to calculate reference adjustment coefficients at positions of vertices of the boundary portion (¶0027, generates different luminance correction coefficients in two regions partitioned using the vertex of the overlapping region B120 that is located on the screen interior side and the vertex that is diagonally opposite thereto).
Yamamoto, Kohara, and Ouchi are analogous to the claimed invention because Yamamoto and Kohara are in the field of generating bird's-eye-view images of vehicles while Ouchi is directed towards adjusting luminance values of overlapping areas between images. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate luminance adjustment techniques of Ouchi into the accumulated images from Kohara and the luminance adjustment of Yamamoto. The suggestion/motivation for doing so would have been to provide uniform luminance, as suggested by Ouchi at ¶0061, According to the present invention, it is possible to provide a projection-type image display apparatus in which uniformity in the luminance.
This method of improving Yamamoto was within the ordinary ability of one of ordinary skill in the art based on the teachings of Kohara and Ouchi.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yamamoto and the teachings of Kohara with the teachings of Ouchi to obtain the invention as specified in claim 2.
Regarding claim 4, in which claim 2 is incorporated, Yamamoto discloses wherein the program instructions further cause the processor to calculate, by linear interpolation using two reference adjustment coefficients calculated at two positions in the direction in which the moving body moves, adjustment coefficients for pixels between the two positions in the [overlapping] image (Fig. 9; ¶0076, In the imaging region 36SL in the vehicular longitudinal direction (Z-axis direction), for example, the front-side region of interest 40FL exhibits luminance of 100 in 256 levels, and the rear-side region of interest 40RL exhibits luminance of 50 in 256 levels. …The linear interpolator 34a generates a linear interpolation formula 42L using the correction value (N=+100) for the region of interest 40FL and the correction value (N=+150) for the region of interest 40RL set by the first setter 32; ¶0096 As described above, the gradient of a linear interpolation formula can be corrected and a linear interpolation formula with a γ-curve coefficient can be calculated. As the linear interpolation spans the entirety of the vehicle length, Examiner considers the formula to calculate coefficients for each pixel in the longitudinal direction of the image and to be in the “moving direction” and ¶0096 indicates that the linear interpolations throughout Yamamoto can be replaced with one utilizing a coefficient).
However, Yamamoto fails to explicitly disclose overlapping image.
Kohara teaches overlapping image (Fig. 3, 4; Col. 8, lines 5-10, The display image creating unit 29 creates a supplementary image D corresponding to the non-imaging area B by sequentially joining partial areas corresponding to the non-imaging area B from the new history image of the accumulated history image group. Fig. 3D illustrates the accumulated images F1-F10 being assembled into overlapping image D while Fig. 4 illustrates the assembly process over time)
Yamamoto, Kohara, and Ouchi are analogous to the claimed invention because Yamamoto and Kohara are in the field of generating bird's-eye-view images of vehicles while Ouchi is directed towards adjusting luminance values of overlapping areas between images. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the accumulated images from Kohara into the luminance adjustment of Yamamoto and the luminance adjustment techniques of Ouchi. The suggestion/motivation for doing so would have been to suppress discontinuity in the final image, as suggested by Kohara at Col. 10, lines 18-21, Therefore, in the display bird's-eye image synthesized from the captured bird's-eye image and the history image, it is possible to suppress discontinuity of images of the road surface rendered in the non-imaging area.
This method of improving Yamamoto was within the ordinary ability of one of ordinary skill in the art based on the teachings of Kohara and Ouchi.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yamamoto and the teachings of Ouchi with the teachings of Kohara to obtain the invention as specified in claim 4.
Regarding claim 5, in which claim 4 is incorporated, Yamamoto discloses wherein the program instructions further cause the processor to calculate, [in the overlapping image], adjustment coefficients for pixels in a direction that is orthogonal to the direction in which the moving body moves, by linear interpolation using the adjustment coefficients calculated at positions in the orthogonal direction (Fig. 7; ¶0072, The linear interpolator 34a generates a linear interpolation formula 42 (42F) using the correction value (N=−50) of the region of interest 40FL and the correction value (N=+50) of the region of interest 40FR set by the first setter 32 (S110) … Then, the luminance setter 34d corrects (sets) the luminance of the region between the region of interest 40FL and the region of interest 40FR with the correction value (individual correction value) calculated by the generated linear interpolation formula 42F; ¶0096 As described above, the gradient of a linear interpolation formula can be corrected and a linear interpolation formula with a γ-curve coefficient can be calculated. Fig. 7 shows the linear interpolation between two points orthogonal to the direction of the vehicle’s motion and ¶0096 indicates that the linear interpolations throughout Yamamoto can be replaced with one utilizing a coefficient).
However, Yamamoto fails to explicitly disclose overlapping image.
Kohara teaches overlapping image (Fig. 3, 4; Col. 8, lines 5-10, The display image creating unit 29 creates a supplementary image D corresponding to the non-imaging area B by sequentially joining partial areas corresponding to the non-imaging area B from the new history image of the accumulated history image group. Fig. 3D illustrates the accumulated images F1-F10 being assembled into overlapping image D while Fig. 4 illustrates the assembly process over time)
Yamamoto, Kohara, and Ouchi are analogous to the claimed invention because Yamamoto and Kohara are in the field of generating bird's-eye-view images of vehicles while Ouchi is directed towards adjusting luminance values of overlapping areas between images. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate the accumulated images from Kohara into the luminance adjustment of Yamamoto and the luminance adjustment techniques of Ouchi. The suggestion/motivation for doing so would have been to suppress discontinuity in the final image, as suggested by Kohara at Col. 10, lines 18-21, Therefore, in the display bird's-eye image synthesized from the captured bird's-eye image and the history image, it is possible to suppress discontinuity of images of the road surface rendered in the non-imaging area.
This method of improving Yamamoto was within the ordinary ability of one of ordinary skill in the art based on the teachings of Kohara and Ouchi.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yamamoto and the teachings of Ouchi with the teachings of Kohara to obtain the invention as specified in claim 5.
Regarding claim 6, in which claim 2 is incorporated, Yamamoto discloses [wherein the program instructions further cause the processor to set a reference adjustment coefficient at a vertex to 1.0], the vertex being located forward in the direction in which the moving body moves (Fig. 5; ¶0055, a first region of interest (e.g., the region of interest 40FL) of the vehicular front region).
However, Yamamoto fails to explicitly disclose wherein the program instructions further cause the processor to set a reference adjustment coefficient at a vertex to 1.0.
Ouchi teaches wherein the program instructions further cause the processor to set a reference adjustment coefficient at a vertex to 1.0 (Fig. 4B; ¶0031, For this reason, as shown in T102 in FIG. 4D, in the overlapping region B102, the luminance correction value for the vertex located on the screen interior side is 1. Fig. 4 displays the coefficient for the top left vertex of the ROI being set to 1).
Yamamoto, Kohara, and Ouchi are analogous to the claimed invention because Yamamoto and Kohara are in the field of generating bird's-eye-view images of vehicles while Ouchi is directed towards adjusting luminance values of overlapping areas between images. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate luminance adjustment techniques of Ouchi into the accumulated images from Kohara and the luminance adjustment of Yamamoto. The suggestion/motivation for doing so would have been to provide uniform luminance, as suggested by Ouchi at ¶0061, According to the present invention, it is possible to provide a projection-type image display apparatus in which uniformity in the luminance.
This method of improving Yamamoto was within the ordinary ability of one of ordinary skill in the art based on the teachings of Kohara and Ouchi.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yamamoto and the teachings of Kohara with the teachings of Ouchi to obtain the invention as specified in claim 6.
Regarding claim 7, Yamamoto in view of Kohara further in view of Ouchi discloses the image processing device according to claim 2.
However, Yamamoto fails to explicitly disclose wherein the program instructions further cause the processor not to the adjustment coefficients unless an image corresponding to one of the vertices of the rectangular boundary portion is included in the overlapping image generated by the overlapping image generating part, and wherein the adjustment part is further configured not to adjust the property of the overlapping image to be merged with the first bird’s-eye-view image by the merging part unless the coefficient calculation part calculates the adjustment coefficients.
Ouchi teaches wherein the program instructions further cause the processor not to calculate the adjustment coefficients (¶0022, The overlapping region correction coefficient generation unit 301 calculates luminance correction coefficients to be applied … with the pixel positions in the image overlapping region generated by the overlapping portion correction timing generation unit 200. Ouchi teaches the calculation of coefficients in only the overlapping area which is equivalent to not calculating when there is no overlap), and wherein the adjustment part is further configured not to adjust the property of the overlapping image to be merged with the first bird’s-eye-view image by the merging part unless the coefficient calculation part calculates the adjustment coefficients (¶0022, The multiplier 303 uses the luminance correction coefficients calculated by the overlapping region correction coefficient generation unit 301 to multiply the input image by the luminance correction coefficients, and thereby performs luminance correction. Since the correction only occurs in the overlapping region where coefficients are calculated, Examiner considers to mean that no adjustment occurs in locations where no coefficients are calculated).
Kohara teaches unless an image corresponding to one of the vertices of the rectangular boundary portion is included in the overlapping image generated by the overlapping image generating part (Fig. 3D; Col. 8, lines 5-10, The display image creating unit 29 creates a supplementary image D corresponding to the non-imaging area B by sequentially joining partial areas corresponding to the non-imaging area B from the new history image of the accumulated history image group C. Fig. 3D illustrates a stack of history images forming a supplementary image. Examiner notes that when combined with the overlapping region generation of Ouchi (Fig. 3 B102), one of the individual images in Fig. 3D must be overlapping with the boundary region).
Yamamoto, Kohara, and Ouchi are analogous to the claimed invention because Yamamoto and Kohara are in the field of generating bird's-eye-view images of vehicles while Ouchi is directed towards adjusting luminance values of overlapping areas between images. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate luminance adjustment techniques of Ouchi and the accumulated images from Kohara into the luminance adjustment of Yamamoto. The suggestion/motivation for incorporating Ouchi would have been to provide uniform luminance, as suggested by Ouchi at ¶0061, According to the present invention, it is possible to provide a projection-type image display apparatus in which uniformity in the luminance. The suggestion/motivation for incorporating Kohara would have been to suppress discontinuity in the final image, as suggested by Kohara at Col. 10, lines 18-21, Therefore, in the display bird's-eye image synthesized from the captured bird's-eye image and the history image, it is possible to suppress discontinuity of images of the road surface rendered in the non-imaging area.
This method of improving Yamamoto was within the ordinary ability of one of ordinary skill in the art based on the teachings of Kohara and Ouchi.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yamamoto with the teachings of Kohara and the teachings of Ouchi to obtain the invention as specified in claim 7.
Claims 3 is rejected under 35 U.S.C. 103 as being unpatentable over Yamamoto et al. (US 2021/0160432) (hereafter, “Yamamoto”) in view of Kohara et al. (US 10,873,725) (hereafter, “Kohara”) as applied to claims 1 and 8-11 above, and further in view of Ishikawa et al. (US 2020/0160526) (hereafter, “Ishikawa”).
Regarding claim 3, in which claim 1 is incorporated, Yamamoto discloses wherein the program instructions further cause the processor to: calculate reference adjustment coefficients at positions of vertices in a direction in which the moving body moves (Fig. 9, 40FL, 40RL; ¶0063, In view of this, the γ-curve coefficient calculator 34c calculates a first coefficient … of the first region of interest (e.g., the region of interest 40FL). Similarly, the γ-curve coefficient calculator 34c calculates a second coefficient … of the second region of interest (e.g., the region of interest 40FR). Fig. 9 shows a linear interpolation in the direction of motion between 40FL and 40 RL. Examiner considers the example in ¶0063 to be applicable to the pairing of 40FL and 40RL as well) [and at least one position between the vertices in the direction in which the moving body moves]; and calculate adjustment coefficients for pixels between two positions that neighbor each other in the direction in which the moving body moves (Fig. 9; ¶0076, In the imaging region 36SL in the vehicular longitudinal direction (Z-axis direction), for example, the front-side region of interest 40FL exhibits luminance of 100 in 256 levels, and the rear-side region of interest 40RL exhibits luminance of 50 in 256 levels. …The linear interpolator 34a generates a linear interpolation formula 42L using the correction value (N=+100) for the region of interest 40FL and the correction value (N=+150) for the region of interest 40RL set by the first setter 32; ¶0096 As described above, the gradient of a linear interpolation formula can be corrected and a linear interpolation formula with a γ-curve coefficient can be calculated. As the linear interpolation spans the entirety of the vehicle length, Examiner considers the formula to calculate coefficients for each pixel in the longitudinal direction of the image and ¶0096 indicates that the linear interpolations throughout Yamamoto can be replaced with one utilizing a coefficient), by using the reference adjustment coefficients calculated at the two positions that neighbor each other (Fig. 9, 40FL, 40RL; Examiner considers the two ROIs 40FL and 40RL as “neighbors”).
However, Yamamoto fails to explicitly disclose at least one position between the vertices in the direction in which the moving body move.
Ishikawa teaches at least one position between the vertices in the direction in which the moving body move (Fig. 5, #4040; ¶0069, In step S20604, the difference obtaining unit 1070 performs processing for obtaining an intensity I2_s_ij in the second image at each of the vertex positions V_ij of the N_sr rectangular parallelepiped areas 4030 obtained by the dividing in step S20602. As Fig. 5 #4040 indicates intermediate vertices on all sides of the ROI #4020, Examiner considers this to include a side in the moving direction of the vehicle).
Yamamoto, Kohara, and Ishikawa are analogous to the claimed invention because Yamamoto and Kohara are in the field of generating bird's-eye-view images of vehicles while Ishikawa is directed towards adjusting luminance when merging images. It would have been obvious to a person of ordinary skill before the effective filing date of the claimed invention to incorporate intermediate vertices of Ishikawa into the accumulated images from Kohara and the luminance adjustment of Yamamoto. The suggestion/motivation for doing so would have been to reduce noise when merging, as suggested by Ishikawa at ¶0004, noise in the subtraction image can be reduced.
This method of improving Yamamoto was within the ordinary ability of one of ordinary skill in the art based on the teachings of Kohara and Ishikawa.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Yamamoto and the teachings of Kohara with the teachings of Ishikawa to obtain the invention as specified in claim 3.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Sugie et al. (US 2020/0082179) discloses luminance correction in bird’s-eye-view images (¶0094, the luminance of the bird's-eye view image is entirely adjusted).
Donishi et al. (US 2014/0085473) discloses utilizing an adjustment curve for luminance correction (This luminance adjustment of the photographed image is effected by using generally a luminance adjustment (correction) curve).
Okuyama (US 2012/0026333) discloses an image brightness adjustment method (¶0146, if a currently set brightness value in the second decoder and brightness correction unit 822 is B2, a newly set brightness value by the brightness correction is B2', a currently set brightness value in the third decoder and brightness correction unit 832 is B3, and a newly set brightness value by the brightness correction is B3').
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIAOMAO DING whose telephone number is (571)272-7237. The examiner can normally be reached Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIAOMAO DING/Examiner, Art Unit 2676
/Henok Shiferaw/Supervisory Patent Examiner, Art Unit 2676
Prosecution Insights