Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
35 USC § 112 (f)
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: "transformation unit", “image generating unit” in claim claims 1-9.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification (¶0119) as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 7-10 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Yamamoto et al. (US Pub 2021/0078496 A1).
As to claim 1, Yamamoto discloses an information processing device comprising:
a viewpoint transformation unit that performs viewpoint transformation on a plurality of images generated by a plurality of imaging units mounted on front, rear, left, and right sides of a vehicle (Fig. 5 to Fig. 8, ¶0025-0026, ¶0038, “generate a bird's-eye view image (periphery image) that displays a vehicle image (own vehicle icon) illustrating the vehicle 1 at a center position, for example, based on the captured image data imaged by the image capturing unit 15, and displays the bird's-eye view image on the display device 8. Further, the CPU 14a can change a position of the virtual viewpoint when generating the bird's-eye view image, and can generate the bird's-eye view image that looks at the vehicle image from directly above or the bird's-eye view image that looks at the vehicle image from an oblique direction.”); and
an overhead image generating unit that generates an overhead image by combining the plurality of images having been subjected to the viewpoint transformation by the viewpoint transformation unit (Fig. 5-7, ¶0038, “generate a bird's-eye view image (periphery image) that displays a vehicle image (own vehicle icon) illustrating the vehicle 1 at a center position, for example, based on the captured image data imaged by the image capturing unit 15, and displays the bird's-eye view image on the display device 8. Further, the CPU 14a can change a position of the virtual viewpoint when generating the bird's-eye view image, and can generate the bird's-eye view image that looks at the vehicle image from directly above or the bird's-eye view image that looks at the vehicle image from an oblique direction.” ¶0050, “the synthesized image control section 36a provides an enlargement display of the synthesized image and performs the movement of the display position of the vehicle image. In FIG. 5, the initial display state, where the synthesized image is in a non-enlarged state and the vehicle image is in a non-moving state, is illustrated. On the other hand, in FIGS. 6 and 7, the processing executing state, where the synthesized image is in an enlarged state and the vehicle image is in a moving state, is illustrated.”), wherein
the overhead image generating unit (Fig. 3, ¶0050)
in a case where a first overhead image to be displayed in the vehicle is generated, executes first synthesis processing (¶0051, “the synthesized image control section 36a generates a first synthesized image G3 including the vehicle image G1 and the periphery image G2 as a first bird's-eye view image of the vehicle 1 viewed from directly above (for example, a virtual viewpoint is set directly above the vehicle 1).”), and
in a case where a second overhead image for executing recognition processing for surroundings of the vehicle is generated, executes second synthesis processing different from the first synthesis processing (¶0051, “the synthesized image control section 36a generates a second synthesized image G6 including the three-dimensional vehicle image G4 and the three-dimensional periphery image G5 as a three-dimensional second bird's-eye view image of the vehicle 1 viewed from the obliquely upper side.” ¶0054, “a semi-transparent display mode so that a display is provided such that an object or the like that is present in the three-dimensional periphery image G5 that is blocked by the three-dimensional vehicle image G4 is easily visible, and the vehicle position information GR that enables the position of the three-dimensional vehicle image G4 to be identified with respect to the road surface is displayed. The vehicle position information GR, for example, can be information for displaying the position where the three-dimensional vehicle image G4 is present on the road surface of the three-dimensional periphery image G5 in grayscale, and can be information in which the position where the three-dimensional vehicle image G4 is present, is displayed with a surrounding line (for example, a broken line). The vehicle position information GR may be displayed with respect to the vehicle image G1 in the periphery image G2. Further, an information display region Ga may be provided in a part of the second synthesized image G6, a message such as “Please check around the vehicle directly.” may be displayed, for example, when the second synthesized image G6 or the first synthesized image G3 is displayed, and the driver or the like may be alerted when an image showing the surrounding situation is displayed.”).
As to claim 7, claim 1 is incorporated and Yamamoto discloses when combining the plurality of images having been subjected to the viewpoint transformation, the second synthesis processing combines the plurality of images having been subjected to the viewpoint transformation at a synthesis position different from a synthesis position in the first synthesis processing (¶0054, “by displaying a three-dimensional periphery image G5 of the vehicle 1 viewed from the rear obliquely upper left side, it is possible to make a display that makes it easy to ascertain the positional relationship between the vehicle 1 and the periphery objects thereof in a state close to the real world (having reality). Similar to the vehicle image G1, the three-dimensional vehicle image G4 is an image composed of a plurality of polygons and illustrating the three-dimensional shape of the vehicle 1. The three-dimensional periphery image G5 is an image using a well-known bird's-eye view image generation technique, and is an image generated by attaching a plurality of captured images obtained by imaging the periphery of the vehicle 1 by the image capturing unit 15 to a bowl shaped or cylindrical shaped three-dimensional surface. Further, the three-dimensional periphery image G5 is displayed in, for example, a semi-transparent display mode so that a display is provided such that an object or the like that is present in the three-dimensional periphery image G5 that is blocked by the three-dimensional vehicle image G4 is easily visible, and the vehicle position information GR that enables the position of the three-dimensional vehicle image G4 to be identified with respect to the road surface is displayed. The vehicle position information GR, for example, can be information for displaying the position where the three-dimensional vehicle image G4 is present on the road surface of the three-dimensional periphery image G5 in grayscale, and can be information in which the position where the three-dimensional vehicle image G4 is present, is displayed with a surrounding line (for example, a broken line). The vehicle position information GR may be displayed with respect to the vehicle image G1 in the periphery image G2. Further, an information display region Ga may be provided in a part of the second synthesized image G6, a message such as “Please check around the vehicle directly.” may be displayed, for example, when the second synthesized image G6 or the first synthesized image G3 is displayed, and the driver or the like may be alerted when an image showing the surrounding situation is displayed.”).
As to claim 8, claim 1 is incorporated and Yamamoto discloses the overhead image generating unit executes the first synthesis processing and the second synthesis processing in parallel (Fig. 12, ¶0079, “when the operation requesting traveling assistance is performed by the driver, it is checked whether the parking target position, the stop target position for turning back, or the like (for example, the target position 40) is set. When the target position 40 is set (Yes in S100), the synthesized image control section 36a generates a first synthesized image G3 and a second synthesized image G6 (S102). That is, the synthesized image control section 36a generates a bird's-eye viewed periphery image G2 based on the captured image captured by the image capturing unit 15 acquired by the image acquisition section 30a. Further, the synthesized image control section 36a superimposes the display data of the vehicle image G1 read from the storage unit such as the ROM 14b on the periphery image G2 to generate the first synthesized image G3. Similarly, the synthesized image control section 36a generates a bird's-eye viewed three-dimensional periphery image G5 based on the captured image captured by the image capturing unit 15 acquired by the image acquisition section 30a. Further, the synthesized image control section 36a superimposes the display data of the three-dimensional vehicle image G4 read from the storage unit such as the ROM 14b on the three-dimensional periphery image G5 to generate the second synthesized image G6.”).
As to claim 9, claim 1 is incorporated and Yamamoto discloses the recognition processing is processing of recognizing a line of a parking space frame, a car stopper, or an obstacle (¶0039, “When performing the traveling assistance such as parking assistance, the CPU 14a provides, for example, a display that makes it easy for the driver to recognize the situation of parking assistance or the surrounding situation of the vehicle 1 during the parking assistance, thereby it is easy to increase the drivers sense of security when traveling assistance, and it is possible to realize an image display that makes it easy for the driver to feel that the burden during the driving is reduced. For example, the display content which is displayed on the display device 8 is automatically changed according to the positional relationship between a target position, such as a parking target position or a stop position for turning back required when moving to the parking target position, and the vehicle 1 (for example, according to at least one of the distance between the current position of the vehicle 1 and the target position, or the period until the vehicle 1 reaches the target position). As will be described later, the CPU 14a executes the enlargement display processing of a synthesized image that includes a vehicle image, executes a movement processing of the display position, indicates a direction indicator indicating the direction of the target position, or indicates a stop indicator that implies a stop when the vehicle 1 reaches the target position.” ¶0083, “When there is no stop target position for the new turn-back and only the parking target position is present, the parking target position is set to the new target position 40 and the flow of FIG. 11 is executed again. When the stop target position for turning back is not present in the movement route set by the route setting section 34, the parking target position is set as the target position 40 from the beginning.”).
As to claim 10, Yamamoto discloses an information processing method in which a computer executes processing of: performing viewpoint transformation on a plurality of images generated by a plurality of imaging units mounted on front, rear, left, and right sides of a vehicle; and generating an overhead image by combining the plurality of images having been subjected to the viewpoint transformation, wherein the processing of generating in a case where a first overhead image to be displayed in the vehicle is generated, executes first synthesis processing, and in a case where a second overhead image for executing recognition processing for surroundings of the vehicle is generated, executes second synthesis processing different from the first synthesis processing (See claim 1 for detailed analysis.).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2-4, 6 are rejected under 35 U.S.C. 103 as being unpatentable over Yamamoto et al. (US Pub 2021/0078496 A1) in view of Jia et al. (US Pub 2018/0033155 A1).
As to claim 2, claim 1 is incorporated and Yamamoto does not disclose wherein the first synthesis processing combines the plurality of images having been subjected to the viewpoint transformation by gradient blending and generates the first overhead image.
Jie teaches the first synthesis processing combines the plurality of images having been subjected to the viewpoint transformation by gradient blending and generates the first overhead image (Jia, ¶0071, “The image blender 138 may blend image gradients and/or may make color consistent along the boundary between images. In some approaches, the image blender 138 may perform Poisson editing. For example, the image blender 138 may perform Poisson editing for color and intensity matching (e.g., solve a Poisson equation) in some approaches to blend the images.”).
Yamamoto and Jie are considered to be analogous art because all pertain to image processing. It would have been obvious before the effective filing date of the claimed invention to have modified Yamamoto with the features of “the first synthesis processing combines the plurality of images having been subjected to the viewpoint transformation by gradient blending and generates the first overhead image” as taught by Jie. The suggestion/motivation would have been in order to an image pair captured from a wide-angle camera and a telephoto camera may be seamlessly composited (e.g., fused) (Jie, ¶0043).
As to claim 3, claim 2 is incorporated and the combination of Yamamoto discloses the gradient blending uses a gradient blending value having a predetermined gradation at a boundary (Jie, ¶0041, “the boundary values of the warped telephoto image may be enforced to be the same as the wide-angle image. A color-corrected warped telephoto image may be obtained by preserving the color gradient. This may be achieved by solving a Poisson equation in some approaches. With color correction, any visible inconsistency may be successfully removed along the composition boundary so that the final composite (e.g., fused) image is free of artifacts.” ¶0092, “the gradient of the first composite image within the co-visible region (e.g., the gradient of the warped second image in the co-visible region) may be transferred to the blended image with a boundary condition that the intensity (e.g., color) at the boundary (e.g., seam) is the same as the corresponding intensity of the first image (e.g., reduced detail wide angle image).” ¶0112, “A boundary condition (e.g., I.sub.out|∂Ω=I.sub.wide|∂Ω) may ensure that the color along the boundary (e.g., seam) between images (e.g., the wide-angle image 948 and the warped telephoto image) is the same (or approximately the same, for example). More detail is provided in relation to one or more of FIGS. 14-17.”).
As to claim 4, claim 1 is incorporated and the combination of Yamamoto discloses the second synthesis processing combines the plurality of images having been subjected to the viewpoint transformation by simple blending and generates the second overhead image (Jie, ¶0042, “It may be observed that the difference between the original warped telephoto image and the color corrected image may vary slowly in spatial domain. Therefore, such gradient-domain image blending on may be applied on downsampled images and the upsampled difference may be added back to the full-resolution image. This approach may be equivalently effective in compensating the color difference and/or may be performed more efficiently (e.g., run faster). “ ¶0071, “The image blender 138 may blend images (e.g., first and second images, a wide-angle image and a telephoto image, etc.).”.¶0125, “gradient-domain image blending may be performed on reduced-detail (e.g., downsized, downsampled, etc.) images in some approaches.” ¶0131, “Performing image blending (e.g., gradient-domain image blending) on reduced-detail images may help to reduce processing complexity and/or processing delay. For example, performing image blending (e.g., gradient-domain image blending) on reduced-detail images may increase processing efficiency and/or speed.”).
As to claim 6, claim 1 is incorporated and the combination of Yamamoto discloses the first synthesis processing performs color correction on each of the plurality of images having been subjected to the viewpoint transformation (Jie, ¶0041, “A color-corrected warped telephoto image may be obtained by preserving the color gradient.” ¶0092, “the gradient of the first composite image within the co-visible region (e.g., the gradient of the warped second image in the co-visible region) may be transferred to the blended image with a boundary condition that the intensity (e.g., color) at the boundary (e.g., seam) is the same as the corresponding intensity of the first image (e.g., reduced detail wide angle image).” ¶0112, “the image blender 938 may blend only image gradients and make color consistent along the boundary (e.g., seam).”).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Yamamoto et al. (US Pub 2021/0078496 A1) in view of Jia et al. (US Pub 2018/0033155 A1) and Pérez, Patrick, Michel Gangnet, and Andrew Blake. "Poisson image editing." "https://cs.brown.edu/courses/cs195-g/asgn/proj2/resources/PoissonImageEditing.pdf" (Year: 2003).
As to claim 5, claim 4 is incorporated and the combination of Yamamoto and Jie discloses the simple blending uses a gradient blending value (Jie, ¶0041, “the boundary values of the warped telephoto image may be enforced to be the same as the wide-angle image.” ¶0071, “The image blender 138 may blend image gradients and/or may make color consistent along the boundary between images.” ¶0092, “the gradient of the first composite image within the co-visible region (e.g., the gradient of the warped second image in the co-visible region) may be transferred to the blended image with a boundary condition that the intensity (e.g., color) at the boundary (e.g., seam) is the same as the corresponding intensity of the first image (e.g., reduced detail wide angle image).”).
Perez discloses uses a gradient blending value having no gradation at a boundary (Perez, Page 313, “under Neumann boundary conditions specifying that the value of the gradient of the new image in the direction normal to the boundary is zero.” Page 314, “The Poisson equation (4) then becomes the following Laplace equation with boundary conditions” Fig. 12, “Setting periodic boundary values on the border of a rectangular region before integrating with the Poisson solver yields a tileable image.”).
Yamamoto, Jie and Perez are considered to be analogous art because all pertain to image processing. It would have been obvious before the effective filing date of the claimed invention to have modified Yamamoto with the features of “uses a gradient blending value having no gradation at a boundary” as taught by Perez. The claim would have been obvious because “a person of ordinary skill has good reason to pursue the known options within his or her technical grasp. If this leads to the anticipated success, it is likely the product not of innovation but of ordinary skill and common sense.”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zhang et al. (US Pub 2024/0265555 A1) discloses during image alignment with homography estimation, a homography estimation between a pair of input images according to the ROI may be determined and then aligned images can be output based on the estimated homography
Saikyo et al. (US Pub 2021/0302977 A1) discloses recognizes the traveling lane by comparing the pattern of road section lines recognized from the accurate map information (e.g., the arrangement of continuous lines and broken lines) and the pattern of the road section lines around the host vehicle recognized from images captured by the cameras.
Pflug (US Pub 2014/0333729 A1) discloses the alpha map (generally) possess a sliding gradient from 100 percent to zero (0) percent for one camera's captured image and zero percent to 100 percent for the other camera's captured image.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YU CHEN whose telephone number is (571)270-7951. The examiner can normally be reached on M-F 8-5 PST Mid-day flex.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached on 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YU CHEN/Primary Examiner, Art Unit 2613
Prosecution Insights