DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
This application claims benefit of foreign priority under 35 U.S.C. 119(a)-(d) of Application No. JP2023-144827, filed in Japan on 09/06/2023.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 08/15/2024 and 11/19/2024 were considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 4, and 6-8 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Gao et al. (NPL: Dual-fisheye omnidirectional stereo, hereafter referred as Gao).
Regarding Claim 1:
Gao teaches an image processing system comprising (Gao: Abstract; novel omnidirectional stereo camera setup): a first imaging unit including a first optical system in which a maximum half angle of view is θa; a second imaging unit including a second optical system in which a maximum half angle of view is θb, the first optical system and the second optical system being configured such that the following expression is satisfied, and θa + θb > 180° an optical axis of the first optical system and an optical axis of the second optical system being disposed in opposite directions to each other (Gao: I. Introduction; we propose a novel configuration where two 245 degree FOV fisheye cameras are installed facing opposite directions on a rigid rod. The configuration is shown in Fig. 1(a). The overlapping FOV between the two fisheye cameras naturally forms a ring-shaped omnidirectional spatial stereo setup as shown in Fig. 4(c). We, therefore, obtain coverage of 65 degrees in the vertical direction and 360 degrees in horizontal directions.); one or more memories storing instructions; and one or more processors executing the instructions to (Gao: A. Fisheye Camera Calibration Results; We use the open source chessboard detector from OpenCV3, as shown in Fig. 5. We modify the CamOdoCal in [26] as our calibration toolbox, based on Ceres Solver4; shows use of processor to execute code): generate composite image information of a celestial sphere based on outputs of the first imaging unit and the second imaging unit (Gao: A. Omnidirectional Stereo Camera Model; The overlapping field of view between the two fisheye cameras naturally forms a ring-shaped omnidirectional spatial stereo setup, as indicated in Fig. 4(c). Using fisheye cameras with a γ FOV, we obtain FOV around (γ−180)-degree vertical and 360-degree horizontal; as per the definition of celestial sphere from Par. [0033] of the specification, celestial sphere (horizontal 360°, vertical 180°)), and generate distance information based on an output of a superimposed viewing angle of the first imaging unit and the second imaging unit (Gao: I. Introduction; For the 65-degree horizontal, and 360-degree vertical ring-shaped regions that are observed by both cameras, we will use spatial stereo matching to obtain depth values directly).
In regards to Claim 2, Gao further teaches the image processing system according to claim 1, wherein at least one of the first optical system and the second optical system is an optical system with a variable angle of view (Gao: Fig. 2; (b) Illustration of fisheye lens. For some fisheye lenses, the incident angles are allowed to be larger than 90 degrees, up to 135 degrees).
In regards to Claim 4, Gao further teaches the image processing system according to claim 1, wherein the one or more processors further executes the instructions to execute trimming processing of cutting an image region of a superimposed viewing angle of the first imaging unit and the second imaging unit (Gao: C. ROI Extraction; We introduce an ROI extraction module for cropping regions of an original fisheye image and converting it into a distortion-free image that is applicable for the standard stereo matching algorithms.), and generate the distance information based on an output of the trimming processing (Gao: Abstract; The overlapping camera views are rectified into stereo image pairs, from which a spatial stereo matching pipeline is developed for depth estimation in all horizontal directions).
Regarding Claim 6:
Gao further teaches a movable apparatus comprising (Gao: VI. Conclusion and Discussion; This stereo system provides a horizontal omnidirectional stereo view directly and it is satisfied with the SWaP constraints on moving robots): a first imaging unit including a first optical system in which a maximum half angle of view is θa; a second imaging unit including a second optical system in which a maximum half angle of view is θb, the first optical system and the second optical system being configured such that the following expression is satisfied, and θa + θb > 180° an optical axis of the first optical system and an optical axis of the second optical system being disposed in opposite directions to each other; (Gao: I. Introduction; we propose a novel configuration where two 245 degree FOV fisheye cameras are installed facing opposite directions on a rigid rod. The configuration is shown in Fig. 1(a). The overlapping FOV between the two fisheye cameras naturally forms a ring-shaped omnidirectional spatial stereo setup as shown in Fig. 4(c). We, therefore, obtain coverage of 65 degrees in the vertical direction and 360 degrees in horizontal directions.); one or more memories storing instructions; and one or more processors executing the instructions to (Gao: A. Fisheye Camera Calibration Results; We use the open source chessboard detector from OpenCV3, as shown in Fig. 5. We modify the CamOdoCal in [26] as our calibration toolbox, based on Ceres Solver4; shows use of processor to execute code): generate composite image information of a celestial sphere based on outputs of the first imaging unit and the second imaging unit (Gao: A. Omnidirectional Stereo Camera Model; The overlapping field of view between the two fisheye cameras naturally forms a ring-shaped omnidirectional spatial stereo setup, as indicated in Fig. 4(c). Using fisheye cameras with a γ FOV, we obtain FOV around (γ−180)-degree vertical and 360-degree horizontal; as per the definition of celestial sphere from Par. [0033] of the specification, celestial sphere (horizontal 360°, vertical 180°)), and generate distance information based on an output of a superimposed viewing angle of the first imaging unit and the second imaging unit (Gao: I. Introduction; For the 65-degree horizontal, and 360-degree vertical ring-shaped regions that are observed by both cameras, we will use spatial stereo matching to obtain depth values directly).
Regarding Claim 7:
Gao further teaches an image processing method using an imaging apparatus (Gao: Abstract; novel omnidirectional stereo camera setup) including a first imaging unit including a first optical system in which a maximum half angle of view is θa, and a second imaging unit including a second optical system in which a maximum half angle of view is θb, the first optical system and the second optical system being configured such that the following expression is satisfied, and θa + θb > 180° an optical axis of the first optical system and an optical axis of the second optical system being disposed in opposite directions to each other (Gao: I. Introduction; we propose a novel configuration where two 245 degree FOV fisheye cameras are installed facing opposite directions on a rigid rod. The configuration is shown in Fig. 1(a). The overlapping FOV between the two fisheye cameras naturally forms a ring-shaped omnidirectional spatial stereo setup as shown in Fig. 4(c). We, therefore, obtain coverage of 65 degrees in the vertical direction and 360 degrees in horizontal directions.), the image processing method comprising: generating composite image information of a celestial sphere based on outputs of the first imaging unit and the second imaging unit (Gao: A. Omnidirectional Stereo Camera Model; The overlapping field of view between the two fisheye cameras naturally forms a ring-shaped omnidirectional spatial stereo setup, as indicated in Fig. 4(c). Using fisheye cameras with a γ FOV, we obtain FOV around (γ−180)-degree vertical and 360-degree horizontal; as per the definition of celestial sphere from Par. [0033] of the specification, celestial sphere (horizontal 360°, vertical 180°)); and generating distance information based on an output of a superimposed viewing angle of the first imaging unit and the second imaging unit (Gao: I. Introduction; For the 65-degree horizontal, and 360-degree vertical ring-shaped regions that are observed by both cameras, we will use spatial stereo matching to obtain depth values directly).
Regarding Claim 8:
Gao further teaches a non-transitory computer-readable storage medium configured to store a computer program (Gao: A. Fisheye Camera Calibration Results; We use the open source chessboard detector from OpenCV3, as shown in Fig. 5. We modify the CamOdoCal in [26] as our calibration toolbox, based on Ceres Solver4; shows use of processor to execute code) for an imaging apparatus (Gao: Abstract; novel omnidirectional stereo camera setup), the imaging apparatus including a first imaging unit including a first optical system in which a maximum half angle of view is θa, and a second imaging unit including a second optical system in which a maximum half angle of view is θb, the first optical system and the second optical system being configured such that the following expression is satisfied, and θa + θb > 180° an optical axis of the first optical system and an optical axis of the second optical system being disposed in opposite directions to each other (Gao: I. Introduction; we propose a novel configuration where two 245 degree FOV fisheye cameras are installed facing opposite directions on a rigid rod. The configuration is shown in Fig. 1(a). The overlapping FOV between the two fisheye cameras naturally forms a ring-shaped omnidirectional spatial stereo setup as shown in Fig. 4(c). We, therefore, obtain coverage of 65 degrees in the vertical direction and 360 degrees in horizontal directions.), wherein the computer program causes the imaging apparatus to execute the following processes (Gao: A. Fisheye Camera Calibration Results; We use the open source chessboard detector from OpenCV3, as shown in Fig. 5. We modify the CamOdoCal in [26] as our calibration toolbox, based on Ceres Solver4; shows use of processor to execute code): generating composite image information of a celestial sphere based on outputs of the first imaging unit and the second imaging unit (Gao: A. Omnidirectional Stereo Camera Model; The overlapping field of view between the two fisheye cameras naturally forms a ring-shaped omnidirectional spatial stereo setup, as indicated in Fig. 4(c). Using fisheye cameras with a γ FOV, we obtain FOV around (γ−180)-degree vertical and 360-degree horizontal; as per the definition of celestial sphere from Par. [0033] of the specification, celestial sphere (horizontal 360°, vertical 180°)); and generating distance information based on an output of a superimposed viewing angle of the first imaging unit and the second imaging unit (Gao: I. Introduction; For the 65-degree horizontal, and 360-degree vertical ring-shaped regions that are observed by both cameras, we will use spatial stereo matching to obtain depth values directly).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Gao et al. (NPL: Dual-fisheye omnidirectional stereo, hereafter referred as Gao) in view of Gao et al. (NPL: Autonomous aerial robot using dual‐fisheye cameras, hereafter referred as Gao 2020).
In regards to Claim 5, Gao fails to further teach the image processing system according to claim 1, wherein the one or more processors further executes the instructions to detect an object present in surroundings based on captured images of the first imaging unit and the second imaging unit and the distance information.
Gao 2020, like Gao, is directed to an image processing system, a movable apparatus, an image processing method, a storage medium, and the like. Gao 2020 does teach wherein the one or more processors further executes the instructions to detect an object present in surroundings based on captured images of the first imaging unit and the second imaging unit and the distance information (Gao 2020; 7.4.1 Indoor experiment; With the dual‐fisheye omnidirectional system, all the surrounding obstacles can be detected, even if such obstacles are parallel to or behind the quadrotor).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Gao to utilize the obstacle detection, as taught by Gao 2020, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. As taught by Gao 2020, the proposed modification allows the system to map out surroundings and find a safe path, even in messy and crowded indoor environments (Gao: 7.4.1 Indoor experiment).
Allowable Subject Matter
Claim 3 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim.
The following is a statement of reasons for the indication of allowable subject matter:
Claim 3 recites wherein, when a focal length of each of the first optical system and the second optical system is denoted as f, an image height is denoted as y, and a half angle of view is denoted as θ, a projection characteristic y(θ) satisfies the following condition. 0.2 < 2 × f × tan(θmax/2) / y(θmax) < 0.92.
Because the cited art of record, alone or in combination, does not teach or suggest each and every feature of dependent Claim 3 this claim would be allowable.
Pertinent Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Van der Auwera et al. (U.S. Patent App. Pub No. 2017/0345136 A1) teaches systems and methods for correcting the distortion present in a fisheye image, and rendering the image for display as 360-degree video.
Kawada (U.S. Patent App. Pub No. 2007/0139793 A1) teaches a technique to constitute a fisheye lens unit with a small number of lenses
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RENAE BITOR whose telephone number is (703)756-5563. The examiner can normally be reached Monday to Friday: 8:00 - 5:30 but off the 1st Friday of the biweek.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, GREG MORSE can be reached on (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RENAE A BITOR/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698
Prosecution Insights