DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-3, 5-11, 13-17 and 19-20 are pending.
Claims 4, 12 and 18 are canceled.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1, 5-9, 16 and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Keselman (US20140029836) in view of Choi et al (US20090060280).
Regarding claim 1, Keselman teaches a computer-implemented method, comprising:
generating downsampled versions of the
(Keselman, Figs. 1 and 3, “In operation, the left and right images may be down-sampled by down-sampling modules 302 to generate a reduced size and lower resolution version of the images. Down-sampling may be accomplished by averaging pixels within a region, sub-sampling pixels or any other suitable technique. In some embodiments the pixels may be down-sampled by a factor of 4”, [0017]; generating downsampled versions of the stereoscopic images)
analyzing the downsampled versions of the
(Keselman, Fig. 3, “The down-sampled images are provided to low-resolution disparity computation module 304, which generates a reduced resolution disparity matrix for the left down-sampled image comprising estimated correspondence pixels from the right down-sampled image. The disparity computation module compares regions of pixels from the left image to regions of pixels from the right image to find a best match based on a cost metric that measures the difference between the two, with a minimum cost selected for the estimated disparity”, [0018]; analyzing the downsampled images using a first image matching process to produce a disparity matrix/map)
processing non-downsampled versions of the
(Keselman, Fig. 3, High-resolution disparity computation module 310 generates high resolution disparity of the input left and right images; “High-resolution disparity computation module 310 may generate a second (or final) full resolution disparity matrix”, [0021]; “A computationally less expensive cost metric may be used in the high-resolution disparity computation”, [0022]; processing non-downsampled (full resolution) versions of the images using a second image matching computation/process)
the dense disparity map providing a set of external hints, comprising one or more disparity measurements between a current image frame and a previous image frame, for use in determining an initial search space to serve as a starting region for the second image matching process,
(Keselman, Fig. 3, "where the search range from minx to maxX is based on the first disparity estimate and the associated quality metric.", [0022]; “The estimated correspondence pixels may be selected from a search range in the right image that is based on the first full resolution disparity matrix and the full resolution quality matrix”, [0021]; “In the third example 306 c, a local variance computation may be performed on the disparity values from a previous image frame which have been stored in memory 402.", [0024]; the first disparity matrix (dense disparity map) and quality metrics comprising disparity values from a previous image frame provide external hints to define the search range (initial search space/starting region) for the second image matching process)
the second image matching process (being) guided by a search area that includes the starting region and an additional region surrounding the starting region; and
(Keselman, Fig. 3, an initial search region selected from the low resolution disparity is formed by pixels; obviously, the region can always be considered as an outer region of pixels surrounding an inner region of pixels (the inner region of pixels => “a starting region”, outer region of pixels => “an additional region surrounding the starting region”); "In this case, the full-resolution search range may center on the pixel from the first full resolution disparity matrix and extend out in either direction by k*σ", [0023]; the second image matching process is guided by a search area that centers on the starting region pixel and extends outward to include an additional surrounding region)
determining distance information for the at least one object using disparity data produced by the second image matching process.
(Keselman, Fig. 1, “a 3-D vision module 130 may generate or interpret 3-D images based on the estimated depth information provided by depth reconstruction module 110”, [0014]; “The disparity can be geometrically related to depth in the image”, [0002]; "The computed pixel disparities are geometrically related to depth in the image at those locations associated with the pixels.", [0014]; "generating a 3-D image based on depth information estimated from the second full resolution disparity matrix.", [0032]; determining depth (distance information) using the disparity data produced by the second full resolution disparity matrix)
Keselman does not expressly disclose but Choi teaches:
rectifying a pair of stereoscopic images including a representation of at least one object;
(Choi, Fig. 2; “The image pre-processing unit 204 rectifies the stereo image of the right and left images using rectification S parameters, makes epipolar lines of the stereo image coincide with each other. That is, the image pre-processing unit 204 rectifies the stereo image of the right and left images using the rectification parameters of the stereo camera so that the epipolar lines are horizontal, corrects a difference between characteristics of right and left lenses of the stereo camera and a difference between brightnesses of the right and left images, and transfers the pre-processed stereo image to the stereo matching unit 206”, [0039]; rectifying stereo images using parameters so that the epipolar lines coincide (are horizontal), which is the standard meaning of image rectification in stereo vision)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Choi into the system or method of Keselman in order to enable an image pre-processing step to rectify the stereo images making epipolar lines of the stereo image of the right and left images coincide with each other, which results in accurate stereo image matching. The combination of Keselman and Choi also teaches other enhanced capabilities.
Regarding claim 5, the combination of Keselman and Choi teaches its/their respective base claim(s).
The combination further teaches the computer-implemented method of claim 1, further comprising:
obtaining at least one additional type of hint for use in determining the initial search space, the at least one additional type of hint including a spatial hint, an internal hint, or a constant hint.
(Keselman, Fig. 4, “the quality metrics may generally provide guidance on the selection of a search size or region to be used in a subsequent higher resolution disparity computation to identify correspondence pixels”, [0019, 0023]; search size and region for full resolution stereo matching is guided by the quality metrics generated from the low resolution disparity map; "the quality metric is a local region variance (or standard deviation, σ) calculation. The variance of pixel values in a region of the right image may provide an indication of noise, texture and edge qualities in that region of the image which may impact the disparity computation and size of the high-resolution search region.", [0023]; "the matching costs generated by the low-resolution disparity computation module 304 are extracted or identified and used as the quality metric ... which may also impact the size of the high-resolution search region.", 0024]; obtaining at least one additional type of hint for use in determining the initial search space; specifically, using an internal hint (the matching costs extracted from the low-resolution disparity computation) or a spatial hint (local region variance mapping texture and edge qualities) to determine the size of the high-resolution search region)
Regarding claim 6, the combination of Keselman and Choi teaches its/their respective base claim(s).
The combination further teaches the computer-implemented method of claim 5, further comprising:
determining a shape of the initial search space based in part upon motion vectors for at least one of the types of hints.
(Choi, Fig. 6, “when the size of the macroblock to be processed in the image compressor 302 is 8 by 8, a disparity resolution of the low-resolution distance information (i.e., a coarse disparity) is obtained by dividing the width and height of an original image by eight by eight in step 614. In this regard, one disparity is acquired per macroblock so that distance information with lower resolution can be obtained”, [0061]; Fig. 8, “extracts a motion vector for the right image, and acquires the low-resolution distance information having a resolution according to the size of a macroblock to be processed in the image compressor 302 using the extracted motion vector, in step 806”, [0079]; “checks detection of an object within a distance range of the low-resolution distance information, in step 808”, [0080]; high (full) resolution distance information (or disparity) in step 812 is only acquired (set GC =0, s810) when an object is detected based on the low resolution distance (parity) information which is further based on the extracted motion vector; the detected object defines a shape of area in which determining high/full resolution distance/parity is activated)
Regarding claims 7 and 20, the combination of Keselman and Choi teaches its/their respective base claim(s).
The combination further teaches the computer-implemented method of claim 1, further comprising:
producing a confidence map corresponding to the dense disparity map and useful for determining errors in subsequent disparity determinations.
(Keselman, Fig. 3, “A quality metric may also be calculated, with increased efficiency, to measure the confidence and quality of the low-resolution disparity computation”, [0012]; “the quality metrics may generally provide guidance on the selection of a search size or region to be used in a subsequent higher resolution disparity computation to identify correspondence pixels”, [0019])
Regarding claim 8, the combination of Keselman and Choi teaches its/their respective base claim(s).
The combination further teaches the computer-implemented method of claim 1, further comprising:
determining an action to take based at least in part upon the distance information for the at least one object, the action relating to navigation of a vehicle or manipulation of a robotic device.
(Keselman, Fig. 1, “a 3-D imaging module 120 and/or a 3-D vision module 130 may generate or interpret 3-D images based on the estimated depth information provided by depth reconstruction module 110”, [0014]; “This can be useful, for example, to enhance the capabilities of computer or robotic systems with 3-D vision”, [0002]; “The estimated image depth information may also be used to enable a 3-D vision system, such as, for example a robotic 3-D vision system with improved perceptual capabilities”, [0013]; a robotic system may be controlled based on the 3D vision enabled by the image depth information)
Regarding claim 9, the combination of Keselman and Choi teaches its/their respective base claim(s).
The combination further teaches the computer-implemented method of claim 1, wherein the second image matching process performs local matching over several ranges of inputs using a set of similarity metrics and selects winning disparity values based in part upon the set of external hints and any additional hints provided as input.
(Keselman, see comments on claim 5)
Regarding claim 16, the combination of Keselman and Choi teaches a teaches a control system, comprising:
a stereoscopic camera assembly;
a control mechanism; at least one processor; and memory including instructions that, when executed by the at least one processor, cause the control system to:
rectify stereoscopic image data captured by the stereoscopic camera;
generate a downsampled version of the rectified stereoscopic image data;
analyze the downsampled version of the stereoscopic image data using a first image matching process to produce a first disparity map;
process a non-downsampled version of the rectified stereoscopic image data using a second image matching process,
the first disparity map providing a set of external hints, comprising one or more disparity measurements between a current image frame and a previous image frame, for use in determining an initial search space to serve as a starting region for the second image matching process,
the second image matching process (being) guided by a search area that includes the starting region and an additional region surrounding the starting region;
determine distance information for at least one object using a second disparity map produced by the second image matching process; and
(Keselman, Choi, see comments on claim 1)
provide at least one instruction to the control mechanism to take an action determined at least in part upon the distance information for the at least one object.
(Keselman, Fig. 1, “a 3-D imaging module 120 and/or a 3-D vision module 130 may generate or interpret 3-D images based on the estimated depth information provided by depth reconstruction module 110”, [0014]; “This can be useful, for example, to enhance the capabilities of computer or robotic systems with 3-D vision”, [0002]; “The estimated image depth information may also be used to enable a 3-D vision system, such as, for example a robotic 3-D vision system with improved perceptual capabilities”, [0013]; a robotic system may be controlled based on the 3D vision enabled by the image depth information)
Regarding claim 19, the combination of Keselman, Choi and Tsai teaches its/their respective base claim(s).
The combination further teaches the control system of claim 16, wherein the instructions when executed further cause the control system to:
obtain at least one additional type of hint for use in determining the initial search space, the at least one additional type of hint including a temporal hint, a spatial hint, an internal hint, or a constant hint; and
(Keselman, Choi, see comments on 5)
determine a shape of the initial search space based in part upon motion vectors for at least one of the types of hints.
(Choi, see comments on 6)
Claim(s) 2-3, 10-11, 13-15 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Keselman (US20140029836) in view of Choi et al (US20090060280) and further in view of Tsai et al (US20190318494).
Regarding claim 2, the combination of Keselman and Choi teaches its/their respective base claim(s).
The combination does not expressly disclose but Tsai teaches the computer-implemented method of claim 1, wherein the first image matching process is a semi-global matching (SGM) process.
(Tsai, “Semi-global matching (SGM) has been proposed as an efficient way to produce a disparity map representing disparity information of pixels at respective pixel locations in a two-dimensional map. SGM performs regional matching-cost aggregation in contrast to the global cost aggregation employed in traditional approaches. However, SGM can be computationally intensive”, [0003]; “Down-sampling may also be used to reduce computation needed to implement an SGM method”, [0020]; the SGM method may be implemented in the low-resolution disparity computation module 304 of Keselman (Fig. 3) for better computational efficiency)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Tsai into the modified system or method of Keselman and Choi in order to use SGM to produce a disparity map from the down-sampled stereo image pair for reducing computational power. The combination of Keselman, Choi and Tsai also teaches other enhanced capabilities.
Regarding claims 3 and 11, the combination of Keselman, Choi and Tsai teaches its/their respective base claim(s).
The combination further teaches the computer-implemented method of claim 2, wherein the first image matching process is performed on
(Tsai, see comments on claim 2)
The combination further teaches:
…dedicated hardware including a programmable vision accelerator (PVA)…,
(Keselman, modules 304, 306 and 308 form a “hint” generation process and may be implemented in a programmable circuitry such as an integrated circuit (=> PVA) for accelerating the computation, “A computationally less expensive cost metric may be used in the high-resolution disparity computation, providing increased speed and efficiency”, [0022], “"Circuitry", as used in any embodiment herein, may comprise, for example, …, programmable circuitry, state machine circuitry”, “A module, as used in any embodiment herein, may be embodied as circuitry. The circuitry may be embodied as an integrated circuit, such as an integrated circuit chip”, [0029])
Regarding claim 10, the combination of Keselman, Choi and Tsai teaches a system comprising: at least one processor; and memory including instructions that, when executed by the at least one processor, cause the system to:
rectify a pair of stereoscopic images including a representation of at least one object;
rectify a pair of stereoscopic images including a representation of at least one object;
process non-downsampled versions of the rectified pair of the stereoscopic images using a second image matching process,
the dense disparity map providing a set of external hints, comprising one or more disparity measurements between a current image frame and a previous image frame, for use in determining an initial search space to serve as a starting region for the second image matching process,
the second image matching process (being) guided by a search area that includes the starting region and an additional region surrounding the starting region; and
determine distance information for the at least one object using disparity data produced by the second image matching process.
(Keselman, Choi, see comments on claim 1)
The combination further teaches:
analyze the downsampled versions of the rectified pair of stereoscopic images using a semi-global matching (SGM) process to produce a dense disparity map;
(Tsai, see comments on claim 2)
Regarding claim 13, the combination of Keselman, Choi and Tsai teaches its/their respective base claim(s).
The combination further teaches the system of claim 10, wherein the instructions when executed further cause the system to:
obtain at least one additional type of hint for use in determining the initial search space, the at least one additional type of hint including a temporal hint, a spatial hint, an internal hint, or a constant hint; and
(Keselman, see comments on claim 5)
determine a shape of the initial search space based in part upon motion vectors for at least one of the types of hints.
(Choi, see comments on claim 6)
Regarding claim 14, the combination of Keselman, Choi and Tsai teaches its/their respective base claim(s).
The combination further teaches the system of claim 10, wherein the instructions when executed further cause the system to:
produce a confidence map corresponding to the dense disparity map and useful for determining errors in subsequent disparity determinations.
(Keselman, see comments on claim 7)
Regarding claim 15, the combination of Keselman, Choi and Tsai teaches its/their respective base claim(s).
The combination further teaches the system of claim 10, wherein the instructions when executed further cause the system to:
perform, as part of the second image matching process, local matching over several ranges of inputs using a set of similarity metrics and selects winning disparity values based in part upon the set of external hints and any additional hints provided as input.
(Keselman, see comments on claim 9)
Regarding claim 17, the combination of Keselman and Choi teaches its/their respective base claim(s).
The combination of Keselman, Choi and Tsai teaches the control system of claim 16, wherein the first image matching process is a semi-global matching (SGM) process performed using a programmable vision accelerator (PVA) for executing the SGM process on the downsampled versions.
(Tsai, see comments on claim 2)
Response to Arguments
Applicant's arguments filed on 1/26/2026 with respect to one or more of the pending claims have been fully considered but they are not persuasive.
Regarding claim(s) 1, 10 and 16, Applicant, in the remarks, argues that the combination of the cited reference(s) fails to teach the newly amended limitations in the claims.
The Examiner respectfully disagreed. The office action has been updated to address applicant’s argument. See the updated review comments for details.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center. for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 3/9/2026