DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 07/15/2024 has/have been considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Camus (US 20050131646 A1) in view of Saleemi et al (US 10891741 B2, hereinafter Saleemi, and further in view of Klaus (US 20190087635 A1).
-Regrading claim 1, Camus discloses a method comprising (Abstract; FIGS. 1-4): converting frames corresponding to sensor data to aligned images (FIG. 2; [0021], “preprocessor 206 calibrates the stereo cameras, captures and digitizes imagery, warps the images into alignment”; [0025], “imagery generated from each of the cameras is warped into alignment to facilitate producing disparity images”; [0026], “disparity images are created for each pair of frames”); generating a disparity image having disparity values corresponding to the difference values and indicating the differences between the pixel values across the aligned pixels ([0022], “Each of the disparity images contains the point-wise motion”; [0026], “The disparity image comprises, in addition to the disparity information, an indication of which of the disparity pixels in the image are deemed valid or invalid. Certain disparity values may be deemed invalid because of image contrast anomalies”); detecting one or more objects in an environment using the disparity image (FIGS. 2-3; [0023]; [0027]; FIG. 4, steps 410-415) and performing one or more operations for a machine based at least on the detecting of the one or more objects (FIG. 3; FIG. 4, steps 420-430).
Camus does not disclose converting image frames into a common plane to generate disparity images.
In the same field of endeavor, Saleemi teaches a method or detecting, tracking and counting objects of interest in video frames (Saleemi: Abstract; FIGS. 1-11). Saleemi further teaches converting image frames into a common plane to generate disparity images (Saleemi: FIG. 2A; Col. 4, lines 2-5, “projects the stereo images onto a common image plane”; FIG. 7A, steps 702-704; Col. 9, lines 1-5). Saleemi also teaches generating a disparity image having disparity values corresponding to the difference values and indicating the differences between the pixel values (Saleemi: FIG.2A; Col. 4, lines 20-26, “disparity module 206 scans … for matching image features. … disparity refers to the difference in coordinates of similar features within two stereo images … the correspondence … can be determined by forming a small image patch around every pixel …”; Col. 6, line 5, “observed disparity per pixel”); detecting one or more objects in an environment using the disparity image and performing one or more operations for a machine based at least on the detecting of the one or more objects (Saleemi: FIGS. 2A, 7A).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Camus with the teaching of Saleemi by converting image frames into a common plane to generate disparity images in order to simplify matching points (correspondence) between the images (Saleemi: Col. 4, lines 4-5).
Camus in view of Saleemi does not teach blending pixel values of aligned pixels between the aligned images to compute difference values indicating differences between the pixel values across the aligned pixels. However, a person of ordinary skills in the art would understand that it a common practice to calculate disparity image by subtracting the aligned images from one another.
However, Klaus is an analogous art pertinent to the problem to be solved in this application and teaches an object detection and avoidance method for aerial vehicles (Klaus: Abstract; FIGS. 1A-7B). Klaus further teaches blending pixel values of aligned pixels between the aligned images to compute difference values indicating differences between the pixel values across the aligned pixels (Klaus: FIGS, 1F, 1H; FIG. 3, steps 320, 335; FIGS. 4-5; [0021], “aligned”; [0027], “disparities between the pixels appearing within the images 145-1, 145-2 may also be used to calculate a difference image 125. The difference image 125 may be calculated by subtracting absolute intensities of the images 145-1, 145-2, or by deriving intensity gradients for the images 145-1, 145-2 and comparing the intensity gradients to one another”). Klaus also teaches generating a disparity image having disparity values corresponding to the difference values and indicating the differences between the pixel values (Klaus: [0027]; [0076]).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teaching of Camus in view of Saleemi with the teaching of Klaus by blending pixel values of aligned pixels generating difference image in order to extract change areas from the difference image to indicate the presence of one or more objects in region of interests for object detection (Klaus: [0013]; [0026]).
-Regarding claim 2, Camus in view of Saleemi, and further in view of Klaus teaches the method of claim 1. The modification further teaches wherein the detecting of the one or more objects includes determining one or more bounding shapes for one or more one or more groups of pixels of the disparity image, and the one or more operations are performed based at least on the one or more bounding shapes (Camus: FIGS. 2-4; [0037], “returns “bounding boxes””; [0038]; [0041]; [0042]).
-Regarding claim 3, Camus in view of Saleemi, and further in view of Klaus teaches the method of claim 1.
Camus does not disclose wherein the converting includes rectifying at least one of the frames to the common image plane.
In the same field of endeavor, Saleemi teaches a method or detecting, tracking and counting objects of interest in video frames (Saleemi: Abstract; FIGS. 1-11). Saleemi further teaches wherein the converting includes rectifying at least one of the frames to the common image plane (Saleemi: FIG. 2A, image rectification module 204, disparity module 206; Col. 4, lines 2-5, “projects the stereo images onto a common image plane”; FIG. 7A, steps 702-704; Col. 9, lines 1-5).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Camus with the teaching of Saleemi by converting image frames into a common plane to generate disparity images in order to simplify matching points (correspondence) between the images (Saleemi: Col. 4, lines 4-5).
Camus in view of Saleemi does not teach blending pixel values of aligned pixels between the aligned images to compute difference values indicating differences between the pixel values across the aligned pixels. However, a person of ordinary skills in the art would understand that it a common practice to calculate disparity image by subtracting the aligned images from one another.
-Regarding claim 4, Camus in view of Saleemi, and further in view of Klaus teaches the method of claim 1. The modification further teaches wherein the detecting the one or more objects includes combining pixels of the disparity image into one or more groups of the pixels based at least on similarities between the disparity values within the one or more groups, and the one or more objects correspond to the one or more groups (Camus: [0015], “The disparity images … processed … a list of possible objects …. corresponding to all nearby high correlation scores …”; [0030], “similarity metric”; [0031], “matching”; [0034], “a measure of similarity”; See also Saleemi: FIG. 9, step 904; Col. 11, lines 25-28; FIG. 11; Col. 12, lines 47-51).
-Regarding claim 6, Camus in view of Saleemi, and further in view of Klaus teaches the method of claim 1.
Camus in view of Saleemi does not teach wherein the blending includes subtracting the aligned images from one another to produce a difference image that corresponds to the disparity image. However, a person of ordinary skills in the art would understand that it a common practice to calculate disparity image by subtracting the aligned images from one another.
However, Klaus is an analogous art pertinent to the problem to be solved in this application and teaches an object detection and avoidance method for aerial vehicles (Klaus: Abstract; FIGS. 1A-7B). Klaus further teaches wherein the blending includes subtracting the aligned images from one another to produce a difference image that corresponds to the disparity image (Klaus: FIGS, 1F, 1H; FIG. 3, steps 320, 335; FIGS. 4-5; [0021], “aligned”; [0027], “disparities between the pixels appearing within the images 145-1, 145-2 may also be used to calculate a difference image 125. The difference image 125 may be calculated by subtracting absolute intensities of the images 145-1, 145-2, or by deriving intensity gradients for the images 145-1, 145-2 and comparing the intensity gradients to one another”).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teaching of Camus in view of Saleemi with the teaching of Klaus by blending pixel values of aligned pixels generating difference image in order to extract change areas from the difference image to indicate the presence of one or more objects in region of interests for object detection (Klaus: [0013]; [0026]).
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Camus (US 20050131646 A1) in view of Saleemi et al (US 10891741 B2, hereinafter Saleemi, and further in view of Klaus (US 20190087635 A1), in view of Liu et al (2014 ICSP), hereinafter Liu.
-Regarding claim 5, Camus in view of Saleemi, and further in view of Klaus teaches the method of claim 1.
Camus in view of Saleemi, and further in view of Klaus does not teach wherein the frames are captured using a single camera.
However, Liu is an analogous art pertinent to the problem to be solved in this application and teaches an object detection method based on disparity (Liu: Abstract; FIGS. 1-4). Liu further teaches wherein the frames are captured using a single camera (Liu: p. 770, 1st Col., 1st paragraph, “motion detection and object extraction method based on disparity or displacement between frames, which applied single camera without calibration”; FIGS. 1-3).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teaching of Camus in view of Saleemi, and further in view of Klaus with the teaching of Liu by using a single camera to capture frames in order to perform motion detection and object detection without using complex calibration of the camera (Liu, p. 770, 1st Col., 1st paragraph).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Camus (US 20050131646 A1) in view of Saleemi et al (US 10891741 B2, hereinafter Saleemi, and further in view of Klaus (US 20190087635 A1), in view of Shu et al (2009 First International Conference on Information Science and Engineering, pp. 1203-1206)., hereinafter Shu.
-Regarding claim 8, Camus in view of Saleemi, and further in view of Klaus teaches the method of claim 1. The modification further teaches wherein the disparity image includes a detection map in which first sets of the disparity values having the differences determined to be greater than a threshold value are encoded with a first value and second sets of the disparity values having the differences determined to be less than the threshold value are encoded with a second value (Camus: [0039]-[0040]; [0043]; See also Klaus: Abstract; [0027]; [0076]; FIG. 3).
Camus in view of Saleemi, and further in view of Klaus does not teach wherein the disparity image includes a binary image.
However, Shu is an analogous art pertinent to the problem to be solved in this application and teaches a method to build dense disparity maps with high accuracy (Shu: Abstract; Figs. 1-8). Shu further teaches wherein the disparity image includes a binary image (Shu: Page 1, 2nd Col., 1st paragraph).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teaching of Camus in view of Saleemi, and further in view of Klaus with the teaching of Shu by using the disparity image which includes a binary image in order to solve the problems of weak texture regions and disparity maps unsmooth.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Camus (US 20050131646 A1) in view of Saleemi et al (US 10891741 B2, hereinafter Saleemi, and further in view of Klaus (US 20190087635 A1), in view of Somanath et al (US 20160188995 A1), hereinafter Somanath.
-Regrading claim 9, Camus in view of Saleemi, and further in view of Klaus teaches the method of claim 1. The modification further teaches determining geometry corresponding to one or more first objects in an environment depicted in the frames of sensor data (Klaus: [0017], “capture one or more images of surfaces within a vicinity … recognize one or more edges, contours, outlines, colors, textures, silhouettes, shapes or other characteristics of the target marker 160”; FIGS. 1A-1I, 3); estimating, using the geometry, one or more features in one or more frames of the frames of sensor data (Klaus: FIGS. 1C, 1E; [0016], “search for obstacles at the landing area when the aerial vehicle 110 reaches a predetermined altitude threshold … evaluate the landing area”; [0019], “define a landing area 165 upon detecting the target marker 160 at the destination 170”; [0023]); matching the one or more features across the frames to determine one or more matched features (Klaus: FIG. 1E. 1G, 1H, 3, 6; [0026], “matching algorithm … construct … landing area 165”; [0027]; [0029]), wherein the converting the frames to the aligned images is based on the one or more matched features (Klaus: FIG.1D, 4-5; [0021], “imaging devices 140-1, 140-2 may be aligned with fields of view that include the target marker 160 and the landing area 165, and overlap at least in part, and may be configured to capture images at any frame rate”).
Camus in view of Saleemi, and further in view of Klaus does not teach estimating a scale of one or more features.
However, Somanath is an analogous art pertinent to the problem to be solved in this application and teaches a method for estimating a disparity map or depth map for the scene that provides values of the disparity in point location from image to image (Somanath: FIGS. 1-11). Somanath further teaches estimating a scale of one or more features (Somanath: FIG. 4A, step 410; [0059]-[0063]; equations (8)-(9)).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teaching of Camus in view of Saleemi, and further in view of Klaus with the teaching of Somanath by scaling region of interest (ROI) in order to determine more accurate disparities of the images.
Claim(s) 10-12, 14-18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Klaus (US 20190087635 A1) in view of Somanath et al (US 20160188995 A1), hereinafter Somanath.
-Regarding claim 10, Klaus discloses a system comprising: one or more processors (FIG. 1I, processor 112; FIG. 2, processor 212) to perform operations including (Abstract; FIGS.1A-7B): determining geometry corresponding to one or more first objects in an environment depicted in frames associated with sensor data ([0017], “capture one or more images of surfaces within a vicinity … recognize one or more edges, contours, outlines, colors, textures, silhouettes, shapes or other characteristics of the target marker 160”; FIGS. 1A-1I, 3); estimating, using the geometry, one or more features in one or more frames of the frames of sensor data (FIGS. 1C, 1E; [0016], “search for obstacles at the landing area when the aerial vehicle 110 reaches a predetermined altitude threshold … evaluate the landing area”; [0019], “define a landing area 165 upon detecting the target marker 160 at the destination 170”; [0023]); matching the one or more features across the frames to determine one or more matched features (Klaus: FIG. 1E. 1G, 1H, 3, 6; [0026], “matching algorithm … construct … landing area 165”; [0027]; [0029]); detecting one or more second objects in the environment using the one or more matched features (FIGS. 1E, G, H, 1I; FIG. 3, steps 360-370; [0014], “A plurality of obstacles (or obstructions) … within a vicinity of the target marker 160”; [0027]); and performing one or more operations corresponding to a machine based at least on the detecting of the one or more second objects (FIG. 1I; FIG. 3, steps 360-370; [0028]-[0029]; [0078]).
Klaus does not disclose estimating a scale of one or more features.
In the same field of endeavor, Somanath teaches a method for estimating a disparity map or depth map for the scene that provides values of the disparity in point location from image to image (Somanath: FIGS. 1-11). Somanath further teaches estimating a scale of one or more features (Somanath: FIG. 4A, step 410; [0059]-[0063]; equations (8)-(9)).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Klaus with the teaching of Somanath by scaling region of interest (ROI) in order to determine more accurate disparities of the images.
-Regarding claim 16, Klaus discloses at least one processor comprising (Abstract; FIGS. 1A-7B): processing circuitry to perform one or more operations for a machine based at least on detecting one or more objects using one or more matched features across frames associated with sensor data ([0017], “capture one or more images of surfaces within a vicinity … recognize one or more edges, contours … shapes or other characteristics of the target marker 160”; [0019], “landing area 165”; [0021], “aligned with fields of view that include the target marker 160 and the landing area 165, and overlap at least in part, and may be configured to capture images at any frame rate …”; [0026]-[0027]; FIGS. 1A-1I, 3-5), the one or more matched features is estimated based at least on analyzing the sensor data (FIGS. 1A-1I, 3-5).
Klaus does not disclose scaling one or more features.
In the same field of endeavor, Somanath teaches a method for estimating a disparity map or depth map for the scene that provides values of the disparity in point location from image to image (Somanath: FIGS. 1-11). Somanath further teaches estimating a scale of one or more features (Somanath: FIG. 4A, step 410; [0059]-[0063]; equations (8)-(9)).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Klaus with the teaching of Somanath by scaling region of interest (ROI) in order to determine more accurate disparities of the images.
-Regarding claim 11, Klaus in view of Somanath teaches the system of claim 10. The combination further teaches wherein the one or more first objects correspond to one or more lanes in the environment or one or more freespace regions in the environment (FIGS. 1A-1I, 3-5).
-Regarding claims 12 and 18, Klaus in view of Somanath teaches the system of claim 10, and the processor of claim 16. The combination teaches wherein the matching uses the one or more instances extracted from the one or more ROIs (Klaus: FIGS. 1A-1I, 3-5; [0032]; [0084]; [0087]-[0088]; See also Somanath: [0025]).
Klaus does not disclose adjusting, based at least on the scale, one or more dimensions of one or more regions of interest (ROIs) in the one or more frames; and based at least on the adjusting, extracting one or more instances of the one or more features from the one or more ROIs.
In the same field of endeavor, Somanath teaches a method for estimating a disparity map or depth map for the scene that provides values of the disparity in point location from image to image (Somanath: FIGS. 1-11). Somanath further teaches adjusting, based at least on the scale, one or more dimensions of one or more regions of interest (ROIs) in the one or more frames; and based at least on the adjusting, extracting one or more instances of the one or more features from the one or more ROIs (Somanath: FIGS. 4A-4B, step 410; [0059]-[0063]; equations (8)-(9)).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Klaus with the teaching of Somanath by scaling region of interest (ROI) in order to determine more accurate disparities of the images.
-Regarding claim 14, Klaus in view of Somanath teaches the system of claim 10.
The combination further teaches converting the frames to aligned images based on the one or more matched features (Klaus: FIGS. 1D, 4; [0021]; [0043]; [0084]); blending pixel values of aligned pixels between the aligned images to compute difference values indicating differences between the pixel values across the aligned pixels Klaus: FIGS, 1F, 1H; FIG. 3, steps 320, 335; FIGS. 4-5; [0021], “aligned”; [0027], “disparities between the pixels appearing within the images 145-1, 145-2 may also be used to calculate a difference image 125. The difference image 125 may be calculated by subtracting absolute intensities of the images 145-1, 145-2, or by deriving intensity gradients for the images 145-1, 145-2 and comparing the intensity gradients to one another”); generating a disparity image having disparity values corresponding to the difference values and indicating the differences between the pixel values across the aligned pixels (Klaus: [0027]; [0076]); and determining one or more bounding shapes for one or more one or more groups of the pixels of the disparity image (Klaus: FIGS. 1A-1I).
-Regarding claims 15 and 20, Klaus in view of Somanath teaches the system of claim 10, and he processor of claim 16. The combination further teaches wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (Klaus: FIGS. 1A-3; [0012]).
Regarding claim 17, Klaus in view of Somanath teaches the processor of claim 16.
Klaus discloses wherein the target marker and landing area (or regions of interest or ROIs) are estimated based at least on one or more of: at least one road profile associated with the one or more features; at least one lane geometry associated with the one or more features; or ego-motion associated with the one or more features (Klaus: FIGS. 1A-5).
Klaus does not disclose scaling one or more features such as regions of interest (ROIs).
In the same field of endeavor, Somanath teaches a method for estimating a disparity map or depth map for the scene that provides values of the disparity in point location from image to image (Somanath: FIGS. 1-11). Somanath further teaches estimating a scale of one or more features (Somanath: FIG. 4A, step 410; [0059]-[0063]; equations (8)-(9)).
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to combine the teaching of Klaus with the teaching of Somanath by scaling region of interest (ROI) in order to determine more accurate disparities of the images.
Claim(s) 13 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Klaus (US 20190087635 A1) in view of Somanath et al (US 20160188995 A1), hereinafter Somanath, and further in view of Liu et al (2014 ICSP), hereinafter Liu.
-Regarding claim 13, Klaus in view of Somanath teaches the system of claim 10, and he processor of claim 16.
Klaus in view of Somanath does not teach wherein the machine is closer to the one or more second objects for a first frame of the frames than for a second frame of the frames.
However, Liu is an analogous art pertinent to the problem to be solved in this application and teaches an object detection method based on disparity wherein the frames are captured using a single camera (Liu: Abstract; FIGS. 1-4; p. 770, 1st Col., 1st paragraph, “motion detection and object extraction method based on disparity or displacement between frames, which applied single camera without calibration”; FIGS. 1-3). Liu future teaches the machine is closer to the one or more second objects for a first frame of the frames than for a second frame of the frames (Liu: p.771, 1st Col., last paragraph, 2nd Col., Sec. 3; FIGS. 1-3; Liu teaches calculating a region disparity between neighborhood frames with the same size as matching blocks for motion detection and object extraction; A person of ordinary skills in the art would understand that due to the motion, one or more objects in one frame will be closer that the, one or more objects in other frame, see FIGS. 1-3. Thus, the regions in the frames must be scaled to achieve same size as matching blocks for disparities computation )
Therefore, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify the teaching of Klaus in view of Somanath with the teaching of Liu by using a single camera to capture frames in order to perform motion detection and object detection without using complex calibration of the camera (Liu, p. 770, 1st Col., 1st paragraph).
Allowable Subject Matter
Claim 7 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIAO LIU whose telephone number is (571)272-4539. The examiner can normally be reached Monday-Thursday and Alternate Fridays 8:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Mehmood can be reached at (571) 272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XIAO LIU/Primary Examiner, Art Unit 2664