DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 02/12/2026 has been considered by the examiner and placed in the Applicant’s file.
Response to Arguments
Applicant’s arguments, filed 02/02/2026, with respect to claims 1-11, have been fully considered but are moot because the arguments do not apply to the current references and current combinations of references being used in the current rejection.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3 and 6-11 are rejected under 35 U.S.C. 103 as being unpatentable over PRICE et al. (US 20190058859 A1), hereinafter referenced as PRICE in view of AMANO et al. (US 20200074212 A1), hereinafter referenced as AMANO.
Regarding claim 1, PRICE explicitly teaches an image processing device (Fig. 1-3 and 12, #100, #200, #300 and #1200 called a system, a head-mounted device and a computer system. Paragraph [0021, 0056, 0060, 0076]. Further at [0076]-PRICE discloses FIG. 12 illustrates a block diagram showing an example computer system 1200 upon which aspects of this disclosure may be implemented. Further in paragraph [0060]-PRICE discloses the techniques described herein may also be applied to other sensing applications such as automotive or vehicular sensors, object scanning sensors, and sensors placed on UAVs (unmanned air vehicles) or other aerial vehicles. Please also see Fig. 1-3) comprising:
at least one memory (Fig. 1-3 and 12, #162, #314, #1206 and #1208, called memory, main memory and read-only memory. Paragraph [0033, 0038, 0066, 0076-0079]) configured to store instructions, and at least one processor (Fig. 1-3 and 12, #220 and #1204, called a controller and a processor. Paragraph [0056, 0066, 0076]) configured to execute the instructions (Fig. 12. Paragraph [0076]-PRICE discloses computer system 1200 includes a main memory 1206, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1202 for storing information and instructions to be executed by processor 1204. In paragraph [0077]-PRICE discloses computer system 1200 can further include a read only memory (ROM) 1208 or other static storage device coupled to bus 1202 for storing static information and instructions for processor 1204. A storage device 1210, such as a flash or other non-volatile memory can be coupled to bus 1202 for storing information and instructions. Please also see Fig. 1-3 and read paragraph [0056 and 0066]) to:
recognize, among multiple different area classes designated regarding photographic subjects, to which area class a photographic subject of each pixel in a captured image (Fig. 1, #142 called a frame. Paragraph [0040]) that has been acquired belongs (Fig. 1. Paragraph [0035]-PRICE discloses the object detector 154 is configured to receive the pixel data 153 for the frame 142 and to identify one or more subsets of the pixels included in the pixel data 153, each subset being associated with a respective physical object detected based on the pixel data 153. Each physical object detected based on the pixel data 153 may be referred to as an “object instance.” Detection of an object instance may include classification of the object, in which one or more classes or types of physical objects are identified for the object instance. A detected object instance may have one or more object classifications with respective confidence scores that the object instance actually corresponds to a respective object class. The object detector 154 may be configured to use one or more threshold scores to determine whether a detected object instance will be processed as an object instance of a particular object class, based on whether a confidence score for the object class is greater than or equal to a respective threshold score. Separate threshold scores may be used for respective object classes or groups of object classes. One or more of the threshold scores may be dynamically set and/or changed according to various conditions detected by the system 100); and
Although PRICE explicitly teaches acquire, from depth map information (Fig. 1. Paragraph [0048]-PRICE discloses the localized depth map calculation module 160 generates, for the current frame 142 (or the frame period for the frame 142), a localized depth map 164 (which may be referred to as “first localized depth map 164”) corresponding to a first object instance detected in pixel data 153 and a second localized depth map 166 corresponding to a second object instance detected in pixel data 153) corresponding to the captured image (Fig. 1, #140, #142, #144 and #146, called frame(s). Paragraph [0025, 0032-0033]), range information of each pixel in an estimated vehicle area (Fig. 1. Paragraph [0040]-PRICE discloses the localized depth map generator 150 includes a region of interest identification module 156. The region of interest identification module 156 is configured to select a respective region of the current frame 142 (or an FOV of the frame 142) corresponding to each subset of pixels identified by the object detector 154 for a respective object instance. A region selected by the region of interest identification module 156 for an object instance may be referred to as an ROI for the object instance. An ROI for an object instance may be used to selectively identify respective pixels included in pixel data 153 and/or pixel data 143 for processing performed by resizing module 158 and/or localized depth map calculation module 160. An ROI may be a rectangular bounding box or at a pixel-level. Please also read paragraph [0029-0030 and 0062]) in the captured image (Fig. 1, #140, #142, #144 and #146, called frame(s). Paragraph [0025, 0032-0033 and 0046]) representing a vehicle area class among the multiple different area classes (Fig. 1. Paragraph [0035]-PRICE discloses the object detector 154 is configured to receive the pixel data 153 for the frame 142 and to identify one or more subsets of the pixels included in the pixel data 153, each subset being associated with a respective physical object detected based on the pixel data 153. Each physical object detected based on the pixel data 153 may be referred to as an “object instance.” Detection of an object instance may include classification of the object, in which one or more classes or types of physical objects are identified for the object instance. In paragraph [0021]-PRICE discloses examples of physical objects 112 and 114 include, but are not limited to, people, animals and vehicles), determine a position at which the range information is discontinuous to be a boundary (Fig. 1. Paragraph [0041]-PRICE discloses by making the ROI slightly larger than the subset of pixels identified by object localization or segmentation, depth discontinuities around an object instance can be included in a localized depth map, so sufficient contrast can be determined between the object instance and background portions of the scene 110 in subsequent processing of the localized depth map (wherein a localized depth map shows depth discontinuities). In paragraph [0046]-PRICE discloses the localized depth map calculation module 160 is configured to selectively generate a localized depth map for each of the object instances identified for the current frame 142 (or the frame period for the current frame 142). A depth map may also be referred to as a “depth image.” A localized depth map for an object instance is calculated within its respective ROI (wherein a localized depth map determines a depth and depth discontinuities for each pixel within a vehicle object instance)).
PRICE is silent on determine a position at which the range information is discontinuous, corresponding to adjacent pixels in which a range difference between the adjacent pixels is greater than a predetermined value among a plurality of pixels included in the estimated vehicle area, to be a boundary between different vehicles.
However, AMANO explicitly teaches determine a position at which the range information is discontinuous (Fig. 3. Paragraph [0082]-AMANO discloses the object recognition device 1 has an imaging function for imaging a traveling direction of the vehicle 70 (wherein the object recognition device performs basic detection processing, separation detection processing, and integration detection processing in any order or in parallel, and the generation unit 500 generates a V-Disparity map, a U-Disparity map, a Real U-Disparity map). In paragraph [0113]-AMANO discloses the V-Disparity map is an example of “information in which a position in the vertical direction is associated with a position in the depth direction”. The U-Disparity map and the Real U-Disparity map are examples of “information in which a position in the horizontal direction is associated with a position in the depth direction. Please also read paragraph [0127-0129]), corresponding to adjacent pixels in which a range difference between the adjacent pixels is greater than a predetermined value among a plurality of pixels included in the estimated vehicle area (Fig. 3. Paragraph [0147]-AMANO discloses at Step S201, the basic detection unit 511 performs 8-neighbor labeling processing for giving the same ID to pixels that are continuous in a vertical, horizontal, or oblique direction for a parallax point as a pixel having a pixel value (frequency of the parallax) equal to or larger than a predetermined value in the real Umap RM. Further in paragraph [0154]-AMANO discloses at Step S301, the integration detection unit 513 performs 4-neighbor labeling processing for giving the same ID to pixels (parallax points) that are continuous in the vertical direction (depth direction) or the lateral direction (horizontal direction) on the small real Umap. Please also read paragraph [0138-0140, 0152]), to be a boundary between different vehicles (Fig. 3. Paragraph [0148]-AMANO discloses basic detection unit 511 sets a rectangle circumscribing each pixel group (each isolated region) to which the same ID is given (Step S202). In paragraph [0155]-AMANO discloses the integration detection unit 513 sets a rectangle circumscribing each pixel group (each isolated region) to which the same ID is given (Step S302). In paragraph [0156]-AMANO discloses the integration detection unit 513 extracts the object such as a vehicle (Step S303). The integration detection unit 513 extracts the region of the object such as a vehicle based on the width, the depth, frequency of the parallax, and the like of each isolated region. Accordingly, the rectangle circumscribing each isolated region is detected as the object region. Please also read paragraph [0158-0165]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of PRICE of having an image processing device comprising: recognize, among multiple different area classes designated regarding photographic subjects, to which area class a photographic subject of each pixel in a captured image that has been acquired belongs, with the teachings of AMANO of having acquire, from depth map information corresponding to the captured image, range information of each pixel in an estimated vehicle area in the captured image representing a vehicle area class among the multiple different area classes, and determine a position at which the range information is discontinuous, corresponding to adjacent pixels in which a range difference between the adjacent pixels is greater than a predetermined value among a plurality of pixels included in the estimated vehicle area, to be a boundary between different vehicles.
Wherein PRICE’s device having acquire, from depth map information corresponding to the captured image, range information of each pixel in an estimated vehicle area in the captured image representing a vehicle area class among the multiple different area classes, and determine a position at which the range information is discontinuous, corresponding to adjacent pixels in which a range difference between the adjacent pixels is greater than a predetermined value among a plurality of pixels included in the estimated vehicle area, to be a boundary between different vehicles.
The motivation behind the modification would have been to obtain a device that improves the performance of object detection and/or depth estimation, since both PRICE and AMANO concern object detection and depth estimation. Wherein PRICE provides systems and methods that improve the performance of object detection and/or depth estimation, while AMANO provides systems and methods improves the performance and accuracy of object recognition when there are multiple potentially device while also improving the safety and reliability of hazard detection. Please see PRICE et al. (US 20190058859 A1), Abstract and Paragraph [0001 and 0030] and AMANO et al. (US 20200074212 A1), Abstract and Paragraph [188-0189].
Regarding claim 2, PRICE in view of AMANO explicitly teaches the image processing device according to claim 1, PRICE further teaches wherein the at least one processor (Fig. 1-3 and 12, #220 and #1204, called a controller and a processor. Paragraph [0056, 0066, 0076]) is further configured to execute the instructions (Fig. 12. Paragraph [0076]-PRICE discloses computer system 1200 includes a main memory 1206, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1202 for storing information and instructions to be executed by processor 1204. In paragraph [0077]-PRICE discloses computer system 1200 can further include a read only memory (ROM) 1208 or other static storage device coupled to bus 1202 for storing static information and instructions for processor 1204. A storage device 1210, such as a flash or other non-volatile memory can be coupled to bus 1202 for storing information and instructions. Please also see Fig. 1-3 and read paragraph [0056 and 0066]) to generate, based on the captured image (Fig. 1, #140, #142, #144 and #146, called frame(s). Paragraph [0025 and 0032-0033]) acquired from the image capture device (Fig. 1-2, #120 and #130 called a first camera and a second camera, respectively. Paragraph [0032]), the depth map information retaining, for each pixel in the captured image, range information from the image capture device to photographic subjects (Fig. 11. Paragraph [0075]-PRICE discloses at a first step 1110, camera measurements are obtained by a camera to produce a frame, much as frame 142 is produced by the first camera 120 in FIG. 1. At a second step 1120, a global depth map corresponding to most or all of the frame produced at step 1110, which involves generating a substantial number of depth estimate values for the frame) appearing in the captured image (Fig. 1, #140, #142, #144 and #146, called frame(s). Paragraph [0025 and 0032-0033]).
Regarding claim 3, PRICE in view of AMANO explicitly teaches the image processing device according to claim 1, PRICE further teaches wherein the at least one processor (Fig. 1-3 and 12, #220 and #1204, called a controller and a processor. Paragraph [0056, 0066, 0076]) is configured to execute the instructions (Fig. 12. Paragraph [0076]-PRICE discloses computer system 1200 includes a main memory 1206, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1202 for storing information and instructions to be executed by processor 1204. In paragraph [0077]-PRICE discloses computer system 1200 can further include a read only memory (ROM) 1208 or other static storage device coupled to bus 1202 for storing static information and instructions for processor 1204. A storage device 1210, such as a flash or other non-volatile memory can be coupled to bus 1202 for storing information and instructions. Please also see Fig. 1-3 and read paragraph [0056 and 0066]) to:
identify a single-vehicle area (Fig. 1. Paragraph [0035]-PRICE discloses the object detector 154 is configured to receive the pixel data 153 for the frame 142 and to identify one or more subsets of the pixels included in the pixel data 153, each subset being associated with a respective physical object detected based on the pixel data 153. Each physical object detected based on the pixel data 153 may be referred to as an “object instance.” Detection of an object instance may include classification of the object, in which one or more classes or types of physical objects are identified for the object instance. In paragraph [0021]-PRICE discloses examples of physical objects 112 and 114 include, but are not limited to, people, animals and vehicles) based on the discontinuous position and the range information (Fig. 1. Paragraph [0040]-PRICE discloses the localized depth map generator 150 includes a region of interest identification module 156. The region of interest identification module 156 is configured to select a respective region of the current frame 142 (or an FOV of the frame 142) corresponding to each subset of pixels identified by the object detector 154 for a respective object instance. A region selected by the region of interest identification module 156 for an object instance may be referred to as an ROI for the object instance. An ROI for an object instance may be used to selectively identify respective pixels included in pixel data 153 and/or pixel data 143 for processing performed by resizing module 158 and/or localized depth map calculation module 160. An ROI may be a rectangular bounding box or at a pixel-level. Please also read paragraph [0029-0030 and 0062]) indicated by each pixel in the estimate vehicle area representing the vehicle area class (Fig. 1. Paragraph [0041]-PRICE discloses by making the ROI slightly larger than the subset of pixels identified by object localization or segmentation, depth discontinuities around an object instance can be included in a localized depth map, so sufficient contrast can be determined between the object instance and background portions of the scene 110 in subsequent processing of the localized depth map (wherein a localized depth map shows depth discontinuities). In paragraph [0046]-PRICE discloses the localized depth map calculation module 160 is configured to selectively generate a localized depth map for each of the object instances identified for the current frame 142 (or the frame period for the current frame 142). A depth map may also be referred to as a “depth image.” A localized depth map for an object instance is calculated within its respective ROI (wherein the localized depth maps determine depth and depth discontinuities for each pixel within vehicle object instances. Please also read paragraph [0040, 0042-0043])).
Regarding claim 6, PRICE explicitly teaches an image processing method (Fig. 1. Paragraph [0021]-PRICE discloses FIG. 1 is a schematic diagram illustrating features included in an example system 100 arranged to generate localized depth maps) that comprises:
recognizing, among multiple different area classes designated regarding photographic subjects, to which area class a photographic subject of each pixel in a captured image that has been acquired belongs (Fig. 1. Paragraph [0035]-PRICE discloses the object detector 154 is configured to receive the pixel data 153 for the frame 142 and to identify one or more subsets of the pixels included in the pixel data 153, each subset being associated with a respective physical object detected based on the pixel data 153. Each physical object detected based on the pixel data 153 may be referred to as an “object instance.” Detection of an object instance may include classification of the object, in which one or more classes or types of physical objects are identified for the object instance. A detected object instance may have one or more object classifications with respective confidence scores that the object instance actually corresponds to a respective object class. The object detector 154 may be configured to use one or more threshold scores to determine whether a detected object instance will be processed as an object instance of a particular object class, based on whether a confidence score for the object class is greater than or equal to a respective threshold score. Separate threshold scores may be used for respective object classes or groups of object classes. One or more of the threshold scores may be dynamically set and/or changed according to various conditions detected by the system 100); and
Although PRICE explicitly teaches acquiring, from depth map information (Fig. 1. Paragraph [0048]-PRICE discloses the localized depth map calculation module 160 generates, for the current frame 142 (or the frame period for the frame 142), a localized depth map 164 (which may be referred to as “first localized depth map 164”) corresponding to a first object instance detected in pixel data 153 and a second localized depth map 166 corresponding to a second object instance detected in pixel data 153) corresponding to the captured image (Fig. 1, #140, #142, #144 and #146, called frame(s). Paragraph [0025, 0032-0033]), range information of each pixel (Fig. 1. Paragraph [0040]-PRICE discloses the localized depth map generator 150 includes a region of interest identification module 156. The region of interest identification module 156 is configured to select a respective region of the current frame 142 (or an FOV of the frame 142) corresponding to each subset of pixels identified by the object detector 154 for a respective object instance. A region selected by the region of interest identification module 156 for an object instance may be referred to as an ROI for the object instance. An ROI for an object instance may be used to selectively identify respective pixels included in pixel data 153 and/or pixel data 143 for processing performed by resizing module 158 and/or localized depth map calculation module 160. An ROI may be a rectangular bounding box or at a pixel-level. Please also read paragraph [0029-0030, 0046 and 0062]) in an estimated vehicle area in the captured image representing a vehicle area class among the multiple different area classes (Fig. 1. Paragraph [0035]-PRICE discloses the object detector 154 is configured to receive the pixel data 153 for the frame 142 and to identify one or more subsets of the pixels included in the pixel data 153, each subset being associated with a respective physical object detected based on the pixel data 153. Each physical object detected based on the pixel data 153 may be referred to as an “object instance.” Detection of an object instance may include classification of the object, in which one or more classes or types of physical objects are identified for the object instance. In paragraph [0021]-PRICE discloses examples of physical objects 112 and 114 include, but are not limited to, people, animals and vehicles), and determining a position at which the range information is discontinuous to be a boundary (Fig. 1. Paragraph [0041]-PRICE discloses by making the ROI slightly larger than the subset of pixels identified by object localization or segmentation, depth discontinuities around an object instance can be included in a localized depth map, so sufficient contrast can be determined between the object instance and background portions of the scene 110 in subsequent processing of the localized depth map (wherein a localized depth map shows depth discontinuities). In paragraph [0046]-PRICE discloses the localized depth map calculation module 160 is configured to selectively generate a localized depth map for each of the object instances identified for the current frame 142 (or the frame period for the current frame 142). A depth map may also be referred to as a “depth image.” A localized depth map for an object instance is calculated within its respective ROI (wherein a localized depth map determines a depth and depth discontinuities for each pixel within a vehicle object instance)).
PRICE is silent on and determining a position at which the range information is discontinuous, corresponding to adjacent pixels in which a range difference between the adjacent pixels is greater than a predetermined value among a plurality of pixels included in the estimated vehicle area, to be a boundary between different vehicles.
However, AMANO explicitly teaches and determining a position at which the range information is discontinuous (Fig. 5. Paragraph [0082]-AMANO discloses the object recognition device 1 has an imaging function for imaging a traveling direction of the vehicle 70 (wherein the object recognition device performs basic detection processing, separation detection processing, and integration detection processing in any order or in parallel, and the generation unit 500 generates a V-Disparity map, a U-Disparity map, a Real U-Disparity map). In paragraph [0113]-AMANO discloses the V-Disparity map is an example of “information in which a position in the vertical direction is associated with a position in the depth direction”. The U-Disparity map and the Real U-Disparity map are examples of “information in which a position in the horizontal direction is associated with a position in the depth direction. Please also read paragraph [0127-0129]), corresponding to adjacent pixels in which a range difference between the adjacent pixels is greater than a predetermined value among a plurality of pixels included in the estimated vehicle area (Fig. 5. Paragraph [0147]-AMANO discloses at Step S201, the basic detection unit 511 performs 8-neighbor labeling processing for giving the same ID to pixels that are continuous in a vertical, horizontal, or oblique direction for a parallax point as a pixel having a pixel value (frequency of the parallax) equal to or larger than a predetermined value in the real Umap RM. Further in paragraph [0154]-AMANO discloses at Step S301, the integration detection unit 513 performs 4-neighbor labeling processing for giving the same ID to pixels (parallax points) that are continuous in the vertical direction (depth direction) or the lateral direction (horizontal direction) on the small real Umap. Please also read paragraph [0138-0140, 0152]), to be a boundary between different vehicles (Fig. 5. Paragraph [0148]-AMANO discloses basic detection unit 511 sets a rectangle circumscribing each pixel group (each isolated region) to which the same ID is given (Step S202). In paragraph [0155]-AMANO discloses the integration detection unit 513 sets a rectangle circumscribing each pixel group (each isolated region) to which the same ID is given (Step S302). In paragraph [0156]-AMANO discloses the integration detection unit 513 extracts the object such as a vehicle (Step S303). The integration detection unit 513 extracts the region of the object such as a vehicle based on the width, the depth, frequency of the parallax, and the like of each isolated region. Accordingly, the rectangle circumscribing each isolated region is detected as the object region. Please also read paragraph [0158-0165]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of PRICE of having an image processing method that comprises: recognizing, among multiple different area classes designated regarding photographic subjects, to which area class a photographic subject of each pixel in a captured image that has been acquired belongs, with the teachings of AMANO of having and determining a position at which the range information is discontinuous, corresponding to adjacent pixels in which a range difference between the adjacent pixels is greater than a predetermined value among a plurality of pixels included in the estimated vehicle area, to be a boundary between different vehicles.
Wherein PRICE’s method having acquiring, from depth map information corresponding to the captured image, range information of each pixel in an estimated vehicle area in the captured image representing a vehicle area class among the multiple different area classes, and determining a position at which the range information is discontinuous, corresponding to adjacent pixels in which a range difference between the adjacent pixels is greater than a predetermined value among a plurality of pixels included in the estimated vehicle area, to be a boundary between different vehicles.
The motivation behind the modification would have been to obtain a method that improves the performance of object detection and/or depth estimation, since both PRICE and AMANO concern object detection and depth estimation. Wherein PRICE provides systems and methods that improve the performance of object detection and/or depth estimation, while AMANO provides systems and methods improves the performance and accuracy of object recognition when there are multiple potentially device while also improving the safety and reliability of hazard detection. Please see PRICE et al. (US 20190058859 A1), Abstract and Paragraph [0001 and 0030] and AMANO et al. (US 20200074212 A1), Abstract and Paragraph [188-0189].
Regarding claim 7, PRICE explicitly teaches a non-transitory computer-readable storage medium (Fig. 1-3 and 12, #162, #314, #1206 and #1208, called memory, main memory and read-only memory. Paragraph [0033, 0038, 0066, 0076-0079]) storing program that makes a computer in an image processing device (Fig. 1-3 and 12, #100, #200, #300 and #1200 called a system, a head-mounted device and a computer system. Paragraph [0021, 0056, 0060, 0076]. Further at [0076]-PRICE discloses FIG. 12 illustrates a block diagram showing an example computer system 1200 upon which aspects of this disclosure may be implemented. Further in paragraph [0060]-PRICE discloses the techniques described herein may also be applied to other sensing applications such as automotive or vehicular sensors, object scanning sensors, and sensors placed on UAVs (unmanned air vehicles) or other aerial vehicles) to execute processes (Fig. 12. Paragraph [0076]-PRICE discloses computer system 1200 includes a main memory 1206, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1202 for storing information and instructions to be executed by processor 1204. In paragraph [0077]-PRICE discloses computer system 1200 can further include a read only memory (ROM) 1208 or other static storage device coupled to bus 1202 for storing static information and instructions for processor 1204. A storage device 1210, such as a flash or other non-volatile memory can be coupled to bus 1202 for storing information and instructions. Please also see Fig. 1-3 and read paragraph [0056 and 0066]), the processes comprising recognizing, among multiple different area classes designated regarding photographic subjects, to which area class a photographic subject of each pixel in a captured image (Fig. 1, #140, #142, #144 and #146, called frame(s). Paragraph [0025, 0032-0033]) that has been acquired belongs (Fig. 1. Paragraph [0035]-PRICE discloses the object detector 154 is configured to receive the pixel data 153 for the frame 142 and to identify one or more subsets of the pixels included in the pixel data 153, each subset being associated with a respective physical object detected based on the pixel data 153. Each physical object detected based on the pixel data 153 may be referred to as an “object instance.” Detection of an object instance may include classification of the object, in which one or more classes or types of physical objects are identified for the object instance. A detected object instance may have one or more object classifications with respective confidence scores that the object instance actually corresponds to a respective object class. The object detector 154 may be configured to use one or more threshold scores to determine whether a detected object instance will be processed as an object instance of a particular object class, based on whether a confidence score for the object class is greater than or equal to a respective threshold score. Separate threshold scores may be used for respective object classes or groups of object classes. One or more of the threshold scores may be dynamically set and/or changed according to various conditions detected by the system 100); and
Although PRICE explicitly teaches acquiring, from depth map information (Fig. 1. Paragraph [0048]-PRICE discloses the localized depth map calculation module 160 generates, for the current frame 142 (or the frame period for the frame 142), a localized depth map 164 (which may be referred to as “first localized depth map 164”) corresponding to a first object instance detected in pixel data 153 and a second localized depth map 166 corresponding to a second object instance detected in pixel data 153) corresponding to the captured image (Fig. 1, #140, #142, #144 and #146, called frame(s). Paragraph [0025, 0032-0033]), range information of each pixel in an estimated vehicle area (Fig. 1. Paragraph [0040]-PRICE discloses the localized depth map generator 150 includes a region of interest identification module 156. The region of interest identification module 156 is configured to select a respective region of the current frame 142 (or an FOV of the frame 142) corresponding to each subset of pixels identified by the object detector 154 for a respective object instance. A region selected by the region of interest identification module 156 for an object instance may be referred to as an ROI for the object instance. An ROI for an object instance may be used to selectively identify respective pixels included in pixel data 153 and/or pixel data 143 for processing performed by resizing module 158 and/or localized depth map calculation module 160. Please also read paragraph [0029-0030, 0046 and 0062]) in the captured image (Fig. 1, #140, #142, #144 and #146, called frame(s). Paragraph [0025, 0032-0033]) representing a vehicle area class among the multiple different area classes (Fig. 1. Paragraph [0035]-PRICE discloses the object detector 154 is configured to receive the pixel data 153 for the frame 142 and to identify one or more subsets of the pixels included in the pixel data 153, each subset being associated with a respective physical object detected based on the pixel data 153. Each physical object detected based on the pixel data 153 may be referred to as an “object instance.” Detection of an object instance may include classification of the object, in which one or more classes or types of physical objects are identified for the object instance. In paragraph [0021]-PRICE discloses examples of physical objects 112 and 114 include, but are not limited to vehicles), and determining a position at which the range information is discontinuous to be a boundary (Fig. 1. Paragraph [0041]-PRICE discloses by making the ROI slightly larger than the subset of pixels identified by object localization or segmentation, depth discontinuities around an object instance can be included in a localized depth map, so sufficient contrast can be determined between the object instance and background portions of the scene 110 in subsequent processing of the localized depth map (wherein a localized depth map shows depth discontinuities). In paragraph [0046]-PRICE discloses the localized depth map calculation module 160 is configured to selectively generate a localized depth map for each of the object instances identified for the current frame 142 (or the frame period for the current frame 142). A depth map may also be referred to as a “depth image.” A localized depth map for an object instance is calculated within its respective ROI (wherein a localized depth map determines a depth and depth discontinuities for each pixel within a vehicle object instance)).
PRICE is silent on and determining a position at which the range information is discontinuous, corresponding to adjacent pixels in which a range difference between the adjacent pixels is greater than a predetermined value among a plurality of pixels included in the estimated vehicle area, to be a boundary between different vehicles.
However, AMANO explicitly teaches and determining a position at which the range information is discontinuous (Fig. 5. Paragraph [0082]-AMANO discloses the object recognition device 1 has an imaging function for imaging a traveling direction of the vehicle 70 (wherein the object recognition device performs basic detection processing, separation detection processing, and integration detection processing in any order or in parallel, and the generation unit 500 generates a V-Disparity map, a U-Disparity map, a Real U-Disparity map). In paragraph [0113]-AMANO discloses the V-Disparity map is an example of “information in which a position in the vertical direction is associated with a position in the depth direction”. The U-Disparity map and the Real U-Disparity map are examples of “information in which a position in the horizontal direction is associated with a position in the depth direction. Please also read paragraph [0127-0129]), corresponding to adjacent pixels in which a range difference between the adjacent pixels is greater than a predetermined value among a plurality of pixels included in the estimated vehicle area (Fig. 3. Paragraph [0147]-AMANO discloses at Step S201, the basic detection unit 511 performs 8-neighbor labeling processing for giving the same ID to pixels that are continuous in a vertical, horizontal, or oblique direction for a parallax point as a pixel having a pixel value (frequency of the parallax) equal to or larger than a predetermined value in the real Umap RM. Further in paragraph [0154]-AMANO discloses at Step S301, the integration detection unit 513 performs 4-neighbor labeling processing for giving the same ID to pixels (parallax points) that are continuous in the vertical direction (depth direction) or the lateral direction (horizontal direction) on the small real Umap. Please also read paragraph [0138-0140, 0152]), to be a boundary between different vehicles (Fig. 5. Paragraph [0148]-AMANO discloses basic detection unit 511 sets a rectangle circumscribing each pixel group (each isolated region) to which the same ID is given (Step S202). In paragraph [0155]-AMANO discloses the integration detection unit 513 sets a rectangle circumscribing each pixel group (each isolated region) to which the same ID is given (Step S302). In paragraph [0156]-AMANO discloses the integration detection unit 513 extracts the object such as a vehicle (Step S303). The integration detection unit 513 extracts the region of the object such as a vehicle based on the width, the depth, frequency of the parallax, and the like of each isolated region. Accordingly, the rectangle circumscribing each isolated region is detected as the object region. Please also read paragraph [0158-0165]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of PRICE of having a non-transitory computer-readable storage medium storing program that makes a computer in an image processing device to execute processes, the processes comprising: recognizing, among multiple different area classes designated regarding photographic subjects, to which area class a photographic subject of each pixel in a captured image that has been acquired belongs, with the teachings of AMANO of having and determining a position at which the range information is discontinuous, corresponding to adjacent pixels in which a range difference between the adjacent pixels is greater than a predetermined value among a plurality of pixels included in the estimated vehicle area, to be a boundary between different vehicles.
Wherein PRICE’s a non-transitory computer-readable storage medium having acquiring, from depth map information corresponding to the captured image, range information of each pixel in an estimated vehicle area in the captured image representing a vehicle area class among the multiple different area classes, and determining a position at which the range information is discontinuous, corresponding to adjacent pixels in which a range difference between the adjacent pixels is greater than a predetermined value among a plurality of pixels included in the estimated vehicle area, to be a boundary between different vehicles.
The motivation behind the modification would have been to obtain a non-transitory computer-readable storage medium that improves the performance of object detection and/or depth estimation, since both PRICE and AMANO concern object detection and depth estimation. Wherein PRICE provides systems and methods that improve the performance of object detection and/or depth estimation, while AMANO provides systems and methods improves the performance and accuracy of object recognition when there are multiple potentially overlapping objects. Please see PRICE et al. (US 20190058859 A1), Abstract and Paragraph [0001 and 0030] and AMANO et al. (US 20200074212 A1), Abstract and Paragraph [188-0189].
Regarding claim 8, PRICE in view of AMANO explicitly teaches the image processing device according to claim 1, PRICE further teaches wherein the at least one processor (Fig. 1-3 and 12, #220 and #1204, called a controller and a processor. Paragraph [0056, 0066, 0076]) is configured to execute the instructions (Fig. 12. Paragraph [0076]-PRICE discloses computer system 1200 includes a main memory 1206, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1202 for storing information and instructions to be executed by processor 1204. In paragraph [0077]-PRICE discloses computer system 1200 can further include a read only memory (ROM) 1208 or other static storage device coupled to bus 1202 for storing static information and instructions for processor 1204. A storage device 1210, such as a flash or other non-volatile memory can be coupled to bus 1202 for storing information and instructions. Please also see Fig. 1-3 and read paragraph [0056 and 0066]) to:
identify a single-vehicle area of one of a plurality of vehicles within the estimated vehicle (Fig. 1. Paragraph [0035]-PRICE discloses the object detector 154 is configured to receive the pixel data 153 for the frame 142 and to identify one or more subsets of the pixels included in the pixel data 153, each subset being associated with a respective physical object detected based on the pixel data 153. Each physical object detected based on the pixel data 153 may be referred to as an “object instance.” Detection of an object instance may include classification of the object, in which one or more classes or types of physical objects are identified for the object instance. In paragraph [0021]-PRICE discloses examples of physical objects 112 and 114 include, but are not limited to, people, animals and vehicles. In paragraph [0040]-PRICE discloses the region of interest identification module 156 is configured to select a respective region of the current frame 142 (or an FOV of the frame 142) corresponding to each subset of pixels identified by the object detector 154 for a respective object instance. A region selected by the region of interest identification module 156 for an object instance may be referred to as an ROI for the object instance. Please also read paragraph [0041-0043]).
Regarding claim 9, PRICE in view of AMANO explicitly teaches the image processing device according to claim 1, PRICE further teaches wherein the at least one processor is configured to execute the instructions (Fig. 12. Paragraph [0076]-PRICE discloses computer system 1200 includes a main memory 1206, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1202 for storing information and instructions to be executed by processor 1204. In paragraph [0077]-PRICE discloses computer system 1200 can further include a read only memory (ROM) 1208 or other static storage device coupled to bus 1202 for storing static information and instructions for processor 1204. A storage device 1210, such as a flash or other non-volatile memory can be coupled to bus 1202 for storing information and instructions. Please also see Fig. 1-3 and read paragraph [0056 and 0066]) to:
output a first detection result indicating a vehicle area (Fig. 4. Paragraph [0032]-PRICE discloses the localized depth map generator 150 is configured to receive the frame 142 (and in some examples, receive frame 146 from the second camera 130) and generate a localized depth map for each instance of a physical object detected in frame 142. The term “localized depth map” refers to a depth map for an object instance that corresponds to, and is essentially limited to, a region of interest (ROI) selected by the region of interest selection module 156 for the object instance. In paragraph [0021]-PRICE discloses examples of physical objects 112 and 114 include vehicles) corresponding to a size of the vehicle among areas obtained by determining the boundary between different vehicles (Fig. 4. Paragraph [0035]-PRICE discloses each physical object detected based on the pixel data 153 may be referred to as an “object instance.” In paragraph [0036]-PRICE discloses the object detector 154 is configured to identify one or more subsets of the pixels included in the pixel data 153 for the frame 142, each subset being associated with a respective one of the object instances detected based on the pixel data 153. Such identification of a subset of pixels associated with a detected object instance may be referred to as “object localization.” An object localization for an object instance results in a rectangular bounding box that closely surrounds multiple pixels identified for the object instance);
output a second detection result (Fig. 4. Paragraph [0035]-PRICE discloses the object detector 154 is configured to receive the pixel data 153 for the frame 142 and to identify one or more subsets of the pixels included in the pixel data 153, each subset being associated with a respective physical object detected based on the pixel data 153. Detection of an object instance may include classification of the object, in which one or more classes or types of physical objects are identified for the object instance) indicating the vehicle area obtained by inputting the captured image into a vehicle recognition model generated by machine learning (Fig. 4. Paragraph [0037]-PRICE discloses a machine-trained model (such as a model for a deep-structured convolutional neural network) trained to detect instances of one or more object classes may be applied for object detection, localization, and/or segmentation based on at least the pixel data 153 for the frame 142); and
PRICE fails to explicitly teach output a result of a matching process which identifies, as a single-vehicle area, an area in which the vehicle area indicated by the first detection result overlaps with the vehicle area indicated by the second detection result by a predetermined size or more.
However, AMANO explicitly teaches output a result of a matching process (Fig. 5. Paragraph [0243]-AMANO discloses the final determination processing unit 1144 receives three results including the basic detection result, the separation detection result, and the integration detection result, calculates a correspondence relation among the detection results, and sets an inclusive frame and a partial frame accompanying the inclusive frame (wherein basic detection, separation detection, and integration detection processing may be performed in any order, or in parallel). The final determination processing unit 1144 corrects the inclusive frame and the partial frame, and selects an output target therefrom. The integration detection result or the basic detection result is set as the inclusive frame. The partial frame stores a result detected through processing having separation performance higher than that of the inclusive frame. The partial frame is a detection frame (an outer frame of the detection result) associated with the inclusive frame, and is a result obtained by separating the inside of the inclusive frame. In this case, the basic detection result or the separation detection result corresponds to the partial frame. The frame indicates a position and a size of the object, and is information associating coordinates of a corner of the rectangle surrounding the object with the height and the width. Please also read paragraph [0142-0144, 0208-0210, 0248 and 0250-0251]) which identifies, as a single-vehicle area, an area in which the vehicle area indicated by the first detection result overlaps with the vehicle area indicated by the second detection result (Fig. 5. Paragraph [0142]-AMANO discloses the selection unit 514 of the clustering processing unit 510 selects the object region to be output to the frame creation unit 515 from among object regions detected through the “basic detection processing”, the “separation detection processing”, and the “integration detection processing”. In paragraph [0147]-AMANO discloses at Step S201, the basic detection unit 511 performs 8-neighbor labeling processing for giving the same ID to pixels that are continuous in a vertical, horizontal, or oblique direction for a parallax point as a pixel having a pixel value (frequency of the parallax) equal to or larger than a predetermined value in the real Umap RM (wherein the basic detection unit 511 sets a rectangle circumscribing each pixel group to which the same ID is given (Step S202)). In paragraph [0154]-AMANO discloses at Step S301, the integration detection unit 513 performs 4-neighbor labeling processing for giving the same ID to pixels (parallax points) that are continuous in the vertical direction (depth direction) or the lateral direction (horizontal direction) on the small real Umap. In the above processing, the 8-neighbor labeling processing may be used (wherein the integration detection unit 513 extracts the region of the object such as a vehicle based on the width, the depth, frequency of the parallax, and the like of each isolated region and sets a rectangle circumscribing each pixel group to which the same ID is given (Step S302)) by a predetermined size or more (Fig. 5. Paragraph [0160]-AMANO discloses the selection unit 514 determines whether the object region detected through the integration detection processing is overlapped with one object region detected through the basic detection processing in a certain degree in the real Umap RM (Step S402). If a value obtained by dividing an area of a region in which the object region detected through the integration detection processing is overlapped with the object region detected through the basic detection processing in the real Umap RM by an area of the object region detected through the basic detection processing is equal to or larger than a predetermined threshold, it is determined that they are overlapped with each other in a certain degree (wherein inclusive and partial frame detection results may be merged). Please also read paragraph [0249-0251]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of PRICE in view of AMANO of having an image processing device comprising: recognize, among multiple different area classes designated regarding photographic subjects, to which area class a photographic subject of each pixel in a captured image that has been acquired belongs, with the teachings of AMANO of having output a result of a matching process which identifies, as a single-vehicle area, an area in which the vehicle area indicated by the first detection result overlaps with the vehicle area indicated by the second detection result by a predetermined size or more.
Wherein PRICE’s device having output a result of a matching process which identifies, as a single-vehicle area, an area in which the vehicle area indicated by the first detection result overlaps with the vehicle area indicated by the second detection result by a predetermined size or more.
The motivation behind the modification would have been to obtain a device that improves the performance of object detection and/or depth estimation, since both PRICE and AMANO concern object detection and depth estimation. Wherein PRICE provides systems and methods that improve the performance of object detection and/or depth estimation, while AMANO provides systems and methods improves the performance and accuracy of object recognition when there are multiple potentially device while also improving the safety and reliability of hazard detection. Please see PRICE et al. (US 20190058859 A1), Abstract and Paragraph [0001 and 0030] and AMANO et al. (US 20200074212 A1), Abstract and Paragraph [188-0189].
Regarding claim 10, PRICE in view of AMANO explicitly teaches the image processing device according to claim 1, PRICE further teaches wherein the at least one processor is configured to execute the instructions to:
detect a vehicle area in the captured image (Fig. 4. Paragraph [0035]-PRICE discloses each physical object detected based on the pixel data 153 may be referred to as an “object instance.” In paragraph [0036]-PRICE discloses the object detector 154 is configured to identify one or more subsets of the pixels included in the pixel data 153 for the frame 142, each subset being associated with a respective one of the object instances detected based on the pixel data 153. Such identification of a subset of pixels associated with a detected object instance may be referred to as “object localization.” An object localization for an object instance results in a rectangular bounding box that closely surrounds multiple pixels identified for the object instance), using at least one of pattern matching, or a vehicle recognition model based on machine learning (Fig. 4. Paragraph [0037]-PRICE discloses a machine-trained model (such as a model for a deep-structured convolutional neural network) trained to detect instances of one or more object classes may be applied for object detection, localization, and/or segmentation based on at least the pixel data 153 for the frame 142);
PRICE fails to explicitly teach compare a first vehicle area included in a result of acquiring the range information of each pixel in the estimated vehicle area from the depth map information, with a second vehicle area included in a result of detecting the vehicle area using the at least one of the pattern matching, or the vehicle recognition model; and identify, as a single-vehicle area, an area in which the first vehicle area and the second vehicle area overlap by a predetermined size or more.
However, AMANO explicitly teaches compare a first vehicle area included in a result of acquiring the range information of each pixel in the estimated vehicle area from the depth map information (Fig. 5. Paragraph [0113]-AMANO discloses the second generation unit 500 is a functional unit generates a V-Disparity map, a U-Disparity map, a Real U-Disparity map, and the like. The V-Disparity map is an example of “information in which a position in the vertical direction is associated with a position in the depth direction”. The U-Disparity map and the Real U-Disparity map are examples of “information in which a position in the horizontal direction is associated with a position in the depth direction”), with a second vehicle area included in a result of detecting the vehicle area (Fig. 5. Paragraph [0142]-AMANO discloses the selection unit 514 of the clustering processing unit 510 selects the object region to be output to the frame creation unit 515 from among object regions detected through the “basic detection processing”, the “separation detection processing”, and the “integration detection processing”. In paragraph [0147]-AMANO discloses at Step S201, the basic detection unit 511 performs 8-neighbor labeling processing for giving the same ID to pixels that are continuous in a vertical, horizontal, or oblique direction for a parallax point as a pixel having a pixel value equal to or larger than a predetermined value in the real Umap RM (wherein the basic detection unit 511 sets a rectangle circumscribing each pixel group to which the same ID is given). In paragraph [0154]-AMANO discloses at Step S301, the integration detection unit 513 performs 4-neighbor labeling processing for giving the same ID to pixels (parallax points) that are continuous in the vertical direction (depth direction) or the lateral direction (horizontal direction) on the small real Umap. In the above processing, the 8-neighbor labeling processing may be used (wherein the integration detection unit 513 extracts the region of the object such as a vehicle based on the width, the depth, frequency of the parallax, and the like of each isolated region and sets a rectangle circumscribing each pixel group to which the same ID is given) using the at least one of the pattern matching, or the vehicle recognition model (Fig. 5. Paragraph [0248]-AMANO discloses at Step S1053, the merge processing unit 1146 performs matching between the integration detection result and the basic detection result. Please also read paragraph [0147, 0154-0155, 0208-0210]); and
identify, as a single-vehicle area, an area in which the first vehicle area and the second vehicle area overlap by a predetermined size or more (Fig. 5. Paragraph [0249]-AMANO discloses the merge processing unit 1146 calculates an overlapping rate of the integration detection result and the basic detection result. In paragraph [0251]-AMANO discloses when the integration detection result overlaps with the basic detection result as a result of matching at Step 51053, the result of Step S1054 is “Yes”. If the result of Step S1054 is “Yes”, the merge processing unit 1146 merges the inclusive frame (integration detection result) with the partial frame (the basic detection result or the separation detection result) (Step S1055), and generates one “detection result”. Please also read paragraph [0243]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of PRICE in view of AMANO of having an image processing device comprising: recognize, among multiple different area classes designated regarding photographic subjects, to which area class a photographic subject of each pixel in a captured image that has been acquired belongs, with the teachings of AMANO of having compare a first vehicle area included in a result of acquiring the range information of each pixel in the estimated vehicle area from the depth map information, with a second vehicle area included in a result of detecting the vehicle area using the at least one of the pattern matching, or the vehicle recognition model; and identify, as a single-vehicle area, an area in which the first vehicle area and the second vehicle area overlap by a predetermined size or more.
Wherein PRICE’s device having compare a first vehicle area included in a result of acquiring the range information of each pixel in the estimated vehicle area from the depth map information, with a second vehicle area included in a result of detecting the vehicle area using the at least one of the pattern matching, or the vehicle recognition model; and identify, as a single-vehicle area, an area in which the first vehicle area and the second vehicle area overlap by a predetermined size or more.
The motivation behind the modification would have been to obtain a device that improves the performance of object detection and/or depth estimation, since both PRICE and AMANO concern object detection and depth estimation. Wherein PRICE provides systems and methods that improve the performance of object detection and/or depth estimation, while AMANO provides systems and methods improves the performance and accuracy of object recognition when there are multiple potentially device while also improving the safety and reliability of hazard detection. Please see PRICE et al. (US 20190058859 A1), Abstract and Paragraph [0001 and 0030] and AMANO et al. (US 20200074212 A1), Abstract and Paragraph [188-0189].
Regarding claim 11, PRICE in view of AMANO explicitly teaches the image processing device according to claim 1, PRICE further teaches wherein the at least one processor (Fig. 1-3 and 12, #220 and #1204, called a controller and a processor. Paragraph [0056, 0066, 0076]) is configured to execute the instructions (Fig. 12. Paragraph [0076]-PRICE discloses computer system 1200 includes a main memory 1206, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1202 for storing information and instructions to be executed by processor 1204. In paragraph [0077]-PRICE discloses computer system 1200 can further include a read only memory (ROM) 1208 or other static storage device coupled to bus 1202 for storing static information and instructions for processor 1204. A storage device 1210, such as a flash or other non-volatile memory can be coupled to bus 1202 for storing information and instructions. Please also see Fig. 1-3 and read paragraph [0056 and 0066]) to:
generate, for each pixel in the captured image, area recognition information having probability in which a pixel belongs to each of the multiple area classes representing multiple different targets (Fig. 1. Paragraph [0035]-PRICE discloses the object detector 154 is configured to receive the pixel data 153 for the frame 142 and to identify one or more subsets of the pixels included in the pixel data 153, each subset being associated with a respective physical object detected based on the pixel data 153. Each physical object detected based on the pixel data 153 may be referred to as an “object instance.” Detection of an object instance may include classification of the object, in which one or more classes or types of physical objects are identified for the object instance. A detected object instance may have one or more object classifications with respective confidence scores that the object instance actually corresponds to a respective object class. The object detector 154 may be configured to use one or more threshold scores to determine whether a detected object instance will be processed as an object instance of a particular object class, based on whether a confidence score for the object class is greater than or equal to a respective threshold score. Separate threshold scores may be used for respective object classes or groups of object classes. One or more of the threshold scores may be dynamically set and/or changed according to various conditions detected by the system 100); and
Although PRICE explicitly teaches acquire the range information of each pixel in the estimated vehicle area (Fig. 1. Paragraph [0040]-PRICE discloses the localized depth map generator 150 includes a region of interest identification module 156. The region of interest identification module 156 is configured to select a respective region of the current frame 142 (or an FOV of the frame 142) corresponding to each subset of pixels identified by the object detector 154 for a respective object instance. A region selected by the region of interest identification module 156 for an object instance may be referred to as an ROI for the object instance. An ROI for an object instance may be used to selectively identify respective pixels included in pixel data 153 and/or pixel data 143 for processing performed by resizing module 158 and/or localized depth map calculation module 160. An ROI may be a rectangular bounding box or at a pixel-level. Please also read paragraph [0029-0030 and 0062]), in which the probability for the vehicle area class is equal to or higher than a threshold value in the captured image, and determine the position at which the range information is discontinuous in the detected estimated vehicle area to be the boundary (Fig. 1. Paragraph [0041]-PRICE discloses by making the ROI slightly larger than the subset of pixels identified by object localization or segmentation, depth discontinuities around an object instance can be included in a localized depth map, so sufficient contrast can be determined between the object instance and background portions of the scene 110 in subsequent processing of the localized depth map. In paragraph [0046]-PRICE discloses the localized depth map calculation module 160 is configured to selectively generate a localized depth map for each of the object instances identified for the current frame 142 (or the frame period for the current frame 142). A depth map may also be referred to as a “depth image.” A localized depth map for an object instance is calculated within its respective ROI (wherein a localized depth map determines a depth and depth discontinuities for each pixel within a vehicle object instance). Please also read paragraph [0035]).
PRICE is silent on and determine the position at which the range information is discontinuous in the detected estimated vehicle area to be the boundary between different vehicles.
However, AMANO explicitly teaches and determine the position at which the range information is discontinuous (Fig. 5. Paragraph [0113]-AMANO discloses the V-Disparity map is an example of “information in which a position in the vertical direction is associated with a position in the depth direction”. The U-Disparity map and the Real U-Disparity map are examples of “information in which a position in the horizontal direction is associated with a position in the depth direction. In paragraph [0147]-AMANO discloses at Step S201, the basic detection unit 511 performs 8-neighbor labeling processing for giving the same ID to pixels that are continuous in a vertical, horizontal, or oblique direction for a parallax point as a pixel having a pixel value (frequency of the parallax) equal to or larger than a predetermined value in the real Umap RM. Further in paragraph [0154]-AMANO discloses at Step S301, the integration detection unit 513 performs 4-neighbor labeling processing for giving the same ID to pixels (parallax points) that are continuous in the vertical direction (depth direction) or the lateral direction (horizontal direction) on the small real Umap. Please also read paragraph [0127-0129]) in the detected estimated vehicle area to be the boundary between different vehicles (Fig. 5. Paragraph [0148]-AMANO discloses basic detection unit 511 sets a rectangle circumscribing each pixel group (each isolated region) to which the same ID is given (Step S202). In paragraph [0155]-AMANO discloses the integration detection unit 513 sets a rectangle circumscribing each pixel group (each isolated region) to which the same ID is given (Step S302). In paragraph [0156]-AMANO discloses the integration detection unit 513 extracts the object such as a vehicle (Step S303). The integration detection unit 513 extracts the region of the object such as a vehicle based on the width, the depth, frequency of the parallax, and the like of each isolated region. Accordingly, the rectangle circumscribing each isolated region is detected as the object region. Please also read paragraph [0138-0140, 0152]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of PRICE in view of AMANO of having an image processing device comprising: recognize, among multiple different area classes designated regarding photographic subjects, to which area class a photographic subject of each pixel in a captured image that has been acquired belongs, with the teachings of AMANO of having and determine the position at which the range information is discontinuous in the detected estimated vehicle area to be the boundary between different vehicles.
Wherein PRICE’s device having generate, for each pixel in the captured image, area recognition information having probability in which a pixel belongs to each of the multiple area classes representing multiple different targets; and acquire the range information of each pixel in the estimated vehicle area, in which the probability for the vehicle area class is equal to or higher than a threshold value in the captured image, and determine the position at which the range information is discontinuous in the detected estimated vehicle area to be the boundary between different vehicles.
The motivation behind the modification would have been to obtain a device that improves the performance of object detection and/or depth estimation, since both PRICE and AMANO concern object detection and depth estimation. Wherein PRICE provides systems and methods that improve the performance of object detection and/or depth estimation, while AMANO provides systems and methods improves the performance and accuracy of object recognition when there are multiple potentially device while also improving the safety and reliability of hazard detection. Please see PRICE et al. (US 20190058859 A1), Abstract and Paragraph [0001 and 0030] and AMANO et al. (US 20200074212 A1), Abstract and Paragraph [188-0189].
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over PRICE et al. (US 20190058859 A1), hereinafter referenced as PRICE in view of AMANO et al. (US 20200074212 A1), hereinafter referenced as AMANO and in further view of PARK et al. (US 20220250624 A1), hereinafter referenced as PARK.
Regarding claim 4, PRICE in view of AMANO explicitly teaches the image processing device according to claim 3, although PRICE explicitly teaches a rectangular region indicating the single-vehicle area (Fig. 1. Paragraph [0040]-PRICE discloses the localized depth map generator 150 includes a region of interest identification module 156. The region of interest identification module 156 is configured to select a respective region of the current frame 142 (or an FOV of the frame 142) corresponding to each subset of pixels identified by the object detector 154 for a respective object instance. A region selected by the region of interest identification module 156 for an object instance may be referred to as an ROI for the object instance. An ROI for an object instance may be used to selectively identify respective pixels included in pixel data 153 and/or pixel data 143 for processing performed by resizing module 158 and/or localized depth map calculation module 160. An ROI may be a rectangular bounding box or at a pixel-level. In paragraph [0021]-PRICE discloses examples of physical objects 112 and 114 include vehicles Please also read paragraph [0035]).
PRICE fails to explicitly teach wherein the at least one processor is configured to execute the instructions to: exclude the single-vehicle area from the estimated vehicle area when, based on a positional relationship between the single-vehicle area and an area in an area class representing a road among the area classes, a pixel adjacent to below a pixel constituting a lower edge of a rectangular region indicating the single-vehicle area is not adjacent to the area in an area class representing a road.
However, PARK explicitly teaches wherein the at least one processor is configured to execute the instructions (Fig. 7, #706 called a CPU. Paragraph [0157]-PARK discloses FIG. 7 is a block diagram of an example computing device(s) 700 suitable for use in implementing some embodiments of the present disclosure. Computing device 700 may include an interconnect system 702 that directly or indirectly couples the following devices: memory 704, one or more central processing units (CPUs) 706, one or more graphics processing units (GPUs) 708. In paragraph [0163]-PARK discloses the CPU(s) 706 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 700 to perform one or more of the methods and/or processes described herein. Please also see Fig. 6A-D and 8) to:
exclude the single-vehicle area from the estimated vehicle area (Fig. 3. Paragraph [0034]-PARK discloses the map generator 130 may generate the depth map of the environment of the vehicle 600 and subsequently carve out regions of interest – such as regions that correspond to the roadway surface (wherein the map generator 130 may use a freespace boundary separating drivable freespace from non-drivable space, such as sidewalks, vehicles, and other objects). In paragraph [0035]-PARK discloses once the depth map (e.g., disparity map) has been generated, the hazard detector 140 may use the map to identify hazards on the roadway (wherein the hazard detection process detects hazards on or within a road surface, and the detection process may use either drivable or non-drivable free-space (i.e. space that either includes or does not include other vehicles). Please also read paragraph [00]) when, based on a positional relationship between the single-vehicle area and an area in an area class representing a road among the area classes (Fig. 3. Paragraph [0035]-PARK discloses when a hazard is present, the hazard detector 140 may detect a discontinuity in disparity values indicative of a distance spanning between a first pixel corresponding to a top of the hazard and a second pixel immediately above the first pixel. The hazard detector 140 may analyze pairs of pixels on the depth map and, when the hazard detector 140 determines that a disparity between a pair of pixels satisfies a threshold disparity, the hazard detector 140 may identify the pixel nearest the vehicle 600 (e.g., the lower pixel of the pair) and label the pixel as a hazard pixel. The hazard detector 140 may further identify pixel pairs neighboring the identified hazard pixel and determine whether those pixel pairs also show a disparity. When the hazard detector 140 identifies one or more hazard pixels, the hazard detector 140 may determine that a hazard exists at the location of the one or more hazard pixels), a pixel adjacent to below a pixel constituting a lower edge of a region indicating the single-vehicle area is not adjacent to the area in an area class representing a road (Fig. 3. Paragraph [0035]-PARK discloses [0037]-PARK discloses in determining the threshold disparity, a maximal distance to the object 312 and/or a minimal pixel size may be considered. For the maximal distance, the threshold disparity must be differentiable on the distance map. As such, the object 312 may not be identified as a hazard unless the disparity difference between neighboring pixels of a same column of pixels in the disparity and/or OF magnitude map are greater than the threshold disparity value. Further in paragraph [0095]-PARK discloses the neural network may take as its input at least some subset of parameters, such as bounding box dimensions, ground plane estimate obtained. Therefore, it would have been obvious to a person of ordinary skill in the art to exclude a single-vehicle area as described in the limitation above. PARK implements rectangular bounding boxes and uses a depth map to improve computational efficiency by excluding non-drivable free-space, such as other vehicles and sidewalks. PARK also designates hazards by using a depth map and disparity thresholds to determine whether pixels adjacent to below a pixel constituting the lower edge of a vehicle are adjacent to an estimated road area. Thus, it would be obvious to combine both processes to determine whether a single-vehicle area should be excluded. This may improve the identification of hazards and computational efficiency).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of PRICE in view of AMANO of having an image processing device comprising: recognize, among multiple different area classes designated regarding photographic subjects, to which area class a photographic subject of each pixel in a captured image that has been acquired belongs, with the teachings of PARK of having wherein the at least one processor is configured to execute the instructions to: exclude the single-vehicle area from the vehicle area when, based on a positional relationship between the single-vehicle area and an area in an area class representing a road among the area classes, the single-vehicle area is not adjacent to the area in an area class representing a road.
Wherein PRICE’s device having wherein the at least one processor is configured to execute the instructions to: exclude the single-vehicle area from the vehicle area when, based on a positional relationship between the single-vehicle area and an area in an area class representing a road among the area classes, the single-vehicle area is not adjacent to the area in an area class representing a road.
The motivation behind the modification would have been to obtain a device that improves the performance of object detection and/or depth estimation, since both PRICE and PARK concern object detection and depth estimation. Wherein PRICE provides systems and methods that improve the performance of object detection and/or depth estimation, while PARK provides systems and methods that simplifies programming and improves performance of a computing device while also improving the safety and reliability of hazard detection. Please see PRICE et al. (US 20190058859 A1), Abstract and Paragraph [0001 and 0030] and PARK et al. (US 20220250624 A1), Abstract and Paragraph [0074, 0105, 0135 and 0145].
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over PRICE et al. (US 20190058859 A1), hereinafter referenced as PRICE in view of AMANO et al. (US 20200074212 A1), hereinafter referenced as AMANO and in further view of TAKEMURA et al. (US 20200349366 A1), hereinafter referenced as TAKEMURA.
Regarding claim 5, PRICE in view of AMANO explicitly teaches the image processing device according to claim 1, PRICE fails to explicitly teach wherein the at least one processor is configured to execute the instructions to: determine whether a size of the single-vehicle area is a size corresponding to a vehicle, and exclude the single-vehicle area from the estimated vehicle area if the size does not correspond to that of a vehicle.
However, TAKEMURA explicitly teaches wherein the at least one processor is configured to execute the instructions (Fig. 4. Paragraph [0032]-TAKEMURA discloses FIG. 1 is a block diagram showing the configuration of the onboard environment recognition device 1) to:
determine whether a size of the single-vehicle area is a size corresponding to a vehicle, and exclude the single-vehicle area from the estimated vehicle area if the size does not correspond to that of a vehicle (Fig. 4. Paragraph [0052]-TAKEMURA discloses in detection of a two-wheeled vehicle or in the detection of a vehicle, when the stereoscopic object is smaller than a predetermined size or larger than a predetermined size with respect to the two-wheeled vehicle and the vehicle respectively, the stereoscopic object is excluded from objects to be identified. Please also read paragraph [0050-0051 and 0054]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of PRICE in view of AMANO of having an image processing device comprising: recognize, among multiple different area classes designated regarding photographic subjects, to which area class a photographic subject of each pixel in a captured image that has been acquired belongs, with the teachings of TAKEMURA of having wherein the at least one processor is configured to execute the instructions to: determine whether a size of the single-vehicle area is a size corresponding to a vehicle, and exclude the single-vehicle area from the estimated vehicle area if the size does not correspond to that of a vehicle.
Wherein PRICE’s device having wherein the at least one processor is configured to execute the instructions to: determine whether a size of the single-vehicle area is a size corresponding to a vehicle, and exclude the single-vehicle area from the estimated vehicle area if the size does not correspond to that of a vehicle.
The motivation behind the modification would have been to obtain a device that improves the performance of object detection and depth estimation, since both PRICE and TAKEMURA concern object detection and depth estimation. Wherein PRICE provides systems and methods that improve the performance of object detection and/or depth estimation, while TAKEMURA provides systems and methods that improve the monocular-vision distance measurement and performance while correcting a road surface and camera heights. Please see PRICE et al. (US 20190058859 A1), Abstract and Paragraph [0001 and 0030] and TAKEMURA et al. (US 20200349366 A1), Abstract and Paragraph [0007 and 0085].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant`s disclosure.
WATANABE (US 20180357495 A1)- An image processing apparatus includes a reducer configured to reduce a number of pixels in a distance image having pixel values corresponding to a distance of a body positioned within a predetermined distance from a plurality of imagers installed in a mobile body, according to frames respectively captured by the plurality of imagers; a detector configured to detect the body based on the reduced distance image; and a rejecter configured to reject data of the detected body, based on a density of pixels having a value of the distance with respect to the detected body in an area overlapping a travel area in which the mobile body travels.......................Please see Fig. 5, 7-8. Abstract.
CAMUS et al. (US 20090195371 A1)- A method and apparatus for performing collision detection is described. An object is detected within a first operational range of an object tracker. A classification of the object is determined using the object tracker. The object tracker tracks the object. The object is detected within a second operational range of a collision detector. A safety measure is activated based on the classification using the collision detector.......................Please see Fig. 1-4. Abstract.
FOWE et al. (US 20200408534 A1)- An approach is provided for inferring vehicle location data using a reference vehicle. The approach includes processing sensor data collected from one or more sensors of the reference vehicle to identify a relative location of one or more vehicles represented in the sensor data, wherein the relative location is with respect to the reference vehicle. The approach also includes computing a vehicle location of the one or more vehicles based on the relative location and a reference location of the reference vehicle. The approach also includes providing the vehicle location of the one or more vehicles as an output comprising the inferred vehicle location data.......................Please see Para. [0035, 0043, 0045 and 0052]. Abstract.
AZUMA et al. (US 20190213746 A1)- A disparity estimation device calculates, for each of first pixels of a first image and each of second pixels of a second image, a first census feature amount and a second census feature amount, calculates, for each of the first pixels, a first disparity value of the first pixel with integer accuracy, extracts, for each of the first pixels, reference pixels located in positions corresponding to the first disparity value and a near disparity value close to the first disparity value from the second pixels, calculates sub-pixel evaluation values based on the relationship between the pixel values of the first pixel and the neighboring pixel and the pixel values of each of the reference pixels and the neighboring pixel, and estimates a second disparity value of the first pixel with sub-pixel accuracy by equiangular fitting........................Please see Para. [0098, 0120, 0128-0129, 0136-0139, 0142-0143, 0147]. Abstract.
MOU et al. (US 20190012808 A1)- Systems and method are provided for controlling a vehicle. In one embodiment, a vehicle includes a camera onboard the vehicle, a lidar device onboard the vehicle, a data storage element onboard the vehicle maintaining one or more transformation parameter values associated with a pairing of the camera and the lidar device, one or more sensors onboard the vehicle, and a controller. The controller detects a stationary condition based on output of the one or more sensors, obtains a first set of image data from the camera during the stationary condition, filters horizontal edge regions from the first set, obtains a second set of the ranging data during the stationary condition, and validates the one or more transformation parameter values based on a relationship between the filtered set of the image data and the second set of the ranging data........................Please see Fig. 6 and 12 and Para. [0069, and 0073-0076]. Abstract.
JIANG et al. (US 20220111839 A1)- An automated vehicle assistance system is provided for supervised or unsupervised vehicle movement. The system includes a control system and a first sensor system. The first sensor system may receive first image data of a scene and may output a first disparity map and a first confidence map based on the first image data. The control system may output a video stream based on the first disparity map and the first confidence map. The vehicle assistance system also may include a second sensor system that receives second image data of at least a portion of the scene that outputs a second confidence map based on second image data. The video stream may include super-frames, with each super-frame including a 2D image of the scene, a depth map corresponding to the 2D image, and a certainty map corresponding to the depth map......................Please see Fig. 1-2, 5 and 7. Abstract.
KUNDU et al. (US 20200250984 A1)- Example implementations described herein are directed to depression detection on roadways (e.g., potholes, horizontal panel lines of a roadway, etc.) through using vision sensor to realize improved safety for advanced driver assistance systems (ADAS) and autonomous driving (AD). Example implementations described herein detect candidate depressions in the roadway in real time and adjust the control of the vehicle system according to the detected depressions......................Please see Fig. 3-4. Abstract.
MOTOHASHI et al. (US 20190019044 A1)- An image processing apparatus includes one or more processors; and a memory, the memory storing instructions, which when executed by the one or more processors, cause the one or more processors to generate vertical direction distribution data indicating a frequency distribution of distance values with respect to a vertical direction of a range image, from the range image having distance values according to distance of a road surface in a plurality of captured images captured by a plurality of imaging parts; estimate a plurality of road surfaces, based on the vertical direction distribution data; and determine a desired road surface, based on the estimated plurality of road surfaces......................Please see Fig. 2-3 and 8-9. Abstract.
SCHAMP et al. (US 20130251194 A1)- Objects in a range map image are clustered into regions of interest responsive to range as determined from either a separate ranging system or from a top-down transformation of a range map image from a stereo vision system. A relatively-central location of each region of interest is transformed to mono-image geometry, and the corresponding portion of an associated mono-image is searched radially outwards from the relatively-central location along a plurality of radial search paths, along which the associated image pixels are filtered using an Edge-Preserving Smoothing filter in order to find an edge of the associated object along the radial search path. Edge locations for each of the radial search paths are combined in an edge profile vector that provides for discriminating the object.....................Please see Fig. 7-11. Abstract.
UENO (US 20190355140 A1)- Systems, methods, and other embodiments described herein relate to a method of determining stereo depth of an object. One method includes obtaining an image captured with a stereo camera arrangement. The stereo camera arrangement can be installed in one of a vehicle and a robotic apparatus, for example. The image captures a portion of the environment associated with the stereo camera arrangement. The method can further include identifying an object in the image, determining an object class for the object, determining a size parameter of the object, determining a size parameter of the object class, determining a maximum disparity for the object with the size parameter of the object and the size parameter of the object class, and determining a stereo depth of the object based on the maximum disparity........................Please see Fig. 2-4. Abstract.
KAKEGAWA (US 20170220877 A1)- The purpose of the present invention is to provide an object detecting device which is capable of accurately detecting an object even far away, and of shortening processing time. Provided is an object detecting device (100), comprising: a disparity acquisition unit (116) which compares each image of two cameras (112, 113) and computes a disparity for each pixel; a near-far boundary setting unit (118) which, in a single image of one of the two cameras, sets a boundary (Rb) between a near region (R1) which is close to a vehicle (110) and a far region (R2) which is distant from the vehicle (110); a near object detecting unit (119) which detects objects (102, 104) of the near region (R1) on the basis of the disparity; and a far object detecting unit (120) which detects objects (103, 104) of the far region (R2) on the basis of the single image.......................Please see Fig. 1 and 5-8. Abstract.
TAKEMAE (US 20150269446 A1)- A boundary detection apparatus includes an acquisition unit, an extraction unit and a detection unit. The acquisition unit acquires a disparity image based on information obtained by capturing an image of a peripheral environment of a vehicle. The extraction unit extracts predetermined pixel regions from first and second pixel regions of the disparity image based on a disparity gradient direction of the first pixel region and a disparity gradient direction of the second pixel region. The detection unit detects a boundary of a step surface existing alongside a road by joining together at least some of the predetermined pixel regions extracted by the extraction unit. The first and second pixel regions sandwich the predetermined pixel region. An angle formed by the disparity gradient direction of the first pixel region and the disparity gradient direction of the second pixel region is within a predetermined angular range of a right angle......................Please see Fig. 1-4. Abstract.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to Aaron Bonansinga whose telephone number is (703) 756-5380 The examiner can normally be reached on Monday-Friday, 9:00 a.m. - 6:00 p.m. ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s
supervisor, Chineyere Wills-Burns can be reached by phone at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON TIMOTHY BONANSINGA/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/ Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights