DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-15 are pending for examination in the Application No. 18/421,211 filed January 24th, 2024.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed as foreign Patent Application No. JP 2023-050934, filed on March 28th, 2023.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on January 24th, 2024, is in compliance with the provisions of 37 CFR 1.97. Accordingly, the IDS is being considered and attached by the examiner.
Specification
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 4, 6-7, and 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Notz (EP 3852063 A1) in view of Choi et al. (Choi; US 2020/0143557 A1).
Regarding claim 1, Notz discloses an object detection device which extracts an object from an image acquired by an imaging device and calculates a position of the object in a real world coordinate system (para(s). [0058] and [0060], recite(s)
[0058] “Known camera systems use machine learning models to detect objects, e.g. vehicles, on images. Such machine learning models, for example, a characterization of an environment by determining 2D or 3D bounding boxes around the objects of interest. The objects thus can be localized in images coordinates but an object's position in geographical coordinates may remain unknown.”
[0060] “A basic idea of the present disclosure is to determine an object's position in world, road or geographical coordinates based on a homography between an image plane of a camera and a road surface.”
, where “determining 2D or 3D bounding boxes around the object of interest” is extracting an object from an image acquired by an imaging device (e.g., “camera systems”)), the object detection device comprising:
an object extraction circuitry which extracts the object from the image and outputs a rectangle enclosing the object in a circumscribing manner (para(s). [0089], recite(s)
[0089] “By way of example, Fig. 3 shows further images 316 of the image data. A skilled person having benefit from the present disclosure will appreciate that the data processing circuitry 120 can detect objects in the image 316 using a trained machine learning model and determine 2D or 3D bounding boxes 172 around the vehicles 170, as depicted in the image 316. Those bounding boxes 172 can be used for an approximate characterization of the vehicles, as stated in more detail below.”
, where “2D… bounding boxes” are rectangles enclosing the object in a circumscribing manner);
a direction calculation circuitry which calculates a direction, on the image, of the object extracted by the object extraction circuitry (para(s). [0099], recite(s)
[0099] “In some applications of the camera system 100, the camera system 100 can be used to determine multiple trajectories 178 and/or velocities of multiple vehicles 170 to analyze a driving behavior of the vehicles 170. For example, this may enable a machine learning based processor circuitry to learn a driving behavior of the vehicles 170 at merging lanes or in case of a lane change.”
, where determining “trajectories or velocities” are calculating directions of the extracted objects on the image); and
a bottom area calculation circuitry which calculates bottom areas of the object on the image and in the real world coordinate system, using a width of the rectangle(para(s). [0090] and [0095], recite(s)
[0090] “The data processing circuitry 120 can determine a width 174 and/or a length 176 of the bounding boxes 172.”
[0095] “Furthermore, if 2D bounding boxes are used the width of the object can be determined, and if 3D bounding boxes are used both the width and the length of the vehicles 170 can be determined.”
, where determining the “width” and “length” of the object in the 2D bounding boxes to determine 3D bounding boxes is calculating a bottom area (i.e., the “width” and “length” of a 3D bounding box is a bottom area of the object) of the object on the image and in the real world coordinate system using at least a width of the rectangle outputted from the object extraction circuitry), wherein
the bottom areas include positions, sizes, and directions of the object on the image and in the real world coordinate system, respectively (para(s). [0097-0098], recite(s)
[0097] “In some embodiments of the present disclosure, the objects/vehicles 170 can be tracked over several camera frames (e.g. multiple images of the image data).”
[0098] “As illustrated in Fig. 4a and Fig. 4b , this allows the data processing circuitry 120 to determine trajectories 178 of the vehicles 170 in the geographical coordinate system 152. For this, the data processing circuitry may interpolate subsequent positions of the vehicles 170. To this end, the data processing circuitry 120 can determine the subsequent positions from subsequent images of the image data in the geographical coordinate system, as described above.”
, where the “3D bounding boxes” (which include bottom areas) are tracked over multiple images of the image data include “determin[ing] trajectories… of the vehicles… in the geographical coordinate system” is the bottom areas (e.g., the “3D bounding boxes”) including positions, sizes, and directions of the object on the image and in the real world coordinate system).
Where Notz does not specifically disclose
a bottom area calculation circuitry which calculates bottom areas of the object on the image and in the real world coordinate system, using a width of the rectangle outputted from the object extraction circuitry and the direction of the object on the image calculated by the direction calculation circuitry;
Choi teaches in the same field of endeavor of extracting an object in an image by enclosing the object in a circumscribing manner and calculating a direction of the extracted object on the image
a bottom area calculation circuitry which calculates bottom areas of the object on the image and in the real world coordinate system, using a width of the rectangle outputted from the object extraction circuitry and the direction of the object on the image calculated by the direction calculation circuitry (para(s). [0003], [0027], [0029], [0030], [0043], and [0051], recite(s)
[0003] “Object detection techniques are for detecting a region containing an object from an image. For example, a two-dimensional (2D) bounding box that surrounds an object may be detected from a 2D image using an object detection technique. A 2D bounding box may be specified by the location and size of the 2D bounding box in an image. Object detection techniques may be performed through image processing based on a neural network. In addition, a three-dimensional (3D) bounding box refers to a volume surrounding an object in a 3D coordinate system and may be, e.g., specified by the location, size, and direction of the 3D bounding box in the 3D coordinate system. Applications requiring 3D bounding boxes may include, e.g., driving applications.”
[0027] “The 2D bounding boxes may be rectangular and may be specified in various ways. For example, the 2D bounding boxes may be specified using the coordinates of four corner points. Alternatively, the 2D bounding boxes may be specified by a location-size combination. A location may be expressed by the coordinate of a corner or a center point, and a size may be expressed by a width or a height.”
[0029] “The 3D bounding box may be a rectangular parallelepiped and may be specified in various ways. For example, the 3D bounding box may be specified using the coordinates of eight corner points. Alternatively, the 3D bounding box may be specified using a combination of locations and sizes. A location may be expressed by the coordinate of a corner point or the coordinate of a center point on a bottom surface, and a size may be expressed by a width, a length, or a height. A direction may be represented by a direction vector of a line normal to a surface. A direction vector may correspond to the degree (e.g., yaw, pitch, roll) of rotation of the 3D bounding box from three axes of the 3D coordinate system (e.g., x-axis, y-axis, z-axis). Direction may also be referred to as orientation.”
[0030] “Since 2D images do not include depth information in the z-axis direction, the 3D bounding box 150 may be detected from the 2D bounding boxes using projective geometry. Projective geometry involves properties in a 2D image that may not vary when a geometric object undergoes projective transformation.”
[0043] “In operation 240 , after the volume direction is detected on the basis of the iterative search results in operation 230 , a detected volumetric region that contains the objection of the 2D image may be specified by its location, size, and orientation in the 3D coordinate system. As described above, the method of specifying the volume may be variously modified, e.g., to a method of specifying the volume using the coordinates of eight corner points.”
[0051] “In FIG. 6A, coordinate (x, y, z) corresponds to a 3D coordinate in the 3D coordinate system. Assuming that a center point 610 of a lower surface of a volume is located at the origin of the 3D coordinate system, the 3D coordinates of eight corners of the volume may be expressed by the size of the volume. For example, if the width of the volume is w, the length of the volume is 1 , and the height of the volume is h, the 3D coordinates of four corners of the lower surface of the volume may be (−w/2, 0, −½), (w/2, 0, −½), (w/2, 0, ½), and (−w/2, 0, ½), and the 3D coordinates of four corners of an upper surface of the volume may be (−w/2, −h, −½), (w/2, −h, −½), (w/2, −h, ½), and (−w/2, −h, ½).”
, where a “three-dimensional (3D) bounding box” may be specified by “location, size, and direction” is calculating at least a bottom area of the object (e.g., “lower surface of the volume” or “bottom surface”) using at least a width of the rectangle outputted from the object extraction circuitry (e.g., “width of the volume” from a “2D bounding box” of the same object) and the direction of the object calculated by the direction calculation circuitry (e.g., “orientation in the 3D coordinate system”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Notz to incorporate calculating the bottom areas of the object on the image in the real world coordinate system using at least the width of the rectangle outputted from the object extraction circuitry and the direction of the object on the image calculated by the direction calculation circuitry to improve detecting a 3D object from a 2D image in the real world coordinate system even if the object is partially visible in the 2D image as taught by Choi (para(s). [0032], recite(s)
[0032] “If a region of an object detected from a 2D image does not entirely include the object, a volume of the object may not be exactly detected in the 3D coordinate system. However, according to an embodiment, even if the object is partially hidden or cut off in the 2D image, the object is to be expressed as a complete object in a 3D coordinate system. Embodiments below describe techniques for accurately detecting a volume in a 3D coordinate system even if an object is partially hidden or cut off in a 2D image by iteratively searching for candidates for the direction of the volume in a 3D coordinate system based on projective geometry.”
).
Regarding claim 4, Notz in view of Choi discloses the object detection device according to claim 1, wherein Notz further discloses the object device according to claim 1 further comprising an object direction map in which an object direction is defined in accordance with a position in the real world coordinate system (para(s). [0080] and [0103], recite(s)
[0080] “In other words, the digital map 160 may represent a globally accurate bird's-eye view perspective, where each location on the digital map 160 is mapped to an exact location on the world.”
[0103] “Subsequently, the trajectories of the vehicles 170 can be mapped onto another HD map. This allows to analyze the driving behavior of different road users and their interactions with each other. Such a behavior analyzation can be useful for a development of autonomous driving.”
, where the “HD map” representing “exact location[s] on the world” comprising of mapping the “trajectories of the vehicles” is an object direction map), wherein
the direction calculation circuitry calculates the direction, on the image, of the object extracted by the object extraction circuitry, using the object direction map (para(s). [0080] and [0103]—see preceding citations immediately above—, where mapping the “trajectories of the vehicles” using an “HD map” is calculating the direction on the image of the object extracted using an object direction map).
Regarding claim 6, Notz in view of Choi discloses an object detection system comprising:
the object detection device according to claim 1 (Notz in view of Choi—see the rejection of claim 1 above); and
the imaging device (Notz further discloses in para(s). [0010]:
[0010] “According to a first aspect the present disclosure relates to a camera system for a characterization of an environment. The camera system comprises a camera configured to record image data of the environment and a data processing circuitry. …”
, where a “camera” is an imaging device).
Regarding claim 7, Notz in view of Choi discloses the object detection system according to claim 6, wherein Notz further discloses
the imaging device includes a road side device provided with a camera (para(s). [0011], recite(s)
[0011] “The environment, for example, includes a road, merging lanes, a crossroad or other public areas. For this, the camera may be installed at an infrastructure of the environment, such as buildings, traffic lights, overpasses, bridges or other static structures.”
, where the camera system includes installing a camera “at an infrastructure of the environment” overseeing an environment such as a “road” is the imaging device including a road side device provided with a camera).
Regarding claim 12, the claim recites similar limitations to claim 6 and is rejected for similar rationale and reasoning (see the analysis for claim 6 above).
Regarding claim 13, the claim recites similar limitations to claim 7 and is rejected for similar rationale and reasoning (see the analysis for claim 7 above).
Claims 2-3, 8-9, and 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over Notz in view of Choi as applied to claim 1 above, and further in view of Lee et al. (Lee; US 2021/0042955 A1).
Regarding claim 2, Notz in view of Choi discloses the object detection device according to claim 1, wherein Choi further teaches
the bottom area calculation circuitry calculates the bottom areas of the object on the image and in the real world coordinate system, further using one of a width of the object, a length thereof(para(s). [0003], [0027], [0029], [0030], [0043], and [0051]—see citations in claim 1 limitation “a bottom area…” above—, where the bottom areas of the object (e.g., “lower surface of the volume” or “bottom surface” determined by “width” and “length”) on the image in a “3D coordinate system” is calculating bottom areas of the object on the image and in the real world coordinate system further using one of a width and length of the object).
Where Notz in view of Choi does not specifically disclose
the bottom area calculation circuitry calculates the bottom areas of the object on the image and in the real world coordinate system, further using one of a width of the object, a length thereof, and a ratio of the width and the length in the real world coordinate system;
Lee teaches in the same field of endeavor of calculating a bottom area of an object on an image
the bottom area calculation circuitry calculates the bottom areas of the object on the image and in the real world coordinate system, further using one of a width of the object, a length thereof, and a ratio of the width and the length in the real world coordinate system (para(s). [0026-0027] and [0069-0070], recite(s)
[0026] “A full length may denote a front-rear length of a vehicle (e.g., a full length of a vehicle from the “hood” of the vehicle to the “trunk” of the vehicle if the vehicle is a passenger sedan). That is, the full length may represent a horizontal length when a camera faces a target vehicle in a lateral direction (e.g., when the camera is facing the side doors of the target vehicle).”
[0027] “A full width may denote a left-right length of a vehicle (e.g., the distance between the “driver-side doors” and the “passenger-side doors” of the vehicle if the vehicle is a passenger sedan). That is, the full width may represent a horizontal length when a camera faces a target vehicle in a rear direction (e.g., when the camera is facing the “trunk” of the target vehicle).”
[0069] “According to various example embodiments, in a case where a k value is arbitrarily determined, a α value may be calculated. The k value may be a ratio of the full width of the target vehicle 310 to the full length of the target vehicle 310 and may differ for each vehicle manufacturer, vehicle classification type, and/or vehicle model, etc.”
[0070] “Referring to Table 1, it may be seen that the k value, which is a ratio of a full width to a full length (e.g., k is a ratio of a real-world width to a real-world length), differs for each vehicle type, but the full width values in Table 1 do not exceed a maximum of 2.5 m. A desired maximum full width value and a desired minimum full width value (e.g., a desired full width value range) may be set to desired values, such as values set in accordance with laws and/or regulations in various countries or territories related to the size and/or dimensions of vehicles. Therefore, the host vehicle 200 may set the k value to a desired value (and/or an arbitrary value). For example, the host vehicle 200 may set the k value to 2.5 and may then calculate the α value.”
, where a “a ratio of a full width to a full length” is calculating the bottom areas of the object on the image (e.g., a vehicle in an image) further using a ratio of the width and the length in the real world coordinate system (e.g., “ratio of a real-world width to a real-world length”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Notz in view of Choi to incorporate calculating the bottom areas of the object on the image further using a ratio of the width and the length in the real world coordinate system to improve calculating bottom areas of the object by constraining the length and width of the object to real-world widths and lengths of vehicle types as taught by Lee above.
Regarding claim 3, Notz, as modified by Choi and Lee, discloses the object detection device according to claim 2, wherein Lee further teaches the object detection device according to claim 2
the object extraction circuitry includes a type determination circuitry for determining a type of the extracted object, and outputs the type of the object determined by the type determination circuitry, as well as outputting the rectangle enclosing the object in the circumscribing manner (para(s). [0085] and [0092], recite(s)
[0085] “In the above-described example embodiments, an example where the k value is set to a desired and/or an arbitrary value has been described, but the example embodiments are not limited thereto. The ISP 120 may identify a type of the target vehicle 310 detected through image processing (e.g., identifying the type of the target vehicle based on the shape of the vehicle or identifying characteristics of the vehicle using image processing, etc.). For example, an identifiable vehicle type may include a car, a special vehicle, a bus, etc. Examples of a car may include a compact car, a quasi-medium car, a medium car, a full-size car, a medium SUV, etc., and examples of a special vehicle may include a truck, a refrigerated vehicle, a tractor-trailer, etc. The ISP 120 may identify a type of the target vehicle 310 and may set a desired and/or predetermined k value on the basis of the identified type. For example, when the identified type corresponds to a car, the ISP 120 may set a k value to a desired and/or an arbitrary value between 2.2 and 2.7, but the example embodiments are not limited thereto. As another example, when the identified type corresponds to a special car or a bus, the ISP 120 may set a k value to desired and/or an arbitrary value between 3.1 and 4.5, but the example embodiments are not limited thereto. When the ISP 120 identifies a type of the target vehicle 310 and sets a k value within a range corresponding to the identified type, a possibility that a world width outside the minimum world width to the maximum world width is calculated may be considerably reduced.”
[0092] “Referring to FIG. 7, the ISP 120 may generate, determine, calculate, and/or obtain a bounding box 320 of a target vehicle 310 in operation S 701 . The bounding box 320 may be performed by an artificial intelligence (AI) algorithm including machine learning and/or deep learning, etc., but the example embodiments are not limited thereto.”
, where “identify[ing] a type of the target vehicle” in addition to outputting a “bounding box” is outputting a type of the extracted object and a rectangle enclosing the object), and
the bottom area calculation circuitry calculates the bottom areas of the object on the image and in the real world coordinate system, using one of the width of the object, the length thereof, and the ratio of the width and the length in the real world coordinate system on the basis of the type of the object determined by the type determination circuitry (para(s). [0069-0070]—see citations in claim 2 above—, where calculating the bottom areas on the image (e.g., a length and width of a vehicle representing a bottom surface of a 3D bounding box or a vehicle viewed from above) includes using “a ratio of a full width to a full length” based on the “type” of vehicle is calculating the bottom areas of the object using a ratio of the width and the length in the real world coordinate system based on at least the type of object determined).
Regarding claim 8, the claim recites similar limitations to claim 6 and is rejected for similar rationale and reasoning (see the analysis for claim 6 above).
Regarding claim 9, the claim recites similar limitations to claim 7 and is rejected for similar rationale and reasoning (see the analysis for claim 7 above).
Regarding claim 10, the claim recites similar limitations to claim 6 and is rejected for similar rationale and reasoning (see the analysis for claim 6 above).
Regarding claim 11, the claim recites similar limitations to claim 7 and is rejected for similar rationale and reasoning (see the analysis for claim 7 above).
Claims 5 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Notz in view of Choi as applied to claim 1 above, and further in view of Kulaste et al. (Kulaste; “Camera Based Framework for Collision Avoidance in Intelligent Vehicles,” 2022).
Regarding claim 5, Notz in view of Choi discloses the object detection device according to claim 1, wherein Kulaste teaches in the same field of endeavor of calculating a direction of an object on the image further comprising an object direction table in which directions on the image and in the real world coordinate system are defined in accordance with a longitudinal-transverse ratio of the rectangle (5th para. of section III(E) on pg. 4, recite(s)
[E. Direction Estimation of Vehicles] “Orientation provides the direction of movement of the vehicle. The partition of frame provides the position of the vehicle relative to the camera. Combining this information gives the accurate estimation of direction of the vehicle. Four orientation and five partition are combined with a ratio of width and length of the 2D points corresponding to 3D box of the vehicle. The width-length ratio provides more accurate results to estimate direction for which the vehicle is heading. Table I II III and IV provide the detailed direction estimation based on the orientation, position of vehicle in partition and width-length ratio of the 2D projected points of 3D box in image.”
, where “Table I II III and IV provide the detailed direction estimation based on the …width-length ratio of the 2D projected points of 3D box in image” are object direction tables in which directions on the image and in the real world coordinate system are defined in accordance with a longitudinal-transverse ratio of the rectangle (e.g., the “ratio of width and length of the 2D points corresponding to 3D box of the vehicle”)), wherein
the direction calculation circuitry calculates the direction, on the image, of the object extracted by the object extraction circuitry, using the object direction table (5th para. of section III(E) on pg. 4—see preceding citation immediately above—, where determining a “detailed direction estimation” of a vehicle using the object direction tables (e.g., “Table I II III and IV”) is calculating the direction of the object extracted (e.g., a vehicle in a “3D box”) using object direction tables).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Notz in view of Choi to incorporate an object direction table in which directions on the image and in the real world coordinate system are defined in accordance with a longitudinal-transverse ratio of the rectangle and using the object direction table to calculate the direction of the extracted object on the image to more accurately estimate the direction the extracted object on the image is heading as taught by Kulaste above.
Regarding claim 14, the claim recites similar limitations to claim 6 and is rejected for similar rationale and reasoning (see the analysis for claim 6 above).
Regarding claim 15, the claim recites similar limitations to claim 7 and is rejected for similar rationale and reasoning (see the analysis for claim 7 above).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JULIA Z YAO whose telephone number is (571)272-2870. The examiner can normally be reached Monday - Friday (8:30AM - 5PM).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571)270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.Z.Y./Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666