DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Prior arts cited in this office action:
Yathirajam et al. (WO 2024062025 A1, hereinafter “Yathirajam”)
Beijbom et al. (CN 111160561 B, hereinafter “Beijbom”)
Oya (US 20240087100 A1, hereinafter “Oya”)
Hartmann et al. (EP 4250031A1, hereinafter “Hartmann”)
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/10/2026 has been entered.
Response to Arguments
Applicant’s Arguments/Remarks filed on 04/10/2026 have been fully considered and are not persuasive.
Applicant’s Arguments/Remarks: Applicant argues that claim 3 sets forth additional elements that makes it further distinguishable in its own right. Claim 3 recites that the three-dimensional polar coordinates include 1) a center coordinate, 2) a depth, 3) a first latitude dimension, 4) a first longitude dimension, 5) a second latitude dimension, and 6) a second longitude dimension. In other words, the three-dimensional polar coordinates include at least six parameters related to a center coordinate, depth, and latitude and longitude dimensions. In rejecting claim 3, the Office Action cites Oya paragraph [0032] and Beijbom paragraphs [0133]-[0139]. However, the Examiner is citing portions of Beijbom that describe cartesian coordinates and not polar coordinates. The cited portions of Beijbom do not describe the specific combination of elements of claim 3, including three-dimensional polar coordinates that include first and second latitude dimensions and first and second longitude dimensions. Accordingly, Beijbom does not disclose or suggest the specific combination of features of claim 3.
Examiner’s Response. Examiner disagrees with applicant assertion above that the combination of the cited prior arts does not teach or suggest applicant invention as claimed. Applicant tries to argue the reference separately while the rejection was based on the combination of the references. For example, Oya teaches Where an elevation angle is defined in an angle direction orthogonal to a polar angle direction in an azimuth of an object point within the three-dimensional space expressed by a polar coordinate system with a pole that is set to an optical axis of the optical system, a first ratio is defined as a ratio of a change rate of an image length relative to a polar angle on the first image relative to the object point to a change rate of an image length relative to the elevation angle on the first image relative to the object point, and a second ratio is defined as a ratio of a change rate of an image length relative to the polar angle on the second image relative to the object point to a change rate of an image length relative to the elevation angle on the second image relative to the object point, the second ratio is closer to 1 than the first ratio at a polar angle of 45° or more and the second ratio is 0.9 or higher and 1.1 or lower. In other words, the object or the bounding box is determined relative to a reference line or a point (Oya [0005], [0018]-[0022], fig. 1). And as applicant pointed out Beijbom teaches When the object is at the center each corner of the bounding box can be obtained and provided based on the corner or the 3D bounding box or any point along the bounding box. All the points are determined in relation to the center point or center line or corresponding angle. In the 3D space more point can always be used to specify the bounding surface.
The cited portions of Beijbom describe that "each data point of the plurality data points is represented by 3D spatial coordinates (x,y,z), a reflectance (r), and a timestamp (t). The system transforms each data point (x, y, Z, r, t) to a respective modified data point (Xoffsct, Yoffsct, Z, r, t, d), where Xoffset and Yoffset are measured based on a relative distance between each data point and the center of each non-empty point pillar, and d is the cylindrical Euclidean distance from the sensor to each data point." See Beijbom at [0133]. Thus, Beijbom describe elements in cartesian space including an offset from the pillar center, a height Z, a reflectance Γ, a timestamp t, and a Euclidean distance. As also explain the coordinate can be transform to polar or any suitable domain by one skill in the art. Applicant has not shown that it would be an undue burden to one of ordinary skill in the art based on the cited references to convert the coordinate to polar coordinate and perform the calculation accordingly.
Furthermore, applicant is reminded that the test for obviousness is not whether the features of a secondary reference may be bodily incorporated into the structure of the primary reference; nor is it that the claimed invention must be expressly suggested in any one or all of the references. Rather, the test is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981). In this case defining a plurality of points to generate a bounding surface based on a reference point or line, in order to locate objects in any given space.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 7-13, 15-21, 23-29 and 31 are rejected under 35 U.S.C. 103 as being unpatentable over Yathirajam et al. (WO 2024062025 A1, hereinafter “Yathirajam”) in view of Beijbom et al. (CN 111160561 B, hereinafter “Beijbom”), in view of Hartmann et al. (EP 4250031A1, hereinafter “Hartmann”) and in view of Oya (US 20240087100 A1, hereinafter “Oya”).
Regarding claims 1, 9, 17 and 25:
Yathirajam teaches a method for image processing (page 1 line 5-14, where Yathirajam teaches a computer-implemented method for automatic envi-ronmental perception based on multi-modal sensor data of a vehicle, wherein a first image of a first environmental sensor modality of the vehicle and a second image of a second environmental sensor modality of the vehicle are received), comprising:
receiving an image frame captured by a fisheye image sensor (Yathirajam page 1 line 5-14, where Yathirajam teaches a computer-implemented method for automatic envi-ronmental perception based on multi-modal sensor data of a vehicle, wherein a first image of a first environmental sensor modality of the vehicle and a second image of a second environmental sensor modality of the vehicle are received);
determining locations for one or more objects depicted within the image frame (Yathirajam page 11 lines 6-24, where Yathirajam teaches a result of the semantic segmentation task comprises a semantically segmented image. In the semantically segmented image, a respective pixel level object class, such as dynamic object, static object, road surface, lane marking, et cetera, is for example assigned to each pixel of the first image. According to several implementations, the at least one visual perception task comprises an object detection task and/or a depth estimation task);
determining bounding surfaces for the one or more objects, wherein the bounding surfaces comprise a three-dimensional representation of a region containing the one or more objects and are defined by three-dimensional spatial coordinates Yathirajam page 11 lines 19-30, Yathirajam teaches the reference coordinate system may also be a sensor coordinate system of one of the environmental sensor modalities. The extrinsic parameters may for example comprise three spatial coordinates specifying a position in the reference coordinate system and three angles specifying the orientation in the coordinate system. The angles may for example be Euler angles, which are denotes as yaw, roll and pitch angle, respectively. the result of the object detection task comprises position information for one or more bounding boxes for respective objects in the environment of the vehicle and a respective object class assigned to the object or the bounding box, respectively. This type of procedure can be applied to any other camera model or sensor model, for example for fisheye cameras. ) , the three-dimensional spatial coordinates being expressed relative to a location of the fisheye image sensor when the image frame was captured (Yathirajam page 11 lines 19-30, Yathirajam teaches in case of a rectangular bounding box, its position may be given by a center position of the rectangle or a corner position of the rectangle or another defined position of the rectangle. In this case, the size of the bounding box may be given by a width and/or height of the rectangle or by equivalent quantities. In particular, the intrinsic parameters define, how a point in the three- dimensional environment is mapped to a pixel in the two-dimensional image plane or sensor plane of the environmental sensor modality. For example, in case of a visible range camera or a thermal camera, the intrinsic parameters may comprise coordinates of a center of projection, commonly denoted as cx, cy, for example, and focal lengths, commonly denoted as fx, fy, for example. The intrinsic parameters may for example be described in terms of a corresponding camera matrix. The distortion parameters of an environmental sensor modality describe the distortion of a respective image generated by the environmental sensor modality, in particular compared to an undistorted two-dimensional representation of the environment. The distortion parameters define, in particular, how a pixel position (u', v') of the respective image is transformed to a corresponding pixel position (u, v) in an undistorted image, also denoted as rectified image. The distortion parameters may be given by a respective model for the environmental sensor modality, such as a pin-hole camera model or a fisheye camera model, et cetera); and
determining control instructions for a vehicle based on the bounding surfaces (Yathirajam page 13 lines 26-31, Yathirajam teaches the method further comprises generating at least one control signal for guiding the vehicle at least in part automatically depending on a result of the at least one visual perception task).
Yathirajam fails to explicitly teach a region containing the one or more objects and are defined by three-dimensional spatial coordinates.
However, Beijbom teaches A set of measurements includes a plurality of data points representing a plurality of objects in a three-dimensional (3D) space around the vehicle. Each of the plurality of data points is a set of 3D spatial coordinates. The location module 408 determines the AV location by calculating the location using data from the sensor 121 and data from the database module 410 (e.g., geographic data) (Beijbom [0004], [0087]).
Therefore, it would have been obvious to one or ordinary skill in the art before the effective filing date of the application to defined the location of the object by three-dimensional spatial coordinates relative to the camera in order to properly and more accurately locate the object in the 3D spatial environment especially when considering the camera mounted on the vehicle is fixed.
The combination fails to teach wherein the bounding surfaces comprise at least one curved bounding surface.
However, Hartmann teaches techniques for detecting objects in the environment around the vehicle. the computer system is configured for receiving a set of measured values from the sensor of the vehicle. The set of measurements comprises a plurality of data points, a plurality of object data point represented in the 3D space around the vehicle. In other words, curved object or surface is included. Furthermore, Hartmann teaches the bounding space is a 3D bounding space that is located within (i.e. defined by) a bounding surface or a 2D bounding space that is located within (i.e. defined by) a bounding "surface" that lies in a plane. Thus, the term "bounding surface" is defined for the purposes of describing the embodiments of the present disclosure as being a 2D surface defined by a set of coordinates that enclose a 3D bounding space or a 1D "surface" (i.e., a continuous line, e.g., a curved line that defines the circumference of a circle, as an exemplary non-limiting example) defined by a set of coordinates that enclose a 2D bounding space (Hartmann [0079]-[0080], [0137]-[0139).
Therefore, taking the teachings of Yathirajam, Beijbom and Hartmann as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to determine bounding surfaces including curved bounding surfaces, in order to be able to determine coordinate regarding objects having different shapes and sizes and/or the smallest bounding space possible.
The combination fails to teach where wherein the image frame is received as fisheye image data from the fisheye image sensor, the fisheye image data being in a fisheye coordinate space.
However, Oya teaches FIG. 1 is a schematic diagram illustrating projection in imaging a three-dimensional space represented by a polar coordinate system through an optical system 103 as a fisheye lens. A figure at the upper part illustrates a relationship between an object point 100 and an object plane 101a passing through an optical axis 104. A figure at the lower part illustrates a relationship viewed from the image plane 101 (Oya [0005], [0018], claim 1).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the application to obtain the image frame as fisheye image data from the fisheye image sensor, the fisheye image data being in a fisheye coordinate space, in order to transform the location of each object or bounding box to the corresponding image space or world coordinate and/or to facilitate further calculation and to take advantage of using polar coordinate because some calculations are better or easier when performed in polar coordinate and some are better when performed in cartesian (Oya [0022]-[0023]).
Regarding claims 2, 10, 18 and 26:
Yathirajam in view of Beijbom in view of Hartmann and in view of Oya teaches wherein the three-dimensional spatial coordinates are three-dimensional polar coordinates representing portions of a viewing area of the fisheye image sensor (Oya [0018], [0022]-[0023]).
Regarding claims 3, 11, 19 and 27:
Yathirajam in view of Beijbom, in view of Hartmann and in view of Oya teaches wherein, for each respective bounding surface of the bounding surfaces, the three-dimensional polar coordinates include a center coordinate for the respective bounding surface, depth of the respective bounding surface, a first latitude dimension of the respective bounding surface, a first longitude dimension of the respective bounding surface, a second latitude dimension of the respective bounding surface, and a second longitude dimension of the respective bounding surface (Oya [0019]-[0022], [0032], [0133]-[0139]).
Regarding claims 4, 12, 20 and 28:
Yathirajam in view of Beijbom, in view of Hartmann and in view of Oya teaches wherein the three-dimensional polar coordinates are converted into cartesian coordinates, and wherein the control instructions are determined based on the cartesian coordinates (Beijbom 0075]-[0076], where X and Y coordinate can be used which is a simple conversion to one of ordinary skill in the art to determine the control of the vehicle on the map (see us patent (12033482 B1) for example).
Regarding claims 5, 13, 21 and 29:
Yathirajam in view of Beijbom, in view of Hartmann and in view of Oya teaches wherein an encoder model is configured to determine the locations for the one or more objects and a decoder model is configured to determine the bounding surfaces for the one or more objects (Yathirajam page 3; Beijbom [0040], [0137]-[0138], [0150]).
Regarding claims 7, 15 and 23:
Yathirajam in view of Beijbom, in view of Hartmann and in view of Oya teaches further comprising training a model based on the bounding surfaces (Yathirajam page 1 line 5-14, page 6 line 17-35; Oya [0033]).
Regarding claims 8, 16 and 24:
Yathirajam in view of Beijbom in view of Hartmann and in view of Oya teaches wherein the locations are determined to identify pixels within the image frame that correspond to the one or more objects (Yathirajam page 11 lines 15-24).
Regarding claim 31:
Yathirajam in view of Beijbom in view of Hartmann and in view of Oya teaches wherein determining the bounding surfaces in the fisheye coordinate space includes determining the bounding surfaces using the fisheye image data without rectifying the fisheye image data ((Beijbom [0092]-[0093]; Oya [0005], [0018], [0022]-[0023], claim 1).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEDNEL CADEAU whose telephone number is (571)270-7843. The examiner can normally be reached Mon-Fri 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chieh Fan can be reached at 571-272-3042. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEDNEL CADEAU/Primary Examiner, Art Unit 2632 April 30, 2026
Prosecution Insights