Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: Fig. 7A, reference sign 702 mentioned in the specification (Para. 64) is absent, 502 is there instead. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-4, 6-9, 12, and 14 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ren et al. (International App. Pub. No. 2023/192752 A1, filed 4/1/2022, pdf attached).
Ren teaches a method of dynamic virtual sensor mapping comprising: receiving a request to obtain a first view associated with a system, the first view being associated with a first distortion key of a plurality of distortion keys (Pg. 13, Line 6, “a virtual camera may be placed in the 3D environment with a specified location and/or orientation and used to render a view of the textured 3D surface from the perspective of the virtual camera into a viewport; and/or the rendered view may be presented on a monitor visible to occupants ( e.g., driver) of the vehicle.”; Pg. 28, Line 7, “In some embodiments, the viewport may be selected based on… a remote command (orienting the viewport in a direction instructed by a remote command)”); obtaining video sensor data from a plurality of video sensors associated with the system (Pg. 14, Line 18, “In an example embodiment, at a high level, 3D object detection may be performed to detect objects from sensor data (e.g., images…) representing an environment surrounding an ego-object such as a vehicle.”); applying the first distortion key to the video sensor data to obtain distorted video sensor data (Pg. 27, Line 6, “Taking an example implementation in which the sensor(s) are camera(s) (e.g., fisheye cameras) that capture image data 105 (e.g., fisheye images), the dewarp module 110 may remove distortion (e.g., barrel distortion, radial distortion) from the image data 105 using any known technique.”); obtaining additional sensor data from a plurality of additional sensors associated with the system (Pg. 22, Line 6, “Taking a surround view visualization as an example, sensor data such as image or LiDAR data may be projected onto a 3D representation”); applying the first distortion key to the additional sensor data to obtain distorted additional sensor data (Pg. 27, Line 19, “For example, the adaptive 3D bowl generator 150 may use sensor data from the one or more sensor(s) to generate an adaptive 3D bowl 170 that models the environment with a shape that depends on distance and/or direction to detected objects in the environment”); combining the distorted video sensor data together with the distorted additional sensor data to generate combined distorted video data (Pg. 27, Line 28, “The projection module 175 may project the stitched image generated by the stitching module 120 to generate a projection image (e.g., a top-down projection image) using estimated depth values (e.g., generated by the depth estimator 155), depth values sampled from a fixed 3D bowl, depth values sampled from the adaptive 3D bowl 170,”); and transmitting the combined distorted video data (Pg. 28, Line 15, “Additionally or alternatively, a representation streaming engine 195 may stream the surround view visualization, the sensor data, and/or some other representation of the environment in and/or around the ego-object to a remote location”).
Regarding claim 2, Ren teaches all of the elements of claim 1, as stated above, as well as wherein the system comprises a driverless vehicle (Pg. 9, Line 17, “Although the present disclosure may be described with respect to an example autonomous vehicle 3500”).
Regarding claim 3, Ren teaches all of the elements of claim 1, as stated above, as well as wherein obtaining the video sensor data from the plurality of video sensors comprises stitching together two or more different sets of video sensor data from two or more of the plurality of video sensors to generate video sensor data having a wider field of view (FOV) than the video sensor data of any single given one of the plurality of video sensors (Pg. 10, Line 13, “Image data representing the environment may be aligned to create an aligned composite image or surface (e.g., a panorama, a 360° image, howl shaped surface) with regions of overlapping image data… the image data may be blended at the seams to create a stitched image or surface”).
Regarding claim 4, Ren teaches all of the elements of claim 3, as stated above, as well as wherein: the two or more different sets of video sensor data include first video sensor data generated by a first video sensor that has a first field of view (FOV) and second video sensor data generated by a second video sensor that has a second FOV that partially overlaps the first FOV; and stitching together the two or more different sets of video sensor data to generate wide-angle video sensor data (Pg. 10, Line 13, “Image data representing the environment may be aligned to create an aligned composite image or surface (e.g., a panorama, a 360° image, howl shaped surface) with regions of overlapping image data… the image data may be blended at the seams to create a stitched image or surface”) comprises removing a portion of the first or second video sensor data that overlaps a corresponding portion of the other of the second or first video sensor data (Pg. 11, Line 31, “In some cases, a seam may be steered through each overlapping region to avoid or otherwise minimize cutting through or intersecting pixels of a salient region detected by multiple sensors”, intersecting pixels are cut).
Regarding claim 6, Ren teaches all of the elements of claim 1, as stated above, as well as wherein obtaining the additional sensor data from the plurality of additional sensors associated with the system comprises obtaining the additional sensor data from one or more of an accelerometer, a gyroscope, a global positioning system (GPS) device, a radar device, a LIDAR device, a thermal infrared device, or an ultrasonic device (Fig. 35A).
Regarding claim 7, Ren teaches all of the elements of claim 1, as stated above, as well as wherein applying the first distortion key to the additional sensor data to obtain distorted additional sensor data comprises formatting, arranging, or otherwise processing some or all of the additional sensor data for combination with the distorted video sensor data. (Pg. 27, Line 19, “For example, the adaptive 3D bowl generator 150 may use sensor data from the one or more sensor(s) to generate an adaptive 3D bowl 170 that models the environment with a shape that depends on distance and/or direction to detected objects in the environment”, a plurality of sensors are disclosed as being used, indicating formatting, arranging, or processing being performed).
Regarding claim 8, Ren teaches all of the elements of claim 1, as stated above, as well as wherein transmitting the combined distorted video data comprises transmitting the combined distorted video data to a recipient device, the recipient device including a teleoperator workstation (Pg. 5, Line 21, “streaming to a remote or fleet operator”; Pg. 28, Line 15, “Additionally or alternatively, a representation streaming engine 195 may stream the surround view visualization, the sensor data, and/or some other representation of the environment in and/or around the ego-object to a remote location”).
Regarding claim 9, Ren teaches all of the elements of claim 1, as stated above, as well as wherein the combined distorted video data includes an area of focus corresponding to the requested first view that has been expanded in the combined distorted video data compared to in the video sensor data obtained from the plurality of video sensors (Pg. 36, Line 1, “In some embodiments, a designated (e.g., active) viewport may be assigned a viewport cost map, such as the one illustrated in FIG. 4F, which assigns a measure of saliency (e.g., higher values) to pixels in the center of the viewport and assigns a measure of non-saliency (e.g., lower values) to pixels toward the edge or boundary of the viewport… As a result, an optimal seam location may be dynamically determined or adjusted to avoid or minimize intersecting or bisecting salient regions, which may effectively identify the shortest candidate seam that crosses the fewest (e.g., projected) pixels of the viewport cost map and/or that encourages placing a seam towards the boundary of a viewport's field of view.”, By using a viewport cost map to determine seam placement and length, the view is inherently expanded by utilizing an optimal seam location).
Regarding claim 12, Ren teaches all of the elements of claim 1, as stated above, as well as wherein: the wide-angle view comprises a front wide-angle view with a field of view (FOV) of at least 180 degrees; the combined distorted video data further includes a second wide-angle view; and the second wide-angle view comprises a rear wide-angle video feed with a FOV of at least 180 degrees (Pg. 26, Line 18, “Generally, any suitable sensor may be used, such as one or more of the stereo camera(s) 3568, wide-view 20 camera(s) 3570 (e.g., fisheye cameras)… surround camera(s) 3574 (e.g., 360° cameras)”; Pg. 28, Line 24, “This may include, for example and without limitation, stitching individual frames captured using multiple sensors in an environment with at least partially overlapping fields of view such that each frame provides a larger field of view than is captured in any individual frame of the frames to be stitched together, such as a 180° or 360° view.”, fisheye lens cameras are well-known in the art and can typically range at 180 degrees, and surround cameras are disclosed which have a FOV of at least 180 degrees).
Regarding claim 14, the non-transitory computer readable storage medium (Pg. 46, Line 1, “The methods may also be embodied as computer-usable instructions stored on computer storage media.”) performs the same function as the method of claim 1. It is rejected under the same analysis.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 10-11 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Ren et al.
Regarding claim 10, Ren teaches all of the elements of claim 9, as stated above, as well as wherein the combined distorted video data further includes an unimportant area of little or no relevance to the system and that is outside the area of focus, the unimportant area having been compressed in the combined distorted video data compared to in the video sensor data obtained from the plurality of video sensors (Pg. 23, Line 27, “may support scalable streams (e.g., using scalable audio or video coders to adjust encoding quality based on bandwidth), and/or may implement a Quality of Service (QoS) mechanism to assign a priority to certain streamed content and/or commands and manage the stream accordingly. In some embodiments, the transport system implements a stream hierarchy that prioritizes particular types of content.”, See analysis of claim 9 above. One of ordinary skill in the art would recognize that a Quality of Service mechanism as described would leave it obvious to perform compression of irrelevant video sensor data, such as low cost areas in the viewport cost map).
Regarding claim 11, Ren teaches all of the elements of claim 1, as stated above, as well as wherein the combined distorted video data includes both a focused view corresponding to the requested first view and a wide-angle view (Pg. 28, Line 7, “In some embodiments, the viewport may be selected based on… a remote command (orienting the viewport in a direction instructed by a remote command)”; Pg. 26, Line 18, “Generally, any suitable sensor may be used, such as one or more of the stereo camera(s) 3568, wide-view camera(s)”; Pg. 79, Line 25, “In some embodiments, the directional audio and/or the directed viewport rendering may be presented in association with some alarm, presented picture-in-picture (e.g., with some other video feed such as one pointing in the direction of travel), and/or otherwise”, multiple video feeds are displayed, as well as the ability to focus the viewport in a specified direction and use wide-view cameras for further viewpoints. One of ordinary skill in the art would understand the benefit of including a secondary wide-angle view alongside the focused view to increase the awareness of the vehicle operator, similar to having rearview and sideview mirrors in a typical vehicle).
Regarding claim 13, Ren teaches all of the elements of claim 11, as stated above, as well as wherein when the combined distorted video data is rendered, the focused view occupies more of a display than the wide-angle video feed (See analysis for claim 11 above. It would have been obvious to set the focus view to occupy more of the screen, especially when considering the viewport cost function as disclosed in Pg. 36).
Claim(s) 5 and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ren et al. (International App. Pub. No. 2023/192752 A1, filed 4/1/2022, pdf attached) in view of Foote et al. (US Patent No. 7,015,954 B1, published 2006).
Regarding claim 5, Ren teaches all of the elements of claim 1, as stated above, as well as wherein applying the first distortion key to the video sensor data to obtain distorted video sensor data comprises one or more of: generating a UV map of a 3D model modeled as at least partially surrounding the system (Pg. 60, Line 2, “As such, the stitched projection image 1950 may be mapped onto the adaptive 3D bowl 1930 (e.g. using UV mapping) to generate a textured 3D bowl 1960, and a view of the textured 3D bowl 1960”).
Ren does not explicitly disclose linearizing the UV map; or warping the UV map or the linearized UV map. However, they do render it back into a 2D image/view.
Foote teaches wherein applying the first distortion key to the video sensor data to obtain distorted video sensor data comprises one or more of: generating a UV map of a 3D model modeled as at least partially surrounding the system (Col. 9, Lin 63, “To calculate a pixel value in the warped coordinate system x, y, the above equations are inverted by solving for u, v in terms of x, y. This allows for what is termed "inverse mapping." For every pixel in the warped coordinate system, the corresponding pixel in the unwarped system is found and its value is copied.”); linearizing the UV map; or warping the UV map or the linearized UV map (Col. 10, Line 1, “The coefficients (of equation 1, for example) are stored in a table and utilized to warp images "on the fly.””).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ren to incorporate the teachings of Foote to include linearizing or warping the UV map. Linearizing or warping a UV map is standard practice in the art, yet it is not explicitly disclosed in the method of Ren. Foote discloses a method that linearizes or warps the generated UV map for improved accuracy of the change in coordinate system between images. One of ordinary skill in the art would have understood this to be a routine optimization.
Regarding claim 15, Ren teaches all of the elements of claim 1, as stated above, and when modified in view of Foote above also teaches a method, comprising unwrapping a 3D model modeled as surrounding a driverless vehicle to generate a UV map (Ren; Pg. 60, Line 2, “As such, the stitched projection image 1950 may be mapped onto the adaptive 3D bowl 1930 (e.g. using UV mapping) to generate a textured 3D bowl 1960, and a view of the textured 3D bowl 1960”); receiving wide-angle video data captured by one or more video sensors of the driverless vehicle (Ren; Pg. 26, Line 18, “Generally, any suitable sensor may be used, such as one or more of the stereo camera(s) 3568, wide-view camera(s)”); receiving a request for a primary view in the wide-angle video data from a requestor (Ren; Pg. 28, Line 7, “In some embodiments, the viewport may be selected based on… a remote command (orienting the viewport in a direction instructed by a remote command); warping the UV map to expand a region corresponding to the primary view (Foote; Fig. 8, Col. 10, Line 16, “Other embodiments include different types of spatial transformations to warp patches from captured images (u, v coordinate system) to a composite grid (x, y coordinate system. Any spatial transformation altering the captured images to fit into a composite grid would be consistent with the present invention. For example, affine, or perspective transformations may be utilized… In addition, hatched columns 851 and 876 (seen in both camera images) are blended, while column 877 and column 852 may either utilized as is, discarded, or blended.”); for each image in a sequence of images in the wide-angle video data, painting the image onto the warped UV map to generate a warped 2D texture map in which a relative size of the primary view is greater in the warped 2D texture map than in the image (Foote; Col. 11, Line 26, “FIGS. 10 and 11 show how multiple images can be integrated into a high-resolution composite. FIG. 10 illustrates images taken from cameras 1000 (CH1), 1010 (CH2), 1020 (CH3), and 1030 (CH4). Each of the images includes a set of quadrilateral grids to be warped into a final panoramic image.”); and transmitting warped video data comprising a sequence of warped 2D texture maps to the requestor (Ren; Pg. 5, Line 21, “streaming to a remote or fleet operator”; Pg. 28, Line 15, “Additionally or alternatively, a representation streaming engine 195 may stream the surround view visualization, the sensor data, and/or some other representation of the environment in and/or around the ego-object to a remote location”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ren to incorporate the teachings of Foote to include warping UV maps to expand a region corresponding to a primary view. One of ordinary skill in the art would understand that applying Foote’s UV unwrapping and warping/painting techniques to Ren’s vehicle-centric projection system in order to generate requested operator views would have predictably yielded improved visualizations for remote operators.
Regarding claim 16, Ren as modified above teaches all of the elements of claim 15, as well as further comprising linearizing the UV map prior to warping the UV map, wherein warping the UV map comprises warping the linearized UV map (Foote; Col. 10, Line 16, “Other embodiments include different types of spatial transformations to warp patches from captured images (u, v coordinate system) to a composite grid (x, y) coordinate system. Any spatial transformation altering the captured images to fit into a composite grid would be consistent with the present invention. For example, affine, or perspective transformations may be utilized… In another embodiment, such abnormalities are corrected by increasing the number of patches (registration points), and the resulting warped images making a better approximation of the actual scene being imaged”).
Regarding claim 17, the recited elements perform variably the same function as that of claims 10 and 11 except it is in the context of UV maps as described in relation to claims 15 and 16 above. It is rejected under the same analyses.
Regarding claim 18, Ren as modified above teaches all of the elements of claim 15, as stated above, as well as further comprising adding additional sensor data to the warped video data prior to transmitting the warped video data, the additional sensor data including at least one of path routing data, GPS data, radar data, driverless vehicle speed data, driverless vehicle directional data, objects in motion data relative to the driverless vehicle, or driverless vehicle trajectory data (Pg. 79, Line 25, “In some embodiments, the directional audio and/or the directed viewport rendering may be presented in association with some alarm, presented picture-in-picture (e.g., with some other video feed such as one pointing in the direction of travel), and/or otherwise.”, non-limiting examples of a presentation to a remote operator are provided. Given the inclusion of numerous sensors such as radar, GPS, lidar, IMU, etc., it would have been obvious to include presentation of these details during transmission).
Regarding claim 19, Ren as modified above teaches all of the elements of claim 15, as stated above, as well as receiving a request for an additional view in the wide-angle video data from the requestor (Ren; Pg. 4, Line 19, “In some embodiments, a state machine is used to select between a default seam placement or dynamic seam placement that avoids salient regions, and to enable and disable dynamic seam placement based on speed of ego-motion, direction of ego-motion, proximity to salient objects, active viewport, driver gaze, and/or other factors”, the system can dynamically modify the presented view based on user interest or request); further warping the UV map to expand a region corresponding to the additional view (Foote; Col. 12, Line 10, “In this system, we can select one or more normal-resolution "virtual camera" images from the panoramic image. Mechanical cameras are constrained by the fact that they can be pointed in only one direction. A camera array suffers no such limitation; an unlimited number of images at different pans and zooms can be extracted from the panoramic image.”); for each image in a second sequence of images in the wide-angle video data, painting the image onto the further warped UV map to generate a further warped 2D texture map in which both a relative size of the primary view and a relative size of the additional view are greater in the further warped 2D texture map than in the image (Col. 12, Line 10, cited above, wherein they disclose zooming of the warped 2D texture map); and transmitting warped video data comprising a sequence of further warped 2D texture maps to the requestor (Ren; Pg. 5, Line 21, “streaming to a remote or fleet operator”; Pg. 28, Line 15, “Additionally or alternatively, a representation streaming engine 195 may stream the surround view visualization, the sensor data, and/or some other representation of the environment in and/or around the ego-object to a remote location”, Same rationale as claim 15 above).
Regarding claim 20, the recited non-transitory computer readable storage medium performs the same function as the method of claim 15. It is rejected under the same analysis.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID A WAMBST whose telephone number is (703)756-1750. The examiner can normally be reached M-F 9-6:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID ALEXANDER WAMBST/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698