DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-12 and 15-22 are pending as of the remarks and amendments filed February 4, 2026.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1 and 10-12 are rejected under 35 U.S.C. 102(a)(1) as being clearly anticipated by Pekkucuksen et al. (hereafter referred to as “Pekkucuksen”, US 2020/0020085).
Regarding claim 1, Pekkucuksen discloses an image processing method, wherein the method comprises:
obtaining a plurality of frames of original images, wherein the plurality of frames of original images are images photographed for a same to-be-photographed scene, the plurality of frames of original images comprise a small field-of-view image and a large field-of-view image, and a field of view corresponding to the large field-of-view image comprises a field of view corresponding to the small field-of-view image (Fig. 2 and pg. [0016]-[0017] “the multiple images could be obtained from two different camera modules on the same device capturing the scene simultaneously”, “having both a wide-angle camera and a tele camera for example, a wide frame will have a larger field of view than the tele frame, but texture detail and edge clarity will be less than that of the tele frame”. The tele frame and the wide frame in the reference corresponds to the “small field-of-view image” and the “large field-of-view image” in the claim, respectively);
segmenting both the small field-of-view image and the large field-of-view image to respectively obtain at least one first image block corresponding to the small field-of-view image and at least one second image block corresponding to the large field-of-view image, wherein the at least one first image block and the at least one second image block have a mapping relationship (Fig. 4 and pg. [0034] “finds local correspondences between blocks of the frames, evaluate the error between the corresponding blocks”) and wherein segmenting comprises semantic segmentation or segmentation based on color or tone ([0018] According to some aspects of the invention, instead of bringing the entirety of tele frame to the wide frame, the circuits and methods set forth below replace portions of the wide image with image data from the tele image. Further, only the luminance data of the pixel data associated with pixels of the tele image are used to replace luminance data for some pixels of the wide image. For example, wide high frequency luminance content may be replaced with that of tele high frequency luminance content where the replacement will lead to improved image quality. By replacing only the luminance content, it is not necessary to correct for disparity or color changes along the camera transition regions. If an image is captured with an RGB camera, the image can be transferred to format having luminance, such as YUV, where Y represents the luminance portion and UV represents the color portion in the pixels of the image. According to other implementations, color content associated with an image could also be replaced.); and
extracting texture information from the at least one first image block (Fig. 4, extract “Tele high pass signal” which represents texture information), and adding the extracted texture information to a target image block to obtain a target image, wherein the target image block is the second image block having a mapping relationship with the at least one first image block (Fig. 4 and pg. [0032] “the blending decision block 416 are coupled to an Adder 418 to control what portions of the Wide low pass signal, the Wide high pass signal, and the tele high pass signal are used to generate the output signal Output Wide Central Y”. pg. [0034] “FIG. 4 provides a low complexity, image detail transfer for each block of a frame”. Pg. [0036] “also enable modulating a transfer level of pixels or a group of pixels”).
Claims 10-12 have been analyzed and are rejected for the same reasons as outlined above in the rejection of claim 1. Pekkucuksen discloses a device/apparatus (Fig. 1) that is computer-based (processor, storage, etc.) and includes two cameras (Fig. 1, cameras 104 and 105, and pg. [0017] “a wide-angle camera and a tele camera”, claims 11-12). Processor 102 receives images from the two cameras (communication between processor 102 and the cameras defines “a receiving interface”, claim 10).
Response To Arguments
Rejections Under 35 USC §102
The Examiner most respectfully disagrees with Applicants assertion that Pekkucuksen does not disclose how it finds local correspondences between blocks of the frames and, in particular, it does not disclose, teach, or suggest that the correspondences between blocks are found with semantic segmentation or with segmentation based on color or tone.
As can be seen from the paragraph [0018] discussions: [0018] According to some aspects of the invention, instead of bringing the entirety of tele frame to the wide frame, the circuits and methods set forth below replace portions of the wide image with image data from the tele image. Further, only the luminance data of the pixel data associated with pixels of the tele image are used to replace luminance data for some pixels of the wide image. For example, wide high frequency luminance content may be replaced with that of tele high frequency luminance content where the replacement will lead to improved image quality. By replacing only the luminance content, it is not necessary to correct for disparity or color changes along the camera transition regions. If an image is captured with an RGB camera, the image can be transferred to format having luminance, such as YUV, where Y represents the luminance portion and UV represents the color portion in the pixels of the image. According to other implementations, color content associated with an image could also be replaced.
Further, Pekkucuksen teaches, [0019] While specific examples set forth below may relate to a wide/tele dual camera arrangement, it should be understood that the circuits and methods for combining information from different images could relate to any arrangement of camera elements that captures two frames, where portions of the two frames may be combined to create a frame having am improved image.
The correspondences between blocks is discussed in relation to Figure 3, reproduced below, discussion beginning paragraph [0027]. Additionally, Pekkucuksen teaches, [0047] It should be noted that the circuits of FIGS. 4 and 5 could be implemented in hardware or software, or a combination of hardware and software, such as by using the processor of FIG. 1. While the circuit of FIG. 4 has been described as generating an improved image, the circuit of FIG. 4 could also be used to implement image processing techniques. For example, instead of being applied to entire common field of view between tele and wide cameras, detail transfer can be selectively applied to specific regions based on an estimated depth to help create a bokeh effect, which is an image processing technique where the background of the photo is blurred. Finally, Pekkucuksen teaches, [0034] The circuit of FIG. 4 enables a soft decision related to the combination of a tele image with a wide image or a portion of the wide image in the spatial domain. The circuit of FIG. 4 generally extracts low and high frequency signals from both the tele and wide frames, finds local correspondences between blocks of the frames, evaluate the error between the corresponding blocks, and determines whether to replace a portion of the wide image with the entire tele image, partially replace a portion of the wide image with the entire tele image, or keep the wide detail signal, depending on the error between the corresponding blocks. The circuit of FIG. 4 provides a low complexity, image detail transfer for each block of a frame in a single step without any dependency on the rest of the frame. By determining whether to transfer detail for each block rather than transferring the entire tele image into a portion of the wide image, an improved output image can be generated with reduced or no bending of straight lines, reduced or no color transition artifacts, and reduced or no suppression of dynamic range.
Therefore, Pekkucuksen teaches segmentation based on color, tone, and texture classification, the rejection is most respectfully maintained.
Allowable Subject Matter
Claims 2-9 and 15-22 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all limitations of the base claim and any intervening claims.
As allowable subject matter has been indicated, applicant's reply must either comply with all formal requirements or specifically traverse each requirement not complied with. See 37 CFR 1.111(b) and MPEP § 707.07(a).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Kroeger U.S. Patent Application Publication US 20200282929 A1.
[0019] At operation 102, the process 100 can include receiving sensor data of an environment. An example accompanying the operation 102 illustrates a vehicle 104 having a sensor 106, e.g., a camera, disposed on the vehicle 104. In the illustrated example, the vehicle 104 is traversing through the environment generally in a direction indicated by an arrow 108, although in other embodiments the vehicle may be stationary or moving in a different direction. As also illustrated, the sensor 106 captures sensor data such as an image 110. In the illustrated embodiment, the sensor 106 may be a camera or other sensor configured to capture the image 110, e.g., using a wide-angle or fish-eye lens. Accordingly, the image 110 may be a distorted image, e.g., having radial distortion caused by the lens of the camera. The sensor 106 can be other types of cameras, e.g., linear cameras, or the like or the sensor 106 can be a different modality of sensor altogether. By way of non-limiting example, the sensor 106 may be a LiDAR sensor or time-of-flight sensor that generates depth information associated with light returns, or the like. The sensor 106 and the image 110 are for example only, and other or additional sensors, e.g., of the same or different modalities may also or alternatively be provided.
[0033] At operation 210, the process 200 can include undistorting the sensor data using one or more sensor parameters. As shown in the example accompanying the operation 210, the image 208 may be undistorted to generate an undistorted image 212. In the undistorted image 212, features in the image 208 are undistorted into a two-dimensional coordinate system to show a true position of detected points. Thus, for example, in the undistorted image 212, the building 114, vehicle 116 and lamppost 116 more accurately reflect their actual position, shape, and orientation in the world. In various examples, the intrinsics of the sensor 206 may be used to generate the undistorted sensor data. For example, the operation 210 may include undistorting the sensed data, e.g., feature locations, according to a distortion model, e.g., to determine undistorted reference locations. In various examples, the distortion model(s) (e.g., polynomial model, field of view model, etc.) may be determined based on a type of lens of the sensor 206. In the example of the ultra-wide-angle lens of FIG. 2, a field of view distortion model, such as the FOV-M4 distortion model, can be used. Other models may additionally or alternatively be utilized to undistort the sensor data.
[0066] The map(s) 438 can be used by the vehicle 402 to navigate within the environment. For the purpose of this discussion, a map can be any number of data structures modeled in two dimensions, three dimensions, or N-dimensions that are capable of providing information about an environment, such as, but not limited to, topologies (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some instances, a map can include, but is not limited to: texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV/HSL color information), and the like), intensity information (e.g., lidar information, radar information, and the like); spatial information (e.g., image data projected onto a mesh, individual “surfels” (e.g., polygons associated with individual color and/or intensity)), reflectivity information (e.g., specularity information, retroreflectivity information, BRDF information, BSSRDF information, and the like). In one example, a map can include a three-dimensional mesh of the environment. In some instances, the map can be stored in a tiled format, such that individual tiles of the map represent a discrete portion of an environment, and can be loaded into working memory as needed. In some examples, the map(s) 438 can include at least one map (e.g., images and/or a mesh). The vehicle 402 can be controlled based at least in part on the map(s) 438. That is, the map(s) 438 can be used in connection with the localization component 420, the perception component 422, the planning component 424, and/or the calibration component 428 to determine a location of the vehicle 402, identify objects in an environment, and/or generate routes and/or trajectories to navigate within an environment. Moreover, and as described herein, the map(s) 438 can include semantic segmentation information about invariant objects in an environment. By way of non-limiting example, semantic segmentation information contained in maps may be used in addition to, or instead of, semantic segmentation information generated by the semantic segmentation component 430. In some examples, one or more of the map(s) 438 may include an aggregation of semantic segmentation information generated using sensor data generated by the vehicle 402 and/or one or more additional vehicles.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Emily C Terrell whose telephone number is (571)270-3717. The examiner can normally be reached Monday - Thursday 7 a.m.-4 p.m.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666
Prosecution Insights