DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/06/2026 has been entered.
Claim Status
Claims 1-12 are pending for examination in the application filed 01/16/2026. Claims 1 and 10-12 have been amended.
Priority
Acknowledgement is made of Applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent application DE102022206379.1, filed on 06/24/2022.
Response to Arguments and Amendments
Applicant’s arguments with respect to independent claims 1, 11, and 12 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-6, 8, and 10-12 are rejected under 35 U.S.C. 103 as being unpatentable over Kundu (US11546503B1) in view of Aluru (US20200304752A1) and Sanguinetti (US20210144274A1).
Regarding claim 1, Kundu teaches a method for providing image recordings in a vehicle ([col. 2 ln. 8-13] Accordingly, systems and methods are disclosed that can dynamically adjust and vary the image compression techniques (ICT) used to compress images, captured by an array of cameras in a vehicle, on a per-camera basis, to reduce the load on the vehicle's ECU and/or other computational resources while maintaining satisfactory performance), the method comprising the following steps performed in an automated manner ([col. 1 ln. 9-11] The embodiments described herein are generally directed to autonomous vehicles, and, more particularly, to dynamic image compression for multiple cameras of an autonomous vehicle):
acquiring a first image recording of a first image sensor of the vehicle and at least one further image recording of at least one further image sensor of the vehicle, the first and further image recordings imaging different spatial regions of surroundings of the vehicle ([col. 5 ln. 40-52] As illustrated, vehicle 100 may comprise an array of a plurality of cameras 120. In the illustrated embodiment, the array of cameras 120 comprises a forward long camera 120FL with a field of view (FOV) from the front of vehicle 100, a forward wide camera 120FW with a wider, but shorter, field of view from the front of vehicle 100 than forward long camera 120FL, a right camera 120R with a field of view from the right of vehicle 100, a left camera 120L with a field of view from the left of vehicle 100, and a rearview camera 120RV with a field of view from the rear of vehicle 100. Each of cameras 120FL, 120FW, 120R, 120L, and 120RV may be monocular cameras, such that the array of cameras 120 comprises five monocular cameras), whereby at least one item of information of relevance to a function of the vehicle is provided and the image recordings differing from each other with regard to their resolution and/or their image format ([col. 16 ln. 27-37] In addition to an ICT configuration, preprocessing module 112 or the ICT configuration sub-module (or other sub-module) of prescriptive analytics 630 may also determine a field of view to be used for each camera 120, based on one or more features. Each camera 120 may then be configured to capture the field of view that has been determined for that camera 120. A field of view may be increased (e.g., to capture lower resolution images of a larger or wider area), decreased (e.g., to capture higher resolution images of a smaller or narrower area), or changed in shape (e.g., to capture images at a different aspect ratio);
performing an adjustment with regard to their resolution and/or their image format to a preset, the preset being dependent on at least one instance of processing performed on execution of the vehicle function ([col. 18 In. 5-20] For example, a vehicle 100 may be driving straight, when perception module 420 recognizes an object in the images from right camera 120R that resembles a speed-limit sign. Responsively, the ICT configuration may be adjusted to prioritize right camera 120R (e.g., by elevating the image compression technique assigned to right camera 120R to lossless) in order to improve the quality of images being provided to perception module 420 by right camera 120R, to thereby facilitate the detection of the current speed limit in subsequent executions of perception module 420. Once the speed limit has been confidently detected or the speed-limit sign can no longer be confidently detected, the ICT configuration may be adjusted to deprioritize right camera 120R (e.g., return the assigned image compression technique to a default image compression technique). [col. 16 ln. 33-37] A field of view may be increased (e.g., to capture lower resolution images of a larger or wider area), decreased (e.g., to capture higher resolution images of a smaller or narrower area), or changed in shape (e.g., to capture images at a different aspect ratio));
and providing the adjusted image recordings for processing ([col. 20 ln. 11-17] In subprocess 1050, the selected ICT configuration is applied, such that subsequent images, received from the array of cameras 120, are compressed according to the assigned image compression techniques until the next redetermination event. The compressed images are provided to AD functions, such as perception module 420).
Kundu does not teach performing an adjustment of the acquired image recordings with regard to their resolution and/or their image format to a preset, the preset being dependent on at least one instance of processing performed on execution of the vehicle function.
Aluru, in the same field of endeavor of image processing in a vehicle, teaches performing an adjustment of the acquired image recordings with regard to their resolution and/or their image format to a preset ([0037] The infotainment module 140 is operative to receive and process the various images and video streams from the video processing module 130 and for coupling to the display. This processing may include video scaling or conversion from one video resolution to another. In this exemplary embodiment, the infotainment module 140 is operative to upconvert the image and video data into a format compatible with the display 150. For example, the video data from the video processing module 130 may be 1280×720 pixels and the display 150 may be 1920×1080 pixels. The infotainment module 140 is then operative to upconvert the video data from the video processing module 130 to 1920×1080 pixels using various upscaling algorithms), the preset being dependent on at least one instance of processing performed on execution of the vehicle function ([0035] The exemplary display 150 may be located within a vehicle passenger compartment and may be operative to show a first top down image 160 of the vehicle wherein the top down image 160 is stitched together from various images taken in different directions around the vehicle. [0037] the infotainment module 140 is operative to upconvert the image and video data into a format compatible with the display 150).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Kundu with the teachings of Aluru to adjust the resolution of the acquired image recordings because "In order to increase the number of cameras in an automotive camera system without significantly increasing costs or increasing system complexity, lower resolution cameras are often used. These lower resolution cameras often do not match the resolution of in vehicle displays and therefore the video resolution must be upscaled for display in the vehicle. Accordingly, it is desirable to upscale the video in a manner that provides the highest quality video for display to the vehicle occupants" [Aluru 0003].
Kundu does not teach performing preparation of each respective image recording of the first and further acquired image recordings, in which part of the image recording is removed and an artificially generated and reproducible replacement fraction is defined by an item of replacement information as an approximation of the removed part in order to prepare the image recording for lossless data compression; performing the lossless data compression on the respectively prepared image recordings, wherein the replacement information is transmitted instead of the removed part during a respective transmission of the first and further acquired image recordings.
Sanguinetti, in the same field of endeavor of lossless data compression for vehicles, teaches performing preparation of each respective image recording of the first and further acquired image recordings, in which part of the image recording is removed and an artificially generated and reproducible replacement fraction is defined by an item of replacement information as an approximation of the removed part in order to prepare the image recording for lossless data compression; performing the lossless data compression on the respectively prepared image recordings, wherein the replacement information is transmitted instead of the removed part during a respective transmission of the first and further acquired image recordings ([0029] FIG. 1 illustrates the underlying steganographic model: A user 101, arbitrarily referred to as Alice, of the method according to the invention is tasked to acquire images of a given subject 100. She can do so by choosing either image sensor SA 105, hereinafter also designated as source sensor SA, or image sensor SB 115, hereinafter also designated as target sensor SB, producing raw image sets A 120 or B 140, respectively. Another user 102, arbitrarily referred to as Bob, will process the images in various ways. He may, among other things, perform some analysis 145 of the subject, try to determine which sensor acquired the images by studying the noise 150, or apply lossless compression 155. [0057] Therefore, a method according to the present invention allows to retrieve, owing to said preparation step, a noise-reduced version of an image, suitably for efficient lossless compression, from a noisy version of the same image. Preferably, this is achieved by replacing the natural noise, whose origin typically is a combination of the quantum noise of the light captured by the image sensor and the electronic noise of the image sensor itself, with artificial pseudo-noise generated with the help of a pseudo-random number generator. While the natural noise is unpredictable, the pseudo-noise is deterministic and can be identically reproduced if the exact parameters and algorithm for the generation of the noise are known. However, an essential part of the present invention is that the pseudo-noise resembles natural noise of a given desired real or ideal target image sensor as closely as possible by imitating its characteristics through the use of a noise model, as will be described below in more detail. Correspondingly, the present invention further provides methods for lossless compression and decompression of images with pseudo-noise).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Kundu with the teachings of Sanguinetti to prepare the image recordings for lossless data compression by removing part of the image recording and replacing it with an artificially generated part because "it should be impossible to tell that the image contains pseudo-noise rather than natural noise, and the image can be used for the same purposes and applications as the original image data 210 [0059] and “Said desired target noise model may be suitable to imitate the source sensor that acquired the original image data, or it may be adapted to a different real or ideal target sensor” [0044].
Regarding claim 2, Kundu, Aluru, and Sanguinetti teach the method of claim 1. Kundu further teaches wherein the adjustment is performed during data compression and/or data decompression of the image recordings ([col. 18 ln. 5-20] For example, a vehicle 100 may be driving straight, when perception module 420 recognizes an object in the images from right camera 120R that resembles a speed-limit sign. Responsively, the ICT configuration may be adjusted to prioritize right camera 120R (e.g., by elevating the image compression technique assigned to right camera 120R to lossless) in order to improve the quality of images being provided to perception module 420 by right camera 120R, to thereby facilitate the detection of the current speed limit in subsequent executions of perception module 420. Once the speed limit has been confidently detected or the speed-limit sign can no longer be confidently detected, the ICT configuration may be adjusted to deprioritize right camera 120R (e.g., return the assigned image compression technique to a default image compression technique). [col. 16 ln. 33-37] A field of view may be increased (e.g., to capture lower resolution images of a larger or wider area), decreased (e.g., to capture higher resolution images of a smaller or narrower area), or changed in shape (e.g., to capture images at a different aspect ratio)).
Regarding claim 3, Kundu, Aluru, and Sanguinetti teach the method of claim 1. Kundu further teaches wherein the adjustment includes standardization of the resolutions of the first and further image recordings to a target resolution and/or of the image formats of the first and further image recordings to a target image format such that the first and further image recordings are available in the target resolution and/or in the target image format during processing ([col. 18 In. 5-20] For example, a vehicle 100 may be driving straight, when perception module 420 recognizes an object in the images from right camera 120R that resembles a speed-limit sign. Responsively, the ICT configuration may be adjusted to prioritize right camera 120R (e.g., by elevating the image compression technique assigned to right camera 120R to lossless) in order to improve the quality of images being provided to perception module 420 by right camera 120R, to thereby facilitate the detection of the current speed limit in subsequent executions of perception module 420. Once the speed limit has been confidently detected or the speed-limit sign can no longer be confidently detected, the ICT configuration may be adjusted to deprioritize right camera 120R (e.g., return the assigned image compression technique to a default image compression technique). [col. 16 In. 33-37] A field of view may be increased (e.g., to capture lower resolution images of a larger or wider area), decreased (e.g., to capture higher resolution images of a smaller or narrower area), or changed in shape (e.g., to capture images at a different aspect ratio. [col. 23 ln. 53-60] FIG. 14A illustrates centralized object recognition, according to an example implementation. In centralized object recognition, images from each camera 120 are compressed according to the image compression technique assigned to that camera 120. The compressed images are stitched together, and the stitched image is then input into a single object-recognition (OR) model (e.g., a deep neural network or rules-based algorithm).
Regarding claim 4, Kundu, Aluru, and Sanguinetti teach the method of claim 1. Kundu further teaches wherein the processing is configured to receive the first and further image recordings ([col. 20 ln. 11-17] In subprocess 1050, the selected ICT configuration is applied, such that subsequent images, received from the array of cameras 120, are compressed according to the assigned image compression techniques until the next redetermination event. The compressed images are provided to AD functions, such as perception module 420).
Kundu does not teach receive the first and further image recordings solely in a uniform resolution calculated by the adjustment and/or a uniform image format calculated by the adjustment including a target resolution and/or a target image format.
Aluru teaches receive the first and further image recordings solely in a uniform resolution calculated by the adjustment and/or a uniform image format calculated by the adjustment including a target resolution and/or a target image format ([0037] The infotainment module 140 is operative to receive and process the various images and video streams from the video processing module 130 and for coupling to the display. This processing may include video scaling or conversion from one video resolution to another. In this exemplary embodiment, the infotainment module 140 is operative to upconvert the image and video data into a format compatible with the display 150. For example, the video data from the video processing module 130 may be 1280×720 pixels and the display 150 may be 1920×1080 pixels. The infotainment module 140 is then operative to upconvert the video data from the video processing module 130 to 1920×1080 pixels using various upscaling algorithms).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Kundu with the teachings of Aluru for the image recordings to be solely in a uniform resolution because "In order to increase the number of cameras in an automotive camera system without significantly increasing costs or increasing system complexity, lower resolution cameras are often used. These lower resolution cameras often do not match the resolution of in vehicle displays and therefore the video resolution must be upscaled for display in the vehicle. Accordingly, it is desirable to upscale the video in a manner that provides the highest quality video for display to the vehicle occupants" [Aluru 0003].
Regarding claim 5, Kundu, Aluru, and Sanguinetti teach the method of claim 1. Kundu further teaches wherein at least one additional item of information is integrated into the first and further image recordings, the additional item of information being specific to the resolution and/or the image format and the adjustment being performed based on the additional item of information ([col. 14 ln. 25-28] Image compression techniques may be assigned to cameras 120 based on various features, such as the position of vehicle 100, the speed of vehicle 100, camera visibility, prior object detection results, current load on ECU 110, and/or the like).
Regarding claim 6, Kundu, Aluru, and Sanguinetti teach the method of claim 1. Kundu further teaches wherein the acquisition of the first and further image recordings is performed decentrally in the vehicle at the first and further image sensors (Fig. 12F), respectively, the first and further image recordings then being transmitted to a processing unit, and the adjustment then being performed by the processing unit, after the transmission ([col. 2 In. 26-39] In embodiments, a system for dynamically assigning image compression techniques to a plurality of cameras in a vehicle is disclosed, wherein the system comprises at least one hardware processor that: receives a plurality of features, wherein each of the plurality of features is a feature of the vehicle or a feature of an environment of the vehicle; prioritizes each of the cameras within the plurality of cameras based on the plurality of features; and assigns one of a plurality of available image compression techniques to each of the plurality of cameras based on the prioritizations, such that an image compression technique assigned to a camera with higher priority has less information loss than an image compression technique that is assigned to a camera with lower priority).
Regarding claim 8, Kundu, Aluru, and Sanguinetti teach the method of claim 1. Kundu further teaches wherein the first and further image recordings are stitched together as a function of imaged surrounding regions, whereby a stitched-together image recording is obtained, the stitched-together image recording being adjusted and/or provided for the processing ([col. 23 ln. 53-60] FIG. 14A illustrates centralized object recognition, according to an example implementation. In centralized object recognition, images from each camera 120 are compressed according to the image compression technique assigned to that camera 120. The compressed images are stitched together, and the stitched image is then input into a single object-recognition (OR) model (e.g., a deep neural network or rules-based algorithm)).
Regarding claim 10, Kundu, Aluru, and Sanguinetti teach the method of claim 1. Sanguinetti teaches wherein performance of the preparation includes the following step: introducing at least one item of additional information into the respective image recording, which information includes an item of metadata information and/or a reference marker, based on which the adjustment is performed ([0072] Finally, the pseudo-noise parameters, metadata 225 and other data 230 are optionally encrypted and packaged 425 with the compressed image data into a compressed file 280 or equivalent data structure. Depending on the application, this concatenation can be performed by simply appending binary data to the same file, by using several files in an archive-like container, or by using the facilities of a special file format such as TIFF. [0017] The pseudo-noisy image data comprising the parameters of said noise model as well as optionally said other data may be stored and/or transmitted as-is or embedded in a container file of any known file format, for example a container file of format tiff, jp2, dng or png).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Kundu with the teachings of Sanguinetti to introduce an additional item of information based on which the adjustment is performed because "illustrated in FIG. 2a, metadata 225 and other relevant data 230 optionally can also be embedded into the pseudo-noisy image, resulting in praw data 260 that is self-contained and can represent additional value compared to the original image data. For example, it is possible to embed other parameters important for optimal processing of the image data, or to embed data that proves the authenticity of the image" [0059].
Regarding claim 11, Kundu teaches a non-transitory computer-readable medium on which is stored a computer program including commands for providing image recordings in a vehicle, the commands, when executed by a computer, causing the computer to perform the following steps in automated manner ([col. 9 ln. 4-6] Internal storage 330 may comprise a non-transitory computer-readable medium that provides storage of instructions and data for software. [col. 1 ln. 9-11] The embodiments described herein are generally directed to autonomous vehicles, and, more particularly, to dynamic image compression for multiple cameras of an autonomous vehicle):
acquiring a first image recording of a first image sensor of the vehicle and at least one further image recording of at least one further image sensor of the vehicle, the first and further image recordings imaging different spatial regions of surroundings of the vehicle ([col. 5 ln. 40-52] As illustrated, vehicle 100 may comprise an array of a plurality of cameras 120. In the illustrated embodiment, the array of cameras 120 comprises a forward long camera 120FL with a field of view (FOV) from the front of vehicle 100, a forward wide camera 120FW with a wider, but shorter, field of view from the front of vehicle 100 than forward long camera 120FL, a right camera 120R with a field of view from the right of vehicle 100, a left camera 120L with a field of view from the left of vehicle 100, and a rearview camera 120RV with a field of view from the rear of vehicle 100. Each of cameras 120FL, 120FW, 120R, 120L, and 120RV may be monocular cameras, such that the array of cameras 120 comprises five monocular cameras), whereby at least one item of information of relevance to a function of the vehicle is provided and the image recordings differing from each other with regard to their resolution and/or their image format ([col. 16 ln. 27-37] In addition to an ICT configuration, preprocessing module 112 or the ICT configuration sub-module (or other sub-module) of prescriptive analytics 630 may also determine a field of view to be used for each camera 120, based on one or more features. Each camera 120 may then be configured to capture the field of view that has been determined for that camera 120. A field of view may be increased (e.g., to capture lower resolution images of a larger or wider area), decreased (e.g., to capture higher resolution images of a smaller or narrower area), or changed in shape (e.g., to capture images at a different aspect ratio);
performing an adjustment of the image recordings with regard to their resolution and/or their image format to a preset, the preset being dependent on at least one instance of processing performed on execution of the vehicle function ([col. 18 ln. 5-20] For example, a vehicle 100 may be driving straight, when perception module 420 recognizes an object in the images from right camera 120R that resembles a speed-limit sign. Responsively, the ICT configuration may be adjusted to prioritize right camera 120R (e.g., by elevating the image compression technique assigned to right camera 120R to lossless) in order to improve the quality of images being provided to perception module 420 by right camera 120R, to thereby facilitate the detection of the current speed limit in subsequent executions of perception module 420. Once the speed limit has been confidently detected or the speed-limit sign can no longer be confidently detected, the ICT configuration may be adjusted to deprioritize right camera 120R (e.g., return the assigned image compression technique to a default image compression technique));
and providing the adjusted image recordings for processing ([col. 20 ln. 11-17] In subprocess 1050, the selected ICT configuration is applied, such that subsequent images, received from the array of cameras 120, are compressed according to the assigned image compression techniques until the next redetermination event. The compressed images are provided to AD functions, such as perception module 420).
Kundu does not teach performing an adjustment of the acquired image recordings with regard to their resolution and/or their image format to a preset, the preset being dependent on at least one instance of processing performed on execution of the vehicle function.
Aluru, in the same field of endeavor of image processing in a vehicle, teaches performing an adjustment of the acquired image recordings with regard to their resolution and/or their image format to a preset ([0037] The infotainment module 140 is operative to receive and process the various images and video streams from the video processing module 130 and for coupling to the display. This processing may include video scaling or conversion from one video resolution to another. In this exemplary embodiment, the infotainment module 140 is operative to upconvert the image and video data into a format compatible with the display 150. For example, the video data from the video processing module 130 may be 1280×720 pixels and the display 150 may be 1920×1080 pixels. The infotainment module 140 is then operative to upconvert the video data from the video processing module 130 to 1920×1080 pixels using various upscaling algorithms), the preset being dependent on at least one instance of processing performed on execution of the vehicle function ([0035] The exemplary display 150 may be located within a vehicle passenger compartment and may be operative to show a first top down image 160 of the vehicle wherein the top down image 160 is stitched together from various images taken in different directions around the vehicle. [0037] the infotainment module 140 is operative to upconvert the image and video data into a format compatible with the display 150).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Kundu with the teachings of Aluru to adjust the resolution of the acquired image recordings because "In order to increase the number of cameras in an automotive camera system without significantly increasing costs or increasing system complexity, lower resolution cameras are often used. These lower resolution cameras often do not match the resolution of in vehicle displays and therefore the video resolution must be upscaled for display in the vehicle. Accordingly, it is desirable to upscale the video in a manner that provides the highest quality video for display to the vehicle occupants" [Aluru 0003].
Kundu does not teach performing preparation of each respective image recording of the first and further acquired image recordings, in which part of the image recording is removed and an artificially generated and reproducible replacement fraction is defined by an item of replacement information as an approximation of the removed part in order to prepare the image recording for lossless data compression; performing the lossless data compression on the respectively prepared image recordings, wherein the replacement information is transmitted instead of the removed part during a respective transmission of the first and further acquired image recordings.
Sanguinetti, in the same field of endeavor of lossless data compression for vehicles, teaches performing preparation of each respective image recording of the first and further acquired image recordings, in which part of the image recording is removed and an artificially generated and reproducible replacement fraction is defined by an item of replacement information as an approximation of the removed part in order to prepare the image recording for lossless data compression; performing the lossless data compression on the respectively prepared image recordings, wherein the replacement information is transmitted instead of the removed part during a respective transmission of the first and further acquired image recordings ([0029] FIG. 1 illustrates the underlying steganographic model: A user 101, arbitrarily referred to as Alice, of the method according to the invention is tasked to acquire images of a given subject 100. She can do so by choosing either image sensor SA 105, hereinafter also designated as source sensor SA, or image sensor SB 115, hereinafter also designated as target sensor SB, producing raw image sets A 120 or B 140, respectively. Another user 102, arbitrarily referred to as Bob, will process the images in various ways. He may, among other things, perform some analysis 145 of the subject, try to determine which sensor acquired the images by studying the noise 150, or apply lossless compression 155. [0057] Therefore, a method according to the present invention allows to retrieve, owing to said preparation step, a noise-reduced version of an image, suitably for efficient lossless compression, from a noisy version of the same image. Preferably, this is achieved by replacing the natural noise, whose origin typically is a combination of the quantum noise of the light captured by the image sensor and the electronic noise of the image sensor itself, with artificial pseudo-noise generated with the help of a pseudo-random number generator. While the natural noise is unpredictable, the pseudo-noise is deterministic and can be identically reproduced if the exact parameters and algorithm for the generation of the noise are known. However, an essential part of the present invention is that the pseudo-noise resembles natural noise of a given desired real or ideal target image sensor as closely as possible by imitating its characteristics through the use of a noise model, as will be described below in more detail. Correspondingly, the present invention further provides methods for lossless compression and decompression of images with pseudo-noise).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Kundu with the teachings of Sanguinetti to prepare the image recordings for lossless data compression by removing part of the image recording and replacing it with an artificially generated part because "it should be impossible to tell that the image contains pseudo-noise rather than natural noise, and the image can be used for the same purposes and applications as the original image data 210 [0059] and “Said desired target noise model may be suitable to imitate the source sensor that acquired the original image data, or it may be adapted to a different real or ideal target sensor” [0044].
Regarding claim 12, Kundu teaches a device configured to provide image recordings in a vehicle, the device configured to ([col. 2 ln. 8-13] Accordingly, systems and methods are disclosed that can dynamically adjust and vary the image compression techniques (ICT) used to compress images, captured by an array of cameras in a vehicle, on a per-camera basis, to reduce the load on the vehicle's ECU and/or other computational resources while maintaining satisfactory performance. [col. 1 ln. 9-11] The embodiments described herein are generally directed to autonomous vehicles, and, more particularly, to dynamic image compression for multiple cameras of an autonomous vehicle):
acquire a first image recording of a first image sensor of the vehicle and at least one further image recording of at least one further image sensor of the vehicle, the first and further image recordings imaging different spatial regions of surroundings of the vehicle ([col. 5 ln. 40-52] As illustrated, vehicle 100 may comprise an array of a plurality of cameras 120. In the illustrated embodiment, the array of cameras 120 comprises a forward long camera 120FL with a field of view (FOV) from the front of vehicle 100, a forward wide camera 120FW with a wider, but shorter, field of view from the front of vehicle 100 than forward long camera 120FL, a right camera 120R with a field of view from the right of vehicle 100, a left camera 120L with a field of view from the left of vehicle 100, and a rearview camera 120RV with a field of view from the rear of vehicle 100. Each of cameras 120FL, 120FW, 120R, 120L, and 120RV may be monocular cameras, such that the array of cameras 120 comprises five monocular cameras), whereby at least one item of information of relevance to a function of the vehicle is provided and the image recordings differing from each other with regard to their resolution and/or their image format ([col. 16 ln. 27-37] In addition to an ICT configuration, preprocessing module 112 or the ICT configuration sub-module (or other sub-module) of prescriptive analytics 630 may also determine a field of view to be used for each camera 120, based on one or more features. Each camera 120 may then be configured to capture the field of view that has been determined for that camera 120. A field of view may be increased (e.g., to capture lower resolution images of a larger or wider area), decreased (e.g., to capture higher resolution images of a smaller or narrower area), or changed in shape (e.g., to capture images at a different aspect ratio);
perform an adjustment of the image recordings with regard to their resolution and/or their image format to a preset, the preset being dependent on at least one instance of processing performed on execution of the vehicle function ([col. 18 ln. 5-20] For example, a vehicle 100 may be driving straight, when perception module 420 recognizes an object in the images from right camera 120R that resembles a speed-limit sign. Responsively, the ICT configuration may be adjusted to prioritize right camera 120R (e.g., by elevating the image compression technique assigned to right camera 120R to lossless) in order to improve the quality of images being provided to perception module 420 by right camera 120R, to thereby facilitate the detection of the current speed limit in subsequent executions of perception module 420. Once the speed limit has been confidently detected or the speed-limit sign can no longer be confidently detected, the ICT configuration may be adjusted to deprioritize right camera 120R (e.g., return the assigned image compression technique to a default image compression technique));
and provide the adjusted image recordings for processing ([col. 20 ln. 11-17] In subprocess 1050, the selected ICT configuration is applied, such that subsequent images, received from the array of cameras 120, are compressed according to the assigned image compression techniques until the next redetermination event. The compressed images are provided to AD functions, such as perception module 420).
Kundu does not teach perform an adjustment of the acquired image recordings with regard to their resolution and/or their image format to a preset, the preset being dependent on at least one instance of processing performed on execution of the vehicle function.
Aluru, in the same field of endeavor of image processing in a vehicle, teaches perform an adjustment of the acquired image recordings with regard to their resolution and/or their image format to a preset ([0037] The infotainment module 140 is operative to receive and process the various images and video streams from the video processing module 130 and for coupling to the display. This processing may include video scaling or conversion from one video resolution to another. In this exemplary embodiment, the infotainment module 140 is operative to upconvert the image and video data into a format compatible with the display 150. For example, the video data from the video processing module 130 may be 1280×720 pixels and the display 150 may be 1920×1080 pixels. The infotainment module 140 is then operative to upconvert the video data from the video processing module 130 to 1920×1080 pixels using various upscaling algorithms), the preset being dependent on at least one instance of processing performed on execution of the vehicle function ([0035] The exemplary display 150 may be located within a vehicle passenger compartment and may be operative to show a first top down image 160 of the vehicle wherein the top down image 160 is stitched together from various images taken in different directions around the vehicle. [0037] the infotainment module 140 is operative to upconvert the image and video data into a format compatible with the display 150).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the device of Kundu with the teachings of Aluru to adjust the resolution of the acquired image recordings because "In order to increase the number of cameras in an automotive camera system without significantly increasing costs or increasing system complexity, lower resolution cameras are often used. These lower resolution cameras often do not match the resolution of in vehicle displays and therefore the video resolution must be upscaled for display in the vehicle. Accordingly, it is desirable to upscale the video in a manner that provides the highest quality video for display to the vehicle occupants" [Aluru 0003].
Kundu does not teach perform preparation of each respective image recording of the first and further acquired image recordings, in which part of the image recording is removed and an artificially generated and reproducible replacement fraction is defined by an item of replacement information as an approximation of the removed part in order to prepare the image recording for lossless data compression; perform the lossless data compression on the respectively prepared image recordings, wherein the replacement information is transmitted instead of the removed part during a respective transmission of the first and further acquired image recordings.
Sanguinetti, in the same field of endeavor of lossless data compression for vehicles, teaches perform preparation of each respective image recording of the first and further acquired image recordings, in which part of the image recording is removed and an artificially generated and reproducible replacement fraction is defined by an item of replacement information as an approximation of the removed part in order to prepare the image recording for lossless data compression; perform the lossless data compression on the respectively prepared image recordings, wherein the replacement information is transmitted instead of the removed part during a respective transmission of the first and further acquired image recordings ([0029] FIG. 1 illustrates the underlying steganographic model: A user 101, arbitrarily referred to as Alice, of the method according to the invention is tasked to acquire images of a given subject 100. She can do so by choosing either image sensor SA 105, hereinafter also designated as source sensor SA, or image sensor SB 115, hereinafter also designated as target sensor SB, producing raw image sets A 120 or B 140, respectively. Another user 102, arbitrarily referred to as Bob, will process the images in various ways. He may, among other things, perform some analysis 145 of the subject, try to determine which sensor acquired the images by studying the noise 150, or apply lossless compression 155. [0057] Therefore, a method according to the present invention allows to retrieve, owing to said preparation step, a noise-reduced version of an image, suitably for efficient lossless compression, from a noisy version of the same image. Preferably, this is achieved by replacing the natural noise, whose origin typically is a combination of the quantum noise of the light captured by the image sensor and the electronic noise of the image sensor itself, with artificial pseudo-noise generated with the help of a pseudo-random number generator. While the natural noise is unpredictable, the pseudo-noise is deterministic and can be identically reproduced if the exact parameters and algorithm for the generation of the noise are known. However, an essential part of the present invention is that the pseudo-noise resembles natural noise of a given desired real or ideal target image sensor as closely as possible by imitating its characteristics through the use of a noise model, as will be described below in more detail. Correspondingly, the present invention further provides methods for lossless compression and decompression of images with pseudo-noise).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the device of Kundu with the teachings of Sanguinetti to prepare the image recordings for lossless data compression by removing part of the image recording and replacing it with an artificially generated part because "it should be impossible to tell that the image contains pseudo-noise rather than natural noise, and the image can be used for the same purposes and applications as the original image data 210 [0059] and “Said desired target noise model may be suitable to imitate the source sensor that acquired the original image data, or it may be adapted to a different real or ideal target sensor” [0044].
Claims 7 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Kundu in view of Aluru, Sanguinetti, and Xiang (US20230377095A1).
Regarding claim 7, Kundu, Aluru, and Sanguinetti teach the method of claim 1. Kundu further teaches transmission from the first and further image sensors being performed from different positions on the vehicle and with different transmission paths (Fig. 12F and Fig. 14A).
Kundu does not teach wherein, after the acquisition and before the adjustment, data compression and transmission of the image recordings are performed.
Xiang, in the same field of endeavor of image compression, teaches wherein, after the acquisition and before the adjustment, data compression and transmission of the image recordings are performed ([0016] The apparatus may segment 102 an image into an object region and a background region, where the image has a first resolution. An image is data that indicates optical information. For instance, an image may be a set of pixel values, an image file, etc. In some examples, an image may have been down-sampled, compressed, and/or may contain artifacts. [0019] The apparatus may generate 104, using a first machine learning model, an enhanced object region with a second resolution that is greater than the first resolution. For example, the object region(s) may be provided to the first machine learning model, which may produce a corresponding enhanced object region or regions).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Kundu with the teachings of Xiang to perform data compression before the adjustment because "a training image (e.g., down-sampled, compressed, etc., image) may be provided to the first machine learning model, which may produce an enhanced training image" [Xiang 0024].
Regarding claim 9, Kundu, Aluru, and Sanguinetti teach the method of claim 1. Xiang teaches wherein the adjustment is performed by a trained algorithm which was obtained by training with image recordings of a different resolution and/or a different data format ([0013] Some examples of the techniques described herein may utilize a machine learning model or models (e.g., deep learning) to increase image resolution and/or quality. For instance, some techniques may be utilized to generate super-resolution images with increased object (e.g., face) rendering quality. [0062] Training data 344 is data to train the machine learning model(s). Examples of training data 344 may include ground truth images and degraded images (e.g., down-sampled and/or compressed images). For example, the training data 344 may include low-resolution degraded images (with a resolution of 500×750 pixels or other resolution, for instance). In some examples, the training data 344 may include high-resolution ground truth data (with a resolution of 1000×1500, 1500×2250, 2000×3000, or other resolution, for instance)).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Kundu with the teachings of Xiang to perform the adjustment by a trained algorithm because "A machine learning model or models (e.g., deep neural networks) may be utilized to enhance (e.g., reconstruct) the region(s) and/or background(s)" [Xiang 0010].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571)272-4077. The examiner can normally be reached M-F 9-5. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666
Prosecution Insights