DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDSs) submitted on December 3, 2024, April 3, 2025 and April 25, 2025 are in compliance with 37 CFR 1.97 and 1.98 and therefore have been considered by the examiner and placed in the file.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation (BRI) using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The BRIs are used for purposes of searching for prior art, but cannot be incorporated into the claims. Claim limitations must be given their plain meaning unless such meaning is inconsistent with the specification. MPEP 2111.01. BRIs for some of the claim limitations are provided below. Should Applicant believe that other interpretations are warranted, Applicant should point to the portions of the present disclosure that clearly show that a different interpretation is appropriate.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publ. Appl. No. 2018/0189574 A1 to Brueckner et al. (hereinafter referred to as “Brueckner”) in view of U.S. Publ. Appl. No. 2020/0043137 A1 to He et al. (hereinafter referred to as “He”).
Regarding claim 1, Brueckner discloses a method comprising:
accessing, by a vehicle device positioned at a vehicle, input image data representing a perspective from the vehicle, the input image data including a first region and a second region, the first and the second region each having an input pixel density (Paras. [0029], [0034], Fig. 1, the image processor 110 of the vehicle control system accesses digital images 114 captured by an image capture device 100 integrated with a vehicle: “[i]n some examples, a vehicle control system configured to analyze digital image outputs from a disclosed image capture device can be provided as an integrated component in an autonomous vehicle.” Para. [0037]: “[r]eferring still to FIG. 1, image capture device 100 also can include one or more image processing devices (e.g., image processors) 110 coupled to image sensor 108. In some examples, image processor 110 can be a field-programmable gate array (FPGA) provided within the image capture device 100. One or more image data links 116 can be provided to couple the one or more image processors 110 to image sensor 108.” The BRI for the term “pixel density”, based on para. [0090] of the present specification, is that it means image resolution. In Brueckner, the input image data 114 captured by the image capture device 100 has the resolution of the raw image capture data and includes a first region 118 corresponding to one or more regions of interest (ROIs) and a second region 119 corresponding to regions of the image outside of the ROIs, paras. [0036]-[0039]);
generating first image data, the first image data at least partially representing the first region and having a first pixel density (The BRI for the term “first pixel density”, based on para. [0091] and Fig. 4 step 410 of the present disclosure, is that it means first resolution and that the first resolution can be the same as the resolution of the captured input image or it can be a reduced resolution obtained by, for example, downsampling the input image. Para. [0039] of Brueckner discloses that first image data corresponding to the region of interest (ROI) 118 in Fig. 1 has a first resolution, referred to in Brueckner as the “second resolution”, is higher than the resolution of the image regions 119 outside of the ROI 118);
generating second image data, the second image data at least partially representing the second region and having a second pixel density less than the first pixel density (Para. [0039], the second image data corresponds to the image data of the regions 119 outside of the ROI 118. The resolution of the second image data in regions 119 outside of the ROI 118 is less than the first resolution of the first image data of the ROI 118. See also Para. [0080], Fig. 7, step 706, discussing that the image data in the ROI can have the same resolution as the raw input image data captured by the image capture device 100 whereas the image data in outside of the ROI can have a sampled down resolution of the raw input image data captured by the image capture device 100 );
generating analysis data by executing at least one image analysis model on the first image data and the second image data (Paras. [0081]-[0082], Fig. 7, step 710, the first image data and the second image data that are output from step 706 as output data are processed by an image analysis model that is executed in step 710 to perform object detection. Para. [0076] discloses that various models are used to perform the analyses);
generating output image data, the output image data representing the first region and the second region and having the second pixel density (The BRI for this limitation, based on para. [0099] and step 410 of Fig. 4 of the present disclosure, is that the output image data made up of the image data representing the first and second regions has a unform resolution equal to the second resolution. Para. [0081], Fig. 7, step 706 of Brueckner are directed to generating output image data representing the first and second regions. However, Brueckner does not explicitly disclose that the output image data has a uniform resolution equal to the second resolution);
outputting the analysis data (Fig. 7, step 712 and para. [0082] disclose outputting the analysis data containing the results of the analysis to the vehicle control system discussed in, for example, para. [0029]: “[i]n response to detection at (710) of at least one object in the digital image or conversely, a detected absence of an object), one or more operational parameters of a vehicle can be controlled at (712). Operational parameters can include vehicle speed, direction, acceleration, deceleration, steering and/or operation of vehicle components to follow a planned motion and/or navigational course. In some examples, operational parameters can be controlled at (712) to navigate a vehicle in an autonomous or semi-autonomous operational mode”); and
outputting the output image data (Para. [0081], Fig. 7, step 708 discloses outputting the output image data to “other computing devices, processors or control devices”).
As indicated above, Brueckner does not explicitly disclose that the generated output image data representing the first region and the second region has a uniform resolution that is equal to the second resolution. Para. [0099] of the present specification discloses that the second resolution is lower than the first resolution, which can be the same resolution as the captured image or a resolution lower than that of the captured image, but does not disclose any other limits on what the second resolution is. Para. [0136] of the present specification states that the purpose of using the second resolution for the output image data is to reduce bandwidth and storage usage by making the output image data smaller in size than the input image data.
Brueckner discloses the importance of reducing the output image file size by using an increased resolution for the ROIs while using a reduced the resolution for the other regions in order to reduce the amount of data that needs to be stored in memory (Para. [0024]: “[f]or some image capture devices having improved resolutions, providing targeted solutions for limiting the high-resolution portions of an image (e.g., image portions within one or more regions of interest) can provide a solution for enhancing the resolution of important portions of an image while maintaining a manageable image file size for data processing and storage.”). Brueckner further discloses that the reduction in the resolution of the non-ROI regions can result in the output image data, referred to in Brueckner as the “target image file”, being a fraction of the file size of the input image generated by a high-resolution image capture device of the vehicle, which significantly reduces data storage usage (Para. [0024]: “[i]n some examples, a target image file size can be less than 50% of a full image frame of a high performance vehicle camera. In some examples, a target image file size can be between about 15-50% of a full image frame of a high performance vehicle camera. In some examples, a target image file size can be between about 15-25% of a full image frame of a high performance vehicle camera.”). However, Brueckner does not explicitly disclose that all of the output image data has the lower, second resolution.
He, in the same field of endeavor, discloses increasing the resolution of an input image to a first resolution in a way that prevents or limits the introduction of artifacts and then decreasing the resolution of the entire image to a second resolution before storing it in memory (Para. [0037]: “[i]n one embodiment, the ability to increase the resolution of an image without introducing noticeable artifacts may allow for images to be stored at a lower resolution, thus saving memory space and/or bandwidth, and restore the image to a higher resolution before displaying the image.”).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the method of Brueckner such that after image analysis is performed in step 710 of Fig. 7 the entire output image data file is reduced from a higher, first resolution to a lower, second resolution before storing the output image data in memory. One of ordinary skill in the art would have been motivated to make the modification to further reduce data storage usage since both Brueckner and He teach the desirability of reducing the resolution of image data captured by a vehicle camera before storing it in memory to reduce data storage usage. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software and/or hardware of Brueckner to downsample or otherwise reduce the resolution of the output image data generated at step 706 before storing it in memory).
Regarding claim 2, Brueckner discloses that outputting the output image data can comprise outputting the output image data to at least one non-transitory processor-readable storage medium at the vehicle device (Para. [0081], Fig. 7, step 708 discloses outputting the output image data to “other computing devices, processors or control devices”. Para. [0024] discloses that the output image data is stored in memory, which is a non-transitory processor-readable storage medium. Fig. 6 shows memory device 626 of the vehicle control system 602, which is “at the vehicle device”, and para. [0070] discusses storing the image data in the memory device 626 and processing of it by the processors 614 and 624 shown in Fig. 6).
Regarding claim 3, Brueckner discloses that outputting the output image data comprises transmitting, by at least one communication interface of the vehicle device, the output image data to a device remote from the vehicle (Fig. 6, para. [0071], discloses any of the data generated by the image capture device 100 or by the vehicle control system 602, including the output image data, can be transmitted by the communication interface 612 of the vehicle device 602 to a device remote from the vehicle 502, such as to “server-based processing or control systems located remotely from a vehicle 502”).
Regarding claim 4, Brueckner discloses that outputting the analysis data comprises transmitting, by at least one communication interface of the vehicle device, the analysis data to a device remote from the vehicle (Fig. 6, para. [0071], discloses any of the data generated by the image capture device 100 or by the vehicle control system 602, including the analysis data, can be transmitted by the communication interface 612 of the vehicle device 602 to a device remote from the vehicle 502, such as to “server-based processing or control systems located remotely from a vehicle 502”).
Regarding claim 5, the BRI for this claim is that it is further limited to: the input image data further including a third region; a further generating step that generates third image data representing the third region and having a third resolution that is less than the first resolution and greater than the second resolution; a further generating step that generates analysis data by executing the at least one image analysis model on the first image data, the second image data, and the third image data; and a further generating step that generates the output image data representing the first, second and third regions at the second resolution.
Therefore, this claim recites the same steps that are recited in claim 1 with reference to the first and second regions, the first and second image data and the first and second resolutions, but with reference the third region, the third image data and the third resolution, with the third resolution being less than the first resolution and greater than the second resolution.
Although Brueckner only explicitly discusses the first regions corresponding to the ROIs 118 (Fig. 1) and having a first resolution and the second regions 119 corresponding to the regions outside of the ROIs and having the second resolution that is less than the first resolution, Brueckner refers to the possibility that there can be multiple different ROIs of different types or importance/relevance levels in different regions of the input image (e.g., “vehicles, pedestrians, roads, buildings, signage, terrain, etc.”) as well as the possibility that there can be different non-ROI regions of different types or importance/relevance levels in different regions of the input image (e.g., “ground surfaces”, “the sky”). Brueckner also discusses the resolution of the images in these different areas being different with higher resolution being used for regions of more importance/relevance and lower resolution being used for regions of less importance/relevance. See, for example, paras. [0024], [0030], [0032], [0057] and [0058]. Para. [0032] of Brueckner also discusses actively shifting ROIs within image frames to “maximize available pixel data for analysis while minimizing overall file size”. Para. [0046] discusses a variety of resolutions that can be used for the different regions of the input image, with resolutions expressed in relative terms as percentages of the resolutions used for other regions.
The combination of all of these teachings of Brueckner teaches that different resolutions can be used for different regions of the input image based on their different levels of importance or relevance. In other words, the teachings of Brueckner are not limited to first and second regions, first and second image data and first and second resolutions, but extend to additional regions, additional image data and additional resolutions.
Consequently, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the present application, to generate third image data representing a third region of the input image data and having a third resolution that is less than the first resolution and greater than the second resolution, to generate the analysis data by performing image analysis on the first, second and third image data, and to generate the output image data representing all three regions. For the same reasons discussed above in the rejection of claim 1 with reference to He, it would have also been obvious to reduce the output image data for all three regions to the lower, second resolution.
A person of skill in the art would have been motivated to make the modifications to allow additional regions in the input image to be analyzed at other resolutions based on the respective levels of importance of the regions, thereby reducing the overall output image data file size for storage while also improving the versatility of the system of Brueckner at handling more regions of other respective importance levels. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software and/or hardware of Brueckner to downsample or otherwise reduce the resolution of image data associated with the third region to a resolution that is in between the first and second resolutions based on the third region having an importance that is less than that of the first region, but greater than that of the second region).
Regarding claim 6, the BRI for this limitation, based on para. [0117] of the present specification, is that the first region represents content in the input image that is at a position farther away from the vehicle than the content represented in the second region. Brueckner discloses that real-world content of the ROIs represented in the first region can be positioned farther away than the real-world content represented in the second region (Fig. 1, duplicated below for convenience, shows the portions of the road and buildings of the ROI represented in the first region 118 being farther from the vehicle than the ground surface nearest the vehicle in the second region 119 near the bottom of the image, paras. [0025], [0027] and [0039]).
PNG
media_image1.png
200
400
media_image1.png
Greyscale
Regarding claim 7, Brueckner discloses that the first image data represents an entirety of the first region and the second image data represents an entirety of the second region (The BRI for this claim, based on para. [0097] and Fig. 5B of the present disclosure, is that the first and second regions have the full width of the captured input image in the horizontal direction, as opposed to being clipped in the horizontal direction. Paras. [0048]-[0054] and Fig. 4 of Brueckner, duplicated below for convenience, disclose setting parameters for the widths of the first and second regions and that the left spacing and right spacing width parameters can be set to zero such that the first and second regions correspond to the entirety of the full image frame area 402 in the horizontal, width-wise direction, as depicted in Fig. 4).
PNG
media_image2.png
200
400
media_image2.png
Greyscale
Regarding claim 8, Brueckner discloses that the first image data represents a first cropped portion of the first region and the second image data represents a second cropped portion of the second region (The BRI for this claim, based on paras. [0119]-[0122] and Figs. 8A and 8B of the present disclosure, is that the first and second regions can be cropped in the horizontal direction such that the first and second regions have less than the full width of the captured input image in the horizontal direction. As indicated above, Brueckner discloses that the left and right spacing width parameters can be set to values that control the widths of the first and second regions relative to the full image frame width dimension 406 of the captured image frame 402, Fig. 4 and paras. [0048]-[0049]. Setting the left and right spacing values to a value greater than zero crops the first and second regions in the horizontal, width-wise direction such that a first cropped portion of the first region and the second image data represents a second cropped portion of the second region).
Regarding claim 9, Brueckner discloses that generating the analysis data by executing at least one image analysis model on the first image data and the second image data comprises executing a trained object detection model on the first image data and the second image data (Para. [0076] discloses that the image analysis model can be an object detection “machine-learned” model).
Regarding claim 10, Brueckner discloses that generating the analysis data by executing at least one image analysis model on the first image data and the second image data comprises executing a following distance detection model on the first image data and the second image data (Para. [0062]: “[e]xample sensed objects that can be determined from sensed object data … can include … distances between vehicle 502 and other vehicles and/or objects, etc.”).
Regarding claim 11, to the extent that claim 11 recites limitations that are recited in claim 1, the rejection of claim 1 applies mutatis mutandis to claim 11. Brueckner discloses that the vehicle device includes at least one processor (Fig. 5, processor(s) 624 of the vehicle control device 602 and at least one non-transitory processor-readable storage medium 626 communicatively coupled to the at least one processor 624, the at least one non-transitory processor-readable storage medium storing processor-executable instructions 630 which, when executed by the at least one processor 624, cause the vehicle device to perform the steps recited in claims 1 and 11).
Regarding claim 12-19, the rejection of claim 2-9 apply mutatis mutandis to claims 12-19, respectively.
Regarding claim 20, the system of Brueckner includes the image capture device (Fig. 6, para. [0065], the system 600 includes the image capture device 100).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. Publ. Appl. No. 2024/0304001 A1 discloses a memory 140 configured to store programming instructions that, when executed by the processor 135, may cause the processor 135 to perform one or more tasks such as, e.g., capturing, using a camera, an image depicting an environment within view of the camera, identifying a first section of the image, wherein the first section depicts an area of the environment spaced within a first distance range from the camera, identifying a second section of the image, wherein the second section depicts an area of the environment spaced within a second distance range from the camera, identifying a third section of the image, wherein the third section depicts an area of the environment spaced within a third distance range from the camera, downsampling the first section of the image to a first image resolution, generating a first processed image, downsampling the second section of the image to a second image resolution, generating a second processed image, downsampling the third section of the image to a third image resolution, generating a third processed image, generating a distance map of the environment of the image, prior to downsampling the first section of the image, the second section of the image, and the third section of the image, downsampling the image to an image resolution lower than an original image resolution, and/or other suitable functions.
U.S. Publ. Appl. No. 2021/0004589 A1 discloses a video analysis subsystem 114C that may be configured to augment a frame determined to include an object (e.g., an object specified by the image-capture task) prior to being added to a training data set. For example, if a frame is determined to include the object at a desired perspective, lighting condition, background, etc., then the frame may be cropped so as to reduce an amount of unneeded data. As another example, a region of interest including the object may identified, and portions of the frame outside the region of interest may be compressed to a lower resolution to converse memory requirements for storing the image, or for performing additional analysis of the image prior to being added to the training data set.
U.S. Publ. Appl. No. 2017/0113664 A1 discloses a video analysis subsystem 114C configured to augment a frame determined to include an object (e.g., an object specified by the image-capture task) prior to being added to a training data set. For example, if a frame is determined to include the object at a desired perspective, lighting condition, background, etc., then the frame may be cropped so as to reduce an amount of unneeded data. As another example, a region of interest including the object may identified, and portions of the frame outside the region of interest may be compressed to a lower resolution to converse memory requirements for storing the image, or for performing additional analysis of the image prior to being added to the training data set. In some embodiments, some or all of the functionality of video analysis subsystem 114C may be offloaded to mobile computing device 104 so as to determine, in real-time, whether the candidate video captured the object. In some cases, where some of the functionality of video analysis subsystem 114C is offloaded to mobile computing device 104, the file size savings obtained by some of the aforementioned frame augmentation schemes may achieve less latency in transmitting image data from mobile computing device 104 to remote server system 110C.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL J SANTOS whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL J. SANTOS/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights