Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-20 were pending. Claims 1, 9, 14 have been amended. No claims have been canceled and no new claims have been added. Thus claims 1-20 are currently pending including independent claims 1, 9, 14.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 3, 5-6, 9, 11, 13-14, 16, 19 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Holub (US 20240412188 A1).
Examiner Note: Holub was effectively filed 12/23/2022 because it claims priority to Provisional application 63/435188 which discloses the subject matter relied upon in this office action (See pages 3-6, 13-15, 20, 45 of PRO 63/435188).
Regarding claim 1, Holub discloses a method of image processing ([0046] image processing), the method comprising:
acquiring a plurality of input image frames ([0052] a plurality of input image frames);
detecting at least one object of interest in individual image frames of the plurality of image frames ([0055] an identified item of interest in each image frame for the purpose of placing a bounding region that encompasses it);
for the individual image frames, producing a respective bounding box corresponding to the at least one object of interest, the bounding box describing coordinates of a boundary of the object of interest within a respective individual image frame ([0055] rectangular bounding regions are placed encompassing items of interest in the frames, the boundary describing coordinates of the region of interest);
temporally averaging corresponding pixel values of pixels within the bounding box over the plurality of image frames to produce a plurality of averaged pixel values ([0058] generate an average pixel value for each pixel within the boundary across multiple frames); and
producing an output image in which pixels within an area of the output image described by coordinates of the bounding box are replaced with the averaged pixel values ([0100] a composite frame (i.e. output image) is generated in which the square regions (i.e. bounding boxes) contain the averaged values of pixels)
while leaving pixels outside of the area described by coordinates of the bounding box unchanged ([0277]-[0278] background subtraction is performed to identify a static background (i.e. pixels outside the bounding box area) and other regions of interest (i.e. bounding box area), wherein only the regions of interest are averaged across accumulated frames and the background is unchanged; [0058] average pixel values are calculated for accumulated values of pixels within the boundary box 31 (Fig. 3A)).
Regarding claim 3, Holub discloses the method of claim 1 as applied above. Holub further discloses wherein detecting the at least one object of interest includes processing the individual image frames using one or more convolutional neural networks ([0145] a convolutional neural network to perform object detection).
Regarding claim 5, Holub discloses the method of claim 1 as applied above. Holub further discloses wherein acquiring the plurality of input image frames includes accessing the plurality of input image frames from at least one non-transitory machine-readable storage medium ([0328] memory in forms of tangible media, [0055] frames are stored in a memory, Fig. 1).
Regarding claim 6, Holub discloses the method of claim 1 as applied above. Holub further discloses wherein acquiring the plurality of input image frames includes receiving the plurality of input image frames from an imaging device ([0052] images are captured by a camera), wherein the individual image frames are acquired by the imaging device under a very low light condition ([0106] useful in low-light conditions).
Regarding claim 9, Holub discloses a computer program product including one or more non-transitory machine-readable mediums having instructions encoded thereon that when executed by at least one processor cause an image processing method to be carried out ([0328] memory in forms of tangible media (i.e. non-transitory machine-readable mediums); [0046] image processing method), the method comprising:
acquiring a plurality of input image frames ([0052] a plurality of input image frames);
detecting at least one object of interest in individual image frames of the plurality of image frames ([0055] an identified in each image frame for the purpose of placing a bounding region that encompasses it);
for the individual image frames, producing a respective bounding box corresponding to the at least one object of interest, the bounding box describing coordinates of a boundary of the object of interest within a respective individual image frame ([0055] rectangular bounding regions are placed encompassing items of interest in the frames, the boundary describing coordinates of the region of interest);
temporally averaging corresponding pixel values of pixels within the bounding box over the plurality of image frames to produce a plurality of averaged pixel values ([0058] generate an average pixel value for each pixel within the boundary across multiple frames); and
producing an output image in which pixels within an area of the output image described by coordinates of the bounding box are replaced with the averaged pixel values ([0100] a composite frame (i.e. output image) is generated in which the square regions (i.e. bounding boxes) contain the averaged values of pixels)
while leaving pixels outside of the area described by coordinates of the bounding box unchanged ([0277]-[0278] background subtraction is performed to identify a static background (i.e. pixels outside the bounding box area) and other regions of interest (i.e. bounding box area), wherein only the regions of interest are averaged across accumulated frames and the background is unchanged; [0058] average pixel values are calculated for accumulated values of pixels within the boundary box 31 (Fig. 3A)).
Regarding claim 11, Holub discloses the computer program product of claim 9 as applied above. Holub further discloses wherein detecting the at least one object of interest includes processing the individual image frames using one or more convolutional neural networks ([0145] a convolutional neural network to perform object detection).
Regarding claim 13, Holub discloses the computer program product of claim 9 as applied above. Holub further discloses wherein acquiring the plurality of input image frames includes accessing the plurality of input image frames from at least one non-transitory machine-readable storage medium ([0328] memory in forms of tangible media, [0055] frames are stored in a memory, Fig. 1).
Regarding claim 14, Holub discloses an image sensor comprising: an imaging device configured to acquire a temporal series of image frames ([0052] camera acquiring a plurality of input image frames); and a digital signal processing module coupled to the imaging device ([0050], [0059] processors coupled to the camera; )and configured to:
process the image frames to detect at least one object of interest in individual image frames of the image frames ([0055] an identified in each image frame for the purpose of placing a bounding region that encompasses it),
for the individual image frames, produce a respective bounding box corresponding to the at least one object of interest, the bounding box describing coordinates of a boundary of the object of interest within a respective individual image frame ([0055] rectangular bounding regions are placed encompassing items of interest in the frames, the boundary describing coordinates of the region of interest),
average corresponding pixel values of pixels within the bounding box over the temporal series of image frames to produce a plurality of averaged pixel values ([0058] generate an average pixel value for each pixel within the boundary across multiple frames), and
produce an output image in which at least some pixels within an area of the output image described by coordinates of the bounding box are replaced with the averaged pixel values ([0100] a composite frame (i.e. output image) is generated in which the square regions (i.e. bounding boxes) contain the averaged values of pixels)
while leaving pixels outside of the area described by coordinates of the bounding box unchanged ([0277]-[0278] background subtraction is performed to identify a static background (i.e. pixels outside the bounding box area) and other regions of interest (i.e. bounding box area), wherein only the regions of interest are averaged across accumulated frames and the background is unchanged; [0058] average pixel values are calculated for accumulated values of pixels within the boundary box 31 (Fig. 3A)).
Regarding claim 16, Holub discloses the image sensor of claim 14 as applied above. Holub further discloses wherein to detect the at least one object of interest, the digital signal processing module is configured to process the individual image frames using one or more convolutional neural networks ([0145] a convolutional neural network to perform object detection).
Regarding claim 19, Holub discloses the image sensor of claim 14 as applied above. Holub further discloses wherein the imaging device is configured to acquire the image frames at a frame rate of between 45 and 90 frames per second ([0102] the frames may be captured at a rate of 60 frames per second).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2, 4, 7, 10, 12, 15, 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Holub (US 20240412188 A1) in view of Mao (US 20210365707).
Examiner Note: Holub was effectively filed 12/23/2022 because it claims priority to Provisional application 63/435188 which discloses the subject matter relied upon in this office action (See pages 3-6, 13-15, 20, 45 of PRO 63/435188).
Regarding claim 2, Holub discloses the method of claim 1 as applied above. Holub fails to disclose wherein producing the bounding box includes: determining a center point and at least one sizing parameter of the bounding box in a first image frame of the plurality of image frames; and constraining the center point of the bounding box in a second image frame of the plurality of image frames based at least in part on a weighting factor applied to the center point of the bounding box in the first image frame.
Mao discloses wherein producing the bounding box includes: determining a center point and at least one sizing parameter of the bounding box in a first image frame of the plurality of image frames ([0162] a center point and a sizing parameter of a bounding box around an object are determined); and constraining the center point of the bounding box in a second image frame of the plurality of image frames based at least in part on a weighting factor applied to the center point of the bounding box in the first image frame ([0199]-[0202] bounding box center point trajectory smoothing across frames including a weighting factor, [equation 00002]).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Mao with Holub and determine a center point and size parameter and constraining the center point of a bounding box based on a weighting factor, as disclosed by Mao, as part of a method of image processing as disclosed by Holub for the purpose of reducing errors in object detection and bounding box locations (See Mao [0258]).
Regarding claim 4, Holub discloses the method of claim 1 as applied above. Holub fails to disclose wherein producing the output image includes producing the output image including the bounding box positioned around the at least one object of interest.
Mao discloses wherein producing the output image includes producing the output image including the bounding box positioned around the at least one object of interest ([0176] a frame is output in which a bounding box is visible around an object of interest).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Mao with Holub and produce an output image including the bounding box, as disclosed by Mao, as part of a method of image processing as disclosed by Holub for the purpose of providing a clear visualization of the results of the object detection (See Mao [0075]-[0076]).
Regarding claim 7, Holub discloses the method of claim 1 as applied above. Holub fails to disclose further comprising displaying the output image on a display.
Mao discloses further comprising displaying the output image on a display ([0176] displaying the output image).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Mao with Holub and display the output image, as disclosed by Mao, as part of a method of image processing as disclosed by Holub for the purpose of providing a clear visualization of the results of the object detection (See Mao [0075]-[0076]).
Regarding claim 10, Holub discloses the computer program product of claim 9 as applied above. Holub fails to disclose wherein producing the bounding box includes: determining a center point and at least one sizing parameter of the bounding box in a first image frame of the plurality of image frames; and constraining the center point of the bounding box in a second image frame of the plurality of image frames based at least in part on a weighting factor applied to the center point of the bounding box in the first image frame.
Mao, in a related system from the same field of endeavor of image processing for object detection (Abstract), discloses wherein producing the bounding box includes: determining a center point and at least one sizing parameter of the bounding box in a first image frame of the plurality of image frames ([0162] a center point and a sizing parameter of a bounding box around an object are determined); and constraining the center point of the bounding box in a second image frame of the plurality of image frames based at least in part on a weighting factor applied to the center point of the bounding box in the first image frame ([0199]-[0202] bounding box center point trajectory smoothing across frames including a weighting factor, [equation 00002]).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Mao with Holub and determine a center point and size parameter and constraining the center point of a bounding box based on a weighting factor, as disclosed by Mao, as part of a computer program product for image processing as disclosed by Holub for the purpose of reducing errors in object detection and bounding box locations (See Mao [0258]).
Regarding claim 12, Holub discloses the computer program product of claim 9 as applied above. Holub fails to disclose wherein producing the output image includes producing the output image including the bounding box positioned around the at least one object of interest.
Mao, in a related system from the same field of endeavor of image processing for object detection (Abstract), discloses wherein producing the output image includes producing the output image including the bounding box positioned around the at least one object of interest ([0176] a frame is output in which a bounding box is visible around an object of interest).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Mao with Holub and produce an output image including the bounding box, as disclosed by Mao, as part of a computer program product for image processing as disclosed by Holub for the purpose of providing a clear visualization of the results of the object detection (See Mao [0075]-[0076]).
Regarding claim 15, Holub discloses the image sensor of claim 14 as applied above. Holub fails to disclose wherein to produce the bounding box, the digital signal processing module is configured to: determine a center point and at least one sizing parameter of the bounding box in a first image frame of the plurality of image frames; and constrain the center point of the bounding box in a second image frame of the plurality of image frames based at least in part on a weighting factor applied to the center point of the bounding box in the first image frame.
Mao, in a related system from the same field of endeavor of image processing for object detection (Abstract), discloses wherein to produce the bounding box, the digital signal processing module is configured to: determine a center point and at least one sizing parameter of the bounding box in a first image frame of the plurality of image frames ([0162] a center point and a sizing parameter of a bounding box around an object are determined); and constrain the center point of the bounding box in a second image frame of the plurality of image frames based at least in part on a weighting factor applied to the center point of the bounding box in the first image frame ([0199]-[0202] bounding box center point trajectory smoothing across frames including a weighting factor, [equation 00002]).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Mao with Holub and determine a center point and size parameter and constraining the center point of a bounding box based on a weighting factor, as disclosed by Mao, as part of an image sensor as disclosed by Holub for the purpose of reducing errors in object detection and bounding box locations (See Mao [0258]).
Regarding claim 17, Holub discloses the image sensor of claim 14 as applied above. Holub fails to disclose wherein to produce the output image, the digital signal processing module is configured to produce the output image including the bounding box positioned around the at least one object of interest.
Mao, in a related system from the same field of endeavor of image processing for object detection (Abstract), discloses wherein to produce the output image, the digital signal processing module is configured to produce the output image including the bounding box positioned around the at least one object of interest ([0176] a frame is output in which a bounding box is visible around an object of interest).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Mao with Holub and produce an output image including the bounding box, as disclosed by Mao, as part of an image sensor as disclosed by Holub for the purpose of providing a clear visualization of the results of the object detection (See Mao [0075]-[0076]).
Regarding claim 18, Holub discloses the image sensor of claim 14 as applied above. Holub fails to disclose further comprising a display coupled to the digital signal processing module and configured to display the output image.
Mao, in a related system from the same field of endeavor of image processing for object detection (Abstract), discloses further comprising a display coupled to the digital signal processing module and configured to display the output image ([0176] displaying the output image on a display device, [0097] the display device is coupled to the processing system).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Mao with Holub and display the output image on a display, as disclosed by Mao, as part of an image sensor as disclosed by Holub for the purpose of providing a clear visualization of the results of the object detection (See Mao [0075]-[0076]).
Claim(s) 8, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Holub (US 20240412188 A1) in view of Gao (US 20220245792).
Examiner Note: Holub was effectively filed 12/23/2022 because it claims priority to Provisional application 63/435188 which discloses the subject matter relied upon in this office action (See pages 3-6, 13-15, 20, 45 of PRO 63/435188).
Regarding claim 8, Holub discloses the method of claim 1 as applied above. Holub further discloses applying a filter to the individual image frames to produce corresponding individual filtered image frames ([0060] the images are filtered to produce filtered photos); determining the coordinates of the bounding box in a respective filtered image frame; and applying the bounding box to the respective individual image frame ([0063] bounding box coordinates are identified in the filtered photo and the box is applied to the photo).
Holub fails to disclose wherein the filter is a spatial averaging filter.
Gao, in a related system from the same field of endeavor of image processing including object detection (Abstract, [0009]), discloses wherein the filter is a spatial averaging filter ([0170] selective blur filter is applied based on a threshold wherein a difference between pixel values that meet the threshold are blurred together).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Gao with Holub and apply a spatial averaging filter to images, as disclosed by Gao, as part of a method of image processing as disclosed by Holub for the purpose of accurately determining features of the images (See Gao [0172]).
Regarding claim 20, Holub discloses the image sensor of claim 14 as applied above. Holub further discloses wherein to detect the at least one object of interest and to produce the respective bounding box, the digital signal processing module is configured to: apply a filter to the individual image frames to produce corresponding individual filtered image frames ([0060] the images are filtered to produce filtered photos); determine the coordinates of the bounding box in a respective filtered image frame; and apply the bounding box to the respective individual image frame ([0063] bounding box coordinates are identified in the filtered photo and the box is applied to the photo).
Holub fails to disclose wherein the filter is a spatial averaging filter.
Gao, in a related system from the same field of endeavor of image processing including object detection (Abstract, [0009]), discloses wherein the filter is a spatial averaging filter ([0170] selective blur filter is applied based on a threshold wherein a difference between pixel values that meet the threshold are blurred together).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to combine Gao with Holub and apply a spatial averaging filter to images, as disclosed by Gao, as part of an image sensor as disclosed by Holub for the purpose of accurately determining features of the images (See Gao [0172]).
Response to Arguments
Applicant's arguments filed 02/20/2026 have been fully considered and they are partially persuasive.
Applicant asserts on page 8 that "Holub therefore does not teach or disclose producing an output image in which pixels within an area of the output image described by coordinates of a bounding box are replaced with the averaged pixel values while leaving pixels outside of the area of described by the bounding box unchanged."
Examiner disagrees. As stated above in the rejection of amended claim 1 under 35 U.S.C. 102, Holub does disclose leaving pixels outside of a designated area unchanged while pixels within a boundary area are averaged (See Holub: [0277]-[0278] and [0058], Fig. 3A). Thus, the rejection of amended claim 1 (and amended claims 9, 14 similarly) under 35 U.S.C. 102 under Holub is maintained.
Applicant asserts on page 9 that "Mao does not teach or suggest producing an output image where pixels only within the bounding box are replaced with averaged pixel values."
Examiner agrees. However, the rejection of amended claim 1 (and amended claims 9, 14 similarly) under 35 U.S.C. 102 under Holub is maintained as applied above.
Applicant asserts on page 10 that "Gao does not teach or suggest producing an output image where pixels only within the bounding box are replaced with averaged pixel values."
Examiner agrees. However, the rejection of amended claim 1 (and amended claims 9, 14 similarly) under 35 U.S.C. 102 under Holub is maintained as applied above.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAROLINE DEPALMA whose telephone number is (571)270-0769. The examiner can normally be reached Mon-Thurs 9:00am-4pm Eastern Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Moyer can be reached at 571-272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CAROLINE E. DEPALMA/Examiner, Art Unit 2675
/SJ Park/Primary Examiner, Art Unit 2675
Prosecution Insights