DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
The instant application claims benefit based on continuation to non-provisional US Application No. 14/977,522 filed on 12/21/2015. Domestic benefit is acknowledged. Thus, the effective filing date of Claims 1-20 is 12/21/2015.
Information Disclosure Statement
The information disclosure statement (“IDS”) filed on 07/09/2024 was reviewed and the listed references were noted.
Drawings
The 3 page drawings have been considered and placed on record in the file.
Status of Claims
Claims 1-20 are currently pending.
Drawings
The drawings are objected to under 37 CFR 1.83(a). The drawings must show every feature of the invention specified in the claims. Therefore, the first through sixth area of pixels disclosed in Claims 1-20 must be shown or the feature(s) canceled from the claim(s). No new matter should be entered.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 112
6. The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 1-20 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Consider independent Claim 1, the specification does not provide teaching for the recited limitations within this claim. For example, among others, “compute, for each pixel of the pixels, a weighted average luminance based on …”, or “perform local tone mapping (LTM) on a first area …”, or “perform global tone mapping on a third area …”. Review of independent Claims 8 and 14 reveals the above-identified issue, and therefore, these claims are rejected in the same manner as Claim 1. Dependent Claims 2-7, 9-13, and 15-20 are rejected under this section of the rules due to their corresponding dependencies from Claims 1, 8, and 14, respectively.
Claim Rejections - 35 USC § 103
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
Determining the scope and contents of the prior art.
Ascertaining the differences between the prior art and the claims at issue.
Resolving the level of ordinary skill in the pertinent art.
Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
8. Claims 1-7 and 14-19 are rejected under 35 U.S.C. 103 as being unpatentable over Batur (US 20110229019) in view of Adsumilli ( US 9571759 B1 with the filing date of 09/30/2015) and further view of Lee (“Noise reduction and adaptive contrast enhancement for local tone mapping” published on July 05, 2012).
Consider Claim 1, Batur teaches “An image capture device comprising: (Batur; [0056]; “Then, the mean luminance value, blkMean, and the mean absolute luminance deviation, blkMAD, are computed for each of the image blocks (504). The blkMean may be computed by summing the pixel values an image block and taking the average. The blkMAD of an image block may be computed as follows. First determine the differences between the blkMean and each luminance value in the image data block. Then, compute the absolute values of all the differences, and average the absolute differences. The dividing of training images into image blocks and computation of the blkMean and blkMAD for each image block is repeated for all training images in the prototype set (506).” (Emphasis added)) “perform local tone mapping (LTM) on a first area of the pixels of the hyper-hemispherical image” (Batur; [0061]; “For each image block, p control point values are retrieved from the p control point tables using the blkMean and blkMAD for the image block and a local tone mapping curve is determined for the image block by interpolating the p control points (606).”) “(Batur; FIG. 2; Examiner notes the input image shows an image divided into a number of blocks that do not overlap with each other.) “perform global tone mapping (GTM) on a third area of the pixels of the hyper- hemispherical image to obtain a processed image,” (Batur; [0061]; “[0061] FIG. 6 is a flow diagram of a method for BCE in accordance with one or more embodiments of the invention. Initially, a digital image is received and global tone mapping curve is determined for the digital image as per steps 402-410 of the BCE method of FIG. 4 (600).”) “wherein an amount of LTM performed progressively converges to an amount of GTM performed in the second area of the pixels that is between the first area of the pixels and the third area of the pixels;” (Batur; [0061]; “The global tone mapping curve is then merged with each of the local tone mapping curves to generate merged tone mapping curves (607). Any suitable technique for merging the tone mapping curves may be used.”; Examiner notes Batur’s merging of local and global tone maps is analogous to the convergence of the local and global tone maps. ) “and output the processed image.” (Batur; [0045]; “ For tuning of each image, the original image, the luminance histogram for that image, a tuning tone mapping curve with m adjustable control points, and an output image showing the effects of adjustments to control points in the tuning tone mapping curve on brightness and contrast may be displayed.”)
Batur does not explicitly disclose “An image capture device comprising: an image sensor configured to obtain a hyper-hemispherical image comprising pixels and a processor configured to:”. However, in an analogous field of endeavor, Adsumilli teaches “An image capture device comprising: an image sensor configured to obtain a hyper-hemispherical image comprising pixels” (Adsumilli; Col 3, Line 53; “…first image sensor captures a first hyper-hemispherical image…”) “and a processor configured to:” (Adsumilli; Col 4, Line 37-38; “The processor 220 controls other components of the camera based on the instructions that are executed.”). Accordingly, before the effective filing date of the instant application, it would have been obvious to one of ordinary skill in the art to combine Batur with the teachings of Adsumilli to further apply tone mapping to hyper hemispherical images. One of ordinary skill in the art would be motivated to combine Batur and Adsumilli to output enhanced and tone mapped images for improved clarity of a captured environment. Accordingly, the combination of Batur and Adsumilli discloses the above described limitations Claim 1.
The combination of Batur and Adsumilli does not explicitly teach “to remove low frequency variations,”. However, in an analogous field of endeavor, Lee teaches “to remove low frequency variations,” (Lee; Section 2.B; “In the proposed local TM algorithm, the decomposed subbands are filtered using a denoising filter, which consists of bilateral filtering and soft-thresholding. The LL subband (low-frequency subband) is filtered using the bilateral filter, whereas LH, HL, and HH subbands (high-frequency subbands) are smoothed using soft-thresholding for effective noise reduction.” ). Accordingly, before the effective filing date of the instant application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Batur and Adsumilli with the teachings of Lee to further remove low frequency variations with local tone mapping. One of ordinary skill in the art would be motivated to combine Batur, Adsumilli, and Lee to remove low frequency variations to reduce the amount of noise and increase image clarity in the outputted image. Accordingly, the combination of Batur, Adsumilli, and Lee discloses the invention of Claim 1.
Consider Claim 2, the combination of Batur, Adsumilli, and Lee teaches “The image capture device of claim 1, wherein the processor is further configured to perform LTM on each pixel of the first area of pixels” (Batur; [0061] “For each image block, p control point values are retrieved from the p control point tables using the blkMean and blkMAD for the image block and a local tone mapping curve is determined for the image block by interpolating the p control points (606).”) “based on the predefined area of pixels.” (Batur; Abstract; “The method may also include dividing the digital image into a plurality of image blocks…”).
Consider Claim 3, the combination of Batur, Adsumilli, and Lee teaches “The image capture device of claim 2, wherein the predefined area of pixels is a 100 x 100 pixel area.” (Batur; [0056]; “Further, any suitable image block size may be used.”)
Consider Claim 4, the combination of Batur, Adsumilli, and Lee teaches “The image capture device of claim 1, wherein the processor is further configured to perform LTM on each pixel of the first area of pixels to” (Batur; [0061] “For each image block, p control point values are retrieved from the p control point tables using the blkMean and blkMAD for the image block and a local tone mapping curve is determined for the image block by interpolating the p control points (606).”) “enhance high frequency details.” (Batur; [0062] “[0062] Once the merged tone mapping curves are determined, each pixel in the image is enhanced based on the merged tone mapping curves (608-616).”).
Consider Claim 5, the combination of Batur, Adsumilli, and Lee teaches “The image capture device of claim 1, wherein the processor is further configured to perform LTM on each pixel of the first area of pixels to” (Batur; [0061] “For each image block, p control point values are retrieved from the p control point tables using the blkMean and blkMAD for the image block and a local tone mapping curve is determined for the image block by interpolating the p control points (606).”) “preserve high frequency details.” (Adsumilli; Col 2, Line 28-31; “ Use of separate range tone mapping for combined images can help minimize the loss of image information in scenes that have different luminance characteristics, i.e., scenes that have both bright and shadowed components.”). The proposed combination as well as the motivation for combining the Batur, Adsumilli, and Lee references presented in the rejection of claim 1, apply to claim 5 and are incorporated herein by reference. Thus, the method recited in claim 5 is met by Batur and Adsumilli and Lee.
Consider Claim 6, the combination of Batur, Adsumilli, and Lee teaches “The image capture device of claim 1, further comprising: a second image sensor configured to obtain a second hyper-hemispherical image; wherein the processor is further configured to (Adsumilli; Col 3, Line 43-46; “ In order to capture spherical content, a spherical camera has at least two lenses that capture overlapping images that can be combined to form a spherical image using a stitching algorithm.”) “output the processed image in a 360 output format.” (Adsumilli; Col 3, Line 41-44; “A camera array can also take the form of a spherical camera system. By definition, a spherical camera system will capture everything in the surrounding environment (e.g., 360 degrees in the horizontal plane and 180 degrees in the vertical plane).”). The proposed combination as well as the motivation for combining the Batur, Adsumilli and Lee references presented in the rejection of claim 1, apply to claim 6 and are incorporated herein by reference. Thus, the method recited in claim 6 is met by Batur, Adsumilli and Lee.
Consider Claim 7, the combination of Batur, Adsumilli, and Lee teaches “The image capture device of claim 6, wherein the 360 output format is a stitched pair fish-eye format, an equi-angular cube map (EAC) format, or an equi-rectangular projection (ERP) format.” (Adsumilli; Col 6, Line 26-35; “Examples of image combination or “stitching” techniques can be found in co-pending U.S. patent application Ser. No. 14/308,507, titled “Image Taping in a Multi-Camera Array”, filed Jun. 18, 2014; ….U.S. patent application Ser. No. 14/637,180, titled “Generation of Video from Spherical Content Using Edit Maps”, filed Mar. 3, 2015, the contents of each of which are incorporated herein by reference in their entirety.”; Examiner notes both referenced patent applications disclose fish-eye cameras and images.). The proposed combination as well as the motivation for combining the Batur, Adsumilli, and Lee references presented in the rejection of claim 1, apply to claim 7 and are incorporated herein by reference. Thus, the method recited in claim 7 is met by Batur, Adsumilli, and Lee.
Consider Claim 14, Batur teaches “(Batur; [0056]; “Then, the mean luminance value, blkMean, and the mean absolute luminance deviation, blkMAD, are computed for each of the image blocks (504). The blkMean may be computed by summing the pixel values an image block and taking the average. The blkMAD of an image block may be computed as follows. First determine the differences between the blkMean and each luminance value in the image data block. Then, compute the absolute values of all the differences, and average the absolute differences. The dividing of training images into image blocks and computation of the blkMean and blkMAD for each image block is repeated for all training images in the prototype set (506).”) “perform local tone mapping (LTM) on a first area of the pixels of the first image portion (Batur; [0061] “For each image block, p control point values are retrieved from the p control point tables using the blkMean and blkMAD for the image block and a local tone mapping curve is determined for the image block by interpolating the p control points (606).”) “wherein the first area of the pixels does not overlap with a second area of the pixels;” (Batur; FIG. 2; Examiner notes the input image shows an image divided into a number of blocks that do not overlap with each other.) “perform global tone mapping (GTM) on a third area of the pixels of the first image portion to obtain a processed image,” (Batur; [0061]; “[0061] FIG. 6 is a flow diagram of a method for BCE in accordance with one or more embodiments of the invention. Initially, a digital image is received and global tone mapping curve is determined for the digital image as per steps 402-410 of the BCE method of FIG. 4 (600).”) “wherein an amount of LTM performed converges to an amount of GTM performed in the second area of the pixels;” (Batur; [0061]; “The global tone mapping curve is then merged with each of the local tone mapping curves to generate merged tone mapping curves (607). Any suitable technique for merging the tone mapping curves may be used.”; Examiner notes Batur’s merging of local and global tone maps is analogous to the convergence of the local and global tone maps.) “and output the processed image.” (Batur; [0045]; “ For tuning of each image, the original image, the luminance histogram for that image, a tuning tone mapping curve with m adjustable control points, and an output image showing the effects of adjustments to control points in the tuning tone mapping curve on brightness and contrast may be displayed.”)
Batur does not explicitly disclose “An image capture device comprising: a first image sensor configured to obtain a first hyper-hemispherical image that has a first image portion comprising pixels; a second image sensor configured to obtain a second hyper-hemispherical image that has a second image portion; and a processor configured to:”. However, in an analogous field of endeavor, Adsumilli teaches “An image capture device comprising: a first image sensor configured to obtain a first hyper-hemispherical image that has a first image portion comprising pixels;” (Adsumilli; Col 3, Line 51-55; “A first lens 102a of the spherical camera system 100 has field of view 110a with boundary 104a, in front of which the first image sensor captures a first hyper-hemispherical image plane from light entering the first lens 102a.”) “a second image sensor configured to obtain a second hyper-hemispherical image that has a second image portion; and” (Adsumilli; Col 3, Line 55-58; “A second lens 102b of the spherical capture system has field of view 112b with boundary 104b, in front of which the second image sensor captures a second hyper-hemispherical image plane from light entering the second lens 102b.”) “a processor configured to:” (Adsumilli; Col 4, Line 37-38; “The processor 220 controls other components of the camera based on the instructions that are executed.”). Accordingly, before the effective filing date of the instant application, it would have been obvious to one of ordinary skill in the art to combine Batur with the teachings of Adsumilli to further apply tone mapping to hyper hemispherical images. One of ordinary skill in the art would be motivated to combine Batur and Adsumilli to output enhanced and tone mapped 360 degree images for improved clarity of a captured environment. Accordingly, the combination of Batur and Adsumilli discloses the above described limitations of Claim 14.
The combination of Batur and Adsumilli does not explicitly teach “to remove low frequency variations,”. However, in an analogous field of endeavor, Lee teaches “to remove low frequency variations,” (Lee; Section 2.B; “In the proposed local TM algorithm, the decomposed subbands are filtered using a denoising filter, which consists of bilateral filtering and soft-thresholding. The LL subband (low-frequency subband) is filtered using the bilateral filter, whereas LH, HL, and HH subbands (high-frequency subbands) are smoothed using soft-thresholding for effective noise reduction.” ). Accordingly, before the effective filing date of the instant application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Batur and Adsumilli with the teachings of Lee to further remove low frequency variations with local tone mapping. One of ordinary skill in the art would be motivated to combine Batur, Adsumilli, and Lee to remove low frequency variations to reduce the amount of noise and increase image clarity in the outputted image. Accordingly, the combination of Batur, Adsumilli, and Lee discloses the above described limitations of invention of Claim 14.
Consider Claim 15, the combination of Batur, Adsumilli, and Lee teaches “The image capture device of claim 14, wherein the processor is further configured to: perform LTM on a fourth area of pixels of the second image portion;” (Batur; [0061]; “For each image block, p control point values are retrieved from the p control point tables using the blkMean and blkMAD for the image block and a local tone mapping ]curve is determined for the image block by interpolating the p control points (606).”) and perform GTM on a fifth area of pixels of the second image portion when” (Batur; [0061]; “[0061] FIG. 6 is a flow diagram of a method for BCE in accordance with one or more embodiments of the invention. Initially, a digital image is received and global tone mapping curve is determined for the digital image as per steps 402-410 of the BCE method of FIG. 4 (600).”) “a portion of the predefined area of pixels overlaps with the first image portion.” (Adsumilli; Abstract; “Multiple tone maps that are applied to overlapping regions of the plurality of images can subsequently be combined to expand the bit depth of the overlapping regions.”). The proposed combination as well as the motivation for combining the Batur, Adsumilli, and Lee references presented in the rejection of claim 14, apply to claim 15 and are incorporated herein by reference. Thus, the method recited in claim 15 is met by Batur, Adsumilli, and Lee.
Consider Claim 16, the combination of Batur, Adsumilli, and Lee teaches “The image capture device of claim 15, wherein the processor is further configured to perform LTM on each pixel of the first area of pixels and the fourth area of pixels based on the predefined area of pixels. (Batur; [0061]; “For each image block, p control point values are retrieved from the p control point tables using the blkMean and blkMAD for the image block and a local tone mapping curve is determined for the image block by interpolating the p control points (606).”)
Consider Claim 17, the combination of Batur, Adsumilli, and Lee teaches “The image capture device of claim 15, wherein the predefined area of pixels is a100 x 100 pixel area.” (Batur; [0056]; “Further, any suitable image block size may be used.”).
Consider Claim 18, the combination of Batur, Adsumilli, and Lee teaches “The image capture device of claim 15, wherein the processor is further configured to perform LTM on each pixel of the fourth area of pixels to remove low frequency variations.” (Batur; [0061] “For each image block, p control point values are retrieved from the p control point tables using the blkMean and blkMAD for the image block and a local tone mapping curve is determined for the image block by interpolating the p control points (606).”).
Consider Claim 19, the combination of Batur, Adsumilli, and Lee teaches “The image capture device of claim 18, wherein the processor is further configured to perform LTM on each pixel of the first area of pixels and the fourth area of pixels” (Batur; [0061]; “For each image block, p control point values are retrieved from the p control point tables using the blkMean and blkMAD for the image block and a local tone mapping curve is determined for the image block by interpolating the p control points (606).”) “to preserve high frequency details.” (Adsumilli; Col 2, Line 28-31; “ Use of separate range tone mapping for combined images can help minimize the loss of image information in scenes that have different luminance characteristics, i.e., scenes that have both bright and shadowed components.”). The proposed combination as well as the motivation for combining the Batur, Adsumilli, and Lee references presented in the rejection of claim 14, apply to claim 19 and are incorporated herein by reference. Thus, the method recited in claim 19 is met by Batur, Adsumilli and Lee.
Allowable Subject Matter
9. Claims 8-13 are not rejected over prior art and could be allowable, if the above described 112(a) rejection is overcome. The following is a statement of reasons for the indication of allowable subject matter:
Consider Claim 8, Adsumilli teaches “A method comprising: obtaining a hyper-hemispherical image that has a first image portion;” (Adsumilli; (42); “The first lens 102a of the spherical camera system 100 captures a first hyper-hemispherical image plane of the environment”), “dividing the hyper-hemispherical image into a plurality of blocks;” (Adsumilli; (38); “…each image is further divided into zones based on position or image content”).
In an analogous field endeavor, Prechtl (US 20070024701) teaches “determining whether a block of the plurality of blocks contains a portion of the first image portion;” (Prechtl; [0061]; “The edges of the two images are detected (step 403), and overlapping pixels are identified (step 405).”) “determining whether the block overlaps a second image portion;” (Prechtl; [0061]; “The edges of the two images are detected (step 403), and overlapping pixels are identified (step 405).”)
In an analogous field of endeavor, Zhou (US 20140119650) teaches “computing a neighborhood luminance of a pixel and” (Zhou; [0186]; “…the correcting section corrects the color-difference signal of the pixel of interest based on color-difference signals and luminance signals of the pixel of interest and its neighboring pixels”) “applying a gain to the pixel based on the neighborhood luminance” (Zhou; [0099]; “In the gain calculating section 20, the integrators 21 to 24 integrate the color signals SR and SB, and the luminance signals SY, and determines the gain G based on the results of the integration.”) “(Zhou; [0186]; “…the correcting section corrects the color-difference signal of the pixel of interest based on color-difference signals and luminance signals of the pixel of interest…”) “applying a gain to the pixel based on the luminance;” (Zhou; [0099]; “In the gain calculating section 20, the integrators 21 to 24 integrate the color signals SR and SB, and the luminance signals SY, and determines the gain G based on the results of the integration.”).
In an analogous field of endeavor, Batur teaches “…(Batur; [0045]; “For tuning of each image, the original image, the luminance histogram for that image, a tuning tone mapping curve with m adjustable control points, and an output image showing the effects of adjustments to control points in the tuning tone mapping curve on brightness and contrast may be displayed.”).
None of the cited prior art references, alone or in combination, provides a motivation to teach the ordered combination of the limitations recited in Claim 8.
Consider Claims 9-13, none of the cited prior art references, alone or in combination, provides a motivation to teach the ordered combination of the limitations recited in Claims 9-13 with the limitations of claims it depends from.
10. Claim 20 is objected to as being dependent upon a rejected base claim, but could be allowable if (1) rewritten in independent form including all of the limitations of the base claim and any intervening claims, and (2) if the above described 112(a) rejection is overcome. The following is a statement of reasons for the indication of allowable subject matter: none of the cited prior art references, alone or in combination, provides a motivation to teach the ordered combination of the limitations recited in Claim 20 with the limitations of claims it depends from.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Annie Pham whose telephone number is (571)272-1673. The examiner can be normally be reached Mon-Fri 9:00a – 5:00p.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANNIE H PHAM/Examiner, Art Unit 2662
/Siamak Harandi/Primary Examiner, Art Unit 2662
Prosecution Insights