DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
Claim(s) 18-37 do not use “means for” (or “step for”) language, or generic placeholders for "means” coupled with functional language without recitation of sufficient structure for carrying out the claimed functions and therefore do not invoke 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 18, 20, 23, 26, 29, 31, 34, 36 and 37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tzur (US 2013/0044254 A1) in view of Official Notice.[claim 18]
Regarding claim 18, Tzur discloses a method for generating an all-in-focus image with a camera, the method comprising:
acquiring a plurality of preview images of a field-of-view of a camera (Figure 1, 102; Paragraphs 0035-0036; capturing several preview images at predefined focus positions or at all focus positions);
generating a depth map of the field-of-view, based at least in part on analysis of the plurality of preview images, the depth map defining focal distances between the mobile camera and objects-of-interest (Figure 1, 102; Paragraphs 0035-0046; calculating depth map from range of focus positions);
determining a selected set of focal distances from the depth map for active autofocus lens sweeping, wherein the determination of the selected set of focal distances is based on the in-focus image information present within the plurality of preview images (Figure 1, 104; Figure 3);
selectively driving the autofocus lens of the mobile camera to the identified selected set of focal distances and capturing a respective sample image at each of the focal distances within the selected set, each sample image comprising an in-focus portion at the respective focal distance (Figure 1, 106); and
producing an all-in-focus image by computationally combining the in-focus portions of each captured sample image with in-focus portions derived from the plurality of preview images (Figure 1, 108).
However, Tzur does not explicitly disclose that the camera is a mobile camera having an autofocus lens.
Official Notice is taken that it is well known in the art to use mobile cameras which include autofocus lenses so that images may be captured in a wide range of locations and the focus for each image may be automatically calculated/set without manual user focusing.
Therefore, it would have been obvious to use a mobile camera with an autofocus lens as the camera of Tzur so that images may be captured in a wide range of locations and the focus for each image may be automatically calculated/set without manual user focusing. [claim 20]
Regarding claim 20, Tzur in view of Official Notice discloses wherein the plurality of preview images are acquired while the autofocus lens is sweeping across a range of focal distances (Tzur, Paragraphs 0035-0036)[claim 23]
Regarding claim 23, Tzur discloses wherein the selected set of focal distances is a subset of all focal distances represented in the real-time depth map (Figure 3; determining principal depths as a subset of all focal distances).[claim 26]
Regarding claim 26, Tzur does not disclose the particular order in which the images are captured. However, it is noted that a finite number of possible capture sequences are possible. Furthermore, one of ordinary skill in the art could have pursued any of these finite sequences with a reasonable expectation of success and no more than predictable results since the particular order of image capture is not critical to the invention of Tzur. Therefore, it would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to perform any of the possible image capture sequences, including one where the first focal distance is nearest a current focal distance of the lens prior to sweeping the autofocus lens of the camera when capturing the additional required images of Tzur.
Further note that since each ordering would be considered obvious, the order which would minimize time that the autofocus lens is sweeping would also be considered obvious. Note that the claim as written does not claim any positive steps of determining the order, merely that an order which happens to have a minimized time is used.[claim 29]
Regarding claim 29, see the rejection of claim 18 above and note that Tzur in view of Official Notice discloses a device comprising a camera and an autofocus lens and processor for performing the recited steps (e.g. Tzur, Paragraph 0063-0064).[claim 31]
Regarding claim 31, Tzur in view of Official Notice discloses an autofocus lens (see rejection of claim 1 above), but does not explicitly disclose that the lens comprises a voice coil motor or MEMS lens.
Official Notice is taken that it is well known in the art to construct autofocus lenses using a voice coil motor or MEMS lens. Such lenses are well understood, commercially available and predictable types of autofocus lenses. Therefore, it would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains to use a voice coil motor lens as the autofocus lens such lenses are well understood, commercially available and predictable types of autofocus lenses. [claim 34]
Regarding claim 34, see the rejection of claim 29 above and note that Tzur discloses a non-transitory computer-readable medium storing instructions for execution by the processor as claimed (Paragraph 0063-0064).[claim 36]
Regarding claim 36 Tzur discloses wherein the instructions to produce the all-in-focus image comprise instructions to extract a portion of each in-focus portion of the captured sample images and the preview images and arrange the extracted portions adjacent to one another to form the all-in-focus image (Paragraphs 0014, 0057-0061; blending portions of the images to form the all-in-focus image).[claim 37]
Regarding claim 37, see the rejection of claim 26 above.
Claim(s) 19 and 35 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tzur (US 2013/0044254 A1) in view of Official Notice in view of Bandwar et al. (US 2018/0343414 A1).[claim 19]
Regarding claim 19 Tzur in view of Official Notice does not disclose wherein the plurality of preview images are acquired during a zero-shutter-lag mode of the mobile camera.
Bandwar discloses a zero-shutter-lag mode for a camera which reduces lag time of image capture (e.g. Paragraphs 0001-0003). Therefore, it would have been obvious to utilize a zero-shutter-lag mode to capture images in the method Brown in view of Tzur so that lag time between capture instruction and image capture may be reduced.[claim 35]
Regarding claim 35, see the rejection of claim 19 above and note that Tzur discloses a non-transitory computer-readable medium storing instructions for execution by the processor as claimed (Paragraph 0063-0064).
Claim(s) 21 and 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tzur (US 2013/0044254 A1) in view of Official Notice in view of Govindarao et al. (US 2017/0374269 A1).[claim 21]
Regarding claim 21, Tzur in view of Official Notice does not teach wherein the analysis of the plurality of preview images is performed using a stereo vision algorithm to generate the real-time depth map.
Govindarao discloses a similar method for generating depth maps for improving focus of an image and further discloses that other systems for capturing depth may be used, such as depth sensors or a stereo camera algorithm may be used to generate a depth map (e.g. Paragraph 0045, 0048, 0083). The use of a stereo camera algorithm for generating depth would allow for depth to be captured faster by capturing multiple frames at once an inferring depth from the stereo frames.
Therefore, it would have been obvious to use a stereo camera algorithm for generating the depth map as taught by Govindarao so that the depth map may be generated faster by using stereo camera preview frames. [claim 30]
Regarding claim 30, see the rejection of claims 29 and 21 above.
Claim(s) 22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tzur (US 2013/0044254 A1) in view of Official Notice in view of Petersson et al. (US 2020/0265622 A1).[claim 22]
Regarding claim 22, Tzur in view of Official Notice does not disclose wherein the determination of the selected set of focal distances is performed by a machine-learned model that is configured to segment the real-time depth map into the selected set of focal distances.
Petersson discloses a method which utilizes a neural network to segment a depth map to identify surfaces, people, and/or other objects (Paragraph 0028, 0037). The use of a neural network would allow for the image to be effectively segmented into component objects without the need to manually determine the optimal method for segmenting the image. Therefore, it would have been obvious to use a neural network to segment the depth map data of Tzur in view of Official Notice to detect principal focus distances for component objects so that the depth map data may be effectively segmented without the need to manually determine the optimal segmentation method.
Claim(s) 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tzur (US 2013/0044254 A1) in view of Official Notice in view of Shin (US 2019/0306404 A1).[claim 24]
Regarding claim 24, Tzur in view of Official Notice does not disclose refining the real-time depth map to normalize a plurality of focal distances into a discrete quantity of focal distances, wherein the selected set of focal distances is based on the discrete quantity of focal distances.
Shin discloses a similar system for determining focal regions from a depth map including refining the real-time depth map to normalize a plurality of focal distances into a discrete quantity of focal distances, wherein the selected set of focal distances is based on the discrete quantity of focal distances (Figures 11-14; normalizing depth data to 1-10 and determining groups based on similar focal lengths for determining a set of focal distances). The normalizing system of Shin would simplify the depth map data for easier processing.
Therefore, it would have been obvious to normalize and process depth map data as taught by Shin so that the depth map processing may be simplified.
Claim(s) 27, 28 and 32 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tzur (US 2013/0044254 A1) in view of Official Notice in view of Chen et al. (US 2005/0100245 A1) in view of Brown (US 2015/0279012 A1).[claim 27]
Regarding claim 27, Tzur in view of Official Notice discloses blending in-focus portions to form an all-in-focus image (Tzur, Paragraphs 0014, 0057-0061), but does not disclose wherein the producing comprises: correcting for geometric distortion and alignment between the captured sample images.
Chen discloses a method for correcting distortions in multi-focus image stacks including correcting for geometric distortions and alignment between the images in the focal stack (Paragraph 0026; Figure 7). Therefore, it would have been obvious to correct the captured images for geometric distortions and alignment so that the captured images may be made to align with each other and the depth map so that the resulting all-in-focus image may be free from errors due to distortions and alignment.
However, Tzur in view of Official Notice in view of Chen does not explicitly disclose that the preview images are used in the generation of the all-in-focus image.
Brown discloses a similar method for generating an all-in-focus image including capturing an image for generation of a depth map, capturing additional images at different focus depths and combining the originally captured image with the additional image to form an all-in-focus image (Figure 4; Paragraph 0040).
By using originally captured images and capturing additional images the number of images used for creation of the all-in-focus image may be increased while reducing the total number of images which must be captured.
Therefore, it would have been obvious to use the originally captured preview images as part of the focus stack for generating the all-in-focus image so that a larger number of images may be used without increasing the number of images which must be captured.
Additionally, following the teachings of Chen, it would have been obvious to correct geometric distortion and alignment such that the preview images and additionally captured images are properly aligned.[claim 28]
Regarding claim 28, Tzur discloses the use of blending functions (see rejection of claim 27 above), but does not explicitly disclose wherein the blending is a multi-band blending algorithm.
Official Notice is taken that it is well known in the art to blend images using a multi-band blending algorithms such as alpha blending for each color channel so that each color channel of the images may be properly blended. Therefore, it would have been obvious to use multi-band blending algorithms when blending images so that a full-color all-in-focus image may be properly formed.[claim 32]
Regarding claim 32, see the rejection of claims 27 and 28 above.
Claim(s) 33 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tzur (US 2013/0044254 A1) in view of Official Notice in view of Parameswaran et al. (US 2018/0227479 A1).[claim 33]
Regarding claim 33, Tzur in view of Official Notice does not teach wherein the processor is further configured to automatically operate the camera in an all-in-focus mode in response to determining that the real- time depth map includes at least two segments with focal distances that are at least a threshold distance apart.
Parameswaran discloses a system which performs scene analysis, including analysis of depth information and automatically selects a mode including an HDR or focal stacking mode such as an extended depth-of-field imaging mode when objects have high variations of depth (Figures 4 and7; Paragraphs 0020, 0069, 0081). Therefore, it would have been obvious to automatically select an all-in-focus or EDoF mode as taught by Parameswaran according to scene analysis including analysis of depth data so that scenes with large variations in depth may be captured with all objects in sharp focus.
Note that the claims do not define the particular threshold.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 18-37 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 13, 15 and 18 of U.S. Patent No. 10,984,513 in view of Tzur, Official Notice, Bandwar, Govindarao, Shin, Chen, Brown or Parameswaran as discussed below. [claim 18]
18. (New) A method for generating an all-in-focus image with a mobile camera, the method comprising:
acquiring a plurality of preview images of a field-of-view of a mobile camera, wherein the mobile camera comprises an autofocus lens;
generating a real-time depth map of the field-of-view, based at least in part on analysis of the plurality of preview images, the depth map defining focal distances between the mobile camera and objects-of-interest;
determining a selected set of focal distances from the real-time depth map for active autofocus lens sweeping, wherein the determination of the selected set of focal distances is based on the in-focus image information present within the plurality of preview images;
selectively driving the autofocus lens of the mobile camera to the identified selected set of focal distances;
capturing a respective sample image at each of the focal distances within the selected set, each sample image comprising an in-focus portion at the respective focal distance; and
producing an all-in-focus image by computationally combining the in-focus portions of each captured sample image with in-focus portions derived from the plurality of preview images.
1. A method for producing an all-in-focus image with a camera of a user equipment, the method comprising:
capturing a plurality of buffer images of a field-of-view of the camera;
storing the buffer images at an image buffer;
determining, based on sensor data, a set of focal distances between the camera and objects-of-interest in the field-of-view;
inferring, based on the sensor data, a plurality of segments in the field-of-view, each segment of the plurality of segments corresponding to a focal distance of the set of focal distances, each segment of the plurality of segments defining a unique focus area within the field-of-view of the camera corresponding with objects-of-interest at similar focal distances;
determining whether a buffer image of the buffer images captures the field-of-view of the camera at any of the focal distances in the set of focal distances;
responsive to determining that a buffer image captures the determined field-of-view of the camera at a focal distance in the set of focal distances and sweeping an autofocus lens of the camera to one or more of the focal distances from the set of focal distances other than the determined focal distance;
capturing sample images, each of the sample images captured at each of the one or more of the focal distances from the set of focal distances swept by the autofocus lens;
combining at least one of the sample images captured at the one or more focal distances swept by the autofocus lens with another image to produce the all-in-focus image; and
outputting, for display, an indication of the all-in-focus image.
Claim 1 of ‘513 does not explicitly claim the use of preview images or generating a depth map using the preview images for determining the focal distances.
However, Tzur teaches a similar method for determining focal distances from a plurality of preview images which are captured and used to calculated a depth map (see rejection of clam 1 above). The use of a depth map provides a convenient way to determine a range of depths of a scene. Therefore, it would have been obvious to form a depth map from a plurality of preview images as taught by Tzur in the method of ‘513 so that the range of depths in the scene may be calculated and stored, and subsequently used for determining a set of focal distances in a convenient manner.[claims 19-24, 27 and 28]
Regarding claims 19-24, 27 and 28, while ‘513 does not explicitly claim these features, they are known the less obvious over the prior art in view of Tzur, Bandwar, Govindarao, Shin and Chen in view of Brown and would be obvious to combine with the method of ‘513 for the same reasons discussed above in the rejections of these claims.
Additionally, regarding claims 20 and 23 it would be obvious to combine these features taught by Tzur so that a depth map may be formed from defocus and a subset of focal distances selected so that the main objects of the scene are in focus without requiring images at every possible focal depth present in the scene to be captured.[claim 25]
Regarding claim 25, see claim 1 of ‘513.[claim 26]
Regarding claim 26, see claim 13 of ‘513.[claim 29]
Regarding claim 29, see the rejection of claim 18 above and claim 14 of ‘513 and note that claim 29 would similarly be obvious over claim 15 of ‘513 in view of Tzur for the same reasons discussed above.[claims 30-33]
Regarding claims 30-33, while ‘513 does not explicitly claim these features, they are known the less obvious over the prior art in view of Official Notice, Govindarao, Chen in view of Brown or Parameswaran and would be obvious to combine with the method of ‘513 for the same reasons discussed above in the rejections of these claims.[claim 34]
Regarding claim 34, see the rejection of claim 18 above and claim 18 of ‘513 and note that claim 34 would similarly be obvious over claim 18 of ‘513 in view of Tzur for the same reasons discussed above.[claims 35-37]
Regarding claims 35-37, they are known the less obvious over the prior art in view of Tzur or Bandwar and would be obvious to combine with the method of ‘513 for the same reasons discussed above in the rejections of these claims.
With respect to claim 36, it would have been obvious to arrange the extracted in-focus regions adjacent to each other so that they may be blended together to form a blended all-in-focus image.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. The following show additional prior art systems/methods for forming depth maps or all-in-focus images:
Kim et al. US 2015/0035855 A1
Lee US 2013/0050430 A1
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIMOTHY J HENN whose telephone number is (571)272-7310. The examiner can normally be reached Monday-Friday ~10-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Twyler Haskins can be reached at (571) 272-7406. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Timothy J Henn/Primary Examiner, Art Unit 2639
Prosecution Insights