DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The IDS(s) submitted on December 27, 2024, has/have been considered and placed in the application file.
Priority
Receipt and acknowledged foreign priority under 35 U.S.C. 119(a)-(d) of certified copies to Japan application No. 2022-103327, filed June 28, 2022, as required by 37 CFR 1.55. Acknowledgment is made of applicant's claim for domestic benefit under 35 U.S.C. 120 and 365(c) to Application No. PCT/JP2023/014410, filed April 7, 2023.
CLAIM INTERPRETATION
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: optical system on which, imaging element which creates, mask installation unit which creates, arithmetic and control unit which outputs in claim(s) 1.
The optical system is disclosed in the PG Pub of the spec ¶¶ 38 and 100; imaging element is described in ¶¶ 38 and 101; the mask installation unit corresponds to the liquid crystal mask unit 40 in the drawings. The specification describes the liquid crystal mask unit 40 as a liquid crystal light shutter 40a having a light shielding portion BM and segments R1 to R5, each segment switched between a light shielding state and a light transmitting state by a voltage on its electrode (¶¶ 106-112). The specification also describes an alternative mechanism that mechanically switches plate-shaped masks, or a multi-diaphragm mechanism with diaphragms that open and close apertures at different positions (¶ 113), and mask plates each provided with an aperture and mechanically switched in front of the optical system (¶ 162). The arithmetic and control unit which outputs a signal for controlling the mask installation unit and the imaging element so that the subject is imaged and acquires a nonmasked captured image of the subject and a masked captured image of the subject together with the non-masked imaging, determination, selection, masked imaging and decoding processing it performs corresponds to a computer. The computer has a processor 11, a memory 12 storing a program P, and an interface 13 (¶¶ 118 to ¶ 120). The processor executes the algorithm of the flowchart of FIG. 2, steps H2 through H8, and of the flowchart of FIG. 13, steps S1 through S10; the determination processing is any of the three methods of FIG.’s 5-7; the selection processing uses the correspondence of FIG. 4 between mask combinations and edge directions.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a “A program for causing a computer to perform” that is non-statutory subject matter. The broadest reasonable interpretation of a claim drawn to a computer-readable medium (also called machine readable medium and other such variations) typically covers forms of non-transitory tangible media and transitory propagating signals per se in view of the ordinary and customary meaning of computer readable media, particularly when the specification is silent. In re Nuijten, 500 F.3d 1346, 1354 (Fed. Cir. 2007). MPEP 2106.03(I).
A claim drawn to such a computer-readable medium that covers both transitory and non-transitory embodiments may be amended to narrow the claim to cover only statutory embodiments to avoid a rejection under 35 U.S.C. § 101 by adding the limitation “non-transitory” to the claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 8-12, 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al., “Scene-adaptive Coded Apertures Imaging,” (hereinafter “Wang”) in view of Zhou et al., “Coded Aperture Pairs for Depth from Defocus and Defocus Deblurring,” (hereinafter “Zhou”) and further in view of Nagahara et al., “Programmable Aperture Camera Using LCoS,” (hereinafter “Nagahara”).
Claims 1, 11, and 20.
An imaging apparatus comprising:
an optical system on which light from a subject is incident (Wang p. 7: “a primary lens (Cannon 50mm/1.8#), a pair of relay lenses (Nikon 50mm/1.8#)”);
an imaging element which receives the light passing through the optical system (Wang p. 7:
“Point Grey Grasshopper sensor … CCD”);
a mask installation unit which creates a state in which any of a plurality of masks prepared in advance is installed (Wang p. 3: “using a Spatial Light Modulator (SLM) consists of a high fill factor Liquid Crystal on Silicon (LCOS) [11] [12] that permits fast code pattern changes”; Wang p. 7: “a Holoeye LC-R 1080 Spatial Light Modulator”; Wang p. 2: “we discretize the edge orientations into nine 20° sectors.”) (Zhou p. 12: “We printed our optimized pair of aperture patterns on high resolution (1 micron) photomasks”; Zhou's aperture patterns are masks prepared in advance since they are optimized and fixed before any image is captured.)
an arithmetic and control unit which outputs a signal for controlling the mask installation unit and the imaging element (Nagahara p. 3: “the aperture pattern must also be updated at the same frame rate and be synchronized with the sensor exposure.”; Nagahara p. 5: “The camera shutter is synchronized with the LCoS device by using an output trigger”; Synchronizing the shutter with the LCoS device controls the sensor and mask to capture the unmasked and coded shots. The non-masked state of the first shot is addressed below.) so that the subject is imaged and acquires a non-masked captured image of the subject and a masked captured image of the subject (Wang p. 3: “we snap N + 1 shots for a dynamic scene with N dominant directions.” Wang supplies the masked images.),
wherein the arithmetic and control unit performs non-masked imaging processing of obtaining the non-masked captured image by imaging the subject (Wang p. 3: “In the first shot we capture the scene with a pinhole aperture and conduct ultra-fast analysis of the scene content to determine the edge directions”) (Wang p. 1: “we capture the scene with a pinhole and analyze the scene content to determine primary edge orientations”; Wang p. 5: “We can obtain the principal directions of the scene and group the pixels into several corresponding regions with a dominant direction”; Calculating edge orientations from the captured scene and grouping pixels gives a dominant direction for each region.),
selection processing of selecting, from among the plurality of the masks prepared in advanceedge direction (Wang p. 5: “the generated 2D patterns are rotated to ensure that the depth discriminative direction coincides the dominant direction of its corresponding region”; Wang p. 6: “each recovered all-focus frame and corresponding estimated depth map only exhibit better performance in the regions corresponding to its aperture pattern than the other shots”; Wang p. 2: “Consider a scene that contains mostly horizontal edge features. Horizontal blurs will provide less useful information in depth discrepancy than vertical blurs.” Wang p. 7: “The results of each single shot displays good depth estimation accuracy in the regions with dominant direction parallel with the aperture code”; The pattern orientation is selected according to the region's edge direction so that the corresponding depth accuracy is best for that, region.)
masked imaging processing of obtaining a plurality of masked captured images by imaging the subject using (Wang p. 3: “we acquire a sequence of defocused images by applying our proposed coding scheme with respect to one of the corresponding principal orientations in each shot”; Applying the coding scheme per principal orientation captures the sequence of masked defocused images.), and decoding processing of obtaining information indicating a depth of the subject at each of a plurality of positions by performing decoding, based on a point spread function unique to each of the masks selected, on the plurality of masked captured images (Wang p. 5: “Denoting the aperture pattern as K, the blur kernels at different depths should be K's scaled versions”; Wang p. 5: “For a non-planar scene, we search for the depth map d that minimizes above energy”; Using each pattern's scaled blur kernel as the PSF and minimizing the energy over depths yields the per pixel depth map.). Claim 11 recites the same non-masked imaging, determination, selection, masked imaging and decoding steps as a subject depth estimation method, and claim 20 recites them as a program for causing a computer to perform them (Wang p. 3: “we develop robust deconvolution algorithms”; The deconvolution algorithms are program instructions that recover the images and estimate depths.). Claims 11 and 20 are rejected on the same grounds as claim 1.
Wang does not teach the strikethrough portions; however, Nagahara in the same field of endeavor teaches “a state in which none of the masks is installed” and imaging the subject “in a state where no mask is installed” (Nagahara p. 13: “We capture a set of defocused images of an IEEE resolution chart using the patterns shown in Figure 1”, the first of those patterns being “(a) Circular Pattern”; Nagahara p. 3: “enables us to dynamically change aperture patterns as needed”; Nagahara p. 6: “works as the aperture stop”; Imaging through the plain circular aperture is imaging with no coded mask installed.). Applicant’s own non-masked state likewise retains fixed aperture BMa in shielding plate BM (SPEC ¶ 108) and is reached when all segments transmit (SPEC ¶ 112), so a plain circular aperture is not one of the prepared masks.
Wang does not teach the strikethrough portions; however, Zhou in the same field of endeavor teaches “a combination of the masks,” imaged one after the other and decoded with the point spread function of each (Zhou p. 4: “we address this problem by taking two images with different coded apertures”; Zhou p. 4: “kd is the point-spread-function (PSF) whose shape is determined by the aperture and whose size d is related to the depth”; Zhou p. 14: "A more compact implementation would be to use a programmable LCD or DMD aperture within a single camera to alternate between the two aperture patterns in quick succession”; Alternating between the two coded apertures yields two images, each associated with its own aperture PSF.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Zhou’s aperture pair on Wang’s LCoS, chosen according to Wang’s dominant edge directions, since Wang teaches that “a scene-adaptive coded aperture is most desirable” (Wang p. 2) and Zhou’s pair recovers “a high quality focused image and a high quality depth map” (Nagahara p. 3). Wang’s concern that the pair “cannot handle dynamic scenes” (Wang p. 2) is handled by Zhou’s suggestion to “alternate between the two aperture patterns in quick succession” on a programmable aperture (Zhou p. 15).
It would also have been obvious to decode each masked image with its own mask’s PSF, because that is how Zhou’s pair works (Zhou p. 4).
Finally, it would have been obvious to take Wang’s analysis shot through Nagahara’s circular apertures instead of a pinhole, because “the pinhole pattern is notorious for its low light efficiency” (Wang p. 2), and the image only needs “large scale edges/structures” (Wang p. 5).
Claims 2 and 12.
Wang, Zhou and Nagahara disclose the imaging apparatus according to claim 1, wherein the mask has a light shielding region in a part of the incident region of the light, and wherein the plurality of the masks is two or more of the masks in which positions of the light shielding regions are different from each other (Wang p. 3: “search for an optimal depth discriminative 1D binary code”; Wang p. 6: “Since each shot is taken with a coded aperture rotated towards to one principal direction.”) (Zhou p. 8: “One example is a pair of small circular patterns with shifted centers (a stereo-like aperture pair)”; A binary code blocks light at its zero cells, and rotating or shifting the aperture moves the blocking regions to different positions.).
Claims 8 and 18.
Wang, Zhou and Nagahara disclose the imaging apparatus according to claim 1, wherein the determination processing is processing of setting a plurality of partial image regions in the non-masked captured image and determining the representative edge direction based on edge images included in the partial image regions (Wang p. 1: “we adopt a spatial partitioning and
temporal propagation scheme”; Wang p. 5: “group the pixels into several corresponding
regions with a dominant direction”; The grouping into corresponding regions by dominant
direction makes those regions’ edge images the source of the direction for each region.).
Claims 9 and 19.
Wang, Zhou and Nagahara disclose the imaging apparatus according to claim 2, wherein the determination processing is processing of setting a plurality of partial image regions in the non-masked captured image and determining the representative edge direction based on edge images included in the partial image regions (Wang p. 1: “we adopt a spatial partitioning and
temporal propagation scheme”; Wang p. 5: “group the pixels into several corresponding
regions with a dominant direction”).
Claim 10.
Wang, Zhou and Nagahara disclose the imaging apparatus according to claim 1, wherein the mask installation unit includes a liquid crystal light shutter (Wang p. 7: “a Holoeye LC-R 1080
Spatial Light Modulator (Reflective.”) (Nagahara p. 3: “LCoS is a reflective liquid crystal
device that has a high fill factor (92%) and high reflectivity(60%)”; The LCoS device is a
liquid crystal element whose cells switch between light passing and blocking.).
Claim(s) 3 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang
in view of Zhou and Nagahara as applied to claims 2 and 12 above, and further in view of
Takeda et al., “Fusing Depth from Defocus and Stereo with Coded Apertures,” (hereinafter “Takeda”).
Claims 3 and 13.
Wang, Zhou and Nagahara disclose the imaging apparatus according to claim 2, wherein the combination of the masks with the relatively highest depth estimation accuracy of the object corresponding to the image representing the edge in the equal direction to the representative edge direction is two or more of the masks (Zhou p. 4: “we address this problem by taking two
images with different coded apertures.”)
Wang, Zhou and Nagahara do not teach the strikethough portion; however, Takeda in the same field of endeavor teaches “two or more of the masks where a direction in which respective main apertures are positionally different is orthogonal to the representative edge direction” (Takeda p. 7: “Zhou et al. provides the best performance since the their technique has similar effect of stereo imaging with a baseline perpendicular to the pattern”; Takeda p. 7: “This is because the aperture shapes (Figure 7(c)) are decentered to the top or bottom of the aperture”; Takeda p. 5: “The horizontal striped pattern had only edges parallel to the epipolar line of stereo rig, but it was deliberately used to assess the performance of defocus depth cue”; Takeda p. 6: “no depth cues were provided since the texture of the scene was parallel to an epipolar line”; The top or bottom aperture decentering gives a vertical positional difference, orthogonal to the horizontal stripe edge direction, and the parallel edge case provides no depth cue.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to orient Zhou's pair so that the direction is orthogonal to Wang's representative edge direction, because the pair “provides the best performance since the their technique has similar effect of stereo imaging with a baseline perpendicular to the pattern” (Takeda p. 7), and texture parallel to the baseline gives “no depth cues” (Takeda p. 6).
Claim(s) 4, 5, 14 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over
Wang in view of Zhou and Nagahara as applied to claims 1, 2, 11 and 12 above, and further in
view of Onuki et al., US 2011/0025904 A1, (hereinafter “Onuki”).
Claims 4, 5, 14, and 15.
Wang, Zhou and Nagahara disclose the imaging apparatus according to claim 1, wherein the determination processing is processing of (Wang p. 5: “We can obtain the principal directions of the scene and group the pixels
into several corresponding regions with a dominant direction.”)
Wang, Zhou and Nagahara do not teach the strikethrough portions; however, Onuki in the same field of endeavor teaches “detecting a predetermined target object in the non-masked captured image” and determining the edge direction “corresponding to a boundary of the target object detected” (Onuki ¶ 97: “detects the presence of a face by applying a known face recognition technique to a shot image, a focus detection area centered on the face area can be set”; Onuki ¶ 98: “can be set using the face area as a center”; Onuki ¶ 114: “recognizes the directional dependence (object pattern) of a change of the object luminance in the focus detection area from the readout imaging pixel information … selects a phase-difference focus detection direction suited to focus detection from the object pattern”; Onuki ¶ 102: “mainly has a vertical line component, that is, a luminance distribution in the horizontal direction … The ridgeline of the mountains on the right side of the frame mainly has a horizontal line component, that is, a luminance distribution in the vertical direction”; The area centered on the detected face spans the face boundary, so the directional pattern it reads from luminance changes corresponds to the object’s own edge (SPEC ¶ 75).).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to take Wang’s representative edge direction from a detected target object, as Onuki does “In the face AF mode, a focus detection area is set in the face area” (Onuki ¶ 109) and “selects a phase-difference focus detection direction suited to focus detection from the object pattern” (Onuki ¶ 114), so that the depth measurement is best on the object of interest.
Allowable Subject Matter
Claims 6, 7, 16, and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record but not relied, yet considered pertinent to the applicant’s disclosure, is listed on the PTO-892 form.
Kane et al., US 2013/0222624 A1, US 2012/0076362 A1 and US 2011/0267477 A1 (range measurement with one or more coded apertures and stored point spread functions);
Piestun et al. , US 2008/0137059 A1 (cited in the international search report);
Yazawa et al., JP 2012-088169 A (structured aperture inserted into and retracted from the subject light beam, a main image captured without it and an auxiliary image with it, and a structured aperture unit holding a plurality of structured apertures);
Magarey et al. , AU 43893/01 A (programmable spatial light modulator in the aperture plane alternately imposing mask functions, including rotated versions, in a single optical path); Koishi et al. , JP 2011-095027 A (cited in the international search report; light-shielding mask with openings of different sizes over adjacent pixels, subject distance estimated from the luminance ratio); and
Kajimura, JP 2015-154112 A (cited in the international search report; light-field reconstruction in which the edge direction detected in each divided region selects the pupil-weighting mask used for that region).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ross Varndell whose telephone number is (571)270-1922. The examiner can normally be reached M-F, 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached at (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ross Varndell/Primary Examiner, Art Unit 2674
Prosecution Insights