DETAILED ACTION
I. Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
II. Priority
A. Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 365(c) or 386(c) is acknowledged.
B. Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). Receipt of the certified copies of papers required by 37 CFR 1.55 is also acknowledged.
III. Claim Interpretation
No claim limitations invoke interpretation under 35 U.S.C. § 112(f). The claims recite a first generation unit, second generation unit, imaging control unit, phase-difference detection unit, control device, first learning unit, second learning unit, learning data generation unit, and imaging processing device. Each recites a generic placeholder (unit, device) and is followed by an associated function, thereby satisfying prongs (A) and (B) of the test for determining whether a claim limitation invokes 112(f) interpretation of MPEP 2181(I). Additionally, none of these limitations is modified by sufficient structure, material, or acts for accomplishing their associated functions (prong (C)). Nevertheless, the examiner submits that these limitations do not invoke interpretation under 35 U.S.C. 112(f) because the disclosure imparts structure to the components associated with the claimed units and devices through internal connection and external connection to other elements. Specifically, Fig. 7 illustrates a control unit/device including first and second generation/learning units that receive an input captured image data and that output data to an imaging control unit. Fig. 7 also illustrates a phase-difference detection unit receiving sensor data and outputting data to the imaging control unit. Fig. 10 illustrates a learning data generation unit as an internal component of the second generation/learning unit, and although Fig. 10 does not show any specific connections associated with this unit, the specification discloses the data that the unit receives and the data that it outputs to the model learning unit. Finally, Fig. 9 illustrates internal, interconnected components embodying an information processing device.
IV. Double Patenting
The claims of commonly-owned, co-pending applications, 18/714,449 and 17/767,917, recite generation of depth information and generation of reliability of that data. However, neither claim set recites depth reliability generation based on intermediate data input from a first estimation model that generates the depth information, required by the instant claims. Therefore, the instant claims and the claims of the ‘499 and ‘917 applications are patentably distinct.
V. Claim Rejections - 35 U.S.C. § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 10 and 14 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claims does/do not fall within at least one of the four categories of patent eligible subject matter because they are directed to an ineligible computer program per se. Pursuant to MPEP 2106.03(I), “[n]on-limiting examples of claims that are not directed to any of the statutory categories include:
Products that do not have a physical or tangible form, such as information (often referred to as “data per se”) or a computer program per se (often referred to as “software per se”) when claimed as a product without any structural recitations….”
Here, both claims 10 and 14 are expressly recited as a program. Therefore, they are ineligible under 35 U.S.C. 101 as explained above. To overcome this rejection, the examiner suggests framing each of claims 10 and 14 as a non-transitory computer-readable media comprising the program.
VI. Claim Rejections - 35 U.S.C. § 112
The following is a quotation of 35 U.S.C. 112:
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 12 is rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention. Specifically, claim 12 lacks antecedent basis for the limitation, “the plurality of images for learning.” To overcome this rejection, the examiner suggests amending “the” to “a.”
VII. Claim Rejections - 35 U.S.C. § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1,2, and 8-14 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Kim et al. (US # 11,727,591 B2).
As to claim 1, Kim et al. teaches a control device (Fig. 9, “Device”) comprising:
a first generation unit (Fig. 9, unit of “Depth probability generator” that generates the “Mean of depth probability distribution”) that generates Depth information (col. 8, lines 49-53, “…first statistical value…,” noting in col. 7, lines 64 and 65 that “a first type of statistical values may include a mean of the depth values of the pixels…”) indicating a distance to each position of a subject appearing in a captured image (col. 7, lines 32-34) on a basis of output of a first estimation model (Fig. 2, model of first network “202” and preceding “Feature extractor,” “Feature analyzer,” and “Depth generator” layers) when the captured image is input (Fig. 2, “Input image”); and
a second generation unit (Fig. 9, unit of “Depth probability generator” that generates the “Variance of depth probability distribution”) that generates reliability information (col. 8, lines 53-55) indicating reliability of the Depth information (col. 8, lines 3-6, “the second statistical value…may indicate information about a confidence level of the estimation of the depth of each pixel.”) on a basis of output of a second estimation model (Fig. 2, model of second network “203”) when intermediate data generated in the first estimation model is input at the time of estimation of the Depth information (Fig. 2, feature map “201” input to second network; {The claimed intermediate data is the feature map, which is generated as an intermediate step before output of the depth mean.}).
As to claim 2, Kim et al. teaches the control device according to claim 1, wherein
the first generation unit generates, as the Depth information, a Depth image having a distance to each position of the subject as a value of each pixel (col. 7, lines 28-31).
As to claim 8, Kim et al. teaches the control device according to claim 1, wherein
the first estimation model is a model generated by learning using an image for learning and correct answer Depth information representing a correct answer distance to each position of a subject appearing in the image for learning (col. 9, line 58 – col. 10, line 1), and the second estimation model is a model generated by learning using reliability of a correct answer of an estimation result of the first estimation model represented by a comparison result between the correct answer Depth information and the Depth information and the image for learning used as input of the first estimation model (col. 9, lines 16-21; {Although Kim et al. is not as detailed regarding training of the second network “203,” the reference is clear that it is a trained network with appropriate meaning output. Inherently, second network “203” must map correct variances to associated pixels of an input image. That is, the variance (reliability) is the comparison result between the actual depth (claimed correct answer Depth information) and the measured depth (claimed Depth information), which would be appropriately output after training when a feature map associated with a specific image is input to second network “203.”}).
Claim 9 is a method claim reciting steps substantially similar to the generation unit functions of claim 1. Therefore, it is rejected as detailed above.
As Kim et al. discloses software instructions executable by a processor to accomplish the reference’s neural network processing (col. 18, lines 16-22), the examiner submits that this disclosure along with the cited passages in the rejection of claim 1 above satisfy the limitations of claim 10.
As to claim 11, Kim et al. teaches an information processing device (Fig. 9, “Device”) comprising:
a first learning unit (Fig. 9, “DNN” unit/layer of “Depth probability generator” that generates the “Mean of depth probability distribution”) that performs learning using an image for learning and correct answer Depth information indicating a correct answer distance to each position of a subject appearing in the image for learning (col. 9, line 58 – col. 10, line 1), and generates a first estimation model (Fig. 2, model of first network “202” and preceding “Feature extractor,” “Feature analyzer,” and “Depth generator” layers) having a captured image as input (Fig. 2, “Input image”) and having, as output, Depth information (col. 8, lines 49-53, “…first statistical value…,” noting in col. 7, lines 64 and 65 that “a first type of statistical values may include a mean of the depth values of the pixels…”) indicating a distance to each position of the subject appearing in the captured image (col. 7, lines 32-34); and
a second learning unit (Fig. 9, “DNN” unit/layer of “Depth probability generator” that generates the “Variance of depth probability distribution”) that performs learning using reliability of a correct answer of an estimation result of the first estimation model indicated by a comparison result between the correct answer Depth information and the Depth information and the image for learning used as input of the first estimation model (col. 9, lines 16-21; {Although Kim et al. is not as detailed regarding training of the second network “203,” the reference is clear that it is a trained network with appropriate meaning output. Inherently, second network “203” must map correct variances with associated pixels of an input image. That is, the variance (reliability) is the comparison result between the actual depth (claimed correct answer Depth information) and the measured depth (claimed Depth information), which would be appropriately output after training when a feature map associated with a specific input image is input to second network “203.”}), and generates a second estimation model (Fig. 2, model of second network “203”) having, as input, intermediate data generated in the first estimation model at the time of estimation of the Depth information (Fig. 2, feature map “201” input to second network; {The claimed intermediate data is the feature map, which is generated as an intermediate step before output of the depth mean.}) and having the reliability as output (col. 8, lines 3-6, “the second statistical value…may indicate information about a confidence level of the estimation of the depth of each pixel.”).
As to claim 12, Kim et al. teaches the information processing device according to claim 11, further comprising
a learning data generation unit (col. 9, lines 58-63, “…video database…”) that repeats generation of a pair of the reliability of the correct answer calculated on a basis of a comparison result between the correct answer Depth information and the Depth information and the image for learning as input of the first estimation model as learning data of the second estimation model on a basis ofa plurality of images for learning and the correct answer Depth information (col. 9, line 63 – col. 10, line 5; {Also, note col. 8, lines 28-37, which states that the neural networks are deep neural networks (DNNs). DNNs inherently require repeated input of training data/images for an optimization.}).
Claim 13 is a method claim reciting steps substantially similar to the learning unit functions of claim 11. Therefore, it is rejected as detailed above.
As Kim et al. discloses software instructions executable by a processor to accomplish the reference’s neural network processing (col. 18, lines 16-22), the examiner submits that this disclosure along with the cited passages in the rejection of claim 11 above satisfies the limitations of claim 14.
VIII. Claim Rejections - 35 U.S.C. § 103
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
A. Claims 1,2, and 8-14 are rejected under 35 U.S.C. 103 as being unpatentable over Katz et al. (US 2022/0377209 A1) in view of Kim et al. (US # 11,727,591 B2)
As to claim 1, Katz et al. teaches a control device (Fig. 1, AR processing system “108”) comprising:
a first generation unit (Fig. 2, stereo vision depth value determination component “) that generates Depth information indicating a distance to each position of a subject appearing in a captured image on a basis of output of a first estimation model ([0042], lines 8-15; {The claimed first estimation model is the triangulation calculation.}) when the captured image is input ([0042], lines 1-8); and
a second generation unit (Fig. 2, confidence score determination component “210”) that generates reliability information indicating reliability of the Depth information ([0045], lines 1-5) on a basis of output of a second estimation model ([0045], lines 5-9; {The claimed second estimation model is the determination of the location of matching features.})
The claim differs from Katz et al. in that it requires that intermediate data be generated and input to the second estimation model at the time of estimation of the Depth information. However, in the same field of endeavor as the instant application, Kim et al. discloses a stereo-vision (Fig. 3) image processor (Fig. 9, “Micro processor”) that trains and deploys a system of neural networks (Figs. 2 and 3) for estimating the depth and depth reliability of each pixel of an input stereo image (col. 7, line 64 – col. 8, line 6). The input stereo image (Fig. 3, “Input left image”) is applied to a plurality of layers of a neural network (Fig. 2, “Feature extractor,” “Feature analyzer,” and “Depth generator” layers) that outputs an intermediate feature map (Fig. 2, feature map “201”). The feature map is input to a first network (Fig. 2, first network “202”) that outputs a depth estimation of each image pixel (col. 8, lines 49-53) and is input to a second network (Fig. 2, second network “203”) that outputs a confidence in the depth estimation (col. 8, lines 53-55).
In light of the teaching of Kim et al., the examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to replace Katz’s triangulation and matching feature location methods with Katz’s system of neural networks to estimate the stereo depth of objects in an image captured by the left or right optical sensor and to estimate confidence in the stereo depth. As Kim et al. notes col. 1, lines 39-42, the neural network-based solution to depth and depth confidence determination improves upon a system that requires analysis of two images. Furthermore, Kim’s system employs the numerous benefits of deep learning networks, like scalability and high detection accuracy of complex features in a scene.
As to claim 2, Katz et al., as modified by Kim et al., teaches the control device according to claim 1, wherein
the first generation unit generates, as the Depth information, a Depth image having a distance to each position of the subject as a value of each pixel (see Katz et al., [0026]; see Kim et al., col. 7, lines 28-31).
As to claim 8, Katz et al., as modified by Kim et al., teaches the control device according to claim 1, wherein
the first estimation model is a model generated by learning using an image for learning and correct answer Depth information representing a correct answer distance to each position of a subject appearing in the image for learning (see Kim et al., col. 9, line 58 – col. 10, line 1), and the second estimation model is a model generated by learning using reliability of a correct answer of an estimation result of the first estimation model represented by a comparison result between the correct answer Depth information and the Depth information and the image for learning used as input of the first estimation model (see Kim et al., col. 9, lines 16-21; {Although Kim et al. does not disclose as much of the detail regarding training of the second network “203,” the reference is clear that it is a trained network with appropriate meaning output. That is, inherently, network “203” must receive correct variances associated with a specific pixel of the input image. That is the variance (reliability) is the comparison result between the actual depth (claimed correct answer Depth information) and the measured depth (claimed {Although Kim et al. is not as detailed regarding training of the second network “203,” the reference is clear that it is a trained network with appropriate meaning output. Inherently, second network “203” must map correct variances with associated pixels of an input image. That is, the variance (reliability) is the comparison result between the actual depth (claimed correct answer Depth information) and the measured depth (claimed Depth information), which would be appropriately output after training when a feature map associated with a specific input image is input to second network “203.”}).
Claim 9 is a method claim reciting steps substantially similar to the generation unit functions of claim 1. Therefore, it is rejected as detailed above.
As Katz et al. discloses software instructions executable by a processor to accomplish the reference’s neural network processing ([0063], lines 1-6), the examiner submits that this disclosure along with the combination of Katz et al. and Kim et al. detailed in the rejection of claim 1 above satisfies the limitations of claim 10.
The combination of Katz et al. and Kim et al. detailed above forms the basis for the rejections of claims 11-14 that follow.
As to claim 11, Katz et al., as modified by Kim et al., teaches an information processing device (see Katz et al., Fig. 1, AR processing system “108”) comprising:
a first learning unit (see Kim et al., Fig. 9, “DNN” unit/layer of “Depth probability generator” that generates the “Mean of depth probability distribution”) that performs learning using an image for learning and correct answer Depth information indicating a correct answer distance to each position of a subject appearing in the image for learning (see Katz et al., [0029]; Kim et al., col. 9, line 58 – col. 10, line 1), and generates a first estimation model (see Kim et al., Fig. 2, model of first network “202” and preceding “Feature extractor,” “Feature analyzer,” and “Depth generator” layers) having a captured image as input (see Katz et al., [0029]; see Kim et al., Fig. 2, “Input image”) and having, as output, Depth information (see Katz et al., [0042], lines 1-3; see Kim et al., col. 8, lines 49-53, “…first statistical value…,” noting in col. 7, lines 64 and 65 that “a first type of statistical values may include a mean of the depth values of the pixels…”) indicating a distance to each position of the subject appearing in the captured image (see Katz et al., [0042], lines 1-3; see Kim et al., col. 7, lines 32-34); and
a second learning unit (see Kim et al., Fig. 9, “DNN” unit/layer of “Depth probability generator” that generates the “Variance of depth probability distribution”) that performs learning using reliability of a correct answer of an estimation result of the first estimation model indicated by a comparison result between the correct answer Depth information and the Depth information and the image for learning used as input of the first estimation model (see Kim et al., col. 9, lines 16-21; {Although Kim et al. is not as detailed regarding training of the second network “203,” the reference is clear that it is a trained network with appropriate meaning output. Inherently, second network “203” must map correct variances with associated pixels of an input image. That is, the variance (reliability) is the comparison result between the actual depth (claimed correct answer Depth information) and the measured depth (claimed Depth information), which would be appropriately output after training when a feature map associated with a specific input image is input to second network “203.”}), and generates a second estimation model (see Kim et al., Fig. 2, model of second network “203”) having, as input, intermediate data generated in the first estimation model at the time of estimation of the Depth information (see Kim et al., Fig. 2, feature map “201” input to second network; {The claimed intermediate data is the feature map, which is generated as an intermediate step before output of the depth mean.}) and having the reliability as output (see Katz et al., [0045], lines 1-5; see Kim et al., col. 8, lines 3-6, “the second statistical value…may indicate information about a confidence level of the estimation of the depth of each pixel.”).
As to claim 12, Katz et al., as modified by Kim et al., teaches the information processing device according to claim 11, further comprising
a learning data generation unit (see Kim et al., col. 9, lines 58-63, “…video database…”) that repeats generation of a pair of the reliability of the correct answer calculated on a basis of a comparison result between the correct answer Depth information and the Depth information and the image for learning as input of the first estimation model as learning data of the second estimation model on a basis ofa plurality of images for learning and the correct answer Depth information (see Kim et al., col. 9, line 63 – col. 10, line 5; {Also, note col. 8, lines 28-37, which states that the neural networks are deep neural networks (DNNs). DNNs inherently require repeated input of training data/images for an optimization.}).
Claim 13 is a method claim reciting steps substantially similar to the learning unit functions of claim 11. Therefore, it is rejected as detailed above.
As Katz et al. discloses software instructions executable by a processor to accomplish the reference’s neural network processing ([0063], lines 1-6), the examiner submits that this disclosure along with the combination of Katz et al. and Kim et al. detailed in the rejection of claim 11 above satisfies the limitations of claim 14.
B. Claims 3-6 are rejected under 35 U.S.C. 103 as being unpatentable over Katz et al. (US 2022/0377209 A1) in view of Kim et al. (US # 11,727,591 B2) and further in view of Gamadia et al. (US 2022/0116544 A1)
As to claim 3, Katz et al., as modified by Kim et al., teaches the control device according to claim 1. Although Katz et al. suggests that the stereo depth information is used for auto-focusing when output in step “312” of Fig. 3 after it demonstrates sufficient confidence in step “310” of Fig. 3, the reference fails to expressly disclose that the optical sensors are connected to an imaging control unit that controls imaging by using the Depth information depending on the reliability. However, in the same field of endeavor as the instant application, Gamadia et al. discloses a camera (Fig. 2, camera “200”) comprising circuitry that controls the focusing position of a lens ([0046], lines 1-7) based on confidence values of different depth estimation techniques (Fig. 6).
In light of the teaching of Gamadia et al., the examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to include circuitry that controls focusing of Katz’s left and/or right sensors using the stereo vision depth when it meets a threshold confidence and to control focusing using phase-difference information when the stereo vision depth does not meet the threshold confidence. One of ordinary skill in the art would recognize that depth information can lead to optimal focusing, and in the case of Katz et al., a depth estimation method for auto-focusing can be appropriately chosen based on the distance of the optical sensors to a primary object to be focused (see Katz et al., [0053], lines 1-11).
As to claim 4, Katz et al., as modified by Kim et al. and Gamadia et al., teaches the control device according to claim 3, wherein
the imaging control unit controls imaging by using the Depth information having the reliability higher than a threshold (see Katz et al., Fig. 3, steps “310” and “312”; see Katz et al., [0060]; see Gamadia et al., Fig. 6).
As to claim 5, Katz et al., as modified by Kim et al. and Gamadia et al., teaches the control device according to claim 4, further comprising
a phase difference detection unit (see Katz et al., Fig. 2, PDAF depth value determination component “204”) that detects a phase difference (see Katz et al., [0039]), wherein
the imaging control unit controls the focus (see Gamadia et al., Fig. 6) on a basis of the Depth information in a case where the reliability is higher than a threshold (see Katz et al., Fig. 3, step “310” - YES), and controls the focus (see Gamadia et al., Fig. 6) on a basis of the phase difference detected by the phase difference detection unit in a case where the reliability is lower than the threshold (see Katz et al., Fig. 3, step “310” - NO).
As to claim 6, Katz et al., as modified by Kim et al. and Gamadia et al., teaches the control device according to claim 5, wherein
the imaging control unit controls imaging by using the Depth information having the reliability higher than a threshold, the Depth information being generated before the Depth information having the reliability lower than the threshold (see Katz et al., [0025], “…video…”; {The examiner reads the claimed Depth information having the reliability higher than a threshold as depth information from a frame of video captured before a frame is that exhibits low stereo vision depth value confidence (i.e., claimed Depth information having the reliability lower than the threshold.}).
C. Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (US # 11,727,591 B2) in view of the WIPO publication of Nayar et al. (WIPO publication number: WO 2013/162747 A1)
As to claim 7, Kim et al. teaches the control device according to claim 1. The claim differs from Kim et al. in that it requires that the first generation unit generates the Depth information on a basis of each of the captured images captured in a sequential manner and that the second generation unit generates the reliability information indicating the reliability of each piece of the Depth information generated on a basis of each of the captured images.
However, in the same field of endeavor as the instant application, Nayar et al. discloses a focusing system for a camera (Fig. 10; [0083]) including a neural network that generates a volume depth map ([0115]) indicating depths of an object determined from a plurality of images captured at different times and with different depths of field ([0050]). In light of the teaching of Nayar et al., the examiner submits that it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant application to input a plurality of images captured at different depths of field to Kim’s neural network system and produce volume mean depth and volume variance information because this would lead to higher quality depth estimation for objects at varying depths within the scene and can lead to creation of an image in which more of the scene is optimally focused (see Nayar et al., [0006]).
IX. Additional Pertinent Prior Art
Xu et al. (US 2024/0121511 A1) discloses another example of a camera that chooses one of depth-based focus or phase-difference-based focus based on the confidence value of collected depth information. Kang (US 2022/0130073 A1) and Mathy et al. (US 2019/0113606 A1) discloses additional examples of a system that generates depth data and reliability information of the depth data.
X. Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J DANIELS whose telephone number is (571)272-7362. The examiner can normally be reached M-F 9:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sinh Tran can be reached at 571-272-7564. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J DANIELS/Primary Examiner, Art Unit 2637
3/28/2026