DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. 2023151510, filed on September 19, 2023.
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 3, 5, 6, 10 is/are rejected under 35 U.S.C. 102a(1) as being anticipated by Sunkavalli (US20190340810A1).
Regarding claim 1, Sunkavalli teaches an optical processing method comprising:
inputting an image including direction information of light from an object as hue information for each pixel into an artificial neural network; “The act 1206 involves providing the set of input digital images to an object relighting neural network trained based on training digital images portraying objects illuminated by training lighting directions” (Sunkavalli, 0127) and “the image relighting system provides each input digital image to the object relighting neural network as a set of three color channels, each channel representing a color value of pixels of the input digital image” (Sunkavalli, 0066). “As shown in FIG. 6, the first input digital image 604 includes the color channels 606 and the light direction channels 608” (0090) and “In particular, for each input digital image, the image relighting system generates a set of color channels representing color values reflecting pixels of the respective input digital image” (Sunkavalli, 0090).
and acquiring depth information of the object for the image based on a result output from the artificial neural network. “The image relighting system modifies the digital object by applying a variable height field” (Sunkavalli, 0076) and “one or more embodiments described herein include an image relighting system that utilizes a neural network to generate digital images portraying objects illuminated from target lighting directions based on a sparse number of input digital images (e.g., five or fewer) portraying the objects illuminated under initial lighting directions. (0021).
“A second set of color channels comprising color values reflecting pixels of the second training digital image; generating a second set of direction channels comprising coordinates corresponding to the second lighting direction; and providing the first set of color channels, the second set of color channels, the first set of direction channels, and the second set of direction channels to the object relighting neural network” (Sunkavalli, 0121).
“The image relighting system generates and provides a set of target direction channels that include coordinates (interpreted as depth information as supported by the applicant’s specification) corresponding to the target lighting direction” (Sunkavalli, 0027).
Regarding claim 3, the optical processing method according to claim 1, Sunkavalli as in Claim 1, teaches wherein: the direction information of the light is an angle of a light beam with respect to a specific axis as a reference. “In particular, a lighting direction includes an orientation of a light source relative to a reference” (Sunkavalli, 0035).
Regarding claim 5, claim 5 is similar in scope to claim 1 except for additional limitations that Sunkavalli discloses: An optical processing apparatus comprising: a processing portion which is configured to input an image including direction information of light from an object as hue information for each pixel into an artificial neural network, and which is configured to acquire depth information of the object for the image based on a result output from the artificial neural network. “The image relighting system configures the object relighting neural network to inherently process the input digital images as if they portrayed the fixed lighting directions” (Sunkavalli, 0067).
Regarding claim 6, Sunkavalli discloses: An optical processing system comprising: the optical processing apparatus according to claim 5;
and an imaging portion which is configured to acquire the image
wherein the imaging portion comprises:
an image formation optical element, ““A lighting direction includes an orientation of a light source relative to a reference (e.g., relative to an object and/or a camera) (interpreted as having an image sensor, a lens, and a color filter). For example, a lighting direction includes a direction between an object and a light source that indicates the direction of light travelled between the light source and the object. Lighting direction can be described in terms of a variety of values (e.g., coordinates of a two-dimensional plane or unit hemisphere or one or more angles)” (Sunkavalli, 0036).
an image sensor,
and a wavelength selection portion that causes the image sensor to acquire direction information of light from the object as hue information. “The image sensor 44 is a line sensor. The longitudinal direction of the image sensor (line sensor) 44 is a direction along the y-axis” (Sunkavalli, 0035) and “the light reflected by the object point may pass through the first wavelength selection region 52 … the image sensor 44 acquires different colors (light beam directions)” (Sunkavalli, 0055).
Regarding claim 10, claim 10 is similar in scope to claim 1 except for additional limitations that Sunkavalli discloses: A non-transitory storage medium storing an optical processing program which causes a computer to execute:
“One or more embodiments described herein provide benefits and/or solve one or more of the foregoing or other problems in the art with systems, methods, and non-transitory computer readable storage media that train and utilize a deep-learning neural network model to generate digital images portraying objects illuminated under novel lighting based on a small sample of input digital images portraying the objects under calibrated lighting” (Sunkavalli, 0005).
- inputting an image including direction information of light from an object as hue information for each pixel into an artificial neural network; and
- acquiring depth information of the object for the image based on a result output from the artificial neural network.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2, 4, 7, 8 are rejected under 35 U.S.C. 103 as being unpatentable over Sunkavalli (US20190340810A1) in view of Ono (JP2022049881A).
Regarding claim 2, Sunkavalli teaches the optical processing method according to claim 1. Sunkavalli does not teach performing iterative calculation so that a three-dimensional shape of the object output when the direction information of the light is input using the artificial neural network satisfies an equation derived based on geometric optics.
However, Ono teaches further comprising:
performing iterative calculation so that a three-dimensional shape of the object output when the direction information of the light is input using the artificial neural network satisfies an equation derived based on geometric optics. “The processing unit 4a calculates the light ray direction of the light ray from the hue pixel value by the above method using the relationship between the previously recorded hue pixel value and the light ray direction (S13)” (Ono, 0042).
Ono and Sunkavalli are combinable because they are in the same field of endeavor of image analysis. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine Ono’s iterative calculation with Sunkavalli’s channel representation technique (organizing per-pixel color and direction data into structured channel sets) in order to predictably improve the neural network’s ability to jointly process hue and direction data in a structured manner, yielding enhanced 3D reconstruction accuracy.
Regarding claim 4, Sunkavalli teaches the optical processing method according to claim 1. Sunkavalli further teaches an image sensor with an image formation optical element and a wavelength selection portion including a plurality of regions: “A lighting direction includes an orientation of a light source relative to a reference (e.g., relative to an object and/or a camera) (interpreted as having an image sensor, a lens, and a color filter). For example, a lighting direction includes a direction between an object and a light source that indicates the direction of light travelled between the light source and the object. Lighting direction can be described in terms of a variety of values (e.g., coordinates of a two-dimensional plane or unit hemisphere or one or more angles)” (Sunkavalli, 0036).
Sunkavalli does not teach a plurality of regions including at least two different transmission wavelength spectra or at least two different reflection wavelength spectra.
However, Ono teaches:
a plurality of regions including at least two different transmission wavelength spectra or at least two different reflection wavelength spectra. “The wavelength selection region 12 is a transmission type wavelength filter, and the transmission wavelength spectra of the first wavelength selection region 12a and the second wavelength selection region 12b are different from each other” (Ono, 0020).
Ono and Sunkavalli are combinable because they are in the same field of endeavor of image analysis. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine Ono’s transmission wavelength spectra with Sunkavalli’s channel representation technique (organizing per-pixel color and direction data into structured channel sets) in order to predictably improve the neural network’s ability to jointly process hue and direction data in a structured manner, yielding enhanced 3D reconstruction accuracy.
Regarding claim 7, Sunkavalli teaches the optical processing system according to claim 6. Sunkavalli does not teach the wavelength selection portion is disposed at a focal plane of the image formation optical element.
However, Ono teaches:
wherein the wavelength selection portion is disposed at a focal plane of the image formation optical element. “The light ray emitting surface 6 of the illumination unit 2 is arranged on the focal plane 9b of the imaging optical element 9” (Ono, 0104).
Ono and Sunkavalli are combinable because they are in the same field of endeavor of image analysis. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine Ono’s imaging optical element with Sunkavalli’s channel representation technique (organizing per-pixel color and direction data into structured channel sets) in order to predictably improve the neural network’s ability to jointly process hue and direction data in a structured manner, yielding enhanced 3D reconstruction accuracy.
Regarding claim 8, Sunkavalli teaches the optical processing system according to claim 6. Sunkavalli does not teach the wavelength selection portion includes a plurality of regions including at least two different transmission wavelength spectra or at least two different reflection wavelength spectra, and the plurality of regions are arranged concentrically.
However, Ono teaches wherein:
the wavelength selection portion includes a plurality of regions including at least two different transmission wavelength spectra or at least two different reflection wavelength spectra, and the plurality of regions are arranged concentrically. “The wavelength selection unit 10 includes at least two, that is, a plurality of wavelength selection regions 12 (regions) … Each wavelength selection region 12 is a transmission type or reflection type wavelength filter” (Ono, 0020).
“It is assumed that the wavelength selection region 12 is axisymmetric and concentric” (Ono, 0094).
Ono and Sunkavalli are combinable because they are in the same field of endeavor of image analysis. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine Ono’s wavelength selection region with Sunkavalli’s channel representation technique (organizing per-pixel color and direction data into structured channel sets) in order to predictably improve the neural network’s ability to jointly process hue and direction data in a structured manner, yielding enhanced 3D reconstruction accuracy.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sunkavalli in view of Zaifuritto (JP2007502443A).
Regarding claim 9, Sunkavalli teaches the optical processing system according to claim 6. Sunkavalli does not teach the wavelength selection portion includes a plurality of regions having at least two different transmission wavelength spectra or at least two different reflection wavelength spectra, and the plurality of regions are arranged in stripes.
However, Zaifuritto teaches wherein:
the wavelength selection portion includes a plurality of regions having at least two different transmission wavelength spectra or at least two different reflection wavelength spectra, and the plurality of regions are arranged in stripes. “Has a plurality of reflection regions and a plurality of transmission regions formed in a stripe shape” (Zaifuritto, 0019).
Sunkavalli and Zaifuritto are combinable because they are in the same field of endeavor of image analysis. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to Sunkavalli’s channel representation technique (organizing per-pixel color and direction data into structured channel sets) with the stripe region of Zaifuritto in order to unite a plurality of light beams having mutually different wavelengths.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIANA SZE whose telephone number is (571)272-9916. The examiner can normally be reached Monday-Thursday 6am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.S./Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights