AfNotice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Some of Applicant’s arguments with respect to claims 1, 8, & 15 have been considered but are moot in view of new grounds of rejection. Since a new ground of rejection is being made against unamended claims, this action is not made final. Applicant’s arguments that Xue fails to teach or suggest “determin[ing]…first intensities of the scene based on camera rays for an image of the scene” have been considered, but are not persuasive. In the broadest reasonable interpretation, the “displayed image 305” Xue refers to in [0056], ln. 3-5 is “based on camera rays for an image of the scene,” and the “first bilateral grid 325 for the non-target region” Xue refers to can be considered “first intensities.”
The remaining Applicant arguments have been considered but are moot in view of new grounds of rejection.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 7, 8, 11, 14, 15, 18, & 20 are rejected under 35 U.S.C. 103 as being unpatentable over Park & Kwak (KR 20200044182 A, hereinafter, "Park") in view of Xue et al (US 20230308769 A1, hereinafter, "Xue"), Oh et al (US 20240212115 A1, hereinafter, "Oh"), and Azarian et al (CN 114467098 A, hereinafter, "Azarian").
Regarding Claim 1, Xue teaches an apparatus for light estimation of a scene, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: determine, using a first neural network, first intensities of the scene based on camera rays for an image of the scene (Xue, Fig. 3, [0056], ln. 3-5, "First HDRnet 310 is applied to displayed image 305 to generate a first bilateral grid 325 for the non-target region that includes at least a portion of (e.g., an entirety of) the image outside the target region." In the broadest reasonable interpretation, the "displayed image 305" Xue refers to is "based on camera rays," and the "first bilateral grid 325 for the non-target region" can be considered "first intensities."). Xue does not teach determine, using one or more second neural networks each comprising a camera response function, second intensities of the scene based on the first intensities of the scene, wherein the one or more second neural networks are trained to learn the camera response function based on a regularization loss. Park teaches determine, using one or more second neural networks each comprising a camera response function (Park, pg. 14, para. 1, ln. 1-3, "…the third artificial intelligence model may be a neural network model that learns shooting setting information [eg, exposure, white balance, and focus] of the camera 101 and RGB images."), wherein the one or more second neural networks are trained to learn the camera response function (Park, pg. 14, para. 1, ln. 1-3, "…the third artificial intelligence model may be a neural network model that learns shooting setting information [eg, exposure, white balance, and focus] of the camera 101 and RGB images."). Park does not teach second intensities of the scene based on the first intensities of the scene based on a regularization loss. Oh teaches second intensities of the scene based on the first intensities of the scene (Oh, Fig. 5, [0084], ln. 3-5, "…the tone mapping module may be a module for generating a tone-mapped LDR image by adjusting the white balance of the HDR image and then applying the camera response function."). Oh does not teach based on a regularization loss. However, Azarian teaches based on a regularization loss (Azarian, Fig. 5, pg. 12, para. 1, ln. 2-4, "As shown in FIG. 5, the frame 502, process 500 based on the classification loss and regularization loss function to determine a trimming threshold for trimming a first set of pre-training weights in a plurality of pre-training weights."). It would have been obvious to a person having ordinary skill in the art at the time of the invention to combine the teachings of Park, Xue, Oh, and Azarian because it is well known in the art to use one or more second (more than one second being a “third”) neural network comprising a camera response function (e.g., exposure, white balance, and focus) which are trained to learned said camera response function, second intensities of a scene based on first intensities of a scene, and training neural networks based on regularization loss function.
Regarding Claim 4, Xue, Park, Oh, and Azarian teach the limitations of dependent Claim 1 as noted above. Oh teaches the first intensities have a higher dynamic range than the second intensities (Oh, Fig. 5, [0084], ln. 3-5, "…the tone mapping module may be a module for generating a tone-mapped LDR image by adjusting the white balance of the HDR image and then applying the camera response function." Secondary LDR [Low Dynamic Range] intensities are generated from a first HDR [High Dynamic Range] intensity.).
Regarding Claim 7, Xue, Park, Oh, and Azarian teach the limitations of dependent Claim 1 as noted above. Oh teaches the first neural network is trained to learn a radiance field model based on a plurality of multi-view images with the second intensities (Oh, [0062]-[0063], all lines, "In step 220, the image processing device may construct 3D HDR radiance fields from the LDR images captured from the various viewpoints. [0063] The 3D HDR radiance fields may include information required to calculate how an object will appear on a screen when the object is viewed from a particular location. The 3D HDR radiance fields may be a concept introduced in a known paper [NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis]."). It would have been obvious to a person having ordinary skill in the art at the time of the invention to combine the teachings of Oh with those of Xue, Park, Oh, and Azarian because it is well known in the art to use neural networks to learn radiance fields based on a plurality of multi-view images with second intensities.
Regarding Claim 8, Xue teaches A method of light estimation of a scene, the method comprising: determining, using a first neural network, first intensities of the scene based on camera rays for an image of the scene (Xue, Fig. 3, [0056], ln. 3-5, "First HDRnet 310 is applied to displayed image 305 to generate a first bilateral grid 325 for the non-target region that includes at least a portion of (e.g., an entirety of) the image outside the target region." In the broadest reasonable interpretation, the "displayed image 305" Xue refers to is "based on camera rays," and the "first bilateral grid 325 for the non-target region" can be considered "first intensities."). Xue does not teach determining, using one or more second neural networks each comprising a camera response function, second intensities of the scene based on the first intensities of the scene, wherein the one or more second neural networks are trained to learn the camera response function based on a regularization loss. Park teaches determining, using one or more second neural networks each comprising a camera response function (Park, pg. 14, para. 1, ln. 1-3, "…the third artificial intelligence model may be a neural network model that learns shooting setting information [eg, exposure, white balance, and focus] of the camera 101 and RGB images.") wherein the one or more second neural networks are trained to learn the camera response function (Park, pg. 14, para. 1, ln. 1-3, "…the third artificial intelligence model may be a neural network model that learns shooting setting information [eg, exposure, white balance, and focus] of the camera 101 and RGB images."). Park does not teach second intensities of the scene based on the first intensities of the scene based on a regularization loss. Oh teaches second intensities of the scene based on the first intensities of the scene (Oh, Fig. 5, [0084], ln. 3-5, "…the tone mapping module may be a module for generating a tone-mapped LDR image by adjusting the white balance of the HDR image and then applying the camera response function."). Oh does not teach based on a regularization loss. However, Azarian teaches based on a regularization loss (Azarian, Fig. 5, pg. 12, para. 1, ln. 2-4, "As shown in FIG. 5, the frame 502, process 500 based on the classification loss and regularization loss function to determine a trimming threshold for trimming a first set of pre-training weights in a plurality of pre-training weights.").
Regarding Claim 11, Xue, Park, Oh, and Azarian teach the limitations of dependent Claim 8 as noted above. Oh teaches the first intensities have a higher dynamic range than the second intensities (Oh, Fig. 5, [0084], ln. 3-5, "…the tone mapping module may be a module for generating a tone-mapped LDR image by adjusting the white balance of the HDR image and then applying the camera response function." Secondary LDR [Low Dynamic Range] intensities are generated from a first HDR [High Dynamic Range] intensity.).
Regarding Claim 14, Xue, Park, Oh, and Azarian teach the limitations of dependent Claim 8 as noted above. Oh teaches the first neural network is trained to learn a radiance field model based on a plurality of multi-view images with the second intensities (Oh, [0062]-[0063], all lines, "In step 220, the image processing device may construct 3D HDR radiance fields from the LDR images captured from the various viewpoints. [0063] The 3D HDR radiance fields may include information required to calculate how an object will appear on a screen when the object is viewed from a particular location. The 3D HDR radiance fields may be a concept introduced in a known paper [NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis].").
Regarding Claim 15, Xue teaches a non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to: determine, using a first neural network, first intensities of a scene based on camera rays for an image of the scene (Xue, Fig. 3, [0056], ln. 3-5, "First HDRnet 310 is applied to displayed image 305 to generate a first bilateral grid 325 for the non-target region that includes at least a portion of (e.g., an entirety of) the image outside the target region." In the broadest reasonable interpretation, the "displayed image 305" Xue refers to is "based on camera rays," and the "first bilateral grid 325 for the non-target region" can be considered "first intensities."). Xue does not teach determining, using one or more second neural networks each comprising a camera response function, second intensities of the scene based on the first intensities of the scene, wherein the one or more second neural networks are trained to learn the camera response function based on a regularization loss. Park teaches determining, using one or more second neural networks each comprising a camera response function (Park, pg. 14, para. 1, ln. 1-3, "…the third artificial intelligence model may be a neural network model that learns shooting setting information [eg, exposure, white balance, and focus] of the camera 101 and RGB images.") wherein the one or more second neural networks are trained to learn the camera response function (Park, pg. 14, para. 1, ln. 1-3, "…the third artificial intelligence model may be a neural network model that learns shooting setting information [eg, exposure, white balance, and focus] of the camera 101 and RGB images."). Park does not teach second intensities of the scene based on the first intensities of the scene based on a regularization loss. Oh teaches second intensities of the scene based on the first intensities of the scene (Oh, Fig. 5, [0084], ln. 3-5, "…the tone mapping module may be a module for generating a tone-mapped LDR image by adjusting the white balance of the HDR image and then applying the camera response function."). Oh does not teach based on a regularization loss. However, Azarian teaches based on a regularization loss (Azarian, Fig. 5, pg. 12, para. 1, ln. 2-4, "As shown in FIG. 5, the frame 502, process 500 based on the classification loss and regularization loss function to determine a trimming threshold for trimming a first set of pre-training weights in a plurality of pre-training weights.").
Regarding Claim 18, Xue, Park, Oh, and Azarian teach the limitations of dependent Claim 15 as noted above. Oh teaches the first intensities have a higher dynamic range than the second intensities (Oh, Fig. 5, [0084], ln. 3-5, "…the tone mapping module may be a module for generating a tone-mapped LDR image by adjusting the white balance of the HDR image and then applying the camera response function." Secondary LDR [Low Dynamic Range] intensities are generated from a first HDR [High Dynamic Range] intensity.).
Regarding Claim 20, Xue, Park, Oh, and Azarian teach the limitations of dependent Claim 15 as noted above. Oh teaches the first neural network is trained to learn a radiance field model based on a plurality of multi-view images with the second intensities (Oh, [0062]-[0063], all lines, "In step 220, the image processing device may construct 3D HDR radiance fields from the LDR images captured from the various viewpoints. [0063] The 3D HDR radiance fields may include information required to calculate how an object will appear on a screen when the object is viewed from a particular location. The 3D HDR radiance fields may be a concept introduced in a known paper [NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis].").
Claims 2, 9, & 16 are rejected under 35 U.S.C. 103 as being unpatentable over Park in view of Xue, Oh, Azarian, and Tao & Kim (US 20160381335 A1, hereinafter, "Tao").
Regarding Claim 2, Xue, Park, Oh, and Azarian teach the limitations of dependent Claim 1 as noted above. Xue teaches based on training the one or more second neural networks using the regularization loss (Xue, Figs. 2 & 3, [0054], ln. 5-7, "For example, a modified version of a trained neural network [e.g., a convolutional neural network or “CNN”], such as Google's HDRnet tone mapping algorithm, may be utilized." Fig. 3 shows second neural network used.). Xue does not teach the camera response function is monotonically increasing. However, Tao teaches the camera response function is monotonically increasing (Tao, Figs. 5A-5C, [0050], ln. 1-3, "FIGS. 5A, 5B, and 5C illustrate example GCI-C and LCI-C tone mapping functions according to various embodiments of the present disclosure. In the various embodiments, the functions ƒ and g are monotonically increasing functions."). It would have been obvious to a person having ordinary skill in the art at the time of the invention to combine the teachings of Xue and Tao with those of Park, Xue, Oh, and Azarian because it is well known in the art to train neural networks based on regularization loss and to monotonically increase a camera function.
Regarding Claim 9, Xue, Park, Oh, and Azarian teach the limitations of dependent Claim 8 as noted above. Xue teaches based on training the one or more second neural networks using the regularization loss (Xue, Figs. 2 & 3, [0054], ln. 5-7, "For example, a modified version of a trained neural network [e.g., a convolutional neural network or “CNN”], such as Google's HDRnet tone mapping algorithm, may be utilized." Fig. 3 shows second neural network used.). Xue does not teach the camera response function is monotonically increasing. However, Tao teaches the camera response function is monotonically increasing (Tao, Figs. 5A-5C, [0050], ln. 1-3, "FIGS. 5A, 5B, and 5C illustrate example GCI-C and LCI-C tone mapping functions according to various embodiments of the present disclosure. In the various embodiments, the functions ƒ and g are monotonically increasing functions.").
Regarding Claim 16, Xue, Park, Oh, and Azarian teach the limitations of dependent Claim 15 as noted above. Xue teaches based on training the one or more second neural networks using the regularization loss (Xue, Figs. 2 & 3, [0054], ln. 5-7, "For example, a modified version of a trained neural network [e.g., a convolutional neural network or “CNN”], such as Google's HDRnet tone mapping algorithm, may be utilized." Fig. 3 shows second neural network used.). Xue does not teach the camera response function is monotonically increasing. However, Tao teaches the camera response function is monotonically increasing (Tao, Figs. 5A-5C, [0050], ln. 1-3, "FIGS. 5A, 5B, and 5C illustrate example GCI-C and LCI-C tone mapping functions according to various embodiments of the present disclosure. In the various embodiments, the functions ƒ and g are monotonically increasing functions.").
Claims 3, 10, & 17 are rejected under 35 U.S.C. 103 as being unpatentable over Park in view of Xue, Oh, Azarian, and Nakata et al (US 20230140768 A1, hereinafter, "Nakata").
Regarding Claim 3, Park, Xue, Oh, and Azarian teach the limitations of dependent Claim 1 as noted above. Nakata teaches a camera ray comprises a ray origin and a ray direction (Nakata, Fig. 2, [0109], ln. 1-2, "The light receiving unit 100 includes, for example, a lens 110, an infrared ray cut filter (IRCF 112), and an imaging element 114." Imaging element 114 is the origin of the camera ray, which extends in the direction of lens 110.). It would have been obvious to a person having ordinary skill in the art at the time of the invention to combine the teachings of Nakata with those of Park, Xue, Oh, and Azarian because it is well known in the art what a ray is.
Regarding Claim 10, Park, Xue, Oh, and Azarian teach the limitations of dependent Claim 8 as noted above. Nakata teaches a camera ray comprises a ray origin and a ray direction (Nakata, Fig. 2, [0109], ln. 1-2, "The light receiving unit 100 includes, for example, a lens 110, an infrared ray cut filter (IRCF 112), and an imaging element 114." Imaging element 114 is the origin of the camera ray, which extends in the direction of lens 110.).
Regarding Claim 17, Park, Xue, Oh, and Azarian teach the limitations of dependent Claim 15 as noted above. Nakata teaches a camera ray comprises a ray origin and a ray direction (Nakata, Fig. 2, [0109], ln. 1-2, "The light receiving unit 100 includes, for example, a lens 110, an infrared ray cut filter (IRCF 112), and an imaging element 114." Imaging element 114 is the origin of the camera ray, which extends in the direction of lens 110.).
Claims 5, 6, 12, 13, & 19 are rejected under 35 U.S.C. 103 as being unpatentable over Park in view of Xue, Oh, Azarian, and Koga et al (US 20210235005 A1, hereinafter, "Koga").
Regarding Claim 5, Park, Xue, Oh, and Azarian teach the limitations of dependent Claim 1 as noted above. Koga teaches the at least one processor is configured to determine the second intensities using the one or more second neural networks further based on an exposure of the image (Koga, Fig. 5, [0065], ln. 3-8, "The first operation procedure example is an example in which a subject to be captured by the monitoring camera 1 is a person and a camera parameter is adjusted [changed] so that a face of the person can be detected by the AI processing. Although exposure time and a gain of the image sensor 12 and a tone curve are exemplified as camera parameters to be adjusted [changed] in FIG. 5, it is needless to say that the camera parameter is not limited thereto."). It would have been obvious to a person having ordinary skill in the art at the time of the invention to combine the teachings of Koga with those of Park, Xue, Oh, and Azarian because it is well known in the art to configure a processor to determine second intensities using one or more second neural networks based on an exposure of the image.
Regarding Claim 6, Park, Xue, Oh, and Azarian teach the limitations of dependent Claim 1 as noted above. Koga teaches the exposure is a combination of an exposure time and a gain of a camera used to capture the image (Koga, Fig. 5, [0065], ln. 3-8, "The first operation procedure example is an example in which a subject to be captured by the monitoring camera 1 is a person and a camera parameter is adjusted [changed] so that a face of the person can be detected by the AI processing. Although exposure time and a gain of the image sensor 12 and a tone curve are exemplified as camera parameters to be adjusted [changed] in FIG. 5, it is needless to say that the camera parameter is not limited thereto."). It would have been obvious to a person having ordinary skill in the art at the time of the invention to combine the teachings of Koga with those of Park, Xue, Oh, and Azarian because it is well known in the art that exposure is a combination of an exposure time and a gain of a camera used to capture the image.
Regarding Claim 12, Park, Xue, Oh, and Azarian teach the limitations of dependent Claim 8 as noted above. Koga teaches the at least one processor is configured to determine the second intensities using the one or more second neural networks further based on an exposure of the image (Koga, Fig. 5, [0065], ln. 3-8, "The first operation procedure example is an example in which a subject to be captured by the monitoring camera 1 is a person and a camera parameter is adjusted [changed] so that a face of the person can be detected by the AI processing. Although exposure time and a gain of the image sensor 12 and a tone curve are exemplified as camera parameters to be adjusted [changed] in FIG. 5, it is needless to say that the camera parameter is not limited thereto.").
Regarding Claim 13, Park, Xue, Oh, and Azarian teach the limitations of dependent Claim 8 as noted above. Koga teaches the exposure is a combination of an exposure time and a gain of a camera used to capture the image (Koga, Fig. 5, [0065], ln. 3-8, "The first operation procedure example is an example in which a subject to be captured by the monitoring camera 1 is a person and a camera parameter is adjusted [changed] so that a face of the person can be detected by the AI processing. Although exposure time and a gain of the image sensor 12 and a tone curve are exemplified as camera parameters to be adjusted [changed] in FIG. 5, it is needless to say that the camera parameter is not limited thereto.").
Regarding Claim 19, Park, Xue, Oh, and Azarian teach the limitations of dependent Claim 15 as noted above. Koga teaches the at least one processor is configured to determine the second intensities using the one or more second neural networks further based on an exposure of the image (Koga, Fig. 5, [0065], ln. 3-8, "The first operation procedure example is an example in which a subject to be captured by the monitoring camera 1 is a person and a camera parameter is adjusted [changed] so that a face of the person can be detected by the AI processing. Although exposure time and a gain of the image sensor 12 and a tone curve are exemplified as camera parameters to be adjusted [changed] in FIG. 5, it is needless to say that the camera parameter is not limited thereto.").
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEVEN DANIEL BARRY whose telephone number is (571)270-0432. The examiner can normally be reached M-Th 0730-1630.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Lin Ye can be reached on 517-272-7372. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STEVEN DANIEL BARRY/Examiner, Art Unit 2638
/LIN YE/Supervisory Patent Examiner, Art Unit 2638