DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. JP2022-132979, filed on 24th August, 2022.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 13 is rejected under 35 U.S.C. § 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because the claim recites a program and the body of the claim recites computer program steps, which are nothing more than just programmed instructions to be performed by a system. Therefore, the steps/elements recited in Claim 13 are non-statutory because a computer program per se, i.e., the descriptions or expressions of the program, are not physical "things" and thus do not fall into either a process, machine, manufacture, and/or composition of matter category and makes it ineligible subject matter under 35 U.S.C. § 101. In contrast, a claimed non-transitory computer-readable medium encoded with a computer program is a computer element which defines structural and functional interrelationships between the computer program and the rest of the computer, which permits the computer program's functionality to be realized, and is thus statutory.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7 and 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Kajimura (US 20220383507 A1), hereinafter referenced as Kajimura, in view of Fujita et al. (US 20220383589 A1), hereinafter referenced as Fujita.
Regarding Claim 1, Kajimura discloses an information processing apparatus (Kajimura, [0054]: teaches an image processing apparatus <read on information processing apparatus> 100) comprising
a processing unit that infers a region of a physical object in a captured image, on a basis of a [[non-extraction-target]] image displayed or projected in an imaging space and the captured image (Kajimura, [0054]: teaches the image processing apparatus 100 and learning device 500 being formed from a computer, which includes CPU 611 <read on processing unit> as shown in FIG. 6A; [0056]: teaches CPU 611 and GPU 610 carrying out processing tasks by inference unit 150; [0023]: teaches generating a virtual viewpoint image based on the viewpoint <read on imaging space> of captured images; [0031]: teaches the inference unit inferring and extracting a foreground region that corresponds to the object <read on physical object> from an input image <read on captured image>) in which
PNG
media_image1.png
292
408
media_image1.png
Greyscale
[[the non-extraction-target image and the physical object present in the imaging space are captured, wherein]]
[[an appearance of the non-extraction-target image depending on a position and an orientation of a camera that images the captured image is known.]]
However, Kajimura does not expressly disclose
a processing unit that infers a region of a physical object in a captured image, on a basis of a non-extraction-target image displayed or projected in an imaging space and the captured image in which
the non-extraction-target image and the physical object present in the imaging space are captured, wherein
an appearance of the non-extraction-target image depending on a position and an orientation of a camera that images the captured image is known.
Fujita discloses
a processing unit that infers a region of a physical object in a captured image, on a basis of a non-extraction-target image displayed or projected in an imaging space and the captured image (Fujita, [0023]: teaches generating "a virtual viewpoint image representing a view from a virtual viewpoint <read on imaging space> based on a plurality of captured images captured and obtained by a plurality of image capture devices from different directions"; [0030]: teaches generating a virtual viewpoint image through projective transformations based on obtaining captured images, which include foreground images and background images <read on non-extraction-target image> that are extracted from foreground and background regions respectively; Note: "non-extraction-target image" is being interpreted as a "background image" as this term is not explicitly defined in the specification) in which
the non-extraction-target image and the physical object present in the imaging space are captured (Fujita, [0041]: teaches captured image 201 being obtain by certain image capture device 2 by capturing an image of objects <read on captured physical object being present in imaging space> as shown in FIGS. 2A-2F), wherein
PNG
media_image2.png
610
417
media_image2.png
Greyscale
an appearance of the non-extraction-target image depending on a position and an orientation of a camera that images the captured image is known (Fujita, [0029]: teaches using virtual viewpoint information for generating a virtual viewpoint image <read on appearance of non-extraction-target image> based on the position and orientation of said virtual viewpoint <read on camera>, which includes "parameters <read on known position and orientation> representing the three-dimensional position of a virtual viewpoint and parameters representing the orientation of the virtual viewpoint in pan, tilt, and roll directions").
Fujita is analogous art with respect to Kajimura because they are from the same field of endeavor, namely processing video data for foreground and background information. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate a neural network that learns and infers camera and shape information to repair virtual viewpoint images as taught by Fujita into the teaching of Kajimura. The suggestion for doing so would result in a neural network that can enhance the output of the foreground and background images, such as denoising, thereby improving results. Therefore, it would have been obvious to combine Fujita with Kajimura.
Regarding Claim 12, it recites the limitations that are similar in scope to Claim 1, but in an information processing method. As shown in the rejection, the combination of Kajimura and Fujita discloses the limitations of Claim 1. Additionally, Kajimura discloses an information processing method comprising by a processor (Kajimura, [0023]: teaches a method for generating a virtual viewpoint image based on 3D shape data and captured images; [0054]: teaches the image processing apparatus 100 and learning device 500 being formed from a computer, which includes CPU 611 <read on processor> as shown in FIG. 6A)…
Thus, Claim 12 is met by Kajimura according to the mapping presented in the rejection of Claim 1, given the information processing apparatus corresponds to an information processing method.
Regarding Claim 13, it recites the limitations that are similar in scope to Claim 1, but in a program. As shown in the rejection, the combination of Kajimura and Fujita discloses the limitations of Claim 1. Additionally, Kajimura discloses a program for causing a computer to function as an information processing apparatus including (Kajimura, [0055]: teaches CPU 611 controlling a computer "by using programs or data stored in any of the ROM 612, the RAM 613, and the like," where CPU 611 causes image processing apparatus <read on information processing apparatus> 100 to function as units)…
Thus, Claim 13 is met by Kajimura according to the mapping presented in the rejection of Claim 1, given the information processing apparatus corresponds to a program.
Regarding Claim 2, the combination of Kajimura and Fujita discloses the information processing apparatus of Claim 1. Additionally, Kajimura further discloses wherein the processing unit includes
an inference unit that outputs a mask image obtained (Kajimura, [0039]: teaches a foreground obtaining unit 160 obtaining foreground information outputted from the first learned model 151 and the second learned model 152 of inference unit 150, where a mask image is generated) by
extracting the region of the physical object from the captured image, on a basis of [[the non-extraction-target image and]] the captured image that have been input (Kajimura, [0036]: teaches the first learned model 151 extracting the foreground region in the captured image based on the first region, the second region, and the candidate region, which then outputs the foreground information; [0038]: teaches the foreground regions being regions that correspond to objects based on the inputted image <read on input>).
However, Kajimura does not expressly disclose
extracting the region of the physical object from the captured image, on a basis of the non-extraction-target image and the captured image that have been input.
Fujita discloses
extracting the region of the physical object from the captured image, on a basis of the non-extraction-target image and the captured image that have been input (Fujita, [0030]: teaches using background images <read on non-extraction target image> to obtain background regions that area not foreground regions, which are then used <read on input> to determine texture information, such as color).
Fujita is analogous art with respect to Kajimura because they are from the same field of endeavor, namely processing video data for foreground and background information. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate a neural network that learns and infers camera and shape information to repair virtual viewpoint images as taught by Fujita into the teaching of Kajimura. The suggestion for doing so would result in a neural network that can enhance the output of the foreground and background images, such as denoising, thereby improving results. Therefore, it would have been obvious to combine Fujita with Kajimura.
Regarding Claim 3, the combination of Kajimura and Fujita discloses the information processing apparatus of Claim 2. Additionally, Kajimura further discloses wherein the inference unit is generated by machine learning to
output the mask image on a basis of the non-extraction-target image to be input and a second captured image obtained [[by imaging the non-extraction-target image]] (Kajimura, [0036]: teaches the first learned model 151 extracting the foreground region in the captured image <read on second captured image> based on the first region, the second region, and the candidate region, which then outputs the foreground information; [0039]: teaches a foreground obtaining unit 160 obtaining foreground information outputted from the first learned model 151 and the second learned model 152 of inference unit 150, where a mask image is generated; Note: "imaging" is being interpreted as "processing").
However, Kajimura does not expressly disclose
output the mask image on a basis of the non-extraction-target image to be input and a second captured image obtained by imaging the non-extraction-target image.
Fujita discloses
output the mask image on a basis of the non-extraction-target image to be input and a second captured image obtained by imaging the non-extraction-target image (Fujita, [0030]: teaches using background images <read on non-extraction target image> to obtain background regions that are not foreground regions, which are then used <read on imaging> to determine texture information, such as color).
Fujita is analogous art with respect to Kajimura because they are from the same field of endeavor, namely processing video data for foreground and background information. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate a neural network that learns and infers camera and shape information to repair virtual viewpoint images as taught by Fujita into the teaching of Kajimura. The suggestion for doing so would result in a neural network that can enhance the output of the foreground and background images, such as denoising, thereby improving results. Therefore, it would have been obvious to combine Fujita with Kajimura.
Regarding Claim 4, the combination of Kajimura and Fujita discloses the information processing apparatus of Claim 3. Additionally, Kajimura further discloses wherein the inference unit is generated by
supervised learning using a zero-image in which values of all pixels are 0 as training data (Kajimura, [0039]: teaches a binarized image, where the foreground regions represented as white pixels, while the background region is represented as black pixels <read on pixel values being 0>; [0048]: teaches learning unit 530 generating the first and second learned models through supervised learning and training data that corresponds to respective images, where the foreground region information is inferred, which creates binarized images; Note: binarized images are black-and-white images, where black pixels are represented as 0, and white pixels are represented as 1).
Regarding Claim 5, the combination of Kajimura and Fujita discloses the information processing apparatus of Claim 1. Kajimura does not expressly disclose the limitations of Claim 5; however, Fujita discloses wherein the non-extraction-target image includes
a rendered image obtained by rendering 3D data on a basis of the position and the orientation of the camera (Fujita, [0068]: teaches generating a background viewpoint image <read on rendered image> by obtaining 3D shape data of a background, such as a stadium, where a CG model is generated based on comparing vector normals of the CG model to the directional vectors of the cameras (i.e., the positions and attitudes <read on orientation>)).
Fujita is analogous art with respect to Kajimura because they are from the same field of endeavor, namely processing video data for foreground and background information. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate a neural network that learns and infers camera and shape information to repair virtual viewpoint images as taught by Fujita into the teaching of Kajimura. The suggestion for doing so would result in a neural network that generates 3D foreground and background models that can be used to generate an output viewpoint image, thereby improving results. Therefore, it would have been obvious to combine Fujita with Kajimura.
Regarding Claim 6, the combination of Kajimura and Fujita discloses the information processing apparatus of Claim 1. Kajimura does not expressly disclose the limitations of Claim 6; however, Fujita discloses wherein
the rendered image is displayed as a background for the physical object (Fujita, [0062]: teaches the virtual viewpoint image generation unit 12 superimposing "the foreground virtual viewpoint image <read on image of physical object> onto the background virtual viewpoint image <read on rendered image>," which generates the virtual viewpoint image).
Fujita is analogous art with respect to Kajimura because they are from the same field of endeavor, namely processing video data for foreground and background information. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate a neural network that learns and infers camera and shape information to repair virtual viewpoint images as taught by Fujita into the teaching of Kajimura. The suggestion for doing so would result in a neural network that generates 3D foreground and background models that can be used to generate an output viewpoint image, thereby improving results. Therefore, it would have been obvious to combine Fujita with Kajimura.
Regarding Claim 7, the combination of Kajimura and Fujita discloses the information processing apparatus of Claim 5. Kajimura does not expressly disclose the limitations of Claim 7; however, Fujita discloses wherein
the rendered image is displayed by a display device that is disposed behind the physical object (Fujita, [0036]: teaches "the display device 4 obtains a virtual viewpoint image from the image generation apparatus 1 and outputs the virtual viewpoint image <read on rendered image> using a display device such as a display"; [0062]: teaches the virtual viewpoint image generation unit 12 superimposing "the foreground virtual viewpoint image <read on image of physical object> onto the background virtual viewpoint image <read on rendered image being behind the physical object>," which generates the virtual viewpoint image),
the camera being a base point (Fujita, [0027]: teaches "the states of an image capture device <read on camera> are the image capture device's position, attitude (orientation and image capture direction), focal length, optical center, distortion, and the like," where "the position and attitude of an image capture device are expressed by a coordinate system having three axes orthogonal to a single origin <read on base point> (hereinafter referred to as a world coordinate system)").
Fujita is analogous art with respect to Kajimura because they are from the same field of endeavor, namely processing video data for foreground and background information. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate a neural network that learns and infers camera and shape information to repair virtual viewpoint images as taught by Fujita into the teaching of Kajimura. The suggestion for doing so would result in a neural network that generates 3D foreground and background models that can be used to generate an output viewpoint image, thereby improving results. Therefore, it would have been obvious to combine Fujita with Kajimura.
Claims 8-11 are rejected under 35 U.S.C. 103 as being unpatentable over Kajimura (US 20220383507 A1), hereinafter referenced as Kajimura, in view of Fujita et al. (US 20220383589 A1), hereinafter referenced as Fujita as applied to Claim 1 above respectively, and further in view of Shi et al. (US 20220377300 A1), hereinafter referenced as Shi.
Regarding Claim 8, the combination of Kajimura and Fujita discloses the information processing apparatus of Claim 3. Additionally, Kajimura further discloses wherein the inference unit is generated by machine learning to
output the mask image [[further on a basis of a lighting condition that is input]] (Kajimura, [0039]: teaches a foreground obtaining unit 160 obtaining foreground information outputted from the first learned model 151 <read on machine learning> and the second learned model 152 of inference unit 150, where a mask image is generated).
However, the combination of Kajimura and Fujita does not expressly disclose
output the mask image further on a basis of a lighting condition that is input.
Shi discloses
output the mask image further on a basis of a lighting condition that is input (Shi, [0062]: teaches object images including lighting label information <read on lighting condition>, where "the lighting label information indicates the lighting condition corresponding to the object image <read on input>"; [0103]: teaches the intermediate data format including lighting condition information that is the most similar to the target lighting condition).
Shi is analogous art with respect to Kajimura, in view of Fujita because they are from the same field of endeavor, namely generating 3D models based on viewpoint images. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to generate a 3D model from viewpoint images using lighting label information as taught by Shi into the teaching of Kajimura, in view of Fujita. The suggestion for doing so would allow the system to determine the most similar lighting condition to the target lighting condition from obtained data, which would result in a generated 3D model with the appropriate lighting data in the output viewpoint image. Therefore, it would have been obvious to combine Shi with Kajimura, in view of Fujita.
Regarding Claim 9, the combination of Kajimura and Fujita discloses the information processing apparatus of Claim 3. Additionally, Kajimura further discloses wherein the processing unit further includes
a second inference unit that outputs a [[quasi-captured]] image on a basis of an input image (Kajimura, [0039]: teaches a foreground obtaining unit 160 obtaining foreground information outputted from the first learned model 151 and the second learned model 152 <read on second inference unit> of inference unit 150, where a mask image is generated; [0038]: teaches the foreground regions being regions that correspond to objects based on the inputted image; [0031]: teaches the second learned model 152 extracting the foreground region from captured image 310 as a result of inference as shown in FIG. 4B), and
PNG
media_image3.png
254
377
media_image3.png
Greyscale
the second inference unit is generated by supervised learning [[using the non-extraction-target image as an input]] (Kajimura, [0058]: teaches generating a second learned model 152 <read on second inference unit>, which is a part of inference unit 150; [0048]: teaches learning unit 530 generating the first learned model 151 and the second learned model <read on second inference unit> 152, which is a part of inference unit 150, perform learning, such as supervised learning), and
the second captured image as training data (Kajimura, [0048]: teaches learning unit 530 generating the first and second learned models through supervised learning and training data that corresponds to respective images <read on second captured image>, where the foreground region information is inferred).
However, the Kajimura does not expressly disclose
a second inference unit that outputs a quasi-captured image on a basis of an input image, and
the second inference unit is generated by supervised learning using the non-extraction-target image as an input.
Fujita discloses
[[a second inference unit that outputs a quasi-captured image on a basis of an input image, and]]
the second inference unit is generated by supervised learning using the non-extraction-target image as an input (Fujita, [0030]: teaches using background images <read on non-extraction target image> to obtain background regions that area not foreground regions, which are then used <read on input> to determine texture information, such as color).
Fujita is analogous art with respect to Kajimura because they are from the same field of endeavor, namely processing video data for foreground and background information. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate a neural network that learns and infers camera information to repair virtual viewpoint images as taught by Fujita into the teaching of Kajimura. The suggestion for doing so would result in a neural network that can enhance the output of the foreground and background images, such as denoising, thereby improving results. Therefore, it would have been obvious to combine Fujita with Kajimura.
However, the combination of Kajimura and Fujita does not expressly disclose
a second inference unit that outputs a quasi-captured image on a basis of an input image.
Shi discloses
a second inference unit that outputs a quasi-captured image on a basis of an input image (Shi, [0049]: teaches generating intermediate images <read on quasi-captured image> based on target pixel point change information and position difference information between the reference viewing angle and the target viewing angle of an object image <read on input image>; Note: "quasi-captured image" is being interpreted as "intermediate image" as this term is not well-defined in the specification).
Shi is analogous art with respect to Kajimura, in view of Fujita because they are from the same field of endeavor, namely generating 3D models based on viewpoint images. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to generate a 3D model from viewpoint images using lighting label information as taught by Shi into the teaching of Kajimura, in view of Fujita. The suggestion for doing so would allow the system to determine the most similar lighting condition to the target lighting condition from obtained data, which would result in a generated 3D model with the appropriate lighting data in the output viewpoint image. Therefore, it would have been obvious to combine Shi with Kajimura, in view of Fujita.
Regarding Claim 10, the combination of Kajimura, Fujita, and Shi discloses the information processing apparatus of Claim 9. The combination of Kajimura and Fujita does not expressly disclose the limitations of Claim 10; however, Shi discloses wherein the processing unit
combines the captured image and the quasi-captured image (Shi, [0051]: teaches superimposing <read on combining> the intermediate image on the scene image <read on captured image>).
Shi is analogous art with respect to Kajimura, in view of Fujita because they are from the same field of endeavor, namely generating 3D models based on viewpoint images. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to generate a 3D model from viewpoint images using lighting label information as taught by Shi into the teaching of Kajimura, in view of Fujita. The suggestion for doing so would allow the system to determine the most similar lighting condition to the target lighting condition from obtained data, which would result in a generated 3D model with the appropriate lighting data in the output viewpoint image. Therefore, it would have been obvious to combine Shi with Kajimura, in view of Fujita.
Regarding Claim 11, the combination of Kajimura, Fujita, and Shi discloses the information processing apparatus of Claim 9. The combination of Kajimura and Fujita does not expressly disclose the limitations of Claim 11; however, Shi discloses wherein the processing unit
combines the region of the physical object and the quasi-captured image (Shi, [0051]: teaches superimposing <read on combining> the intermediate image on the scene image <read on region of physical object>).
Shi is analogous art with respect to Kajimura, in view of Fujita because they are from the same field of endeavor, namely generating 3D models based on viewpoint images. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to generate a 3D model from viewpoint images using lighting label information as taught by Shi into the teaching of Kajimura, in view of Fujita. The suggestion for doing so would allow the system to determine the most similar lighting condition to the target lighting condition from obtained data, which would result in a generated 3D model with the appropriate lighting data in the output viewpoint image. Therefore, it would have been obvious to combine Shi with Kajimura, in view of Fujita.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Furukawa (US 20190102643 A1) discloses an image processing apparatus that extracts an object region corresponding to a predetermined object from an extraction target image;
Kang et al. (US 20210248795 A1) discloses using neural networks to complete coloring images; and
Langlotz et al. (US 20130314442 A1) discloses extracting a desired object from a source video stream.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KARL TRUONG whose telephone number is (703)756-5915. The examiner can normally be reached 10:30 AM - 7:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.D.T./Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights