Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities: Spec [0024] contains a grammatical error, it should be “a distance” and not “an distance.”
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 9- 11 and 13-15 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Furukawa (US 20260017874 A1).
Regarding claim 1, Furukawa discloses An information processing apparatus comprising: (Fig. 1 Item 10, discloses an information processing apparatus that creates a virtual viewpoint image using a plurality of cameras)
one or more memories storing instructions; and (Spec [0365], “The storage unit includes, for example, a hard disk, a RAM disk, a nonvolatile memory, and the like”)
one or more processors executing the instructions to: (Spec [0366], “The RAM 1002 also appropriately stores data and the like necessary for the CPU 1001 to execute various processing”)
obtain a first virtual viewpoint image, wherein the first virtual viewpoint image is generated based on a group of images (Spec [0013], “generates a virtual viewpoint image by synthesizing multi-viewpoint images”) obtained by shooting a subject (Fig. 1 Item 32, discloses a subject) from a plurality of directions; (Fig. 1 Items 31-1 to 31-8, discloses a plurality of cameras that takes images of the subject in which a virtual viewpoint image is created using these images)
obtain at least one image coordinate in a specific region in the first virtual viewpoint image; (Spec [0110] “when a position where a failure has occurred in a virtual viewpoint image is designated as a failure region by a user”)
determine, based on a pixel value at the at least one image coordinate, (Spec [0110] “when a position where a failure has occurred in a virtual viewpoint image is designated as a failure region by a user”) a parameter for generating a virtual viewpoint image; (Spec [0111], “the virtual viewpoint image is generated with the corrected color synthesis weight”)
and generate a second virtual viewpoint (Spec [0257], “plurality of virtual viewpoint images”) image using the determined parameter (Spec [0164], “a plurality of color synthesis weights based on the set color synthesis weights W0 to W2 may be set”).
Regarding claim 9, Furukawa discloses The information processing apparatus according to claim 1, wherein (Fig. 1 Item 11, discloses an information processing apparatus that creates a virtual viewpoint image using a plurality of cameras)
the one or more processors further execute the instructions to generate the second virtual viewpoint image (Spec [0257], “plurality of virtual viewpoint images”) based on pixel values in the group of images (Spec [0259], “Another parameter may be, for example, reflectance or absorptivity of light of the subject, or in a case where a pixel is used as a unit, a pixel value of a pixel in the vicinity thereof”) and shape data of the subject through image-based rendering (Spec [0256], “The candidate image re-synthesis unit 194 substantially generates the multi-viewpoint images used to generate the virtual viewpoint image on the basis of the information of the shape and texture of the 3D model”).
Regarding claim 10, Furukawa discloses The information processing apparatus according to claim 1, wherein (Fig. 1 Item 11, discloses an information processing apparatus that creates a virtual viewpoint image using a plurality of cameras).
the one or more processors further execute the instructions to: (Spec [0366], “The RAM 1002 also appropriately stores data and the like necessary for the CPU 1001 to execute various processing”).
generate a three-dimensional model of the subject (Spec [0173], “The 3D model generation unit 112 generates a model having three-dimensional information of the subject 131”) based on pixel values in the group of images (Spec [0181], “Data of color information is also generated as a texture in association with the 3D shape data”), the parameter (Spec [0222], “the three-dimensional image synthesis unit 151 sets the color synthesis weight according to an angle between the normal direction of the subject and the line-of-sight direction from the viewpoint position for each pixel of the multi-viewpoint image restored by the information of the shape and texture of the 3D model according to the virtual viewpoint position by the rendering processing”), and shape data of the subject; (Specs [0179], “The 3D model can express shape information representing the surface shape of the subject 131 in the form of mesh data expressed by a connection between vertices called polygon mesh”) and
generate the second virtual viewpoint image using the three-dimensional model through model-based rendering (Spec [0256], “The candidate image re-synthesis unit 194 substantially generates the multi-viewpoint images used to generate the virtual viewpoint image on the basis of the information of the shape and texture of the 3D model”).
Regarding claim 11, Furukawa discloses The information processing apparatus according to claim 1, wherein (Fig. 1 Item 11, discloses an information processing apparatus that creates a virtual viewpoint image using a plurality of cameras)
the one or more processors further execute the instructions to (Spec [0366], “The RAM 1002 also appropriately stores data and the like necessary for the CPU 1001 to execute various processing”) obtain the at least one image coordinate based on inputting (Spec [0120], “the user inputs a mark M including a circle or the like in the drawing so as to surround the failure portion BL in order to designate the position of the recognized failure portion BL”) performed on an indication apparatus (Spec [0365], “The input unit 1006 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, and the like”) that accepts an indication for designating image coordinates in the first virtual viewpoint image (Spec [0120], “since the region of the mark M is a failure region designated by the user”).
Regarding claim 13, Furukawa discloses The information processing apparatus according to claim 1, wherein (Fig. 1 Item 11, discloses an information processing apparatus that creates a virtual viewpoint image using a plurality of cameras)
the specific region is a noise region that appears in the first virtual viewpoint image (Fig. 7 Q(t) and Spec [0093], “The third failure is a failure that occurs when the color of the object to be the foreground is synthesized with the color of the object on the background side at the boundary between the foreground and the background.”).
Regarding claim 14, it recites the limitations of claim 1 but in method form and is therefore
rejected using the same rationale as claim 1.
Regarding claim 15, Furukawa discloses A non-transitory computer-readable storage medium storing a program for causing a computer to execute a control method for an information processing apparatus, the control method comprising: (Spec [0366], “the CPU 1001 loads a program stored in the storage unit 1008 into the RAM 1002 via the input/output interface 1005 and the bus 1004 and executes the program”)
The rest of the limitations in claim 15 are recitations of the limitations of claim 1 and are therefore rejected using the same rationale as claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Furukawa (US 20260017874 A1) in view of Nerayoff, et al. (US 20140039987 A1).
Regarding claim 2, Furukawa teaches The information processing apparatus according to claim 1, wherein (Fig. 1 Item 11, discloses an information processing apparatus that creates a virtual viewpoint image using a plurality of cameras)
the one or more processors further execute the instructions to: (Spec [0366], “The RAM 1002 also appropriately stores data and the like necessary for the CPU 1001 to execute various processing”)
determine the parameter based on a pixel value at the updated image coordinates (Spec [0111] Lines 3-5, teaches correcting a pixel using color synthesis weights).
However, Furukawa does not teach search for a pixel with a pixel value similar to the pixel value at the at least one image coordinate, and update the image coordinates; and
Nerayoff teaches search for a pixel with a pixel value similar to the pixel value at the at least one image coordinate (Spec [0034], “the algorithm may be configured to detect groups of pixels with similar colors (a range of colors that be considered to be similar may be specified in a system configuration, such as that similar colors may be a maximum of 2 shades difference, with reference to shades indicated by a color chart)”), and update the image coordinates; and (Spec [0035], “A further refinement could be made to accommodate "holes" or imperfections in a detected shape due to, for example, a reflection of the sun, a windshield area, or a roof rack.”).
It would be obvious for a person having ordinary skill in the art to implement Nerayoff’s algorithm to Furukawa’s apparatus. The person possessing ordinary skill in the art would be motivated to automate finding noise regions.
Claims 3-4 are rejected under 35 U.S.C. 103 as being unpatentable over Furukawa (US 20260017874 A1) in view of Nerayoff, et al. (US 20140039987 A1) and Jia, et al. (US 20200286293 A1).
Regarding claim 3, Furukawa in view of Nerayoff teaches The information processing apparatus according to claim 2, wherein
However, Furukawa in view of Nerayoff does not teach the one or more processors further execute the instructions to search pixels around a pixel at the at least one image coordinate
Jia teaches the one or more processors further execute the instructions to search pixels around a pixel at the at least one image coordinate (Spec [0055], “for each texture hole region as indicated in the texture hole mask (118-1), bordering pixels of the hole region can be located first”).
It would be obvious for a person having ordinary skill in the art to configure Furukawa’s CPU to use a Nerayoff’s algorithm combined with Jia’s techniques to search around the image coordinate. The person possessing ordinary skill in the art would be motivated to find if the region containing the image coordinate is a noise region.
Regarding claim 4, Furukawa in view of Nerayoff teaches The information processing apparatus according to claim 2, wherein
However, Furukawa in view of Nerayoff does not teach the one or more processors further execute the instructions to search pixels in frames that precede and succeed the first virtual viewpoint image
Jia teaches the one or more processors further execute the instructions to search pixels in frames that precede and succeed the first virtual viewpoint image (Spec [0031], “a reference image frame (e.g., a synthesized or non-synthesized image, etc.) at a reference time point (e.g., a previous time point, a subsequent time point, etc.) different from the target time point may be used”).
It would be obvious for a person having ordinary skill in the art to configure Furukawa’s CPU to use Nerayoff’s algorithm combined with Jia’s technique to search preceding and succeeding frames. The person possessing ordinary skill in the art would be motivated to find if there are noise regions in preceding and succeeding frames and fix them if they do exist.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Furukawa (US 20260017874 A1) in view of Kamiyama (US 20210272284 A1).
Regarding claim 5, Furukawa teaches The information processing apparatus according to claim 1, wherein (Fig. 1 Item 11, discloses an information processing apparatus that creates a virtual viewpoint image using a plurality of cameras)
the one or more processors further execute the instructions to: (Spec [0366], “The RAM 1002 also appropriately stores data and the like necessary for the CPU 1001 to execute various processing”)
However, Furukawa does not disclose obtain a training model that outputs image coordinates in the specific region with respect to inputting of the first virtual viewpoint image;
output the image coordinates in the specific region based on the training model; and
obtain the output image coordinates.
Kamiyama discloses obtain a training model that outputs image coordinates in the specific region with respect to inputting of the first virtual viewpoint image; (Fig. 5 Item 20, discloses that a training model is obtained)
output the image coordinates in the specific region based on the training model (Spec [0060], “The training model 20 after the training outputs position information with a higher probability of the position in a region where the annotations TG1 to TG3 overlap, and with a lower probability of the position in a region where the annotations TG1 to TG3 do not overlap”); and
obtain the output image coordinates (Spec [0074], “The detection section 5 detects first to third position information MK1 to MK3 about the object OBJD from the detection image IMGD”).
It would be obvious for a person having ordinary skill in the art to add Kamiyama’s training model to Furukawa’s apparatus. The person possessing ordinary skill in the art would be motivated to combine the prior art to automate detecting noise regions in the virtual viewpoint image.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Furukawa (US 20260017874 A1) in view of Kamiyama (US 20210272284 A1) and Lu, et al. (US 20180322648 A1).
Regarding claim 6, Furukawa in view of Kamiyama teaches The information processing apparatus according to claim 5, wherein (Furukawa, Fig. 1 Item 11, discloses an information processing apparatus that creates a virtual viewpoint image using a plurality of cameras)
the one or more processors further execute the instructions to: (Spec [0366], “The RAM 1002 also appropriately stores data and the like necessary for the CPU 1001 to execute various processing”)
obtain the generated training model (Kamiyama, Fig. 5 Item 20, discloses obtaining a training model).
However, Furukawa in view of Kamiyama does not disclose generate a training model that outputs image coordinates in the specific region with respect to inputting of the first virtual viewpoint image; and
Lu teaches generate a training model that outputs image coordinates in the specific region with respect to inputting of the first virtual viewpoint image; and (Lu, Spec [0060], “Training model generation unit 222 may be configured to generate one or more training models that are used to generate a first detection information.”).
It would be obvious for a person having ordinary skill in the art to add Lu’s training model generation unit to Furukawa’s apparatus to generate Kamiyama’s training model. The person possessing ordinary skill in the art would be motivated to combine the prior art to automate the creation of a training model for detecting noise regions.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over (US 20260017874 A1) in view of Fukuda (US 20160081645 A1).
Regarding claim 7, Furukawa teaches The information processing apparatus according to claim 1, wherein (Fig. 1 Item 11, discloses an information processing apparatus that creates a virtual viewpoint image using a plurality of cameras)
However, Furukawa does not teach the one or more processors further execute the instructions to, in a case where a degree of similarity between a pixel value in the specific region and a pixel value in the group of images that contributes to determination of the pixel value in the specific region is equal to or higher than a threshold, determine the parameter so that an extent of contribution is small compared to a case where the degree of similarity is lower than the threshold.
Fukuda teaches the one or more processors further execute the instructions to, in a case where a degree of similarity between a pixel value in the specific region and a pixel value in the group of images that contributes to determination of the pixel value in the specific region is equal to or higher than a threshold, determine the parameter so that an extent of contribution is small compared to a case where the degree of similarity is lower than the threshold (Spec [0168], “having a pixel value whose absolute difference value exceeds a predetermined threshold value, the pixel value of the regression surface at the coordinate position with the small correlation is determined as an outlier and is removed or weighted with a small weight”).
It would be obvious for a person having ordinary skill in the art to implement Fukuda’s algorithm on Furukawa’s apparatus. The person possessing ordinary skill in the art would be motivated to combine the prior art to prevent pixels from having their values changed when they do not need to.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Furukawa (US 20260017874 A1) in view of Baqai (US 11189017 B1).
Regarding claim 8, Furukawa teaches The information processing apparatus according to claim 1, wherein (Fig. 1 Item 11, discloses an information processing apparatus that creates a virtual viewpoint image using a plurality of cameras)
However, Furukawa does not teach the one or more processors further execute the instructions to, in a case where a degree of similarity between a pixel value in the group of images that contributes to determination of a pixel value in the specific region and a pixel value in a region different from the specific region is equal to or higher than a threshold, determine the parameter so that an extent of contribution is large compared to a case where degree of similarity is lower than the threshold.
Baqai teaches the one or more processors further execute the instructions to, in a case where a degree of similarity between a pixel value in the group of images that contributes to determination of a pixel value in the specific region and a pixel value in a region different from the specific region is equal to or higher than a threshold, determine the parameter so that an extent of contribution is large compared to a case where degree of similarity is lower than the threshold (Paragraph (40), “Thus, rather than completely discarding a corresponding pixel from a candidate image that does not meet a pixel similarity threshold (e.g., by setting its similarity weight to ‘0’), the process may simply lower its similarity weight value to where it does not have a large influence on the pixel value in the resultant fused image, but it still provides at least some contribution to the resultant fused image (e.g., by setting its similarity weight to ‘0.1’)”).
It would be obvious for a person having ordinary skill in the art to combine Furukawa’s apparatus with Baqai’s process. The person possessing ordinary skill in the art would be motivated to combine the prior art to determine pixel values with the parameter more user-friendly.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Furukawa (US 20260017874 A1) in view of Fukuda, et al. (US 20210335010 A1).
Regarding claim 12, Furukawa discloses The information processing apparatus according to claim 11, wherein
However, Furukawa does not teach the image coordinates in the first virtual viewpoint image are designated as a quadrilateral region.
Fukuda teaches the image coordinates in the first virtual viewpoint image are designated as a quadrilateral region (Spec [0184], “generate quadrilateral search windows”).
It would be obvious for a person having ordinary skill in the art to either replace Furukawa’s teaching of using a circular region with Fukuda’s teaching of using a quadrilateral region or add Fukuda’s teachings as an option. The person possessing ordinary skill in the art would be motivated to combine the prior art to allow the user more options to use shapes for more flexible regions.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to IRVING SHI whose telephone number is (571)272-9613. The examiner can
normally be reached Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use
the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
Tammy Goddard can be reached at (571) 272-7773. The fax phone number for the organization where
this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users. To
file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or
571-272-1000.
/IRVING NMN SHI/Examiner, Art Unit 2611
/TAMMY PAIGE GODDARD/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights