DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 08/17/2026 has been entered.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 08/25/2026 has been made record of and considered by the examiner.
Response to Arguments
Applicant’s arguments, see pages 8-11, filed 08/17/2026, with respect to the rejection(s) of claim(s) 1 and 10-16 under 35 USC 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of the amendment of 08/17/2026.
Response to Amendment
The amendment of 08/17/2026 has been accepted.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 10-11, 13-19, and 22-24 are rejected under 35 U.S.C. 103 as being unpatentable over Konuma (JP2020126393A), and in further view of Mora (US 2015/0178988 A1).
Consider claims 1, 13 and 14, Konuma discloses an image processing apparatus generating three-dimensional shape data corresponding to a foreground object for which synchronous image capturing is performed by a plurality of imaging apparatuses (¶12-13, 18-23; “the plurality of image pickup units 1 perform image pickup in synchronization with each other with high accuracy based on the synchronization signal from the synchronization unit 2. It is assumed that the plurality of imaging units 1 are installed so as to surround a subject including the ground 20 (competition space) of the stadium, as shown in FIG… generates the foreground mask image and the foreground texture image in which the region of the foreground image including the subject is extracted”), the image processing apparatus comprising:
one or more hardware processors (¶36-37); and
one or more memories storing one or more programs configured to be executed by the one or more hardware processors, the one or more programs including instructions for (¶36-37; “CPU 91 controls the entire computer using computer programs and data stored in the RAM92 and the ROM 93, and also executes each process performed by the image processing apparatus”):
obtaining captured images from the plurality of imaging apparatuses and image capturing parameters of the plurality of imaging apparatuses (¶12; “the plurality of image pickup units 1 perform image pickup in synchronization with each other with high accuracy based on the synchronization signal from the synchronization unit 2. It is assumed that the plurality of imaging units 1 are installed so as to surround a subject including the ground 20 (competition space) of the stadium, as shown in FIG.”; ¶21; “the deletion unit 10 uses the external parameter indicating the position and orientation of the imaging unit 1 and the internal parameter indicating the angle of view of the three-dimensional shape of the subject estimated by the shape estimation unit 5”);
obtaining a mask image, the mask image being prepared in advance based on image capturing parameters of an imaging apparatus corresponding to the mask image, a position of a first generation space in a virtual space corresponding to an image capturing-target space, and which the three-dimensional shape data is to be generated, wherein the mask image includes a mask area corresponding to an image area in which an actually existing space corresponding to an unnecessary space would be captured and a non-mask area corresponding to an image area in which an actually existing space corresponding to the second generation space would be captured, the unnecessary space being a portion of the first generation space outside the second generation space, and (¶24, 25; “it is possible to delete the images of apparently unnecessary parts such as the audience seats”; ¶27-31; “As shown in FIG. 7, in the second embodiment, the shape estimation unit 5 is supplied with the information of the shape estimation region from the region designation unit 12… the shape estimation region is a region for limiting the range in which shape estimation is performed, and in the present embodiment, for example, a rectangular parallelepiped region 81… The deleting unit 10 projects the shape estimation region on the image capturing angle of view of each image capturing unit 1 based on the external parameter and the internal parameter of each image capturing unit 1,… Such a reprojection area mask image 82 is generated… the deletion unit 10 masks the foreground mask image 42 and the foreground texture image 43 extracted by the extraction unit 3 using the reprojection area mask image 82. As a result, the masked re-extracted foreground texture image 83 and the re-extracted foreground mask image 84 are generated… the re-projection area mask image may be generated once, and the subsequent processing is a mask processing of a two-dimensional image); and
generating the three-dimensional shape data corresponding to the foreground object in the generation space, based on the obtained image capturing parameters of the plurality of imaging apparatuses, the obtained captured images from the plurality of imaging apparatuses, and the obtained mask image (¶20; “shape estimation unit 5 estimates the three-dimensional shape of each subject in the foreground based on the foreground mask image 42”; ¶27-31; “As shown in FIG. 7, in the second embodiment, the shape estimation unit 5 is supplied with the information of the shape estimation region from the region designation unit 12… the shape estimation region is a region for limiting the range in which shape estimation is performed, and in the present embodiment, for example, a rectangular parallelepiped region 81… The deleting unit 10 projects the shape estimation region on the image capturing angle of view of each image capturing unit 1 based on the external parameter and the internal parameter of each image capturing unit 1,… Such a reprojection area mask image 82 is generated… the deletion unit 10 masks the foreground mask image 42 and the foreground texture image 43 extracted by the extraction unit 3 using the reprojection area mask image 82. As a result, the masked re-extracted foreground texture image 83 and the re-extracted foreground mask image 84 are generated… the re-projection area mask image may be generated once, and the subsequent processing is a mask processing of a two-dimensional image… since the foreground mask image input to the shape estimation unit 5 has been deleted, the number of silhouette inside/outside determinations is reduced when performing the visual volume intersection method”).
However, Konuma fails to explicitly disclose a position of a second generation space included in the first generation space, wherein the unnecessary space and the second generation space are distinguished from each other based on the mask image and the image capturing parameters.
In related art, Mora discloses a position of a second generation space included in the first generation space, wherein the unnecessary space and the second generation space are distinguished from each other based on [occupancy] and the image capturing parameters (¶Mora ¶110-116; “Discretize the volume into voxels… Check occupancy for each voxel: Project voxel into each silhouette image. If the projection is outside at least one silhouette the voxel is empty. If the projection is inside all silhouettes the voxel is occupied.”, a bounded 3D reconstruction volume within which camera projection is used to classify voxels).
Konuma states, “in the background subtraction method, the audience seats and other image noises are also extracted as subjects when shooting a stadium competition or the like. The images of these audience seats and other image noises are data that are not substantially used unless they are converted into three-dimensional shape data. It is wasteful to process, transmit, and store such data that has no use destination.” Mora discloses an independent computation that determines if each voxel is inside or outside the Visual Hull (Mora ¶109). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the predefined bounded reconstruction volume of Mora into the target volume of Konuma to predictably yield wherein the unnecessary space and the second generation space are distinguished from each other based on the mask image and the image capturing parameters, thereby avoiding unnecessary space increasing processing load.
Consider claim 10, Konuma, as modified by Mora, discloses the claimed invention wherein the one or more programs further include an instruction for:
obtaining distance information indicating respective distances from a position of the imaging apparatus corresponding to the mask image to a plurality of points on a boundary surface between the unnecessary space and the second generation space (Konuma ¶26-31; Mora ¶110-116), and wherein
the unnecessary space and the second generation space are distinguished from each other further based on the distance information (Konuma ¶26-32; Mora ¶110-116).
Consider claim 11, Konuma, as modified by Mora, discloses the claimed invention wherein the three-dimensional shape data is generated in the second generation space by a visual hull method using silhouette images generated based on the obtained captured images, each of the silhouette images indicating a foreground area in which the foreground object is captured (Konuma ¶26-32; Mora ¶110-116).
Consider claim 15, Konuma, as modified by Mora, discloses the claimed invention herein the image capturing parameters of the imaging apparatus corresponding to the mask image and the mask image are stored in advance in a storage device and are obtained by reading the image capturing parameters and the mask image from the storage device (Konuma ¶18-32; 36-42; Mora ¶97, 166-169).
Consider claim 16, Konuma, as modified by Mora, discloses the claimed invention wherein the one or more programs further include an instruction for storing voxel data indicating the second generation space, a generation space after deletion, and wherein the stored voxel data is reused when generating the three-dimensional shape data for one or more subsequent frames (Konuma ¶18-32; 36-42; Mora ¶97, 109-120, 166-169).
Consider claim 17, Konuma, as modified by Mora, discloses the claimed invention wherein the mask image is one of a plurality of mask images respectively corresponding to a plurality of imaging apparatuses, and wherein the unnecessary space includes a spatial area corresponding to a mask area of at least one of the plurality of mask images based on image capturing parameters of a corresponding one of the plurality of imaging apparatuses (Konuma ¶18-32; Mora ¶97, 109-116).
Consider claim 18, Konuma, as modified by Mora, discloses the claimed invention wherein the first generation space is represented by a plurality of voxels, and wherein, for each of the plurality of voxels, whether the voxel belongs to the unnecessary space or the second generation space is determined based on whether a position at which the voxel is projected onto the mask image based on the image capturing parameters is located in the mask area or the non-mask area of the mask image (Konuma ¶26-32; Mora ¶109-116).
Consider claim 19, Konuma, as modified by Mora, discloses the claimed invention wherein the one or more programs further include instructions for generating information indicating the second generation space based on the mask image and the image capturing parameters before generating the three-dimensional shape data corresponding to the foreground object, and wherein the three-dimensional shape data is generated within the second generation space indicated by the generated information (Konuma ¶18-32; 36-42; Mora ¶97, 166-169).
Consider claim 22, Konuma, as modified by Mora, discloses the claimed invention wherein the one or more programs further include instructions for:
generating, based on at least one of the obtained captured images, a temporary silhouette image indicating an image area in which the foreground object is captured (Konuma ¶18-32; Mora ¶109-116);
obtaining a captured-image mask image including a captured-image mask area corresponding to an actually existing space corresponding to the unnecessary space (Konuma ¶18-32; Mora ¶109-116); and
generating a silhouette image in which an area of the temporary silhouette image corresponding to the captured-image mask area is set as a background area, wherein the three-dimensional shape data is generated using the generated silhouette image (Konuma ¶26-32; Mora ¶109-120).
Consider claim 23, Konuma, as modified by Mora, discloses the claimed invention wherein the distance information is a depth map in which each pixel value indicates a distance from the position of the imaging apparatus corresponding to the mask image to a corresponding point on the boundary surface, and wherein, for each of a plurality of voxels in the first generation space, whether the voxel belongs to the unnecessary space or the second generation space is determined by comparing a distance from the position of the imaging apparatus to the voxel with a pixel value at a position onto which the voxel is projected in the depth map (Konuma ¶18-32; Mora ¶93, 109-120, 125-131).
Consider claim 24, Konuma, as modified by Mora, discloses the claimed invention wherein the boundary surface includes a curved surface, and respective distances from the position of the imaging apparatus corresponding to the mask image to points on the curved surface vary according to respective positions of the points on the curved surface (Konuma ¶18-32; Mora ¶41, 93, 109-120, 125-131).
Claims 12 and 20-21 are rejected under 35 U.S.C. 103 as being unpatentable over Konuma, in view of Mora, as applied to claims 1, 10-11, 13-19, and 22-24 above, and further in view of Feng (‘Bottom and Concave Surface Rendering in Image-based Visual Hull’).
Consider claim 12, Konuma, as modified by Mora, discloses the claimed invention wherein the one or more programs further include an instruction for:
generating a virtual viewpoint image based on the generated three-dimensional shape data, the obtained captured images, and virtual viewpoint information including information indicating a position of a virtual viewpoint and a direction of a line-of-sight from the virtual viewpoint (Konuma ¶26-32; Mora ¶93, 109-120, 136).
In related art, Feng explicitly supports generating a virtual viewpoint image based on the generated three-dimensional shape data, the obtained captured images, and virtual viewpoint information including information indicating a position of a virtual viewpoint and a direction of a line-of-sight from the virtual viewpoint (Feng 4, 5).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the virtual camera of Feng into the unnecessary space method of Konuma, as modified by Mora, to predictably yield virtual viewpoint information including information indicating a position of a virtual viewpoint and a direction of a line-of-sight from the virtual viewpoint.
Consider claim 20, Konuma, as modified by Mora, discloses the claimed invention wherein the imaging apparatus corresponding to the mask image is an (Konuma ¶26-32; Mora ¶93, 109-120, 136).
In related art, Feng discloses wherein the imaging apparatus is an imaginary imaging apparatus that does not actually exist, and the image capturing parameters of the imaginary imaging apparatus include a position, a direction of an optical axis, and a viewing angle of the imaginary imaging apparatus (Feng 4. Bottom Rendering in IBVH).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the virtual camera of Feng into the unnecessary space method of Konuma, as modified by Mora, to predictably yield image capturing parameters of the imaginary imaging apparatus including a position, a direction of an optical axis, and a viewing angle of the imaginary imaging apparatus.
Consider claim 21, Konuma, as modified by Mora, discloses the claimed invention wherein the image capturing parameters of the imaginary imaging apparatus include distortion parameters set based on an assumption that the imaginary imaging apparatus has no lens distortion (Konuma ¶26-32; Mora ¶93, 109-120, 152; Feng 4, 5).
Relevant Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
JP5295044B2 discloses a method and program for extracting a mask image with high accuracy, including an unnecessary portion removal process capable of sequentially removing unnecessary portions in the vicinity of the floor surface and on the silhouette outline, and the mask. JP6914734B2 discloses a method for silhouette extraction.
JP2020187528A (previous action).
US 2013/0315471 A1 discloses a method for concave surface modeling.
Guan, 'Visual Hull Construction in the Presence of Partial Occlusion.'
Furukawa, "Carved visual hulls for image-based modeling.'
Kleinkort, 'Visual Hull Method for Realistic 3D Particle Shape Reconstruction Based on High- Resolution Photographs of Snowflakes in Free Fall from Multiple Views.'
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ASHLEY HYTREK whose telephone number is (703)756-4562. The examiner can normally be reached M-F 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Steve Koziol can be reached at (408)918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ASHLEY HYTREK/ Examiner, Art Unit 2665
/Stephen R Koziol/ Supervisory Patent Examiner, Art Unit 2665
Prosecution Insights