DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2-15 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 2, 4, and 12 recite the limitation “based on a determined degree of importance of the object” in the last clauses. The limitation renders the claims indefinite because it is unclear whether this degree of importance corresponds to the earlier-recited degree or a new/different degree. For the purpose of further examination, the limitation has been interpreted as “based on [[a]]the determined degree of importance of the object.”
Claims 3, 5-11, and 13-15 depend from claim 2 and therefore inherit the deficiency of claim 2 discussed above.
Claims 7 and 9-11 recite the limitation “similar.” The is a relative and/or subjective term which renders the claim indefinite. The term “similar” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
A claim that requires the exercise of subjective judgment without restriction renders the claim indefinite. In re Musgrave, 431 F.2d 882, 893, 167 USPQ 280, 289 (CCPA 1970). Claim scope cannot depend solely on the unrestrained, subjective opinion of a particular individual purported to be practicing the invention. Datamize LLC v. Plumtree Software, Inc., 417 F.3d 1342, 1350, 75 USPQ2d 1801, 1807 (Fed. Cir. 2005)); see also Interval Licensing LLC v. AOL, Inc., 766 F.3d 1364, 1373, 112 USPQ2d 1188 (Fed. Cir. 2014).
For the purpose of further examination, the limitation has been interpreted as “matching.”
Claims 8-11 depends from claim 7 and therefore inherit the deficiency of claim 7 discussed above.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 16, 17, 18, and 19 is/are rejected under 102(a)(2) as being anticipated by Rematas et al. (US 2023/0281913 A1), hereinafter referred to as Rematas.
Regarding claim 1, Rematas teaches an image processing apparatus comprising:
one or more hardware processors (Rematas ¶¶0005: “The system can include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations”); and
one or more memories storing one or more programs configured to be executed by the one or more hardware processors, the one or more programs including instructions (Rematas ¶¶0005 discussed above) for:
obtaining image capturing parameters of each of a plurality of imaging apparatuses arranged at positions different from one another (Rematas ¶¶0032: “the plurality of images can include images depicting a same, or similar, scene of the environment captured from different positions and angles”; Rematas ¶¶0094: “using camera parameters provided with the dataset”);
obtaining data of a captured image obtained by image capturing by each of the plurality of imaging apparatuses (Rematas Fig. 6: 602; Rematas ¶¶0100: “At 602, a computing system can obtain input data. The input data can be descriptive of a three-dimensional position and a two-dimensional view direction”);
obtaining virtual viewpoint information including at least one of information indicating a position of a virtual viewpoint and information indicating a viewing direction from the virtual viewpoint (Rematas ¶¶0102: “The view synthesis output can include a novel view synthesis that differs from the plurality of panoramic images”; Rematas ¶¶0104: “a computing system can obtain a three-dimensional position and a two-dimensional view direction. The three-dimensional position and the two-dimensional view direction can be associated with a location and direction in an environment”; Rematas ¶¶0159: “enable great flexibility in placing a virtual camera for novel view synthesis”; Rematas Fig. 6: 604; Rematas Fig. 7: 702);
determining a learning condition of a learning model estimating radiance fields corresponding to an object existing in an image capturing area of the plurality of imaging apparatuses based on the virtual viewpoint information (Rematas ¶¶0105: “the computing system can process the three-dimensional position and a two-dimensional view direction with a machine-learned view synthesis model to generate a view synthesis output. The machine-learned view synthesis model can include a neural radiance field model (e.g., a Mip-NERF model). The machine-learned view synthesis model can be trained on a plurality of images of the environment along with depth data descriptive of depths in the environment”; Rematas Fig. 7: 704-706); and
performing learning of the learning model based on the learning condition, the image capturing parameters, and data of the captured image (Rematas ¶¶0105 discussed above; also see Rematas ¶¶0064: “The training computing system 150 can include a model trainer 160 that trains the machine-learned models 120”; Rematas Fig. 10).
Regarding claim 16, Rematas teaches the image processing apparatus according to claim 1, wherein the one or more programs further include instructions for:
estimating radiance fields corresponding to the object by using a learned model, the learning model for which learning has been performed (Rematas ¶¶0035: “the machine-learned view synthesis model can include a neural radiance field model”; Rematas ¶¶0105: “The machine-learned view synthesis model can include a neural radiance field model (e.g., a Mip-NERF model)”); and
generating a virtual viewpoint image corresponding to an appearance from the virtual viewpoint (Rematas ¶¶0035 & ¶¶0105 discussed above; Rematas ¶¶0106: “the computing system can provide the view synthesis output for display in which the view synthesis output includes a novel view synthesis of the environment”; Rematas Figs. 2-3, 5, 9),
the virtual viewpoint information includes information indicating a position of the virtual viewpoint and information indicating a viewing direction from the virtual viewpoint (Rematas Fig. 7 & ¶¶0103-¶¶0106: “a computing system can obtain a three-dimensional position and a two-dimensional view direction. The three-dimensional position and the two-dimensional view direction can be associated with a location and direction in an environment … the computing system can process the three-dimensional position and a two-dimensional view direction with a machine-learned view synthesis model to generate a view synthesis output”), and
the generating of the virtual viewpoint image is performed by inputting the virtual viewpoint information to the learning model (Rematas Fig. 7 & ¶¶0103-¶¶0106 discussed above).
Regarding claim 17, Rematas teaches the image processing apparatus according to claim 16, wherein the one or more programs further include instructions for:
displaying and outputting the virtual viewpoint image on a display device by performing display control (Rematas Fig. 7: 706; Rematas Figs. 2-3, 5, 9).
Regarding claim 18, Rematas teaches an image processing method (Rematas Abstract: “Systems and methods for view synthesis and three-dimensional reconstruction can learn an environment by utilizing a plurality of images of an environment and depth data”) comprising the steps described in claim 1. Therefore claim 18 is rejected using the same rationale as claim 1 discussed above.
Regarding claim 19, Rematas teaches a non-transitory computer readable storage medium storing a program for causing a computer to perform a control method of an image processing apparatus (Rematas ¶¶0005 discussed above), the control method comprising the steps described in claim 1. Therefore, claim 19 is rejected using the same rationale as applied to claim 1 discussed above.
Allowable Subject Matter
Claim 2-15 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 2, the prior art of record teaches that it was known at the time the application was filed to use the image processing apparatus according to claim 1, wherein the one or more programs further include instructions for:
obtaining a virtual viewpoint image corresponding to an appearance from the virtual viewpoint based on the virtual viewpoint information (Rematas Figs. 2 & 9); and
determining that the learning of the learning model is performed based on an object area, which is an image area corresponding to the object in the obtained virtual viewpoint image (Rematas Fig. 2: 204 – shows objects in the virtual viewpoint; Rematas Fig. 9; Rematas ¶¶0076: “The image processing task may be object detection, where the image processing output identifies one or more regions in the one or more images and, for each region, a likelihood that region depicts an object of interest”).
However, the prior art, alone or in combination, does not appear to each or suggest determining a degree of importance of the object and determining that the learning condition is performed based on the determined degree of importance of the object.
Claims 3, 5-11, and 13-15 depend from claim 2 and therefore are objected to for the same reason as claim 2 discussed above.
Regarding claim 4, the prior art of record teaches that it was known at the time the application was filed to use the image processing apparatus according to claim 1, wherein the one or more programs further include instructions for:
estimating a three-dimensional shape of the object by using at least part of the obtained data of a plurality of the captured images (Rematas ¶¶0106: “the machine-learned view synthesis model may generate a three-dimensional reconstruction output that can include a geometry aware representation (e.g., a mesh representation)”); and
using at least one of information indicating a volume of the object obtained based on an estimated three-dimensional shape of the object and information indicating a distance from a position of the virtual viewpoint obtained based on an estimated three-dimensional shape of the object to the object (Rematas ¶¶0122-¶¶0124: “the system can introduce a series of lidar-based losses that allow accurate surface estimation both for solid structures like buildings and for volumetric formations such as trees/vegetation … Neural radiance fields can fit a coordinate-based neural network with parameters θ to describe a volumetric scene from a set of posed images … NeRF can use ray marching to sample the volumetric radiance field and can composite the sampled density and color to render the incoming radiance of a particular ray”).
However, the prior art, alone or in combination, does not appear to each or suggest determining a degree of importance of the object and determining that the learning condition is performed based on the determined degree of importance of the object.
Regarding claim 12, the prior art of record teaches that it was known at the time the application was filed to use the image processing apparatus according to claim 1, but does not appear to explicitly teach or suggest determining a degree of importance of the object in a case where the learning of the learning model is performed based on the virtual viewpoint information and information indicating a type of the object, and the determining of the learning conditions is performed based on a determined degree of importance of the object.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1 and 16-19 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 and 2-17 of copending Application No. 18/820,353 (reference application). Although the claims at issue are not identical, they are not patentably distinct from each other because both applications are directed to multi-camera methods, systems, and media for obtaining camera parameters, image data, and virtual viewpoint information, determining a learning condition by estimating radiance fields, and performing learning based on the learning conditions.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOO J SHIN whose telephone number is (571)272-9753. The examiner can normally be reached M-F; 10-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Soo Shin/Primary Examiner, Art Unit 2667 571-272-9753
soo.shin@uspto.gov
Prosecution Insights