DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 4 is objected to because of the following informalities: Claim 4, limitation 2 recites, “selecting the learning ray based on the number of selectins”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2-8 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 2-4, 6 and 8 each recite the limitation “the number of selections”. There is insufficient antecedent basis for this limitation in the claim.
Claim 5 and 7 recite the limitation “the number of selections” and “the determined number of selections”. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 8-12 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Kumar et al (US 20250157178 A1).
Regarding claim 1, Kumar discloses an image processing apparatus (Kumar fig. 2; [0056], “FIG. 2 shows a block diagram of an example image processing configuration”) comprising:
one or more hardware processors (Kumar [0056], “a processor (e.g., CPU) 204”); and
one or more memories storing one or more programs configured to be executed by the one or more hardware processors, the one or more programs including instructions (Kumar [0061], “execute instructions from a memory, such as instructions 208 from the memory 206, instructions stored in a separate memory coupled to or included in the image signal processor”) for:
obtaining data of a plurality of captured images obtained by image capturing from a plurality of viewpoints (Kumar [0079], “image frames 420 may be received from a plurality of cameras (obtaining data of a plurality of captured images), and more particularly from the respective image sensors of the plurality of cameras … a scene may span across two or more of the image frames 420 (image capturing from a plurality of viewpoints). For example, a first portion of the object may be depicted in a first image frame and a second portion of the object may be depicted in a second image frame.”);
Kumar does not expressly disclose in exact words the following limitations. However, as cited below Kumar reads on the limitations.
obtaining an object area corresponding to a representation of an object in each of the plurality of captured images (Kumar [0079], “an object (e.g., a truck) in a scene may span across two or more of the image frames 420. For example, a first portion of the object may be depicted in a first image frame and a second portion of the object may be depicted in a second image frame (portion/an object area corresponding to a representation of an object in each of the plurality of captured images).”);
setting a learning ray group corresponding to pixels of each of the plurality of captured images, which is used for learning of information relating to a three-dimensional field of an image capturing space that is an image capturing target from the plurality of viewpoints, based on the obtained object area (Kumar [0085], “Ordering the neural rays 603 into the ordered set 605 (a learning ray group) creates groups of neural rays having a similar characteristic, which can assist object detection (based on the object area) … ordered set 605 of neural rays that are close to each other in the 3D space (relating to a three-dimensional field of an image capturing space) and likely to be representing the same objects in a scene … the ordered set 605 also ensures that neural rays from different image frames taken by different cameras are aligned with each other (setting a learning ray group corresponding to pixels of each of the plurality of captured images, that is an image capturing target from the plurality of viewpoints), which contributes to determining a complete and accurate representation of a scene”
and
performing learning of information relating to the three-dimensional field based on the set learning ray group (Kumar [0089], “Determining the ordered set 605 of neural rays 603 thereby enables efficiently applying a graph neural network to the neural rays 603. Applying graph network 607 allows learning features that aggregate information from neighboring points”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention retrieve scene and object information from images generate a ray group and learn information about the captured scene. This would have been done to create representation that are accurate and free of errors. See for example, Kumar [0089], “Applying graph network 607 allows learning features that aggregate information from neighboring points, which results in features that are robust to noise/occlusions and incorporate 3D context.”.
Regarding claim 2, Kumar discloses the image processing apparatus according to claim 1, wherein the one or more programs further include instructions for:
controlling the number of selections of the learning ray corresponding to the object area in a case where a learning ray that is used for learning of information relating to the three-dimensional field is selected from among the learning ray group (Kumar [0085], “groups are formed in the ordered set 605 of neural rays that are close to each other in the 3D space and likely to be representing the same objects in a scene (number of selections of the learning ray corresponding to the object area in a case where a learning ray that is used for learning of information relating to the scene/three-dimensional field is selected from among the learning ray group)”).
Regarding claim 3, Kumar discloses the image processing apparatus according to claim 1, wherein the one or more programs further include instructions for:
obtaining information relating to the number of selections of the learning ray as learning parameters in a case where a learning ray that is used for learning of information relating to the three-dimensional field is selected from among the learning ray group (Kumar [0097], “The neural rays 603 are organized into the ordered set 605 based on a parameter. For example, the parameter may be a respective azimuth angle associated with each of the neural rays 603. In such aspects, method 700 may include determining the respective azimuth angle associated with each of the neural rays 603.”).
Regarding claim 4, Kumar discloses the image processing apparatus according to claim 3, wherein the one or more programs further include instructions for:
obtaining the number of selections of the learning ray corresponding to the object area as the learning parameters in a case where a learning ray that is used for learning of information relating to the three-dimensional field is selected from among the learning ray group (Kumar [0083], “parameters may include weights (e.g., priorities) for different features and combinations of features (e.g., image features of a vehicle, pedestrian, stationary object, the sky, etc.).”; and
selecting the learning ray based on the number of selectins of the learning ray corresponding to the object area obtained as the learning parameters in a case where the learning ray that is used for learning of information relating to the three-dimensional field is selected from among the set learning ray group (Kumar [0103], “Each neural ray of the ordered set of neural rays represents three-dimensional positions of pixels of an image frame of the plurality of image frames, each point is associated with a node of a plurality of nodes of the graph network, and the feature set includes features of each of the plurality of image frames.”).
Regarding claim 8, Kumar discloses the image processing apparatus according to claim 1, wherein the one or more programs further include instructions for:
determining the number of selections of the learning ray corresponding to the object area in a case where a learning ray that is used for learning of information relating to the three-dimensional field is selected from among the set learning ray group for each captured image in the plurality of captured images (Kumar [0085], “Ordering the neural rays 603 (a learning ray that is used for learning of information relating to the three-dimensional field) into the ordered set 605 creates groups of neural rays having a similar characteristic, which can assist object detection (determining a number of selections of the learning ray corresponding to an object area for object detection)”).
Regarding claim 9, Kumar discloses the image processing apparatus according to claim 1, wherein the one or more programs further include instructions for:
setting a learning area in each of the plurality of captured images (Kumar [0086], “parameter for ordering, the 3D points representing each of the pixels of image frames 420 (plurality of captured images) … The neural ray's direction vector in 3D space (setting a learning area in each of the plurality of captured images is first determined).”);
selecting a learning ray that is used for learning of information relating to the three-dimensional field from among the learning ray group corresponding to the learning area (Kumar figs. 5,6; [0080], “The first image sensor configuration 500 shows a potential arrangement of near-field image sensors 502, 504, 506, 508 with a field of view of 180 degrees.”; [0084], “Neural ray creation 602 includes projecting each pixel of each image frame of image frames 420 onto a 3D space” – for object detection a neural ray is selected from neural rays that represent the captured area); and
performing learning of information relating to the three-dimensional field by using the selected learning ray (Kumar [0089], “Determining the ordered set 605 of neural rays 603 thereby enables efficiently applying a graph neural network to the neural rays 603. Applying graph network 607 allows learning features”).
Regarding claim 10, Kumar discloses the image processing apparatus according to claim 1, wherein the one or more programs further include instructions for:
setting a learning area in each of the plurality of captured images (Kumar [0086], “parameter for ordering, the 3D points representing each of the pixels of image frames 420 (plurality of captured images) … The neural ray's direction vector in 3D space (setting a learning area in each of the plurality of captured images is first determined).”);
setting the learning ray group corresponding to pixels included in the learning area (Kumar [0097], “Each neural ray of the ordered set 605 of neural rays 603 represents three-dimensional positions of pixels of an image frame of the image frames 420.”); and
performing learning of information relating to the three-dimensional field based on the set learning ray group (Kumar [0047], “structuring the neural rays as a graph network enables a deep learning model to reason about the relationships between different parts of the scene”).
Regarding claim 11, The image processing apparatus according to claim 9, wherein the one or more programs further include instructions for:
obtaining learning area information indicating the learning area in each of the plurality of captured images (Kumar [0079], “an object (e.g., a truck) in a scene may span across two or more of the image frames 420. For example, a first portion of the object may be depicted in a first image frame and a second portion of the object may be depicted in a second image frame (exemplary learning area information).”); and
setting the learning area in each of the plurality of captured images based on the learning area information (Kumar [0085], “groups of neural rays having a similar characteristic, which can assist object detection accuracy”).
Regarding claim 12, The image processing apparatus according to claim 9, wherein the one or more programs further include instructions for:
setting the learning area in each of the plurality of captured images by calculating the learning area in each of the plurality of captured image based on the object area in each of the plurality of captured images (Kumar [0083], “one or more training datasets may be used that contain image frames captured by the various cameras … one or more parameter updates may be computed (calculated) based on differences between the predicted outputs and the expected outputs. In particular, the parameters may include weights (e.g., priorities) for different features and combinations of features (e.g., image features of a vehicle, pedestrian, stationary object, the sky, etc. (computing/calculating the learning area in each of the plurality of captured image based on the object area in each of the plurality of captured images))”).
Claim 14 recites a method which corresponds to the function performed by the image processing apparatus of claim 1. As such, the mapping and rejection of claim 1 above is considered applicable to the method of claim 14.
Claim 15 recites a non-transitory computer readable storage medium which corresponds to the function performed by the image processing apparatus of claim 1. As such, the mapping and rejection of claim 1 above is considered applicable to the non-transitory computer readable storage medium of claim 15.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Kumar in view of Chakraborty et al (US 20240242423 A1).
Regarding claim 13, Kumar discloses the image processing apparatus according to claim 1, wherein the one or more programs further include instructions for:
But does not disclose obtaining virtual viewpoint information including at least viewpoint position information indicating a position of a virtual viewpoint and viewing direction information indicating a viewing direction at the virtual viewpoint; and
generating a virtual viewpoint image corresponding to the virtual viewpoint based on the obtained virtual viewpoint information and information relating to the three-dimensional field obtained as results of the learning.
However, Chakraborty discloses
obtaining virtual viewpoint information including at least viewpoint position information indicating a position of a virtual viewpoint and viewing direction information indicating a viewing direction at the virtual viewpoint (Chakraborty [0034], “3D spatial data is derived from a plurality of previously captured digital images 504A-504C of real-world object 202, captured from various angles and viewpoints”); and
generating a virtual viewpoint image corresponding to the virtual viewpoint based on the obtained virtual viewpoint information and information relating to the three-dimensional field obtained as results of the learning (Chakraborty fig. 8; [0047], “FIG. 8, again showing neural radiance model 500. As shown, based on an estimate 800 of the imaged pose of the real-world object, the neural radiance model outputs a virtual object view 802 (generating a virtual viewpoint image obtained as a result of learning) depicting virtual object 208, having an appearance and pose that is consistent with the digital image of the real-world object. As discussed above, the imaged pose of the real-world object is estimated in any suitable way—e.g., via an inference model trained on synthetic object views output by the neural radiance model, or via the neural radiance model itself.”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Kumar with Chakraborty to generate virtual images corresponding to user viewpoints using learning techniques. This would have been done to easily and efficiently generate a variety of desired views of objects based on learning techniques. See, for example, Chakraborty [0073], “generating the plurality of synthetic object views includes varying one or more of lighting conditions, background conditions, occlusions, and a virtual object pose between one or more of the plurality of synthetic object views.”).
Allowable Subject Matter
Claims 5-7 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten or amended to overcome the rejection(s) under 35 U.S.C. 112(b), set forth in this Office action and also rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 5, none of the prior art of record, alone or in combination, disclose the claim as recited, including, “obtaining a ratio of the learning rays corresponding to the object area as the learning parameters; determining the number of selections of the learning ray corresponding to the object area in a case where the learning ray that is used for learning of information relating to the three-dimensional field is selected based on the obtained ratio”.
Claim 7 is allowable for depending from claim 5.
Regarding claim 6, none of the prior art of record, alone or in combination, disclose the claim as recited.
Conclusion
See the notice of references cited (PTO-892) for prior art made of record, including art that is not relied upon but considered pertinent to applicant's disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JITESH PATEL whose telephone number is (571)270-3313. The examiner can normally be reached 8am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said A. Broome can be reached at (571) 272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JITESH PATEL/Primary Examiner, Art Unit 2612
Prosecution Insights