DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 17 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because
Claim 17 is directed to a program and it doesn’t fall into four statutory category of inventions: process, machine, manufacture, or composition of matter.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: a 3D shape data generation unit, an interpolation unit, an image recognition unit, a position estimation unit and a texture generation unit followed by functional language generate, interpolate, recognize, estimate and calculate in claims 1-17.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
Specification indicates the above units are implemented in software/program and executed by a CPU. Support for this in the specification PGPUB “[0085] FIG. 20 is a block diagram illustrating a configuration example of the hardware of the computer in a case where the computer executes each processing executed by the information processing system 1 or the 3D model generation unit 12 with a program. …………..[0089] In the computer configured as described above, for example, the CPU 401 loads a program stored in the storage unit 408 into the RAM 403 via the input/output interface 405 and the bus 404 and executes the program, so that the above-described series of processing is performed”.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-3, and 13-17 are rejected under 35 U.S.C. 102(a)(1)/102(a)(2) as being anticipated by YOSHIKAWA et al. (US Pat. Pub. No. 20190174109 “Yoshikawa”).
Regarding claim 1 Yoshikawa teaches An information processing device (Fig. 1) comprising:
a 3D shape data generation unit configured to generate 3D shape data representing a 3D shape of a modeling target, which is a target for generating a 3D model, on a basis of a plurality of captured images obtained by capturing the modeling target from different viewpoint positions (“[0040] A video processing device according to an aspect of the present disclosure includes: a model generation unit that generates, from a multi-view video of a same scene captured from different viewpoints, a three-dimensional model for the scene; [0131] Analyzer ex112 detects a blind spot by using Structure from Motion (SfM), for example. SfM is a technique of restoring a three-dimensional shape of a subject from a plurality of videos captured from different positions”); and
an interpolation unit configured to interpolate 3D shape data of a missing section of the modeling target, the missing section being missing in 3D shape data generated by the 3D shape data generation unit (“[0083] At this point, rendering unit 124 may determine a region three-dimensional model 152 of which is not generated, a region of a low quality, or a region difficult to reproduce, such as a surface of water, and interpolate these regions using CG or the like”).
Regarding claim 2 Yoshikawa teaches wherein the 3D shape data generation unit generates the 3D shape data on a basis of an image region of the modeling target in each of the plurality of captured images (“0057] First, video processing device 102 acquires multi-view video 151 (S111). Next, three-dimensional model generation unit 121 uses multi-view video 151 to generate three-dimensional model 152 of a target scene that is a capturing space (S112)”).
Regarding claim 3 Yoshikawa teaches wherein the 3D shape data generation unit generates the 3D shape data using a visual-volume intersection method (“[0057]….. Next, three-dimensional model generation unit 121 uses the acquired positional information on the plurality of cameras 111 to perform visual cone intersection or the like, generating voxel data”).
Regarding claim 13 Yoshikawa teaches wherein the modeling target includes a plurality of types of objects (“[0182] For example, when soccer is played in a stadium, analyzer ex112 may restore the three-dimensional shapes of players and a ball”).
Regarding claim 14 Yoshikawa teaches wherein the modeling target includes a person (“[0182] For example, when soccer is played in a stadium, analyzer ex112 may restore the three-dimensional shapes of players and a ball”).
Regarding claim 15 Yoshikawa teaches wherein the modeling target includes a tool used by a person (“[0182] For example, when soccer is played in a stadium, analyzer ex112 may restore the three-dimensional shapes of players and a ball”).
Claim 16 is directed to a method claim and its steps are similar in scope and functions performed by the elements of the device claim 1 and therefore claim 16 is also rejected with the same rationale as specified in the rejection of claim 1.
Claim 17 is directed to a program claim (Yoshikawa “[0008] Note that these general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or may be realized by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium”) and its elements are similar in scope and functions performed by the elements of the device claim 1 and therefore claim 17 is also rejected with the same rationale as specified in the rejection of claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over Yoshikawa in view of SUGIMURA et al. (US PAT. PUB. No. 20220044027 “Sugimura”).
Regarding claim 4 Yoshikawa is silent about an image recognition unit configured to recognize the modeling target in each of the plurality of captured images and detect the image region of the modeling target that has been recognized.
Sugimura teaches an image recognition unit (Fig. 1 element 132) configured to recognize the modeling target in each of the plurality of captured images and detect the image region of the modeling target that has been recognized (“[0054] In another example, whether or not there is a target within the image is recognized for each of the plurality of images by means of deep learning using an image recognition process that is represented by a convolutional neural network (CNN). In the deep learning, the images where the distortion of the target is small, the images where the target is close to the center, the images where the target is not partially shown, the images where the target is only slightly obstructed and the like are included in correct training data (positive images), whereas the images that do not correspond to these are included in incorrect training data (negative images) so as to generate a learning model in advance, which is utilized for the recognition of the target within the images”);
Sugimura and Yoshikawa are analogous art as both of them are related to image processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Yoshikawa by recognizing the modeling target in each of the plurality of captured images and detect the image region of the modeling target that has been recognized as taught by Sugimura.
The motivation for the above is to use automation in image recognition.
Regarding claim 5 Yoshikawa modified by Sugimura teaches wherein the image recognition unit recognizes the modeling target by an inference model having a structure of a neural network generated by machine learning (Sugimura “[0054] In another example, whether or not there is a target within the image is recognized for each of the plurality of images by means of deep learning using an image recognition process that is represented by a convolutional neural network (CNN). In the deep learning, the images where the distortion of the target is small, the images where the target is close to the center, the images where the target is not partially shown, the images where the target is only slightly obstructed and the like are included in correct training data (positive images), whereas the images that do not correspond to these are included in incorrect training data (negative images) so as to generate a learning model in advance, which is utilized for the recognition of the target within the images”).
The motivation is same as the motivation of claim 4.
Claim(s) 6-10 are rejected under 35 U.S.C. 103 as being unpatentable over Yoshikawa in view of Fleischman et al. (US Pat. Pub. No. 20210375062 “Fleischman”) and Kar et al. (US Pat. Pub. No. 20200257862 “Kar”).
Regarding claim 6 Yoshikawa teaches wherein the interpolation unit specifies a three-dimensional region in which the missing section should originally exist (“[0083] At this point, rendering unit 124 may determine a region three-dimensional model 152 of which is not generated”) but is silent about on a basis of a three-dimensional position of a predetermined portion of the modeling target estimated by a position of a predetermined portion of the modeling target in each of the plurality of captured images;
Fleischman teaches specifying a three-dimensional region in which an identified section should originally exist on a basis of predetermined portion of modeling target estimated by a position of a predetermined portion of the modeling target in each of the plurality of captured images (“[0115]……. The spatial indexing system receives 830 a video comprising a plurality of image frames from a camera system, where the video is captured as the camera system is moved through the portion of the building. The spatial indexing system identifies 840 one or more objects in the plurality of image frames, where each of the one or more objects is associated with an object type and a location within the portion of the building where the object is disposed. The spatial indexing system generates 850 a 3D model of the portion of the building from the plurality of image frames. For each of the one or more identified objects, the spatial indexing system modifies 860 a region of the 3D model corresponding to the location within the portion of the building where the identified object is disposed to include the identified object”) and
Kar teaches a three-dimensional position of a predetermined portion of the modeling target estimated by a position of a predetermined portion of the modeling target (“[0213] A corresponding position on the 3D object mesh is determined at 1014. According to various embodiments, the position may be determined by applying the barycentric coordinates as weights for the vertices of the corresponding intersection face in the object 3D mesh”);
Fleischman, Kar and Yoshikawa are analogous art as both of them are related to image processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Yoshikawa by specifying a three-dimensional region in which the missing section should originally exist on a basis of a three-dimensional position of a predetermined portion of the modeling target estimated by a position of a predetermined portion of the modeling target in each of the plurality of captured images similar to specifying a three-dimensional region in which an identified section should originally exist on a basis of predetermined portion of modeling target estimated by a position of a predetermined portion of the modeling target in each of the plurality of captured images as taught by Fleischman and by having a three-dimensional position of a predetermined portion of the modeling target estimated by a position of a predetermined portion of the modeling target as taught by Kar.
The motivation for the above is to correctly identify the position of missing portion of model so that texture can be provided at correct location.
Yoshikawa modified by Fleischman and Kar teaches adds 3D shape data of the missing section to the three-dimensional region that has been specified (Yoshikawa “[0083] At this point, rendering unit 124 may determine a region three-dimensional model 152 of which is not generated, a region of a low quality, or a region difficult to reproduce, such as a surface of water, and interpolate these regions using CG or the like”).
Regarding claim 7 Yoshikawa modified by Fleischman and Kar teaches position estimation unit configured to estimate the three-dimensional position of the predetermined portion on a basis of a position of the predetermined portion of the modeling target in each of the plurality of captured images (Kar “[0213] A corresponding position on the 3D object mesh is determined at 1014. According to various embodiments, the position may be determined by applying the barycentric coordinates as weights for the vertices of the corresponding intersection face in the object 3D mesh”).
The motivation is same as the motivation of claim 6.
Regarding claim 8 Yoshikawa modified by Fleischman and Kar teaches wherein the position estimation unit calculates a plurality of three-dimensional positions as the three-dimensional position of the predetermined portion on a basis of a position of the predetermined portion of the modeling target in each of the plurality of captured images, and estimates the three-dimensional position of the predetermined portion on a basis of the plurality of three-dimensional positions that have been calculated (Fleischman “[0115]……. The spatial indexing system receives 830 a video comprising a plurality of image frames from a camera system, where the video is captured as the camera system is moved through the portion of the building. The spatial indexing system identifies 840 one or more objects in the plurality of image frames, where each of the one or more objects is associated with an object type and a location within the portion of the building where the object is disposed. The spatial indexing system generates 850 a 3D model of the portion of the building from the plurality of image frames. For each of the one or more identified objects, the spatial indexing system modifies 860 a region of the 3D model corresponding to the location within the portion of the building where the identified object is disposed to include the identified object”
Kar “[0213] A corresponding position on the 3D object mesh is determined at 1014. According to various embodiments, the position may be determined by applying the barycentric coordinates as weights for the vertices of the corresponding intersection face in the object 3D mesh”).
The motivation is same as the motivation of claim 6.
Regarding claim 9 Yoshikawa modified by Fleischman and Kar teaches wherein the position estimation unit estimates a barycentric position of the plurality of the three-dimensional positions that have been calculated, as the three-dimensional position of the predetermined portion (Kar “[0213] A corresponding position on the 3D object mesh is determined at 1014. According to various embodiments, the position may be determined by applying the barycentric coordinates as weights for the vertices of the corresponding intersection face in the object 3D mesh”).
The motivation is same as the motivation of claim 6.
Regarding claim 10 Yoshikawa modified by Fleischman and Kar teaches wherein the predetermined portion of the modeling target is a portion included in the missing section. (Yoshikawa “[0083] At this point, rendering unit 124 may determine a region three-dimensional model 152 of which is not generated, a region of a low quality, or a region difficult to reproduce, such as a surface of water, and interpolate these regions using CG or the like”).
The motivation is same as the motivation of claim 6.
Claim(s) 11 is rejected under 35 U.S.C. 103 as being unpatentable over Yoshikawa in view of Byrod et al. (US Pat. Pub. No. 20240394960 “Byrod”).
Regarding claim 11 Yoshikawa is silent about a texture generation unit configured to generate a texture to be pasted to the 3D shape data on a basis of an image of the modeling target in each of the plurality of captured images.
Byrod teaches a texture generation unit (integral part of element 120) configured to generate a texture to be pasted to the 3D shape data on a basis of an image of the modeling target in each of the plurality of captured images (“[0080] In some implementations, server device 120 can process the map data collected for each image capture point 1104-1110 to generate textures for the three-dimensional models generated for each image capture point. For example, server device 120 can combine the images captured for each perspective A, B, C, D at each image capture point 1104-1110 to generate a panoramic image (e.g., 360-degree image) of each image capture point. This panoramic image, or portions thereof, corresponding to an image capture point can be applied to the three-dimensional model generated for the image capture point to add texture (e.g., imagery representing objects, colors, textures, etc.) to the three-dimensional model”);
Byrod and Yoshikawa are analogous art as both of them are related to image processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Yoshikawa by having a texture generation unit configured to generate a texture to be pasted to the 3D shape data on a basis of an image of the modeling target in each of the plurality of captured images as taught by Byrod.
The motivation for the above is to create realistic model.
Claim(s) 12 is rejected under 35 U.S.C. 103 as being unpatentable over Yoshikawa in view of Somasundaram et al. (US Pat. Pub. No. 20180302543 “Somasundaram”).
Regarding claim 12 Even though Yoshikawa teaches plurality of cameras (Fig. 1 elements 101) but is silent about wherein any one or more of the plurality of captured images are captured images captured by a rolling shutter type.
Somasundaram teaches wherein any one or more of the plurality of captured images are captured images captured by a rolling shutter type (“[0029] In some embodiments, as illustrated in FIG. 3, images may be captured using a rolling shutter”);
Somasundaram and Yoshikawa are analogous art as both of them are related to image processing.
Therefore it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Yoshikawa by having any one or more of the plurality of captured images are captured images captured by a rolling shutter type as taught by Somasundaram.
The motivation for the above is to use a camera that is compact in designs and has high-resolution sensors providing high sensitivity, lower noise, and wide dynamic range
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Tapas Mazumder whose telephone number is (571)270-7466. The examiner can normally be reached M-F 8:00 AM-5:00 PM PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at 571-272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TAPAS MAZUMDER/Primary Examiner, Art Unit 2615