Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Use of indicates a limitation is not explicitly disclosed by the reference alone.
Claim(s) 1-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Takahashi (US 2022/0172474) in view of Izumi (US 2022/0084300).
Claim 1
Takahashi discloses an image processing apparatus (abstract: “image processing apparatus and an image processing method”) comprising:
one or more hardware processors (Takahashi, ¶ 209: “n the computer, a central processing unit (CPU) 301, a read only memory (ROM) 302, and a random access memory (RAM) 303 are connected to each other by a bus 304.”); and
one or more memories storing one or more programs configured to be executed by the one or more hardware processors (Takahashi, ¶ 209: “n the computer, a central processing unit (CPU) 301, a read only memory (ROM) 302, and a random access memory (RAM) 303 are connected to each other by a bus 304.”), the one or more programs including instructions for:
obtaining data of a plurality of captured images obtained by capturing an object from a plurality of directions (Takahashi, ¶ 42: “As illustrated in FIG. 1, the image capturing devices CAM-1 to CAM-N(N>1) are arranged so as to surround a person who is a subject, capture images of the subject, and supply the captured images obtained as the result to the image processing apparatus 11.”) and image capturing camera parameters, which are camera parameters corresponding to each of the plurality of captured images (Takahashi, ¶ 68: “The characteristic parameter representing a characteristic of the single image capturing device CAM includes, for example, internal parameters such as a focal length, a principal point, and an angle of view of the image capturing device CAM, and optical characteristic parameters such as distortion (aberration). The external parameter includes a relative coordinate value (relative position) of the self image capturing device CAM with respect to another image capturing device CAM.”);
generating (Takahashi, ¶ 99: “Then, the 3D model calculation unit 38 generates a 3D model of the object by means of a method such as Visual Hull by using a plurality of silhouette images in different directions and camera parameters of the N respective image capturing devices CAM. The silhouette images and the camera parameters may be acquired from the calibration processing unit 32 and the silhouette extraction unit 34, or may be calculated separately. Details of generation of the 3D model of the object are omitted herein.”); and
obtaining spatial information representing a space in which the object exists based on the plurality of object images and the object image camera parameters corresponding to each of the plurality of object images (Takahashi, ¶ 43: “The image processing apparatus 11 generates a 3D model of an object which is the subject from a plurality of captured images supplied from the image capturing devices CAM-1 to CAM-N. The data of the generated 3D model of the object includes, for example, image data of the captured images of the subject obtained from the respective image capturing devices CAM and 3D shape data representing the 3D shape of the subject. The 3D shape data may be, for example, a depth image corresponding to captured images captured by the plurality of image capturing devices CAM, or is represented by a point cloud representing a three-dimensional position of the object using a set of points, a polygon mesh representing the three-dimensional position using connection between vertices, or the like.”).
Takahashi does not explicitly disclose, but Izumi discloses
generating a plurality of object images by extracting an image region corresponding to an image of the object from each of the plurality of captured images (Izumi, ¶ 52: 57, 69: “The camera parameters include at least an external parameter and an internal parameter….The display device 26 or the viewing position detection device 27 can also supply, to the reproduction device 25 as necessary, information regarding a display function of the display device 26, such as an image size and an angle of view of an image displayed by the display device 26…The reproduction side can make a request for only an object to be viewed among a large number of objects existing in an imaging space, and cause a display device to display the object. For example, the reproduction side assumes a virtual camera having an imaging range that coincides with a viewing range of a viewer, makes a request for, among a large number of objects existing in the imaging space, only objects that can be captured by the virtual camera, and causes the display device to display the objects. The viewpoint of the virtual camera can be set to any position so that the viewer can see the field from any viewpoint in the real world.”);
object image camera parameters (Izumi, ¶ 52: 57, 69: “The camera parameters include at least an external parameter and an internal parameter….The display device 26 or the viewing position detection device 27 can also supply, to the reproduction device 25 as necessary, information regarding a display function of the display device 26, such as an image size and an angle of view of an image displayed by the display device 26…The reproduction side can make a request for only an object to be viewed among a large number of objects existing in an imaging space, and cause a display device to display the object. For example, the reproduction side assumes a virtual camera having an imaging range that coincides with a viewing range of a viewer, makes a request for, among a large number of objects existing in the imaging space, only objects that can be captured by the virtual camera, and causes the display device to display the objects. The viewpoint of the virtual camera can be set to any position so that the viewer can see the field from any viewpoint in the real world.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider object parameters.
One of ordinary skill in the art would have motivation to “reduce a processing load of drawing processing on a reproduction side.” (Izumi, ¶ 5). One of ordinary skill in the art would have had a reasonable expectation of success because both references consider segmentation of Multiview images into objects and creation of three-dimensional models.
Claim 2
Takahashi does not explicitly disclose, but Izumi discloses wherein the one or more programs further include instructions for: generating an image corresponding to an appearance from a virtual viewpoint based on the object image camera parameters, based on imaginary spatial information and the object image camera parameters and obtaining of the spatial information is performed by repeating updating of the imaginary spatial information so that a difference between the generated image and the object image corresponding to the object image camera parameters becomes small (Izumi, ¶ 98: “The distortion/color correction unit 41 corrects lens distortion and color of each imaging device 21 for N texture images supplied from the N imaging devices 21. As a result, the distortion and color variation between the N texture images are corrected, so that it is possible to suppress a feeling of strangeness when colors of a plurality of texture images are blended at the time of drawing. The image data of the corrected N texture images is supplied to the silhouette extraction unit 42 and the image transmission unit 48.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider minimizing the difference.
One of ordinary skill in the art would have motivation to “reduce a processing load of drawing processing on a reproduction side.” (Izumi, ¶ 5) while minimizing distortion. One of ordinary skill in the art would have had a reasonable expectation of success because both references consider segmentation of Multiview images into objects and creation of three-dimensional models.
Claim 3
Takahashi discloses wherein the spatial information includes information representing density at each position in the space (Takahashi, ¶ 43: “. The 3D shape data may be, for example, a depth image corresponding to captured images captured by the plurality of image capturing devices CAM, or is represented by a point cloud representing a three-dimensional position of the object using a set of points, a polygon mesh representing the three-dimensional position using connection between vertices, or the like.”)
Claim 4
Takahashi discloses wherein the spatial information includes information representing a signed distance from the surface of the object at each position in the space (Takahashi, ¶ 84: “Furthermore, in a case where a depth image in which the distance to the subject is stored as a depth value is also acquired, the silhouette image can be generated by separating a foreground region, which is a subject region, from a background region on the basis of the distance information of the depth image.”).
Claim 5
Takahashi discloses wherein the spatial information includes information representing a color at each position in the space (e.g. white/red coding for the capture subject; . Takahashi, ¶ 59: “In the example in C of FIG. 2, in the subject region of the silhouette image expressed in white, a foot region is colored in red, and the user is notified that the foot region is out of the image capturing range. Note that, in FIG. 2, the red region is expressed by hatching due to drawing restrictions.”)
Claim 5
Takahashi discloses wherein the spatial information includes information representing a color different for different directions at each position in the space (e.g. for camera 5 with associated direction; Takahashi, ¶ 59: “In the example in C of FIG. 2, in the subject region of the silhouette image expressed in white, a foot region is colored in red, and the user is notified that the foot region is out of the image capturing range. Note that, in FIG. 2, the red region is expressed by hatching due to drawing restrictions.”)
Claim 7
Takahashi does not explicitly disclose, but Izumi discloses wherein the one or more programs further include instructions for: generating an image corresponding to an appearance from an arbitrary viewpoint based on the spatial information (Izumi, ¶ 52: “The reproduction side can make a request for only an object to be viewed among a large number of objects existing in an imaging space, and cause a display device to display the object. For example, the reproduction side assumes a virtual camera having an imaging range that coincides with a viewing range of a viewer, makes a request for, among a large number of objects existing in the imaging space, only objects that can be captured by the virtual camera, and causes the display device to display the objects. The viewpoint of the virtual camera can be set to any position so that the viewer can see the field from any viewpoint in the real world.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider an arbitrary viewpoint.
One of ordinary skill in the art would have motivation to “reduce a processing load of drawing processing on a reproduction side.” (Izumi, ¶ 5). One of ordinary skill in the art would have had a reasonable expectation of success because both references consider segmentation of Multiview images into objects and creation of three-dimensional models.
Claim 8
Takahashi discloses wherein the image capturing camera parameters include information on a position, an orientation, a focal length, and a principal point of an imaging apparatus used for the image capturing and information on a size of the captured image (Takahashi, ¶ 68: “The characteristic parameter representing a characteristic of the single image capturing device CAM includes, for example, internal parameters such as a focal length, a principal point, and an angle of view of the image capturing device CAM, and optical characteristic parameters such as distortion (aberration). The external parameter includes a relative coordinate value (relative position) of the self image capturing device CAM with respect to another image capturing device CAM.”).
Claim 9
Takahashi does not explicitly disclose, but Izumi discloses wherein generation of the object image camera parameters is performed by changing information on a size of the captured image and information on a principal point of an imaging apparatus used for the image capturing, both being included in the image capturing camera parameters corresponding to the captured image, based on information indicating a position of an image region corresponding to an image of the object in the captured image (Izumi, ¶ 57, 69: “The camera parameters include at least an external parameter and an internal parameter….The display device 26 or the viewing position detection device 27 can also supply, to the reproduction device 25 as necessary, information regarding a display function of the display device 26, such as an image size and an angle of view of an image displayed by the display device 26.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider camera parameters as claimed.
One of ordinary skill in the art would have motivation to “reduce a processing load of drawing processing on a reproduction side.” (Izumi, ¶ 5). One of ordinary skill in the art would have had a reasonable expectation of success because both references consider segmentation of Multiview images into objects and creation of three-dimensional models.
Claim 10
Takahashi discloses wherein the one or more programs further include instructions for:
designating resolution of the object image (Takahashi, ¶ 129: “Furthermore, by setting the resolution of points or vertices, the voxel size, and the like representing the three-dimensional shape of the object at the time of calculating the 3D region to coarser setting values than those in the first embodiment, the processing load may further be reduced”); and
dividing an extracted image region corresponding to an image of the object into a plurality of unit regions based on designated resolution and generation of the object image is performed by extracting each of the plurality of divided unit regions from the captured image (Takahashi, ¶ 157: “The projection unit 92 projects the coordinates of the eight vertices of the truncated quadrangular pyramid of each of the N image capturing devices CAM supplied from the 3D region calculation unit 91 onto the viewpoint of the target camera to generate N truncated quadrangular pyramid projection regions obtained by projecting simple 3D regions of the N truncated quadrangular pyramids onto the projection plane of the target camera.”).
Claim 11
Takahashi discloses wherein the one ore more programs further include instructions for: classifying the captured image into a foreground region and a background region and generation of the object image is performed by extracting the classified foreground region from the captured image (Takahashi, ¶ 84: “Furthermore, in a case where a depth image in which the distance to the subject is stored as a depth value is also acquired, the silhouette image can be generated by separating a foreground region, which is a subject region, from a background region on the basis of the distance information of the depth image.”).
Claim 12
Takahashi discloses wherein the one or more programs further include instructions for:
obtaining a semantic label designating the processing-target object (Takahashi, ¶ 86: “. For example, in a case where the subject is a person, the semantic processing unit 35 can identify and classify the subject region of the entire person into respective parts such as the head, the hands (the right hand and the left hand), the arms (the right arm and the left arm), the feet (the right foot and the left foot), the torso, and the like by the semantic segmentation processing”); and
appending information on a label for semantic classification to part of image regions or pixels in the captured image and generation of the object image is performed by extracting image regions or pixels to which information on a label corresponding to the sematic label is appended from the captured image (Takahashi, ¶ 86: “The semantic processing unit 35 performs semantic segmentation processing of identifying semantic information of an object appearing as a subject in the captured image and adding the semantic information to each predetermined region. For example, in a case where the subject is a person, the semantic processing unit 35 can identify and classify the subject region of the entire person into respective parts such as the head, the hands (the right hand and the left hand), the arms (the right arm and the left arm), the feet (the right foot and the left foot), the torso, and the like by the semantic segmentation processing. In a case where the subject is not a person, the semantic processing unit 35 can identify the type of the object such as a ball, a racket, and a car and add the semantic information to it. The semantic processing unit 35 supplies the semantic information added to each predetermined region of the identified subject to the determination unit 36. Instance semantic segmentation, in which the semantic information includes individual information, may be performed. By utilizing the individual information, it is possible to perform processing focusing on the number of subjects such as one person and two persons or a specific subject, for example.”).
Claim 13
Takahashi discloses wherein generation of the object image is performed based on part of regions or pixels of the captured image designated by a user (Takahashi, ¶ 52: “Here, the user is a person who is an image capturing person or a subject.”).
Claim 14
The same teachings and rationales in claim 1 are appliable to claim 14.
Claim 15
Examiner’s Interpretation:
Machine readable media can encompass forms of signal transmission media that falls outside of the four statutory categories of invention. MPEP 2106; citing In re Nuijten, 500 F.3d 1346, 84 USPQ2d 1495 (Fed. Cir. 2007). A claim whose BRI covers both statutory and non-statutory embodiments embraces subject matter that is not eligible for patent protection and therefore is directed to non-statutory subject matter. MPEP 2106.
Claim 15 as drafted recites A non-transitory computer readable storage medium
Because non-transitory without additional definition excludes signal media and the like, the broadest reasonable interpretation of the claimed medium in view of Applicant’s specification covers only eligible subject matter.
Claim Mapping:
The same teachings and rationales in claim 1 are appliable to claim 15.
Additional Prior Art
Additional prior art relevant to Applicant’s disclosure but not relied upon:
Sugano (US 2024/0414427) considers segmentation and extraction of objects.
PNG
media_image1.png
359
387
media_image1.png
Greyscale
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN M GRAY whose telephone number is (571)272-4582. The examiner can normally be reached on Monday through Friday, 9:00am-5:30pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached on (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN M GRAY/Primary Examiner, Art Unit 2611
Prosecution Insights