DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: means for receiving, means for generating, means for decoding, and means for displaying in claim 17.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Dependent claims not mentioned specifically above also interpreted under 35 U.S.C. 112 (f) interpretation due to the dependency of claim on which the depend.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 – 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (Publication: US 2024/0144578 A1) in view of Neal et al. (Publication: US 2024/0112394).
Regarding claim 1, Kim discloses a method comprising ([0114] – computer system includes memory stores instruction processed by the processor to perform the following: ):
receiving, by a processing device, a three-dimensional representation of an object ([0008] – receiving the 3D mesh and update the 3D mesh based on the function. [0114] – methods are executed by the processor.);
generating, by the processing device, maps based on the three-dimensional representation, the maps including encoded geometry information for the object (
[0008] - generating a texture map of a three-dimensional (3D) mesh according to encoding a texture map of a 3D mesh, generating maps based on the mesh.
[0114] – methods are executed by the processor.);
generating, by the processing device, a set of textured views of the object by decoding the encoded geometry information from the maps ( [0008] - decoding the encoded, quantized, texture map, performing rendering using the decoded texture map, and updating the texture map of the 3D mesh based on the value of a loss function.
[0114] – methods are executed by the processor.);
set of textured views of the object ([0008] - decoding the encoded, quantized, texture map, performing rendering using the decoded texture map, and updating the texture map of the 3D mesh based on the value of a loss function.).
Kim does not Neal discloses
for using a machine learning model to generate views of the object ([0032] Where the non-immersive image is a series of images, the super-resolution machine learning model may process the series of images as a group to generate a series of high-resolution images.);
displaying, by the processing device, the object in a user interface ([0062] - The immersive projection and corresponding depthmap resulting from the method 100 may then be provided to the user device 302, such as in the form of an automatically generated rendering of the immersive projection with simulated camera movement (steps 120-124), display in a VR or AR head set (steps 118, 126, and 128), display on an LED wall (step 130), or on a glasses-free holographic 3D display.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Kim with for using a machine learning model to generate views of the object; displaying, by the processing device, the object in a user interface as taught by Neal. The motivation for doing so is to provided a better “immersive” experience for the user.
Regarding claim 2, see rejection on claim 11.
Regarding claim 3, see rejection on claim 12.
Regarding claim 4, Kim in view of Neal disclose all the limitations of claim 3.
Kim discloses receiving an input specifying an editing operation related to a visual feature of the concatenated textured image (
[0064] Referring to FIG. 4 and FIG. 5, it can be seen that the empty regions in FIG. 4 are filled by referring to neighboring colors in FIG. 5. By padding the empty regions, the degree of discontinuity in the texture map may be decreased, “concatenated textured image”, and the compression efficiency may be improved.
[0061] - there is a method of padding empty regions in the texture map using a Smoothed Push-Pull (SPP) );
generating an updated concatenated textured image based on the editing operation ([0064] Referring to FIG. 4 and FIG. 5, it can be seen that the empty regions in FIG. 4 are filled by referring to neighboring colors in FIG. 5. By padding the empty regions, the degree of discontinuity in the texture map may be decreased, “concatenated textured image”, and the compression efficiency may be improved.
[0061] - there is a method of padding empty regions in the texture map using a Smoothed Push-Pull (SPP)); and
rendering the updated concatenated textured image ([0105] - performing rendering using the decoded texture map, and a learning unit 250 for updating the texture map of the 3D mesh based on the value of a loss function.).
Neal discloses receiving an input specifying an editing operation ([0063] A user may then wish to make a change to the immersive projection and corresponding depthmap. The user may then provide an additional text-generation prompt to the LLM 300. , the user can instruct the LLM 300 to remove an entity, add an entity, specify additional attributes for an entity, specify additional actions, specify additional or different thematic elements, or provide any other replacement or additional instructions to the LLM 300. The LLM 300 will receive the subsequent text-generation prompt, generate an updated text-to-image prompt based on the subsequent text-generation prompt and the state 304 of the conversation with the user and again input the text-to-image prompt to the method 100.)
generating an image based on the editing operation ([0063] A user may then wish to make a change to the immersive projection and corresponding depthmap. The user may then provide an additional text-generation prompt to the LLM 300. , the user can instruct the LLM 300 to remove an entity, add an entity, specify additional attributes for an entity, specify additional actions, specify additional or different thematic elements, or provide any other replacement or additional instructions to the LLM 300. The LLM 300 will receive the subsequent text-generation prompt, generate an updated text-to-image prompt based on the subsequent text-generation prompt and the state 304 of the conversation with the user and again input the text-to-image prompt to the method 100.);
rendering the image in the user interface ([0062] The text-generation prompt will then be processed according to a text-to-immersive projection method 100 described above with respect to FIG. 1. The immersive projection and corresponding depthmap resulting from the method 100 may then be provided to the user device 302, such as in the form of an automatically generated rendering of the immersive projection with simulated camera movement (steps 120-124), display in a VR or AR head set (steps 118, 126, and 128), display on an LED wall (step 130), or on a glasses-free holographic 3D display.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Kim in view of Neal with receiving an input specifying an editing operation; generating an image based on the editing operation; rendering the image in the user interface as taught by Neal. The motivation for doing so is to provided a better “immersive” experience for the user.
Regarding claim 5, see rejection on claim 13.
Regarding claim 6, Kim in view of Neal disclose all the limitations of claim 1.
Kim discloses wherein a texture of the set of textured views is defined by information decoded from the maps by the model (
[0079] - the rendered image from the viewpoint v, which is acquired by sampling the decoded texture map to the texture coordinate map C.sub.v, may be represented by the Equation (4) . ).
Neal discloses defined by depth information by the machine learning model ([0044] The size of the portion 214 may vary. For example, the portion 214 may be from 1 to 200 rows or columns of pixels in one or two dimensions. A depthmap of the out-painted image may also be generated 218, either for the whole image or for the portion 214 that was out-painted.
[0039] - In a first approach, the immersive projection is out-painted, i.e., information for pixels is added using an out-painting machine learning model. The first approach has the deficiency that the out-painting machine learning model is most likely trained with rectilinear images and therefore may not accurately generate pixel values for an immersive projection. Likewise, the resolution of the out-painting machine learning model may be inadequate for some applications. However, for an out-painting machine learning model trained with immersive projections (e.g., F-theta), the first approach may be adequate.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Kim in view of Neal with defined by depth information by the machine learning model as taught by Neal. The motivation for doing so is to provided a better “immersive” experience for the user.
Regarding claim 7, see rejection on claim 15.
Regarding claim 8, see rejection on claim 16.
Regarding claim 9, Kim in view of Neal disclose all the limitations of claim 1.
Neal discloses wherein the machine learning model is a diffusion model ([0030] The method 100 may include processing 104 the text prompt with a text-to-image artificial image model to obtain a non-immersive image, such as DALL-E 1 or DALLE-2, Midjourney, Stable Diffusion or other machine learning model. It is known that Stable diffusion is a machine learning model.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Kim in view of Neal with wherein the machine learning model is a diffusion model as taught by Neal. The motivation for doing so is to provided a better “immersive” experience for the user.
Regarding claim 10, Kim discloses a non-transitory computer-readable storage medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising ([0114] – computer system includes memory stores instruction processed by the processor to perform the following:):
receiving a three-dimensional representation of an object ([0008] – receiving the 3D mesh and update the 3D mesh based on the function. [0114] – methods are executed by the processor.);
generating maps that include encoded information related to features of the three-dimensional representation ([0008] - generating a texture map of a three-dimensional (3D) mesh according to encoding a texture map of a 3D mesh, generating maps based on the mesh.
[0114] – methods are executed by the processor.);
generating a set of textured views of the object having the three-dimensional representation by decoding the encoded information from the maps using a model ([0008] - decoding the encoded, quantized, texture map, performing rendering using the decoded texture map, and updating the texture map of the 3D mesh based on the value of a loss function.
[0114] – methods are executed by the processor.);
set of textured views of the object ([0008] - decoding the encoded, quantized, texture map, performing rendering using the decoded texture map, and updating the texture map of the 3D mesh based on the value of a loss function.).
Kim does not Neal discloses
the object having a level of resolution that is higher than a level of resolution of the three-dimensional using a diffusion model (
[0030], [0032] Where the non-immersive image, using Stable Diffusion mode, is a series of images, the super-resolution machine learning model may process the series of images as a group to obtain a series of high-resolution images.
[0023] - Super resolution models produce a higher-resolution version of an image given a lower resolution version. );
and displaying the object in a user interface ([0062] - The immersive projection and corresponding depthmap resulting from the method 100 may then be provided to the user device 302, such as in the form of an automatically generated rendering of the immersive projection with simulated camera movement (steps 120-124), display in a VR or AR head set (steps 118, 126, and 128), display on an LED wall (step 130), or on a glasses-free holographic 3D display.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Kim with the object having a level of resolution that is higher than a level of resolution of the three-dimensional using a diffusion model; and displaying the object in a user interface as taught by Neal. The motivation for doing so is to provided a better “immersive” experience for the user.
Regarding claim 11, Kim in view of Neal disclose all the limitations of claim 10.
Kim discloses encoded information for individual pixels of the three-dimensional representation of the object ( Fig 7, image encoder. [0077] The differentiable mesh renderer in FIG. 7 calculates texture coordinates (UV coordinates) corresponding to the image in a viewing direction for each pixel. Also, the differentiable mesh renderer performs rendering by sampling the coordinates stored in the pixels using an interpolation function, such as bilinear or nearest interpolation, in the texture map.
[0008] – receiving the 3D mesh and update the 3D mesh based on the function. [0114] – methods are executed by the processor.).
Neal discloses information specifies depths ([0023] - Monoscopic depth estimation models produce an estimated depthmap given a single input image (e.g., MiDaS).)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Kim in view Neal with information specifies depths as taught by Neal. The motivation for doing so is to provided a better “immersive” experience for the user.
Regarding claim 12, Kim in view of Neal disclose all the limitations of claim 10.
Kim discloses combining the set of textured views into a concatenated textured image and generating content between textured views of the concatenated textured image ([0064] Referring to FIG. 4 and FIG. 5, it can be seen that the empty regions in FIG. 4 are filled by referring to neighboring colors in FIG. 5. By padding the empty regions, the degree of discontinuity in the texture map may be decreased, “concatenated textured image”, and the compression efficiency may be improved.
[0061] - there is a method of padding empty regions in the texture map using a Smoothed Push-Pull (SPP) algorithm.).
Neal discloses generating for in-painting gaps ([0023] -In-painting and out-painting models fill in missing parts of an image, while leaving other parts alone based on a mask of what is to be painted (e.g., Stable Diffusion, various generative adversarial networks (GANs)).).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Kim in view Neal with generating for in-painting gaps as taught by Neal. The motivation for doing so is to provided a better “immersive” experience for the user.
Regarding claim 13, Kim in view of Neal disclose all the limitations of claim 10.
Kim discloses wherein the maps include at least one of a depth map, a normal map, or a position map ([0028] FIG. 6 is an example of coordinate transformation between multi-view images, a UV texture map, and a rendered image, “position map”).
Regarding claim 14, Kim in view of Neal disclose all the limitations of claim 10.
Kim discloses one texture for the set of textured views ( [0002], [0105] - encoding unit 210 for encoding a texture map of a 3D mesh, for improving the compression performance of a texture map of a three-dimensional (3D) mesh reconstructed from multi-view images.).
Regarding claim 15, Kim in view of Neal disclose all the limitations of claim 10.
Kim discloses generating a grid mesh of the object by based on the encoded information ([0105] Referring to FIG. 19, an apparatus for generating a texture map of a 3D mesh, an encoding unit 210 for encoding a texture map of a 3D mesh, a quantization unit 220 for quantizing the encoded texture map, a decoding unit 230 for decoding the quantized texture map, a rendering unit 240 for performing rendering using the decoded texture map, and a learning unit 250 for updating the texture map of the 3D mesh based on the value of a loss function.).
Neal discloses by calculating warping for portions of the object ([0040] In a second approach, the high-resolution non-immersive image is out-painted first to obtain an extended image that is then warped to obtain the immersive projection and corresponding depth map with the extended image occupying substantially all, e.g., from 90 to 100 percent, of the immersive image. In a third approach the non-immersive image is out-painted to obtain an extended image, the extended image is processed with the super resolution machine learning model to obtain a high-resolution extended image, and the high-resolution extended image is then warped to obtain the immersive projection and corresponding depth map with the extended image occupying substantially all, e.g., from 90 to 100 percent, of the immersive image.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Kim in view Neal with by calculating warping for portions of the object as taught by Neal. The motivation for doing so is to provided a better “immersive” experience for the user.
Regarding claim 16, Kim in view of Neal disclose all the limitations of claim 15.
Neal discloses further comprising projecting pixels onto a view of the views based on the warping ([0035] Generating the immersive projection may result in warping of the high-resolution non-immersive image in correspondence with warping induced by a specified lens or by a particular media format. This warping may be implemented by building a lookup table which specifies, for each pixel in the output projection, which pixel in the input image (the high-resolution non-immersive image) to sample (possibly with non-integer coordinates). The output projection may then be obtained by sampling pixels from the input projection GPU. The pixel coordinates to sample can be derived from the equations for a lens projection and/or media projection.
[0038] - the immersive projection may be a “partial” projection in the sense that the projection of the high-resolution non-immersive image will only occupy part of the field of view of the immersive projection. Specifically, the rectilinear image will not have 180 degree or 360 degree field of view (FOV) sufficient to cover half or all of a sphere. Instead, the high-resolution non-immersive image will have an FOV that is less than 180.) .
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Kim in view Neal with comprising projecting pixels onto a view of the views based on the warping by calculating warping for portions of the object as taught by Neal. The motivation for doing so is to provided a better “immersive” experience for the user.
Regarding claim 17, Kim discloses a system comprising ([0114] – computer system includes memory stores instruction processed by the processor to perform the following: ):
means for receiving a mesh that is a three-dimensional representation of an object ([0008] – receiving the 3D mesh and update the 3D mesh based on the function. [0114] – methods are executed by the processor.);
means for generating maps based on the mesh, the maps including encoded geometry information for the object (
[0008] - generating a texture map of a three-dimensional (3D) mesh according to encoding a texture map of a 3D mesh, generating maps based on the mesh.
[0114] – methods are executed by the processor.);
means for decoding the encoded geometry information from the maps ( [0008] - decoding the encoded, quantized, texture map, performing rendering using the decoded texture map, and updating the texture map of the 3D mesh based on the value of a loss function.
[0114] – methods are executed by the processor.).
Kim does not Neal discloses
means for using a machine learning model to generate views of the object ([0032] Where the non-immersive image is a series of images, the super-resolution machine learning model may process the series of images as a group to generate a series of high-resolution images.
[0067] – processor that performs the method above.);
and means for displaying the set of textured views of the object in a user interface ([0062] - The immersive projection and corresponding depthmap resulting from the method 100 may then be provided to the user device 302, such as in the form of an automatically generated rendering of the immersive projection with simulated camera movement (steps 120-124), display in a VR or AR head set (steps 118, 126, and 128), display on an LED wall (step 130), or on a glasses-free holographic 3D display.
[0067] – processor that performs the method above.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Kim with means for using a machine learning model to generate views of the object; and means for displaying the set of textured views of the object in a user interface as taught by Neal. The motivation for doing so is to provided a better “immersive” experience for the user.
Regarding claim 18, see rejection on claim 11.
Regarding claim 19, see rejection on claim 12.
Regarding claim 20, see rejection on claim 13.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ming Wu whose telephone number is (571)270-0724. The examiner can normally be reached on Monday - Friday: 9:30am - 6:00pm EST .
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached on 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MING WU/
Primary Examiner, Art Unit 2618
Prosecution Insights