Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 2-3 and 5 are objected to because of the following informalities:
Regarding claim 2 (and similarly claim 3), there appears to be a discrepancy between the “another machine-learning model” defined in claim 2 and the “other machine-learning model” defined in claim 3. The Examiner recommends that consistent terminology be used for referring to the same machine learning model (e.g., the upsampling machine-learning model from claim 1 as a “first machine-learning model” and the 3D generative model in claims 2-3 as a “second machine-learning model”).
Regarding claim 5, the examiner notes that the language “wherein the first video includes multiple first videos…” is unclear as there is no clear distinction between the overarching first video and the multiple sub-videos when they both utilize the same term (i.e., “first”). The Examiner recommends that the language be modified such that there is a clear distinction the overarching “first video” and the sub-videos.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7 and 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 7, the language “between adjacent frames of the first video is sufficiently small for the machine-learning model…” is indefinite as there are no clear indications for what defines “sufficiently small”.
Regarding claim 19, the use of “or” leads to two potential interpretations of the claim and therefore is unclear what is the intended interpretation of the limitation:
Interpretation 1: wherein the object comprises a scene with multiple items OR an environment with multiple items (i.e., both the scene and environment must require multiple items).
Interpretation 2: wherein the object comprises a scene (which does not necessarily need to contain any items) OR an environment which contains multiple items.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4, 6-17, and 20 are rejected as being unpatentable over Yang et al. (“GaussianObject: Just Taking Four Images to Get A High-Quality 3D Object with Gaussian Splatting”, DOI: 10.48550/arXiv.2402.10259, Publication Year: 2024, as noted in document# 57 on IDS filed 11/25/2024; hereinafter “Yang”) in view of Xu et al. (“VideoGigaGAN: Towards Detail-rich Video Super-Resolution”, DOI: 10.48550/arXiv.2404.12388; Publication Year: 2024, as noted in document# 56 on IDS filed 11/25/2024; hereinafter “Xu”).
Regarding Claim 1, Yang discloses a method comprising (see Fig. 2):
receiving, by a processing device, a first three-dimensional (3D) representation of an object in a first resolution (3.2 Overall Framework, Yang discloses obtaining a series of N reference images captured within a 360° range to obtain a 3D representation
G
by performing 3D Gaussian splatting.);
generating, by the processing device, a first video of the object from multiple viewpoints of the first 3D representation (The Examiner notes that the 3D representation
G
is a series of images which make up a 3D video representation of the object (where the video can be seen on the GaussianObject project page (https://gaussianobject.github.io/) as well as the corresponding Youtube video for the project (https://www.youtube.com/watch?v=s5arAXdgdZQ));
outputting, by the processing device, a second 3D representation of the object in a second resolution higher than the first resolution and based on a 3D reconstruction of the object (Fig. 2, Yang discloses a Gaussian Repair Model which takes input a degraded 3D object representation and repairs (i.e., higher resolution) the 3D object representation.).
Yang does not explicitly disclose generating, using a machine-learning model and from the first video, a second video of the object in a higher resolution than the first video.
Xu discloses generating, using a machine-learning model and from the first video, a second video of the object in a higher resolution than the first video (Fig. 3, Xu discloses a video super resolution model which can up sample an input video.).
Yang and Xu are considered to be analogous to the claimed invention as they are in the same field of processing a series of image frames using a deep learning model in order to improve resolution/detail. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang by utilizing the VideoGigaGAN disclosed by Xu in order to increase the resolution of the 3D model, and consequently processing the output produced by Xu (i.e., the up-sampled video output) utilizing the methods disclosed by Yang in order to generate the final 3D repaired model. The motivation for this combination being the ability to utilize a specific GAN network which can maintain temporal consistency (i.e., consistency between different viewpoints of the object).
Claim 20 is the non-transitory computer storage medium claim corresponding to claim 1, and is similarly rejected (see 4.1. Implementation Details, specifically the usage of a GeForce RTX 3090 GPU.).
Regarding Claim 13, Yang discloses a system comprising: a memory component; and one or more processing devices coupled to the memory component, the one or more processing devices to perform operations comprising (see 4.1. Implementation Details, specifically the usage of a GeForce RTX 3090 GPU.):
receive a first three-dimensional (3D) representation of an object in a first resolution (3.2 Overall Framework, Yang discloses obtaining a series of N reference images captured within a 360° range to obtain a 3D representation
G
by performing 3D Gaussian splatting.);
generate a first video of the object from multiple viewpoints of the first 3D representation (The Examiner notes that the 3D representation
G
is a series of images which make up a 3D video representation of the object (where the video can be seen on the GausianObject project page (https://gaussianobject.github.io/) as well as the corresponding Youtube video for the project (https://www.youtube.com/watch?v=s5arAXdgdZQ));
generate a second 3D representation of the object in a second resolution higher than the first resolution by fitting Gaussian splats to the object (Fig. 2, Yang discloses a Gaussian Repair Model which takes input a degraded 3D object representation and repairs (i.e., higher resolution) the 3D object representation.).
Yang does not explicitly disclose generate, using a machine-learning model and from the first video, a second video of the object in a higher resolution than the first video.
Xu discloses generate, using a machine-learning model and from the first video, a second video of the object in a higher resolution than the first video (Fig. 3, Xu discloses a video super resolution model which can up sample an input video.).
Yang and Xu are considered to be analogous to the claimed invention as they are in the same field of processing a series of image frames using a deep learning model in order to improve resolution/detail. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang by utilizing the VideoGigaGAN disclosed by Xu in order to increase the resolution of the 3D model, and consequently processing the output produced by Xu (i.e., the up-sampled video output) utilizing the methods disclosed by Yang in order to generate the final 3D repaired model. The motivation for this combination being the ability to utilize a specific GAN network which can maintain temporal consistency (i.e., consistency between different viewpoints of the object).
Regarding Claim 2, Yang in view of Xu teaches the method of claim 1, wherein a format of the first 3D representation is one of Gaussian splats (3.2 Overall Framework, Yang discloses obtaining a series of N reference images captured within a 360° range to obtain a 3D representation
G
by performing 3D Gaussian splatting.), neural radiance fields (NeRFs), a low-poly mesh, a digital video, a sensor scan of a lidar or radar system, or a 3D object generated by another machine-learning model.
Claim 14 is the system claim corresponding to claim 2, and is similarly rejected (see 4.1. Implementation Details, specifically the usage of a GeForce RTX 3090 GPU.).
Regarding Claim 4, Yang in view of Xu teaches the method of claim 1, wherein the multiple viewpoints of the first video follow a trajectory around at least a portion of the object (3.2 Overall Framework, Yang discloses obtaining a series of N reference images captured within a 360° range to obtain a 3D representation
G
).
Regarding Claim 6, Yang in view of Xu teaches the method of claim 4, wherein the first video comprises a sequence of red-green-blue (RGB) images of the object (the Examiner notes the Github and Youtube pages referenced above, as well as the images provided in Fig. 5 which all show an input of multi-view RGB images.).
Regarding Claim 7, Yang in view of Xu teaches the method of claim 4.
The current combination of Yang in view of Xu does not explicitly teach wherein camera movement between adjacent frames of the first video is sufficiently small for the machine-learning model to assume temporal alignment in the first video.
Xu further discloses wherein camera movement between adjacent frames of the first video is sufficiently small for the machine-learning model to assume temporal alignment in the first video (4.2 Traj-DiT as Cinematographer, Xu discloses obtaining video sequences which “necessitates a learnable temporal embedding” which models sequential dependencies inherent in real-world data.).
It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the current combination of Yang in view of Xu to further include the temporal requirement established by Xu. The motivation for this combination being the ability to ensure that the input video satisfies temporal requirements which will ensure an accurate output from the machine-learning model.
Regarding Claim 8, Yang in view of Xu teaches the method of claim 1, wherein the machine-learning model is a video-based generative upsampler (Fig. 3, Xu discloses a video super resolution model which can up sample an input video.).
Claim 15 is the system claim corresponding to claim 8, and is similarly rejected (see 4.1. Implementation Details, specifically the usage of a GeForce RTX 3090 GPU.).
Regarding Claim 9, Yang in view of Xu teaches the method of claim 8, wherein the video-based generative upsampler is fine-tuned to reduce artifacts of a format of the first 3D representation (Fig. 2, Xu discloses a VideoGigaGAN model which produces video results with high-frequency details and temporal consistency while artifacts (like aliasing) are mitigated.).
Claim 16 is the system claim corresponding to claim 9, and is similarly rejected (see 4.1. Implementation Details, specifically the usage of a GeForce RTX 3090 GPU.).
Regarding Claim 10, Yang in view of Xu teaches the method of claim 8, wherein the video-based generative upsampler is trained on a dataset of video pairs, each video pair including a low-resolution video and a corresponding high-resolution video (3.6 Loss functions, 4.1 Setup, Xu discloses using low resolution input frames and high resolution frames as ground truth.).
Regarding Claim 11, Yang in view of Xu teaches the method of claim 1, wherein a format of the second 3D representation is Gaussian splats (3.1 Preliminary, Yang discloses 3D gaussian splatting, wherein a scene/objects is defined by a set of Gaussians
G
=
{
G
i
:
μ
i
,
q
i
,
s
i
,
σ
i
,
s
h
i
}
), each Gaussian splat indicating a position in space (see
μ
i
, a center location), a size value (see
s
i
, a scaling factor), an orientation (see
q
i
, a rotation quaternion), a color (see Equation 1), and an opacity value (see
σ
i
, an opacity).
Claim 17 is the system claim corresponding to claim 11, and is similarly rejected (see 4.1. Implementation Details, specifically the usage of a GeForce RTX 3090 GPU.).
Regarding Claim 12, Yang in view of Xu teaches the method of claim 11, wherein the second 3D representation is generated by fitting the Gaussian splats to the object from the second video (Fig. 2, the Examiner notes that the input and output from the work disclosed by Yang is in the form of a Gaussian splat.).
Claims 3 and 19 are rejected as being unpatentable over Yang in view of Xu in view of Li et al. (“Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text”, DOI: 10.48550/arXiv.2406.17601, Publication Year: 2024; hereinafter “Li”).
Regarding Claim 3, Yang in view of Xu teaches the method of claim 2.
Yang in view of Xu does not explicitly teach wherein the other machine-learning model is a text-to-3D generative model that generates the first 3D representation based on a text prompt.
Li discloses wherein the other machine-learning model is a text-to-3D generative model that generates the first 3D representation based on a text prompt (Fig. 1, Abstract, Li discloses a machine learning model (Director3D) which takes textual input to generate a 3D scene/object).
Yang, Xu, and Li are considered to be analogous to the claimed invention as they are in the same field of generating and processing 3D representation using machine learning models. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang in view of Xu such that it further incorporates Li’s disclosure of using the Director3D model to produce 3D representations. The motivation for this combination being the ability to increase the domain of input methods which can be used and processed.
Regarding Claim 19, Yang in view of Xu teaches the system of claim 13.
Yang in view of Xu does not explicitly teach wherein the object comprises a scene or environment with multiple items.
Li discloses wherein the object comprises a scene or environment with multiple items (Fig. 2, Li discloses generating scenes consisting of multiple objects (i.e., a marketplace with multiple types of fruits).).
Yang, Xu, and Li are considered to be analogous to the claimed invention as they are in the same field of generating and processing 3D representation using machine learning models. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang in view of Xu such that it further incorporates Li’s disclosure of using the Director3D model to produce 3D scenes consisting of multiple objects. The motivation for this combination being the ability to increase the domain of input methods which can be used and processed.
Claims 5 and 18 are rejected as being unpatentable over Yang in view of Xu in view of Kuang et al. (“Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control”, DOI: 10.48550/arXiv.2405.17414, Publication Year: 2024; hereinafter “Kuang”).
Regarding Claim 5, Yang in view of Xu teaches the method of claim 4.
Yang in view of Xu does not explicitly teach wherein the first video includes multiple first videos that each follow a respective trajectory around at least a respective portion of the object.
Kuang discloses wherein the first video includes multiple first videos that each follow a respective trajectory around at least a respective portion of the object (Fig. 2, Fig. 4, 4. Collaborative Video Diffusion with Camera Control, Kuang discloses utilizing two different camera trajectories to generate two different video sequences.).
Yang, Xu, and Kuang are considered to be analogous to the claimed invention as they are in the same field of generating and processing 3D representation using machine learning models. Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Yang in view of Xu such that it further incorporates Kuang’s methods of utilizing multiple camera trajectories to generate multiple video sequences, such that they are input into the video upsampler model taught by Yang in view of Xu. The motivation for this combination being able to provide additional data/image sequences from other camera trajectories, to improve the output from the video upsampler model.
Claim 18 is the system claim corresponding to claim 5, and is similarly rejected (see 4.1. Implementation Details, specifically the usage of a GeForce RTX 3090 GPU.).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PROMOTTO TAJRIAN ISLAM whose telephone number is (703)756-5584. The examiner can normally be reached Monday - Friday 8:30 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PROMOTTO TAJRIAN ISLAM/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights