Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 11 objected to because of the following informalities: Claim 11 recites the limitation “3-dimensional (3D) photo data”. Parent claim 8 recites a 2D photo. Inserting an “a” or “the” in front of 3-dimensional. Appropriate correction is required.
Claim Rejection – 35 USC § 112 (b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 11 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite in that it fails to point out what is included or excluded by the claim language. This claim is an omnibus type claim. Claim 11 recites the limitation “sending”. It is unclear what is doing the sending. The claim is indefinite. In the interest of compact prosecution, examiner will interpret “sending” as sending from the client to the server.
Claim 12 recites the limitation "user global MLP weights". There is insufficient antecedent basis for this limitation in the claim. In the interest of compact prosecution, examiner will interpret “user global MLP weights” as global MLP weights.
The term “securely aggregated” in claim 12 is a relative term which renders the claim indefinite. The term “securely aggregated” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Claim Rejection – 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, and 5-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Holden (Federated Neural Radiance Fields), hereinafter referenced as Holden, in view of Martin-Brualla (NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections) hereinafter referenced as Martin-Brualla, and Poorebrahim (US11991156B2) hereinafter referenced as Poorebrahim.
Regarding claim 1, Holden teaches a method comprising: receiving, at a server, from each of a plurality of user devices, user global multi-layer perceptron (MLP) weights generated from one or more images of a shared scene,
“In this paper, we consider training NeRFs in a federated manner, whereby multiple compute nodes, each having acquired a distinct set of observations of the overall scene, learn a common NeRF in parallel…Our contribution is the first federated learning algorithm for NeRF, which splits the training effort across multiple compute nodes and obviates the need to pool the images at a central node.” (Holden, Abstract); “Our experiments validate that the structure of traditional MLP-based NeRF is amenable to optimisation across multiple compute nodes using standard federated averaging techniques, where each node tunes the model using an independent set of image observations of the overall scene… To reconcile the resultant models at the different nodes, only the optimised weights need to be communicated to a central node for aggregation into an overall NeRF;” (Holden, Section 1);
PNG
media_image1.png
222
546
media_image1.png
Greyscale
Figure 1 (b)
Holden teaches of how to train NeRFs in a federated manner (reads on a method). The server receives the data from the client devices, see Figure 1 (b) (reads on receiving, at a server, from each of a plurality of user devices). Training NeRFs involves a traditional MLP-based NeRF in which only the optimised weights (reads on user global multi-layer perceptron (MLP) weights) need to be communicated to a central node for aggregation into an overall NeRF where each node tunes the model using an independent set of image observations of the overall scene (generated from one or more images of a shared scene).
aggregating the user global MLP weights using Federated Neural Radiance Fields; and
PNG
media_image1.png
222
546
media_image1.png
Greyscale
“Figure 1 (b) Federated learning: our proposed FedNeRF splits the training effort across the clients, each using only its own data on board. Only the network weights are transferred between client and server for aggregation.” (Holden, Figure 1 b); “The transfer of the individual refined weights back to the server to be aggregated into an overall model” (Holden, Section 2.3);
Holden teaches aggregating the network weights (reads on (reads aggregating the user global MLP weights) using FedNeRF, Federated Neural Radiance Fields
sending, from the server to the plurality of user devices, updated weights, wherein the updated weights comprise aggregated global MLP weights.
PNG
media_image2.png
741
720
media_image2.png
Greyscale
(Holden, Algorithm 2)
“The transfer of the individual refined weights back to the server to be aggregated into an overall model” (Holden, Section 2.3);
Holden teaches at the start of each merge round, the server sends to client (reads on sending, from the server to the plurality of user devices) the updated weights which were aggregated into an overall model (updated weights, wherein the updated weights comprise aggregated global MLP weights.).
Holden fails to teach the following: wherein the user global MLP weights are generated so as to not include personal content of a user; and aggregating the user global MLP weights using secure multi-party computation (SMPC) to further ensure exclusion of personal content.
However, Martin-Brualla teaches
wherein the user global MLP weights are generated so as to not include personal content of a user;
“Real-world landmarks are rarely captured in isolation, without moving objects or occluders around them. Tourist photos of landmarks are particularly challenging, as they often contain posing human subjects and other pedestrians.” (Martin-Brualla, Section 4); “First, we designate the color-emitting MLP (Equation (4)) used in NeRF as the “static” head of our model, and we add an additional “transient” head that emits its own color and density, where that density is allowed to vary across training images. This enables NeRF-W to re construct images containing occluders without introducing artifacts into the static scene representation.” (Martin-Brualla, Section 4.2)
Martin-Brualla teaches of excluding occluders which comprises of moving objects, posing human subjects and other pedestrians, (reads on not include personal content of a user) from the static head of the model (reads on user global MLP weights are generated. The static head has weights which are on parameter set fit jointly to every training image.) The static head of the multi-layer perceptron acts as a global head because it maps the entire 3D scene into a single, shared coordinate system).
Martin-Brualla BASE is analogous art with respect to Holden because they are from the same field of endeavor, namely neural radiance field. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Holden with the feature of Martin-Brualla to exclude occludes which comprises of moving objects, posing human subjects and other pedestrians, from the “static” head of the model. A person of ordinary skill in the art would do such in order to improve image rendering and photorealism.
Holden in view of Martin-Brualla fail to teach the following: aggregating the user global MLP weights using secure multi-party computation (SMPC) to further ensure exclusion of personal content.
But Poorebrahim does, Poorebrahim teahces
aggregating the user global MLP weights using secure multi-party computation (SMPC) to further ensure exclusion of personal content; and
“(the server) averages the encrypted models to generate the global model. In this way, the server cannot exploit sensitive data from any specific client's model. Each client can therefore receive an average of all of the client models without seeing or having any data about any other client model weights. (Poorebrahim, ¶ 19); “This setup is needed to use the SMPC PROTOCOL which uses both the SecureMult and SecureCompare algorithms” (Poorebrahim, ¶146);
Poorebrahim teaches of averaging the encrypted models (reads on aggregating the user global MLP weights) using secure multi-party computation (SMPC) to prevent the server from exploiting sensitive data from any specific client's model (reads on to further ensure exclusion of personal content).
Poorebrahim BASE is analogous art with respect to Holden in view of Martin-Brualla because they are from the same field of endeavor, namely federated learning. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Holden in view of Martin-Brualla with the feature of Poorebrahim to incorporate averaging the encrypted models using secure multi-party computation (SMPC) to prevent the server from exploiting sensitive data from any specific client's model. A person of ordinary skill in the art would do such in order to improve data security.
Regarding claim 2, Holden in view of Martin-Brualla and Poorebrahim teaches the method of claim 1, and additionally teaches the following. Holden teaches the method of claim 1, further comprising sending, from the server to the plurality of user devices, initial weights.
PNG
media_image3.png
417
433
media_image3.png
Greyscale
(Holden, Figure 2a)
Holden teaches the server sends initial weights to a fleet of quadcopters (reads on the plurality of user devices).
Regarding claim 3, Holden in view of Martin-Brualla and Poorebrahim teaches the method of claim 1, and additionally teaches the following. Holden teaches wherein the one or more images of the shared scene comprise one or more 2-dimensional (2D) images of the shared scene.
PNG
media_image4.png
507
718
media_image4.png
Greyscale
(Martin-Brualla, Figure 1);
“enabling accurate reconstructions from unstructured image collections taken from the internet. We apply our system, dubbed NeRF-W, to internet photo collections of famous landmarks, and demonstrate temporally consistent novel view renderings that are significantly closer to photorealism than the prior state of the art.” (Martin-Brualla, Abstract);
Martin-Brualla teaches a series of internet photo collections of famous landmarks (reads on one or more images of the shared scene) which comprise of unstructured image collections taken from the internet (reads on one or more 2-dimensional (2D) images of the shared scene). See Figure 1.
Regarding claim 5, Holden in view of Martin-Brualla and Poorebrahim teaches the method of claim 1, and additionally teaches the following. Martin-Brualla teaches wherein the one or more images of the shared scene comprise the personal content and global content.
“While NeRF works well on images of static subjects captured under controlled settings, it is incapable of modeling many ubiquitous, real-world phenomena in uncontrolled images, such as variable illumination or transient occluders. We introduce a series of extensions to NeRF to address these issues…We apply our system, dubbed NeRF-W, to internet photo collections of famous landmarks, and demonstrate temporally consistent novel view renderings that are significantly closer to photorealism than the prior state of the art” (Martin-Brualla, Abstract); “Real-world landmarks are rarely captured in isolation, without moving objects or occluders around them. Tourist photos of landmarks are particularly challenging, as they often contain posing human subjects and other pedestrians.” (Martin-Brualla, Section 4);
Martin-Brualla teaches of a series of images of static subjects of real-world landmarks (reads on global content) which include posing human subjects and other pedestrians (reads on personal content);
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Holden in view of Martin-Brualla and Poorebrahim with the feature of Martin-Brualla to incorporate a series of images of static subjects of real-world landmarks which include posing human subjects and other pedestrians. A person of ordinary skill in the art would do such in order to improve photorealism.
Regarding claim 6, Holden in view of Martin-Brualla and Poorebrahim teaches the method of claim 5, and additionally teaches the following. Martin-Brualla teaches wherein the global content is static content across a plurality of images.
PNG
media_image5.png
527
696
media_image5.png
Greyscale
(Martin-Brualla, Figure 2);
PNG
media_image6.png
343
1512
media_image6.png
Greyscale
(Martin-Brualla, Figure 4);
“enabling accurate reconstructions from unstructured image collections taken from the internet. We apply our system, dubbed NeRF-W, to internet photo collections of famous landmarks, and demonstrate temporally consistent novel view renderings that are significantly closer to photorealism than the prior state of the art.” (Martin-Brualla, Abstract); “In Section 4.2 we further extend this model by allowing transient objects to be jointly estimated and disentangled from a static representation of the 3D world. Figure 3 shows an overview of the proposed model architecture.” (Martin-Brualla, Section 4);
Martin-Brualla teaches of a series of images from the internet (across a plurality of images) with a static representation of the world (reads on global content is static content).
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Holden in view of Martin-Brualla and Poorebrahim with the feature of Martin-Brualla to incorporate a series of images from the internet with a static representation of the world. A person of ordinary skill in the art would do such in order to improve dataset.
Regarding claim 7, Holden in view of Martin-Brualla and Poorebrahim teaches the method of claim 1, and additionally teaches the following. Martin-Brualla teaches wherein the personal content is dynamic content across a plurality of images.
PNG
media_image5.png
527
696
media_image5.png
Greyscale
(Martin-Brualla, Figure 2);
PNG
media_image6.png
343
1512
media_image6.png
Greyscale
(Martin-Brualla, Figure 4);
“Transient objects: Real-world landmarks are rarely captured in isolation, without moving objects or occluders around them. Tourist photos of landmarks are particularly challenging, as they often contain posing human subjects and other pedestrians.” (Martin-Brualla, Section 4);
Martin-Brualla teaches of a series of internet images (reads on plurality of images) which include transient objects containing posing human subjects and other pedestrians (reads on the personal content is dynamic content).
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Holden in view of Martin-Brualla and Poorebrahim with the feature of Martin-Brualla to a series of internet images which include transient objects containing posing human subjects and other pedestrians. A person of ordinary skill in the art would do such in order to improve dataset.
Regarding claim 8, Holden in view of Martin-Brualla and Poorebrahim teaches the method of claim 1, and additionally teaches the following. Martin-Brualla teaches the method of claim 1, further comprising: taking at least one 2-dimensional (2D) photo on a first user device of the user devices;
PNG
media_image7.png
472
586
media_image7.png
Greyscale
(Holden, Figure 2b)
Holden teaches of a fleet of quadcopters each (reads on a first user device of the user devices) taking new photos (reads on taking at least one 2-dimensional (2D) photo).
processing, by the first user device, the at least one 2D photo to train associated user global MLP weights, wherein training is performed with a neural radiance field (NeRF) pipeline learns associated user global MLP weights and personal MLP weights
“we consider training NeRFs in a federated manner, whereby multiple compute nodes, each having acquired a distinct set of observations of the overall scene, learn a common NeRF in parallel.” (Holden, Abstract) “Training is performed distributed over the clients on data collected by each client” (Holden, Section 2.3); “where each node tunes the model using an independent set of image observations of the overall scene” (Holden, Section 7);
PNG
media_image8.png
1002
577
media_image8.png
Greyscale
(Holden, Algorithm 2);
Holden teaches each client trains (reads on processing, by the first user device) the data collected by each client in which each node tunes the model. In Algorithm 2 Holden teaches each client evaluates its own local environment image data sets (reads on at least one 2D photo) in which the clients receive the latest model state from the server and train the local NeRF model on their respective datasets for a specified number of training iterations (reads on train associated user global MLP weights).
sending, from the first user device to the server, the associated user global MLP weights while keeping personal MLP weights optimised.
PNG
media_image9.png
586
1082
media_image9.png
Greyscale
(Holden, Figure 1b);
“To reconcile the resultant models at the different nodes, only the optimised weights need to be communicated to a central node for aggregation into an overall NeRF” (Holden, Section 1);
Holden teaches of sending only the optimised weights (reads on the associated user global MLP weights) from the client devices the server (reads on sending, from the first user device to the server).
Martin-Brualla also teaches of:
one 2-dimensional (2D) photo
PNG
media_image10.png
833
1110
media_image10.png
Greyscale
(Martin-Brualla, Figure 2);
Martin-Brualla teaches a series of in-the-wild photography of landmarks (reads on one 2-dimensional (2D) photo)
and wherein processing separates personal content from the at least one 2D photo; and
PNG
media_image6.png
343
1512
media_image6.png
Greyscale
(Martin-Brualla, Figure 4);
“we allow our transient head to emit a field of uncertainty (much like our existing fields of color and density), which allows our model to adapt its reconstruction loss to ignore unreliable pixels and 3D locations that are likely to contain occluders.” (Martin-Brualla, Section 4.2);
Martin-Brualla teaches of a transient head which emit a field of uncertainty containing occluders (reads on processing separates personal content from the at least one 2D photo);
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Holden in view of Martin-Brualla and Poorebrahim with the feature of Martin-Brualla to incorporate a transient head which emit a field of uncertainty containing occluders. A person of ordinary skill in the art would do such in order to improve image reconstruction.
Poorebrahim further teaches the following:
sending, from the first user device to the server while keeping personal MLP weights local to the first user device.
“the secure averaging encrypts the model of each client before sending it to the server, which then (the server) averages the encrypted models to generate the global model. In this way, the server cannot exploit sensitive data from any specific client's model. Each client can therefore receive an average of all of the client models without seeing or having any data about any other client model weights.” (Poorebrahim, ¶ 19); “Every device is training the same model and the only difference is that the respective client is training on the data local to them.” (Poorebrahim, ¶ 19);
Poorebrahim teaches of a secure averaging function for clients to send data to the server (reads on sending, from the first user device to the server) in which the server cannot see or have any data about any other client model weights (reads on personal MLP weights local to the first user device).
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Holden in view of Martin-Brualla and Poorebrahim with the feature of Poorebrahim to incorporate a secure averaging function for clients to send data to the server in which the server cannot see or have any data about any other client model weights. A person of ordinary skill in the art would do such in order to improve data security.
Regarding claim 9, Holden in view of Martin-Brualla and Poorebrahim teaches the method of claim 8, and additionally teaches the following. Holden teaches the method of claim 8, further comprising: receiving, from the server at the plurality of user devices, the updated weights;
PNG
media_image8.png
1002
577
media_image8.png
Greyscale
(Holden, Algorithm 2);
Holden teaches the clients (reads on the plurality of user devices) receive the aggregated weights (reads on the updated weights) from the server back each iteration, see algorithm 2 (reads on receiving, from the server at the plurality of user devices, the updated weights).
processing, by the first user device, the at least one 2D photo to generate updated user global MLP weights using the updated weights; and
PNG
media_image7.png
472
586
media_image7.png
Greyscale
(Holden, Figure 2b)
“The sharing of NeRF weights to each client which are then refined onboard using collected data.” (Holden, Section 5); Holden teaches of onboard refinement from the received weights (reads on generate updated user global MLP weights using the updated weights) is done by each client (reads on processing, by the first user device).
sending, from the first user device to the server, the updated user global MLP weights.
PNG
media_image11.png
438
447
media_image11.png
Greyscale
(Holden, Figure 2c)
“The transfer of the individual refined weights back to the server to be aggregated into an overall model.” (Holden, Section 5);
Holden teaches each client device sends (reads on sending, from the first user device to the server) refined weights back to the server to be aggregated into an overall model (reads on the updated user global MLP weights).
Regarding claim 10, Holden in view of Martin-Brualla and Poorebrahim teaches the method of claim 9, and additionally teaches the following. Holden teaches the method of claim 9, further comprising updated user global MLP weights
PNG
media_image8.png
1002
577
media_image8.png
Greyscale
(Holden, Algorithm 2);
Holden teaches of aggregated weights (reads on the updated weights) with each iteration, see algorithm 2.
Poorebrahim further teaches:
obfuscating the updated user global MLP weights before sending the updated user global MLP weights to the server.
“the secure averaging encrypts the model of each client before sending it to the server, which then (the server) averages the encrypted models to generate the global model. In this way, the server cannot exploit sensitive data from any specific client's model. Each client can therefore receive an average of all of the client models without seeing or having any data about any other client model weights.” (Poorebrahim, ¶ 19);
Poorebrahim teaches of encrypt weights (reads on updated user global MLP weights) before sending it from client to server (reads on obfuscating the updated user global MLP weights before sending the updated user global MLP weights to the server. Obfuscating is to deliberately making something unclear, confusing, or harder to understand. By encrypting the weights before sending it from client to server, Poorebrahim ensures the weights are secure and unclear, confusing, or harder to understand. The weights of the user device which are being encrypted are the updated user global MLP weights).
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Holden in view of Martin-Brualla and Poorebrahim with the feature of Poorebrahim to incorporate encrypting weights before sending it from client to server. A person of ordinary skill in the art would do such in order to improve data security.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Holden in view of Martin-Brualla, Poorebrahim, and Pasupaleti (US20100194851A1) hereinafter referenced as Pasupaleti.
Regarding claim 4, Holden in view of Martin-Brualla and Poorebrahim teaches the method of claim 1, and additionally teaches the following. Martin-Brualla teaches wherein the one or more images of the shared scene comprise a plurality of images taken in uncontrolled settings.
“While NeRF works well on images of static subjects captured under controlled settings, it is incapable of modeling many ubiquitous, real-world phenomena in uncontrolled images, such as variable illumination or transient occluders. We introduce a series of extensions to NeRF to address these issues” (Martin-Brualla, Abstract);
Martin-Brualla teaches of images of static subject (reads on the one or more images of the shared scene) in under controlled settings comprise of variable illumination or transient occluders.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Holden in view of Martin-Brualla and Poorebrahim with the feature of Martin-Brualla to incorporate images of static subject in under controlled settings which comprise of variable illumination or transient occluders. A person of ordinary skill in the art would do such in order to improve dataset.
Holden in view of Martin-Brualla and Poorebrahim fail to teach the images taken at different angles, distances, and times.
But Pasupaleti does. Pasupaleti teaches of images taken at different angles, distances, and times
“the plurality of images is registered based on spatial relations of image data in an overlap region between the images. The spatial relations may correspond to distance and angle between a plurality of features in the first and the second images respectively” (Pasupaleti, Abstract); Image registration or alignment matches two or more images of the scene acquired at different instances of time, from different viewpoints and/or from different sensors.” (Pasupaleti ¶ 4);
Pasupaleti teaches of images taken at different instances of time, from different viewpoints and/or from different sensors which capture different distance and angle (reads on images taken at different angles, distances, and times).
Pasupaleti BASE is analogous art with respect to Holden in view of Martin-Brualla and Poorebrahim because they are from the same field of endeavor, namely image processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Holden in view of Martin-Brualla and Poorebrahim with the feature of Pasupaleti to incorporate images taken at different instances of time, from different viewpoints and/or from different sensors which capture different distance and angle. A person of ordinary skill in the art would do such in order to improve dataset and image quality.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Holden in view of Martin-Brualla, Poorebrahim, and Paris (US20230306682A1) hereinafter referenced as Paris.
Regarding claim 11, Holden in view of Martin-Brualla and Poorebrahim teaches the method of claim 8, but teaches wherein sending, from the first user device to the server, metadata and features associated with 3-dimensional (3D) photo data for camera pose estimation.
Paris teaches wherein sending, from the first user device to the server, metadata and features associated with 3-dimensional (3D) photo data for camera pose estimation.
“From the mobile device 321, the registration server 311 receives images and corresponding image metadata. For example, the image metadata can include information associated with the image such as camera pose data (i.e., position and orientation), GPS data, compass information, inertial measurement unit (IMU) data, or some combination of these and other metadata.” (Paris, ¶ 43);
Paris teaches of sending from the mobile device to the registration server (reads on sending, from the first user device to the server) image metadata associated with camera pose data (i.e., position and orientation), GPS data, compass information, inertial measurement unit (IMU) data (reads on metadata and features associated with 3-dimensional (3D) photo data for camera pose estimation. Position, orientation, GPS data, compass information, and inertial measurement unit are all metadata and features associated with 3-dimensional (3D) photo data which are used to determine the camera pose.)
Paris BASE is analogous art with respect to Holden in view of Martin-Brualla and Poorebrahim because they are from the same field of endeavor, namely image data processing. Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Holden in view of Martin-Brualla and Poorebrahim with the feature Paris to incorporate sending from the mobile device to the registration server image metadata associated with camera pose data (i.e., position and orientation), GPS data, compass information, inertial measurement unit (IMU) data. A person of ordinary skill in the art would do such in order to improve image rendering and accuracies.
Claim(s) 12-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Poorebrahim in view of Holden and Martin-Brualla.
Regarding claim 12, Poorebrahim teaches a server comprising at least one processor; and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the server to perform at least the following:
“An embodiment of this disclosure can also include a system 102. The system can include a processor and a computer-readable storage device storing instructions which, when executed by the processor, cause the processor to perform operations” (Poorebrahim, ¶ 180); “one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.” (Poorebrahim, ¶ 190); “Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as compact disk (CD) or digital versatile disk (DVD), flash memory, memory or memory devices.” (Poorebrahim, ¶ 191);
Holden further teaches:
receive, at the server, from each of a plurality of user devices, at least one global multilayer perceptron (MLP),
PNG
media_image9.png
586
1082
media_image9.png
Greyscale
(Holden, Figure 1b);
PNG
media_image12.png
266
666
media_image12.png
Greyscale
(Holden, Algorithm 2);
“To reconcile the resultant models at the different nodes, only the optimised weights need to be communicated to a central node for aggregation into an overall NeRF; see Fig. 1b.” (Holden, Section 1);
Holden teaches the server receives from the clients (reads on receive, at the server, from each of a plurality of user devices) optimised weights (reads on at least one global multilayer perceptron (MLP)).
wherein at least one global MLP includes weights used by the user device; and at least one updated user global MLP weights
PNG
media_image12.png
266
666
media_image12.png
Greyscale
(Holden, Algorithm 2);
Holden teaches in Algorithm 2 each client processes, trains, and updates its own weights with each iteration (reads on at least one global MLP includes weights used by the user device). (Each weight reads on at least one updated user global MLP weights).
combine the received global MLP to generate a securely aggregated global MLP of the shared scene;
PNG
media_image7.png
472
586
media_image7.png
Greyscale
PNG
media_image11.png
438
447
media_image11.png
Greyscale
(Holden, Figure 2b, Figure 2c);
PNG
media_image13.png
221
652
media_image13.png
Greyscale
(Holden, Algorithm 2);
“To reconcile the resultant models at the different nodes, only the optimised weights need to be communicated to a central node for aggregation into an overall NeRF;” (Holden Section 1); “The transfer of the individual refined weights back to the server to be aggregated into an overall model.” (Holden, Section 5);
Holden teaches the server receives the updated weights from each client device (reads on received global MLP) to aggerate, see Algorithm 2 (reads on generate a securely aggregated). The updated weights from each client is derived from new photos taken by each device, see Figure 2b (reads on global MLP of the shared scene).
determine updated weights based on the securely aggregated global MLP; and
PNG
media_image2.png
741
720
media_image2.png
Greyscale
(Holden, Algorithm 2)
Holden teaches for each iteration the updated weights are trained (reads on determine updated weights) on the averaged/aggregated weights (reads the securely aggregated global MLP) see algorithm 2.
send, from the server to the plurality of user devices, the updated weights, wherein the updated weights comprise global MLP weights.
PNG
media_image11.png
438
447
media_image11.png
Greyscale
(Holden, Figure 2c)
PNG
media_image2.png
741
720
media_image2.png
Greyscale
(Holden, Algorithm 2)
Holden teaches each updated aggerated/average weights (reads on the updated weights, wherein the updated weights comprise global MLP weights) are sent from the server to the client devices (reads on send, from the server to the plurality of user devices).
Holden is analogous art with respect to Poorebrahim because they are from the same field of endeavor, namely Neural Radiance Fields (NeRF). Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Poorebrahim with the feature of Holden to incorporate server receives from the clients optimised weights in which each client processes, trains, and updates its own weights with each iteration. The server receives the updated weights from each client device to aggerate. The updated weights from each client is derived from new photos taken by each device. For each iteration the updated weights are trained on the averaged/aggregated weights. Each updated aggerated/average weights are sent from the server to the client devices. A person of ordinary skill in the art would do such in order to improve data privacy and lower bandwidth usage.
Poorebrahim in view of Holden fail to disclose wherein the at least one global MLP comprises 3-dimensional (3D) data of a shared scene, and wherein the at least one global MLP includes weights used by the user device to remove personal content from a source document during generation of updated user global MLP weights;
But Martin-Brualla does. Martin-Brualla teaches
wherein the at least one global MLP comprises 3-dimensional (3D) data of a shared scene, and wherein the at least one global MLP includes weights used by the user device to remove personal content from a source document during generation of updated user global MLP weights;
“We build on Neural Radiance Fields (NeRF), which uses the weights of a multilayer perceptron to model the density and color of a scene as a function of 3D coordinates.” (Martin-Brualla, Abstract);
PNG
media_image6.png
343
1512
media_image6.png
Greyscale
(Martin-Brualla, Figure 4);
“we allow our transient head to emit a field of uncertainty (much like our existing fields of color and density), which allows our model to adapt its reconstruction loss to ignore unreliable pixels and 3D locations that are likely to contain occluders.” (Martin-Brualla, Section 4.2);
Martin-Brualla teaches of Neural Radiance Fields (NeRF), which uses the weights of a multilayer perceptron (reads on one global MLP) to model the density and color of a scene as a function of 3D coordinates (read on 3-dimensional (3D) data of a shared scene). Martin-Brualla teaches the NeRF-W separately renders the static and transient elements of the scene, in which a transient head (reads on one global MLP includes weights) emits a field of uncertainty from the set in-the-wild photographs from the Phototourism dataset containing occluders (remove personal content from a source document) for each iteration (reads on during generation of updated user global MLP weights).
Martin-Brualla is analogous art with respect to REFERENCE because they are from the same field of endeavor, namely Neural Radiance Fields (NeRF). Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Poorebrahim in view of Holden with the feature of Martin-Brualla to incorporate Neural Radiance Fields (NeRF), which uses the weights of a multilayer perceptron to model the density and color of a scene as a function of 3D coordinates. The NeRF-W separately renders the static and transient elements of the scene, in which a transient head emits a field of uncertainty from the set in-the-wild photographs from the Phototourism dataset containing occluders for each iteration. A person of ordinary skill in the art would do such in order to improve photorealism.
Regarding claim 13, Poorebrahim in view of Holden and Martin-Brualla teaches the method of claim 12, and additionally teaches the following. Martin-Brualla teaches wherein the personal content is dynamic content across a plurality of images
PNG
media_image10.png
833
1110
media_image10.png
Greyscale
(Martin-Brualla, Figure 2);
“Transient objects: Real-world landmarks are rarely captured in isolation, without moving objects or occluders around them. Tourist photos of landmarks are particularly challenging, as they often contain posing human subjects and other pedestrians.” (Martin-Brualla, Section 4);
Martin-Brualla teaches of transient objects (reads on personal content) which include posing human subjects and other pedestrians (reads dynamic content) across a series of in-the-wild photographs (reads on across a plurality of images)
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Poorebrahim in view of Holden and Martin-Brualla with the feature of Martin-Brualla to incorporate transient objects which include posing human subjects and other pedestrians across a series of in-the-wild photographs. A person of ordinary skill in the art would do such in order to improve photorealism.
Regarding claim 14, Poorebrahim teaches a user device, comprising at least one processor; and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the user device to perform at least the following:
“FIG. 15 illustrates example computer device that can be used in connection with any of the systems disclosed herein. In this example, FIG. 11 illustrates a computing system 1100 including components in electrical communication with each other using a connection 1105, such as a bus. System 1100 includes a processing unit (CPU or processor) 1110 and a system connection 1105 that couples various system components including the system memory 1115, such as read only memory (ROM) 1120 and random access memory (RAM) 1125, to the processor 1110.” (Poorebrahim, ¶ 183); “one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.” (Poorebrahim, ¶ 190);
Poorebrahim fails to teach the following: take at least one photo of a shared scene; receive initial global multi-layer perceptron (MLP) weights from a server; process the at least one photo to train user global MLP weights and personal MLP weights, wherein training separates personal content from global content from the at least one photo; send, to the server, the user global MLP weights; and receive, from the server, updated global MLP weights.
But Holden does. Holden teaches the following:
take at least one photo of a shared scene;
PNG
media_image7.png
472
586
media_image7.png
Greyscale
(Holden, Figure 2b);
Holden teaches of a fleet of quadcopters which take new pictures, see Figure 2b (reads on take at least one photo of a shared scene).
receive initial global multi-layer perceptron (MLP) weights from a server;
PNG
media_image14.png
110
653
media_image14.png
Greyscale
(Holden, Algorithm 2);
Holden teaches in the first merge round the clients will receive from the server (reads on receive from a server) the first initial data, C(m-1) where m=1 for the first merge round C(m-1) = C0 representing the first initial data (reads on initial global multi-layer perceptron (MLP) weights)
send, to the server, the user global MLP weights; and
PNG
media_image12.png
266
666
media_image12.png
Greyscale
(Holden, Algorithm 2);
Holden teaches client transits to the server the updated weights (reads on user global MLP weights).
receive, from the server, updated global MLP weights.
PNG
media_image2.png
741
720
media_image2.png
Greyscale
(Holden, Algorithm 2)
Holden teaches at the start of each merge round, the client receives from the server the updated/averaged weights (reads on receive, from the server, updated global MLP weights).
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Poorebrahim with the feature of Holden to incorporate a fleet of quadcopters clients take new pictures. In the first merge round the clients will receive from the server the initial data. The client will transit to the server the updated weights, and at the start of each merge round, and the client will receive from the server the updated/averaged weights. A person of ordinary skill in the art would do such in order to improve federate learning.
Poorebrahim in view of Holden fails to teach the following: process the at least one photo to train user global MLP weights and personal MLP weights, wherein training separates personal content from global content from the at least one photo;
But Martin-Brualla does. Martin-Brualla teaches
process the at least one photo to train user global MLP weights and personal MLP weights, wherein training separates personal content from global content from the at least one photo;
PNG
media_image6.png
343
1512
media_image6.png
Greyscale
(Martin-Brualla, Figure 4);
“we allow our transient head to emit a field of uncertainty (much like our existing fields of color and density), which allows our model to adapt its reconstruction loss to ignore unreliable pixels and 3D locations that are likely to contain occluders.” (Martin-Brualla, Section 4.2);
Martin-Brualla the NeRF-W separately renders the static and transient elements of the scene (reads on process the at least one photo to train user global MLP weights and personal MLP weights), in which training minimizes the difference between the composite and the true image weighted by uncertainty which is simultaneously optimized to identify and discount anomalous image regions (reads on training separates personal content from global content from the at least one photo).
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Poorebrahim in view of Holden with the feature of Martin-Brualla to incorporate a NeRF-W separately rendering a static and transient elements of the scene in which training minimizes the difference between the composite and the true image weighted by uncertainty which is simultaneously optimized to identify and discount anomalous image regions. A person of ordinary skill in the art would do such in order to improve photorealism.
Claim 15 is rejected using the same rationale or bases as applied to claim 3.
Regarding claim 16, Poorebrahim in view of Holden and Martin-Brualla teaches the method of claim 14, and additionally teaches the following. Martin-Brualla teaches wherein the at least one photo of the shared scene comprises a plurality of images, and the personal content is dynamic content across the plurality of images.
PNG
media_image10.png
833
1110
media_image10.png
Greyscale
(Martin-Brualla, Figure 2);
“Transient objects: Real-world landmarks are rarely captured in isolation, without moving objects or occluders around them. Tourist photos of landmarks are particularly challenging, as they often contain posing human subjects and other pedestrians.” (Martin-Brualla, Section 4);
Martin-Brualla teaches a series of in-the-wild photography of landmarks (reads on wherein the at least one photo of the shared scene comprises a plurality of images) which comprises of transient objects (reads on personal content) wherein transient objects comprises of posing human subjects and other pedestrians (reads dynamic content) across a series of in-the-wild photographs (reads on across a plurality of images), see Figure 2.
Before the effective filling date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to modify Poorebrahim in view of Holden with the feature of Martin-Brualla to incorporate teaches a series of in-the-wild photography of landmarks which comprises of transient objects wherein transient objects comprises of posing human subjects and other pedestrians across a series of in-the-wild photographs. A person of ordinary skill in the art would do such in order to improve photorealism.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUNE NGUYEN whose telephone number is (571)272-8919. The examiner can normally be reached M-TH 7:00AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at (571) 272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DUNE NGOC NGUYEN/Examiner, Art Unit 2618 /DEVONA E FAULK/Supervisory Patent Examiner, Art Unit 2618
Prosecution Insights