Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1-4, 7, 8, 9, are rejected under 35 U.S.C. 103 as being unpatentable over Chen (Camera and LiDAR Fusion for Urban Scene Reconstruction and Novel View Synthesis via Voxel-Based Neural Radiance Fields), in view of Yun (CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features), in view of Fisher (Logical AND/NAND) and further in view of Fukuda (US 20220318998 A1).
Regarding claims 1, 8,
Chen teaches:
An information processing apparatus comprising: at least one memory storing instructions; and at least one processor (Chen 3 “We use camera images, the corresponding camera poses, and LiDAR point clouds as the input to construct a sparse voxel set. Upon sparse voxel sampling, the voxel embedding is queried and interpolated to yield a feature representation based on eight corresponding voxel vertices. Finally, the MLP is trained and optimized based on the results of volume rendering.” Note: Chen teaches that a model is trained using a set of image data, to perform the task of training a model and inputting images as training data implicitly requires a processor and memory, thus Chen teaches a processor and memory.) configured to execute the instructions to acquire three-dimensional point cloud data (Chen Abstract “First, our approach leverages LiDAR odometry to refine frame-by-frame LiDAR point cloud alignment and derive accurate initial camera pose through joint LiDAR-camera calibration.”
PNG
media_image1.png
592
1650
media_image1.png
Greyscale
Note: Chen teaches a system that leverages 3D point cloud data captured via LiDAR as well as a corresponding image capture from a camera of the same area, examples of both can be seen in Fig. 1.) and a captured image captured by an imaging apparatus under a first imaging condition, (Chen Fig. 1, cited above, teaches a captured image via an imaging apparatus, a camera, under a first imaging condition. Note: An “imaging condition”, as defined by the specifications and claim 2 below, is a position and direction from which the image is captured, and is thus taught by Chen’s image capture.) generate a projection image under a second imaging condition being different from the first imaging condition, based on the three-dimensional point cloud data, (Chen 1 “
We can obtain accurate initial camera pose and apriori 3D point cloud models through LiDAR odometry and LiDAR-camera calibration, which can reduce the artifacts in synthesizing novel views and enhance the reconstruction quality.
We propose a novel NeRF 3D reconstruction algorithm that employs sparse voxel partitioning. By dividing space into sparse voxels and constructing a voxel octree structure, we can accelerate 3D reconstruction for urban scenes and enhance scene geometric consistency.
Experimental results on four urban outdoor datasets indicate that our method can reduce the training time and significantly improve 3D reconstruction quality compared with the latest NeRF methods”
Note: Chen teaches novel view synthesis, where 2D views, or images, of a scene/area can be generated using the 3D point cloud models and the initial camera pose. Fig. 1, cited above, shows an example input image and LiDAR point cloud data, and on the right shows novel views synthesized from different poses. As the novel views are from their own camera angles and positions, Chen teaches generating the claims “projection image” from a different imaging condition than the first.)
The claim then states a pseudo-projection image based on the projection image and captured image is generated. A “pseudo-projection image” said to be based on these two images has no inherent definition, and when defined by the specifications and claim 4 below it can be seen that multiple steps and other pieces of data must be used to make a “pseudo-projection image”. To properly show that this portion of the claim is taught:
and generate a pseudo-projection image, based on the projection image and the captured image,
The rejection of the relevant portions of claim 4 which clarify how a “pseudo-projection image” is actually formed are provided here:
generate a first projection image and a second projection image each having a different imaging condition, based on the three-dimensional point cloud data;(Chen Fig. 1, cited above, shows an example where six different projection images each from different imaging conditions are generated from a single captured image and 3D LiDAR point cloud data, thus teaching a first and second projection image with different imaging conditions.)
While Chen teaches generating multiple projection images with different imaging conditions based on the 4D point cloud data it does not teach generating a third projection image from an existing first and second projection image replacing pixels of the first with the second. This concept of generating a third image from a first two by replacing the pixels the first image with the second is taught by Yun which teaches generate a third image in which a specific ratio of pixels among pixels in the first image is replaced with pixels in the second image;(Yun 3.1
PNG
media_image2.png
746
734
media_image2.png
Greyscale
PNG
media_image3.png
636
542
media_image3.png
Greyscale
Note: The specification the generation of a third image with a specific ratio of pixels from the first replaced with pixels in the second image as ¶36 “Then, the generation unit 12 generates, for example, the projection image 503 acquired by extracting and combining ratio X of each pixel among the pixels of the projection image 501 and ratio 1-X of each pixel among the pixels of the projection image 502. As a result, the projection image 503 in which ratio X of the pixels among the pixels in the projection image 501 are replaced with the pixels in the projection image 502 is generated.” This described method, where an X ratio/group of pixels is in one image, and the ratio of pixels in the other image is 1-X are combined to produce a third image is the same approach taught by Yun. As seen in the screenshot of Yun 3.1 two images xa and xb are combined in order to produce a third image. The combining algorithm, labelled (1), teaches image xa has binary mask M applied to it and image xb has the inverse of the binary mask 1-M applied to it. Thus, every pixel excluded by mask M in xa will be filled by the pixels in xb, the same method described in the specifications ¶36.) based on a logical conjunction of each pixel value (Yun 3.1, cited above, teaches that the third image is produced from applying a binary mask to one image and the inverse of the binary mask to another. This process of applying a binary mask to filter out pixels in an image is analogous to an AND gate, thus teaching a logical conjunction on each pixel value is done.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Chen with Yun where a third image generated from replacing the pixels of a first image with a second is a third projection image generated from two projection images.
There are several reasons that would motivate one to do so, the present invention teaches generating projection images to obtain more training data, Yun teaches a similar concept where the image produced from the pixel replacement of two images is obtained to train the model to further enhance its performance. If one sought to further enhance a models performance the two present methods of artificially generating more training images could be used together to make more training data.
While Yun teaches performing a logical conjunction on pixel values of images, as Yun teaches a binary mask (analogous to an AND gate), it does not teach performing a logical conjunction between a captured image and a projection image to produce a ‘pseudo-projection image’. Performing a logical conjunction between every pixel in two images rather than between images and a mask/filter to create a third image. This concept can be found in Fisher which teaches generate the pseudo-projection image, based on a logical conjunction of each pixel value of the third image and each pixel value of the captured image. (Fisher Brief Description “The AND (and similarly the NAND) operator typically takes two binary or integer graylevel images as input, and outputs a third image whose pixel values are just those of the first image, ANDed with the corresponding pixels from the second. ”
PNG
media_image4.png
628
610
media_image4.png
Greyscale
Note: The specifications handle the logical conjunction of each pixel value between the two images as ¶37 “the generation unit 12 may use the logical conjunction of the pixel value of each coordinate in the projection image and the pixel value of each associated coordinate (for example, having the same coordinates) in the captured image, as the pixel value of each associated coordinate in the pseudo-projection image. As a result, a pseudo-projection image is generated in which the value of each pixel of the captured image is black (pixel value is 0) when the value of each associated pixel in the projection image is black (pixel value is 0).” ¶38 “In the example of Fig. 5, the generation unit 12 generates a pseudo- projection image 521, based on the projection image 503 and a captured image 511” ¶42 “image 531 having a large ratio of a white color (pixel value is 1)”.
PNG
media_image5.png
632
872
media_image5.png
Greyscale
The specifications teach the logical conjunction between two pixels is done via their “pixel value”, where the pixel value is 0 if black and 1 if white. An example of what it means to perform logical conjunctions on each pixel value is seen in Fig. 5 above, the output image 521 contains all black/nonwhite (value 0, or false) of both images, as if either input in an AND operation is false the output will be false. It is only the portions that are white (value 1, or true) in both images that are also white in the output image. This is the exact approach and method where pixels are ANDed by their greyscale pixels values is seen above in the screenshot of Fisher. Thus, Fisher teaches that a new image can be produced, the claims “pseudo-projection image”, from the logical conjunction of each pixel in two images.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Chen with Fisher where a system that generates projection images and combines them to produce more training data can further produce more images by performing a logical conjunction between the pixels of a combined image and a captured image.
There are several reasons that would motivate one to do so, it has previously been established that one may seek to combine projection images to produce more examples of data that can be used for training by taking regions of one image and swapping them in another. If even more data is desired for use in training to further enhance the model’s accuracy the already existing and useful pixel data from captured images and projection images could be used to make more images by performing a logical conjunction on the two to produce the claims ‘pseudo-projection image’.
Now that rejection of a “pseudo-projection image” has been established the final portion of claim 1 can be handled. While Chen teaches the creation of a pseudo-projection image, it does not teach that it is to be used as an explanatory variable input to a model for supervised learning where the captured image is the ground truth data. This can be found in Fukuda which teaches set learning data for performing supervised learning by using the pseudo-projection image as an explanatory variable (Fukuda ¶11 “The processor performs pseudo-projection on a three-dimensional structure with radiation at a plurality of virtual irradiation positions, using three-dimensional data indicating the three-dimensional structure, to generate a plurality of virtual projection images onto which the three-dimensional structure has been projected” ¶65 “An example of a learning phase in which the tomographic image estimation model 63 is trained by machine learning will be described with reference to FIG. 5. The tomographic image estimation model 63 is trained by the learning data 100. The learning data 100 is composed of a set of the virtual tomographic image group 98 generated from the virtual projection images 91 and of the correct answer data 102.” Note: It is not claimed that Fukuda teaches the generation of a “pseudo-projection image” as defined by the claims exactly, Fukuda simply teaches a projection image it refers to as made from “pseudo-projection” which will stand in for the claims “pseudo-projection image”. Fukuda teaches in ¶65 that during a learning phase a model will be trained on learning data, where the learning data is composed of the projection images. As Fukuda teaches the learning involves the learning data and “correct answer data”, aka ground-truth data, it is known that this is a supervised learning process.) and using the captured image as ground truth data. (Fukuda ¶65 “The learning data 100 is composed of a set of the virtual tomographic image group 98 generated from the virtual projection images 91 and of the correct answer data 102. The correct answer data 102 … is image data indicating a plurality of correct answer tomographic images corresponding to different tomographic planes.” Note: Fukuda teaches that “correct answer data”, aka ground truth data, is composed of tomographic images that have been captured. Thus, Fukuda teaches using a captured image as ground truth data.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Chen with Fukuda where the generated pseudo-projection image is used as an explanatory variable in a supervised learning process where the captured image is the ground-truth data.
There are several reasons that would motivate one to do so, supervised learning is known to train high accuracy, reliable models but requires a large amount of training data and known ground truth data. As Chen already teaches pseudo-projection images and captured images that would make good ground truth data, a model could be accurately trained by simply using the existing pieces of pseudo-projection image data as explanatory variables and the captured images as ground truth data.
Regarding claim 2,
Chen teaches:
The information processing apparatus according to claim 1, wherein a combination of a position and a direction being included in the first imaging condition is different from a combination of a position and a direction being included in the second imaging condition. (
PNG
media_image1.png
592
1650
media_image1.png
Greyscale
Note: As seen in Chen Fig. 1, a captured image from one position and direction along with LiDAR point cloud data is input to a model, and generated 2D images of the same scene with a different camera position and direction output as seen to the right of Fig. 1.)
Regarding claim 3,
Chen teaches:
The information processing apparatus according to claim 1, wherein the projection image includes a plurality of projection images, (Chen Fig. 1, cited previously, teaches multiple “projection images”, or novel views, of the scene can be output by the model.) and the at least one processor is further configured to execute the instructions to generate the plurality of projection images having different imaging conditions, based on the three-dimensional point cloud data,(
PNG
media_image1.png
592
1650
media_image1.png
Greyscale
Note: The 6 novel views, or projection images, output by Chen’s model can be seen to all be from different camera distances and positions, teaching that the plurality of projection images being output all have different imaging conditions.) and generate the pseudo- projection image, based on the plurality of projection images and the captured image.
The generation of a pseudo-projection image from a first and second projection images, i.e., a plurality of projection images, and the captured image, has already been shown to be taught in claim 1, and is here rejected under the same rationale.
Regarding claim 4,
Chen teaches:
The information processing apparatus according to claim 3, wherein the at least one processor is further configured to execute the instructions to: generate a first projection image and a second projection image each having a different imaging condition, based on the three-dimensional point cloud data;(Chen Fig. 1, cited above, shows an example where six different projection images each from different imaging conditions are generated from a single captured image and 3D LiDAR point cloud data, thus teaching a first and second projection image with different imaging conditions.)
While Chen teaches generating multiple projection images with different imaging conditions based on the 4D point cloud data it does not teach generating a third projection image from an existing first and second projection image replacing pixels of the first with the second. This concept of generating a third image from a first two by replacing the pixels the first image with the second is taught by Yun which teaches generate a third image in which a specific ratio of pixels among pixels in the first image is replaced with pixels in the second image;(Yun 3.1
PNG
media_image2.png
746
734
media_image2.png
Greyscale
PNG
media_image3.png
636
542
media_image3.png
Greyscale
Note: The specification the generation of a third image with a specific ratio of pixels from the first replaced with pixels in the second image as ¶36 “Then, the generation unit 12 generates, for example, the projection image 503 acquired by extracting and combining ratio X of each pixel among the pixels of the projection image 501 and ratio 1-X of each pixel among the pixels of the projection image 502. As a result, the projection image 503 in which ratio X of the pixels among the pixels in the projection image 501 are replaced with the pixels in the projection image 502 is generated.” This described method, where an X ratio/group of pixels is in one image, and the ratio of pixels in the other image is 1-X are combined to produce a third image is the same approach taught by Yun. As seen in the screenshot of Yun 3.1 two images xa and xb are combined in order to produce a third image. The combining algorithm, labelled (1), teaches image xa has binary mask M applied to it and image xb has the inverse of the binary mask 1-M applied to it. Thus, every pixel excluded by mask M in xa will be filled by the pixels in xb, the same method described in the specifications ¶36.) based on a logical conjunction of each pixel value (Yun 3.1, cited above, teaches that the third image is produced from applying a binary mask to one image and the inverse of the binary mask to another. This process of applying a binary mask to filter out pixels in an image is analogous to an AND gate, thus teaching a logical conjunction on each pixel value is done.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Chen with Yun where a third image generated from replacing the pixels of a first image with a second is a third projection image generated from two projection images.
There are several reasons that would motivate one to do so, the present invention teaches generating projection images to obtain more training data, Yun teaches a similar concept where the image produced from the pixel replacement of two images is obtained to train the model to further enhance its performance. If one sought to further enhance a models performance the two present methods of artificially generating more training images could be used together to make more training data.
While Yun teaches performing a logical conjunction on pixel values of images, as Yun teaches a binary mask (analogous to an AND gate), it does not teach performing a logical conjunction between a captured image and a projection image to produce a ‘pseudo-projection image’. Performing a logical conjunction between every pixel in two images rather than between images and a mask/filter to create a third image. This concept can be found in Fisher which teaches generate the pseudo-projection image, based on a logical conjunction of each pixel value of the third image and each pixel value of the captured image. (Fisher Brief Description “The AND (and similarly the NAND) operator typically takes two binary or integer graylevel images as input, and outputs a third image whose pixel values are just those of the first image, ANDed with the corresponding pixels from the second. ”
PNG
media_image4.png
628
610
media_image4.png
Greyscale
Note: The specifications handle the logical conjunction of each pixel value between the two images as ¶37 “the generation unit 12 may use the logical conjunction of the pixel value of each coordinate in the projection image and the pixel value of each associated coordinate (for example, having the same coordinates) in the captured image, as the pixel value of each associated coordinate in the pseudo-projection image. As a result, a pseudo-projection image is generated in which the value of each pixel of the captured image is black (pixel value is 0) when the value of each associated pixel in the projection image is black (pixel value is 0).” ¶38 “In the example of Fig. 5, the generation unit 12 generates a pseudo- projection image 521, based on the projection image 503 and a captured image 511” ¶42 “image 531 having a large ratio of a white color (pixel value is 1)”.
PNG
media_image5.png
632
872
media_image5.png
Greyscale
The specifications teach the logical conjunction between two pixels is done via their “pixel value”, where the pixel value is 0 if black and 1 if white. An example of what it means to perform logical conjunctions on each pixel value is seen in Fig. 5 above, the output image 521 contains all black/nonwhite (value 0, or false) of both images, as if either input in an AND operation is false the output will be false. It is only the portions that are white (value 1, or true) in both images that are also white in the output image. This is the exact approach and method where pixels are ANDed by their greyscale pixels values is seen above in the screenshot of Fisher. Thus, Fisher teaches that a new image can be produced, the claims “pseudo-projection image”, from the logical conjunction of each pixel in two images.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Chen with Fisher where a system that generates projection images and combines them to produce more training data can further produce more images by performing a logical conjunction between the pixels of a combined image and a captured image.
There are several reasons that would motivate one to do so, it has previously been established that one may seek to combine projection images to produce more examples of data that can be used for training by taking regions of one image and swapping them in another. If even more data is desired for use in training to further enhance the model’s accuracy the already existing and useful pixel data from captured images and projection images could be used to make more images by performing a logical conjunction on the two to produce the claims ‘pseudo-projection image’.
Regarding claim 7,
Chen teaches:
The information processing apparatus according to claim 1,
While Chen teaches a trained model is used it does not detail generating a trained model. This is taught by Fuduku which teaches wherein the at least one processor is further configured to execute the instructions to perform supervised learning, based on the learning data, (Fukuda ¶11 “The processor performs pseudo-projection on a three-dimensional structure with radiation at a plurality of virtual irradiation positions, using three-dimensional data indicating the three-dimensional structure, to generate a plurality of virtual projection images onto which the three-dimensional structure has been projected” ¶65 “An example of a learning phase in which the tomographic image estimation model 63 is trained by machine learning will be described with reference to FIG. 5. The tomographic image estimation model 63 is trained by the learning data 100. The learning data 100 is composed of a set of the virtual tomographic image group 98 generated from the virtual projection images 91 and of the correct answer data 102. The correct answer data 102 … is image data indicating a plurality of correct answer tomographic images corresponding to different tomographic planes.” Note: Fukuda teaches supervised learning is performed where captured tomographic images are the ground truth data, and projection images generated via pseudo projection are the training data the model learns from) and generate a trained model that infers a captured image from a projection image. ¶67 “The tomographic image estimation model 63 outputs the estimated tomographic image group 92 corresponding to the virtual tomographic image group 98. Loss calculation using a loss function is performed on the basis of the estimated tomographic image group 92 and the correct answer data 102. Then, an update of various coefficients of the tomographic image estimation model 63 is set according to the result of the loss calculation, and the tomographic image estimation model 63 whose update has been set is updated.” Note: Fukuda in ¶65, cited previously, teaches that projection images are input to the model as its learning data. This input learning data is used by the model to estimate a correct tomographic image. Fukuda uses the real captured image to evaluate how accurate the output image by the model is, the difference between the two is the model’s error/loss and is used to readjust the model to make it more accurate. This shows what is being estimated (aka predicted or inferred) by the model is a captured image from the input projection image. Thus, Fukuda teaches generating a trained model that infers a captured image from a projection image)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Chen with Fukuda where a model that uses a projection image as learning data/an explanatory variable and captured images as a ground truth data to train a model with supervised learning will train the model into one that infers captured images from projection images.
There are several reasons that would motivate one to do so, it has already been shown how Chen teaches using projection images as explanatory variables to train the model and evaluate its outputs with the captured images as ground truth data. If the model is already being trained as such, it is implicit that it will result in a trained model capable of doing the task it was trained on.
Regarding claim 9,
Chen teaches:
An information processing apparatus comprising: at least one memory storing instructions; and at least one processor (Chen 3, cited in the rejection of claim 1, teaches a processor and memory are used.) configured to execute the instructions to acquire first three-dimensional point cloud data, generate a first projection image, based on the first three- dimensional point cloud data, (Chen 1, cited in the rejection of claim 1, teaches generating a first projection image from 3D point cloud data.) and infer a second captured image, based on the first projection image, by using a trained model
Claim 7, rejected previously, has already been shown to teach inferring a captured image, making it the second captured image as the first is a real captured image by using a trained model, thus the above portion is rejected under the same rationale. generated by supervised learning using a pseudo- projection image being generated based on a first captured image captured by an imaging apparatus under a first imaging condition and a second projection image under a second imaging condition generated based on second three-dimensional point cloud data, as an explanatory variable, and using the first captured image as ground truth data.
The above-described steps of generating the trained model is contained entirely in the rejection of claim 1 and is thus rejected under the same rationale.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Chen (Camera and LiDAR Fusion for Urban Scene Reconstruction and Novel View Synthesis via Voxel-Based Neural Radiance Fields), in view of Yun (CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features), in view of Fisher (Logical AND/NAND), in view of Fukuda (US 20220318998 A1), and further in view of Cubuk (AutoAugment: Learning Augmentation Strategies from Data).
Regarding claim 5,
Chen teaches:
The information processing apparatus according to claim 1, wherein the at least one processor is further configured to execute the instructions to: generate the pseudo-projection image, based on the projection image and the captured image,
While it has been previously shown how generating the pseudo-projection image is taught, a process where the generated image used for training can alternatively be one with noise injected is not taught. This concept, where a probability outcome determines whether or not we will generate normal data for use in training or generate data with random noise is taught by Cubuk, output an image with a specific probability; and generate the pseudo-projection image, based on a random noise image and the captured image, with a remaining probability of the specific probability. (Cubuk 3 “Search space details: In our search space, a policy consists of 5 sub-policies with each sub-policy consisting of two image operations to be applied in sequence. Additionally, each operation is also associated with two hyperparameters: 1) the probability of applying the operation, and 2) the magnitude of the operation.” 3 “The operations we searched over are ShearX/Y, TranslateX/Y, Rotate, AutoContrast, Invert, Equalize, Solarize, Posterize, Contrast, Color, Brightness, Sharpness, Cutout [12], Sample Pairing [24].2 In total, we have 16 operations in our search space … The probability of applying ShearX is 0.9, and when applied, has a magnitude of 7 out of 10. We then apply Invert with probability of 0.8. The Invert operation does not use the magnitude information. We emphasize that these operations are applied in the specified order.” Note:
PNG
media_image5.png
632
872
media_image5.png
Greyscale
The specifications ¶38 define the probability as “In the example of Fig. 5, the generation unit 12 generates a pseudo- projection image 521, based on the projection image 503 and a captured image 511 with a specific probability p (p is a value in a range of 0 to 1), and generates a pseudo-projection image 522, based on a random noise image 531 and the captured image 511 with the remaining probability (1-p) of the specific probability.” The specifications clarify that probability p is the chance we will generate the image to be used in training normally and if 1-p occurs then we make an image with noise added to it; this the exact concept taught by Cubuk. Cubuk teaches a number of augmentations can be made to training images that include things such as cutting out portions of the image, shearing(cutting) out parts along the x or y axis, etc… These all fit the definition of adding noise, as noise in machine learning is the presence of something that makes the data more random, irrelevant, or inaccurate in some way. Thus, to cut out a square from an image and remove that useful data is also to “add” noise by adding an image with missing pixel. Cubuk 3 clearly teaches that the decision to add noise or not to a training image is done based on a probability p from 0 to 1, giving an example the “probability of applying ShearX is 0.9”.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Chen with Cubuk where the process of producing pseudo-projection images can alternatively produce an image with noise added as opposed the normal pseudo-projection image intended for use as training data.
There are several reasons that would motivate one to do so, while the presence of noise in data is generally a negative quality, for a model to be “robust” and able to handle real data that will have noise it is important that a model receives training data that is not perfect and has noise so it can learn how to handle it. Thus, using a probability outcome to determine whether we produce normal training data or noisy data provides a way to enhance the model’s robustness and accuracy by giving it “hard” examples to learn from.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Chen (Camera and LiDAR Fusion for Urban Scene Reconstruction and Novel View Synthesis via Voxel-Based Neural Radiance Fields), in view of Yun (CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features), in view of Fisher (Logical AND/NAND), in view of Fukuda (US 20220318998 A1), in view of Cubuk (AutoAugment: Learning Augmentation Strategies from Data), and further in view of Wei (Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learning)
Regarding claim 6,
Chen teaches:
The information processing apparatus according to claim 5,
Chen does not however teach changing the value of a probability which will determine whether or not noise is added as learning progresses. This is taught in Cubuk which teaches wherein the at least one processor is further configured to execute the instructions to change a value of the specific probability as the learning progresses. (Cubuk 3 “Search algorithm details: The search algorithm that we used in our experiment uses Reinforcement Learning, inspired by [71, 4, 72, 5]. The search algorithm has two components: a controller, which is a recurrent neural network, and the training algorithm, which is the Proximal Policy Optimization algorithm [53]. At each step, the controller predicts a decision produced by a softmax; the prediction is then fed into the next step as an embedding. In total the controller has 30 softmax predictions in order to predict 5 sub-policies, each with 2 operations, and each operation requiring an operation type, magnitude and probability.” Note: Previously it was shown how Cubuk determines whether or not to add noise to training image based on a probability. Here, Cubuk teaches a training/learning process where a model learns to predict the probability of applying an augmentation, or adding noise. During this learning/training process Cubuk teaches that a prediction is made at each step. Thus, Cubuk teaches changing the value of the specific probability as learning progresses.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Chen with Cubuk where the process of producing pseudo-projection images that can alternatively produce an image with noise added will change the probability as learning progresses.
There are several reasons that would motivate one to do so, while a model learns it changes how it processes its inputs; it can be evaluated with ground truth data to measure the current progress of the model. Depending on the development of a model it may be ready for an increased amount of more difficult/noisy examples, or alternatively it may be the case that there are too many difficult examples and its next training phase needs more clean samples. Regardless of the situation the training can be optimized to produce a more accurate final model if the probability of producing noisy training data can be changed as the model learns.
While Cubuk teaches changing the values of the specific probability that will determine whether or not we will add noise to the image it does not teach that the value is only increased as the learning progresses. To increase the probability is to increase the amount of training data which will have noise injected into it. This concept, to increase the amount of data that has noise added as learning progresses is taught by Wei which teaches wherein the at least one processor is further configured to execute the instructions to increase number of training data with noise added as the supervised learning progresses (Wei 2 “As such, we can create an artificial curriculum in the data by leveraging controlled application of data augmentation, starting by training on only original data and then adding augmented data with a higher levels of noising as training progresses” 1 “Instead of simply augmenting data, PDA explores an alternative perspective by progressively aggregating, augmenting, and injecting adversarial noises during adversarial training.”
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Chen with Wei where changing the probability that determines whether or not to add noise to an image to produce an image to be used in training as learning progresses increases the probability that noise will be added as learning progresses.
There are several reasons that would motivate one to do so, this general idea to have little to no instances of data with noise added early in learning then increase the amount as the model learns more is referred to a “curriculum learning” and has been a staple of machine learning training for years since its debut paper in 2009. If one wished to enhance the accuracy of their model by making it more robust, the known method of curriculum learning leveraged by Wei could be used by increasing the probability that an image used in training will have noise added to it.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALAN GREGORY HAKALA whose telephone number is (571)272-7863. The examiner can normally be reached 8:00am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALAN GREGORY HAKALA/Examiner, Art Unit 2617 /KING Y POON/ Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights