DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-9 and 11-19 are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Zuev (US 20190294961 A1).
Regarding to claim 1, Zuev discloses a data augmentation method ([0044]: augment an image using an autoencoder; [0052]: augment the image data set; Fig. 5; [0053]: a method 500 for generating augmentation of image data; the method 500 is performed by the random augmentation engine; computer system; Fig. 11; [0067]: a computer system; Fig. 11; [0069]: processor 1102 executes instructions 1126 for performing the operations; [0071]: a computer-readable storage medium 1124 stores one or more sets of instructions 1126), comprising:
obtaining an input image (Fig. 3; [0038]: Input layer 220 receives one or more images; Fig. 5; [0054]: obtain one or more first images; Fig. 8; [0064]: receive single input image 810); and
generating a plurality of output images corresponding to the input image ([0026]: an unlimited number of randomly distorted images are derived and generated from a single image; Fig. 3; [0038]: input layer 220 generates and produces an output 226 in the format, i.e. number of images in a batch, number of channels, image height, image width;
PNG
media_image1.png
156
488
media_image1.png
Greyscale
; Fig. 5; [0057]: generate and provide a first output image; Fig. 5; [0060]: the computer system generates and provides a second output image; Fig. 5; [0061]: the computer system generates and provides a third output image; the computer system provides a third output of the third set of layers of the computational units as one or more second images; the third output is based on the one or more first images and the distortion parameters; Fig. 8; [0064]: Images 821, 822, 823, and 824 represent an augmented data set generated from a single input mage 810;
PNG
media_image2.png
712
578
media_image2.png
Greyscale
; generate Images 821, 822, 823, and 824 based a single input image 810),
wherein at least one first pixel of the input image is displaced to form one of the output images ([0025]: the input images are restored and superimposed with random distortions; [0039]: perform filtering of an input image; each fragment includes multiple pixel values; [0040]: pixel-wise displacement; [0057]: a third matrix of displacement values; Fig. 8; [0064]: Images 821, 822, 823, and 824 are modified version of input image 810 that have each been modified with a random distortion each time the input image 810 passed through autoencoder 801; the pixels of image 810 are displaced to form Images 821, 822, 823, and 824 as illustrated in Fig. 8;
PNG
media_image2.png
712
578
media_image2.png
Greyscale
), and a displacement of each of the at least one first pixel is randomized ([0024]: impose random distortions to images; [0025]: the input images are restored and superimposed with random distortions; [0041]: produce random distortion parameters to modify the image; Fig. 6; [0058]: the input 244 is processed using the features of the random convolutional layer 240 and producing an output 246; Fig. 8; [0064]: Images 821, 822, 823, and 824 are modified version of input image 810 that have each been modified with a random distortion each time the input image 810 passed through autoencoder 801.).
Regarding to claim 2, Zuev discloses the data augmentation method of claim 1, further comprising:
generating a first matrix, wherein at least one element of the first matrix is a random number (Zuev; [0025]: matrices with learnable parameters include a matrix of mean values, a matrix of standard deviation values, and a matrix of displacement values; epsilon matrix with non-learned parameters; the matrix of mean values are initialized with random values; [0046]: kernel_mean matrix 420 is a matrix of mean values; the kernel_mean matrix 420 is initialized with random values; [0048]: bias matrix 440 is a matrix of displacement values; generate the bias matrix 440 based on a number of filters to apply to the input of the random layer; [0049]: the epsilon matrix 450 is initialized with random values; [0051]: the weights or parameter sets generated for the kernel matrix 410 are random); and
determining a deformation matrix based on the first matrix ([0041]: produce random distortion parameters, i.e. deformation matrix, to modify the image; obtain the distortion parameters, i.e. deformation matrix, from the random layer; [0045]: a kernel matrix 410 represents a randomized kernel matrix; generate the kernel matrix based on one or more matrices; [0049]: epsilon matrix 450 is generated from a normal distribution with a mean value of zero and a standard deviation value of sampling_std 460; [0051]), wherein the deformation matrix comprise either at least one displacement of the at least one first pixel of the input image or at least one coordinate in the output image for the at least one first pixel (or is optional; Zuev; [0051]: represent random image distortion parameters superimposed to a representation of the input image; [0052]: generate one or more randomly distorted images from a single input image each time a representation of the input image is passed through the random convolution layer after; [0059]: the computer system obtains a distortion parameter for each of the plurality of portions of the first image; represent random image distortion parameters superimposed to a representation of the input image; Fig. 8; [0064]: Images 821, 822, 823, and 824 correspond to naturally distorted images of input 810 and are different from each other).
Regarding to claim 3, Zuev discloses the data augmentation method of claim 2, wherein the at least one element of the first matrix follows a normal distribution, a mean of the normal distribution is related to an equivalent displacement degree, and a standard deviation of the normal distribution is related to an equivalent deformation degree (Zuev; [0025]: the epsilon matrix is based on a normal distribution value and an arbitrary standard deviation value; [0049]: epsilon matrix 450 is a matrix that is based on an arbitrary standard deviation value and a normal distribution value; epsilon matrix 450 is generated from a normal distribution with a mean value of zero and a standard deviation value of sampling_std 460; [0051]: the randomized kernel matrix 410 is a normal vector with a mean value of “kernel_mean 420” and the standard deviation value of “exp (Kernel_stddev/2); [0057]: the arbitrary standard deviation value specifies the roughness, i.e. degree, of the image distortions).
Regarding to claim 4, Zuev discloses the data augmentation method of claim 2, further comprising:
transforming the first matrix or a third matrix into a second matrix using at least one filter (Zuev; [0045]: a kernel matrix 410 represents a randomized kernel matrix; generate the kernel matrix based on one or more matrices; [0051]: local transformation of the input image for each portion; [0057]: the first matrix, the second matrix, and the third matrix each may include learnable parameters),
wherein determining the deformation matrix based on the first matrix comprises determining the deformation matrix based on the second matrix (Zuev; [0040]: the bias matrix includes unique scalar values that are added to the output of each layer's filter for each pixel to add a specified offset or displacement to the output values; [0041]: produce random distortion parameters to modify the image; obtain the distortion parameters from the random layer; [0043]: the weights or parameter sets used for the one or more matrices for the random layer are different; [0045]: a kernel matrix 410 represents a randomized kernel matrix; generate the kernel matrix based on one or more matrices; [0057]: the first matrix, the second matrix, and the third matrix each may include learnable parameters).
Regarding to claim 5, Zuev discloses the data augmentation method of claim 4, wherein one of the at least one filter is a Gaussian filter, and a standard deviation or a size of the Gaussian filter is related to equivalent smoothness (or is optional; Zuev; [0025]: image filters; iterative filtering of the one or more images; a matrix of standard deviation values; an arbitrary standard deviation value; [0037]: size of filter; [0039]: apply a filter to a fragment of an input image; [0040]: the bias matrix may include unique scalar values that are added to the output of each layer's filter for each pixel to add a specified offset or displacement to the output values; [0044]: select a sampling standard deviation value; [0045]: filter width represents the size of a filter; [0051]: the randomized kernel matrix 410 is a normal vector with a mean value of “kernel_mean 420” and the standard deviation value of “exp (Kernel_stddev/2)).
Regarding to claim 6, Zuev discloses the data augmentation method of claim 2, further comprising:
vector-integrating the first matrix or a second matrix to generate a third matrix (Zuev; [0043]: the weights or parameter sets used for the one or more matrices for the random layer are different; [0045]: a kernel matrix 410 represents a randomized kernel matrix; generate the kernel matrix based on one or more matrices; [0051]: local transformation of the input image for each portion; [0057]: the first matrix, the second matrix, and the third matrix each may include learnable parameters),
wherein determining the deformation matrix based on the first matrix comprises determining the deformation matrix based on the third matrix (Zuev; [0025]: the random layer includes matrices with learnable parameters, such as a matrix of mean values, a matrix of standard deviation values, a matrix of displacement values, and an “epsilon” matrix with non-learned parameters; obtain one or more images with the superimposed random distortions as an output of the deconvolution layer; [0041]: produce random distortion parameters to modify the image; obtain the distortion parameters from the random layer; [0043]: the weights or parameter sets used for the one or more matrices for the random layer are different; [0045]: a kernel matrix 410 represents a randomized kernel matrix; generate the kernel matrix based on one or more matrices; [0057]: the first matrix, the second matrix, and the third matrix each may include learnable parameters).
Regarding to claim 7, Zuev discloses the data augmentation method of claim 1, further comprising: training a deep learning model using the input image or the output images (Zuev; [0003]: deep learning; [0021]: train a machine learning model; [0024]: effectively create useful synthetic training dataset for a machine learning model; [0025]: the one or more images with the superimposed random distortions are added to a training set of images to train a machine learning model; [0031]: The set of machine learning models 114 is a deep network, i.e., a machine learning model that is composed of multiple levels of non-linear operations).
Regarding to claim 8, Zuev discloses the data augmentation method of claim 1, wherein the at least one first pixel comprises all or part of pixels of the input image (or is optional; Zuev; [0039]: obtain a fragment by dividing an input image into a plurality of portions; apply a filter to a fragment of an input image; [0040]: perform filtering of the image by fragments, e.g., a specified portion, of the image; [0059]: obtain a distortion parameter for each of the plurality of portions of the first image).
Regarding to claim 9, Zuev discloses the data augmentation method of claim 1, further comprising:
dividing the input image into the at least one first pixel and at least one second pixel (Zuev; [0039]: obtain a fragment by dividing an input image into a plurality of portions; apply a filter to a fragment of an input image; [0051]: Determination of the distortion parameters is performed for each portion of the input image); and
performing first image processing on the at least one first pixel to form a first region image, wherein the first image processing comprises individually displacing the at least one first pixel (Zuev; [0026]: an unlimited number of randomly distorted images are derived from a single image; Fig. 3; [0038]: input layer 220 generates and produces an output 226 in the format, i.e. number of images in a batch, number of channels, image height, image width;
PNG
media_image1.png
156
488
media_image1.png
Greyscale
; [0040]: perform filtering of the image by fragments, e.g., a specified portion, of the image; [0051]: Determination of the distortion parameters is performed for each portion of the input image; Fig. 5; [0057]: generate and provide a first output image; [0059]: obtain a distortion parameter for each of the plurality of portions of the first image; Fig. 5; [0060]: the computer system generates and provides a second output image; Fig. 5; [0061]: the computer system generates and provides a third output image; the computer system provides a third output of the third set of layers of the computational units as one or more second images; the third output is based on the one or more first images and the distortion parameters; [0062]: the one or more second images; Fig. 8; [0064]: Images 821, 822, 823, and 824 represent an augmented data set generated from a single input mage 810);
wherein generating the output images corresponding to the input image comprises performing image synthesis based on the first region image and the at least one second pixel to generate one of the output images (Zuev; [0024]: effectively create useful synthetic training dataset for a machine learning model; the systems herein provide for using a single original image and superimpose nearly natural random distortions on the original image; the roughness or coarseness of distortions are applied on the original image; [0051]: the convolution performed with generated randomized kernel matrix applied to the image data on the input of the random layer may represent random image distortion parameters superimposed to a representation of the input image, i.e. performing image synthesis with image distortion parameters and input image).
Regarding to claim 11, Zuev discloses a computing device ([0044]: augment an image using an autoencoder; [0052]: augment the image data set; Fig. 5; [0053]: a method 500 for generating augmentation of image data; the method 500 is performed by the random augmentation engine; computer system; Fig. 11; [0067]: an example computer system; [0071]: a computer-readable storage medium 1124 stores one or more sets of instructions 1126), comprising:
a storage circuit, configured to store an instruction, wherein the instruction (Fig. 11; [0068]: a computer system 1100 includes a processor 1102, a main memory 1104, e.g., read-only memory (ROM) or dynamic random access memory (DRAM), and a data storage device 1118; Fig. 11; [0069]: processor 1102 executes instructions 1126 for performing the operations; [0071]: a computer-readable storage medium 1124 stores one or more sets of instructions 1126) comprises:
a processing circuit, coupled to the storage circuit and configured to execute the instruction (Fig. 11; [0069]: processor 1102 executes instructions 1126 for performing the operations; [0071]: a computer-readable storage medium 1124 stores one or more sets of instructions 1126).
the rest claim limitations are similar to claim limitations recited in claim 1. Therefore, same rational used to reject claim 1 is also used to reject claim 11.
Regarding to claim 12, Zuev discloses the computing device of claim 11, wherein the instruction further comprises:
The rest claim limitations are similar to claim limitations recited in claim 2. Therefore, same rational used to reject claim 2 is also used to reject claim 12.
Regarding to claim 13, Zuev discloses the computing device of claim 12,
The rest claim limitations are similar to claim limitations recited in claim 3. Therefore, same rational used to reject claim 3 is also used to reject claim 13.
Regarding to claim 14, Zuev discloses the computing device of claim 12, wherein the instruction further comprises:
The rest claim limitations are similar to claim limitations recited in claim 4. Therefore, same rational used to reject claim 4 is also used to reject claim 14.
Regarding to claim 15, Zuev discloses the computing device of claim 14,
The rest claim limitations are similar to claim limitations recited in claim 5. Therefore, same rational used to reject claim 5 is also used to reject claim 15.
Regarding to claim 16, Zuev discloses the computing device of claim 12, wherein the instruction further comprises:
The rest claim limitations are similar to claim limitations recited in claim 6. Therefore, same rational used to reject claim 6 is also used to reject claim 16.
Regarding to claim 17, Zuev discloses the computing device of claim 11, wherein the instruction further comprises:
The rest claim limitations are similar to claim limitations recited in claim 7. Therefore, same rational used to reject claim 7 is also used to reject claim 17.
Regarding to claim 18, Zuev discloses the computing device of claim 11,
The rest claim limitations are similar to claim limitations recited in claim 8. Therefore, same rational used to reject claim 8 is also used to reject claim 18.
Regarding to claim 19, Zuev discloses the computing device of claim 11, wherein the instruction further comprises:
The rest claim limitations are similar to claim limitations recited in claim 9. Therefore, same rational used to reject claim 9 is also used to reject claim 19.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 10 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zuev (US 20190294961 A1) and in view of Michiels (US 20230029578 A1).
Regarding to claim 10, Zuev discloses the data augmentation method of claim 9, further comprising:
performing second image processing according to the at least one second pixel to form a second region image (Zuev; [0026]: an unlimited number of randomly distorted images are derived from a single image; Fig. 3; [0038]: input layer 220 generates and produces an output 226 in the format, i.e. number of images in a batch, number of channels, image height, image width; ; [0040]: perform filtering of the image by fragments, e.g., a specified portion, of the image; Fig. 5; [0057]: generate and provide a first output image; [0059]: obtain a distortion parameter for each of the plurality of portions of the first image; Fig. 5; [0060]: the computer system generates and provides a second output image; Fig. 5; [0061]: the computer system generates and provides a third output image; the computer system provides a third output of the third set of layers of the computational units as one or more second images; the third output is based on the one or more first images and the distortion parameters; [0062]: the one or more second images; Fig. 8; [0064]: Images 821, 822, 823, and 824 represent an augmented data set generated from a single input mage 810);
wherein performing image synthesis based on the first region image and the at least one second pixel comprises combining the first region image and the second region image (Zuev; [0024]: effectively create useful synthetic training dataset for a machine learning model; the systems herein provide for using a single original image and superimpose nearly natural random distortions on the original image; the roughness or coarseness of distortions are applied on the original image),
Zuev fails to explicitly disclose:
wherein the first image processing comprises removing at least one edge pixel from the at least one first pixel after displacement to form the first region image, and the at least one edge pixel is located at one or more edge of the at least one first pixel after displacement.
In same field of endeavor, Michiels teaches:
wherein the first image processing comprises removing at least one edge pixel from the at least one first pixel after displacement to form the first region image, and the at least one edge pixel is located at one or more edge of the at least one first pixel after displacement (Fig. 16; [0057]: the pixel pattern is displaced 44 in the horizontal and/or vertical direction by a random number of pixels (D); If the translation results in the pixel pattern extending beyond the edge of the image when the pixel pattern is overlaid, the portions that overhang the edge of the image may be removed from the pixel pattern;
PNG
media_image3.png
208
304
media_image3.png
Greyscale
).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Zuev to include wherein the first image processing comprises removing at least one edge pixel from the at least one first pixel after displacement to form the first region image, and the at least one edge pixel is located at one or more edge of the at least one first pixel after displacement as taught by Michiels. The motivation for doing so would have been to train the ML model using a set of training images; to remove the portions that overhang the edge of the image from the pixel pattern as taught by Michiels in paragraphs [0017] and [0057].
Regarding to claim 20, Zuev discloses the computing device of claim 19, wherein the instruction further comprises:
The rest claim limitations are similar to claim limitations recited in claim 10. Therefore, same rational used to reject claim 10 is also used to reject claim 20.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hai Tao Sun whose telephone number is (571)272-5630. The examiner can normally be reached 9:00AM-6:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 5712727642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAI TAO SUN/Primary Examiner, Art Unit 2616
Prosecution Insights