Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 10/27/2025 has been entered.
Response to Arguments
Regarding the rejection of claims under 35 U.S.C. 103, Applicant’s arguments are directed towards claim amendments that have not been previously examined, and for which new grounds of rejection are given below.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language (specifically, “configured to”) without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “a de-identification unit configured to encode a first image represented by a vector of n-th dimensions into a second image of predetermined p-th dimensions, and then decode the second image into a third image of q-th dimensions, wherein the third image corresponds to the de-identified image of the first image,” and “a training unit configured to input the third image to the neural network and extract an object information included in the third image to train at least one parameter information used for a computation in the neural network” in claims 6-8.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-9 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation “and wherein, during service operation, the received first image is de-identified to generate the second image, the first image is deleted, and only the second image is stored in a database, and a plurality of the stored second images are bulk-decoded to generate the third image, the third image being used for training parameter information of the neural network.”
The plain reading of this limitation is that, as second images are stored, first images are deleted; second images are used to generate third images, which are later used to train the neural network; and therefore, at the time of training, the first images have been deleted. However, the training procedure recited by the claim requires not only the third image, but also a corresponding first image (“inputting the third image to the neural network and extracting object information included in the third image; and training at least one parameter information used for a computation in the neural network by using an error of the extracted object information compared with a target value of the first image”). Thus, the claim appears to be deleting data before it is used. Therefore, a person having ordinary skill in the art would be unable to determine a definite meaning for the limitation and thereby determine the metes and bounds of the claim.
Claim 6 recites the same limitations references above, and therefore is indefinite by the same reasoning. Claims 2-5 and 7-9 depend either on claim 1 or claim 6 and are therefore indefinite by the same reasoning.
In further examination below, the limitation “and wherein, during service operation, the received first image is de-identified to generate the second image, the first image is deleted, and only the second image is stored in a database, and a plurality of the stored second images are bulk-decoded to generate the third image, the third image being used for training parameter information of the neural network” will be read as stating that the first image is deleted at some point after the second image is generated.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5-6, and 8-9 rejected under 35 U.S.C. 103 over Kim, US Pre-Grant Publication No. 2020/0034520 (hereafter Kim) in view of Baluja et al., “Task-specific color spaces and compression for machine-based object recognition", 2019, Technical Disclosure Commons, https://www.tdcommons.org/dpubs_series/2067 (hereafter Baluja) and Long et al., US Pre-Grant Publication No. 2019/0138748 (hereafter Long).
Regarding claim 1:
Kim teaches:
“An iterative neural network training method using a de- identified image, performed by a neural network training server, the method comprising”: Kim, paragraph 0008, “Accordingly, the inventors of the present disclosure propose a method for generating obfuscated data such that the obfuscated data is different from the original data while an output result of inputting the original data into a machine learning model and an output result of inputting the obfuscated data into the learning model are same or similar to each other [training method using a de-identified image]”; Kim, paragraph 0068, “And, the machine learning network may include at least one of a k-Nearest Neighbors, a Linear Regression, a Logistic Regression, a Support Vector Machine (SVM), a Decision Tree and Random Forest, a Neural Network [neural network], a Clustering, a Visualization and a Dimensionality Reduction, an Association Rule Learning, a Deep Belief Network, a Reinforcement Learning, and a Deep learning algorithm, but the machine learning network is not limited thereto and may include various learning algorithms”; Kim, paragraph 0086, “And, the learning device 100, while increasing an integer k from 2 to n, may repeat the processes above up to the n-th learning network Fn, to thereby acquire an n-th obfuscation network On [iterative neural network training method].”
“during each iteration of the iterative neural network training method: encoding a first image represented by a vector of n-th dimensions into a second image of predetermined pth-dimensions; decoding the second image into a third image of q-th dimensions, wherein the third image corresponds to the de-identified image of the first image”: Kim, paragraph 0086, “And, the learning device 100, while increasing an integer k from 2 to n, may repeat the processes above up to the n-th learning network Fn, to thereby acquire an n-th obfuscation network On [during each iteration of the iterative neural network training method]”; Kim, paragraph 0068, “Meanwhile, as one example, the obfuscation network O may include an encoder having one or more convolutional layers for applying one or more convolution operations to images as the training data x [encoding a first image represented by a vector of n-th dimensions into a second image of predetermined pth-dimensions], and a decoder having one or more deconvolutional layers for applying one or more deconvolution operations to at least one feature map outputted from the encoder [decoding the second image into a third image of q-th dimensions] and for generating the obfuscated training data x' [wherein the third image corresponds to the de-identified image of the first image], but the scope of the present disclosure is not limited thereto, and may include any learning networks having various structures capable of obfuscating the inputted training data.”
“inputting the third image to the neural network and extracting object information included in the third image”: Kim, paragraph 0079, “Next, the learning device 100 may input the obfuscated training data x' into each of the 1-st learning network F1 to the n-th learning network Fn [inputting the third image to the neural network], and allow each of the 1-st learning network Fl to the n-th learning network Fn to (i) apply its corresponding network operation to the obfuscated training data x' using respectively the 1-st learned parameters to the n-th learned parameters of the 1-st learning network F1 to the n-th learning network Fn, and thus to (ii) generate each piece of (1_1)-st characteristic information Fl(x') to (l_n)-th characteristic information Fn(x') corresponding to the obfuscated training data x' [extracting object information included in the third image]”; Kim, paragraph 0068, “And, the machine learning network may include at least one of a k-Nearest Neighbors, a Linear Regression, a Logistic Regression, a Support Vector Machine (SVM), a Decision Tree and Random Forest, a Neural Network [the neural network], a Clustering, a Visualization and a Dimensionality Reduction, an Association Rule Learning, a Deep Belief Network, a Reinforcement Learning, and a Deep learning algorithm, but the machine learning network is not limited thereto and may include various learning algorithms.”
(bold only) “and training at least one parameter information used for a computation in the neural network by using an error of the extracted object information compared with a target value of the first image by using a backpropagated error obtained by backpropagating the error of the extracted object information through the neural network”: Kim, paragraph 0069, “Next, the learning device 100 may learn [training at least one parameter information used for a computation in the neural network] the obfuscation network O such that (i) at least one 1-st error is minimized which is calculated by referring to at least part of (i-1) at least one (1_1)-st error acquired by referring to the 1-st characteristic information F(x') and the 2-nd characteristic information F(x), and (i-2) at least one (1_2)-nd error acquired by referring to at least one task specific output generated by using the 1-st characteristic information F(x') and by further referring to at least one ground truth corresponding to the task specific output [using … the error of the extracted object information], and such that (ii) at least one 2-nd error is maximized which is calculated by referring to the training data x and the obfuscated training data x'.”
“training a de-identification parameter information used for the encoding or decoding computation in a subsequent iteration”: Kim, paragraph 0069, “Next, the learning device 100 may learn the obfuscation network O such that (i) at least one 1-st error is minimized which is calculated by referring to at least part of (i-1) at least one (1_1)-st error acquired by referring to the 1-st characteristic information F(x') and the 2-nd characteristic information F(x), and (i-2) at least one (1_2)-nd error acquired by referring to at least one task specific output generated by using the 1-st characteristic information F(x') and by further referring to at least one ground truth corresponding to the task specific output, and such that (ii) at least one 2-nd error is maximized which is calculated by referring to the training data x and the obfuscated training data x'. That is, the learning device 100 may learn the obfuscation network O, such that the obfuscation network O outputs the obfuscated training data x' much different from the training data x by using the 2-nd error [training a de-identification parameter information used for the encoding or decoding computation, interpreted as altering the encoding or decoding computation used for de-identification based on computations using the de-identified or obfuscated data], and such that the obfuscation network O obfuscates the training data by using the 1-st error, in order for the learning network F to recognize the obfuscated training data x' as same or similar to the training data x, to thereby output the obfuscated training data x'”; Kim, paragraph 0086, “And, the learning device 100, while increasing an integer k from 2 to n, may repeat the processes above up to the n-th learning network Fn, to thereby acquire an n-th obfuscation network On [computation in a subsequent iteration].”
“wherein the error of the extracted object information updates the at least one parameter information in the neural network”: Kim, paragraph 0069, “Next, the learning device 100 may learn the obfuscation network O such that (i) at least one 1-st error is minimized which is calculated by referring to at least part of (i-1) at least one (1_1)-st error acquired by referring to the 1-st characteristic information F(x') and the 2-nd characteristic information F(x), and (i-2) at least one (1_2)-nd error acquired by referring to at least one task specific output generated by using the 1-st characteristic information F(x') and by further referring to at least one ground truth corresponding to the task specific output, and such that (ii) at least one 2-nd error is maximized which is calculated by referring to the training data x and the obfuscated training data x'. That is, the learning device 100 may learn the obfuscation network O, such that the obfuscation network O outputs the obfuscated training data x' much different from the training data x by using the 2-nd error [wherein the error of the extracted object information updates the at least one parameter information in the neural network], and such that the obfuscation network O obfuscates the training data by using the 1-st error, in order for the learning network F to recognize the obfuscated training data x' as same or similar to the training data x, to thereby output the obfuscated training data x'.”
(bold only) “and the backpropagated error updates the de-identification parameter information to determine a compression rate of the encoding computation”: Kim, paragraph 0069, “Next, the learning device 100 may learn the obfuscation network O such that (i) at least one 1-st error is minimized which is calculated by referring to at least part of (i-1) at least one (1_1)-st error acquired by referring to the 1-st characteristic information F(x') and the 2-nd characteristic information F(x), and (i-2) at least one (1_2)-nd error acquired by referring to at least one task specific output generated by using the 1-st characteristic information F(x') and by further referring to at least one ground truth corresponding to the task specific output, and such that (ii) at least one 2-nd error is maximized which is calculated by referring to the training data x and the obfuscated training data x'. That is, the learning device 100 may learn the obfuscation network O, such that the obfuscation network O outputs the obfuscated training data x' much different from the training data x by using the 2-nd error [and the … error updates the de-identification parameter information to determine a compression rate of the encoding computation, interpreted as altering the encoding or decoding computation used for de-identification based on computations using the de-identified or obfuscated data], and such that the obfuscation network O obfuscates the training data by using the 1-st error, in order for the learning network F to recognize the obfuscated training data x' as same or similar to the training data x, to thereby output the obfuscated training data x'.”
“and wherein, during service operation, the received first image is de-identified to generate the second image”: Kim, paragraph 0068, “Meanwhile, as one example, the obfuscation network O may include an encoder having one or more convolutional layers for applying one or more convolution operations to images as the training data x [and wherein, during service operation, the received first image is de-identified to generate the second image], and a decoder having one or more deconvolutional layers for applying one or more deconvolution operations to at least one feature map outputted from the encoder and for generating the obfuscated training data x', but the scope of the present disclosure is not limited thereto, and may include any learning networks having various structures capable of obfuscating the inputted training data.”
“and a plurality of the stored second images are bulk-decoded to generate the third image”: Kim, paragraph 0068, “Meanwhile, as one example, the obfuscation network O may include an encoder having one or more convolutional layers for applying one or more convolution operations to images as the training data x, and a decoder having one or more deconvolutional layers for applying one or more deconvolution operations to at least one feature map outputted from the encoder [a plurality of the stored second images are bulk-decoded to generate the third image, bulk-decoded interpreted as processed without user interactions] and for generating the obfuscated training data x', but the scope of the present disclosure is not limited thereto, and may include any learning networks having various structures capable of obfuscating the inputted training data.”
“the third image being used for training parameter information of the neural network”: Kim, paragraph 0079, “Next, the learning device 100 may input the obfuscated training data x' into each of the 1-st learning network F1 to the n-th learning network Fn [the third image being used for training parameter information of the neural network], and allow each of the 1-st learning network Fl to the n-th learning network Fn to (i) apply its corresponding network operation to the obfuscated training data x' using respectively the 1-st learned parameters to the n-th learned parameters of the 1-st learning network F1 to the n-th learning network Fn, and thus to (ii) generate each piece of (1_1)-st characteristic information Fl(x') to (l_n)-th characteristic information Fn(x') corresponding to the obfuscated training data x'.”
Kim does not explicitly teach:
(bold only) “training a de-identification parameter information used for the encoding or decoding computation in a subsequent iteration by using a backpropagated error obtained by backpropagating the error of the extracted object information through the neural network”
(bold only) “and the backpropagated error updates the de-identification parameter information to determine a compression rate of the encoding computation”
“the first image is deleted, and only the second image is stored in a database”
Baluja teaches (bold only) “training a de-identification parameter information used for the encoding or decoding computation in a subsequent iteration by using a backpropagated error obtained by backpropagating the error of the extracted object information through the neural network” and
(bold only) “and the backpropagated error updates the de-identification parameter information to determine a compression rate of the encoding computation”: Baluja, page 6, paragraph 2 and following,
“Fig. 2 illustrates the training of task-specific image compression networks, per techniques of this disclosure. An encoder (202) and a decoder (204) are trained together, guided by an error signal emanating from the classifier or other task-specific unit (206). As mentioned earlier, minimization of distortion, e.g., using L1 or L2 metrics, between the original and reconstructed images is not a criterion for optimizing the networks. Rather, it is the ability of the classifier to correctly classify the image that serves as an error signal to train the networks. The neural network on a mobile device is relatively lightweight, and is a small fraction of the computation required for machine recognition. The task of machine recognition is performed by the trained network (206). Although Fig. 2 illustrates the encoder and decoder as being of the same size, there is no requirement for their dimensions to be similar. The classifier is already fully trained, e.g., there is no retraining of the classifier; rather, it is the classifier that is used to train the encoder-decoder network. Each component of the encoder-decoder network is trained simultaneously, with the classification error signal passing through the entire network, e.g., through the levels of decoder to the levels of the encoder. Based on the error (training signal) provided by the classifier, the encoder-decoder network rejects or retains to appropriate resolution selected portions and colors of the image that are relevant to image reconstruction. Approach 2: direct training of image-compression neural networks to achieve task-specific goals For mobile devices that are not compute-limited, instead of discretizing color spaces, images are compressed in a manner similar to standard compression. Image compression is performed not with the goal of faithful reconstructability, but with the goal of accurately accomplishing certain machine-based tasks, e.g., scene classification, object detection, action recognition, image segmentation, etc. The machine-based task unit, e.g., scene classifier, generates an error signal that is propagated backwards through the image compression network [training … parameter information used for the encoding or decoding computation in a subsequent iteration by using a backpropagated error obtained by backpropagating the error of the extracted object information through the neural network] [and the backpropagated error updates the de-identification parameter information to determine a compression rate of the encoding computation]. As mentioned earlier, such an error signal is not a conventional error such as a sum-of-squares error, L1-error, etc.”
Baluja and Kim are analogous arts as they are both related to the image compression models. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the task-based compression of Baluja with the use of task-based obfuscation in Kim to arrive at the present invention, in order to tailor compression to the post-compression task, as stated in Baluja, Abstract, “In some recent contexts, images are generated that are not necessarily intended for human viewership. For example, such images are generated for the purposes of machine-based tasks such as action-detection, scene-recognition, etc. In such cases, compression that is driven by fidelity of the decompressed image to the original can be sub-optimal. This disclosure describes techniques to compress images based on the end use of the image. For example, if an image is used for the purposes of detecting particular objects within it, then image compression is driven by an object detector. Portions of the image that are irrelevant to detecting the sought objects are excised during compression. The result is a more efficient, task-specific, encoding of the image.”
Long teaches “the first image is deleted, and only the second image is stored in a database”: Long, paragraph 0053, “At 406, the personally identifiable data is removed, such as by a configured edge processor that abstracts or redacts the personally identifiable data. For example, and continuing with the people counting application, the images are analyzed to determine a number of people in the images and a value representing the number of people in the public space is generated for transmission instead of the image, thereby abstracting the personally identifiable data”; Long, paragraph 0054, “At 408, the device transmits the data to an external location, such as a cloud computing system or cloud storage system remote from the device. As should be appreciated, the data that is communicated outside of the device no longer contains the personally identifiable data [only the second image is stored in a database]. That is, the device communicates abstracted or redacted data to the cloud service. In one example, only the data having the personally identifiable aspects removed is maintained in a memory of the device. For example, the original data, such as images of faces, is deleted after being processed [the first image is deleted] or after the personally identifiable aspects have been removed and that data transmitted outside the device. In some examples, after the processed data that is representative of non-personally identifiable data has been transmitted from the device, that data is also deleted from the device.”
Long and Kim are analogous arts as they are both related to de-identifying data. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the original data deletion of Long with the teachings of Kim to arrive at the present invention, in order to reduce privacy and security risk, as stated in Long, paragraph 0018, “For example, in an IoT device that communicates with other IoT devices or data collection and/or processing systems that store and/or process data in a data storage cloud or server remote from the IoT devices, personally identifiable data is removed before transmission of that data from the IoT device. This results in mitigating the security risk of transmitting the personally identifiable data.”
Regarding claim 5:
Kim as modified by Baluja and Long teaches the method of claim 1.
Kim further teaches “wherein the decoding comprises decoding the second image to have a data value different from that of the first image when decoding the second image in the q-th dimensions“: Kim, paragraph 0008, “Accordingly, the inventors of the present disclosure propose a method for generating obfuscated data such that the obfuscated data is different from the original data [decoding the second image to have a data value different from that of the first image] while an output result of inputting the original data into a machine learning model and an output result of inputting the obfuscated data into the learning model are same or similar to each other.”
Regarding claim 6:
Kim teaches:
“An iterative neural network training server using a de- identified image, the server comprising”: Kim, paragraph 0014, “In accordance with one aspect of the present disclosure, there is provided a method for learning an obfuscation network used for concealing original data to protect personal information, including steps of: (a) a learning device, if training data is acquired, inputting the training data into an obfuscation network and instructing the obfuscation network to obfuscate the training data and thus to generate obfuscated training data; (b) the learning device (i) inputting the obfuscated training data into a learning network having its own one or more learned parameters [training server using a de-identified image], and allowing the learning network to (i-1) apply a network operation to the obfuscated training data using the learned parameters and thus to (i-2) generate I-st characteristic information corresponding to the obfuscated training data”; Kim, paragraph 0068, “And, the machine learning network may include at least one of a k-Nearest Neighbors, a Linear Regression, a Logistic Regression, a Support Vector Machine (SVM), a Decision Tree and Random Forest, a Neural Network [neural network], a Clustering, a Visualization and a Dimensionality Reduction, an Association Rule Learning, a Deep Belief Network, a Reinforcement Learning, and a Deep learning algorithm, but the machine learning network is not limited thereto and may include various learning algorithms”; Kim, paragraph 0086, “And, the learning device 100, while increasing an integer k from 2 to n, may repeat the processes above up to the n-th learning network Fn, to thereby acquire an n-th obfuscation network On [iterative neural network training].”
“during each iteration of the iterative neural network training method: a de-identification unit configured to encode a first image represented by a vector of n-th dimensions into a second image of predetermined p-th dimensions, and then decode the second image into a third image of q-th dimensions, wherein the third image corresponds to the de- identified image of the first image”: Kim, paragraph 0086, “And, the learning device 100, while increasing an integer k from 2 to n, may repeat the processes above up to the n-th learning network Fn, to thereby acquire an n-th obfuscation network On [during each iteration of the iterative neural network training method]”; Kim, paragraph 0068, “Meanwhile, as one example, the obfuscation network O may include an encoder having one or more convolutional layers for applying one or more convolution operations to images as the training data x [encode a first image represented by a vector of n-th dimensions into a second image of predetermined p-th dimensions], and a decoder having one or more deconvolutional layers for applying one or more deconvolution operations to at least one feature map outputted from the encoder [decode the second image into a third image of q-th dimensions] and for generating the obfuscated training data x' [wherein the third image corresponds to the de- identified image of the first image], but the scope of the present disclosure is not limited thereto, and may include any learning networks having various structures capable of obfuscating the inputted training data.”
“and a training unit configured to input the third image to the neural network and extract an object information included in the third image”: Kim, paragraph 0079, “Next, the learning device 100 may input the obfuscated training data x' into each of the 1-st learning network F1 to the n-th learning network Fn [input the third image to the neural network], and allow each of the 1-st learning network Fl to the n-th learning network Fn to (i) apply its corresponding network operation to the obfuscated training data x' using respectively the 1-st learned parameters to the n-th learned parameters of the 1-st learning network F1 to the n-th learning network Fn, and thus to (ii) generate each piece of (1_1)-st characteristic information Fl(x') to (l_n)-th characteristic information Fn(x') corresponding to the obfuscated training data x' [extract an object information included in the third image]”; Kim, paragraph 0068, “And, the machine learning network may include at least one of a k-Nearest Neighbors, a Linear Regression, a Logistic Regression, a Support Vector Machine (SVM), a Decision Tree and Random Forest, a Neural Network [the neural network], a Clustering, a Visualization and a Dimensionality Reduction, an Association Rule Learning, a Deep Belief Network, a Reinforcement Learning, and a Deep learning algorithm, but the machine learning network is not limited thereto and may include various learning algorithms.”
“to train at least one parameter information used for a computation in the neural network by using an error of the extracted object information compared with a target value of the first image”: Kim, paragraph 0069, “Next, the learning device 100 may learn [train at least one parameter information used for a computation in the neural network] the obfuscation network O such that (i) at least one 1-st error is minimized which is calculated by referring to at least part of (i-1) at least one (1_1)-st error acquired by referring to the 1-st characteristic information F(x') and the 2-nd characteristic information F(x), and (i-2) at least one (1_2)-nd error acquired by referring to at least one task specific output generated by using the 1-st characteristic information F(x') and by further referring to at least one ground truth corresponding to the task specific output [by using an error of the extracted object information compared with a target value of the first image], and such that (ii) at least one 2-nd error is maximized which is calculated by referring to the training data x and the obfuscated training data x'.”
“and training a de-identification parameter information used for the encoding or decoding computation in a subsequent iteration”: Kim, paragraph 0069, “Next, the learning device 100 may learn the obfuscation network O such that (i) at least one 1-st error is minimized which is calculated by referring to at least part of (i-1) at least one (1_1)-st error acquired by referring to the 1-st characteristic information F(x') and the 2-nd characteristic information F(x), and (i-2) at least one (1_2)-nd error acquired by referring to at least one task specific output generated by using the 1-st characteristic information F(x') and by further referring to at least one ground truth corresponding to the task specific output, and such that (ii) at least one 2-nd error is maximized which is calculated by referring to the training data x and the obfuscated training data x'. That is, the learning device 100 may learn the obfuscation network O, such that the obfuscation network O outputs the obfuscated training data x' much different from the training data x by using the 2-nd error [training a de-identification parameter information used for the encoding or decoding computation in a subsequent iteration, interpreted as altering the encoding or decoding computation used for de-identification based on computations using the de-identified or obfuscated data], and such that the obfuscation network O obfuscates the training data by using the 1-st error, in order for the learning network F to recognize the obfuscated training data x' as same or similar to the training data x, to thereby output the obfuscated training data x'”; Kim, paragraph 0086, “And, the learning device 100, while increasing an integer k from 2 to n, may repeat the processes above up to the n-th learning network Fn, to thereby acquire an n-th obfuscation network On [computation in a subsequent iteration].”
(bold only) “by using a backpropagated error obtained by backpropagating the error of the extracted object information through the neural network”: Kim, paragraph 0069, “Next, the learning device 100 may learn the obfuscation network O such that (i) at least one 1-st error is minimized which is calculated by referring to at least part of (i-1) at least one (1_1)-st error acquired by referring to the 1-st characteristic information F(x') and the 2-nd characteristic information F(x), and (i-2) at least one (1_2)-nd error acquired by referring to at least one task specific output generated by using the 1-st characteristic information F(x') and by further referring to at least one ground truth corresponding to the task specific output [using … the error of the extracted object information], and such that (ii) at least one 2-nd error is maximized which is calculated by referring to the training data x and the obfuscated training data x'.”
“wherein the error of the extracted object information updates the at least one parameter information in the neural network”: Kim, paragraph 0069, “Next, the learning device 100 may learn the obfuscation network O such that (i) at least one 1-st error is minimized which is calculated by referring to at least part of (i-1) at least one (1_1)-st error acquired by referring to the 1-st characteristic information F(x') and the 2-nd characteristic information F(x), and (i-2) at least one (1_2)-nd error acquired by referring to at least one task specific output generated by using the 1-st characteristic information F(x') and by further referring to at least one ground truth corresponding to the task specific output, and such that (ii) at least one 2-nd error is maximized which is calculated by referring to the training data x and the obfuscated training data x'. That is, the learning device 100 may learn the obfuscation network O, such that the obfuscation network O outputs the obfuscated training data x' much different from the training data x by using the 2-nd error [wherein the error of the extracted object information updates the at least one parameter information in the neural network], and such that the obfuscation network O obfuscates the training data by using the 1-st error, in order for the learning network F to recognize the obfuscated training data x' as same or similar to the training data x, to thereby output the obfuscated training data x'.”
(bold only) “and the backpropagated error updates the de-identification parameter information to determine a compression rate of the encoding computation”: Kim, paragraph 0069, “Next, the learning device 100 may learn the obfuscation network O such that (i) at least one 1-st error is minimized which is calculated by referring to at least part of (i-1) at least one (1_1)-st error acquired by referring to the 1-st characteristic information F(x') and the 2-nd characteristic information F(x), and (i-2) at least one (1_2)-nd error acquired by referring to at least one task specific output generated by using the 1-st characteristic information F(x') and by further referring to at least one ground truth corresponding to the task specific output, and such that (ii) at least one 2-nd error is maximized which is calculated by referring to the training data x and the obfuscated training data x'. That is, the learning device 100 may learn the obfuscation network O, such that the obfuscation network O outputs the obfuscated training data x' much different from the training data x by using the 2-nd error [and the … error updates the de-identification parameter information to determine a compression rate of the encoding computation, interpreted as altering the encoding or decoding computation used for de-identification based on computations using the de-identified or obfuscated data], and such that the obfuscation network O obfuscates the training data by using the 1-st error, in order for the learning network F to recognize the obfuscated training data x' as same or similar to the training data x, to thereby output the obfuscated training data x'.”
“and wherein, during service operation, the received first image is de-identified to generate the second image”: Kim, paragraph 0068, “Meanwhile, as one example, the obfuscation network O may include an encoder having one or more convolutional layers for applying one or more convolution operations to images as the training data x [and wherein, during service operation, the received first image is de-identified to generate the second image], and a decoder having one or more deconvolutional layers for applying one or more deconvolution operations to at least one feature map outputted from the encoder and for generating the obfuscated training data x', but the scope of the present disclosure is not limited thereto, and may include any learning networks having various structures capable of obfuscating the inputted training data.”
“and a plurality of the stored second images are bulk-decoded to generate the third image”: Kim, paragraph 0068, “Meanwhile, as one example, the obfuscation network O may include an encoder having one or more convolutional layers for applying one or more convolution operations to images as the training data x, and a decoder having one or more deconvolutional layers for applying one or more deconvolution operations to at least one feature map outputted from the encoder [a plurality of the stored second images are bulk-decoded to generate the third image, bulk-decoded interpreted as processed without user interactions] and for generating the obfuscated training data x', but the scope of the present disclosure is not limited thereto, and may include any learning networks having various structures capable of obfuscating the inputted training data.”
“the third image being used for training parameter information of the neural network”: Kim, paragraph 0079, “Next, the learning device 100 may input the obfuscated training data x' into each of the 1-st learning network F1 to the n-th learning network Fn [the third image being used for training parameter information of the neural network], and allow each of the 1-st learning network Fl to the n-th learning network Fn to (i) apply its corresponding network operation to the obfuscated training data x' using respectively the 1-st learned parameters to the n-th learned parameters of the 1-st learning network F1 to the n-th learning network Fn, and thus to (ii) generate each piece of (1_1)-st characteristic information Fl(x') to (l_n)-th characteristic information Fn(x') corresponding to the obfuscated training data x'.”
Kim does not explicitly teach:
(bold only) “training a de-identification parameter information used for the encoding or decoding computation in a subsequent iteration by using a backpropagated error obtained by backpropagating the error of the extracted object information through the neural network”
(bold only) “and the backpropagated error updates the de-identification parameter information to determine a compression rate of the encoding computation”
“the first image is deleted, and only the second image is stored in a database”
Baluja teaches (bold only) “training a de-identification parameter information used for the encoding or decoding computation in a subsequent iteration by using a backpropagated error obtained by backpropagating the error of the extracted object information through the neural network” and
(bold only) “and the backpropagated error updates the de-identification parameter information to determine a compression rate of the encoding computation”: Baluja, page 6, paragraph 2 and following,
“Fig. 2 illustrates the training of task-specific image compression networks, per techniques of this disclosure. An encoder (202) and a decoder (204) are trained together, guided by an error signal emanating from the classifier or other task-specific unit (206). As mentioned earlier, minimization of distortion, e.g., using L1 or L2 metrics, between the original and reconstructed images is not a criterion for optimizing the networks. Rather, it is the ability of the classifier to correctly classify the image that serves as an error signal to train the networks. The neural network on a mobile device is relatively lightweight, and is a small fraction of the computation required for machine recognition. The task of machine recognition is performed by the trained network (206). Although Fig. 2 illustrates the encoder and decoder as being of the same size, there is no requirement for their dimensions to be similar. The classifier is already fully trained, e.g., there is no retraining of the classifier; rather, it is the classifier that is used to train the encoder-decoder network. Each component of the encoder-decoder network is trained simultaneously, with the classification error signal passing through the entire network, e.g., through the levels of decoder to the levels of the encoder. Based on the error (training signal) provided by the classifier, the encoder-decoder network rejects or retains to appropriate resolution selected portions and colors of the image that are relevant to image reconstruction. Approach 2: direct training of image-compression neural networks to achieve task-specific goals For mobile devices that are not compute-limited, instead of discretizing color spaces, images are compressed in a manner similar to standard compression. Image compression is performed not with the goal of faithful reconstructability, but with the goal of accurately accomplishing certain machine-based tasks, e.g., scene classification, object detection, action recognition, image segmentation, etc. The machine-based task unit, e.g., scene classifier, generates an error signal that is propagated backwards through the image compression network [training … parameter information used for the encoding or decoding computation in a subsequent iteration by using a backpropagated error obtained by backpropagating the error of the extracted object information through the neural network] [and the backpropagated error updates the de-identification parameter information to determine a compression rate of the encoding computation]. As mentioned earlier, such an error signal is not a conventional error such as a sum-of-squares error, L1-error, etc.”
Baluja and Kim are analogous arts as they are both related to the image compression models. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the task-based compression of Baluja with the use of task-based obfuscation in Kim to arrive at the present invention, in order to tailor compression to the post-compression task, as stated in Baluja, Abstract, “In some recent contexts, images are generated that are not necessarily intended for human viewership. For example, such images are generated for the purposes of machine-based tasks such as action-detection, scene-recognition, etc. In such cases, compression that is driven by fidelity of the decompressed image to the original can be sub-optimal. This disclosure describes techniques to compress images based on the end use of the image. For example, if an image is used for the purposes of detecting particular objects within it, then image compression is driven by an object detector. Portions of the image that are irrelevant to detecting the sought objects are excised during compression. The result is a more efficient, task-specific, encoding of the image.”
Long teaches “the first image is deleted, and only the second image is stored in a database”: Long, paragraph 0053, “At 406, the personally identifiable data is removed, such as by a configured edge processor that abstracts or redacts the personally identifiable data. For example, and continuing with the people counting application, the images are analyzed to determine a number of people in the images and a value representing the number of people in the public space is generated for transmission instead of the image, thereby abstracting the personally identifiable data”; Long, paragraph 0054, “At 408, the device transmits the data to an external location, such as a cloud computing system or cloud storage system remote from the device. As should be appreciated, the data that is communicated outside of the device no longer contains the personally identifiable data [only the second image is stored in a database]. That is, the device communicates abstracted or redacted data to the cloud service. In one example, only the data having the personally identifiable aspects removed is maintained in a memory of the device. For example, the original data, such as images of faces, is deleted after being processed [the first image is deleted] or after the personally identifiable aspects have been removed and that data transmitted outside the device. In some examples, after the processed data that is representative of non-personally identifiable data has been transmitted from the device, that data is also deleted from the device.”
Long and Kim are analogous arts as they are both related to de-identifying data. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the original data deletion of Long with the teachings of Kim to arrive at the present invention, in order to reduce privacy and security risk, as stated in Long, paragraph 0018, “For example, in an IoT device that communicates with other IoT devices or data collection and/or processing systems that store and/or process data in a data storage cloud or server remote from the IoT devices, personally identifiable data is removed before transmission of that data from the IoT device. This results in mitigating the security risk of transmitting the personally identifiable data.”
Regarding claim 8:
Kim as modified by Baluja and Long teaches the server of claim 6.
Kim further teaches “wherein the de-identification unit is configured to decode the second image to have a data value different from that of the first image when decoding the second image in the q-th dimensions “: Kim, paragraph 0008, “Accordingly, the inventors of the present disclosure propose a method for generating obfuscated data such that the obfuscated data is different from the original data [decode the second image to have a data value different from that of the first image] while an output result of inputting the original data into a machine learning model and an output result of inputting the obfuscated data into the learning model are same or similar to each other.”
Regarding claim 9:
Kim as modified by Baluja and Long teaches the method of claim 1.
Kim further teaches “One or more computers and a program stored in a non-transitory computer-readable recording medium to allow the one or more computers to perform operations of each method of claim 1 when executed by the one or more computers”: Kim, paragraph 0058, “Such description of the computing device does not exclude an integrated device including any combination of a processor, a memory, a medium, or any other computing components for implementing the present disclosure.”
Claims 2-3 rejected under 35 U.S.C. 103 over Kim as modified by Baluja and Long in view of Akyazi et al., “Learning-Based Image Compression using Convolutional Autoencoder and Wavelet Decomposition,” 2019, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (hereafter Akyazi).
Regarding claim 2:
Kim as modified by Baluja and Long teaches the method of claim 1.
Kim further teaches “wherein the compression rate of the encoding computation is associated with a degree of a de-identification of the de-identified image”: Kim, paragraph 0067, “Next, the learning device 100 may learn the obfuscation network O such that (i) at least one 1-st error is minimized which is calculated by referring to at least part of (i-1) at least one (1_1)-st error acquired by referring to the 1-st characteristic information F(x') and the 2-nd characteristic information F(x), and (i-2) at least one (1_2)-nd error acquired by referring to at least one task specific output generated by using the 1-st characteristic information F(x') and by further referring to at least one ground truth corresponding to the task specific output, and such that (ii) at least one 2-nd error is maximized which is calculated by referring to the training data x and the obfuscated training data x'. That is, the learning device 100 may learn the obfuscation network O, such that the obfuscation network O outputs the obfuscated training data x' much different from the training data x by using the 2-nd error, and such that the obfuscation network O obfuscates the training data by using the 1-st error [the compression rate of the encoding computation is associated with a degree of a de-identification of the de-identified image, interpreted as altering the computation in order to control a degree of obfuscation], in order for the learning network F to recognize the obfuscated training data x' as same or similar to the training data x, to thereby output the obfuscated training data x'.”
Kim as modified by Baluja and Long does not explicitly teach “wherein a size of the p-th dimensions is determined based on the compression rate of the encoding computation.”
Akyazi teaches “wherein a size of the p-th dimensions is determined based on the compression rate of the encoding computation”: Akyazi, section 3, paragraph 3, “In the analysis block, the number of outputs from each wavelet scale is 32. It is possible to attenuate the high frequency artifacts by changing the contribution of outputs from coarse to fine scales. When the outputs of the coarsest scale are doubled to 64 at the fifth convolutional layer, the high frequency artifacts become weaker, however the decoded images have increased low frequency noise. This results in lower subjective and objective quality averaged over the validation and test images, depicted on the example in Figure 4. Despite the reduced quality, such changes demonstrate how the use of wavelets can be beneficial in order to adjust the frequency characteristics of the output image. Optimization of the contribution from different scales to the latent representation can adjust the decoded images to have better subjective and objective quality [wherein a size of the p-th dimensions is determined based on the compression rate of the encoding computation].”
Akyazi and Kim are both related to the same field of endeavor (image compression). Kim teaches a method of de-identification of images that employs an encoder-decoder pair, acting as compression and decompression, where the encoder-decoder operators are learned according to the error rate from the use of the resulting de-identified image. Akyazi teaches a relationship between dimensionality and desired compression levels in the context of image compression using an encoder/decoder pair. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the relationship of compression rate to dimensionality in Akyazi to the teachings of Kim to arrive at the present invention, in order to control compression, and thus, de-identification, levels through dimensionality, as stated in Akyazi, section 3, paragraph 3, “Optimization of the contribution from different scales to the latent representation can adjust the decoded images to have better subjective and objective quality.”
Regarding claim 3:
Kim as modified by Baluja, Long and Akyazi teaches the method of claim 2.
Akyazi further teaches “wherein the n-th dimensions and the q-th dimensions have the same size“: Akyazi, Fig. 1,
PNG
media_image1.png
232
566
media_image1.png
Greyscale
[showing input image of dimensions 3xHxW returning to same dimensions after encoding/decoding, hence, wherein the n-th dimensions and the q-th dimensions have the same size ].
Akyazi and Kim are both related to the same field of endeavor (image compression). Kim teaches a method of de-identification of images for use in later processing, in which the de-identified image is intended to be as good as the original image in the later processing (Kim, paragraph 0065, “Also, the obfuscated training data x' may be recognized as data different from the training data x by a human, but may be recognized as data similar or same as the training data x by the learning network”). Akyazi explicitly teaches that the dimensions of the image after the initial compression/decompression can be of the same dimensions, which would be required to enable to substitution of one image for another as input to the same secondary model as performed in Kim. It would therefore have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the maintenance of dimensionality in Akyazi with the teachings of Kim to arrive at the present invention.
Claims 4 and 7 rejected under 35 U.S.C. 103 over Kim as modified by Baluja and Long in view of Kuo et al., US Pre-Grant Publication No. 2020/0177470 (hereafter Kuo).
Regarding claim 4:
Kim as modified by Baluja and Long teaches the method of claim 1.
Long further teaches “further comprising storing the second image encoded in the predetermined p-th dimensions”: Long, paragraph 0054, “At 408, the device transmits the data to an external location, such as a cloud computing system or cloud storage system remote from the device. As should be appreciated, the data that is communicated outside of the device no longer contains the personally identifiable data. That is, the device communicates abstracted or redacted data to the
cloud service. In one example, only the data having the personally identifiable aspects removed is maintained in a memory of the device [storing the second image encoded in the predetermined p-th dimensions]. For example, the original data, such as images of faces, is deleted after being processed or after the personally identifiable aspects have been removed and that data transmitted outside the device. In some examples, after the processed data that is representative of non-personally identifiable data has been transmitted from the device, that data is also deleted from the device.”
Long and Kim are combinable for the rationale given under claim 1.
Kim as modified by Baluja and Long does not explicitly teach “wherein the decoding comprises decoding the stored second image into the third image when the neural network performs the training”
Kuo teaches “wherein the decoding comprises decoding the stored second image into the third image when the neural network performs the training”: Kuo, paragraphs 0079-0080, “STEP 403: decoding the training encoded images into a plurality of training decoded images by using the training decoder of the server [decoding comprises decoding the stored second image into the third image when the neural network performs the training]; STEP 404: the artificial neural network module accepting the training decoded images and processing the training decoded images one by one by using at least one training algorithm in order to generate a plurality of training output images.”
Kuo and Kim are both related to the same field of endeavor (image processing with encoder-decoder pairs). Kim teaches using an encoder-decoder pair to de-identify images for later processing. Kuo teaches the storage of an encoder image, for later decoding and post-decoding processing. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the encoded image storage of Kuo to the teachings of Kim to arrive at the present invention, in order to avoid the overhead of encoding during the downstream processing of images, as stated in Kuo, paragraph 0026, “In a preferred embodiment, the encoded images accepted by the client device are a video e-file stored in a portable memory medium in advance and can be directly accessed by the client device to obtain the encoded images by reading the video c-file.”
Regarding claim 7:
Kim as modified by Baluja and Long teaches the server of claim 6.
Long further teaches “further comprising storing the second image encoded in the predetermined p-th dimensions”: Long, paragraph 0054, “At 408, the device transmits the data to an external location, such as a cloud computing system or cloud storage system remote from the device. As should be appreciated, the data that is communicated outside of the device no longer contains the personally identifiable data. That is, the device communicates abstracted or redacted data to the
cloud service. In one example, only the data having the personally identifiable aspects removed is maintained in a memory of the device [storing the second image encoded in the predetermined p-th dimensions]. For example, the original data, such as images of faces, is deleted after being processed or after the personally identifiable aspects have been removed and that data transmitted outside the device. In some examples, after the processed data that is representative of non-personally identifiable data has been transmitted from the device, that data is also deleted from the device.”
Long and Kim are combinable for the rationale given under claim 1.
Kim as modified by Baluja and Long does not explicitly teach: “wherein the de-identification unit is configured to decode the stored second image into the third image when the neural network performs the training”
Kuo teaches “wherein the de-identification unit is configured to decode the stored second image into the third image when the neural network performs the training”: Kuo, paragraphs 0079-0080, “STEP 403: decoding the training encoded images into a plurality of training decoded images by using the training decoder of the server [decode the stored second image into the third image when the neural network performs the training]; STEP 404: the artificial neural network module accepting the training decoded images and processing the training decoded images one by one by using at least one training algorithm in order to generate a plurality of training output images.”
Kuo and Kim are both related to the same field of endeavor (image processing with encoder-decoder pairs). Kim teaches using an encoder-decoder pair to de-identify images for later processing. Kuo teaches the storage of an encoder image, for later decoding and post-decoding processing. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date of the claimed invention to have combined the encoded image storage of Kuo to the teachings of Kim to arrive at the present invention, in order to avoid the overhead of encoding during the downstream processing of images, as stated in Kuo, paragraph 0026, “In a preferred embodiment, the encoded images accepted by the client device are a video e-file stored in a portable memory medium in advance and can be directly accessed by the client device to obtain the encoded images by reading the video c-file.”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Hanina et al., US Pre-Grant Publication No. 2018/0052971, discloses a method for de-identifying medical images while preserving the usefulness of the images in answering medical questions, in particular, the method including only the de-identified images in long-term storage.
Venkataraman et al., US Pre-Grant Publication No. 2020/0372180, discloses de-identifying raw surgical videos, and only retaining the de-identified videos for use in later training (Venkataraman, paragraph 0003, “Hence, before raw surgical videos can be used for various research purposes such as for building machine-learning tools”).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VINCENT SPRAUL whose telephone number is (703) 756-1511. The examiner can normally be reached M-F 9:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MICHAEL HUNTLEY can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VAS/ Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129