DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
As per the instant Application having Application number 17/455,075 the examiner acknowledges the applicant's submission of the amendment dated 09/29/2025. At this point, claims 1, 8, and 10 have been amended. Claims 1-12 are pending.
Response to Arguments
Applicant’s arguments, see Applicant Arguments/Remarks Made in an Amendment, filed 09/25/2025, with respect to the 35 U.S.C. 101 rejection of claims 1-12 have been fully considered and are persuasive. The 35 U.S.C. 101 rejection of claims 1-12 has been withdrawn.
Applicant’s arguments with respect to claim(s) the 35 U.S.C. 103 rejection of claims 1-12 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, and 4 are rejected under 35 U.S.C. 103 as being unpatentable over Tremblay (US 11475613 B1, hereinafter Tremblay), in view of Nam (US 20200302286 A1, hereinafter Nam) and Hu et al., (US 2020/0168320 A1, hereinafter Hu).
Regarding claim 1, Tremblay teaches “A computer-implemented training method for a convolutional neural network” (Col. 3, lines 13-20, “In some implementations, the detected edges can also be utilized for digital image segmentation based on identifying various objects, shapes, and/or regions of interest. In some implementations, one or more digital image segmentations operations may be performed by one or more trainable models (e.g., convolutional neural networks) configured to detect certain objects, shapes, and/or regions of interest.” Tremblay utilizes convolutional neural networks when performing the following operations to identify objects, shapes, and regions of interest), comprising: receiving first data (Fig. 1, 105, shows receiving the first source digital image as the first step in the process.); performing a first stylization on the received first data (Fig. 1, 130, the source digital image is fed into the stylization module; Col. 2, lines 28-29, "the source digital image 105 is fed to the stylization module 130," Col. 2, lines 35-45, "The stylization module 130 may transform the source digital image 105 based on one or more parameters and/or sample images of the chosen visual style 115. The visual style sample images and/or visual style parameters may specify one or more textures, shapes, color palettes, and/or various other visual digital image aspects that characterize the selected visual style. Accordingly, the stylization module 130 may perform one or more digital image stylization operations, by employing trainable models (also referred to as "machine learning-based models") and/or rule-based stylization methods." Tremblay teaches that the first image is fed into the stylization module, where the first stylization occurs. Tremblay teaches that the stylization that occurs can perform any number of image transformations, such as focusing on the shape characteristic over the texture characteristic, which aligns with how stylization is defined in the present application.); receiving second data after the first stylization is performed on the first data; (Col. 2, lines 65-67, "The styled digital image (also referred to as "underpainting") 135 produced by the stylization module 130 is fed to the digital image analysis module 140" Tremblay teaches that the second data, or the underpainting, is being used as a second data in future operations.)
Tremblay fails to teach about training the convolutional neural network based on first and second data and normalization layers. However, Nam teaches and training the convolutional neural network based on the first data and the second data, wherein the convolutional neural network has a first normalization layer used for the first data, and wherein the convolutional neural network has a second normalization layer used for the second data. (Paragraph 0011, " According to an aspect of the present disclosure, there is provided a method of training a neural network comprising obtaining output data of a first layer of the neural network regarding a training sample, transforming the output data into first normalized data using a first normalization technique, transforming the output data into second normalized data using a second normalization technique, transforming the output data into second normalized data using a second normalization technique, generating third normalized data by aggregating the first normalized data and the second normalized data based on a learnable parameter and providing the third normalized data as an input to a second layer of the neural network." Nam teaches that the training of the neural network is based on the first normalization layer, which is obtained the first data, and second normalization layer, which is obtained from an aggregation of the first and second data. Thus, the convolutional neural network is trained based on the first and second normalization layer, which are gathered from the first and second data.) Tremblay and Nam are both considered to be analogous to the claimed invention because they are both in the same field of training of convolutional neural networks based on data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tremblay to incorporate the teachings of Nam. This is because Nam teaches a faster way of training a convolutional neural network by using normalization layers. Normalization layers stabilize training by keeping activation in a stable range, which in turn accelerates training by using larger learning rates without instability.
Tremblay and Nam also fail to teach “data representing an object having a texture feature and a shape feature; … the first stylization configured to make the object less characterized by the texture feature and to retain characterization of the object according to the shape feature;… and wherein the convolutional neural network is used to detect the object in further received data.”
However, Hu teaches “data representing an object having a texture feature and a shape feature; … the first stylization configured to make the object less characterized by the texture feature and to retain characterization of the object according to the shape feature;… and wherein the convolutional neural network is used to detect the object in further received data.” (Paragraph [0008]: “Additionally, or alternatively, the system may provide for improvements in autonomous imaging which lacks specific features, requires semantic labels, and/or needs nonlinear adjustments. For example, the system may normalize test data by first processing it through a customization layer (e.g., that features a generative artificial neural network prior to processing the test data through a trained discriminatory artificial neural network). In such cases, the generative artificial neural network may reconstruct portions of an inputted image with missing features, the absence of which may prevent the trained discriminatory artificial neural network from properly classifying objects in the image. In another example, the system … information determined based on the geometric artificial neural network (e.g., such as the three-dimensional dimensions of an object in the image) is used by a convolutional neural network to properly classify the object and/or identify the bounds of features of the object. In another example, the system may normalize test data by first processing it through a customization layer that applies non-linear adjustments (e.g., coloring, texture mapping, etc.) to the test data. The adjusted test-data may then be input into an artificial neural network that is trained on non-linearly adjusted data (e.g., trained on objects with the same coloring, texture mapping, etc.).”)
Tremblay, Nam, and Hu are analogous in the arts because Tremblay, Nam, and Hu all describe of convolutional neural networks based on data.
Therefore, it would be obvious to one of ordinary skill in the art at the filing date of the instant application, having the teachings of Tremblay, Nam, and Hu before him or her, to modify the teachings of Tremblay and Nam to include the teachings of Hu in order to receive a wider range of training data that needs adjustments and can support non-linear adjusted training data of objects and thereby increase the training ability of Tremblay and Nam (see Hu paragraphs [0007] and [0008]).
Regarding claim 3, Tremblay, in view of Nam and Hu, teaches the subject matter of claim 1. Tremblay further teaches the training method according to Claim 1, wherein the first data comprises at least one of image data, audio data, and text data (Fig. 1, 105, shows receiving the first source digital image as the first step in the process. Tremblay teaches that the first data is an image, thus image data.)
Regarding claim 4, Tremblay, in view of Nam and Hu, teaches the subject matter of claim 1. Nam further teaches, the training method according to Claim 1, wherein: the first normalization layer comprises a first batch normalization layer; and/or the second normalization layer comprises a second batch normalization layer (Paragraph 0040, “The neural network may be composed of a plurality of layers, and at least some of the plurality of layers may include a normalization layer which normalizes input (or output) data of the neural network,” Paragraph 0043, "As shown in FIG. 2, according to the batch normalization technique, a batch normalization layer 1 interposed between specific layers normalizes output data of a previous layer and performs an affine transformation using learnable parameters 3 and 5," Paragraph 0051, "Therefore, in order to ensure performance of the neural network, it is very important to select an appropriate normalization technique according to the relationship between a target task and style information contained in an image," Paragraph 0053, “According to various embodiments of the present disclosure, when there is a strong relationship between the target task of the neural network and style information of an image (i.e., when the style information is required to perform the target task), a normalization layer may perform normalization by mainly using the batch normalization technique,” Paragraph 0059, “In operation S120, the output data is transformed into first normalized data with statistical information of a batch to which the training samples 44 belong. In other words, the output data is transformed into the first normalized data through the batch normalization technique.” Nam teaches that the CNN will be composed of layers, which include normalization layers. Nam teaches that batch normalization should be used on the normalization layer when "style information" needs to be preserved. Style information is defined as the texture of the image. In the present application, the style information, or texture of the image should be preserved, so batch normalization should be the method used. Nam teaches that the appropriate normalization technique should be used, so for the present application, batch normalization should be used because style needs to be preserved. Therefore, the batch normalization technique performed on the first and second normalization layer would result in a first and second batch normalization layer).
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tremblay, Nam, and Hu as applied to claim 1 above, and further in view of Son (US 20180285629 A1, hereinafter Son).
Regarding claim 2, the combination of Tremblay, Nam, and Hu teach the subject matter of claim 1. However, Tremblay, Nam, and Hu do not teach the training method according to Claim 1, further comprising: receiving third data after a second stylization is performed on the first data, the second stylization different from the first stylization; and training the convolutional neural network based on the third data, wherein the convolutional neural network further has a third normalization layer used for the third data.
Son though teaches the training method according to Claim 1, further comprising: receiving third data after a second stylization is performed on the first data, the second stylization different from the first stylization; (Fig. 4, 403 is the region of interest, Paragraph 0121, "In operations 411 and 421, respective normalizations, for example, a first normalization and a second normalization are performed on the face region, e.g., to be suitable for the respective inputs of each recognizer, and based on the detected landmarks," Paragraph 0122, "The second recognizer 427 outputs one or more second registered features 429 based on input image information of the normalized face region 423" Son teaches that a first and second stylization is done on the first given data, then receiving a first data from the first stylization and second data from the second stylization. This is comparable to the present application where the “third data” in the present application is equivalent to the second data received from the “second” stylization from Son’s prior art. This is because in the present application, the “third” data is the data received from the second stylization, but in the prior art, the “second” data is the data received from the second stylization, thus they are equivalent as they are both data resulting from the second stylization. In the prior art, the data from the second stylization is received from the second stylization and sent to a database;) and training the convolutional neural network based on the third data, wherein the convolutional neural network further has a third normalization layer used for the third data (Paragraph 0163, "As an example of such a larger collection of layers, the above discussion with respect to FIG. 5 discusses combining first feature information and second feature information, which may be trained and subsequently implemented by parallel layers configured to implement the respective first and second recognizers and which may each be providing respectively extracted features to an example same subsequent layer for combined consideration, as a non-limiting example." Paragraph 0163, "For example, in examples where the neural network is trained for the image verification or rejection, the neural network may include convolutional layers or be representative of a convolutional neural network (CNN), and thus the respective convolutional kernel elements," Son teaches that information from the second stylization will be implemented into layers, normalized or parallel, to train a convolutional neural network.)
Tremblay, Nam, Hu, and Son are all considered to be analogous to the claimed invention because they are all in the same field of training of identifying a region of interest, performing stylizations on it, and using it to train a convolutional neural network. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tremblay, Nam, and Hu to incorporate the teachings of Son. This is because Son teaches a more accurate way of training a convolutional neural network by using two stylizations on the region of interest and normalizing both outputs. This leads to more accurate results when working with face verification because both stylizations look for different facial features.
Claim(s) 5 and 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tremblay, Nam, and Hu as applied to claim 1 above, and further in view of Narayanan (US 11514319 B2, hereinafter Narayanan).
Regarding claim 5, the combination of Tremblay, Nam, and Hu teach the subject matter of claim 1. However, the combination does not teach the training method according to Claim 1, further comprising: calculating a first loss for the first data; weighting the calculated first loss; and performing backpropagation based on the weighted first loss.
Narayanan though teaches the training method according to Claim 1, further comprising: calculating a first loss for the first data; weighting the calculated first loss; and performing backpropagation based on the weighted first loss (Col. 10, lines 25-37, "During a training phase, the first LSTM output of the first LSTM may represent a first loss, the second LSTM output of the second LSTM may represent a second loss, and the third LSTM output of the third LSTM may represent a third loss, etc. According to one aspect, a first weight associated with the first GCN 212 may be adjusted based on the first loss, a second weight associated with the second GCN 228 may be adjusted based on the second loss, etc. In this way, backpropagation may be utilized to compute gradients or weights associated with the respective GCNs with respect to a loss function during the training phase," GCN is defined as graph convolution network and LTSM is defined as long short term memory of pyramid layer of the convolution layer. During the training phase, a first LSTM output of a first long-short-term- memory (LSTM) gate of the first pyramid layer may represent a first loss and a second LSTM output of a second LSTM of the second pyramid layer may represent a second loss The first loss is calculated and the weight associated with it can be adjusted. Finally, backpropagation can be performed on the weights associated with the respective loss function.)
Tremblay, Nam, and Hu, and Narayanan are all considered to be analogous to the claimed invention because they are all in the same field of training a convolutional neural network. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tremblay and Nam to incorporate the teachings of Lakshimi Narayanan. This is because Lakshimi Narayanan teaches a faster way of training a convolutional neural network by using two layers, one associated with images, and an output summation operator.
Regarding claim 6, the combination of Tremblay, Nam, and Hu teach the subject matter of claim 1. However, the combination does not teach the training method according to Claim 5, further comprising: calculating a second loss for the second data; weighting the calculated second loss; and performing backpropagation based on the weighted second loss.
Narayanan though teaches the training method according to Claim 5, further comprising: calculating a second loss for the second data; weighting the calculated second loss; and performing backpropagation based on the weighted second loss (Col. 10, lines 25-37, "During a training phase, the first LSTM output of the first LSTM may represent a first loss, the second LSTM output of the second LSTM may represent a second loss, and the third LSTM output of the third LSTM may represent a third loss, etc. According to one aspect, a first weight associated with the first GCN 212 may be adjusted based on the first loss, a second weight associated with the second GCN 228 may be adjusted based on the second loss, etc. In this way, backpropagation may be utilized to compute gradients or weights associated with the respective GCNs with respect to a loss function during the training phase," GCN is defined as graph convolution network and LTSM is defined as long short term memory of pyramid layer of the convolution layer. During the training phase, a first LSTM output of a first long-short-term- memory (LSTM) gate of the first pyramid layer may represent a first loss and a second LSTM output of a second LSTM of the second pyramid layer may represent a second loss The second loss is calculated and the weight associated with it can be adjusted. Finally, backpropagation can be performed on the weights associated with the respective loss function).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Tremblay, Nam, and Hu, and Narayanan as applied to claim 6 above, and further in view of Zhou (US 20220083868 A1, hereinafter Zhou).
Regarding claim 7, the combination of Tremblay, Nam, and Hu, and Narayanan do teach “the training method according to Claim 6.” However, the combination of Tremblay, Nam, and Hu, and Narayanan do not teach “further comprising: determining a total loss based on the weighted first loss and the weighted second loss; and performing backpropagation based on the determined total loss.”
Zhou though teaches the training method according to Claim 6, further comprising: determining a total loss based on the weighted first loss and the weighted second loss; and performing backpropagation based on the determined total loss (Paragraph 0079, "In step S142, a weighted sum of the first loss function value and the second loss function value is calculated as a total loss function value. Similarly, those skilled in the art can understand that the first loss function value and the second loss function value can also be combined in other ways to calculate the total loss function value." Paragraph 0080, "In step S143, the parameters of the second neural network are updated in a manner that the total loss function value is backpropagated." Zhou teaches that the total loss can determined from the weighted first loss and weighted second loss. Zhou then teaches that backpropagation can be performed on the total loss).
Tremblay, Nam, Hu, and Narayanan, and Zhou are all considered to be analogous to the claimed invention because they are all in the same field of training neural networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tremblay, Nam, and Lakshimi Narayanan to incorporate the teachings of Zhou. This is because Zhou teaches a faster way to train neural networks by combining the first weighted loss and second weighted loss, then performing backpropagation on the total loss.
Claim(s) 8 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Tremblay (US 11475613 B1, hereinafter Tremblay), in view of Nam (US 20200302286 A1, hereinafter Nam) and Hu et al., (US 2020/0168320 A1, hereinafter Hu).
Regarding claim 8, Tremblay teaches a computer-implemented method for detecting an object, comprising: receiving data of the object; and detecting the object based on the received data of the object using a convolutional neural network, (Col. 3, lines 16-20, "In some implementations, the detected edges can also be utilized for digital image segmentation based on identifying various objects, shapes, and/or regions of interest. In some implementations, one or more digital image segmentations operations may be performed by one or more trainable models (e.g., convolutional neural networks) configured to detect certain objects, shapes, and/or regions of interest" Tremblay teaches that one can use a convolutional neural network to detect objects or shapes based on the received data, which in the present application refers to images.); wherein the convolutional neural network is trained by (i) receiving first data, (ii) performing a first stylization on the received first data, (iii) receiving second data after the first stylization is performed on the first data, (Fig. 1, 130, the source digital image is fed into the stylization module. Col. 2, lines 28-29, "the source digital image 105 is fed to the stylization module 130" Col. 2, lines 35-45, "The stylization module 130 may transform the source digital image 105 based on one or more parameters and/or sample images of the chosen visual style 115. The visual style sample images and/or visual style parameters may specify one or more textures, shapes, color palettes, and/or various other visual digital image aspects that characterize the selected visual style. Accordingly, the stylization module 130 may perform one or more digital image stylization operations, by employing trainable models (also referred to as "machine learning-based models") and/or rule-based stylization methods" Tremblay teaches that the first image is fed into the stylization module, where the first stylization occurs. Tremblay teaches that the stylization that occurs can perform any number of image transformations, such as focusing on the shape characteristic over the texture characteristic, which aligns with how stylization is defined in the present application).
Tremblay fails to teach about training the convolutional neural network based on first and second data and normalization layers. However, Nam teaches and (iv) training the convolutional neural network based on the first data and the second data, wherein the convolutional neural network has a first normalization layer used for the first data, and wherein the convolutional neural network has a second normalization layer used for the second data. (Paragraph 0011, “According to an aspect of the present disclosure, there is provided a method of training a neural network comprising obtaining output data of a first layer of the neural network regarding a training sample, transforming the output data into first normalized data using a first normalization technique, transforming the output data into second normalized data using a second normalization technique, transforming the output data into second normalized data using a second normalization technique, generating third normalized data by aggregating the first normalized data and the second normalized data based on a learnable parameter and providing the third normalized data as an input to a second layer of the neural network” Nam teaches that the training of the neural network is based on the first normalization layer, which is obtained the first data, and second normalization layer, which is obtained from an aggregation of the 1st and second data. Thus, the convolutional neural network is trained based on the first and second normalization layer).
Tremblay and Nam are both considered to be analogous to the claimed invention because they are both in the same field of training of convolutional neural networks based on data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tremblay to incorporate the teachings of Nam. This is because Nam teaches a faster way of training a convolutional neural network by using normalization layers. Normalization layers stabilize training by keeping activation in a stable range, which in turn accelerates training by using larger learning rates without instability.
Tremblay and Nam also fail to teach “… the first data representing a texture feature of the object and a shape feature of the object, …. the first stylization configured to make the object less characterized by the texture feature and to retain characterization of the object according to the shape feature”
However, Hu teaches “… the first data representing a texture feature of the object and a shape feature of the object, …. the first stylization configured to make the object less characterized by the texture feature and to retain characterization of the object according to the shape feature.” (Paragraph [0008]: “Additionally, or alternatively, the system may provide for improvements in autonomous imaging which lacks specific features, requires semantic labels, and/or needs nonlinear adjustments. For example, the system may normalize test data by first processing it through a customization layer (e.g., that features a generative artificial neural network prior to processing the test data through a trained discriminatory artificial neural network). In such cases, the generative artificial neural network may reconstruct portions of an inputted image with missing features, the absence of which may prevent the trained discriminatory artificial neural network from properly classifying objects in the image. In another example, the system … information determined based on the geometric artificial neural network (e.g., such as the three-dimensional dimensions of an object in the image) is used by a convolutional neural network to properly classify the object and/or identify the bounds of features of the object. In another example, the system may normalize test data by first processing it through a customization layer that applies non-linear adjustments (e.g., coloring, texture mapping, etc.) to the test data. The adjusted test-data may then be input into an artificial neural network that is trained on non-linearly adjusted data (e.g., trained on objects with the same coloring, texture mapping, etc.).”)
Tremblay, Nam, and Hu are analogous in the arts because Tremblay, Nam, and Hu all describe of convolutional neural networks based on data.
Therefore, it would be obvious to one of ordinary skill in the art at the filing date of the instant application, having the teachings of Tremblay, Nam, and Hu before him or her, to modify the teachings of Tremblay and Nam to include the teachings of Hu in order to receive a wider range of training data that needs adjustments and can support non-linear adjusted training data of objects and thereby increase the training ability of Tremblay and Nam (see Hu paragraphs [0007] and [0008]).
Regarding claim 9, Nam further teaches the method according to Claim 8, further comprising: inputting the data of the object to the first normalization layer. (Paragraph 0058, "For example, referring to a neural network 40 shown in FIG. 6, the first layer may correspond to one layer #k 41 among a plurality of layers constituting the neural network 40. Also, operation S100 and operations S120 to S180 to be described below may be considered to be performed in a normalization layer 42." Paragraph 0059, "In operation S 120, the output data is transformed into first normalized data with statistical information of a batch to which the training samples 44 belong. In other words, the output data is transformed into the first normalized data through the batch normalization technique." In Figure 6, the starting data, 44, is transformed into the first normalized data, then subsequently inputted to the first normalization layer.)
Claim(s) 10-12 are rejected under 35 U.S.C. 103 as being unpatentable over Tremblay (US 11475613 B1, hereinafter Tremblay), in view of Nam (US 20200302286 A1, hereinafter Nam) and Hu et al., (US 2020/0168320 A1, hereinafter Hu).
Regarding claim 10, Nam teaches a computer system, comprising: one or more processors; and one or more storage devices storing computer-executable instructions, wherein the computer-executable instructions, when executed by the one or more processors, cause the one or more processors to perform a method for detecting an object including (i) receiving data of the object, and (ii) detecting the object based on the received data of the object using a convolutional neural network, wherein the convolutional neural network is trained by (Fig. 19, shows a processor 210 and storage device 290. Paragraph 0133, "In other words, the processor 210 may execute methods according to various embodiments of the present disclosure by performing the one or more instructions." Paragraph 0035, "In particular, FIG. 1 shows a case in which the target task is a task related to an image (e.g., object recognition) as an example," Paragraph 0040, “A training sample may indicate a data unit for training and may include various kinds of data. For example, a training sample may be one image and may include various kinds of data in addition to the image according to a training target or a task,” Nam teaches that the computer system has a processor, storage, and executable instructions. This computer system can receive data in the form of images and detect objects when the processor executes the instructions,) and (iv) training the convolutional neural network based on the first data and the second data, wherein the convolutional neural network has a first normalization layer used for the first data, and wherein the convolutional neural network has a second normalization layer used for the second data (Paragraph 0011, “According to an aspect of the present disclosure, there is provided a method of training a neural network comprising obtaining output data of a first layer of the neural network regarding a training sample, transforming the output data into first normalized data using a first normalization technique, transforming the output data into second normalized data using a second normalization technique, transforming the output data into second normalized data using a second normalization technique, generating third normalized data by aggregating the first normalized data and the second normalized data based on a learnable parameter and providing the third normalized data as an input to a second layer of the neural network.” Nam teaches that the training of the neural network is based on the first normalization layer, which is obtained the first data, and second normalization layer, which is obtained from an aggregation of the 1st and second data. Thus, the convolutional neural network is trained based on the first and second normalization layer.)
Nam fails to teach about stylizations and image transformations. However, Tremblay teaches (i) receiving first data, (ii) performing a first stylization on the received first data, (iii) receiving second data after the first stylization is performed on the first data, (Fig. 1, 130, the source digital image is fed into the stylization module. Col. 2, lines 28-29, "the source digital image 105 is fed to the stylization module 130" Col. 2, lines 35-45, "The stylization module 130 may transform the source digital image 105 based on one or more parameters and/or sample images of the chosen visual style 115. The visual style sample images and/or visual style parameters may specify one or more textures, shapes, color palettes, and/or various other visual digital image aspects that characterize the selected visual style. Accordingly, the stylization module 130 may perform one or more digital image stylization operations, by employing trainable models (also referred to as "machine learning-based models") and/or rule-based stylization methods." Col. 2, lines 65-67, "The styled digital image (also referred to as "underpainting") 135 produced by the stylization module 130 is fed to the digital image analysis module 140" Tremblay teaches that the first image is fed into the stylization module, where the first stylization occurs. Tremblay teaches that the stylization that occurs can perform any number of image transformations, such as focusing on the shape characteristic over the texture characteristic, which aligns with how stylization is defined in the present application. Tremblay teaches that the second data, or the underpainting, is being used as a second data in future operations).
Tremblay and Nam are both considered to be analogous to the claimed invention because they are both in the same field of training of convolutional neural networks based on data. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tremblay to incorporate the teachings of Nam. This is because Nam teaches a faster way of training a convolutional neural network by using normalization layers. Normalization layers stabilize training by keeping activation in a stable range, which in turn accelerates training by using larger learning rates without instability.
Tremblay and Nam also fail to teach “(i) receiving first data representing the object, the first data representing a texture feature of the object and a shape feature of the object, (ii) performing a first stylization on the received first data, the first stylization configured to make the object less characterized by the texture feature and to retain characterization of the object according to the shape feature,”
However, Hu teaches “(i) receiving first data representing the object, the first data representing a texture feature of the object and a shape feature of the object, (ii) performing a first stylization on the received first data, the first stylization configured to make the object less characterized by the texture feature and to retain characterization of the object according to the shape feature,” (Paragraph [0008]: “Additionally, or alternatively, the system may provide for improvements in autonomous imaging which lacks specific features, requires semantic labels, and/or needs nonlinear adjustments. For example, the system may normalize test data by first processing it through a customization layer (e.g., that features a generative artificial neural network prior to processing the test data through a trained discriminatory artificial neural network). In such cases, the generative artificial neural network may reconstruct portions of an inputted image with missing features, the absence of which may prevent the trained discriminatory artificial neural network from properly classifying objects in the image. In another example, the system … information determined based on the geometric artificial neural network (e.g., such as the three-dimensional dimensions of an object in the image) is used by a convolutional neural network to properly classify the object and/or identify the bounds of features of the object. In another example, the system may normalize test data by first processing it through a customization layer that applies non-linear adjustments (e.g., coloring, texture mapping, etc.) to the test data. The adjusted test-data may then be input into an artificial neural network that is trained on non-linearly adjusted data (e.g., trained on objects with the same coloring, texture mapping, etc.).”)
Tremblay, Nam, and Hu are analogous in the arts because Tremblay, Nam, and Hu all describe of convolutional neural networks based on data.
Therefore, it would be obvious to one of ordinary skill in the art at the filing date of the instant application, having the teachings of Tremblay, Nam, and Hu before him or her, to modify the teachings of Tremblay and Nam to include the teachings of Hu in order to receive a wider range of training data that needs adjustments and can support non-linear adjusted training data of objects and thereby increase the training ability of Tremblay and Nam (see Hu paragraphs [0007] and [0008]).
Regarding claim 11, the combination of Tremblay, Nam, and Hu teaches the system of claim 10, and Nam further teaches the computer system according to Claim 10, wherein the computer-executable instructions are included in a computer program product. (Paragraph 0138, “According to the above-described embodiments, it should not be understood that the separation of various configurations is necessarily required, and it should be understood that the described program components and systems may generally be integrated together into a single software product or be packaged into multiple software products.” Nam teaches that the components stated previously like processors and computer executable instructions are stored on a computer program product.)
Regarding claim 12, the combination of Tremblay, Nam, and Hu teaches the system of claim 10, and Tremblay further teaches the computer system according to Claim 11, wherein the computer program product is stored on a non-transitory computer-readable medium (Col. 9, lines 30-43, "This apparatus may be specially constructed for the required purposes, or it may be a general purpose computer system selectively programmed by a computer program stored in the computer system. Such a computer program may be stored in a computer readable storage medium, such as, but not limited to, any type of disk including optical disks, CD-ROMs, and magnetic-optical disks, read- only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic disk storage media, optical storage media, flash memory devices, other type of machine-accessible storage media, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus." Tremblay teaches that this product will be stored on a non-transitory computer readable medium).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Velic et al., (US 2017/0304732 A1), part of the prior art made of record, describes the object features of claims 1, 8, and 10 through object region of interest classification to detect toy objections in paragraph [0042].
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHANE D WOOLWINE whose telephone number is (571)272-4138. The examiner can normally be reached M-F 9:30-6:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MIRANDA HUANG can be reached at (571) 270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SHANE D. WOOLWINE
Primary Examiner
Art Unit 2124
/SHANE D WOOLWINE/Primary Examiner, Art Unit 2124
Prosecution Insights