Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Remarks
Remarks page 9-13, Applicant contends:
Amended claims are not directed towards an abstract idea or at least integrate into a practical application.
Response:
The applicant’s arguments regarding the claims under 101 are seen as persuasive. The amended claim limitations are not seen as being directed towards an abstract idea, and the amended claims contain elements that can direct towards an improvement in the performance of a machine learning model. As a result, the 101 rejections are removed.
Remarks page 13-17, Applicant contends:
Claimed limitations are not disclosed in Date.
Wang does not teach limiting candidate data to learning data.
Response:
As the applicant notes Date discloses aspects related to applying user preferences to input data, but the examiner disagrees that Date does not disclose applying qualitative patterns to the input data. Date, as noted in previous claim 9, teaches that elements of the input include aspects such as the shape and preference. The claims required at least one candidate data for the adaptive pattern. Under BRI the aspects related to the shape and/or preference aspects of the input are seen as fulfilling the requirement, as the description of qualitative pattern does not appear to exclude such data under BRI. (MPEP 2111: "Because applicant has the opportunity to amend the claims during prosecution, giving a claim its broadest reasonable interpretation will reduce the possibility that the claim, once issued, will be interpreted more broadly than is justified. In re Yamamoto, 740 F.2d 1569, 1571 (Fed. Cir. 1984); In re Zletz, 893 F.2d 319, 321, 13 USPQ2d 1320, 1322 (Fed. Cir. 1989) ('During patent examination the pending claims must be interpreted as broadly as their terms reasonably allow.')") The purpose of Date and the current application do not have to be the same in order to read as prior art, and what is written in the specification is not read to limit the claims (MPEP 2145(VI) states: “Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims.”) Amending the claims can limit the BRI of the claims, thus altering aspects, such as details around the transformation or qualitative patterns, could be methods of ensuring the interpretation of the claims is closer to the wanted interpretation.
Applicant argues Wang is silent regarding the data being limited to within the learning data used to train the neural network model. Aspects related to the limitations related to the argument of not teaching data being within the data used to train the neural network model, such as limitations previously in claim 10, are taught by Date where an interpretation of the references and application is noted such as “Examiner interprets this to mean that the user must input candidate data that is the same as the preferences trained on.”. Aspects believed to be related to Wang are argued in previous claim 8 with Wang, such as the motivation to combine with Wang noting utilizing data that is similar enough or meeting a threshold. The amended limitations regarding using learning data used to train the neural network model are seen as taught when incorporating the teachings of previous claim 8 involving Date and Wang and the teachings of Date in previous claim 10.
As aspects of the claim limitations are taught by Date and not solely by Wang, aspects of the applicant’s argument are not seen as convincing regarding Wang ([Form Paragraph 7.37.13]: "In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).").
Status of Claims
Claims 9-10, 16-16, 20, 23-24 were canceled.
Claims 8, 11-15, 18-19, 21-22, 25-27 are pending and are examined herein.
Claims 8, 11-15, 18-19, 21-22, 25-27 are rejected under 35 USC 103.
Priority
Should applicant desire to obtain the benefit of foreign priority under 35 U.S.C. 119(a)-
(d) prior to declaration of an interference, a certified English translation of the foreign
application must be submitted in reply to this action. 37 CFR 41.154(b) and 41.202(e).
Failure to provide a certified translation may result in no benefit being accorded for the
non-English application.
Claim Objections
Claim 8 is object to for reciting “used to learn the deep neural network model”. “learn” is interpreted as intending to mean “train”.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 8, 13, 15-18, 22-25, and 27 are rejected under 35 U.S.C. 103 as being unpatentable over “Date” (“Fashioning with Networks: Neural Style Transfer to Design Clothes”, Published 2017) in view of “Wang” (US 20220138398 A1, “Style Transfer”).
Regarding Claim 8, Date teaches
receiving the target data from a communicator or external image storage unit
([Date page 3 Figure 2 description]: “In the first phase the user provides the system access to his / her closet images from where the user’s fashion preferences are learned.” [Date Related Work page 2]: “Liu et. al [19] create a ro bust fashion dataset of about 800,000 images that contains annotations for various types of clothes, their attributes and the location of landmarks as well as cross-domains pairs.” Examiner interprets the use of an image storage or the use of a collection of images to receive data from as “receiving the target data from a communicator or external image storage unit”)
applying adaptive pattern transformation to the target data based on a qualitative pattern of the at least one candidate data (Date, 1 Introduction recites “Figure 1 shows a sample clothing item generated using neural style transfer. The first clothing item given by the user provides the shape for the new dress. The second is initially provided by the user from his/her closet to learn their preference.” Examiner interprets using the neural style transfer as applying an adaptive pattern transformation and the preference of shape as the candidate data.)
transferring the target data to which the adaptive pattern transformation is applied, to the deep neural network model (Date, 1 Introduction recites “Figure 1 shows a sample clothing item generated using neural style transfer. The first clothing item given by the user provides the shape for the new dress. The second is initially provided by the user from his/her closet to learn their preference. The third is the final generated design for the user (the generated sample contains styles from multiple pieces of the user's clothing).” Examiner recognizes the use of a deep neural network model being used at the end of the process as indicated by ‘generate the design for the user’. Date, 3 Preliminaries confirms this fact, “Consider an input image x and convolutional neural network N N. Every convolution layer l in the convolutional network has Nl distinct filters.”)
acquiring the output value based on the target data to which the adaptive pattern transformation is applied, from the deep neural network model (Date, 1 Introduction recites “Figure 1 shows a sample clothing item generated using neural style transfer. The first clothing item given by the user provides the shape for the new dress. The second is initially provided by the user from his/her closet to learn their preference. The third is the final generated design for the user (the generated sample contains styles from multiple pieces of the user's clothing).” The generated design for the user is derived by using a neural style transfer process that takes in the user’s closet via preferences.)
retrieving, from reference data, at least one candidate data having a similarity to the target data that is greater than a first threshold value, wherein the reference data is selected only from the learning data that was used to learn the deep neural network model
(Date, 4.2 Creating a Personal Style Store recites “To learn the user's fashion preferences, the user initially provides the set of clothes from his / her closet. The Gram matrices Gl (eq.1) of all the clothes with their annotated attributes are calculated.” Figure 2 shows “User’s Personal Style Store” to the right, and the same user inputting “User Chosen Outline” to the left. Examiner interprets this to mean that the user must input candidate data that is the same as the preferences trained on.)
Date does not appear to explicitly teach
Retrieving, from reference data, at least one candidate data having a similarity to the target data that is greater than a first threshold value, wherein the reference data is selected only from the learning data that was used to learn the deep neural network model
However, Wang, directed to analogous art (explicitly on transferring a style to an image), teaches
Retrieving, from reference data, at least one candidate data having a similarity to the target data that is greater than a first threshold value, wherein the reference data is selected only from the learning data that was used to learn the deep neural network model (Wang, Paragraph [0046] recites “In some implementations, a plurality of predefined editable objects can be associated with a plurality of various categories, such as science category, finance category and the like. In such case, one or more data sets whose similarity with the target data set is below a predefined threshold can be respectively determined from the plurality of categories. Thus, styles in various categories can be recommended to the users.” Examiner interprets the one or more data sets as potential candidate states. Wang, Paragraph [0047] proceeds to state “Alternatively or additionally, the style corresponding to the data set with the highest similarity can be directly applied into the target editable object.” Examiner recognizes the word ‘retrieving’ in the instant limitation as continuing with the rest of the method having this information. Examiner sees that ‘can be directly applied’ found within Wang as to also having this context of having the information readily available.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Date in view of Wang to apply only candidate date with the highest similarity or greater than a threshold to the target data in the process of generating the image. Date, Conclusions & Future Work recites “In the future we will like to improve the performance of the pipeline as it is time consuming to generate a new design. Also, we plan to experiment with better methods to personalize and generate designs with higher resolutions.” The reader is lead to improve Date specifically in generating the image closer to the user’s choice. Wang, Paragraph [0049] states “In this way, some better styles can be conveniently recommended to the user for them to choose, thereby enhancing the convenient level of the style transfer.” In Wang, the context is a recommendation. However, said person of ordinary skill in the art would recognize that this same process could be used for other inventions in the art. Wang, Paragraph [0044] recites “The predefined editable objects have respective styles, which can be the styles matching the data at a higher aesthetic degree.” Some styles are more natural with certain images and look better. It would be obvious to said person of ordinary skill to use the combination to generate images that have better quality and match better to the chosen styles.
Regarding Claim 13, the rejection of Claim 8 is incorporated herein. Date teaches
Wherein the adaptive pattern transformation transforms the target data to have the qualitative pattern corresponding to the qualitative pattern of the at least on candidate data (Date, 4.2 Creating a Personal Style Store recites “The Gram matrices Gl (eq.1) of all the clothes with their annotated attributes are calculated. Tensor ow [1] allows us to get the partially computed functions El in 2 (where the gram matrices for Gl are computed first and then Gl later). The style losses El are thus stored in a dictionary with the associated attributes.” Date, 4.3 Style Transfer states “Although the style's extracted from the user's closet as a whole represent the his/her fashion sense, we pick the style functions of the chosen attributes because we assume the user's mental model of dress is likely to be similar to the styles extracted for those attributes.” The qualitative patterns in this art are the styles learned from the closet. These same styles are picked by the user when selecting a certain target.)
It would have been obvious to a person of ordinary skill in the art before the effective
filing date of the claimed invention to modify Date in view of Wang as described above with respect to Claim 8.
Claims 11 are rejected under 35 U.S.C. 103 as being unpatentable over “Date” (“Fashioning with Networks: Neural Style Transfer to Design Clothes”, Published 2017) in view of “Wang” (US 20220138398 A1, “Style Transfer”), in further view of David (US 20190370665 A1), and in further view of Gorin et al (US 8095363 B1), referred to as Gorin.
Claims 15-18 recite a system which performs substantially similar steps as listed by the method in Claims 8-11, respectively, and are rejected with the same rationale, mutatis mutandis. Additionally, Date teaches
A computing apparatus for improving reproduction performance of an output value for target data having a different qualitative pattern from learning data related to a deep neural network model, the computing apparatus comprising: a processor configured to perform processes (Date, System Architecture recites “Figure 2 shows the entire pipeline to personalize and de-sign custom clothes for the user. There are four modules to the architecture, namely, preprocessing, personal style store creation, style transfer and post-processing to generate the final design.” Date, Abstract states “In this paper, the neural style transfer algorithm is applied to fashion so as to synthesize new custom clothes.” Examiner sees no explicit definition of the equipment used in order to perform the disclosed method; thus, the broadest reasonable interruption of the equipment is used. Among such equipment would be a computer system having a processor in order to process the algorithm.)
Regarding Claim 11, the rejection of Claim 8 is incorporated herein. David and Gorin teaches
Date notes aspects as mentioned in previous office action which are now further supported by David and Gorin: (Date, 3.1 Style Extraction recites “The dot product computes the similarities between feature maps. Thus the Gram matrix Gl invariably contains image points that are consistent between the maps while inconsistent features become 0. Consider two images x (input image used to transfer the style) and ^x (a randomly generated image from white noise).” The first threshold value is 0 as that is when inconsistent features become nulled in the adaptive pattern transformation process.)
when the at least one candidate data whose similarity to the target data is greater than the first threshold value is not retrieved from the reference data terminating the adaptive pattern transformation…([David 0027]: “Embodiments of the invention may test the similarity between the new and pre-trained target models based on the random probe dataset or by probing both models with a new random or semi-random set of test inputs to determine the similarity between the corresponding outputs generated by the new and target models. Training may terminate after a measure of such similarity exceeds a threshold or after a predetermined number of epochs. A predetermined number of input/output training pairs, distribution of training pairs, number or diversity or epochs, may result in a sufficient or above threshold match (or lower than threshold error or distance) between the new and target models (e.g., when probed with random test data).”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Date in view of David in order to terminate based on similarity threshold to prevent training or processing past a wanted point [David 0027]. Date and David are in the same field of endeavor of machine learning.
with classifying the target data as impossible to judge ([Gorin Column 9 Line 66]: “However, in step 5300, if the NLU monitor 180 determines that the task classification processor 440 cannot classify the user’s request, in step 5500, the NLU monitor 180 determines whether the probability of correctly understanding the user’s input communication exists above a predetermined threshold.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Date in view of Gorin in order to indicate that a classification fails when not meeting a threshold to be able to communicate such to a user or perform something else as a result [Gorin Column 9 Line 66]. Date and Gorin are in the same field of endeavor of machine learning.
Claims 22-25 and 27 recite a non-transitory medium which stores substantially similar as listed by the method in Claims 8-11 and 13, respectively, and are rejected with the same rationale, mutatis mutandis. Additionally, Date teaches
A computer program stored in a non-transitory machine-readable recording medium, including instructions that cause a computing apparatus to perform a method of improving reproduction performance of an output value for target data having a different qualitative pattern from learning data related to a deep neural network model (Date, System Architecture recites “Figure 2 shows the entire pipeline to personalize and de-sign custom clothes for the user. There are four modules to the architecture, namely, preprocessing, personal style store creation, style transfer and post-processing to generate the final design.” Date, Abstract states “In this paper, the neural style transfer algorithm is applied to fashion so as to synthesize new custom clothes.” Examiner sees no explicit definition of the equipment used in order to perform the disclosed method; thus, the broadest reasonable interruption of the equipment is used. Among such equipment would be a computer system having a processor in order to process the algorithm. The algorithm would have to be stored on a machine-readable medium in order to conduct the steps.)
Claims 12, 14, 19, 21, and 26 are rejected under 35 U.S.C. 103 as being unpatentable over “Date” (“Fashioning with Networks: Neural Style Transfer to Design Clothes”, Published 2017) in view of “Wang” (US 20220138398 A1, “Style Transfer”) and “Chen” (“Unsupervised Stylish Image Description Generation via Domain Layer Norm”, Published in 2018).
Regarding Claim 12, the rejection of Claim 8 is incorporated herein. Date teaches
Date teach reference data represents data having a lower similarity which works in combination with Chen as noted below (Date, 3.1 Style Extraction recites “The dot product computes the similarities between feature maps. Thus the Gram matrix Gl invariably contains image points that are consistent between the maps while inconsistent features become 0. Consider two images x (input image used to transfer the style) and ^x (a randomly generated image from white noise).”)
Date does not appear to explicitly teach
wherein the reference data includes images selected from among a plurality of images included in the learning data, the selected images having similarity to each other less than a second threshold value
However, Chen, directed to analogous art (explicitly on transferring a style to an image), teaches
wherein the reference data includes images selected from among a plurality of images included in the learning data, the selected images having similarity to each other less than a second threshold value (Chen, Figure 2 has a description stating “We make several assumptions to deal with the challenging unsupervised stylish image description generation problem. We first assume there exists a shared latent space Z so that a latent code z ∈ Z can be mapped to the source description space DS and the target stylish description space DT via GS and GT. We also assume there exists a stylish image description embedding function that can map a stylish description to a latent code. Finally, we assume there exists an image embedding.” Figure 2 shows a latent space Z having a latent point z which is a specific similarity between the source image and target description. Examiner interprets the context of Date to be similarities between the characteristics/features. The combination of Chen with Date would arrive at what the Examiner would classify as latent features between the reference data and learning data. In that same figure, Examiner interprets the line E1 that indicates what is an acceptable generated image as a threshold. If the latent features are distant, then the image would not be generated with those features.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Date in view of Wang and Chen. The modifications would be to apply only candidate date with the highest similarity to the target data in the process of generating the image and to form latent spaces representing each feature. The rational for combining with Wang is described in Claim 8. Chen, Abstract recites “It can learn to generate stylish image descriptions that are more related to image content and can be trained with the arbitrary monolingual corpus without collecting new paired image and stylish descriptions. Moreover, it enables users to generate various stylish descriptions by plugging in style-specific parameters to include new styles into the existing model.” This would customize the style they want even if it is not a trained/known style yet, giving the user more personalization of the process. Examiner interprets the modifications to be congruent in personalizing images generated from Date’s art: Said person of ordinary skill in the art looking to generate images that match the user’s intentions would see the obvious combination of these arts.
Regarding Claim 14, the rejection of Claim 8 is incorporated herein. Date teaches
a number of the at least one candidate data is greater than 2 wherein the qualitative pattern of the at least one candidate data is based on a combination of the at least one candidate data (Date, 4.3 Style Transfer recites “The number of images for every attribute picked depends on the distribution of the particular attribute across the entire list of images present. The higher the frequency of the attribute in the distribution, the higher is the bias towards a certain label and suppresses the effect of the others. This makes certain image characteristics more pronounced in the final dress than others.” Examiner interprets the images belonging to the user’s closet as candidate data while the characteristics as the qualitative patterns.)
Date does not appear to explicitly teach
or an average value of the at least one candidate data, in a latent space of the deep neural network model
However, Chen, directed to analogous art (explicitly on transferring a style to an image), teaches
or an average value of the at least one candidate data, in a latent space of the deep neural network model (Chen, Figure 2 has a description stating “We make several assumptions to deal with the challenging unsupervised stylish image description generation problem. We first assume there exists a shared latent space Z so that a latent code z ∈ Z can be mapped to the source description space DS and the target stylish description space DT via GS and GT. We also assume there exists a stylish image description embedding function that can map a stylish description to a latent code. Finally, we assume there exists an image embedding.” Figure 2 shows a latent space Z having a latent point z which is a specific similarity between the source image and target description. Chen, Experiment recites “The overall content similarity score is averaged over the testing data. This is because we assume stylish descriptions should at least contain objects which appear in the image.” Examiner recognizes that the latent space of a certain space is averaged. The latent space belong to the candidate data would also follow this implementation and be averaged together within the combination of the two arts.)
It would have been obvious to a person of ordinary skill in the art before the effective
filing date of the claimed invention to modify Date in view of Wang and Chen as described above with respect to Claim 12.
Claims 19 and 21 recite a system which performs substantially similar steps as listed by the method in Claims 12 and 14, respectively, and are rejected with the same rationale, mutatis mutandis. Additionally, Date teaches
A computing apparatus for improving reproduction performance of an output value for target data having a different qualitative pattern from learning data related to a deep neural network model, the computing apparatus comprising: a processor configured to perform processes (Date, System Architecture recites “Figure 2 shows the entire pipeline to personalize and de-sign custom clothes for the user. There are four modules to the architecture, namely, preprocessing, personal style store creation, style transfer and post-processing to generate the final design.” Date, Abstract states “In this paper, the neural style transfer algorithm is applied to fashion so as to synthesize new custom clothes.” Examiner sees no explicit definition of the equipment used in order to perform the disclosed method; thus, the broadest reasonable interruption of the equipment is used. Among such equipment would be a computer system having a processor in order to process the algorithm.)
Claim 26 recites a non-transitory medium which stores substantially similar as listed by the method in Claim 12, respectively, and is rejected with the same rationale, mutatis mutandis. Additionally, Date teaches
A computer program stored in a non-transitory machine-readable recording medium, including instructions that cause a computing apparatus to perform a method of improving reproduction performance of an output value for target data having a different qualitative pattern from learning data related to a deep neural network model (Date, System Architecture recites “Figure 2 shows the entire pipeline to personalize and de-sign custom clothes for the user. There are four modules to the architecture, namely, preprocessing, personal style store creation, style transfer and post-processing to generate the final design.” Date, Abstract states “In this paper, the neural style transfer algorithm is applied to fashion so as to synthesize new custom clothes.” Examiner sees no explicit definition of the equipment used in order to perform the disclosed method; thus, the broadest reasonable interruption of the equipment is used. Among such equipment would be a computer system having a processor in order to process the algorithm. The algorithm would have to be stored on a machine-readable medium in order to conduct the steps.)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Tasdizen et al ("Improving the robustness of convolutional networks to appearance variability in biomedical images") is relevant art that discusses aspects noted in the current applications specification such as medical image recognition and the need for a form of transformation to assist with data different than training data.
Afridi et al (WO 2019/025909 A1) is relevant art that discusses transferring the style of an image to a target image. Relevant as the current application discusses transforming a target image using another image as a reference.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTOPHER D DEVORE whose telephone number is (703)756-1234. The examiner can normally be reached Monday-Friday 7:30 am - 5 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.D.D./Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights