DETAILED ACTION
This communication is a Final Office Action rejection on the merits. Claims 1-20 are currently pending and have been addressed below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Response to Arguments
Applicant's arguments filed on 11/25/2025 (related to the 101 Rejection) have been fully considered but they are not persuasive.
Applicant states, on pages 7-11, that predicting customer sentiment and optimizing product design is not the same as advertising or marketing. In fact, developing a product that will be appealing to customers is a basic cornerstone of product design, and does not involve the acts of advertising or marketing at all. The MPEP states that if a claim is based on or involves an abstract idea, but does not recite it, then the claim is not directed to an abstract idea. (MPEP 2106.04(a)(l).) The 2019 PEG further stipulates that "only when a claim recites a judicial exception does the claim require further analysis in order to determine its eligibility."
Examiner respectfully disagrees with Applicant. Examiner notes that the predictions are based on performance of sales activities or behaviors. In this case, a sentiment of the customer toward a product is considered a performance of sales activities or behaviors (e.g., understanding customer needs or likes). Also, using an algorithm for predicting customer sentiments is considered a mathematical relationship (e.g., relationship between features and sentiments). If a claim limitation, under its broadest reasonable interpretation, covers sales behaviors or mathematical relationships, then it falls within the “certain methods of organizing human activity” or “mathematical concepts” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
The mere nominal recitation of generic computer components does not take the claim out of the abstract idea grouping. The computer is merely used to execute instructions (Paragraph 0107). The pre-trained image processing model is merely used to learn rich feature representations from a wide range of images (Paragraph 0054). The pre-trained natural language processing model is merely used to analyze user reviews presented in the form of unstructured data (Paragraph 0047). The DMDE model is merely used to predict the overall and attribute-level sentiment values based on the features extracted from orthographic product images and from product descriptions (Paragraph 0004). The generative design model is merely used to edit design concepts at the attribute level automatically (Paragraph 0043). In this case, the claim does not provide any details about how the DMDE is initially trained and/or improved over time (see 2024 AI Guidance, Example 47, claim 2). Also, although the generative design model is used to generate a design (e.g., an image), the claim does not provide any specific details of how the model/computer generated the image (MPEP 2106.05(f)).
Lastly, the claim fails to recite any improvements to another technology or technical field, improvements to the functioning of the computer itself, use of a particular machine, effecting a transformation or reduction of a particular article to a different state or thing, adding unconventional steps that confine the claim to a particular useful application, and/or meaningful limitations beyond generally linking the use of an abstract idea to a particular environment. See 84 Fed. Reg. 55. Viewed individually or as a whole, these additional claim element(s) do not provide meaningful limitation(s) to transform the abstract idea into a patent eligible application of the abstract idea such that the claim(s) amounts to significantly more than the abstract idea itself. Thus, the claim is not patent eligible.
Examiner recommends to follow Example 47, claim 3 of the 2024 AI Guidance and/or further specify how the machine learning operates (if supported by the specification).
Applicant's arguments filed on 11/25/2025 (related to the 103 Rejection) have been fully considered but are moot in view of new grounds of rejection. Applicant's amendments necessitated the new ground(s) of rejection presented in this Office action. Rejection based on a newly cited reference(s) follows.
Although Arguments are moot in view of new grounds of rejection, Examiner wanted to address arguments presented on pages 15-16. Applicant states, on pages 15-16, that Song does not describe "providing a new product design comprising one or more orthographic renderings to the DMDE model to generate ... a different product design having one or more attributes having favorable predicted customer sentiments." It is not clear how the new design generated by the method described by Song would be combined with the process as described by Lu to result in the generation of new, different designs.
Examiner respectfully disagrees with Applicant. Although Applicant claims providing a new product design, Examiner notes that the Applicant does not specify how the new product design is generated by a computer. In this case, “providing a new product design” may be interpreted as a new product design generated by a designer and uploaded to the model or a new product design generated by a computer. Further clarification is requested.
Further, Examiner notes that Song et al. discloses a new product design generated by a computer having one or more attributes having favorable predicted customer sentiments (see Paragraphs 0044, 0047-0048, and 0052, Paragraph 0048, Also, as another embodiment of generating a new design element, there is a method of synthesizing different elements among design elements to generate a new design for a specific item. Specifically, a new design can be generated by merging different elements among elements of each trendy design; Examiner notes that an estimated preference and/or a trendy product based on customer feedback is a type of sentiment). Although Song et al. discloses wherein the new design is generated using one or more attributes having favorable predicted customer sentiments, Song et al. does not specifically disclose wherein the predicted customer sentiments are predicted using a DMDE model with a self-attention fusion mechanism. However, the combination of Lu et al. and Huang discloses a deep multimodal design evaluation (DMDE) model with a self-attention fusion to predict customer sentiment for new product designs (Lu et al., Paragraph 0041, According to one embodiment, the NLP component 220 may be used to determine (e.g., infer) the user's opinion or perception as relating to the product 234 based on textual statement 238. In at least one embodiment, the association component 224 may also be implemented to link the user-defined product feature 242 to the textual statement 238 corresponding to the user-defined product feature 242; Huang, Page 26, 1. Introduction, Sentiment analysis of such large-scale multimodal data can help better understand people’s attitude or opinion toward certain events or topics. For example, companies are interested in understanding how their products or brands is perceived among their customers [1–3]). Page 28, 3. Deep Multimodal Attentive Fusion, The details of the overall architecture of the proposed model Deep Multimodal Attentive Fusion are shown in Fig. 2. First, two separate unimodal attention models are proposed to learn the most discriminative features in image and text respectively. The visual attention mechanism is used to automatically focus on the affectional regions, while the semantic attention mechanism is used to highlight the most emotional words. Then, a deep intermediate fusion-based multimodal attention model is proposed to exploit the complementary and non-redundant information in different modalities. It employs a multi-layer perceptron to mine the non-linear correlation between different modalities of features. Finally, a late fusion scheme upon the three models, i.e., visual attention model, semantic attention model, and multimodal attention model, is proposed to obtain the final decision of sentiment classification). Therefore, the combination of Lu et al. and Huang discloses another model used to predict customer sentiments, which is merely a simple substitution of one known prediction model (e.g., estimate product or product attribute preference and/or trend) for another prediction model (e.g., predict customer sentiment of a product by analyzing images and textual descriptions).
Examiner recommends to further include specific steps of how the DMDE is trained and/or specific steps of how the design is generated or edited using the predicted sentiments (see Applicant’s specification, Paragraph 0103, train with orthographic images and product attributes; Paragraph 0105, model the connection between the images and descriptive language).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., an abstract idea) without reciting significantly more.
Independent Claim 1
Step One - First, pursuant to step 1 in the January 2019 Revised Patent Subject Matter Eligibility Guidance (“2019 PEG”) on 84 Fed. Reg. 53, the claim 1 is directed to a method which is a statutory category.
Step 2A, Prong One - Claim 1 recites: A method of predicting customer sentiment for a product and aspects thereof, comprising the steps of: receiving customer data for a plurality of products, and generating a vector of customer sentiments associated with different aspects of each of the plurality of products based on the customer data; receiving images for each of the plurality of products, and generating a latent vector for the images for each product; receiving a textual description for each of the plurality of products, and generating a latent vector for the textual description for each product; applying a model to integrate the latent vectors of the images and the textual descriptions for each product to predict customer sentiment for new product designs based on their images and textual descriptions; and providing a new product design comprising one or more orthographic renderings to the model to generate sentiments for one or more attributes of the new product design, and a second new product design having one or more attributes having favorable predicted customer sentiments using the model and a design model, based solely on the one or more orthographic renderings. These claim elements are considered to be abstract ideas because they are directed to “certain methods of organizing human activity” which include “commercial or legal interactions.” In this case, designing a product based on previous customer preferences/sentiments is considered a sales activity (e.g., optimizing a product design). If a claim limitation, under its broadest reasonable interpretation, covers commercial or legal interactions, then it falls within the “certain methods of organizing human activity” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2 - The judicial exception is not integrated into a practical application. Claim 1 includes additional elements: a computer; by fine-tuning a pre-trained image processing model; by fine-tuning a pre-trained natural language processing model; a deep multimodal design evaluation (DMDE) model with a self-attention fusion mechanism; and a generative design model.
The computer is merely used to execute instructions (Paragraph 0107). The pre-trained image processing model is merely used to learn rich feature representations from a wide range of images (Paragraph 0054). The pre-trained natural language processing model is merely used to analyze user reviews presented in the form of unstructured data (Paragraph 0047). The DMDE model is merely used to predict the overall and attribute-level sentiment values based on the features extracted from orthographic product images and from product descriptions (Paragraph 0004). The generative design model is merely used to edit design concepts at the attribute level automatically (Paragraph 0043). Merely stating that the step is performed by a computer component results in “apply it” on a computer (MPEP 2106.05f). These elements of “computer,” “image processing model,” “natural language processing model,” “DMDE model,” and “generative design model” are recited at a high level of generality such that it amounts no more than mere instructions to apply the exception using a generic computer element. Accordingly, alone and in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Therefore, the claim is directed to an abstract idea.
Step 2B - The claim does not include additional elements that are sufficient to amount significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the claims describe how to generally “apply” the concept of obtaining predicted customer sentiments for one or more attributes of the new product design. The specification shows that the computer is merely used to execute instructions (Paragraph 0107). The pre-trained image processing model is merely used to learn rich feature representations from a wide range of images (Paragraph 0054). The pre-trained natural language processing model is merely used to analyze user reviews presented in the form of unstructured data (Paragraph 0047). The DMDE model is merely used to predict the overall and attribute-level sentiment values based on the features extracted from orthographic product images and from product descriptions (Paragraph 0004). The generative design model is merely used to edit design concepts at the attribute level automatically (Paragraph 0043). In this case, the claim does not provide any details about how the DMDE is initially trained and/or improved over time (see 2024 AI Guidance, Example 47, claim 2). Further, although the generative design model is used to generate a design (e.g., an image), the claim does not provide any specific details of how the model/computer generated the image (MPEP 2106.05(f)). Thus, nothing in the claim adds significantly more to the abstract idea. The claim is ineligible.
Independent claim 9 is directed to an article of manufacture at step 1, which is a statutory category. Claim 9 recites similar limitations as claim 1 and is rejected for the same reasons at step 2a, prong one; step 2a, prong 2; and step 2b. Claim 9 further recites “computer readable storage medium” and “processor” – which are treated as just an explicit “processor/computer” for executing the operations and are treated under MPEP 2106.05f in the same manner as claim 1. Accordingly, these limitations are viewed as “apply it on a computer” at step 2a, prong 2 and step 2b. Thus, the claim is ineligible.
Independent claim 17 is directed to an apparatus at step 1, which is a statutory category. Claim 17 recites similar limitations as claim 1 and is rejected for the same reasons at step 2a, prong one; step 2a, prong 2; and step 2b. Claim 17 further recites “processor” and “memory” – which are treated as just an explicit “processor/computer” for executing the operations and are treated under MPEP 2106.05f in the same manner as claim 1. Accordingly, these limitations are viewed as “apply it on a computer” at step 2a, prong 2 and step 2b. Thus, the claim is ineligible.
Dependent claims 2-3, 10-11, and 18-19 are not directed to any additional claim elements. Rather, these claims offer additional functions of elements found in the independent claims and addressed above - such as: wherein the customer data comprises customer reviews or customer survey results; and wherein the customer reviews comprise online reviews posted by customers scraped from online sources. The additional function of “scraped from online sources” is considered an insignificant extra-solution activity at step 2A, Prong 2; since it’s just “mere data gathering” to use it for a sentiment analysis (MPEP 2106.05g). At Step 2B, this is a conventional computer function of “receiving or transmitting data over a network” (MPEP 2106.05d). Thus, nothing in the claims add significantly more to the abstract idea. The claim is ineligible.
Dependent claims 4-5, 12-13, and 20 are not directed to any additional claim elements. Rather, these claims offer additional functions of elements found in the independent claims and addressed above - such as: how the latent vector of the images are generated (e.g., based on a sentiment expressed for each attribute of the product). In this case, the claims do not provide any details about how the model identifies sentiments expressed for each attribute of the product (see 2024 AI Guidance, Example 47, claim 2). Merely stating that the step is performed by a particular technological environment (e.g., DMDE model) results in “apply it” on a computer (MPEP 2106.05f) being applicable at both Step 2A, Prong 2 and Step 2B. Mere instructions to apply an exception using a generic technological environment cannot provide an inventive concept. Thus, the claim is ineligible.
Dependent claims 6 and 14 are directed to an additional element such as: a multimodal data concatenation process. The multimodal data concatenation process is merely used to integrate the features associated with different modalities (Paragraphs 0058-0059). However, the claims do not provide any details about how the features are integrated (see 2024 AI Guidance, Example 47, claim 2). Merely stating that the step is performed by a particular technological environment (e.g., multimodal data concatenation process) results in “apply it” on a computer (MPEP 2106.05f) being applicable at both Step 2A, Prong 2 and Step 2B. Mere instructions to apply an exception using a generic technological environment cannot provide an inventive concept. Thus, the claim is ineligible.
Dependent claims 7-8 and 15-16 are not directed to any additional claim elements. Rather, these claims offer additional descriptions of elements found in the independent claims and addressed above - such as: wherein the DMDE model and the generative design model comprise neural network models; and wherein the images comprise multiple different views of each of the plurality of products. In this case, the claims do not provide any details about how the neural network models operate (see 2024 AI Guidance, Example 47, claim 2). Also, the claims do not provide any details of how the multiple different views of each of the plurality of products are used to train the image processing model. Merely stating that the step is performed by a particular technological environment (e.g., neural network) results in “apply it” on a computer (MPEP 2106.05f) being applicable at both Step 2A, Prong 2 and Step 2B. Mere instructions to apply an exception using a generic technological environment cannot provide an inventive concept. Thus, the claim is ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Lu et al. (US 2022/0092652 A1), in view of Huang (Huang, F., Zhang, X., Zhao, Z., Xu, J. and Li, Z., 2019. Image–text sentiment analysis via deep multimodal attentive fusion. Knowledge-Based Systems, 167, pp.26-37), in further view of Song et al. (US 2020/0372193 A1) and Desai (Desai, Y., Shah, N., Shah, V., Bhavathankar, P. and Katchi, K., 2021, September. Markerless augmented reality based application for e-commerce to visualise 3D content. In 2021 Third International Conference on Inventive Research in Computing Applications (ICIRCA) (pp. 756-760). IEEE).
Regarding claim 1 (Currently Amended), Lu et al. discloses a computer-implemented method of predicting customer sentiment for a product and aspects thereof, comprising the steps of (Paragraph 0003, Embodiments of the present invention disclose a method, computer system, and a computer program product for user feedback visualization; Paragraph 0041, According to one embodiment, the NLP component 220 may be used to determine (e.g., infer) the user's opinion or perception as relating to the product 234 based on textual statement 238. In one embodiment, the NLP component 220 may utilize sentiment analysis and topic modeling techniques to characterize an orientation of the sentiment expressed in the user's opinions. In embodiments, the sentiment orientation may include, the polarity, tone, and/or emotions expressed in the user's opinions. In various embodiments, the sentiment orientation may be clustered into three main categories: positive, negative, and neutral sentiment):
receiving customer data for a plurality of products, and generating a vector of customer sentiments associated with different aspects of each of the plurality of products based on the customer data (Paragraph 0038, According to one embodiment, the user feedback database 208 may include one or more textual statements 238, one or more image data 240, and one or more user-defined product features 242 received from user device 212; Paragraph 0042, According to one embodiment, the aggregation component 222 may receive the textual statement 238 tagged with one or more topics (e.g., product feature) and corresponding sentiment orientations. The aggregation component 222 may apply a statistical accumulation of the sentiment orientations for each product feature to determine an aggregated feedback rating value or score (e.g., three out of five) based on the sentiment or overall evaluation of the product feature);
receiving images for each of the plurality of products, and generating a latent vector for the images for each product by … image processing model (Paragraph 0038, According to one embodiment, the user feedback database 208 may include one or more textual statements 238, one or more image data 240, and one or more user-defined product features 242 received from user device 212. In one embodiment, image data 240 may include one or more photographs of an object (e.g., product 234 or components thereof) received from the user device 212. As will be described further, in embodiments, the feedback program 110a, 110b may implement image processing techniques to generate pictorial representations of a product 234 based on the image data 240 received from user device 212 corresponding to the product 234. These pictorial representations of the products 234 may be referred to as a user image-based product representation 244 and stored in output database 210);
receiving a textual description for each of the plurality of products, and generating a latent vector for the textual description for each product by fine-tuning a pre-trained natural language processing model (Paragraph 0038, According to one embodiment, the user feedback database 208 may include one or more textual statements 238, one or more image data 240, and one or more user-defined product features 242 received from user device 212. In one embodiment, the textual statements 238 may include natural language input corresponding to: a description and/or opinion of product 234 as a whole, a description and/or opinion of one or more user-defined product features 242 of product 234, or a description and/or opinion of both—product 234 as a whole and one or more user-defined product features 242 of product 234; Paragraph 0042, According to one embodiment, the aggregation component 222 may receive the textual statement 238 tagged with one or more topics (e.g., product feature) and corresponding sentiment orientations. The aggregation component 222 may apply a statistical accumulation of the sentiment orientations for each product feature to determine an aggregated feedback rating value or score (e.g., three out of five) based on the sentiment or overall evaluation of the product feature);
applying a … model … to integrate the latent vectors of the images and the textual descriptions for each product to predict customer sentiment for new product designs based on their images and textual descriptions (Paragraph 0039, The feedback program 110a, 110b may enable the user to graphically select or annotate (e.g., via cursor control device; touchscreen) a portion of the pictorial representation (e.g., retail image 236; user image-based product representation 244) of the product 234 to dynamically register the selected portion as the user-defined product feature 242. In one embodiment, the feedback program 110a, 110b may electronically link the user-defined product feature 242 (e.g., the selected pixels) to segments of the textual statement 238 such that the descriptions/opinions in the textual statement 238 may be associated with the user-defined product feature 242. In some embodiments, the feedback program 110a, 110b may enable the user to enter a feature name for the user-defined product feature 242. In other embodiments, the feedback program 110a, 110b may automatically determine the feature name for the user-defined product feature 242 based on one or more segments of the textual statement 238 associated with the user-defined product feature 242; Paragraph 0041, According to one embodiment, the NLP component 220 may be used to determine (e.g., infer) the user's opinion or perception as relating to the product 234 based on textual statement 238. In one embodiment, the NLP component 220 may utilize sentiment analysis and topic modeling techniques to characterize an orientation of the sentiment expressed in the user's opinions. In embodiments, the sentiment orientation may include, the polarity, tone, and/or emotions expressed in the user's opinions. In various embodiments, the sentiment orientation may be clustered into three main categories: positive, negative, and neutral sentiment. In at least one embodiment, the sentiment orientation may be clustered into any number of categories. Using topic modeling, the NLP component 220 may draw out and identify the product features or components of product 234 mentioned in the textual statement 238. According to one embodiment, the association component 224 may be implemented to link the sentiment to the respective product features or components. In at least one embodiment, the association component 224 may also be implemented to link the user-defined product feature 242 to the textual statement 238 corresponding to the user-defined product feature 242; Examiner notes that Lu et al. is linking the textual statements 238 with the image user-defined features 242, wherein the linked information is used to predict/infer customer sentiment for a product and/or features of the product);
and … a … new product design having one or more attributes having favorable predicted customer sentiments using the … model … (Paragraph 0047, In at least one embodiment, the collaboration component 232 may enable the user to communicate how they fixed a problem with the product (e.g., communicating that replacing a bolt would make the product work more efficiently for a specific scenario). This may enable users to customize products to meet specific needs and share that customization with other users. In one embodiment, the collaboration component 232 may also enable the user to link to other parts which may be used to fix a broken product feature; Paragraph 0069, As described previously with reference to FIGS. 2 and 3, the feedback program 110a, 110b may employ NLP techniques to associate segments of the textual statement 520 to the product 502 and/or the user-defined product feature 510 and determine the sentiment of the user corresponding to the product 502 and/or the user-defined product feature 510; Examiner interprets “customizing the products based on the feedback” as the “new product design having one or more attributes having favorable predicted customer sentiments”).
Although Lu et al. discloses applying a model to integrate/link images and the textual descriptions for each product to predict customer sentiment (Paragraph 0069), Lu et al. does not specifically disclose wherein the model is a deep multimodal model with a self-attention fusion mechanism.
However, Huang discloses … receiving images for each of the plurality of products, and generating a latent vector for the images for each product by fine-tuning a pre-trained image processing model (Page 26, 1. Introduction, Sentiment analysis of such large-scale multimodal data can help better understand people’s attitude or opinion toward certain events or topics. For example, companies are interested in understanding how their products or brands is perceived among their customers [1–3]); Page 28, 2.2. Image Sentiment Analysis, Motivated by the powerful performance of deep models on extracting high level image features, Xu et al. [39] transferred VGG networks trained on ImageNet dataset into visual sentiment analysis on the sentiment datasets);
receiving a textual description for each of the plurality of products, and generating a latent vector for the textual description for each product by fine-tuning a pre-trained natural language processing model (Page 26, 1. Introduction, Sentiment analysis of such large-scale multimodal data can help better understand people’s attitude or opinion toward certain events or topics. For example, companies are interested in understanding how their products or brands is perceived among their customers [1–3]); Page 27, 2.1. Text Sentiment Analysis, Text sentiment analysis is a well-studied research area in NLP. These methods can be divided into two groups: lexicon-based methods [24–26,20] and machine learning-based methods [27– 30]; Page 29, 3.2. Semantic attention model for text sentiment analysis, Similar to image regions, some words in the text are usually more important to sentiment presentation compared to other words. Recently, semantic attention mechanism has been proven to be beneficial for many natural language processing related tasks, such as machine translation [45,46], text sentiment analysis [47, 48]. Different from these work, our semantic attention model for sentiment classification is also formulated in an end-to-end process, which can directly highlight the most important words);
applying a deep multimodal design evaluation (DMDE) model with a self-attention fusion mechanism to integrate the latent vectors of the images and the textual descriptions for each product to predict customer sentiment for new product designs based on their images and textual descriptions (Page 28, 3. Deep Multimodal Attentive Fusion, The details of the overall architecture of the proposed model Deep Multimodal Attentive Fusion are shown in Fig. 2. First, two separate unimodal attention models are proposed to learn the most discriminative features in image and text respectively. The visual attention mechanism is used to automatically focus on the affectional regions, while the semantic attention mechanism is used to highlight the most emotional words. Then, a deep intermediate fusion-based multimodal attention model is proposed to exploit the complementary and non-redundant information in different modalities. It employs a multi-layer perceptron to mine the non-linear correlation between different modalities of features. Finally, a late fusion scheme upon the three models, i.e., visual attention model, semantic attention model, and multimodal attention model, is proposed to obtain the final decision of sentiment classification);
and providing a new product design … to the DMDE model to generate customer sentiments for one or more attributes of the new product design, and … (Page 27, 2. Related Work, Sentiment analysis is an important task which has been rapidly developed in recent years. It has been applied to a broad set of applications, including product evaluation; 3. Deep Multimodal Attentive Fusion, The details of the overall architecture of the proposed model Deep Multimodal Attentive Fusion are shown in Fig. 2. First, two separate unimodal attention models are proposed to learn the most discriminative features in image and text respectively. The visual attention mechanism is used to automatically focus on the affectional regions, while the semantic attention mechanism is used to highlight the most emotional words. Then, a deep intermediate fusion-based multimodal attention model is proposed to exploit the complementary and non-redundant information in different modalities. It employs a multi-layer perceptron to mine the non-linear correlation between different modalities of features. Finally, a late fusion scheme upon the three models, i.e., visual attention model, semantic attention model, and multimodal attention model, is proposed to obtain the final decision of sentiment classification; Examiner interprets “evaluating the product” as “providing a new product design … to the DMDE model to generate customer sentiments for one or more attributes of the new product design).
It would have been obvious to one ordinary skill in the art before the effective filing date to modify the method for predicting customer sentiment for a product of the invention of Lu et al. to further incorporate wherein the method uses a deep multimodal model with a self-attention fusion mechanism of the invention of Huang because doing so would allow the method to use a deep intermediate fusion-based multimodal attention model to exploit the complementary and non-redundant information in different modalities (see Huang, Figure 2 & 3. Deep Multimodal Attentive Fusion). Further, the claimed invention is merely a combination of old elements, and in combination each element would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
The combination of Lu et al. and Huang discloses applying a Deep Multimodal Attentive Fusion model to predict customer sentiment for one or more attributes based on their images and textual descriptions (see Huang, Figure 2 & 3. Deep Multimodal Attentive Fusion). Although the combination of Lu et al. and Huang further discloses wherein the predicted customer sentiments for the one or more attributes may be used for designing a new product (e.g., product customization/evaluation), the combination of Lu et al. and Huang does not specifically disclose wherein the new product is generated using a generative design model (e.g., generative adversarial networks).
However, Song et al. discloses and providing a new product design comprising one or more [images] to the … model to generate customer sentiments for one or more attributes of the new product design (Paragraph 0052, Ranking new designs or design elements may be determined based on the frequency of exposure of a specific design or design element on images exposed to social media or websites in general, the volume and amount of sales of items that contain a specific design or design element, and the preference (determined based on all data that can estimate the preference for the image, such as clicks, feedback, and sharing) for the corresponding image exposed on the Internet, such as a specific website or social media; Paragraph 0093, The evaluation unit 406 evaluates a newly generated design or design element in the design changing unit 402 or the design synthesis unit 404 on a constant basis. Specifically, the evaluation unit 406 may rank new designs or design elements extracted in consideration of various factors; Examiner notes that an estimated preference and/or a trendy product based on customer feedback is a type of sentiment), and a second new product design having one or more attributes having favorable predicted customer sentiments and a generative design model, based solely on the one or more [images] (Paragraph 0044, In order to generate a new design, generative adversarial networks (GAN) with category-specific and random generation, Variational Autoencoder (VAE)+GAN that can randomly transform a specific design and generate a new design in a form with high similarity in characteristics of a specific design, genetic algorithm+GAN that recognizes a specific design element as a human genetic trait (looks like), generates various crosses, and repeatedly transfers new, more productive variants between generations based on feedback, conditional GAN for design change, and a style transfer technology that transforms style while maintaining the appearance of an existing design by extracting inspiration for a new design from various images and data can be used; Paragraph 0047, As an embodiment of generating a design, there is a method of generating a new design element by changing a vector value of the design element. The change of the vector value can be input as a set value, and can be made by a learned design generation model. Specifically, for example, referring to FIG. 5, if a floral dress is fashionable, the vector values of the existing floral patterns in the floral dress extracted from the collected; Paragraph 0048, Also, as another embodiment of generating a new design element, there is a method of synthesizing different elements among design elements to generate a new design for a specific item. Specifically, a new design can be generated by merging different elements among elements of each trendy design; Paragraph 0052, Ranking new designs or design elements may be determined based on the frequency of exposure of a specific design or design element on images exposed to social media or websites in general, the volume and amount of sales of items that contain a specific design or design element, and the preference (determined based on all data that can estimate the preference for the image, such as clicks, feedback, and sharing) for the corresponding image exposed on the Internet, such as a specific website or social media; Examiner notes that an estimated preference and/or a trendy product based on customer feedback is a type of sentiment).
It would have been obvious to one ordinary skill in the art before the effective filing date to modify the method for predicting customer sentiment for a product (e.g., by applying a DMDE model with a self-attention fusion mechanism) of the invention of Lu et al. and Huang to further incorporate wherein the method uses the favorable predicted customer sentiments (e.g., trends) to generate an image of the invention of Song et al. because doing so would allow the method to generate a new design based on trendy design elements (see Song et al., Paragraph 0090). Further, the claimed invention is merely a combination of old elements, and in combination each element would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Although the combination of Lu et al., Huang, and Song et al. discloses providing a new product design comprising one or more [images] to the DMDE model to generate customer sentiments for one or more attributes of the new product design (Huang, Figure 2 & 3. Deep Multimodal Attentive Fusion to obtain a final decision of sentiment classification; Song et al., Paragraph 0093, The evaluation unit 406 evaluates a newly generated design or design element in the design changing unit 402. Specifically, the evaluation unit 406 may rank new designs or design elements extracted in consideration of various factors), and a second new product design having one or more attributes having favorable predicted customer sentiments using the DMDE model and a generative design model, based solely on the one or more [images] (Huang, Figure 2 & 3. Deep Multimodal Attentive Fusion to obtain a final decision of sentiment classification; Song et al., Paragraph 0044, GAN to generate a new design; Paragraph 0090, One blouse design can be generated by merging “cuff sleeves” of the blouse, “flower-shaped collar” of the blouse, and “mint color” of the blouse, which are the elements of the trendy design). Although the combination of Lu et al., Huang, and Song et al. further discloses wherein the design generated by the generative design model is an image (e.g., generate a new image using a generative design model such as a GAN), the combination of Lu et al., Huang, and Song et al. does not specifically disclose wherein the image comprises one or more orthographic renderings.
However, Desai discloses … comprising one or more orthographic renderings …, based solely on the one or more orthographic renderings (Abstract, Augmented Reality in E-commerce allows customers to view products or experience services in their physical space before purchasing the required items. Current online shopping services only allow customers to see 2D images of the products they are buying. This type of experience is not personalized and sometimes leads to bad shopping choices; the customers find it difficult to shop only with a static image view available; Page 760, Conclusion, The proposed system helps the user to view any 2D image of a product in AR view. This proposed system uses Kudan SDK to provide markerless augmented reality. The user using this application will be able to view a 3D model in various orthographic different views i.e. front view, back view, side view).
It would have been obvious to one ordinary skill in the art before the effective filing date to modify the method for predicting customer sentiment for a product (e.g., by applying a DMDE model with a self-attention fusion mechanism), wherein a new product design is generated (e.g., an image of the product that has one or more favorable predicted attributes) of the invention of Lu et al., Huang, and Song et al. to further incorporate wherein the product comprises one or more orthographic renderings of the invention of Desai because doing so would allow the customers to view a product in various orthographic different views, which helps them visualize the products better (see Desai, Abstract). Further, the claimed invention is merely a combination of old elements, and in combination each element would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claim 9 (Currently Amended), Lu et al. discloses a computer program product for predicting customer sentiment for a product and aspects thereof, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method comprising (Paragraph 0016, he present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention; Paragraph 0041, According to one embodiment, the NLP component 220 may be used to determine (e.g., infer) the user's opinion or perception as relating to the product 234 based on textual statement 238. In one embodiment, the NLP component 220 may utilize sentiment analysis and topic modeling techniques to characterize an orientation of the sentiment expressed in the user's opinions. In embodiments, the sentiment orientation may include, the polarity, tone, and/or emotions expressed in the user's opinions. In various embodiments, the sentiment orientation may be clustered into three main categories: positive, negative, and neutral sentiment):
receiving customer data for a plurality of products, and generating a vector of customer sentiments associated with different aspects of each of the plurality of products based on the customer data (Paragraph 0038, According to one embodiment, the user feedback database 208 may include one or more textual statements 238, one or more image data 240, and one or more user-defined product features 242 received from user device 212; Paragraph 0042, According to one embodiment, the aggregation component 222 may receive the textual statement 238 tagged with one or more topics (e.g., product feature) and corresponding sentiment orientations. The aggregation component 222 may apply a statistical accumulation of the sentiment orientations for each product feature to determine an aggregated feedback rating value or score (e.g., three out of five) based on the sentiment or overall evaluation of the product feature);
receiving images for each of the plurality of products, and generating a latent vector for the images for each product by … image processing model (Paragraph 0038, According to one embodiment, the user feedback database 208 may include one or more textual statements 238, one or more image data 240, and one or more user-defined product features 242 received from user device 212. In one embodiment, image data 240 may include one or more photographs of an object (e.g., product 234 or components thereof) received from the user device 212. As will be described further, in embodiments, the feedback program 110a, 110b may implement image processing techniques to generate pictorial representations of a product 234 based on the image data 240 received from user device 212 corresponding to the product 234. These pictorial representations of the products 234 may be referred to as a user image-based product representation 244 and stored in output database 210);
receiving a textual description for each of the plurality of products, and generating a latent vector for the textual description for each product by fine-tuning a pre-trained natural language processing model (Paragraph 0038, According to one embodiment, the user feedback database 208 may include one or more textual statements 238, one or more image data 240, and one or more user-defined product features 242 received from user device 212. In one embodiment, the textual statements 238 may include natural language input corresponding to: a description and/or opinion of product 234 as a whole, a description and/or opinion of one or more user-defined product features 242 of product 234, or a description and/or opinion of both—product 234 as a whole and one or more user-defined product features 242 of product 234; Paragraph 0042, According to one embodiment, the aggregation component 222 may receive the textual statement 238 tagged with one or more topics (e.g., product feature) and corresponding sentiment orientations. The aggregation component 222 may apply a statistical accumulation of the sentiment orientations for each product feature to determine an aggregated feedback rating value or score (e.g., three out of five) based on the sentiment or overall evaluation of the product feature);
applying a … model … to integrate the latent vectors of the images and the textual descriptions for each product to predict customer sentiment for new product designs based on their images and textual descriptions (Paragraph 0039, The feedback program 110a, 110b may enable the user to graphically select or annotate (e.g., via cursor control device; touchscreen) a portion of the pictorial representation (e.g., retail image 236; user image-based product representation 244) of the product 234 to dynamically register the selected portion as the user-defined product feature 242. In one embodiment, the feedback program 110a, 110b may electronically link the user-defined product feature 242 (e.g., the selected pixels) to segments of the textual statement 238 such that the descriptions/opinions in the textual statement 238 may be associated with the user-defined product feature 242. In some embodiments, the feedback program 110a, 110b may enable the user to enter a feature name for the user-defined product feature 242. In other embodiments, the feedback program 110a, 110b may automatically determine the feature name for the user-defined product feature 242 based on one or more segments of the textual statement 238 associated with the user-defined product feature 242; Paragraph 0041, According to one embodiment, the NLP component 220 may be used to determine (e.g., infer) the user's opinion or perception as relating to the product 234 based on textual statement 238. In one embodiment, the NLP component 220 may utilize sentiment analysis and topic modeling techniques to characterize an orientation of the sentiment expressed in the user's opinions. In embodiments, the sentiment orientation may include, the polarity, tone, and/or emotions expressed in the user's opinions. In various embodiments, the sentiment orientation may be clustered into three main categories: positive, negative, and neutral sentiment. In at least one embodiment, the sentiment orientation may be clustered into any number of categories. Using topic modeling, the NLP component 220 may draw out and identify the product features or components of product 234 mentioned in the textual statement 238. According to one embodiment, the association component 224 may be implemented to link the sentiment to the respective product features or components. In at least one embodiment, the association component 224 may also be implemented to link the user-defined product feature 242 to the textual statement 238 corresponding to the user-defined product feature 242; Examiner notes that Lu et al. is linking the textual statements 238 with the image user-defined features 242, wherein the linked information is used to predict/infer customer sentiment for a product and/or features of the product);
and … a … new product design having one or more attributes having favorable predicted customer sentiments using the … model … (Paragraph 0047, In at least one embodiment, the collaboration component 232 may enable the user to communicate how they fixed a problem with the product (e.g., communicating that replacing a bolt would make the product work more efficiently for a specific scenario). This may enable users to customize products to meet specific needs and share that customization with other users. In one embodiment, the collaboration component 232 may also enable the user to link to other parts which may be used to fix a broken product feature; Paragraph 0069, As described previously with reference to FIGS. 2 and 3, the feedback program 110a, 110b may employ NLP techniques to associate segments of the textual statement 520 to the product 502 and/or the user-defined product feature 510 and determine the sentiment of the user corresponding to the product 502 and/or the user-defined product feature 510; Examiner interprets “customizing the products based on the feedback” as the “new product design having one or more attributes having favorable predicted customer sentiments”).
Although Lu et al. discloses applying a model to integrate/link images and the textual descriptions for each product to predict customer sentiment (Paragraph 0069), Lu et al. does not specifically disclose wherein the model is a deep multimodal model with a self-attention fusion mechanism.
However, Huang discloses … receiving images for each of the plurality of products, and generating a latent vector for the images for each product by fine-tuning a pre-trained image processing model (Page 26, 1. Introduction, Sentiment analysis of such large-scale multimodal data can help better understand people’s attitude or opinion toward certain events or topics. For example, companies are interested in understanding how their products or brands is perceived among their customers [1–3]); Page 28, 2.2. Image Sentiment Analysis, Motivated by the powerful performance of deep models on extracting highlevel image features, Xu et al. [39] transferred VGG networks trained on ImageNet dataset into visual sentiment analysis on the sentiment datasets);
receiving a textual description for each of the plurality of products, and generating a latent vector for the textual description for each product by fine-tuning a pre-trained natural language processing model (Page 26, 1. Introduction, Sentiment analysis of such large-scale multimodal data can help better understand people’s attitude or opinion toward certain events or topics. For example, companies are interested in understanding how their products or brands is perceived among their customers [1–3]); Page 27, 2.1. Text Sentiment Analysis, Text sentiment analysis is a well-studied research area in NLP. These methods can be divided into two groups: lexicon-based methods [24–26,20] and machine learning-based methods [27– 30]; Page 29, 3.2. Semantic attention model for text sentiment analysis, Similar to image regions, some words in the text are usually more important to sentiment presentation compared to other words. Recently, semantic attention mechanism has been proven to be beneficial for many natural language processing related tasks, such as machine translation [45,46], text sentiment analysis [47, 48]. Different from these work, our semantic attention model for sentiment classification is also formulated in an end-to-end process, which can directly highlight the most important words);
applying a deep multimodal design evaluation (DMDE) model with a self-attention fusion mechanism to integrate the latent vectors of the images and the textual descriptions for each product to predict customer sentiment for new product designs based on their images and textual descriptions (Page 28, 3. Deep Multimodal Attentive Fusion, The details of the overall architecture of the proposed model Deep Multimodal Attentive Fusion are shown in Fig. 2. First, two separate unimodal attention models are proposed to learn the most discriminative features in image and text respectively. The visual attention mechanism is used to automatically focus on the affectional regions, while the semantic attention mechanism is used to highlight the most emotional words. Then, a deep intermediate fusion-based multimodal attention model is proposed to exploit the complementary and non-redundant information in different modalities. It employs a multi-layer perceptron to mine the non-linear correlation between different modalities of features. Finally, a late fusion scheme upon the three models, i.e., visual attention model, semantic attention model, and multimodal attention model, is proposed to obtain the final decision of sentiment classification);
and providing a new product design … to the DMDE model to generate customer sentiments for one or more attributes of the new product design, and … (Page 27, 2. Related Work, Sentiment analysis is an important task which has been rapidly developed in recent years. It has been applied to a broad set of applications, including product evaluation; 3. Deep Multimodal Attentive Fusion, The details of the overall architecture of the proposed model Deep Multimodal Attentive Fusion are shown in Fig. 2. First, two separate unimodal attention models are proposed to learn the most discriminative features in image and text respectively. The visual attention mechanism is used to automatically focus on the affectional regions, while the semantic attention mechanism is used to highlight the most emotional words. Then, a deep intermediate fusion-based multimodal attention model is proposed to exploit the complementary and non-redundant information in different modalities. It employs a multi-layer perceptron to mine the non-linear correlation between different modalities of features. Finally, a late fusion scheme upon the three models, i.e., visual attention model, semantic attention model, and multimodal attention model, is proposed to obtain the final decision of sentiment classification; Examiner interprets “evaluating the product” as “providing a new product design … to the DMDE model to generate customer sentiments for one or more attributes of the new product design).
It would have been obvious to one ordinary skill in the art before the effective filing date to modify the method for predicting customer sentiment for a product of the invention of Lu et al. to further incorporate wherein the method uses a deep multimodal model with a self-attention fusion mechanism of the invention of Huang because doing so would allow the method to use a deep intermediate fusion-based multimodal attention model to exploit the complementary and non-redundant information in different modalities (see Huang, Figure 2 & 3. Deep Multimodal Attentive Fusion). Further, the claimed invention is merely a combination of old elements, and in combination each element would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
The combination of Lu et al. and Huang discloses applying a Deep Multimodal Attentive Fusion model to predict customer sentiment for one or more attributes based on their images and textual descriptions (see Huang, Figure 2 & 3. Deep Multimodal Attentive Fusion). Although the combination of Lu et al. and Huang further discloses wherein the predicted customer sentiments for the one or more attributes may be used for designing a new product (e.g., product customization/evaluation), the combination of Lu et al. and Huang does not specifically disclose wherein the new product is generated using a generative design model (e.g., generative adversarial networks).
However, Song et al. discloses and providing a new product design comprising one or more [images] to the … model to generate customer sentiments for one or more attributes of the new product design (Paragraph 0052, Ranking new designs or design elements may be determined based on the frequency of exposure of a specific design or design element on images exposed to social media or websites in general, the volume and amount of sales of items that contain a specific design or design element, and the preference (determined based on all data that can estimate the preference for the image, such as clicks, feedback, and sharing) for the corresponding image exposed on the Internet, such as a specific website or social media; Paragraph 0093, The evaluation unit 406 evaluates a newly generated design or design element in the design changing unit 402 or the design synthesis unit 404 on a constant basis. Specifically, the evaluation unit 406 may rank new designs or design elements extracted in consideration of various factors; Examiner notes that an estimated preference and/or a trendy product based on customer feedback is a type of sentiment), and a second new product design having one or more attributes having favorable predicted customer sentiments and a generative design model, based solely on the one or more [images] (Paragraph 0044, In order to generate a new design, generative adversarial networks (GAN) with category-specific and random generation, Variational Autoencoder (VAE)+GAN that can randomly transform a specific design and generate a new design in a form with high similarity in characteristics of a specific design, genetic algorithm+GAN that recognizes a specific design element as a human genetic trait (looks like), generates various crosses, and repeatedly transfers new, more productive variants between generations based on feedback, conditional GAN for design change, and a style transfer technology that transforms style while maintaining the appearance of an existing design by extracting inspiration for a new design from various images and data can be used; Paragraph 0047, As an embodiment of generating a design, there is a method of generating a new design element by changing a vector value of the design element. The change of the vector value can be input as a set value, and can be made by a learned design generation model. Specifically, for example, referring to FIG. 5, if a floral dress is fashionable, the vector values of the existing floral patterns in the floral dress extracted from the collected; Paragraph 0048, Also, as another embodiment of generating a new design element, there is a method of synthesizing different elements among design elements to generate a new design for a specific item. Specifically, a new design can be generated by merging different elements among elements of each trendy design; Paragraph 0052, Ranking new designs or design elements may be determined based on the frequency of exposure of a specific design or design element on images exposed to social media or websites in general, the volume and amount of sales of items that contain a specific design or design element, and the preference (determined based on all data that can estimate the preference for the image, such as clicks, feedback, and sharing) for the corresponding image exposed on the Internet, such as a specific website or social media; Examiner notes that an estimated preference and/or a trendy product based on customer feedback is a type of sentiment).
It would have been obvious to one ordinary skill in the art before the effective filing date to modify the method for predicting customer sentiment for a product (e.g., by applying a DMDE model with a self-attention fusion mechanism) of the invention of Lu et al. and Huang to further incorporate wherein the method uses the favorable predicted customer sentiments (e.g., trends) to generate an image of the invention of Song et al. because doing so would allow the method to generate a new design based on trendy design elements (see Song et al., Paragraph 0090). Further, the claimed invention is merely a combination of old elements, and in combination each element would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Although the combination of Lu et al., Huang, and Song et al. discloses providing a new product design comprising one or more [images] to the DMDE model to generate customer sentiments for one or more attributes of the new product design (Huang, Figure 2 & 3. Deep Multimodal Attentive Fusion to obtain a final decision of sentiment classification; Song et al., Paragraph 0093, The evaluation unit 406 evaluates a newly generated design or design element in the design changing unit 402. Specifically, the evaluation unit 406 may rank new designs or design elements extracted in consideration of various factors), and a second new product design having one or more attributes having favorable predicted customer sentiments using the DMDE model and a generative design model, based solely on the one or more [images] (Huang, Figure 2 & 3. Deep Multimodal Attentive Fusion to obtain a final decision of sentiment classification; Song et al., Paragraph 0044, GAN to generate a new design; Paragraph 0090, One blouse design can be generated by merging “cuff sleeves” of the blouse, “flower-shaped collar” of the blouse, and “mint color” of the blouse, which are the elements of the trendy design). Although the combination of Lu et al., Huang, and Song et al. further discloses wherein the design generated by the generative design model is an image (e.g., generate a new image using a generative design model such as a GAN), the combination of Lu et al., Huang, and Song et al. does not specifically disclose wherein the image comprises one or more orthographic renderings.
However, Desai discloses … comprising one or more orthographic renderings …, based solely on the one or more orthographic renderings (Abstract, Augmented Reality in E-commerce allows customers to view products or experience services in their physical space before purchasing the required items. Current online shopping services only allow customers to see 2D images of the products they are buying. This type of experience is not personalized and sometimes leads to bad shopping choices; the customers find it difficult to shop only with a static image view available; Page 760, Conclusion, The proposed system helps the user to view any 2D image of a product in AR view. This proposed system uses Kudan SDK to provide markerless augmented reality. The user using this application will be able to view a 3D model in various orthographic different views i.e. front view, back view, side view).
It would have been obvious to one ordinary skill in the art before the effective filing date to modify the method for predicting customer sentiment for a product (e.g., by applying a DMDE model with a self-attention fusion mechanism), wherein a new product design is generated (e.g., an image of the product that has one or more favorable predicted attributes) of the invention of Lu et al., Huang, and Song et al. to further incorporate wherein the product comprises one or more orthographic renderings of the invention of Desai because doing so would allow the customers to view a product in various orthographic different views, which helps them visualize the products better (see Desai, Abstract). Further, the claimed invention is merely a combination of old elements, and in combination each element would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claim 17 (Currently Amended), Lu et al. discloses a computer system, comprising: at least one processor; memory associated with the at least one processor; and a program stored in the memory for predicting customer sentiment for a product and aspects thereof, the program containing a plurality of instructions which, when executed by the at least one processor, cause the at least one processor to (Paragraph 0016, he present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention; Paragraph 0041, According to one embodiment, the NLP component 220 may be used to determine (e.g., infer) the user's opinion or perception as relating to the product 234 based on textual statement 238. In one embodiment, the NLP component 220 may utilize sentiment analysis and topic modeling techniques to characterize an orientation of the sentiment expressed in the user's opinions. In embodiments, the sentiment orientation may include, the polarity, tone, and/or emotions expressed in the user's opinions. In various embodiments, the sentiment orientation may be clustered into three main categories: positive, negative, and neutral sentiment):
receiving customer data for a plurality of products, and generating a vector of customer sentiments associated with different aspects of each of the plurality of products based on the customer data (Paragraph 0038, According to one embodiment, the user feedback database 208 may include one or more textual statements 238, one or more image data 240, and one or more user-defined product features 242 received from user device 212; Paragraph 0042, According to one embodiment, the aggregation component 222 may receive the textual statement 238 tagged with one or more topics (e.g., product feature) and corresponding sentiment orientations. The aggregation component 222 may apply a statistical accumulation of the sentiment orientations for each product feature to determine an aggregated feedback rating value or score (e.g., three out of five) based on the sentiment or overall evaluation of the product feature);
receiving images for each of the plurality of products, and generating a latent vector for the images for each product by … image processing model (Paragraph 0038, According to one embodiment, the user feedback database 208 may include one or more textual statements 238, one or more image data 240, and one or more user-defined product features 242 received from user device 212. In one embodiment, image data 240 may include one or more photographs of an object (e.g., product 234 or components thereof) received from the user device 212. As will be described further, in embodiments, the feedback program 110a, 110b may implement image processing techniques to generate pictorial representations of a product 234 based on the image data 240 received from user device 212 corresponding to the product 234. These pictorial representations of the products 234 may be referred to as a user image-based product representation 244 and stored in output database 210);
receiving a textual description for each of the plurality of products, and generating a latent vector for the textual description for each product by fine-tuning a pre-trained natural language processing model (Paragraph 0038, According to one embodiment, the user feedback database 208 may include one or more textual statements 238, one or more image data 240, and one or more user-defined product features 242 received from user device 212. In one embodiment, the textual statements 238 may include natural language input corresponding to: a description and/or opinion of product 234 as a whole, a description and/or opinion of one or more user-defined product features 242 of product 234, or a description and/or opinion of both—product 234 as a whole and one or more user-defined product features 242 of product 234; Paragraph 0042, According to one embodiment, the aggregation component 222 may receive the textual statement 238 tagged with one or more topics (e.g., product feature) and corresponding sentiment orientations. The aggregation component 222 may apply a statistical accumulation of the sentiment orientations for each product feature to determine an aggregated feedback rating value or score (e.g., three out of five) based on the sentiment or overall evaluation of the product feature);
applying a … model … to integrate the latent vectors of the images and the textual descriptions for each product to predict customer sentiment for new product designs based on their images and textual descriptions (Paragraph 0039, The feedback program 110a, 110b may enable the user to graphically select or annotate (e.g., via cursor control device; touchscreen) a portion of the pictorial representation (e.g., retail image 236; user image-based product representation 244) of the product 234 to dynamically register the selected portion as the user-defined product feature 242. In one embodiment, the feedback program 110a, 110b may electronically link the user-defined product feature 242 (e.g., the selected pixels) to segments of the textual statement 238 such that the descriptions/opinions in the textual statement 238 may be associated with the user-defined product feature 242. In some embodiments, the feedback program 110a, 110b may enable the user to enter a feature name for the user-defined product feature 242. In other embodiments, the feedback program 110a, 110b may automatically determine the feature name for the user-defined product feature 242 based on one or more segments of the textual statement 238 associated with the user-defined product feature 242; Paragraph 0041, According to one embodiment, the NLP component 220 may be used to determine (e.g., infer) the user's opinion or perception as relating to the product 234 based on textual statement 238. In one embodiment, the NLP component 220 may utilize sentiment analysis and topic modeling techniques to characterize an orientation of the sentiment expressed in the user's opinions. In embodiments, the sentiment orientation may include, the polarity, tone, and/or emotions expressed in the user's opinions. In various embodiments, the sentiment orientation may be clustered into three main categories: positive, negative, and neutral sentiment. In at least one embodiment, the sentiment orientation may be clustered into any number of categories. Using topic modeling, the NLP component 220 may draw out and identify the product features or components of product 234 mentioned in the textual statement 238. According to one embodiment, the association component 224 may be implemented to link the sentiment to the respective product features or components. In at least one embodiment, the association component 224 may also be implemented to link the user-defined product feature 242 to the textual statement 238 corresponding to the user-defined product feature 242; Examiner notes that Lu et al. is linking the textual statements 238 with the image user-defined features 242, wherein the linked information is used to predict/infer customer sentiment for a product and/or features of the product);
and … a … new product design having one or more attributes having favorable predicted customer sentiments using the … model … (Paragraph 0047, In at least one embodiment, the collaboration component 232 may enable the user to communicate how they fixed a problem with the product (e.g., communicating that replacing a bolt would make the product work more efficiently for a specific scenario). This may enable users to customize products to meet specific needs and share that customization with other users. In one embodiment, the collaboration component 232 may also enable the user to link to other parts which may be used to fix a broken product feature; Paragraph 0069, As described previously with reference to FIGS. 2 and 3, the feedback program 110a, 110b may employ NLP techniques to associate segments of the textual statement 520 to the product 502 and/or the user-defined product feature 510 and determine the sentiment of the user corresponding to the product 502 and/or the user-defined product feature 510; Examiner interprets “customizing the products based on the feedback” as the “new product design having one or more attributes having favorable predicted customer sentiments”).
Although Lu et al. discloses applying a model to integrate/link images and the textual descriptions for each product to predict customer sentiment (Paragraph 0069), Lu et al. does not specifically disclose wherein the model is a deep multimodal model with a self-attention fusion mechanism.
However, Huang discloses … receiving images for each of the plurality of products, and generating a latent vector for the images for each product by fine-tuning a pre-trained image processing model (Page 26, 1. Introduction, Sentiment analysis of such large-scale multimodal data can help better understand people’s attitude or opinion toward certain events or topics. For example, companies are interested in understanding how their products or brands is perceived among their customers [1–3]); Page 28, 2.2. Image Sentiment Analysis, Motivated by the powerful performance of deep models on extracting highlevel image features, Xu et al. [39] transferred VGG networks trained on ImageNet dataset into visual sentiment analysis on the sentiment datasets);
receiving a textual description for each of the plurality of products, and generating a latent vector for the textual description for each product by fine-tuning a pre-trained natural language processing model (Page 26, 1. Introduction, Sentiment analysis of such large-scale multimodal data can help better understand people’s attitude or opinion toward certain events or topics. For example, companies are interested in understanding how their products or brands is perceived among their customers [1–3]); Page 27, 2.1. Text Sentiment Analysis, Text sentiment analysis is a well-studied research area in NLP. These methods can be divided into two groups: lexicon-based methods [24–26,20] and machine learning-based methods [27– 30]; Page 29, 3.2. Semantic attention model for text sentiment analysis, Similar to image regions, some words in the text are usually more important to sentiment presentation compared to other words. Recently, semantic attention mechanism has been proven to be beneficial for many natural language processing related tasks, such as machine translation [45,46], text sentiment analysis [47, 48]. Different from these work, our semantic attention model for sentiment classification is also formulated in an end-to-end process, which can directly highlight the most important words);
applying a deep multimodal design evaluation (DMDE) model with a self-attention fusion mechanism to integrate the latent vectors of the images and the textual descriptions for each product to predict customer sentiment for new product designs based on their images and textual descriptions (Page 28, 3. Deep Multimodal Attentive Fusion, The details of the overall architecture of the proposed model Deep Multimodal Attentive Fusion are shown in Fig. 2. First, two separate unimodal attention models are proposed to learn the most discriminative features in image and text respectively. The visual attention mechanism is used to automatically focus on the affectional regions, while the semantic attention mechanism is used to highlight the most emotional words. Then, a deep intermediate fusion-based multimodal attention model is proposed to exploit the complementary and non-redundant information in different modalities. It employs a multi-layer perceptron to mine the non-linear correlation between different modalities of features. Finally, a late fusion scheme upon the three models, i.e., visual attention model, semantic attention model, and multimodal attention model, is proposed to obtain the final decision of sentiment classification);
and providing a new product design … to the DMDE model to generate customer sentiments for one or more attributes of the new product design, and … (Page 27, 2. Related Work, Sentiment analysis is an important task which has been rapidly developed in recent years. It has been applied to a broad set of applications, including product evaluation; 3. Deep Multimodal Attentive Fusion, The details of the overall architecture of the proposed model Deep Multimodal Attentive Fusion are shown in Fig. 2. First, two separate unimodal attention models are proposed to learn the most discriminative features in image and text respectively. The visual attention mechanism is used to automatically focus on the affectional regions, while the semantic attention mechanism is used to highlight the most emotional words. Then, a deep intermediate fusion-based multimodal attention model is proposed to exploit the complementary and non-redundant information in different modalities. It employs a multi-layer perceptron to mine the non-linear correlation between different modalities of features. Finally, a late fusion scheme upon the three models, i.e., visual attention model, semantic attention model, and multimodal attention model, is proposed to obtain the final decision of sentiment classification; Examiner interprets “evaluating the product” as “providing a new product design … to the DMDE model to generate customer sentiments for one or more attributes of the new product design).
It would have been obvious to one ordinary skill in the art before the effective filing date to modify the method for predicting customer sentiment for a product of the invention of Lu et al. to further incorporate wherein the method uses a deep multimodal model with a self-attention fusion mechanism of the invention of Huang because doing so would allow the method to use a deep intermediate fusion-based multimodal attention model to exploit the complementary and non-redundant information in different modalities (see Huang, Figure 2 & 3. Deep Multimodal Attentive Fusion). Further, the claimed invention is merely a combination of old elements, and in combination each element would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
The combination of Lu et al. and Huang discloses applying a Deep Multimodal Attentive Fusion model to predict customer sentiment for one or more attributes based on their images and textual descriptions (see Huang, Figure 2 & 3. Deep Multimodal Attentive Fusion). Although the combination of Lu et al. and Huang further discloses wherein the predicted customer sentiments for the one or more attributes may be used for designing a new product (e.g., product customization/evaluation), the combination of Lu et al. and Huang does not specifically disclose wherein the new product is generated using a generative design model (e.g., generative adversarial networks).
However, Song et al. discloses and providing a new product design comprising one or more [images] to the … model to generate customer sentiments for one or more attributes of the new product design (Paragraph 0052, Ranking new designs or design elements may be determined based on the frequency of exposure of a specific design or design element on images exposed to social media or websites in general, the volume and amount of sales of items that contain a specific design or design element, and the preference (determined based on all data that can estimate the preference for the image, such as clicks, feedback, and sharing) for the corresponding image exposed on the Internet, such as a specific website or social media; Paragraph 0093, The evaluation unit 406 evaluates a newly generated design or design element in the design changing unit 402 or the design synthesis unit 404 on a constant basis. Specifically, the evaluation unit 406 may rank new designs or design elements extracted in consideration of various factors; Examiner notes that an estimated preference and/or a trendy product based on customer feedback is a type of sentiment), and a second new product design having one or more attributes having favorable predicted customer sentiments and a generative design model, based solely on the one or more [images] (Paragraph 0044, In order to generate a new design, generative adversarial networks (GAN) with category-specific and random generation, Variational Autoencoder (VAE)+GAN that can randomly transform a specific design and generate a new design in a form with high similarity in characteristics of a specific design, genetic algorithm+GAN that recognizes a specific design element as a human genetic trait (looks like), generates various crosses, and repeatedly transfers new, more productive variants between generations based on feedback, conditional GAN for design change, and a style transfer technology that transforms style while maintaining the appearance of an existing design by extracting inspiration for a new design from various images and data can be used; Paragraph 0047, As an embodiment of generating a design, there is a method of generating a new design element by changing a vector value of the design element. The change of the vector value can be input as a set value, and can be made by a learned design generation model. Specifically, for example, referring to FIG. 5, if a floral dress is fashionable, the vector values of the existing floral patterns in the floral dress extracted from the collected; Paragraph 0048, Also, as another embodiment of generating a new design element, there is a method of synthesizing different elements among design elements to generate a new design for a specific item. Specifically, a new design can be generated by merging different elements among elements of each trendy design; Paragraph 0052, Ranking new designs or design elements may be determined based on the frequency of exposure of a specific design or design element on images exposed to social media or websites in general, the volume and amount of sales of items that contain a specific design or design element, and the preference (determined based on all data that can estimate the preference for the image, such as clicks, feedback, and sharing) for the corresponding image exposed on the Internet, such as a specific website or social media; Examiner notes that an estimated preference and/or a trendy product based on customer feedback is a type of sentiment).
It would have been obvious to one ordinary skill in the art before the effective filing date to modify the method for predicting customer sentiment for a product (e.g., by applying a DMDE model with a self-attention fusion mechanism) of the invention of Lu et al. and Huang to further incorporate wherein the method uses the favorable predicted customer sentiments (e.g., trends) to generate an image of the invention of Song et al. because doing so would allow the method to generate a new design based on trendy design elements (see Song et al., Paragraph 0090). Further, the claimed invention is merely a combination of old elements, and in combination each element would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Although the combination of Lu et al., Huang, and Song et al. discloses providing a new product design comprising one or more [images] to the DMDE model to generate customer sentiments for one or more attributes of the new product design (Huang, Figure 2 & 3. Deep Multimodal Attentive Fusion to obtain a final decision of sentiment classification; Song et al., Paragraph 0093, The evaluation unit 406 evaluates a newly generated design or design element in the design changing unit 402. Specifically, the evaluation unit 406 may rank new designs or design elements extracted in consideration of various factors), and a second new product design having one or more attributes having favorable predicted customer sentiments using the DMDE model and a generative design model, based solely on the one or more [images] (Huang, Figure 2 & 3. Deep Multimodal Attentive Fusion to obtain a final decision of sentiment classification; Song et al., Paragraph 0044, GAN to generate a new design; Paragraph 0090, One blouse design can be generated by merging “cuff sleeves” of the blouse, “flower-shaped collar” of the blouse, and “mint color” of the blouse, which are the elements of the trendy design). Although the combination of Lu et al., Huang, and Song et al. further discloses wherein the design generated by the generative design model is an image (e.g., generate a new image using a generative design model such as a GAN), the combination of Lu et al., Huang, and Song et al. does not specifically disclose wherein the image comprises one or more orthographic renderings.
However, Desai discloses … comprising one or more orthographic renderings …, based solely on the one or more orthographic renderings (Abstract, Augmented Reality in E-commerce allows customers to view products or experience services in their physical space before purchasing the required items. Current online shopping services only allow customers to see 2D images of the products they are buying. This type of experience is not personalized and sometimes leads to bad shopping choices; the customers find it difficult to shop only with a static image view available; Page 760, Conclusion, The proposed system helps the user to view any 2D image of a product in AR view. This proposed system uses Kudan SDK to provide markerless augmented reality. The user using this application will be able to view a 3D model in various orthographic different views i.e. front view, back view, side view).
It would have been obvious to one ordinary skill in the art before the effective filing date to modify the method for predicting customer sentiment for a product (e.g., by applying a DMDE model with a self-attention fusion mechanism), wherein a new product design is generated (e.g., an image of the product that has one or more favorable predicted attributes) of the invention of Lu et al., Huang, and Song et al. to further incorporate wherein the product comprises one or more orthographic renderings of the invention of Desai because doing so would allow the customers to view a product in various orthographic different views, which helps them visualize the products better (see Desai, Abstract). Further, the claimed invention is merely a combination of old elements, and in combination each element would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claims 2, 10, and 18 (Original), which are dependent of claims 1, 9, and 17, the combination of Lu et al., Huang, Song et al., and Desai discloses all the limitations in claims 1, 9, and 17. Lu et al. further discloses wherein the customer data comprises customer reviews or customer survey results (Paragraph 0038, According to one embodiment, the user feedback database 208 may include one or more textual statements 238, one or more image data 240, and one or more user-defined product features 242 received from user device 212. In one embodiment, the textual statements 238 may include natural language input corresponding to: a description and/or opinion of product 234 as a whole, a description and/or opinion of one or more user-defined product features 242 of product 234, or a description and/or opinion of both—product 234 as a whole and one or more user-defined product features 242 of product 234; Paragraph 0042, According to one embodiment, the aggregation component 222 may receive the textual statement 238 tagged with one or more topics (e.g., product feature) and corresponding sentiment orientations. The aggregation component 222 may apply a statistical accumulation of the sentiment orientations for each product feature to determine an aggregated feedback rating value or score (e.g., three out of five) based on the sentiment or overall evaluation of the product feature; It can be noted that the claim language is written in alternative form. The limitation taught by Lu et al. is based on “customer reviews/opinions").
Regarding claims 3, 11, and 19 (Original), which are dependent of claims 2, 10, and 18, the combination of Lu et al., Huang, Song et al., and Desai discloses all the limitations in claims 2, 10, and 18. Lu et al. further discloses wherein the customer reviews comprise online reviews posted by customers scraped from online sources (Paragraph 0036, According to one embodiment, the product catalog database 204 may include a list of products 234 that may be provided for sale to the user by an E-commerce service; Paragraph 0038, According to one embodiment, the user feedback database 208 may include one or more textual statements 238, one or more image data 240, and one or more user-defined product features 242 received from user device 212. In one embodiment, the textual statements 238 may include natural language input corresponding to: a description and/or opinion of product 234 as a whole, a description and/or opinion of one or more user-defined product features 242 of product 234, or a description and/or opinion of both—product 234 as a whole and one or more user-defined product features 242 of product 234).
Regarding claims 4, 12, and 20 (Original), which are dependent of claims 1, 9, and 17, the combination of Lu et al., Huang, Song et al., and Desai discloses all the limitations in claims 1, 9, and 17. Lu et al. further discloses wherein generating the latent vector for the images comprises, for the customer data for each product, identifying the attributes of the product discussed in the customer data, identifying the sentiments expressed for each attribute of the product, and identifying an intensity and polarity of each sentiment (Paragraph 0039, The feedback program 110a, 110b may enable the user to graphically select or annotate (e.g., via cursor control device; touchscreen) a portion of the pictorial representation (e.g., retail image 236; user image-based product representation 244) of the product 234 to dynamically register the selected portion as the user-defined product feature 242. In one embodiment, the feedback program 110a, 110b may electronically link the user-defined product feature 242 (e.g., the selected pixels) to segments of the textual statement 238 such that the descriptions/opinions in the textual statement 238 may be associated with the user-defined product feature 242. In some embodiments, the feedback program 110a, 110b may enable the user to enter a feature name for the user-defined product feature 242. In other embodiments, the feedback program 110a, 110b may automatically determine the feature name for the user-defined product feature 242 based on one or more segments of the textual statement 238 associated with the user-defined product feature 242; Paragraph 0041, According to one embodiment, the NLP component 220 may be used to determine (e.g., infer) the user's opinion or perception as relating to the product 234 based on textual statement 238. In one embodiment, the NLP component 220 may utilize sentiment analysis and topic modeling techniques to characterize an orientation of the sentiment expressed in the user's opinions. In embodiments, the sentiment orientation may include, the polarity, tone, and/or emotions expressed in the user's opinions. In various embodiments, the sentiment orientation may be clustered into three main categories: positive, negative, and neutral sentiment. In at least one embodiment, the sentiment orientation may be clustered into any number of categories. Using topic modeling, the NLP component 220 may draw out and identify the product features or components of product 234 mentioned in the textual statement 238. According to one embodiment, the association component 224 may be implemented to link the sentiment to the respective product features or components. In at least one embodiment, the association component 224 may also be implemented to link the user-defined product feature 242 to the textual statement 238 corresponding to the user-defined product feature 242; Examiner notes that Lu et al. is linking the textual statements 238 with the image user-defined features 242, wherein the linked information is used to predict/infer customer sentiment for a product and/or features of the product).
Regarding claims 5 and 13 (Original), which are dependent of claims 4 and 12, the combination of Lu et al., Huang, Song et al., and Desai discloses all the limitations in claims 4 and 12. Lu et al. further discloses wherein generating the latent vector for the images further comprises aggregating customer sentiments identified for each product (Paragraph 0039, The feedback program 110a, 110b may enable the user to graphically select or annotate (e.g., via cursor control device; touchscreen) a portion of the pictorial representation (e.g., retail image 236; user image-based product representation 244) of the product 234 to dynamically register the selected portion as the user-defined product feature 242. In one embodiment, the feedback program 110a, 110b may electronically link the user-defined product feature 242 (e.g., the selected pixels) to segments of the textual statement 238 such that the descriptions/opinions in the textual statement 238 may be associated with the user-defined product feature 242. In some embodiments, the feedback program 110a, 110b may enable the user to enter a feature name for the user-defined product feature 242. In other embodiments, the feedback program 110a, 110b may automatically determine the feature name for the user-defined product feature 242 based on one or more segments of the textual statement 238 associated with the user-defined product feature 242; Paragraph 0041, According to one embodiment, the NLP component 220 may be used to determine (e.g., infer) the user's opinion or perception as relating to the product 234 based on textual statement 238. In one embodiment, the NLP component 220 may utilize sentiment analysis and topic modeling techniques to characterize an orientation of the sentiment expressed in the user's opinions. In embodiments, the sentiment orientation may include, the polarity, tone, and/or emotions expressed in the user's opinions. In various embodiments, the sentiment orientation may be clustered into three main categories: positive, negative, and neutral sentiment. In at least one embodiment, the sentiment orientation may be clustered into any number of categories. Using topic modeling, the NLP component 220 may draw out and identify the product features or components of product 234 mentioned in the textual statement 238. According to one embodiment, the association component 224 may be implemented to link the sentiment to the respective product features or components. In at least one embodiment, the association component 224 may also be implemented to link the user-defined product feature 242 to the textual statement 238 corresponding to the user-defined product feature 242; Examiner notes that Lu et al. is linking the textual statements 238 with the image user-defined features 242, wherein the linked information is used to predict/infer customer sentiment for a product and/or features of the product).
Regarding claims 6 and 14 (Original), which are dependent of claims 1 and 9, the combination of Lu et al., Huang, Song et al., and Desai discloses all the limitations in claims 1 and 9. Although Lu et al. discloses designing a model to combine/link images and the textual descriptions for each product to predict customer sentiment for new product designs based on their images and textual descriptions (Paragraph 0069), Lu et al. does not specifically disclose wherein the images and the textual description for each product are combined prior to designing the DMDE model using a multimodal data concatenation process.
However, Huang discloses combining the latent vector for the textual description for each product and the latent vector for the images for each product prior to designing the DMDE model using a multimodal data concatenation process (Page 28, 3. Deep Multimodal Attentive Fusion, The details of the overall architecture of the proposed model Deep Multimodal Attentive Fusion are shown in Fig. 2. First, two separate unimodal attention models are proposed to learn the most discriminative features in image and text respectively. The visual attention mechanism is used to automatically focus on the affectional regions, while the semantic attention mechanism is used to highlight the most emotional words. Then, a deep intermediate fusion-based multimodal attention model is proposed to exploit the complementary and non-redundant information in different modalities. It employs a multi-layer perceptron to mine the non-linear correlation between different modalities of features. Finally, a late fusion scheme upon the three models, i.e., visual attention model, semantic attention model, and multimodal attention model, is proposed to obtain the final decision of sentiment classification; Examiner notes that the features are concatenated/fused prior to designing the model).
PNG
media_image1.png
270
552
media_image1.png
Greyscale
Regarding claims 7 and 15 (Original), which is dependent of claims 1 and 9, the combination of Lu et al., Huang, Song et al., and Desai discloses all the limitations in claims 1 and 9. Although Lu et al. discloses designing a model to combine/link images and the textual descriptions for each product to predict customer sentiment for new product designs based on their images and textual descriptions (Paragraph 0069), Lu et al. does not specifically disclose wherein the model is a deep multimodal model.
However, Huang further discloses wherein the DMDE model … comprise neural network models (Page 26, 1. Introduction, Multimodal sentiment analysis has gained increasing attention in recent years. Based on the combination strategies for the multimodal contents, these methods can be categorized into three groups, namely early fusion [7–9], intermediate fusion [10–12] and late fusion [13–15]. The early fusion-based methods integrate multiple sources of data into a single feature vector before being used as the input to a machine learning algorithm. However, early fusion of multimodal data is not effective to capture the complementary nature of the modalities involved and may lead to large input vectors that may contain redundancies. Late fusion refers to the aggregation of decisions from multiple sentiment classifiers, each trained on separate modalities. This method cannot effectively capture the correlation between different modalities. Intermediate fusion, which is mainly implemented with neural networks, refers to that the fusion process is conducted in the intermediate layers of the whole networks).
It would have been obvious to one ordinary skill in the art before the effective filing date to modify the method for predicting customer sentiment for a product of the invention of Lu et al. to further incorporate wherein the method uses a deep multimodal model with a self-attention fusion mechanism of the invention of Huang because doing so would allow the method to conduct the fusion process in the intermediated layers of the whole networks (see Huang, Page 26, 1. Introduction). Further, the claimed invention is merely a combination of old elements, and in combination each element would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
The combination of Lu et al. and Huang discloses designing a Deep Multimodal Attentive Fusion model to predict customer sentiment for one or more attributes based on their images and textual descriptions (see Huang, Page 26, 1. Introduction, Intermediated fusion implemented with neural networks). Although the combination of Lu et al. and Huang further discloses wherein the deep multimodal model comprises neural network models, the combination of Lu et al. and Huang does not specifically disclose a generative design model (e.g., generative adversarial networks), wherein the generative design model comprises neural network models.
However, Song et al. discloses wherein the … the generative design model comprise neural network models (Paragraph 0044, In order to generate a new design, generative adversarial networks (GAN) with category-specific and random generation, Variational Autoencoder (VAE)+GAN that can randomly transform a specific design and generate a new design in a form with high similarity in characteristics of a specific design, genetic algorithm+GAN that recognizes a specific design element as a human genetic trait (looks like), generates various crosses, and repeatedly transfers new, more productive variants between generations based on feedback, conditional GAN for design change, and a style transfer technology that transforms style while maintaining the appearance of an existing design by extracting inspiration for a new design from various images and data can be used; Paragraph 0090, In detail, referring to FIG. 6, among the trendy design elements, a knit “puff sleeve,” a shirt “ultra violet,” and a “long dress” are synthesized, so that designs with ultra-violet index long dresses with puff sleeves can be generated. It is also possible to synthesize different elements of the same item. For example, one blouse design can be generated by merging “cuff sleeves” of the blouse, “flower-shaped collar” of the blouse, and “mint color” of the blouse, which are the elements of the trendy design; Paragraph 0093, The evaluation unit 406 evaluates a newly generated design or design element in the design changing unit 402 or the design synthesis unit 404 on a constant basis. Specifically, the evaluation unit 406 may rank new designs or design elements extracted in consideration of various factors; Examiner notes that the GAN is a type of neural network architecture).
It would have been obvious to one ordinary skill in the art before the effective filing date to modify the method for predicting customer sentiment for a product of the invention of Lu et al. and Huang to further incorporate wherein the method uses the favorable predicted customer sentiments to generate an image of the invention of Song et al. because doing so would allow the method to generate a new design based on trendy design elements (see Song et al., Paragraphs 0044 & 0090, generate a new design using GAN). Further, the claimed invention is merely a combination of old elements, and in combination each element would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claims 8 and 16 (Original), which are dependent of claims 1 and 9, the combination of Lu et al., Huang, Song et al., and Desai discloses all the limitations in claims 1 and 9. Lu et al. further discloses wherein the images comprise multiple different views of each of the plurality of products (Paragraph 0040, According to one embodiment, the feedback program 110a, 110b may implement the user interaction component 218 to enable the user (e.g., via UI 216 of user device 212) to interact directly with the pictorial representations (e.g., retail image 236; user image-based product representation 244) of the product 234. In one embodiment, the user may directly manipulate (e.g., via rotation control; zoom control) the pictorial representations of the product 234 (e.g., via UI 216 of user device 212) to glean user feedback information regarding the products 234. For example, the user may zoom in and pinpoint a product feature (e.g., in the pictorial representation) to extract user feedback information corresponding to that product feature; Examiner interprets “rotating images” as the “multiple different views of each of the plurality of products”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Clark et al. (US 2019/0139058 A1) – discloses ingesting the product reviews for multiple products by a processor using a natural language processing, in which each product embodies a specific form for each of the product features. The ingested product reviews are analyzed for sentiments associated with the specific forms by the processor. A sentiment score for each product feature is generated based on the analyzing by the processor and the product features are ranked based on the sentiment scores by the processor (see at least Abstract).
Bao et al. (US 2017/0061454 A1) – discloses a method, device and computer program product for product design based on user reviews. Reviews on a product are obtained from a plurality of users. These reviews are analyzed to determine sentiments of the users with respect to a property of the product. A plurality of candidate product designs are generated by changing a first value of the property based on the sentiments of the users. Then one or more new product designs are obtained based on the candidate product designs (see at least Abstract).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARJORIE PUJOLS-CRUZ whose telephone number is (571)272-4668. The examiner can normally be reached Mon-Thru 7:30 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Patricia H Munson can be reached at (571)270-5396. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.P./Examiner, Art Unit 3624 /PATRICIA H MUNSON/Supervisory Patent Examiner, Art Unit 3624
Prosecution Insights