DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Applicant’s response filed on 10/06/2025 has been fully considered.
Claims 1 and 15-16 are amended.
Claims 1-16 are currently pending and have been examined.
Priority
Examiner acknowledges Applicant’s claim of foreign priority to Japanese Patent Application No. JP 2021-101057, filed on 6/17/2021. Examiner accepts this priority date.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-16 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception without significantly more. The claims recite an abstract idea. The judicial exception is not integrated into a practical application. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Under Step 1 of the Subject Matter Eligibility Test for Products and Processes, the claims must be directed to one of the four statutory categories (See MPEP 2106.03(II)). In the instant application, claims 1-14 are directed to an apparatus, claim 15 is directed to a process, and claim 16 is directed to a manufacture. All the claims are directed to one of the four statutory categories (YES).
Under Step 2A of the Patent Subject Matter Eligibility Test (see MPEP 2106.04), it is determined whether the claims are directed to a judicially recognized exception. Step 2A is a two-prong inquiry.
Under Prong 1, it is determined whether the claim recites a judicial exception (See MPEP 2106.04). (YES). Taking Claim 1 as representative, the claim recites limitations that fall within the certain methods of organizing human activity groupings of abstract ideas, including:
An information processing device comprising:
a processor, wherein the processor is configured to:
obtain a base image, the base image being a photograph that includes a transaction
generate an illustration image from the base image using a machine-learning model, wherein the machine-learning model is learned using a convolution neural network (CNN) and the illustration image includes an illustration-prepared transaction target, wherein the machine-learning model comprises a generation model that outputs the illustration image from the base image;
receive editing performed by the user on the illustration-prepared transaction target and analyze the editing to determine user emphasis on specific features of the transaction target; and
perform inverse conversion on an edited illustration image to generate a converted image using the CNN, the converted image being a photographic image converted from the edited illustration image;
provide the converted image to a search server that conducts an image search, wherein the edited illustration image being the illustration image that has been edited by the user.
Independent claims 15 and 16 recite similar processes as claim 1. Claim 16 further recites the abstract idea of a generation process of generating rankings that correspond to price or selling history of the one or more transaction targets.
Certain methods of organizing human activity include:
fundamental economic principles or practices (including hedging, insurance, and mitigating risk)
commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; and business relations)
managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions)
The limitations, as emphasized above, recite the concept of performing a search for a transaction target (i.e. a product for purchase). These limitations, under their broadest reasonable interpretations, fall with the “Certain Methods of Organizing Human Activity” grouping of abstract ideas, enumerated in the MPEP because these limitations recite a commercial interaction (such as marketing or sales activities). Specifically, the claims recite obtaining a base image including a transaction target, receiving editing performed on an illustration of the transaction target, converting the image, and conducting a search based on the image. Applicant’s specification describes that “the transaction target may be any kind of target for transaction, such as products or services, and examples include products, services or the like that have been exhibited in a cybermall. Description is provided below by using sneakers (shoes) serving as a product, as an example of a transaction target. However, the transaction target may be any type of target…such as the layout of a real estate property” (see specification page 6) and also recites that the search is performed via a cybermall, i.e. an e-commerce platform (see specification pages 10-11). The dependent claims clarify that the search is performed in order to obtain “a similar transaction target that is similar to the illustration-prepared transaction target” (claim 3). Accordingly, independent claims 1 and 15-16 as a whole, are directed towards certain methods of organizing human activity reciting commercial interaction. The claims recite an abstract idea.
Under Prong 2, it is determined whether the claim recites additional elements that integrate the exception into a practical application of the exception. This judicial exception is not integrated into a practical application (NO).
Independent claims 1 and 15-16 recite additional elements beyond the abstract idea, emphasized below, including:
an information processing device comprising: a processor, wherein the processor is configured to perform operations
a machine-learning model, wherein the machine-learning model is learned using a convolution neural network (CNN)
a generation model
a search server
a computer
a non-transitory computer readable storage medium having stored an information processing program that causes a computer to execute a method
a Cycle Generative Adversarial Network (CycleGAN)
The additional elements of claims 1 and 15-16 are recited at a high level of generality (i.e. as generic computing hardware) in Applicant’s specification without meaningful detail regarding their structure, configuration, or function. As such, the additional elements amount to nothing more than mere instructions to implement or apply the abstract idea on a generic computing hardware (or, merely use a computer as a tool to perform an abstract idea.) Specifically, the additional element of the search server is recited at a high level of generality (i.e., a generic computer hardware performing a search query) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Further, the additional elements do no more than generally link the use of the judicial exception to a particular technological environment or field of use (such as computers or computing networks). For example, stating that the illustration image is generated “using a machine learning model” that is “learned using a convolution neural network (CNN)” only generally links the commercial interactions to a computer environment. Employing well-known computer functions to execute an abstract idea, even when limiting the use of the idea to one particular environment, does not integrate the exception into a practical application.
Additionally, the additional elements are insufficient to integrate the abstract idea into a practical application because the claim fails to i) reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, ii) apply the judicial exception with, or use the judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim, iii) effect a transformation or reduction of a particular article to a different state or thing, or iv) apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment.
Accordingly, the judicial exception is not integrated into a practical application.
Under Step 2B, it is determined whether the claims recite additional elements that amount to significantly more than the judicial exception. The claims of the present application do not include additional elements that are sufficient to amount to significantly more than the judicial exception (NO).
As discussed above with respect to Prong Two of Step 2A, although additional computer related elements are recited, the claim merely invokes such additional elements as a tool to perform the abstract idea. See MPEP 2106.05(f). Moreover, the limitations of claims 1 and 15-16 are manual processes (e.g., obtaining an image, preparing an illustration, editing the illustration, etc.). The courts have indicated that mere automation of manual processes is not sufficient to show an improvement in computer-functionality (see MPEP 2106.05(a)(I)). Furthermore, as discussed above with respect to Prong Two of Step 2A, claims 1 and 15-16 merely recite the additional elements in order to further define the field of use of the abstract idea, therein attempting to generally link the use of the abstract idea to a particular technological environment, such as the Internet or computing networks (see Ultramercial, Inc. v. Hulu, LLC. (Fed. Cir. 2014); Bilski v. Kappos (2010); MPEP 2106.05(h)). Similar to FairWarning v. Iatric Sys., claims 1 and 15-16 specifying that the abstract idea of performing a search for a product merely indicates a field of use in which to apply the abstract idea because this requirement merely limits the claim to the computer field, i.e., to execution on a generic computer.
Even when considered as an ordered combination, the additional elements do not add anything that is not already present when they are considered individually. In Alice Corp., the Court considered the additional elements “as an ordered combination,” and determined that “the computer components…‘[a]dd nothing…that is not already present when the steps are considered separately’ and simply recite intermediated settlement as performed by a generic computer.” Id. (citing Mayo, 566 U.S. at 79, 101 USPQ2d at 1972). Similarly, viewed as a whole, claims 1 and 15-16 simply convey the abstract idea itself facilitated by generic computing components. Therefore, under Step 2B of the Alice/Mayo test, there are no meaningful limitations in claims 1 and 15-16 that transform the judicial exception into a patent eligible application such that the claim amounts to significantly more than the judicial exception itself.
Even considered as an ordered combination (as a whole), the additional elements do not add anything significantly more than when considered individually. Therefore, claims 1 and 15-16 do not provide an inventive concept and do not qualify as eligible subject matter.
Claims 2-14 are dependencies of claims 1. When analyzed, as a whole, the dependent claims are held to be patent-ineligible under 35 U.S.C. 101 because they are directed to the judicial exception, do not integrate the judicial exception into a practical application, nor do they add “significantly more” to the judicial exception. More specifically, the claims 2-14 further fall within the “Certain Methods of Organizing Human Activity” grouping of abstract ideas, enumerated in the MPEP, in that they further recite commercial or legal interactions, such as advertising, marketing, or sales activities or behaviors and managing personal behavior or relationships or interactions between people. Dependent claims 2-14 do not recite further additional elements but rather recite limitations that further define the abstract idea. Therefore, claims 2-14 are not indicative of integration into a practical application and are not significantly more than the judicial exception.
Accordingly, under the Alice/Mayo test, the Examiner concludes that there are no meaningful
limitations in claims 1-16 that transform the judicial exception into a patent eligible application
such that the claim amounts to significantly more than the judicial exception itself. The analysis
above applies to all statutory categories of invention.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-5, 8, and 10-16 are rejected under 35 U.S.C. 103 as being unpatentable over previously cited Chow et al. (U.S. 2022/0172123) A1 in view of previously cited NPL Reference Yi et al. (initially cited in the Office action dated 10/25/2024) in view of previously cited Zhang et al. (US 20200073968 A1).
Regarding claim 1, Chow et al. (hereinafter Chow) discloses:
An information processing device comprising (See a computer 400 that may be configured as a device for executing the methods of Fig. 2, according to exemplary embodiments [0080], Figs. 2, 4. See further that the operations of Fig. 2 may be performed by one or more processors of a computer systems, where a processor may be a central processing unit [0078]):
a processor, wherein the processor is configured to (See a computer 400 that may be configured as a device for executing the methods of Fig. 2, according to exemplary embodiments [0080], Figs. 2, 4. See further that the operations of Fig. 2 may be performed by one or more processors of a computer systems, where a processor may be a central processing unit [0078]):
obtain a base image (See 205 that the image-based search system 130 may provide a visual prompt to the user, where the visual prompt is based on the selected sub-category for the product category. The image-based search system 130 may provide at least apportion of the prompt physically or may provide a portion of the visual prompt electronically (via a display of the user device). In some embodiments, the visual prompt includes a template associated with the product category and/or the product sub-category. The template may be selected from a plurality of templates that correspond to different view angles of a product [0038-0039], Figs. 1-2. Examiner interprets the template as the base image), the base image being a photograph that includes a transaction(See a process for identifying one or more products that correspond to product features of interest to a customer. A customer may desire information or may desire to purchase a particular product in a particular product category having a particular loo or feel, and/or particular features. The user may select a category of products (e.g., vehicles, real estate, jewelry, food, etc.) or may select a sub-category of products (e.g., sport car, sedan, truck, van, etc.). The visual prompt provided to the user at step 205 may be based on the category or sub-category selected [0035-0038], Fig. 2. Examiner interprets the user selection of the category and sub-category as the transaction target specified by the user);
generate an illustration image from the base image (See that the template may be provided to the user and the template may comprise features associated with the category and/or sub-category. In digital form, the features may be in the form of electronic objects that may be manipulated by the user [0039]. See the customer making a freehand or sketched marking on the template in order to form an illustration [0043], Fig. 1. Examiner interprets performing the initial sketch as generating the illustration image) using a machine-learning model, wherein the machine-learning model is learned using a convolution neural network (CNN) and the illustration image includes an illustration-prepared transaction target (See that the freehand and/or sketched portions may include one or more features for the product, one or more colors, shadings, line weights, levels of detail, etc. (i.e., the illustration-prepared transaction target). At step 210, the system 130 may receive the freeform or sketched illustration [0042-0044], Fig. 2), wherein the machine-learning model comprises a generation model that outputs the illustration image from the base image (See that the template may be provided to the user and the template may comprise features associated with the category and/or sub-category. In digital form, the features may be in the form of electronic objects that may be manipulated by the user [0039]. See the customer making a freehand or sketched marking on the template in order to form an illustration [0043], Fig. 1. Examiner interprets performing the initial sketch as generating the illustration image);
receive editing performed by the user on the illustration-prepared transaction target and analyze the editing to determine user emphasis on specific features of the transaction target (See that the user may manipulate one or more features onto the electronic depiction in order to modify the template. The user may generate an at least partially freeform or sketched illustration of the desired product. The user may make a freehand or sketched marking on the template to form the illustration. In some embodiment, the user may use a mouse, keyboard, touch screen, stylus, or the like to interact with, for example, an electronic drawing application associated with the user device 105, the vendor system 110, the image-based search system 130, or the like. The freehand and/or sketched portions may include one or more features for the product, one or more colors, shadings, line weights, levels of detail, etc. At step 210, the system 130 may receive the freeform or sketched illustration [0042-0044], Fig. 2. See identifying one or more products that correspond to product features of interest to a user may include: a memory storing instructions and a machine-learning model trained, using (i) at least partially freeform or sketched illustrations from various users depicting products in a product category and (ii) feature labels assigned to the illustrations, to output one or more features of a depicted product in the product category in an input illustration; and a processor operatively connected to the memory and configured to execute the instructions to perform acts. The acts may include: receiving an at least partially freeform or sketched illustration of a product from a user device associated with a user; determining at least one feature of the product depicted in the illustration by employing the machine-learning model, wherein the at least one feature corresponds to a freeform or sketched portion of the received illustration [0008]. Examiner interprets the user making multiple sketched and freeform illustrations as reading on the editing); and
perform inverse conversion on an edited illustration image to generate a converted image using the CNN, the converted image being a photographic image converted from the edited illustration image ( See that the image based search system 130 may receive the illustration from the user device 105 (i.e. providing the edited illustration image to a search system) [0044], Fig. 1, 2. See further determining at least one feature of the product depicted in the illustration [0047], Fig. 2. the first machine-learning model may be and/or include a convolutional neural network (a “CNN”). A CNN may include one or more layers, e.g., neural nodes, that may include one or more filters that are convolved over an input image in order to identify one or more features present in the input image [0053]; In response, 225 the image-based search system 130 may generate a search query based on a determined at least one feature from the illustration received at 210. The search query may include and/or be based on information associated with one or more aspects of the determined at least one feature (e.g., orientation, size, etc.). The search query may be based on emphasis score determined for one or more of the at least one features [0065], Fig. 2);
provide the converted image to a search server that conducts an image search (See that the image based search system 130 may receive the illustration from the user device 105 (i.e. providing the edited illustration image to a search system) [0044], Fig. 1, 2. See further determining at least one feature of the product depicted in the illustration [0047], Fig. 2. In response, 225 the image-based search system 130 may generate a search query based on a determined at least one feature from the illustration received at 210. The search query may include and/or be based on information associated with one or more aspects of the determined at least one feature (e.g., orientation, size, etc.). The search query may be based on emphasis score determined for one or more of the at least one features [0065], Fig. 2), wherein the edited illustration image being the illustration image that has been edited by the user (See that in some embodiment, the user may use a mouse, keyboard, touch screen, stylus, or the like to interact with, for example, an electronic drawing application associated with the user device 105, the vendor system 110, the image-based search system 130, or the like. The freehand and/or sketched portions may include one or more features for the product, one or more colors, shadings, line weights, levels of detail, etc. At step 210, the system 130 may receive the freeform or sketched illustration [0043-0044], Fig. 2).
Chow does not explicitly disclose:
the base image being a photograph;
generate an illustration image from the base image using a machine-learning model, wherein the machine-learning model is learned using a convolution neural network (CNN);
wherein the machine-learning model comprises a generation model that outputs the illustration image from the base image;
perform inverse conversion on an edited illustration image to generate a converted image using the CNN, the converted image being a photographic image converted from the edited illustration image;
provide the converted image to a search server that conducts an image search
However, Chow does disclose loading from one computer or processor into another, for example, from a management server or host computer of a mobile communications network into the computer platform of a server and/or from a server to the mobile device (see Chow [0081]). Chow also discloses providing an in illustration from a mobile device 105 to an image-based search system 103 (see Chow [0044]) and that the image-based search system may be comprised by a computer 400 with a processor 402, memory 404, display, 410, computer readable medium 422, etc. communicating over a network 125 (see Chow [0080], Fig. 4). Thus, while the search system 130 is not explicitly described as a server, one could interpret the search system 130 to teach the server. Nevertheless, additional art has been cited to teach a search server.
Further, while Chow does not explicitly disclose the use of machine learning to generate an illustration image, Chow does disclose generating an illustration/sketchpad image (see Chow [0039], [0042]) as well as utilizing a first machine learning model being trained using at least partially freeform or sketched illustrations from various users depicting products and feature labels assigned to the illustrations, where the first machine learning model may include and/or be a convolutional neural network (CNN) (see Chow [0052-0053]). This machine learning model is used to recognize items that are sketched by the user to perform a search query (see Chow [0025]). Chow does not explicitly disclose that the machine learning model is used to generate the illustration image but does disclose the presence of a machine learning model trained by a CNN. Additional art has been cited to teach using a machine learning model to generate a modifiable image from a photograph.
Previously cited NPL Reference Ran Yi et al. (hereinafter Yi), on the other hand, teaches:
the base image being a photograph (Transforming input photos into drawings. Page 3465);
generate an illustration image from the base image using a machine-learning model, wherein the machine-learning model is learned using a convolution neural network (CNN), wherein the machine-learning model comprises a generation model that outputs the illustration image from the base image (See applying a convolutional neural network, specifically a generative adversarial network (GAN), “APDrawingGAN++” to transform a test photo into a line drawing. Pages 3462-3463, Fig. 1(a))
PNG
media_image1.png
453
706
media_image1.png
Greyscale
Chow and Yi disclose systems and methods directed to image recognition and modification. These references are in the same field of endeavor and are therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of Chow with the teachings of Yi. The claimed invention is merely a combination of existing elements, and in combination, each element would have functioned the same as they would separately. One of ordinary skill in the art would have recognized that the results were repeatable and would have allowed for a higher-quality image style transfer that makes small features clearly visible (see Yi Pages 3462-3463).
Additionally, previously cited Zhang et al. (hereinafter Zhang), on the other hand, teaches:
perform inverse conversion on an edited illustration image to generate a converted image using the CNN, the converted image being a photographic image converted from the edited illustration image (Zhang: [0029] – “sketch queries received by the SBIR system 150 may initially be converted to synthetic images 180 prior to searching for authentic images 190 corresponding to the sketches 170 that are subject of the queries”);
provide the converted image to a search server that conducts an image search (Zhang: [0029] – “sketch queries received by the SBIR system 150 may initially be converted to synthetic images 180 prior to searching for authentic images 190 corresponding to the sketches 170 that are subject of the queries. This domain-migration function enables the SBIR system 150 to generate more effective mappings between the sketch and image domains, thus allowing for more accurate identification of the retrieval results 160”; [0009] and Fig. 4 – “Hash codes can be generated from the synthetic images to be used as a basis for querying an image database”; [0025] and Fig. 1 – “an SBIR system 150 is stored on one or more servers 120”).
Chow/Yi and Zhang disclose systems and methods directed to image recognition and searching. These references are in the same field of endeavor and are therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of Chow/Yi with the teachings of Zhang. The claimed invention is merely a combination of existing elements, and in combination, each element would have functioned the same as they would separately. One of ordinary skill in the art would have recognized that the results were repeatable and would have allowed for more accurate identification of the retrieval results (Zhang: [0029]).
Regarding claim 2, the combination of Chow/Yi/Zhang teaches the information processing device (hereinafter the device) of claim 1. Chow further discloses:
wherein the processor is further configured to: obtain, from the search server, a search result based on the edited illustration image (See 225 generating a search query based on a determined at least one feature, e.g., orientation, position, size, etc. 230 that the image-based system 130 may identify at least one product in an inventory, e., the inventory of the vendor system 110 or the like, that corresponds to the search query. In some embodiments, the inventory may include information associated with a plurality of products and respective one or more features associated with each product [0065-0066], Figs. 1-2. See 215 that the determined at least one feature is a feature of a product depicted in the illustration [0047], Figs. 1-2).
Chow does not explicitly disclose: a search server.
However, Chow does disclose loading from one computer or processor into another, for example, from a management server or host computer of a mobile communications network into the computer platform of a server and/or from a server to the mobile device (see Chow [0081]). Chow also discloses providing an in illustration from a mobile device 105 to an image-based search system 103 (see Chow [0044]) and that the image-based search system may be comprised by a computer 400 with a processor 402, memory 404, display, 410, computer readable medium 422, etc. communicating over a network 125 (see Chow [0080], Fig. 4). Thus, while the search system 130 is not explicitly described as a server, one could interpret the search system 130 to teach the server. Nevertheless, additional art has been cited.
Zhang, on the other hand, teaches: a search server ([0025] and Fig. 1 – “an SBIR system 150 is stored on one or more servers 120”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed inventions to combine Zhang with Chow/Yi for the reasons identified above with respect to claim 1.
Regarding claim 3, the combination of Chow/Yi/Zhang teaches the device of claim 2. Chow further discloses: wherein the processor is further configured to: obtain, from the search server, the search result indicating a similar transaction target that is similar to the illustration-prepared transaction target (See 230 that the image-based system 130 may identify at least one product in an inventory, e., the inventory of the vendor system 110 or the like, that corresponds to the search query. In some embodiments, the inventory may include information associated with a plurality of products and respective one or more features associated with each product [0066], Figs. 1-2).
Chow does not explicitly disclose: a search server.
However, Chow does disclose loading from one computer or processor into another, for example, from a management server or host computer of a mobile communications network into the computer platform of a server and/or from a server to the mobile device (see Chow [0081]). Chow also discloses providing an in illustration from a mobile device 105 to an image-based search system 103 (see Chow [0044]) and that the image-based search system may be comprised by a computer 400 with a processor 402, memory 404, display, 410, computer readable medium 422, etc. communicating over a network 125 (see Chow [0080], Fig. 4). Thus, while the search system 130 is not explicitly described as a server, one could interpret the search system 130 to teach the server. Nevertheless, additional art has been cited.
Zhang, on the other hand, teaches: a search server ([0025] and Fig. 1 – “an SBIR system 150 is stored on one or more servers 120”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed inventions to combine Zhang with Chow/Yi for the reasons identified above with respect to claim 1.
Regarding claim 4, the combination of Chow/Yi/Zhang teaches the device of claim 1. Chow further discloses:
wherein the processor is further configured to: receive a change in color that has been performed by the user on the illustration-prepared transaction target (See the template may include one or more electronic objects that represent one or more features available with various products, and that the customer may manipulate, e.g., position, orient, color, resize the one or more features to modify the template [0042]. See that the freehand and/or sketched portion of the illustration may include one or more colors, shadings, line weights, levels of detail, etc. [0043]), and
provide the search server with the edited illustration image in which the color of the illustration-prepared transaction target has been changed by the user (See after receiving the freeform sketched illustration of the product at step 210, the search is performed in step 225. Fig. 2. See that the search query is generated based on information associated with at least one feature emphasized by the user [0065], Fig. 2).
Chow does not explicitly disclose: a search server.
However, Chow does disclose loading from one computer or processor into another, for example, from a management server or host computer of a mobile communications network into the computer platform of a server and/or from a server to the mobile device (see Chow [0081]). Chow also discloses providing an in illustration from a mobile device 105 to an image-based search system 103 (see Chow [0044]) and that the image-based search system may be comprised by a computer 400 with a processor 402, memory 404, display, 410, computer readable medium 422, etc. communicating over a network 125 (see Chow [0080], Fig. 4). Thus, while the search system 130 is not explicitly described as a server, one could interpret the search system 130 to teach the server. Nevertheless, additional art has been cited.
Zhang, on the other hand, teaches: a search server ([0025] and Fig. 1 – “an SBIR system 150 is stored on one or more servers 120”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed inventions to combine Zhang with Chow/Yi for the reasons identified above with respect to claim 1.
Regarding claim 5, the combination of Chow/Yi/Zhang teaches the device of claim 1. Chow further discloses:
wherein the processor is further configured to: receive a change in a shape that has been performed by the user on the illustration-prepared transaction target (See 220 determining a respective emphasis score for one or more of at least one determined feature depicted in the illustration. The respective emphasis score may be based on one or more of a coloration, a size, a line wight, a line style, a shading, a level of detail, or a predetermined shape of a freeform or sketched portion of the illustration corresponding to the at least one feature [0056-0057], Fig. 2), and
provide the search server with the edited illustration image in which the shape of the illustration-prepared transaction target has been changed by the user (See after receiving the freeform sketched illustration of the product at step 210, the search is performed in step 225. Fig. 2. See that the search query is generated based on information associated with at least one feature emphasized by the user [0065], Fig. 2).
Chow does not explicitly disclose: a search server.
However, Chow does disclose loading from one computer or processor into another, for example, from a management server or host computer of a mobile communications network into the computer platform of a server and/or from a server to the mobile device (see Chow [0081]). Chow also discloses providing an in illustration from a mobile device 105 to an image-based search system 103 (see Chow [0044]) and that the image-based search system may be comprised by a computer 400 with a processor 402, memory 404, display, 410, computer readable medium 422, etc. communicating over a network 125 (see Chow [0080], Fig. 4). Thus, while the search system 130 is not explicitly described as a server, one could interpret the search system 130 to teach the server. Nevertheless, additional art has been cited.
Zhang, on the other hand, teaches: a search server ([0025] and Fig. 1 – “an SBIR system 150 is stored on one or more servers 120”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed inventions to combine Zhang with Chow/Yi for the reasons identified above with respect to claim 1.
Regarding claim 8, the combination of Chow/Yi/Zhang teaches the device of claim 1. Chow further discloses:
wherein the processor is further configured to: obtain the base image of the transaction target having a shape specified by the user (See when customer may desire a sports car having a particular look and feel, the image-based search system 130 provides the customer, e.g., via a display associated with the user device 105 or vendor system 110, an electronic depiction of a template associated with sports cars. The template may include a driver, a driver's seat, at least a portion of a chassis of a car, and four wheels. The template may also include one or more electronic objects that represent one or more features available with various sports cars, e.g., various different doors, front hoods, trunks, headlights, exhausts, grills, etc. [0042], Fig. 1. Examiner interprets the template as the base image, the desired sports car as the transaction target, and the seat, chassis, wheels, and electronic objects as making up the shape).
Regarding claim 10, the combination of Chow/Yi/Zhang teaches the device of claim 1. Chow further discloses:
wherein the processor is further configured to: obtain the base image that corresponds to the transaction target being a product specified by the user (See 205 that the image-based search system 130 may provide a visual prompt to the user, where the visual prompt is based on the selected sub-category for the product category. The image-based search system 130 may provide at least apportion of the prompt physically or may provide a portion of the visual prompt electronically (via a display of the user device). In some embodiments, the visual prompt includes a template associated with the product category and/or the product sub-category. The template may be selected from a plurality of templates that correspond to different view angles of a product [0038-0039], Figs. 1-2. Examiner interprets the template as the base image).
Regarding claim 11, the combination of Chow/Yi/Zhang teaches the device of claim 1. Chow further discloses:
wherein the processor is further configured to: generate the illustration image from the base image (See the user/customer generating an at least partially freeform or sketched illustration of the desired product. The user may make a freehand or sketched marking on the template (i.e. the base image) to form the illustration [0043]), wherein
receive editing that has been performed by the user on the illustration image (See that in some embodiment, the user may use a mouse, keyboard, touch screen, stylus, or the like to interact with, for example, an electronic drawing application associated with the user device 105, the vendor system 110, the image-based search system 130, or the like. The freehand and/or sketched portions may include one or more features for the product, one or more colors, shadings, line weights, levels of detail, etc. At step 210, the system 130 may receive the freeform or sketched illustration [0043-0044], Fig. 2).
Regarding claim 12, the combination of Chow/Yi/Zhang teaches the device of claim 11. Chow further discloses:
wherein the processor is further configured to: generate the illustration image (See that the user, e.g., the customer, may generate an at least partially freeform or sketched illustration of the desired product. The user may make a freehand or sketched marking on the template to form the illustration. The freehand and/or sketched portions of the illustration may include one or more colors, shadings, line weights, level of detail, etc. [0043]) in which a style of the base image has been changed (See 220 determining a respective emphasis score for one or more of at least one determined feature depicted in the illustration. The respective emphasis score may be based on one or more of a coloration, a size, a line wight, a line style, a shading, a level of detail, or a predetermined shape of a freeform or sketched portion of the illustration corresponding to the at least one feature [0056-0057], Fig. 2).
Regarding claim 13, the combination of Chow/Yi/Zhang teaches the device of claim 11. Chow further discloses:
wherein the processor is further configured to: generate the illustration image being a line drawing (See that the user, e.g., the customer may generate an at least partially freeform or sketched illustration of the desired product. In some embodiments, the user may make a freehand or sketched marking on the template to form the illustration. The freehand and/or sketched portions of the illustration may include one or more labels for one or more features in the illustration [0043]) obtained by removing color from the base image.
Chow does not explicitly teach:
wherein the processor is further configured to: generate the illustration image being a line drawing obtained by removing color from the base image.
Yi, on the other hand, teaches:
wherein the processor is further configured to: generate the illustration image being a line drawing obtained by removing color from the base image (See that the methods transform face photos to APDrawings as a function which maps the face photo domain into a black-and-white line-stroke-based APDrawing Domain. The model is based on the GAN framework which comprises convolutional neural networks (CNN) specifically designed for APDrawings with line-stroke-based artist drawing style. Pages 3464-3465, Figs. 1-2. See further that the APDrawingsGAN transforms a face photo (i.e., color pixel information) into an APDrawing Page 3466)
PNG
media_image1.png
453
706
media_image1.png
Greyscale
Chow and Yi disclose systems and methods directed to image recognition and modification. These three references are in the same field of endeavor and are therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of Chow with the teachings of Yi. The claimed invention is merely a combination of existing elements, and in combination, each element would have functioned the same as they would separately. One of ordinary skill in the art would have recognized that the results were repeatable and would have allowed for a higher-quality image style transfer that makes small features clearly visible (see Yi Pages 3462-3463).
Regarding claim 14, the combination of Chow/Yi/Zhang teaches the device of claim 11. Chow further discloses:
wherein the processor is further configured to: generate the illustration image (See that the user, e.g., the customer may generate an at least partially freeform or sketched illustration of the desired product. In some embodiments, the user may make a freehand or sketched marking on the template to form the illustration. The freehand and/or sketched portions of the illustration may include one or more labels for one or more features in the illustration [0043]) in which a first color of the base image has been changed into a second color.
Chow does not explicitly teach:
wherein the processor is further configured to: generate the illustration image in which a first color of the base image has been changed into a second color.
Yi, on the other hand, teaches:
wherein the processor is further configured to: generate the illustration image in which a first color of the base image has been changed into a second color (See that the methods transform face photos to APDrawings as a function which maps the face photo domain into a black-and-white line-stroke-based APDrawing Domain. The model is based on the GAN framework which comprises convolutional neural networks (CNN) specifically designed for APDrawings with line-stroke-based artist drawing style. Pages 3464-3465, Figs. 1-2. See further that the APDrawingsGAN transforms a face photo (i.e., color pixel information) into an APDrawing Page 3466. Examiner notes that transforming a color photograph into a black and white drawing reads on changing a first color into a second color).
PNG
media_image1.png
453
706
media_image1.png
Greyscale
Chow and Yi disclose systems and methods directed to image recognition and modification. These three references are in the same field of endeavor and are therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of Chow with the teachings of Yi. The claimed invention is merely a combination of existing elements, and in combination, each element would have functioned the same as they would separately. One of ordinary skill in the art would have recognized that the results were repeatable and would have allowed for a higher-quality image style transfer that makes small features clearly visible (see Yi Pages 3462-3463).
Regarding claim 15, Chow discloses:
An information processing method performed by a computer, the information processing method comprising (See a computer 400 that may be configured as a device for executing the methods of Fig. 2, according to exemplary embodiments [0080], Figs. 2, 4):
an obtaining process of obtaining a plurality of base images (See 205 that the image-based search system 130 may provide a visual prompt to the user, where the visual prompt is based on the selected sub-category for the product category. The image-based search system 130 may provide at least apportion of the prompt physically or may provide a portion of the visual prompt electronically (via a display of the user device). In some embodiments, the visual prompt includes a template associated with the product category and/or the product sub-category. The template may be selected from a plurality of templates that correspond to different view angles of a product [0038-0039], Figs. 1-2. Examiner interprets the template as the base image), each of the plurality of base images includes an element of a transaction target specified by a user (See a process for identifying one or more products that correspond to product features of interest to a customer. A customer may desire information or may desire to purchase a particular product in a particular product category having a particular loo or feel, and/or particular features. The user may select a category of products (e.g., vehicles, real estate, jewelry, food, etc.) or may select a sub-category of products (e.g., sport car, sedan, truck, van, etc.). The visual prompt provided to the user at step 205 may be based on the category or sub-category selected [0035-0038], Fig. 2. Examiner interprets the user selection of the category and sub-category as the transaction target specified by the user);
a generating process of generating an illustration image from the plurality of base images (See that the template may be provided to the user and the template may comprise features associated with the category and/or sub-category. In digital form, the features may be in the form of electronic objects that may be manipulated by the user [0039]. See the customer making a freehand or sketched marking on the template in order to form an illustration [0043], Fig. 1. Examiner interprets performing the initial sketch as generating the illustration image) using a machine-learning model, wherein the machine-learning model is learned using a convolution neural network (CNN) and the illustration image includes an illustration-prepared transaction target with elements of the plurality of base images (See that the freehand and/or sketched portions may include one or more features for the product, one or more colors, shadings, line weights, levels of detail, etc. (i.e., the illustration-prepared transaction target). At step 210, the system 130 may receive the freeform or sketched illustration [0042-0044], Fig. 2), wherein the machine-learning model comprises a generation model that outputs the illustration image from the base image (See that the template may be provided to the user and the template may comprise features associated with the category and/or sub-category. In digital form, the features may be in the form of electronic objects that may be manipulated by the user [0039]. See the customer making a freehand or sketched marking on the template in order to form an illustration [0043], Fig. 1. Examiner interprets performing the initial sketch as generating the illustration image);
a reception process of receiving editing performed by the user on the illustration-prepared transaction target and analyze the editing to determine user emphasis on specific features of the transaction target (See that the user may manipulate one or more features onto the electronic depiction in order to modify the template. The user may generate an at least partially freeform or sketched illustration of the desired product. The user may make a freehand or sketched marking on the template to form the illustration. In some embodiment, the user may use a mouse, keyboard, touch screen, stylus, or the like to interact with, for example, an electronic drawing application associated with the user device 105, the vendor system 110, the image-based search system 130, or the like. The freehand and/or sketched portions may include one or more features for the product, one or more colors, shadings, line weights, levels of detail, etc. At step 210, the system 130 may receive the freeform or sketched illustration [0042-0044], Fig. 2. See identifying one or more products that correspond to product features of interest to a user may include: a memory storing instructions and a machine-learning model trained, using (i) at least partially freeform or sketched illustrations from various users depicting products in a product category and (ii) feature labels assigned to the illustrations, to output one or more features of a depicted product in the product category in an input illustration; and a processor operatively connected to the memory and configured to execute the instructions to perform acts. The acts may include: receiving an at least partially freeform or sketched illustration of a product from a user device associated with a user; determining at least one feature of the product depicted in the illustration by employing the machine-learning model, wherein the at least one feature corresponds to a freeform or sketched portion of the received illustration [0008]. Examiner interprets the user making multiple sketched and freeform illustrations as reading on the editing); and
a conversion process of performing inverse conversion on an edited illustration image to generate a converted image using the CNN, the converted image being a photograph converted from the edited illustration image ( See that the image based search system 130 may receive the illustration from the user device 105 (i.e. providing the edited illustration image to a search system) [0044], Fig. 1, 2. See further determining at least one feature of the product depicted in the illustration [0047], Fig. 2. the first machine-learning model may be and/or include a convolutional neural network (a “CNN”). A CNN may include one or more layers, e.g., neural nodes, that may include one or more filters that are convolved over an input image in order to identify one or more features present in the input image [0053]; In response, 225 the image-based search system 130 may generate a search query based on a determined at least one feature from the illustration received at 210. The search query may include and/or be based on information associated with one or more aspects of the determined at least one feature (e.g., orientation, size, etc.). The search query may be based on emphasis score determined for one or more of the at least one features [0065], Fig. 2);
a provision process of providing the converted image to a search server that conducts an image search (See that the image based search system 130 may receive the illustration from the user device 105 (i.e. providing the edited illustration image to a search system) [0044], Fig. 1, 2. See further determining at least one feature of the product depicted in the illustration [0047], Fig. 2. In response, 225 the image-based search system 130 may generate a search query based on a determined at least one feature from the illustration received at 210. The search query may include and/or be based on information associated with one or more aspects of the determined at least one feature (e.g., orientation, size, etc.). The search query may be based on emphasis score determined for one or more of the at least one features [0065], Fig. 2), wherein the edited illustration image being the illustration image that has been edited by the user (See that in some embodiment, the user may use a mouse, keyboard, touch screen, stylus, or the like to interact with, for example, an electronic drawing application associated with the user device 105, the vendor system 110, the image-based search system 130, or the like. The freehand and/or sketched portions may include one or more features for the product, one or more colors, shadings, line weights, levels of detail, etc. At step 210, the system 130 may receive the freeform or sketched illustration [0043-0044], Fig. 2).
Chow does not explicitly disclose:
an obtaining process of obtaining a plurality of base images, each of the plurality of base images;
a generating process of generating an illustration image from the plurality of base images using a machine-learning model, wherein the machine-learning model is learned using a convolution neural network (CNN) and the illustration image includes an illustration-prepared transaction target with elements of the plurality of base images, wherein the machine-learning model comprises a generation model that outputs the illustration image from the base image;
a conversion process of performing inverse conversion on an edited illustration image to generate a converted image using the CNN, the converted image being a photograph converted from the edited illustration image;
a provision process of providing the converted image to a search server that conducts an image search
Previously cited NPL Reference Ran Yi et al. (hereinafter Yi), on the other hand, teaches:
an obtaining process of obtaining a plurality of base images, each of the plurality of base images (Transforming input photos into drawings. Page 3465);
a generating process of generating an illustration image from the plurality of base images using a machine-learning model, wherein the machine-learning model is learned using a convolution neural network (CNN), wherein the machine-learning model comprises a generation model that outputs the illustration image from the base image (See applying a convolutional neural network, specifically a generative adversarial network (GAN), “APDrawingGAN++” to transform a test photo into a line drawing. Pages 3462-3463, Fig. 1(a))
PNG
media_image1.png
453
706
media_image1.png
Greyscale
Chow and Yi disclose systems and methods directed to image recognition and modification. These references are in the same field of endeavor and are therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of Chow with the teachings of Yi. The claimed invention is merely a combination of existing elements, and in combination, each element would have functioned the same as they would separately. One of ordinary skill in the art would have recognized that the results were repeatable and would have allowed for a higher-quality image style transfer that makes small features clearly visible (see Yi Pages 3462-3463).
Additionally, previously cited Zhang et al. (hereinafter Zhang), on the other hand, teaches:
a conversion process of performing inverse conversion on an edited illustration image to generate a converted image using the CNN, the converted image being a photographic image converted from the edited illustration image (Zhang: [0029] – “sketch queries received by the SBIR system 150 may initially be converted to synthetic images 180 prior to searching for authentic images 190 corresponding to the sketches 170 that are subject of the queries”);
a provision process of providing the converted image to a search server that conducts an image search (Zhang: [0029] – “sketch queries received by the SBIR system 150 may initially be converted to synthetic images 180 prior to searching for authentic images 190 corresponding to the sketches 170 that are subject of the queries. This domain-migration function enables the SBIR system 150 to generate more effective mappings between the sketch and image domains, thus allowing for more accurate identification of the retrieval results 160”; [0009] and Fig. 4 – “Hash codes can be generated from the synthetic images to be used as a basis for querying an image database”; [0025] and Fig. 1 – “an SBIR system 150 is stored on one or more servers 120”).
Chow/Yi and Zhang disclose systems and methods directed to image recognition and searching. These references are in the same field of endeavor and are therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of Chow/Yi with the teachings of Zhang. The claimed invention is merely a combination of existing elements, and in combination, each element would have functioned the same as they would separately. One of ordinary skill in the art would have recognized that the results were repeatable and would have allowed for more accurate identification of the retrieval results (Zhang: [0029]).
Regarding claim 16, Chow discloses:
A non-transitory computer readable storage medium having stored an information processing program that causes a computer to execute (See a computer 400 that may be configured as a device for executing the methods of Fig. 2, according to exemplary embodiments. The computer 400 may include a storage unit 406 (such as ROM, HDD, SDD, etc.) that may store data on a computer-readable medium 422, although the computer 400 may receive programming data via network communications. The computer 400 may also have a memory 404 (such as a RAM) storing instructions for executing the techniques presented herein, although the instructions may be stored temporarily or permanently within other modules of the computer (e.g., processor 402 and/or computer readable medium 422) [0080], Figs. 2, 4. Examiner notes that a read-only memory (ROM) and a hard drive disk (HDD) are non-transitory):
an obtaining procedure of obtaining a base image (See 205 that the image-based search system 130 may provide a visual prompt to the user, where the visual prompt is based on the selected sub-category for the product category. The image-based search system 130 may provide at least apportion of the prompt physically or may provide a portion of the visual prompt electronically (via a display of the user device). In some embodiments, the visual prompt includes a template associated with the product category and/or the product sub-category. The template may be selected from a plurality of templates that correspond to different view angles of a product [0038-0039], Figs. 1-2. Examiner interprets the template as the base image), the base image being a photograph that includes a transaction target specified by a user (See a process for identifying one or more products that correspond to product features of interest to a customer. A customer may desire information or may desire to purchase a particular product in a particular product category having a particular loo or feel, and/or particular features. The user may select a category of products (e.g., vehicles, real estate, jewelry, food, etc.) or may select a sub-category of products (e.g., sport car, sedan, truck, van, etc.). The visual prompt provided to the user at step 205 may be based on the category or sub-category selected [0035-0038], Fig. 2. Examiner interprets the user selection of the category and sub-category as the transaction target specified by the user);
a generating procedure of generating an illustration image from the base image (See that the template may be provided to the user and the template may comprise features associated with the category and/or sub-category. In digital form, the features may be in the form of electronic objects that may be manipulated by the user [0039]. See the customer making a freehand or sketched marking on the template in order to form an illustration [0043], Fig. 1. Examiner interprets performing the initial sketch as generating the illustration image) using a machine-learning model, wherein the machine-learning model is learned using a Cycle Generative Adversarial Network (CycleGAN) and the illustration image (See that the freehand and/or sketched portions may include one or more features for the product, one or more colors, shadings, line weights, levels of detail, etc. (i.e., the illustration-prepared transaction target). At step 210, the system 130 may receive the freeform or sketched illustration [0042-0044], Fig. 2) comprises a line drawing of the transaction target with color automatically removed without user input, wherein the machine-learning model comprises a generation model that outputs the illustration image from the base image (See that the template may be provided to the user and the template may comprise features associated with the category and/or sub-category. In digital form, the features may be in the form of electronic objects that may be manipulated by the user [0039]. See the customer making a freehand or sketched marking on the template in order to form an illustration [0043], Fig. 1. Examiner interprets performing the initial sketch as generating the illustration image);
a reception procedure of receiving editing performed by the user on the illustration image (See that the user may manipulate one or more features onto the electronic depiction in order to modify the template. The user may generate an at least partially freeform or sketched illustration of the desired product. The user may make a freehand or sketched marking on the template to form the illustration. In some embodiment, the user may use a mouse, keyboard, touch screen, stylus, or the like to interact with, for example, an electronic drawing application associated with the user device 105, the vendor system 110, the image-based search system 130, or the like. The freehand and/or sketched portions may include one or more features for the product, one or more colors, shadings, line weights, levels of detail, etc. At step 210, the system 130 may receive the freeform or sketched illustration [0042-0044], Fig. 2. Examiner interprets the user making multiple sketched and freeform illustrations as reading on the editing);
a conversion procedure of performing inverse conversion on an edited illustration image to generate a converted image using the CycleGAN, the converted image being a photographic image converted from the edited illustration image ( See that the image based search system 130 may receive the illustration from the user device 105 (i.e. providing the edited illustration image to a search system) [0044], Fig. 1, 2. See further determining at least one feature of the product depicted in the illustration [0047], Fig. 2. the first machine-learning model may be and/or include a convolutional neural network (a “CNN”). A CNN may include one or more layers, e.g., neural nodes, that may include one or more filters that are convolved over an input image in order to identify one or more features present in the input image [0053]; In response, 225 the image-based search system 130 may generate a search query based on a determined at least one feature from the illustration received at 210. The search query may include and/or be based on information associated with one or more aspects of the determined at least one feature (e.g., orientation, size, etc.). The search query may be based on emphasis score determined for one or more of the at least one features [0065], Fig. 2);
a provision procedure of providing the converted image to a search server that conducts an image search to identify one or more transaction targets (See that the image based search system 130 may receive the illustration from the user device 105 (i.e. providing the edited illustration image to a search system) [0044], Fig. 1, 2. See further determining at least one feature of the product depicted in the illustration [0047], Fig. 2. In response, 225 the image-based search system 130 may generate a search query based on a determined at least one feature from the illustration received at 210. The search query may include and/or be based on information associated with one or more aspects of the determined at least one feature (e.g., orientation, size, etc.). The search query may be based on emphasis score determined for one or more of the at least one features [0065], Fig. 2; [0072] – “the image-based search system 130 may provide information associated with a plurality of identified products to the customer); and
a generation process of generating ranking that correspond to price or selling history of the one or more transaction targets by the search server (Chow: [0072] – “the plurality of identified products may be ranked and/or ordered based on…price”).
Chow does not explicitly disclose:
the base image being a photograph;
a generating procedure of generating an illustration image from the base image using a machine-learning model, wherein the machine-learning model is learned using a Cycle Generative Adversarial Network (CycleGAN) and the illustration image comprises a line drawing of the transaction target with color automatically removed without user input, wherein the machine-learning model comprises a generation model that outputs the illustration image from the base image;
a conversion procedure of performing inverse conversion on an edited illustration image to generate a converted image using the CycleGAN, the converted image being a photographic image converted from the edited illustration image;
a provision procedure of providing the converted image to a search server that conducts an image search;
a generation process of generating ranking that correspond to price or selling history of the one or more transaction targets by the search server
However, Chow does disclose loading from one computer or processor into another, for example, from a management server or host computer of a mobile communications network into the computer platform of a server and/or from a server to the mobile device (see Chow [0081]). Chow also discloses providing an in illustration from a mobile device 105 to an image-based search system 103 (see Chow [0044]) and that the image-based search system may be comprised by a computer 400 with a processor 402, memory 404, display, 410, computer readable medium 422, etc. communicating over a network 125 (see Chow [0080], Fig. 4). Thus, while the search system 130 is not explicitly described as a server, one could interpret the search system 130 to teach the server. Nevertheless, additional art has been cited to teach a search server.
Further, while Chow does not explicitly disclose the use of machine learning to generate an illustration image, Chow does disclose generating an illustration/sketchpad image (see Chow [0039], [0042]) as well as utilizing a first machine learning model being trained using at least partially freeform or sketched illustrations from various users depicting products and feature labels assigned to the illustrations, where the first machine learning model may include and/or be a convolutional neural network (CNN) (see Chow [0052-0053]). This machine learning model is used to recognize items that are sketched by the user to perform a search query (see Chow [0025]). Chow does not explicitly disclose that the machine learning model is used to generate the illustration image but does disclose the presence of a machine learning model trained by a CNN. Additional art has been cited to teach using a machine learning model to generate a modifiable image from a photograph.
Yi, on the other hand, teaches:
the base image being a photograph (Transforming input photos into drawings. Page 3465);
a generating procedure of generating an illustration image from the base image using a machine-learning model, wherein the machine-learning model is learned using a Cycle Generative Adversarial Network (CycleGAN), wherein the machine-learning model comprises a generation model that outputs the illustration image from the base image (See applying a CycleGAN to transform a test photo into a line drawing. Pages 3462-3463, Fig. 1(a)) and the illustration image comprises a line drawing of the transaction target with color automatically removed without user input (See that the methods transform face photos by mapping the face photo domain into a black-and-white line-stroke-based Domain. The model is based on the GAN framework which comprises convolutional neural networks (CNN) specifically designed for APDrawings with line-stroke-based artist drawing style. Pages 3464-3465, Figs. 1-2. See further that the APDrawingsGAN transforms a face photo (i.e., color pixel information) into an APDrawing Page 3466); and
a conversion procedure of performing inverse conversion on an edited illustration image to generate a converted image using the CycleGAN (See applying a CycleGAN to transform a test photo into a line drawing. Pages 3462-3463, Fig. 1(a)).
PNG
media_image1.png
453
706
media_image1.png
Greyscale
Chow and Yi disclose systems and methods directed to image recognition and modification. These references are in the same field of endeavor and are therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of Chow with the teachings of Yi. The claimed invention is merely a combination of existing elements, and in combination, each element would have functioned the same as they would separately. One of ordinary skill in the art would have recognized that the results were repeatable and would have allowed for a higher-quality image style transfer that makes small features clearly visible (see Yi Pages 3462-3463).
Additionally, previously cited Zhang et al. (hereinafter Zhang), on the other hand, teaches:
a conversion procedure of performing inverse conversion on an edited illustration image to generate a converted image using the CycleGAN, the converted image being a photographic image converted from the edited illustration image (Zhang: [0029] – “sketch queries received by the SBIR system 150 may initially be converted to synthetic images 180 prior to searching for authentic images 190 corresponding to the sketches 170 that are subject of the queries”);
a provision procedure of providing an edited illustration image to a search server that conducts an image search; a generation process of generating ranking that correspond to price or selling history of the one or more transaction targets by the search server (Zhang: [0029] – “sketch queries received by the SBIR system 150 may initially be converted to synthetic images 180 prior to searching for authentic images 190 corresponding to the sketches 170 that are subject of the queries. This domain-migration function enables the SBIR system 150 to generate more effective mappings between the sketch and image domains, thus allowing for more accurate identification of the retrieval results 160”; [0009] and Fig. 4 – “Hash codes can be generated from the synthetic images to be used as a basis for querying an image database”; [0025] and Fig. 1 – “an SBIR system 150 is stored on one or more servers 120”).
Chow/Yi and Zhang disclose systems and methods directed to image recognition and searching. These references are in the same field of endeavor and are therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of Chow/Yi with the teachings of Zhang. The claimed invention is merely a combination of existing elements, and in combination, each element would have functioned the same as they would separately. One of ordinary skill in the art would have recognized that the results were repeatable and would have allowed for more accurate identification of the retrieval results (Zhang: [0029]).
Claims 6 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Chow, in view of Yi, in view of Zhang, in view of previously cited Piramuthu et al. (U.S. 2019/0205962 A1).
Regarding claim 6, the combination of Chow/Yi/Zhang teaches the device of claim 1. Chow further discloses:
wherein the processor is further configured to: receive a change in a pattern that has been performed by the user on the illustration-prepared transaction target (See 220 determining a respective emphasis score for one or more of at least one determined feature depicted in the illustration. The respective emphasis score may be based on one or more of a coloration, a size, a line wight, a line style, a shading, a level of detail, or a predetermined shape of a freeform or sketched portion of the illustration corresponding to the at least one feature [0056-0057], Fig. 2), and
provide the search server with the edited illustration image in which the pattern of the illustration-prepared transaction target has been changed by the user (See after receiving the freeform sketched illustration of the product at step 210, the search is performed in step 225. Fig. 2. See that the search query is generated based on information associated with at least one feature emphasized by the user [0065], Fig. 2).
Chow does not explicitly disclose:
wherein the processor is further configured to: receive a change in a pattern that has been performed by the user on the illustration-prepared transaction target, and
provide the search server with the edited illustration image in which the pattern of the illustration-prepared transaction target has been changed by the user
However, Chow does disclose loading from one computer or processor into another, for example, from a management server or host computer of a mobile communications network into the computer platform of a server and/or from a server to the mobile device (see Chow [0081]). Chow also discloses providing an in illustration from a mobile device 105 to an image-based search system 103 (see Chow [0044]) and that the image-based search system may be comprised by a computer 400 with a processor 402, memory 404, display, 410, computer readable medium 422, etc. communicating over a network 125 (see Chow [0080], Fig. 4). Thus, while the search system 130 is not explicitly described as a server, one could interpret the search system 130 to teach the server. Nevertheless, additional art has been cited.
Zhang, on the other hand, teaches: a search server ([0025] and Fig. 1 – “an SBIR system 150 is stored on one or more servers 120”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed inventions to combine Zhang with Chow/Yi for the reasons identified above with respect to claim 1.
Piramuthu, on the other hand, teaches:
wherein the processor is further configured to: receive a change in a pattern that has been performed by the user on the illustration-prepared transaction target (See at the first stage 352, receiving a digital image of a dress (i.e. a product) having a pattern. At the second stage 354, a digital image 360 of a pattern is captured. From this, the camera platform manager module may determine, using machine learning (e.g., object recognition) that the pattern, texture, and/or materials are of interest in this digital image 360. As a result, the overall shape from the digital image 358, and the texture, materials, and/or pattern of the digital image are used to locate digital content 362 (e.g., another digital image in a product listing) of a dress having a similar shape from the digital image 358 and a pattern from the digital image 360 [0054-0055], Fig. 3B), and
provide the search server with the edited illustration image in which the pattern of the illustration-prepared transaction target has been changed by the user (See at the first stage 352, receiving a digital image of a dress (i.e. a product) having a pattern. At the second stage 354, a digital image 360 of a pattern is captured. From this, the camera platform manager module may determine, using machine learning (e.g., object recognition) that the pattern, texture, and/or materials are of interest in this digital image 360. As a result, the overall shape from the digital image 358, and the texture, materials, and/or pattern of the digital image are used to locate digital content 362 (e.g., another digital image in a product listing) of a dress having a similar shape from the digital image 358 and a pattern from the digital image 360 [0054-0055], Fig. 3B)
Both Chow/Yi/Zhang and Piramuthu disclose systems and methods directed to image-based product searching. These references are in the same field of endeavor and are therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of Chow/Yi/Zhang with the pattern-based searching of Piramuthu. The claimed invention is merely a combination of existing elements, and in combination, each element would have functioned the same as they would separately. One of ordinary skill in the art would have recognized that the results were repeatable and would have allowed for pattern-based authentication in order to determine whether a captured pattern matches a known authentic pattern, which would prevent a user from purchasing counterfeit products (see Piramuthu [0101-0102]).
Regarding claim 9, the combination of Chow/Yi/Zhang teaches the device of claim 1. Chow does not explicitly disclose:
wherein the processor is further configured to: obtain the base image of the transaction target having a pattern specified by the user.
Piramuthu, on the other hand, teaches:
wherein the processor is further configured to: obtain the base image of the transaction target having a pattern specified by the user (See at the first stage 352, receiving a digital image of a dress (i.e. a transaction target) having a pattern. The digital image 358 includes an entire outline of a dress [0054], Fig. 3B).
Both Chow/Yi/Zhang and Piramuthu disclose systems and methods directed to image-based product searching. These references are in the same field of endeavor and are therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of Chow/Yi/Zhang with the pattern-based searching of Piramuthu. The claimed invention is merely a combination of existing elements, and in combination, each element would have functioned the same as they would separately. One of ordinary skill in the art would have recognized that the results were repeatable and would have allowed for pattern-based authentication in order to determine whether a captured pattern matches a known authentic pattern, which would prevent a user from purchasing counterfeit products (see Piramuthu [0101-0102]).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Chow in view of Yi in view of Zhang view of previously cited Yada et al. (US 2020/0356591 A1).
Regarding claim 7, the combination of Chow/Yi/Zhang teaches the device of claim 1. Chow further discloses:
wherein the processor is further configured to: obtain the base image of the transaction target (See 205 that the image-based search system 130 may provide a visual prompt to the user, where the visual prompt is based on the selected sub-category for the product category. The image-based search system 130 may provide at least apportion of the prompt physically or may provide a portion of the visual prompt electronically (via a display of the user device). In some embodiments, the visual prompt includes a template associated with the product category and/or the product sub-category. The template may be selected from a plurality of templates that correspond to different view angles of a product [0038-0039], Figs. 1-2. Examiner interprets the template as the base image) having a color specified by the user.
The combination of Chow and Piramuthu does not explicitly teach:
wherein the processor is further configured to: obtain the base image of the transaction target having a color specified by the user.
Chow does disclose that a customer may, via electronic objects, modify a template image by coloration before generating the freeform or sketched illustration (see Chow [0042-0043]), which could be interpreted to teach obtaining the base image (i.e. the template) featuring a color specified by the user. Nevertheless, additional art has been cited to teach the base image.
Yada et al. (hereinafter Yada), on the other hand, teaches:
wherein the processor is further configured to: obtain the base image of the transaction target having a color specified by the user (See that a user may textually specify a desired color or pattern to be used in a query image. The computing environment can then process the text based information items along with the image based information items to retrieve and present one or more candidate images [0040-0044]).
Chow/Yi/Zhang and Yada disclose systems and methods directed to image analysis, manipulation, or searching. These references are in the same field of endeavor and are therefore analogous. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of Chow with the searching methods of Yada. The claimed invention is merely a combination of existing elements, and in combination, each element would have functioned the same as they would separately. One of ordinary skill in the art would have recognized that the results were repeatable and would have improved over traditional image-based searching because traditional image-based searching does not allow revising of previously-submitted images, which wastes computing resources (see Yada [0035]).
Response to Arguments
35 U.S.C. 101
Applicant argues the claims are patent eligible because the examiner “oversimplifies the technical nature of the claim” (Remarks page 8). The examiner disagrees. The claims recite a process of analyzing image and illustration information in order to search for images. Therefore, the claims being directed to an abstract idea of performing a search for a transaction target is not a description of the claims at a high level of abstraction untethered from the language of the claim. The examiner points to the various recent court decisions, such as Ultramercial Inc. v. Hulu and Cyberfone Systems v. CNN Interactive Group, for examples of claims that, on their face, recited complex ideas/interactions but were construed more broadly in terms of the abstract idea they directed themselves to. The MPEP further highlights that a claim recites a judicial exception (e.g., an abstract idea) when the judicial exception is “set forth” or “described” in the claim. Claims can “describe” an abstract concept without ever explicitly stating the abstract concept, e.g., the claims in Alice “described” the concept of intermediated settlement without ever explicitly using the words “intermediated” or “settlement.”
Applicant argues the claims are patent eligible because “[t]hese steps represent technological improvements in image searching capabilities” (Remarks pages 8-9). The examiner disagrees. The examiner disagrees. The MPEP sets forth, in Step 2A Prong Two, that a claim that recites a judicial exception is not directed to that judicial exception, if the claim as a whole “integrates the recited judicial exception into a practical application of that exception.” The evaluation of Prong Two requires the use of the considerations (e.g. improving technology, effecting a particular treatment or prophylaxis, implementing with a particular machine, etc.) identified by the Supreme Court and the Federal Circuit, to ensure that the claim as a whole ‘integrates [the] judicial exception into a practical application [that] will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception.’ In the instant case, the claims include additional elements such as an information processing device comprising: a processor, wherein the processor is configured to perform operations, a machine-learning model, a generation model, wherein the machine-learning model is learned using a convolution neural network (CNN), a search server, a computer, a non-transitory computer readable storage medium having stored an information processing program that causes a computer to execute a method, and a Cycle Generative Adversarial Network (CycleGAN). While these elements are recited, they are merely peripherally incorporated in order to implement the abstract idea. Put another way, these additional elements are merely used to apply the abstract idea of performing a search for a transaction target (i.e. a product for purchase) in a technological environment without effectuating any improvement or change to the functioning of the additional elements or other technology. Applicant’s disclosure does not articulate or suggest how these additional elements function, individually or in combination, in any manner other than using generic functionality nor does the disclosure articulate how the elements provide a technical improvement. Accordingly, the additional elements do not integrate the abstract idea into a practical application because they merely amount to using the software architecture as a tool to perform the abstract idea.
Accordingly, the rejection under 35 U.S.C. 101 is maintained.
35 U.S.C. 103
Applicant’s arguments with respect to the rejection under 35 U.S.C. 103 have been fully considered but are not persuasive. Applicant argues that the cited art has differences from the invention and does not teach, disclose, or suggest “a specific generation model that outputs illustration images from input images, with inverse conversion that restores edited illustrations back to photographic images for searching. The search server then uses registered illustration images prepared from registered transaction target images to conduct the search and extract relevant transaction information” (remarks page 8). Examiner disagrees. Initially, the examiner notes the arguments do not cite to specific claim language. Furthermore, the applied art may be directed to, or embody, different “ideas” but may still be applicable as applied art that teaches the specific limitations of the invention. As discussed in the 103 rejection above, the combination of the cited art teaches the limitations of the claims. Thus, the cited art teaches the invention.
Accordingly, the claims remain rejected under 35 U.S.C. 103.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Previously cited Becherer et al. (US 2014/0089295 A1) discloses a system and method directed to performing image-based product searches using a morphing search tool that provides an interface through which a user may modify a displayed image and perform image similarity queries.
Previously cited Wang et al. (US 2016/0132498 A1) discloses systems and methods for image searching using a color sketchpad.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANNA MAE MITROS whose telephone number is (571)272-3969. The examiner can normally be reached Monday-Friday from 9:30-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Marissa Thein can be reached at 571-272-6764. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANNA MAE MITROS/Examiner, Art Unit 3689
/MARISSA THEIN/Supervisory Patent Examiner, Art Unit 3689
Prosecution Insights