DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on September 19, 2023, is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
Applicant’s Preliminary Amendment to the Specification and Claims filed on September 19, 2023, has been entered and made of record.
Currently Pending Claim(s) 1-9
Independent Claim(s) 1 and 8-9
Amended Claim(s) 1-9
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Objections
Claim 6 is objected to because of minor informalities. The Examiner recommends removing redundant wording to promote clarity of the claims. For example, the last phrase of claim 6 could be removed as follows: “The image processing apparatus according to claim 1, wherein the operations further comprise processing the image, and thereby inferring the product and/or a similar product
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim Interpretation: Under the broadest reasonable interpretation, the terms of the claim are presumed to have their plain meaning consistent with the specification as it would be interpreted by one of ordinary skill in the art. See MPEP 2111.
Claims 1 and 8-9 do not fall within at least one of the four categories of patent eligible subject matter. In these claims, several steps are recited:
acquiring an image that is a captured image of an image capture subject and includes a product in a part of an area
processing the image and thereby generating subject inference data indicating an inference result of a type of the image capture subject
performing output based on the subject inference data
Step 1: This part of the eligibility analysis evaluates whether the claim falls within any statutory category. See MPEP 2106.03. Claims 1 and 8-9 recite a computer system that implements a method for recognizing images of products and correlating user demographics to products and advertising mediums, so the claims do fall within one of the statutory categories of invention.
Step 2A, Prong One: This part of the eligibility analysis evaluates whether the claim recites a judicial exception. As explained in MPEP 2106.04, subsection II, a claim “recites” a judicial exception when the judicial exception is “set forth” or “described” in the claim.
The broadest reasonable interpretation of steps (a), (b), and (c) is that those steps fall within the mental process groupings of abstract ideas, because they cover concepts performed in the human mind, including observation, evaluation, judgment, and opinion. See MPEP 2106.04(a)(2), subsection III.
More specifically, step (a) involves acquiring an image of a product, which is a step of data gathering. Step (b) involves determining the type of the image capture subject, which is viewing the image and determining the context of the image (whether the image contains an email, commercial, social media post, etc.). A human can easily observe an image and identify the image capture subject without the need for a specific computer system. Step (C) involves “performing output” without providing any details about what data is output or how images are processed.
Step 2A, Prong Two: This part of the eligibility analysis evaluates whether the claim as a whole integrates the recited judicial exception into a practical application of the exception or whether the claim is “directed to” the judicial exception. This evaluation is performed by (1) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (2) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. See MPEP 2106.04(d).
Claims 1 and 8-9 recite the additional elements of generic computer components. Claim 1 includes an apparatus comprising memory and a processor, claim 8 includes a computer, and claim 9 includes a non-transitory storage medium for storing a computer program. The claims amount to no more than instructions to apply the method using a generic computer. See MPEP 2106.05(f).
Step 2B: This part of the eligibility analysis evaluates whether the claim as a whole amounts to significantly more than the recited exception i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. See MPEP 2106.05.
Even when considered in combination, these additional elements represent mere instructions to implement an abstract idea or other exception on a computer and insignificant extra-solution activity, which do not provide an inventive concept.
Overall, the Examiner recommends providing more technical details about the apparatus and the image processing method beyond the claimed steps of acquiring an image, inferring the subject (email advertisement, commercial, social media post, etc.) of the image, and “performing output” using the subject inference data. To provide a practical application for the independent claims, technical details about the image processing method and apparatus should be provided so that concrete, technological improvements which require the claimed apparatus are present. Currently, the independent claims provide a method which could be completed by any generic image analysis system or computer. Furthermore, the process of “performing output” is vague and does not clearly explain what data is output by the invention or how the output is determined.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1 and 8-9 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. These claims include the limitation of “performing output based on the subject inference data” without any explanation of what values the output contains or how the output is determined.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-4, 6, and 8-9 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Shihadah et al. (US 2013/0282532 A1), hereafter Shihadah.
Regarding claim 1, Shihadah teaches an image processing apparatus (Fig. 13) comprising:
at least one memory configured to store instructions; and at least one processor configured to execute the instructions to perform operations (Fig. 13 shows the system. Label 1300 represents a user’s mobile device which captures an input image or video of a product. 1302 shows a web server which receives the input image or video to identify the product and return online shopping options to the user for buying that product or other corresponding products. Fig. 14 further shows a processing system including a CPU and memory.), the operations comprising:
acquiring an image that is a captured image of an image capture subject and includes a product in a part of an area (Shihadah teaches a system for capturing an image or video containing a product, and the system recognizes the product and returns related product listings. The input can contain a product seen in real life or as seen in an advertisement/commercial. For example, Fig. 2B shows a user capturing an image of a football game on a television screen. Here, the jersey worn by a player is the product and the football game on a television screen is the image capture subject. [0008-0009] “Consumers are then encouraged to capture and send in portions of media streams they are currently watching, listening to, etc. The portions of the media streams that users capture and send in are analyzed… The mobile device may operate to capture video, audio, and/or images, and relate them to particular goods and/or services. The user may be presented with more information about the goods/services and/or may be presented with options to purchase the goods and/or services.” Additionally, the user can manually select products present in the input video or image to focus on. [0163] “There may be multiple different ways to select an item/object that is displayed in a video stream. For example, a user can draw on the smart device (e.g., a circle) or may tap a specific area, etc. The selection and/or recognition of objects within a video scene can be combined with the below discussed social aspect (e.g., objects can be highlighted based on previously requests and/or purchases).”);
processing the image and thereby generating subject inference data indicating an inference result of a type of the image capture subject (Shihadah teaches methods for interpreting the input image or video to determine the product, and the methods determine if the product is within the context of an advertisement. [0082] “In certain example embodiments, additional processing may take place to automatically identify products, screens, etc. The various types of information captured may be fed into an example media analysis system for a processing of data and/or media at 316.” [0083] “FIG. 4 shows an exemplary process for handling advertising content according to certain example embodiments. The processing of data/media at 400 may involve a determination that the media is an advertisement at 402.”); and
performing output based on the subject inference data (Shihadah teaches that output (webpages to buy the identified product or related products) may be determined based on the context of the input image or video. For example, if the input is recognized as a football player in a video game on television rather than a real-life video of a football game, the system may provide output information based on the video game itself rather than the football player. [0155] “In certain example embodiments, the snippet that is shared may be edited before sending (e.g., in terms of size, or tagged information). In certain example embodiments, the information retrieval and/or shopping options may be based on the object within the context of the video/show/movie/game/etc or outside of such context. For example, information on particular football player may be retrieved generally (outside of the current context) or specifically (within the context of the current game being played).”).
Regarding claim 2, Shihadah teaches the image processing apparatus according to claim 1, wherein the operations further comprise generating the subject inference data by processing an area around the product in the image (Shihadah teaches a system capable of recognizing screens. This way, the system can determine if the product is being shown in real-life or within an advertisement on a screen. [0082] “In certain example embodiments, additional processing may take place to automatically identify products, screens, etc. The various types of information captured may be fed into an example media analysis system for a processing of data and/or media at 316.” [0083] “FIG. 4 shows an exemplary process for handling advertising content according to certain example embodiments. The processing of data/media at 400 may involve a determination that the media is an advertisement at 402.” Additionally, [0155] mentions using context around the product for affecting the output.).
Regarding claim 3, Shihadah teaches the image processing apparatus according to, claim 1, wherein the operations further comprise, acquiring, as the image, a moving image, and processing a plurality of frame images included in the moving image, and thereby generating the subject inference data ([0078] “In certain example embodiments, instead of taking a picture a user may record a video or short clip (e.g., 5 seconds). This may apply to “real world” content (e.g., where person 104 obtains a video clip of car 100 driving by) or content broadcast through a television, radio, etc.”).
Regarding claim 4, Shihadah teaches the image processing apparatus according claim 1, wherein a type of the image capture subject includes at least one of: a screen displayed on a display, based on broadcast; a screen displayed on a display, based on a social networking service (SNS); a screen displayed on a display, based on an email; a printed matter; and the product arranged in a physical space ([0082] “The T.V. content that is fed into an example media analysis system may capture video at 310, scenes, frames, clips, etc at 314, and/or audio at 312… In certain example embodiments, additional processing may take place to automatically identify products, screens, etc. The various types of information captured may be fed into an example media analysis system for a processing of data and/or media at 316.” [0083] “FIG. 4 shows an exemplary process for handling advertising content according to certain example embodiments. The processing of data/media at 400 may involve a determination that the media is an advertisement at 402.” See the paragraphs following 0083 which discuss determining the type of advertisement from the input image/video. [0111] “FIG. 12 is a block diagram of an example processing server that includes multiple different modules according to certain example embodiments. A processing server 1200 may be one or more physical servers (e.g., a cluster or server farm). A media platform module 1202 may be provided. This module may tag media content though the automatic processing of channels (e.g., TV channels, internet sources, etc). Tagged information may include: Program type, for example: movies, documentary, sports (e.g., NFL, NBA), news, music, T.V. shows, commercials (e.g., Advertisements); Time of transmission or broadcast, for example: country, region, city, channel #, network (e.g., HBO, NBC, ABC). In certain instances, the module 1202 may provide libraries for content/data to search over. For example, the libraries may include: cars, bicycles, different types of sports (e.g., golf, football, etc).”).
Regarding claim 6, Shihadah teaches the image processing apparatus according to claim 1, wherein the operations further comprise processing the image, and thereby inferring the product and/or a similar product similar to the product ([0009] “…automatic identification of TV or movie content and association with product or service offers or location aids is provided via a mobile device. The content may include for example movies, TV shows, advertisements, sports, news, music, entertainment channels, or any form of stimuli that can be sensed and captured electronically…. The user may be presented with more information about the goods/services and/or may be presented with options to purchase the goods and/or services.” [0086] “In certain example embodiments, various actions may be taken once a product, person, place, service, or the like is identified. In FIG. 5 a report is given to the user as to whether or not a match 512 based on the content submitted has been found… In certain example embodiments, when a no match report is returned to the user, additional options 514 may be presented to the user. For example, a related goods search may be offered to the user. This may include presenting other car types to the user if the user submitted media that included a car.”).
Regarding claim 8, Shihadah teaches an image processing method (Figs. 4-8) performing:
by a computer (Fig. 13 shows the system. Label 1300 represents a user’s mobile device which captures an input image or video of a product. 1302 shows a web server which receives the input image or video to identify the product and return online shopping options to the user for buying that product or other corresponding products. Fig. 14 further shows a processing system including a CPU and memory.),
acquiring an image that is a captured image of an image capture subject and includes a product in a part of an area (Shihadah teaches a system for capturing an image or video containing a product, and the system recognizes the product and returns related product listings. The input can contain a product seen in real life or as seen in an advertisement/commercial. For example, Fig. 2B shows a user capturing an image of a football game on a television screen. Here, the jersey worn by a player is the product and the football game on a television screen is the image capture subject. [0008-0009] “Consumers are then encouraged to capture and send in portions of media streams they are currently watching, listening to, etc. The portions of the media streams that users capture and send in are analyzed… The mobile device may operate to capture video, audio, and/or images, and relate them to particular goods and/or services. The user may be presented with more information about the goods/services and/or may be presented with options to purchase the goods and/or services.” Additionally, the user can manually select products present in the input video or image to focus on. [0163] “There may be multiple different ways to select an item/object that is displayed in a video stream. For example, a user can draw on the smart device (e.g., a circle) or may tap a specific area, etc. The selection and/or recognition of objects within a video scene can be combined with the below discussed social aspect (e.g., objects can be highlighted based on previously requests and/or purchases).”);
processing the image and thereby generating subject inference data that indicate an inference result of a type of the image capture subject (Shihadah teaches methods for interpreting the input image or video to determine the product, and the methods determine if the product is within the context of an advertisement. [0082] “In certain example embodiments, additional processing may take place to automatically identify products, screens, etc. The various types of information captured may be fed into an example media analysis system for a processing of data and/or media at 316.” [0083] “FIG. 4 shows an exemplary process for handling advertising content according to certain example embodiments. The processing of data/media at 400 may involve a determination that the media is an advertisement at 402.”); and
performing output based on the subject inference data (Shihadah teaches that output (webpages to buy the identified product or related products) may be determined based on the context of the input image or video. For example, if the input is recognized as a football player in a video game on television rather than a real-life video of a football game, the system may provide output information based on the video game itself rather than the football player. [0155] “In certain example embodiments, the snippet that is shared may be edited before sending (e.g., in terms of size, or tagged information). In certain example embodiments, the information retrieval and/or shopping options may be based on the object within the context of the video/show/movie/game/etc or outside of such context. For example, information on particular football player may be retrieved generally (outside of the current context) or specifically (within the context of the current game being played).”).
Regarding claim 9, Shihadah teaches a non-transitory computer-readable medium [0055] “The system includes a memory storage medium.”) storing a program for causing a computer to perform operations (Fig. 13 shows the system. Label 1300 represents a user’s mobile device which captures an input image or video of a product. 1302 shows a web server which receives the input image or video to identify the product and return online shopping options to the user for buying that product or other corresponding products. Fig. 14 further shows a processing system including a non-transitory computer readable medium.), the operations comprising:
acquiring an image that is a captured image of an image capture subject and includes a product in a part of an area (Shihadah teaches a system for capturing an image or video containing a product, and the system recognizes the product and returns related product listings. The input can contain a product seen in real life or as seen in an advertisement/commercial. For example, Fig. 2B shows a user capturing an image of a football game on a television screen. Here, the jersey worn by a player is the product and the football game on a television screen is the image capture subject. [0008-0009] “Consumers are then encouraged to capture and send in portions of media streams they are currently watching, listening to, etc. The portions of the media streams that users capture and send in are analyzed… The mobile device may operate to capture video, audio, and/or images, and relate them to particular goods and/or services. The user may be presented with more information about the goods/services and/or may be presented with options to purchase the goods and/or services.” Additionally, the user can manually select products present in the input video or image to focus on. [0163] “There may be multiple different ways to select an item/object that is displayed in a video stream. For example, a user can draw on the smart device (e.g., a circle) or may tap a specific area, etc. The selection and/or recognition of objects within a video scene can be combined with the below discussed social aspect (e.g., objects can be highlighted based on previously requests and/or purchases).”);
processing the image and thereby generating subject inference data that indicate an inference result of a type of the image capture subject (Shihadah teaches methods for interpreting the input image or video to determine the product, and the methods determine if the product is within the context of an advertisement. [0082] “In certain example embodiments, additional processing may take place to automatically identify products, screens, etc. The various types of information captured may be fed into an example media analysis system for a processing of data and/or media at 316.” [0083] “FIG. 4 shows an exemplary process for handling advertising content according to certain example embodiments. The processing of data/media at 400 may involve a determination that the media is an advertisement at 402.”); and
performing output based on the subject inference data (Shihadah teaches that output (webpages to buy the identified product or related products) may be determined based on the context of the input image or video. For example, if the input is recognized as a football player in a video game on television rather than a real-life video of a football game, the system may provide output information based on the video game itself rather than the football player. [0155] “In certain example embodiments, the snippet that is shared may be edited before sending (e.g., in terms of size, or tagged information). In certain example embodiments, the information retrieval and/or shopping options may be based on the object within the context of the video/show/movie/game/etc or outside of such context. For example, information on particular football player may be retrieved generally (outside of the current context) or specifically (within the context of the current game being played).”).
Allowable Subject Matter
Claims 5 and 7 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Additionally, the rejections under 35 U.S.C. 101 and 35 U.S.C. 112(b) applied to claim 1 must first be overcome.
Regarding claim 5, the closest prior art of record, Shihadah (US 2013/0282532 A1), teaches
the image processing apparatus according to claim 1, wherein the image is generated by a terminal (Fig. 13 shows a terminal 1300, which is the user’s end device for capturing images and initiating the image-search for the product. [0119] “Section 1300 may include those systems that are operated by a user. For example, a mobile device that is used for uploading media content that is to be analyzed by the systems in section 1302. Thus, the mobile device in section 1300 may communicate with one or more of the systems in section 1302 (e.g., for uploading content) and/or 1304 (e.g., for payment processing).”),
the operations further comprise acquiring the image from a plurality of the terminals, generating the subject inference data for each of the plurality of terminals ([0082] “In certain example embodiments, additional processing may take place to automatically identify products, screens, etc. The various types of information captured may be fed into an example media analysis system for a processing of data and/or media at 316.” [0083] “FIG. 4 shows an exemplary process for handling advertising content according to certain example embodiments. The processing of data/media at 400 may involve a determination that the media is an advertisement at 402.”).
Furthermore, similar systems for receiving input images/video, identifying a product and subject of the input, returning a page to buy the product and/or similar products, and recording user profile information and purchase history are taught throughout the prior art. For example, Kanegaki (JP 2006/085392 A; from the IDS dated September 19, 2023) teaches a system where users can upload an image of a live TV commercial, and the system recognizes the TV commercial and the product within the commercial to provide a webpage for the user to purchase the product. Similarly, Kannan (US 2011/0082735 A1) teaches a system where users can upload images of a commercial, and the system recognizes the product in the commercial and recommends products for the user to buy. Additionally, Schiffman (US 10,600,060 B1) teaches a system for recognizing products in user uploaded images and determining contexts that the product often appears in; here, context refers to how and where people typically use the product in images.
However, the prior art of record fails to teach outputting first relation data indicating a relation between attribute information of a user of each of the plurality of terminals and the subject inference data. Although determining relationships between user demographics and advertising/searching subjects (through email, commercial, social media, etc.) is well-known, determining such relation data is typically completed for the purpose of targeted advertising. For example, Chura (US 20200242656A1) teaches a method of tracking user interactions with advertisements, so user demographics and interaction history can be analyzed to determine which advertisement subjects (email ad, web ad, or phone call) were most often used by different user demographics. Similarly, Wardell (US 2007/0061190 A1) teaches methods of analyzing user browsing behavior, purchase history, and responsiveness to different advertisements so that the most effective methods for targeted advertisements can be determined. This involves determining whether email, web ads, or phone calls are most commonly used by different user demographics for finding a product to purchase. See 0067-0068 of Wardell.
Overall, the cited prior art and many more patents by Google, Microsoft, etc., in the field of advertising, teach determining which products users search for most and which advertising subjects (through email, commercial, social media, etc.) users interact with most, so companies can determine which advertisements will be most effective for different user demographics. This involves analyzing click-through rates, responsiveness, and search frequency across different advertising subjects. However, the prior art fails to teach these methods applied specifically to a system for visual product searching only. The claimed invention analyzes images which are uploaded to a server for visual product searching, and first relation data is output. The first relation data is interpreted as a relationship between the user demographic groups and the subject inference data (See 0037 of the Specification and Fig. 9), such as the number of times a specific user demographic searched for a product using a specific image subject (visual product search using an image of an email, commercial, real-life image, etc.). Thus, the claimed invention tracks how different demographics search for a product rather than how different demographics respond to advertising.
Regarding claim 7, Shihadah teaches the image processing apparatus according to claim 6, wherein the operations further comprise acquiring purchase result information indicating whether the product and/or the similar product has been purchased ([0039] “User profile that may include information such as, for example, Name, email, telephone number (e.g., home, work, cell), payment options, billing information, the devices that are registered (e.g., iPhone, Android mobile devices), buying history, wish list, interests, etc.”). Similarly, Kanegaki, Kannan, and Shiffman also teach acquiring purchase result information.
However, the prior art of record fails to teach outputting relation data in the specific context of visual product searching. Thus, the prior art of record fails to teach outputting second relation data indicating a relation between the subject inference data and the purchase result information, for the same reasons discussed above regarding claim 5.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Black et al. (US 8,255,948 B1) teaches receiving media content from a user, classifying the media content to determine salient features, correlating content to different user demographics, and targeting related content to members of the demographic group.
Wardell (US 2007/0061190 A1) teaches methods for performing and optimizing marketing campaigns by analyzing user purchase history information for determining effective advertising methods to market to a specific user.
Lee et al. (US 2019/0065911 A1) teaches methods for analyzing, segmenting, and classifying multimedia content.
Kannan et al. (US 2011/0082735 A1) teaches methods for visual product searching where a product is recognized in a user-uploaded image, a recommendation image determines similar products to a product identified in the image, and the system provides webpages to the user for purchasing the product or similar products.
Chura et al. (US 2020/0242656 A1) teaches methods of tracking user interaction with different advertisements which led to the purchase of a product. The method includes determining which demographic of users are most responsive to different types of advertisements.
Schiffman et al. (US 10,600,060 B1) teaches methods for predictive analytics which includes analyzing user uploaded images of products and determining the context that products are typically observed in. For example, the method may track a specific product across many social media posts to determine how users typically utilize the product.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERIC JAMES SHOEMAKER whose telephone number is (571)272-6605. The examiner can normally be reached Monday through Friday from 8am to 5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner' s supervisor, JENNIFER MEHMOOD, can be reached at (571)272-2976. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Eric Shoemaker/
Patent Examiner
/XIAO LIU/Primary Examiner, Art Unit 2664
Prosecution Insights