DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant claims priority to provisional U.S. Patent Application No. 63/048,707, filed 7/7/2020.
Information Disclosure Statement
The IDSs submitted on 8/9/2022, 3/21/2023, 10/6/2023, and 10/12/2023 were previously considered.
Election/Restrictions
Applicant's election with traverse of Invention I in the reply filed on 3/26/2026 is acknowledged. The traversal is on the ground(s) that the claims are not independent or distinct, overlap in scope, and no serious search or examination burden exists. This is not found persuasive because, as previously noted, Inventions I and II have material different function such as Invention I functioning to generate a list of matched search results for products detected and presenting/displaying the products. Invention II functioning to create marketing emails containing information about the plurality of products identified in the user set of digital photos, embed an image or video showing at least one of the identified plurality of products in a correct context and usage, allow a customer or a vendor to set times for a set of product-specific marketing emails to be sent, providing a product purchasing reminder service. Furthermore, the inventions as claimed do not encompass overlapping subject matter and there is nothing of record to show them to be obvious variants. Additionally, there would be a serious search and/or examination burden if restriction were not required because the invention have acquired a separate status in the art in view of their recognized divergent subject matters, which require unique considerations with respect to the applicable prior art as well as requiring uniquely different fields of search (e.g., for employing different search queries, consulting searching different classes/subclasses). The divergent subject matter of the groups raises unique consideration with respect to the prior art such that the body of art applied to one group would not likely be applicable to the other group (as a whole or at least in part).
The requirement is still deemed proper and is therefore made FINAL.
Claims 34-40 are withdrawn from further consideration pursuant to 37 CFR 1.142(b), as being drawn to a nonelected Invention II, there being no allowable generic or linking claim. Applicant timely traversed the restriction (election) requirement in the reply filed on 3/26/2026.
Examiner’s Comment
For purposes of compact prosecution, Examiner has examined claims 21-33. However, Examiner notes the listing of claims does not include the text of all pending claims (including withdrawn claims). Claims 34-40 have the proper status identifier as (Withdrawn) however the listing of claims 34-40 do not include the text of the claims. Applicant is reminded to include the text of all pending claims (including withdrawn claims) as the text of all pending claims under examination and withdrawn claims must be submitted each time any claim is amended. See MPEP 714. Further, while claim 21 has the limitation “scene” underlined in line 4, this limitation was present in the claims filed 9/10/2025, and therefore is not added subject matter in the claim and should not be underlined.
Status of Claims
Applicant’s amended claims, filed 3/26/2026, have been entered. Claims 1-20 were previously canceled. Claim 21 has been amended. Claims 34-40 have been withdrawn from consideration as being directed to a non-elected invention. See 37 CFR 1.142(b) and MPEP § 821.03. Claims 21-40 are currently pending in this application and claims 21-33 have been examined.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
Claim 21:
a communication interface configured to link
The underlined terms above represent generic placeholders that are coupled to functional language using the transition term “configured to”, and do not recite sufficient structure.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof. A review of the specification reveals the following:
[44] Further, while described in the context of functional modules and illustrated using block diagram format, it is to be understood that, unless otherwise stated to the contrary, one or more of the described functions and/or features may be integrated in a single physical device and/or a software module, or one or more functions and/or features may be implemented in separate physical devices or software modules.
Per MPEP 2181(II)(B):
Often the supporting disclosure for a computer-implemented invention discusses the implementation of the functionality of the invention through hardware, software, or a combination of both. In this situation, a question can arise as to which mode of implementation supports the means-plus-function limitation. The language of 35 U.S.C. 112(f) requires that the recited “means” for performing the specified function shall be construed to cover the corresponding “structure or material” described in the specification and equivalents thereof. Therefore, by choosing to use a means-plus-function limitation and invoke 35 U.S.C. 112(f) applicant limits that claim limitation to the disclosed structure, i.e., implementation by hardware or the combination of hardware and software, and equivalents thereof. Therefore, the examiner should not construe the limitation as covering pure software implementation.
In accordance with the above, the limitations indicated above invoke the means-plus-function interpretation and have been interpreted as hardware or the combination of hardware and software, and equivalents thereof. “Hardware” is understood as any type of general-purpose hardware – e.g., processors, memory, or the like (e.g., Fig. 1; ¶23).
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 21-33 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim(s) recite(s) an abstract idea. This judicial exception is not integrated into a practical application. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Under Step 1 of the Alice/Mayo test the claims are directed to statutory categories. Specifically, the system, as claimed in claims 21-26, are directed to a machine, the method, as claimed in claims 27-33, are directed to a process (see MPEP 2106.03).
Under Step 2A (prong 1), claim 21 recites at least the following limitations (emphasis added) that recite an abstract idea:
A visual search system for identifying multiple objects within a crowded scene comprising a plurality of objects, further comprising:
being moved around the crowded scene;
detect a plurality of objects within the video of the crowded scene;
generate and store a list of matched search results for products detected in real-time as the camera is moved around an area of interest;
present the list of matched search results; and
link the list of matched products to an archived version of the captured video.
Claim 27 recites at least the following limitations (emphasis added) that recite an abstract idea:
A method for video-based product identification, the method comprising:
moving around an area of interest;
detecting a plurality of products within the video without requiring a user interaction;
generating a matched search results for the detected products;
associating the matched search results with at least a temporal position within the video;
storing an archived version of the captured video; and
enabling concurrent viewing of the video with display of located products, wherein specific products within a displayed listing are highlighted when a corresponding product appears within the video.
These limitations recite certain methods of organizing human activity, such as performing commercial interactions (see MPEP 2106.04(a)(2)(II)). Certain methods of organizing human activity are defined by MPEP 2106.04 as including “fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions).” In this case, the abstract ideas recited in representative claims 21 and 27 are certain methods of organizing human activity because detecting products and displaying information regarding matched products is a commercial or legal interaction because it is a advertising, marketing or sales activity, or business relations.
Thus, claims 21 and 27 recite an abstract idea.
Under Step 2A (prong 2), if it is determined that the claims recite a judicial exception, it is then necessary to evaluate whether the claims recite additional elements that integrate the judicial exception into a practical application of that exception (see MPEP 2106.04). As stated in the MPEP, when “an additional element merely recites the words ‘apply it (or an equivalent) with the judicial exception, or merely uses a computer as a tool to perform an abstract idea,” the judicial exception has not been integrated into a practical application.
In this case, claim 21 includes additional elements such as (additional elements are bolded):
A visual search system for identifying multiple objects within a crowded scene comprising a plurality of objects, further comprising:
a mobile computing device having a camera configured to capture video in a video capturing mode while being moved around the crowded scene;
a processing unit configured to execute stored instructions to detect a plurality of objects within the video of the crowded scene;
a memory configured to generate and store a list of matched search results for products detected in real-time as the camera is moved around an area of interest;
a display configured to present the list of matched search results; and
a communication interface configured to link the list of matched products to an archived version of the captured video.
Claim 27 includes additional elements such as (additional elements are bolded):
A computer-implemented method for video-based product identification, the method comprising:
capturing a digital video while moving a camera around an area of interest;
detecting a plurality of products within the video without requiring a user interaction with a camera display;
generating a matched search results for the detected products;
associating the matched search results with at least a temporal position within the digital video;
storing an archived version of the captured video; and
enabling concurrent viewing of the video with display of located products, wherein specific products within a displayed listing are highlighted when a corresponding product appears within the video.
Although reciting these additional elements, taken alone or in combination these elements are not sufficient to integrate the abstract idea into a practical application. These additional elements merely amount to the general application of the abstract idea to a technical environment (“a mobile computing device having a camera configured to capture video in a video capturing mode”, “a processing unit configured to execute stored instructions”, “a memory configured to generate and store”, “a display configured to present”, “a communication interface configured to link”, “computer-implemented”, “digital video”) and insignificant pre-and-post solution activity (generating information, storing information, presenting information, linking/associating information). The specification makes clear the general-purpose nature of the technological environment. This is because the additional elements of claims 21 and 27 are recited at a high level of generality (i.e., as generic computing hardware) such that they amount to nothing more than the mere instructions to implement or apply the abstract idea on generic computing hardware (or, merely uses a computer as a tool to perform the abstract idea) (see Fig. 1; paragraphs [20]- [25]). The specification indicates that while exemplary general-purpose systems may be specific for descriptive purposes, any elements capable of implementing the claimed invention are acceptable. That is, the technology used to implement the invention is not specific or integral to the claim. The description demonstrates that these additional elements are merely generic devices such as a generic computer. Further, the additional elements do no more than generally link the use of a judicial exception to a particular environment or field of use (such as the Internet or computing networks).
Therefore, considered both individually and as an ordered pair, the additional elements do no more than generally link the use of the abstract idea to a particular technological environment or field of use. That is, given the generality with which the additional elements are recited, the limitations do not implement the abstract idea with, or use the abstract idea in conjunction with, a particular machine or manufacture that is integral to the claim. Additionally, the claims do not reflect an improvement in the functioning of a computer, or an improvement to other technology or technical field, do not transform or reduction of a particular article to a different state or thing; and do not apply or use the abstract idea in some other meaningful way beyond generally linking the use of the abstract idea to a particular technology environment, such that the claim as a whole is more than a drafting effort designed to monopolize the abstract idea into a practical application, and is therefore “directed to” the abstract idea.
In addition to the above, the recited receiving, storing, displaying, linking/associating and transmitting/communicating steps (even assuming arguendo they do not form part of the abstract idea, which the Examiner does not acquiesce), are at best little more than extra-solution activity (e.g., data gathering, presentation of data) that contributes nominally or insignificantly to the execution of the claimed system (see MPEP 2106.05(g)).
In view of the above, under Step 2A (prong 2), claims 21 and 27 do not integrate the recited exception into a practical application.
Under Step 2B, examiners should evaluate additional elements individually and in combination to determine whether they provide an inventive concept (i.e., whether the additional elements amount to significantly more than the exception itself). In this case, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Returning to representative claims 21 and 27, taken individually or as a whole the additional elements of claims 21 and 27 do not provide an inventive concept (i.e. they do not amount to “significantly more” than the exception itself). As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements used to perform the claimed process amount to no more than the mere instructions to apply the exception using a generic computer and/or no more than a general link to a technological environment.
Furthermore, the additional elements fail to provide significantly more also because the claim simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception. For example, the additional elements of claims 21 and 27 utilize operations the courts have held to be well-understood, routine, and conventional (see: MPEP 2106.05(d)(II)), including at least:
receiving or transmitting data over a network,
storing or retrieving information from memory,
presenting offers
Additionally, the Specification recites capturing video using a camera of a mobile computing device is a well-understood, routine, and conventional activities previously known to the industry (see paragraph 2 and paragraphs 20-25).
Even considered as an ordered combination (as a whole), the additional elements of claims 21 and 27 do not add anything further than when they are considered individually.
In view of the above, representative claims 21 and 27 do not provide an inventive concept (“significantly more”) under Step 2B, and is therefore ineligible for patenting.
Regarding claims 22-25 and 28-33
Dependent claim(s) 22-25 and 28-33, when analyzed as a whole, are held to be patent ineligible under 35 U.S.C. 101 because they do not add “significantly more” to the abstract idea. More specifically, dependent claim(s) 22-25 and 28-33 merely further define the abstract limitations of claim(s) 21 and 27 or provide further embellishments of the limitations recited in independent claim claim(s) 21 and 27.
Claims 22-25 and 28-33 set forth:
enable application of filters to limit the displayed search results, wherein the filters further comprise a product manufactured by a discrete class of manufacturer, an exact match to a product previously purchased by a user, a cost limit, a brand, or a product purchased using a specific cost center.
wherein the area of interest comprises a tool area, and wherein the display is configured to enable a tool check for a sold product status before ordering and a verification that appropriate products were returned as the list of matched search results.
receive a product identifying information prior to a performance of a real-time search and provide an immediate notification when a specified product is detected in the video being captured.
use a set of metadata captured with the video to assist in a product identification, wherein the metadata further comprises at least one of a time, a geolocation, a camera owner identification, an internet protocol (IP) address, or a Media Access Control (MAC) address.
processing the digital video to identify at least a product in a background location within the video that is not in a camera focus, wherein the product in the background photo is also matched.
identifying at least a person within the video to associate the identified person's products of interest with a past purchasing behavior of the person.
wherein the method is performed across a plurality of different product vendors, each one of the plurality of different product vendors selling a plurality of products, the method alerting the user.
wherein detecting the plurality of products comprises processing a set of video frames in near real-time concurrent with the processing of the product in focus.
providing a notification when a pre-specified product is detected, wherein the notification comprises at least one of an audible alert, a tactile alert, or a visual alert.
a time tag attached to each matched search results and highlighting the products in captured video to indicate when a product of interest was detected for later viewing by a user.
Such recitations merely embellish the abstract idea of detecting products and displaying information regarding matched products. The claims do not set forth any further additional limitations, and therefore such abstract embellishments are applied to the additional limitations recited in claim(s) 22-25 and 28-33, which do no more than generally link the use of the abstract idea to a particular technological environment, do not integrate the abstract idea into a practical application, and do not provide an inventive concept. Accordingly, the claims do not confer eligibility on the claimed invention and is ineligible for similar reasons to claim(s) 21 and 27.
Thus, dependent 22-25 and 28-33 are ineligible.
Regarding claim 26
Dependent claim(s) 26 sets forth:
wherein the camera is mounted to at least one mobile robot to capture a set of image data from a hard-to-access location, and wherein the matched search results are provided to a system user on an at least another device for review.
Such recitations merely embellish the abstract idea of detecting products and displaying information regarding matched products. While the claim(s) do set forth the additional elements of “wherein the camera is mounted to at least one mobile robot”, these recitations are similar to the additional limitations in claims 21 and 27, as they do no more than generally link the use of the abstract idea to a particular technological environment. That is these additional elements merely amount to the general application of the abstract idea to a technical environment (“wherein the camera is mounted to at least one mobile robot”). The specification makes clear the general-purpose nature of the technological environment. Paragraph [43] indicates that while exemplary general-purpose systems may be specific for descriptive purposes, any elements capable of implementing the claimed invention are acceptable. That is, the technology used to implement the invention is not specific or integral to the claim. Therefore, these additional elements do not integrate the abstract idea into a practical application because they merely amount to using a computer to apply the abstract idea and no more than a general link of the use of the abstract idea to a particular technological environment or field of use and thus do not act to integrate the abstract idea into a practical application of the abstract idea. Further, the “mobile robot” is recited at a high level and amounts to merely applying the abstract idea.
Additionally, the additional elements do not amount to significantly more because they merely amount to using a computer to apply the abstract idea and amount to no more than a general link of the use of the abstract idea to a particular technological environment.
Thus, dependent claim 26 is also ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 21-25, 27, 28, and 30-33 is/are rejected under 35 U.S.C. 103 as being unpatentable over Grossman (US 2018/0197220 A1 [previously recited]) in view of Wakim (US 9,760,778 B1 [previously recited]).
Regarding claim 21, Grossman discloses a visual search system for identifying multiple objects within a crowded scene comprising a plurality of objects (Figs. 1-3; abstract; ¶0040 [identifying one or more objects within the electronic image and providing information regarding each of the identified objects]), further comprising:
a mobile computing device having a camera configured to capture video in a video capturing mode while being moved around the crowded scene (Figs. 1-3; ¶¶0025-0026 [front-end components 102 allow a user to capture or select images, indicate objects within images, view purchase or other options, and/or purchase products or services via a client computing device 110… The client computing device 110 may include… a camera 115… The camera 115 may be a digital camera configured to capture still and/or video images, which may be stored or displayed by the client computing device 110] and ¶0038 [At block 302, the client computing device 110 may obtain an electronic image to process. The electronic image may be obtained by various means. In some embodiments, the user may take a digital photograph using a camera of the client computing device 110. For example, the user may position the camera to capture an image of an object of interest (e.g., a watch) and control the timing of image capture (e.g., by pressing a physical or virtual button). In some such embodiments, the image may be captured using a special-purpose application operating on the client computing device 110 that is configured to perform part or all of the method 300. Thus, the user may operate the camera from within the special-purpose application to capture the image. In some embodiments, the special-purpose application may cause the client computing device 110 to automatically capture a series of images in order to obtain multiple views of the object.]);
a processing unit configured to execute stored instructions to detect a plurality of objects within the video of the crowded scene (Figs. 1-3, 8; ¶0039 [The image may include a still image or a part of a video], ¶0040 [the method 300 may continue by identifying one or more objects within the electronic image and providing information regarding each of the identified objects], ¶¶0045-0047 [The server 140 may perform the image analysis or may send the image (or portions thereof) to one or more image services 240 for analysis… keywords obtained by analysis of the image…may be used by the server 140 to further search for related products or services… context feature 504 may be used to estimate an approximate size of the object, which may be used to identify or confirm the identification of the object as a basketball jersey… at block 310, the server 140 may search for related products or services similar to or associated with the identified object based upon the keywords. The similar products may be items for sale that are generally of similar types as the identified object (e.g., shirts, jerseys, licensed jerseys for a team). Alternatively, in some embodiments, the similar products may be limited to products that are identical to the identified object (e.g., a home jersey for a particular player on a particular team).] and ¶0100 [implemented using software (code embodied on a non-transitory, tangible machine-readable medium) configuring and controlling computer hardware components]);
a memory configured to generate and store a list of matched search results for products detected around an area of interest (Figs. 1-3, 8; ¶0039-0042 [The image may include a still image or a part of a video…the method 300 may continue by identifying one or more objects within the electronic image and providing information regarding each of the identified objects], ¶0047 [at block 310, the server 140 may search for related products or services similar to or associated with the identified object based upon the keywords. The similar products may be items for sale that are generally of similar types as the identified object (e.g., shirts, jerseys, licensed jerseys for a team). Alternatively, in some embodiments, the similar products may be limited to products that are identical to the identified object (e.g., a home jersey for a particular player on a particular team… results from the search may be stored in a custom data table or list in the database 146 for presentation of information to the user. By generating a custom data table from the search results, purchase options corresponding to results from a plurality of vendors may be seamlessly combined for presentation to the user)]);
a display configured to present the list of matched search results (Figs. 1-3, 8; ¶0026 [The client computing device 110 may include a display component 112… The display component 112 may be a display screen integrated into or connected to the client computing device 110], ¶0040 [the method 300 may continue by identifying one or more objects within the electronic image and providing information regarding each of the identified objects]), ¶¶0047-0049 [results from the search may be stored in a custom data table or list in the database 146 for presentation of information to the user. By generating a custom data table from the search results, purchase options corresponding to results from a plurality of vendors may be seamlessly combined for presentation to the user… At block 314, the client computing device 110 may receive the purchase option data and present related purchase option information to the user. The purchase option information may be presented to the user as a list of purchase options for products or services].
While Grossman discloses storing the search results in a list in the database (¶0047), Grossman does not explicitly disclose storing a list of matched search results for products detected in real-time as the camera is moved around an area of interest and a communication interface configured to link the list of matched products to an archived version of the captured video. However, in the field of object recognition on images in an image series (abstract) Wakim teaches storing a list of matched search results for products detected in real-time as the camera is moved around an area of interest (Figs. 13-15; col. 2, lines 34-53 [The system may match the recognized object to other information sources and retrieve information from those sources to present to the user. This matching may occur substantially in real-time, to provide the user with an experience of having supplemental content/functionality available while engaging with primary content (such as the video feed).], col. 4, lines 25-51 [FIG. 1B illustrates an interface that enables a device, such as a tablet 110 c, to convey which objects in an image have been recognized…The visually enhanced representation may also include displaying a set of virtual “fireflies” or other types of graphical elements over a live view of image (e.g., video) data being acquired by the device or over previously stored or otherwise non-live content to indicate that certain objects in the content have been recognized.], col. 16, line 65 to col. 17, line 29 [object information may be stored for later retrieval], col. 18, lines 5-38 [The server 140 may further include a recognition module 1330 that performs recognition on content captured by or available to the device 110, such as the recognition operations discussed above. For example, the device 110 may send image data (either raw or processed) to the server 140 for the server to perform recognition on the image data. The image data may include single still images, a video feed, or portions of still images or a video feed, such as isolated or processed portions thereof.], col. 20, lines 16-29 [an image and/or video capture component such as camera(s) 1416] in view of col. 1, line 66 to col. 2, line 5 [Certain devices have been configured to recognize objects in an image feed from a device-associated camera, such as a built-in camera in a phone or tablet device]) and a communication interface configured to link the list of matched products to an archived version of the captured video (Figs. 12A-15; col. 16, line 65 to col. 17, line 29 [object information may be stored for later retrieval… When later browsing an object list, the system may redisplay portions of the video signal (including image and/or audio data) from where the object was recognized, thus improving user recall of the object. Thus individual object data may be stored with other information, such as a timestamp (which may relate to a broadcast time, viewing time, or other time), associated program, product/object ID, user ID, video signal information, etc.)], col. 18, lines 22-38 [The server 140 may further include a recognition module 1330 that performs recognition on content captured by or available to the device 110, such as the recognition operations discussed above. For example, the device 110 may send image data (either raw or processed) to the server 140 for the server to perform recognition on the image data. The image data may include single still images, a video feed, or portions of still images or a video feed, such as isolated or processed portions thereof. The recognition module 1330 may include a variety of components, including an image recognition engine 1360, OCR engine 1362, recognition database 1330, or other components such as an audio recognition module (not shown) or the like. The recognition database 1368 may store data used by the various engines/processors of the recognition module 1330 such as image data], and col. 21, lines 53-56). The system of Wakim is applicable to the system of Grossman as they share characteristics and capabilities, namely, they are directed to identifying objects within images. It would have been obvious to one of ordinary skill in the art at the time of filing to modify the identified search results as taught by Grossman with the real-time processing and linking the identified objects with stored video as taught by Wakim. One of ordinary skill in the art at the time of filing would have been motivated to expand the system of Grossman in order to improve user recall of the object when later browsing an object list (col. 17, lines 21-29).
Regarding claim 22, Grossman in view of Wakim teaches the visual search system of claim 21, Grossman further discloses wherein the processing unit is configured to enable application of filters to limit the displayed search results, wherein the filters further comprise a product manufactured by a discrete class of manufacturer, an exact match to a product previously purchased by a user, a cost limit, a brand, or a product purchased using a specific cost center (¶0048 [Depending upon user preferences, the purchase option data may include products generally similar to the identified object in the electronic image or may include only information regarding products identical to the identified object within the electronic image… the purchase option data may be filtered based upon user preferences, user selection of product or service searching, product or service price, vendor characteristics (e.g., location, shipping options, reliability or rating, etc.), or vendor status (e.g., preferred sellers or preferred service providers).], ¶0078 [a subset of the purchase options may be selected for presentation to the user. Seller or service provider identity may be used to select or order the purchase options... information regarding the user or the user's purchase preferences may be used in determining the purchase options to be presented to the user or the order in which to present the purchase options. For example, user brand preferences from previous purchases or user location], ¶0077 [the server 140 may utilize data from previous user purchases or from a user profile to automatically set default values for purchase options, which may be subsequently changed by the user]).
Regarding claim 23, Grossman in view of Wakim teaches the visual search system of claim 21, Grossman further discloses wherein the area of interest comprises a tool area (Figs. 1-10; ¶¶0040-0042 [the user indication may include a selection of an area, such as by a user circling an area or cropping the image to focus on an area. For example, the user may create a rectangular user indication of an area of particular interest within the image by clicking or tapping on a first point, then dragging a pointer to a second point before releasing a mouse button or withdrawing a finger or stylus from a touchscreen display 112]; Examiner notes selection area is comparable to tool area), and wherein the display is configured to enable a tool check for a sold product status before ordering (Figs. 1-5A; ¶0050 [facilitating the purchase may include providing additional information and/or confirming a purchase decision, then process an order by coordinating payment and other aspects of the transaction (e.g., shipping of the product or scheduling of the service). In some embodiments, additional order options associated with the purchase may be presented to the user, and additional data needed for the purchase may be obtained. Such additional order options may depend upon and may be determined based upon a genre of the product or service. For example, clothing size or color options, delivery or installation options, or service scheduling options may be presented to the user. User responses may be received from the client computing device 110 and used by the server 140 to facilitate the purchase]) and a verification that appropriate products were returned as the list of matched search results (Fig. 5A; ¶0046 [alternative keyword phrases may be presented to the user upon request by the user when a previous search failed to produce purchase options desired by the user] and ¶0050 [facilitating the purchase may include providing additional information and/or confirming a purchase decision, then process an order by coordinating payment and other aspects of the transaction (e.g., shipping of the product or scheduling of the service). In some embodiments, additional order options associated with the purchase may be presented to the user, and additional data needed for the purchase may be obtained. Such additional order options may depend upon and may be determined based upon a genre of the product or service. For example, clothing size or color options, delivery or installation options, or service scheduling options may be presented to the user. User responses may be received from the client computing device 110 and used by the server 140 to facilitate the purchase]).
Regarding claim 24, Grossman in view of Wakim teaches the visual search system of claim 21, Grossman further discloses wherein the processing unit is configured to receive a product identifying information prior to a performance of a real-time search and provide an immediate notification when a specified product is detected in the video being captured (Figs. 1-2; ¶0062 [the server 140 may store previously identified labels in the database 146 for future use when an identical image is again indicated by the same or a different user. In such embodiments, the server 140 may calculate a checksum value for each image using an appropriate algorithm, which checksum may be stored with the set of labels associated with the image in the database 146. When an image is obtained upon a user search request, the server 140 may first calculate the checksum of the image using the same algorithm. The newly calculated checksum may then be compared against checksums stored in the database 146 to identify identical images previously analyzed. If an identical image is identified by comparison of checksums, the server 140 may access the associated labels in the database 146, which may be used instead of newly identified labels from the image services 240. Such embodiments may be used to reduce the delay caused by communication with and processing of the image by the image services 240. In further embodiments, keywords identified with an image (as described further below) may similarly be stored in the database 146 and used for later-received identical images, based upon checksum comparison.]).
Regarding claim 25, Grossman in view of Wakim teaches the visual search system of claim 21, Grossman further discloses wherein the processing unit is configured to use a set of metadata captured with the video to assist in a product identification, wherein the metadata further comprises at least one of a time, a geolocation, a camera owner identification, an internet protocol (IP) address, or a Media Access Control (MAC) address (Figs. 1-10; ¶0092 [At block 1006, in some embodiments, the server 140 may further identify one or more locations associated with the image or with the user. Such locations may include the current location of the user, which may also be the location of the image when the image has been recently captured by the user via the camera 115 of the client computing device 110. The user's current location may be determined based upon the location of the client computing device 113. For example, the user's location may be determined by the geolocation unit 113 of the client computing device 110 and transmitted to the sever 140 via the network 130, or the user's location may be determined with lower accuracy based upon the location of a wireless node (e.g., a cell tower) through which the client computing device 110 is connected to the network 130. One or more locations may instead be determined as locations associated with the user based upon additional information regarding the user, such as a user mailing address from prior purchases or from a user profile stored in the database 146. In some embodiments, the user's location may be identified from a location associated with the electronic image, which may include location (e.g., GPS coordinate) metadata associated with an image captured using the camera 115 of the client computing device 110. The location metadata may be automatically determined and associated with the electronic image by the client computing device 110 using the geolocation component 113, such as by determining and adding GPS coordinates as EXIF data to an electronic image file], ¶0026 [The camera 115 may be a digital camera configured to capture still and/or video images, which may be stored or displayed by the client computing device 110], ¶0039 [The image may include a still image or a part of a video], ¶0077 [the server 140 may utilize data from previous user purchases or from a user profile to automatically set default values for purchase options, which may be subsequently changed by the user]).
Regarding claim 27, Grossman discloses a computer-implemented method for video-based product identification (Figs. 1-3, 8; abstract; ¶0040 [identifying one or more objects within the electronic image and providing information regarding each of the identified objects] and ¶0026 [The camera 115 may be a digital camera configured to capture still and/or video images, which may be stored or displayed by the client computing device 110.]), the method comprising:
capturing a digital video while moving a camera around an area of interest (Figs. 1-3; ¶¶0025-0026 [front-end components 102 allow a user to capture or select images, indicate objects within images, view purchase or other options, and/or purchase products or services via a client computing device 110… The client computing device 110 may include… a camera 115… The camera 115 may be a digital camera configured to capture still and/or video images, which may be stored or displayed by the client computing device 110] and ¶0038 [At block 302, the client computing device 110 may obtain an electronic image to process. The electronic image may be obtained by various means. In some embodiments, the user may take a digital photograph using a camera of the client computing device 110. For example, the user may position the camera to capture an image of an object of interest (e.g., a watch) and control the timing of image capture (e.g., by pressing a physical or virtual button). In some such embodiments, the image may be captured using a special-purpose application operating on the client computing device 110 that is configured to perform part or all of the method 300. Thus, the user may operate the camera from within the special-purpose application to capture the image. In some embodiments, the special-purpose application may cause the client computing device 110 to automatically capture a series of images in order to obtain multiple views of the object.]);
detecting a plurality of products within the video without requiring a user interaction with a camera display (Figs. 1-3, 8; ¶0039 [The image may include a still image or a part of a video], ¶0040 [If no user indication is received, the method 300 may continue by identifying one or more objects within the electronic image and providing information regarding each of the identified objects], ¶¶0045-0047 [The server 140 may perform the image analysis or may send the image (or portions thereof) to one or more image services 240 for analysis… keywords obtained by analysis of the image…may be used by the server 140 to further search for related products or services… context feature 504 may be used to estimate an approximate size of the object, which may be used to identify or confirm the identification of the object as a basketball jersey… at block 310, the server 140 may search for related products or services similar to or associated with the identified object based upon the keywords. The similar products may be items for sale that are generally of similar types as the identified object (e.g., shirts, jerseys, licensed jerseys for a team). Alternatively, in some embodiments, the similar products may be limited to products that are identical to the identified object (e.g., a home jersey for a particular player on a particular team).]);
generating a matched search results for the detected products (Figs. 1-3, 8; ¶0039 [The image may include a still image or a part of a video], ¶0040 [the method 300 may continue by identifying one or more objects within the electronic image and providing information regarding each of the identified objects], ¶0047 [at block 310, the server 140 may search for related products or services similar to or associated with the identified object based upon the keywords. The similar products may be items for sale that are generally of similar types as the identified object (e.g., shirts, jerseys, licensed jerseys for a team). Alternatively, in some embodiments, the similar products may be limited to products that are identical to the identified object (e.g., a home jersey for a particular player on a particular team… results from the search may be stored in a custom data table or list in the database 146 for presentation of information to the user. By generating a custom data table from the search results, purchase options corresponding to results from a plurality of vendors may be seamlessly combined for presentation to the user)]);
storing an archived version of the captured video (¶0026 [The camera 115 may be a digital camera configured to capture still and/or video images, which may be stored or displayed by the client computing device 110.]); and
enabling display of located products, wherein specific products within a displayed listing are highlighted when a corresponding product appears within the video (Figs. 1-3, 8; ¶0026 [The client computing device 110 may include a display component 112… The display component 112 may be a display screen integrated into or connected to the client computing device 110], ¶0040 [the method 300 may continue by identifying one or more objects within the electronic image and providing information regarding each of the identified objects]), ¶¶0047-0049 [results from the search may be stored in a custom data table or list in the database 146 for presentation of information to the user. By generating a custom data table from the search results, purchase options corresponding to results from a plurality of vendors may be seamlessly combined for presentation to the user… Depending upon user preferences, the purchase option data may include products generally similar to the identified object in the electronic image or may include only information regarding products identical to the identified object within the electronic image… At block 314, the client computing device 110 may receive the purchase option data and present related purchase option information to the user. The purchase option information may be presented to the user as a list of purchase options for products or services], ¶0060 [the purchase options may be presented to a user to enable user selection and purchase of products or services (block 816)], ¶¶0077-0078, and ¶0090 [From the results of such searching, a list of service recommendations may be generated (block 1010) and presented to the user (block 1012).]; Examiner notes displaying purchase option data of identified products based on user preferences is comparable to wherein specific products within a displayed listing are “highlighted” when a corresponding product appears within the video).
While Grossman discloses storing results from the search in a database (¶0047) and displaying specific products that appear within the video (Figs. 1-3, 8; ¶0026, ¶0040, ¶¶0047-0049, ¶0060, ¶¶0077-0078, and ¶0090), Grossman does not explicitly disclose associating the matched search results with at least a temporal position within the digital video and enabling concurrent viewing of the video with display of located products. However, in the field of object recognition on images in an image series (abstract), Wakim teaches associating the matched search results with at least a temporal position within the digital video (col. 16, line 65 to col. 17, line 29 [object information may be stored for later retrieval… When later browsing an object list, the system may redisplay portions of the video signal (including image and/or audio data) from where the object was recognized, thus improving user recall of the object. Thus individual object data may be stored with other information, such as a timestamp (which may relate to a broadcast time, viewing time, or other time), associated program, product/object ID, user ID, video signal information, etc.)] and Wakim teaches enabling concurrent viewing of the video with display of located products, wherein specific products within a displayed listing are highlighted when a corresponding product appears within the video (col. 4, line 25 to col. 5, line 12 [FIG. 1B illustrates an interface that enables a device, such as a tablet 110 c, to convey which objects in an image have been recognized. This may include displaying a visually enhanced representation of objects that are recognized by the system. The visually enhanced representation may include a highlighting of an object, surrounding it with a border (for example a border that is of a color or texture that makes the object stand out from the remainder of the image. The visually enhanced representation may also include making the object increase and decrease in size (for example, in a pulsing effect). The visually enhanced representation may also include displaying a set of virtual “fireflies” or other types of graphical elements over a live view of image (e.g., video) data being acquired by the device or over previously stored or otherwise non-live content to indicate that certain objects in the content have been recognized… The visually enhanced representation may be achieved by overlaying an image from an image feed with certain visual elements or may be achieved by altering an underlying image. Further, the visually enhanced representation may be presented on a still image (for example, a paused image from a video signal) or on a “moving” image, for example, on multiple images of a continuing video signal.]). The steps of Wakim are applicable to the steps of Grossman as they share characteristics and capabilities, namely, they are directed to identifying objects within images. It would have been obvious to one of ordinary skill in the art at the time of filing to modify the matched results and the display of located products as taught by Grossman with associating the matched results with a temporal position in an archived video and enabling concurrent viewing of the video with display of located products as taught by Wakim. One of ordinary skill in the art at the time of filing would have been motivated to expand the system of Grossman in order to indicate within a live view that certain objects in the content have been recognized (col. 4, line 25 to col. 5, line 12) and improve user recall of the object when later browsing an object list (col. 17, lines 21-29).
Regarding claim 28, Grossman in view of Wakim teaches the computer-implemented method of claim 27, Grossman further discloses further comprising processing the digital video to identify at least a product in a background location within the video that is not in a camera focus, wherein the product in the background photo is also matched (Figs. 1-10; ¶0043 [a background may be identified and separate from a foreground or object, such as by identifying a sky, wall, or objects out of focus], ¶0040 [If no user indication is received, the method 300 may continue by identifying one or more objects within the electronic image and providing information regarding each of the identified objects]).
Regarding claim 30, Grossman in view of Wakim teaches the computer-implemented method of claim 27, Grossman further discloses wherein the method is performed across a plurality of different product vendors, each one of the plurality of different product vendors selling a plurality of products (Figs. 1-3; ¶0004 [The identification of products or services may be used to facilitate electronic commerce transactions by presenting options to a user, which may be purchase options for acquiring the identified products or services. The techniques involve identifying relevant keywords based upon an image. The image may be analyzed using computer image recognition techniques to generate labels associated with the image. In some embodiments, labels associated with objects indicated by the user within the image may be identified. Keywords may then be identified from the labels associated with the image, which keywords may be refined or filtered to identify particularly salient keywords. The keywords may be used to search information associated with vendors, such as web sites associated with product sellers or service providers. Data from sites matching the keywords may be used to generate datasets of information relating to options for acquiring the relevant products or services. Such options may then be presented to the user for selection.], ¶0025 [the server 140 may request or receive information from one or more data sources 170, which may be associated with vendors selling products or services. For example, the data sources 170 may be associated with online retailers selling products associated with an object identified in an electronic image], ¶0029 [The server 140 may further include a database 146, which may be adapted to store data related to the system 100, such as a database associating images of known objects with labels or information regarding vendors], ¶0031 [The one or more data sources 170 may further include databases or servers associated with vendors, providing information regarding products or services (e.g., vendor web sites).], ¶0036, ), the method alerting the user (Figs. 1-3; ¶¶0047-0049 [By generating a custom data table from the search results, purchase options corresponding to results from a plurality of vendors may be seamlessly combined for presentation to the user… Once generated, the server 140 may send the purchase option data to the client computing device 110 via the network 130. At block 314, the client computing device 110 may receive the purchase option data and present related purchase option information to the user. The purchase option information may be presented to the user as a list of purchase options for products or services]).
Regarding claim 31, Grossman in view of Wakim teaches the computer-implemented method of claim 27, Grossman further discloses wherein detecting the plurality of products comprises processing a set of video frames in near real-time concurrent with the processing of the product in focus (Fig. 3; ¶0039 [image may include a still image or a part of a video], ¶0043 [a background may be identified and separate from a foreground or object, such as by identifying a sky, wall, or objects out of focus], ¶0099 [one or more of the individual operations may be performed concurrently]).
Regarding claim 32, Grossman in view of Wakim teaches the computer-implemented method of claim 27. While Grossman further discloses further comprising providing a notification when a product is detected, wherein the notification comprises at least one of an audible alert, a tactile alert, or a visual alert (Figs. 1-3; ¶¶0047-0049 [By generating a custom data table from the search results, purchase options corresponding to results from a plurality of vendors may be seamlessly combined for presentation to the user… Once generated, the server 140 may send the purchase option data to the client computing device 110 via the network 130. At block 314, the client computing device 110 may receive the purchase option data and present related purchase option information to the user. The purchase option information may be presented to the user as a list of purchase options for products or services]), Grossman does not explicitly disclose the product is pre-specified. However, Wakim further teaches providing a notification when a pre-specified product is detected, wherein the notification comprises at least one of an audible alert, a tactile alert, or a visual alert (col. 4, lines 25-51 [FIG. 1B illustrates an interface that enables a device, such as a tablet 110 c, to convey which objects in an image have been recognized…The visually enhanced representation may also include displaying a set of virtual “fireflies” or other types of graphical elements over a live view of image (e.g., video) data being acquired by the device or over previously stored or otherwise non-live content to indicate that certain objects in the content have been recognized.], col. 9, lines 15-20 [if a quick match on the device can identify an object as something already recognized, for which there is data stored on the device, the graphical elements may use a color, shape, and/or size appropriate for that object, for example, and a corresponding ribbon can be presented without need to contact a remote server or service, etc.]). It would have been obvious to one of ordinary skill in the art at the time of filing to modify the notification as taught by Grossman with a notification of previously recognized objects (i.e., pre-specified) as taught by Wakim. One of ordinary skill in the art at the time of filing would have been motivated to expand the steps of Grossman in order to provide a quick match and present the information without a need to contact a remote server or service (col. 4, lines 25-51).
Regarding claim 33, Grossman in view of Wakim teaches the computer-implemented method of claim 27. Wakim further teaches a time tag attached to each matched search results and highlighting the products in captured video to indicate when a product of interest was detected for later viewing by a user (Figs. 12A-15; col. 16, line 65 to col. 17, line 29 [object information may be stored for later retrieval… When later browsing an object list, the system may redisplay portions of the video signal (including image and/or audio data) from where the object was recognized, thus improving user recall of the object. Thus individual object data may be stored with other information, such as a timestamp (which may relate to a broadcast time, viewing time, or other time), associated program, product/object ID, user ID, video signal information, etc.)], col. 18, lines 22-38 [The server 140 may further include a recognition module 1330 that performs recognition on content captured by or available to the device 110, such as the recognition operations discussed above. For example, the device 110 may send image data (either raw or processed) to the server 140 for the server to perform recognition on the image data. The image data may include single still images, a video feed, or portions of still images or a video feed, such as isolated or processed portions thereof. The recognition module 1330 may include a variety of components, including an image recognition engine 1360, OCR engine 1362, recognition database 1330, or other components such as an audio recognition module (not shown) or the like. The recognition database 1368 may store data used by the various engines/processors of the recognition module 1330 such as image data], and col. 21, lines 53-56). The steps of Wakim is applicable to the steps of Grossman as they share characteristics and capabilities, namely, they are directed to identifying objects within images. It would have been obvious to one of ordinary skill in the art at the time of filing to modify the identified search results as taught by Grossman with the time tagging the identified objects with stored video as taught by Wakim. One of ordinary skill in the art at the time of filing would have been motivated to expand the system of Grossman in order to improve user recall of the object when later browsing an object list (col. 17, lines 21-29).
Claim(s) 26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Grossman in view of Wakim and Skidmore (US 2017/0243400 A1 [previously recited]).
Regarding claim 26, Grossman in view of Wakim teaches the visual search system of claim 21. Wakim further teaches wherein the matched search results are provided to a system user on an at least another device for review (Figs. 12A-15; col. 15 line 63 to col. 16, line 64 [The device may indicate (1212) recognized objects on a display of a first device. The system (which may include the device or another component of the system) may receive (1214) an indication to display object information on a second device. The system may determine (1216) a user ID associated with the second device and then, if the user ID matches the user ID associated with the first device, send (1218) the object data to the second device. This process is illustrated in FIGS. 12B-12D.]). It would have been obvious to one of ordinary skill in the art at the time of filing to modify the identified search results as taught by Grossman with providing the results to another device as taught by Wakim. One of ordinary skill in the art at the time of filing would have been motivated to expand the method of Grossman in order to enable display of identified objects on a number of devices and enhance the user experience when performing object recognition (col. 15 line 63 to col. 16, line 64). However, Grossman in view of Wakim do not explicitly teach wherein the camera is mounted to at least one mobile robot to capture a set of image data from a hard-to-access location. However, in the field of augmented reality (abstract), Skidmore teaches a camera mounted to a drone or robot to capture perspectives in the field which a human cannot ordinarily achieve (Fig. 8; ¶0083). The system of Skidmore is applicable to the system of Grossman in view of Wakim as they share characteristics and capabilities, namely, they are directed to capturing image data with a camera. It would have been obvious to one of ordinary skill in the art at the time of filing to modify the camera as taught by Grossman in view of Wakim with the camera mounted to at least one mobile robot as taught by Skidmore. One of ordinary skill in the art at the time of filing would have been motivated to expand the system of Grossman in view of Wakim in order to access a wide variety of spaces, including many spaces where humans cannot themselves go do to size, safety, or practicality constraints because a robotic device is useable to access a confined space such as a pipe, ductwork, or the space above ceiling tiles (¶0083).
Claim(s) 29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Grossman in view of Wakim and Li et al. (US 2019/0205965 A1 [previously recited]).
Regarding claim 29, Grossman in view of Wakim teaches the computer-implemented method of claim 27. While Grossman further discloses further comprising the method identifying at least a person within the video (Figs. 1-10; ¶0024 [the term “object” means any tangible thing that can be identified within an electronic representation of a physical environment. Examples include, without limitation, the following: a vehicle, person, shirt, tree, or building represented within an electronic image file.], ¶0043 [regions may be separately analyzed to identify relevant features, such as objects within the regions… isolating an object may include identifying one or more context features within the image. Such context features may include faces… a face may be identified within the image], ¶0059) and associating products of interest with a past purchasing behavior (¶0077 [the server 140 may utilize data from previous user purchases or from a user profile to automatically set default values for purchase options, which may be subsequently changed by the user], ¶0092), Grossman in view of Wakim does not explicitly teach associating the identified person's products of interest with a past purchasing behavior of the person. However, in the field of recommending items to a customer (abstract), Li et al., hereinafter Li, teaches identifying a user in a captured image and associates products of interest (i.e., recommendations) based on the past purchases of the identified person (Fig. 2; ¶¶0056-0066). The method of Li is applicable to the method of Grossman in view of Wakim as they share characteristics and capabilities, namely, they are directed to capturing image data with a camera. It would have been obvious to one of ordinary skill in the art at the time of filing to modify the object identification as taught by Grossman in view of Wakim with the association of the identified person’s past purchasing behavior as taught by Li. One of ordinary skill in the art at the time of filing would have been motivated to expand the system of Grossman in view of Wakim in order to provide the system actual purchase records of the customer to identify a purchase tendency of the customer while detecting and tracking the customer (¶¶0064-0066).
Response to Arguments
Applicant’s arguments, on pages 18-19 of the Remarks filed 3/26/2026, with respect to the previous 35 USC §112(f) interpretations have been fully considered and are not persuasive. While Applicant notes that the specification provides adequate structural support in paragraphs [20]-[25] for the communication interface which describe the communication components of the mobile computing device, Examiner notes the claims do not recite the communication interface as being part of a computing device. Accordingly the previous 35 USC §112(f) interpretation of the claims are maintained.
Applicant’s arguments, on pages 7-8 of the Remarks filed 3/26/2026, with respect to the previous 35 USC §112(b) rejections have been fully considered and are persuasive in view of the amendments to the claims. Accordingly the previous 35 USC §112(b) rejections are withdrawn.
Applicant’s arguments, on pages 8-12 of the Remarks filed 3/26/2026, with respect to the previous 35 USC §101 rejections have been fully considered but they are not persuasive. Applicant argues the amended claims are directed to an improvement in specific computer vision and video processing technologies. Examiner respectfully disagrees.
Specifically, Applicant argues that the claims are eligible over Step 2A, prong 1 as the limitations describe a technical system that solves concrete problems in computer vision and real-time video processing. Examiner respectfully disagrees.
Applicant is reminded that in Prong One examiner evaluate whether the claim recites a judicial exception, i.e., whether a law of nature, natural phenomenon, or abstract idea is set forth or described in the claim. Despite Applicant’s assertion to the contrary, the Examiner maintains that the claims clearly set forth or describe abstract idea(s) for those reasons set forth previously. Therefore, the instant claims are unlike the claims in Enfish and McRO, as the instant claims recite an abstract idea. Further, detecting products and displaying information regarding matched products is an abstract idea known from the pre-Internet world and the claimed solution is not necessarily rooted in computer technology in order to overcome a problem specifically arising in the realm of computer networks.
Moreover, Applicant’s own assertion that the claims are directed towards “an improved shopping experience for customers by allowing a customer to find a product's replacement (usually an exact match replacement or near exact replacement)” (specification paragraph [38]) is itself an abstract idea and underscores the Examiner’s findings under Prong One. Examiner notes the arguments directed to practical applications and computer improvements are analyzed under Step 2A, Prong Two and not within Step 2A, Prong One.
Accordingly, Examiner maintains the claims recite an abstract idea.
Applicant argues that the amended claim integrates any abstract idea into a practical application. Examiner respectfully disagrees. While the Examiner agrees that the limitations including a mobile computing device, processing unit, memory, display, and communication interface do not fall within the abstract idea, the Examiner disagrees that these elements impose meaningful limits on the judicial exception. As claimed, these elements represent the mere use of generic computing components to facility the abstract idea. Examiner additionally notes some of the arguments on page 9 of the remarks are not claimed. Specifically, the claims do not recite “automatically detecting multiple products in a continuously moving video stream without user interaction.” While claim 21 recites “a mobile device having a camera configured to capture video in a video capturing mode while being moved around the crowded scene”, “a processing unit configured to execute stored instructions to detect a plurality of objects within the video of the scene”, and “a memory configured to generate and store a list of matched search results for products detected in real-time as the camera is moved around an area of interest”, these limitations do not recite “automatically detecting multiple products in a continuously moving video stream without user interaction.” Similar analysis applies to independent claim 27. While claim 27 recites “capturing a digital video while moving a camera around an area of interest”, “detecting a plurality of products within the video without requiring a user interaction with a camera display”, and “generating a matched search results for the detected products”, these limitations do not recite “automatically detecting multiple products in a continuously moving video stream without user interaction.” While claim 27 recites a user does not interact with a camera display, claim 27 does not exclude all user interaction, only interaction with a camera display. Further, the arguments directed to “synchronizing search results with temporal positions in a video so a user can review which products appeared at which moment” are directed to the abstract idea. MPEP 2106.04(d) uses the term additional elements to refer to claim features, limitations, and/or steps that are recited in the claim beyond the identified judicial exception. The above argued limitation is part of the abstract idea of “detecting products and displaying information regarding matched products”, and does not contain any additional elements, such as hardware, beyond the abstract idea itself. Abstract ideas are not patent eligible, therefore this limitation cannot provide integration.
While Applicant argues the claims recite improvements to the technology of video-based object detection and enabling concurrent live and archived video viewing with product highlighting as well as argues the specific combination of technical components in concert are operating multiple specialized technologies in a coordinated, temporally synchronized manner on a live moving video feed, Examiner notes these elements represent the mere use of generic computing components to facility the abstract idea. Notably, the specification provides only a brief description of object detection and enabling concurrent live and archived video viewing with product highlighting (see Fig. 1; paragraphs [20]- [25], [28], [37], and [39]) as well as provides only a brief description of a processing unit executing instructions, a camera, communication interface, and concurrent viewing of video.
If it is asserted that the invention improves upon conventional function of a computer, or upon conventional technology or technological processes, or a specific arrangement of generic additional elements in order to provide a technical improvement, a technical explanation as to how to implement the invention should be present in the specification. That is, the disclosure must provide sufficient details such that one of ordinary skill in the art would recognize the claimed invention as providing an improvement. Although the specification need not explicitly set forth the improvement, it must describe the invention such that the improvement would be apparent to one of ordinary sill in the art. Conversely, if the specification explicitly sets forth an improvement but in a conclusory manner (i.e., a bare assertion of an improvement without the detail necessary to be apparent to a person of ordinary skill in the art), the examiner should not determine the claim improves technology (see MPEP 2106.05(a); MPEP 2106.04(d)(1)).
Applicant’s specification does not provide the requisite detail necessary such that one of ordinary skill in the art could recognize the claimed invention as providing an improvement. Applicant’s specification does not provide sufficient detail with respect to video-based object detection or a specific arrangement of generic additional elements to provide a technical improvement (i.e., a non-conventional and non-generic arrangement of known, conventional pieces), and is specific only in their use in facilitating the abstract idea of detecting products and displaying information regarding matched products.
The manner in which the currently pending claims are written is akin to ineligible decisions such as Affinity Labs of Texas v. DirecTV, LLC (Fed. Cir. 2016) (the court relied on the specification’s failure to provide details regarding the manner in which the invention accomplished the alleged improvement when holding the claimed methods of delivering broadcast content to cellphones ineligible), or, Internet Patents Corp. v. Active Network, Inc. (Fed. Cir. 2015) (claims contained no restriction on the manner in which the additional elements perform these claimed functions). The alleged improvement by Applicant is at best a bare assertion of an improvement sans sufficient detail to demonstrate that Applicant has provided the alleged improvement to the technical field.
While Applicant argues the instant claims are similar to the eligible claims in McRO, Enfish, and Bascom Examiner respectfully disagrees. With respect to McRO, the claimed improvement, as confirmed by the originally filed specification, was “…allowing computers to produce ‘accurate and realistic lip synchronization and facial expressions in animated characters…” and it was “…the incorporation of the claimed rules, not the use of the computer, that “improved [the] existing technological process” by allowing the automation of further tasks.” McRO, Inc. v. Bandai Namco Games America Inc., 837 F3d 1299, (Fed. Cir. 2016). In Enfish, the specification provided teaching that the claimed invention achieves benefits over conventional databases, such as increased flexibility, faster search times, and smaller memory requirements. Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335-36 (Fed. Cir. 2016). In Bascom the inventive concept provided improvement in internet technology by the non-conventional and non-generic arrangement of known, conventional pieces. Specifically, the Federal Circuit identified that the Bascom specification describes how the claims particular arrangement of elements is a technical improvement over prior art ways of filtering such content, and was therefore considered an improvement to an existing technological process.
Here again, the Examiner emphasizes the failure of the disclosure to set forth or describe the amended features, or any improvements that are achieved from or made relative to another technology or technical field. Contrary to Applicant’s assertion, the improvements manifested by the claimed invention are improvements to the abstract idea itself, not the computer or another technology or technical field. Applicant’s own disclosure reveal the impetus is improving the commercial process, not in technology:
[38] In view of foregoing, it will be appreciated that the described systems and methods for providing tap-less, real-time visual search provide, among other things, an improved shopping experience for customers by allowing a customer to find a product's replacement (usually an exact match replacement or near exact replacement) where the only user interaction needed is pointing a camera towards a product of interest. Furthermore, as seen by the sample screen images illustrated in Fig. 6, which depict a computing device 100 being used to execute multiple product searches and to display the corresponding search results, the subject system and method has the advantage of seamlessly providing information about plural objects within a crowded scene simply in response to a customer pointing a camera towards each of the objects in turn.
The character of the claims as a whole is not directed to improving computer performance and do not recite any such benefit. The claims of the instant application, however, merely represent the use of generic computing technology used as a tool to perform the abstract idea in an online environment. The claims lack any restriction on the manner in which the computing operations are to be performed.
In review of the claimed invention, and in consideration of the specification as originally filed, the Examiner asserts that:
(i) the claimed invention does not reflect an improvement in the functioning of a computer, or an improvement to other technology or technical field, but instead improves an abstract, commercial process, and,
(ii) the specification, as originally filed, does not provide sufficient discloser or technical explanation such that one of ordinary skill in the art would have determined that the disclosed invention provided an improvement to the functioning of a computer or another technology or technical field.
Even assuming a relationship of the claimed invention to another technology or technical field, if it is asserted that the invention improves upon conventional functioning of a computer, or upon conventional technology or technological process, a technical explanation as to how to implement the invention should be present in the specification. That is, the disclosure most provide sufficient details such that one of ordinary skill in the art would recognize the claimed invention as providing an improvement (see MPEP 2106.05(a)). Even when a specification explicitly asserts an improvement, examiner should not determine a claim improves technology when only a bare assertion of an improvement is present without the detail necessary to be apparent to a person of ordinary skill in the art (see MPEP 2106.04(d)(1)).
Further, the instant claims are not directed to improving “the existing technological process” requiring the generic components to operate in an unconventional manner to achieve an improvement in computer functionality or requiring the non-conventional and non-generic arrangement of known, conventional pieces to improve a technical process. As currently recited, the instant claims are directed to improving the business task of detecting products and displaying information regarding matched products (i.e., the abstract idea).
Therefore, the instant claims are unlike the claims in McRO, Enfish, and Bascom and the Examiner maintains the claims do not recite additional elements that integrate the judicial exception into a practical application of that exception and maintains the rejection Step 2A, Prong Two.
Applicant argues on pages 11-12 that the amended claims provides an inventive concept under Step 2B. Examiner respectfully disagrees. As noted above in the full rejection of the claims, the claimed additional elements were evaluated individually and in combination to determine whether they provide an inventive concept (i.e., whether the additional elements amount to significantly more than the exception itself). In this case, the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Taken individually or as a whole the additional elements of the claims do not provide an inventive concept (i.e. they do not amount to “significantly more” than the exception itself). As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements used to perform the claimed process amount to no more than the mere instructions to apply the exception using a generic computer and/or no more than a general link to a technological environment. Unlike Bascom, the Specification does not provide support that one of ordinary skill in the art would recognize a technical improvement in the non-conventional and non-generic arrangement of known, conventional pieces. Further, while Applicant contents that the Examiner has failed to provide factual evidence addressing Berkheimer, the Examiner respectfully disagrees. The Examiner points specifically to the previous rejection, as well as that updated above. As referenced to both previously and above, with respect to the claimed limitations, the Examiner underscores that these limitations are being performed by a generic processor and merely confines the use of the abstract idea to a particular technological environment and thus fails to add an inventive concept to the claims. See MPEP 2106.05(h). The background also states that the generic processor performs these limitations at a high level of generality (see Fig. 1; paragraphs [20]- [25], [28], [37], and [39]). This description demonstrates that these additional elements are merely generic devices such as a generic computer.
Further, as referenced to both previously and above, MPEP 2106.05(d)(II) outlines various computer functions that he courts have recognized as well-understood, routine, and conventional functions. Most notably, the Examiner pointed out that the claimed invention utilized receiving or transmitting data over a network, storing and retrieving information in memory, and presenting offers. As these operations have been expressly noted by the courts, the Examiner has indeed provided factual evidence to the Examiner’s findings. Berkheimer is thereby satisfied. As noted above, the Examiner emphasizes the failure of the disclosure to set forth or describe the claimed features as providing an improvement, or an improvement for the specific technical combination, or any improvements that are achieved from or made relative to another technology or technical field. Contrary to Applicant’s assertion, the improvements manifested by the claimed invention are improvements to the abstract idea itself, not the computer or another technology or technical field. Applicant’s own disclosure reveal the impetus is improving the commercial process, not in technology. Further, as noted above, Applicant argues a more narrow interpretation of the limitations than as currently claimed.
Even considered as an ordered combination (as a whole), the additional elements of the claims do not add anything further than when they are considered individually and do not provide an inventive concept (“significantly more”) under Step 2B, and is therefore ineligible for patenting.
Accordingly, the Examiner maintains the 101 rejection of the claims.
Applicant’s arguments, on pages 13-18 of the Remarks filed 3/26/2026, with respect to the 35 USC §103 rejections have been fully considered but are not persuasive.
Specifically, Applicant argues on pages 13-15 that Grossman does not disclose real-time detection from a continuously moving camera without user interaction. In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “real-time detection from a continuously moving camera without user interaction”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). During patent examination, the pending claims must be "given their broadest reasonable interpretation consistent with the specification" (see MPEP 2111). The only exceptions to giving the words in a claim their ordinary and customary meaning in the art are (1) when the applicant acts as his own lexicographer; and (2) when the applicant disavows or disclaims the full scope of a claim term in the specification. To act as his own lexicographer, the applicant must clearly set forth a special definition of a claim term in the specification that differs from the plain and ordinary meaning it would otherwise possess (see MPEP 2111).
As noted above, while claim 21 recites “a mobile device having a camera configured to capture video in a video capturing mode while being moved around the crowded scene”, “a processing unit configured to execute stored instructions to detect a plurality of objects within the video of the scene”, and “a memory configured to generate and store a list of matched search results for products detected in real-time as the camera is moved around an area of interest”, these limitations do not recite “real-time detection from a continuously moving camera without user interaction” requiring “the real-time, continuous, moving camera vide detection” that is argued. Similar analysis applies to independent claim 27. While claim 27 recites “capturing a digital video while moving a camera around an area of interest”, “detecting a plurality of products within the video without requiring a user interaction with a camera display”, and “generating a matched search results for the detected products”, these limitations do not recite “real-time detection from a continuously moving camera without user interaction.” While claim 27 recites a user does not interact with a camera display, claim 27 does not exclude all user interaction, only interaction with a camera display.
Accordingly, Examiner maintains under the broadest reasonable interpretation, Grossman discloses the argued limitations as currently claimed.
Applicant argues on pages 15-16 that Grossman does not disclose linking matched products to an archived version of the captured video. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As noted in the previous rejection of the claims and above in the rejection of the claims, Grossman discloses a mobile computing device having a camera configured to capture video in a video capturing mode while being moved around the crowded scene (Figs. 1-3; ¶¶0025-0026, ¶0038), a processing unit configured to execute stored instructions to detect a plurality of objects within the video of the crowded scene (Figs. 1-3, 8; ¶¶0039-0040, ¶¶0045-0047, and ¶0100), a memory configured to generate and store a list of matched search results for products detected around an area of interest (Figs. 1-3, 8; ¶0039-0042, ¶0047), and a display configured to present the list of matched search results (Figs. 1-3, 8; ¶0026, ¶0040, ¶¶0047-0049). While Grossman does not explicitly disclose storing a list of matched search results for products detected in real-time as the camera is moved around an area of interest and a communication interface configured to link the list of matched products to an archived version of the captured video, Wakim was used to teach storing a list of matched search results for products detected in real-time as the camera is moved around an area of interest (Figs. 13-15; col. 2, lines 34-53, col. 4, lines 25-51, col. 16, line 65 to col. 17, line 29, col. 18, lines 5-38 in view of col. 1, line 66 to col. 2, line 5 [Certain devices have been configured to recognize objects in an image feed from a device-associated camera, such as a built-in camera in a phone or tablet device]) and a communication interface configured to link the list of matched products to an archived version of the captured video (Figs. 12A-15; col. 16, line 65 to col. 17, line 29, col. 18, lines 22-38, and col. 21, lines 53-56). The system of Wakim is applicable to the system of Grossman as they share characteristics and capabilities, namely, they are directed to identifying objects within images. Accordinlgy, Examiner maintains Grossman’s architecture is compatible with Wakim. In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, Examiner cited a portion of Wakim as a motivation to combine. It would have been obvious to one of ordinary skill in the art at the time of filing to modify the identified search results as taught by Grossman with the real-time processing and linking the identified objects with stored video as taught by Wakim. One of ordinary skill in the art at the time of filing would have been motivated to expand the system of Grossman in order to improve user recall of the object when later browsing an object list (Wakim col. 17, lines 21-29).
Accordingly, Examiner maintains the motivation to combine is supported.
Applicant argues on pages 16-17 that the combination fails to teach temporal synchronization of products with video position (claim 27). Examiner respectfully disagrees. In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., “visual feedback during live acquisition of content”) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As noted in the previous rejection of the claims and above in the rejection of the claims, Grossman discloses capturing a video using a camera, detecting a plurality of objects within the video of the crowded scene, generating and storing a list of matched search results for products detected in an area of interest, and presenting the list of matched search results (see claim mapping in claim 27). However, while Grossman discloses storing results from the search in a database (¶0047) and displaying specific products that appear within the video (Figs. 1-3, 8; ¶0026, ¶0040, ¶¶0047-0049, ¶0060, ¶¶0077-0078, and ¶0090), Grossman does not explicitly disclose associating the matched search results with at least a temporal position within the digital video and enabling concurrent viewing of the video with display of located products. Wakim was used to teach associating the matched search results with at least a temporal position within the digital video (col. 16, line 65 to col. 17, line 29) and enabling concurrent viewing of the video with display of located products, wherein specific products within a displayed listing are highlighted when a corresponding product appears within the video (col. 4, line 25 to col. 5, line 12). One of ordinary skill in the art at the time of filing would have been motivated to expand the system of Grossman in order to indicate within a live view that certain objects in the content have been recognized (col. 4, line 25 to col. 5, line 12) and improve user recall of the object when later browsing an object list (col. 17, lines 21-29).
Accordingly, Examiner maintains Grossman in view of Wakin teaches the presently claimed associating the matched search results with at least a temporal position within the digital video and enabling concurrent viewing of the video with display of located products.
Applicant argues on pages 17-18 claims 26 and 29 are not rendered obvious because the base combination of Grossman and Wakim fails to teach independent claims 21 and 27. Examiner respectfully disagrees. For the reasons noted above, Examiner maintains Grossman in view of Wakim teach the limitations of claims 21 and 27.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LINDSEY B SMITH whose telephone number is (571)272-0519. The examiner can normally be reached Monday - Friday 9-6 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jeff Smith can be reached at 571-272-6763. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
LINDSEY B. SMITH
Examiner
Art Unit 3688
/LINDSEY B SMITH/ Examiner, Art Unit 3688
/Jeffrey A. Smith/ Supervisory Patent Examiner, Art Unit 3688
Prosecution Insights